Agile Testing: A Practical Guide for Testers and Agile Teams

The most basic reason a team wants to automate is that it simply takes too long to complete all of the necessary testing manually. As your application gets bigger and bigger, the time to test everything grows longer and longer, sometimes exponentially, depending on the complexity of the AUT (Application under test).

Agile teams are able to deliver production-ready software at the end of each short iteration by having production-ready software every day. Running a full suite of passing regression tests at least daily is an indispensable practice, and you can’t do it with manual regression testing. If you don’t have any automation now, you’ll have to regression test manually, but don’t let that stop you from starting to automate it.

If you execute your regression testing manually, it takes more and more time testing every day, every iteration. In order for testing to keep pace with coding, either the programmers have to take time to help with manual regression testing, or the team has to hire more testers. Inevitably, both technical debt and frustration will grow.

If the code doesn’t even have to pass a suite of automated unit level regression tests, the testers will probably spend much of their time researching, trying to reproduce and report those simple bugs, and less time finding potentially serious system level bugs. In addition, because the team isn’t doing test-first development, code design is more likely to be less testable and may not provide the functionality desired by the business.

Manually testing a number of different scenarios can take a lot of time, especially if you’re keying inputs into a user interface. Setting up data for a variety of complex scenarios can be an overwhelming task if you have no automated way to speed it up. As a result, only a limited number of scenarios may be tested, and important defects can be missed.

Manual Processes Are Error Prone

Manual testing gets repetitive, especially if you’re following scripted tests, and manual tests get boring very quickly. It’s way too easy to make mistakes and overlook even simple bugs. Steps and even entire tests will be skipped. If the team’s facing a tight deadline, there’s a temptation to cut corners, and the result is a missed problem.

Because manual testing is slow, you might still be testing at midnight on the last day of the iteration. How many bugs will you notice then?

Automated builds, deployment, version control, and monitoring also go a long way toward mitigating risk and making your development process more consistent. Automating these scripted tests eliminate the possibility of errors, because each test is done exactly the same way every time.

The adage of “build once, deploy to many” is a tester’s dream come true. Automation of the build and deploy processes allow you to know exactly what you are testing on any given environment.

Automation Frees People to Do Their Best Work

Writing code test-first helps programmers understand requirements and design code accordingly. Having continual builds run all of the unit tests and the functional regression tests means more time to do interesting exploratory testing. Automating the setup for exploratory tests means even more time to probe into potentially weak parts of the system. Because you didn’t spend time executing tedious manual scripts, you have the energy to do a good job, thinking of different scenarios and learning more about how the application works.

If we’re thinking constantly about how to automate tests for a fix or new feature, we’re more likely to think of testability and a quality design rather than a quick hack that might prove fragile. That means better code and better tests.

Automating tests can actually help with consistency across the application.

Janet’s Story

Jason (one of my fellow testers) and I were working on some GUI automation scripts using Ruby and Watir, and were adding constants for button names for the tests. We quickly realized that the buttons on each page were not consistently named. We were able to get them changed and resolved those consistency issues very quickly, and had an easy way to enforce the naming conventions.

—Janet

See Chapter 9, “Toolkit for Business-Facing Tests that Support the Team,” Chapter 12, “Summary of Testing Quadrants,” and Chapter 14, “An Agile Test Automation Strategy,” for more information about Ruby and Watir.

Books such as Pragmatic Project Automation [2004] can guide you in automating daily development chores and free your team for important activities such as exploratory testing.

Giving Testers Better Work

Chris McMahon described the benefits he’s experienced due to regression test automation in a posting to the agile-testing mailing list in November 2007: