Agile Testing: A Practical Guide for Testers and Agile Teams

Test automation won’t pay off unless other good development practices are in place. Continuous integration running a robust suite of unit tests is a first step toward automating other tests. Code that’s continually refactored for maintainability and good design will help increase the ROI on automation. Refactoring can’t happen without that good unit test coverage. These development practices also need to be applied to the automated functional test scripts.

Multi-Layered Approach

While we recommend mastering one tool at a time, don’t expect too much out of any one tool. Use the right tool for each need. The tool that works best for unit tests may or may not be appropriate to automate functional tests. GUI, load, performance, and security testing may each require a different tool or tools.

Mike Cohn’s test pyramid concept (see Figure 14-2) has helped our teams put their automation efforts where they do the most good. We want to maximize the tests that have the best ROI. If the system architecture is designed for testability, test automation will be less expensive, especially at the unit level.

Tests that go through the user interface usually have the lowest ROI, because they’re expensive to maintain, but we do need them. They make up the small tip of our pyramid We may choose to automate some of these tests, but the majority of GUI tests are defined in business terms and probably are best left as human interaction tests (i.e., manual tests).

The middle layer represents the functional tests that work directly with production code, without a GUI or other layer in between. While they’re not as inexpensive to automate as unit-level tests, and provide feedback a bit more slowly, the right tools allow them to have a good ROI. The fact that these tests can be written in a language the business experts understand adds to their value.

There are many different layers in the application that can be tested independently. In his book xUnit Test Patterns [2007], Gerard Meszaros refers to this as the Layer Test pattern. He cautions that when trying to test all of the layers of the application separately, we still have to verify that the layers are hooked up correctly, and this may require at least one test of the business logic through the presentation layer.

Lisa’s Story

As my team built our automation framework one step at a time, we gathered an arsenal of tools. After implementing a continuous build framework with Ant and CruiseControl, we mastered JUnit for unit testing. We knew that unit test automation is the quickest and cheapest way to automate, and provides the fastest feedback to the programmers.

Our legacy system had no automated tests, so we built a GUI regression test suite with Canoo WebTest. This provided good payback because the WebTest scripts were specified, not programmed. They were quick to write and easy to maintain.

After JUnit and WebTest were in place, we experimented with FitNesse and found it worked well for functional testing behind the GUI. We found automating with FitNesse to go relatively quickly. Although FitNesse tests are significantly more expensive to produce and maintain than unit tests, their value in driving development and promoting collaboration among customers, programmers, and testers kept the ROI high.

All of these tools were easy to learn, implement, and integrate with the build process, and provided continual feedback about our regression issues. They were important considerations when we were deciding on our test automation strategy.

—Lisa

When evaluating the payback of your automation efforts, consider less tangible aspects such as whether the tool promoted collaboration between the technical and customer teams. A primary reason to write tests is to help guide development. If the process of writing your automated acceptance tests results in a thorough understanding of business requirements, that’s plenty of payback, even if the tests never find a single regression bug later on.

Think about Test Design and Maintenance

Think about all of the manual test scripts you’ve written in your life. Don’t you just wish those all would have been automated? Wouldn’t your life have been a lot easier? We believe that all scripted tests should be automated. Let’s get started converting those manual scripted tests.