Agile Testing: A Practical Guide for Testers and Agile Teams

After an automated test for a piece of functionality passes, it must continue to pass until the functionality is intentionally changed. When we plan changes in the application, we change the tests to accommodate them. When an automated test fails unexpectedly, a regression defect may have been introduced by a code change. Running an automated suite of tests every time new code is checked in helps ensure that regression bugs will be caught quickly. Quick feedback means the change is still fresh in some programmer’s mind, so troubleshooting will go more quickly than if the bug weren’t found until some testing phase weeks later. Failing fast means bugs are cheaper to fix.

Automated tests run regularly and often act as your change detector. They allow the team an opportunity to know what has changed since the last build. For example, were there any negative side effects with the last build? If your automation suite has sufficient coverage, it can easily tell far-reaching effects that manual testers can never hope to find.

More often than not, if regression tests are not automated, they won’t get run every iteration, let alone every day. The problem arises very quickly during the end game, when the team needs to complete all of the regression tests. Bugs that would have been caught early are found late in the game. Many of the benefits of testing early are lost.

Tests and Examples that Drive Coding Can Do More

In Chapter 7, “Technology-Facing Tests that Support the Team,” we talked about using tests and examples to drive coding. We’ve talked about how important it is to drive coding with both unit and customer tests. We also want to stress that if these tests are automated, they become valuable for a different reason. They become the base for a very strong regression suite.

Lisa’s Story

After my team got a handle on unit tests, refactoring, continuous integration, and other technology-facing practices, we were able to catch regression bugs and incorrectly implemented functionality during development.

Of course, this didn’t mean our problems were completely solved; we still sometimes missed or misunderstood requirements. However, having an automation framework in place enabled us to start focusing on doing a better job of capturing requirements in up-front tests. We also had more time for exploratory testing. Over time, our defect rate declined dramatically, while our customers’ delight in the delivered business value went up.

—Lisa

TDD and SDD (story test-driven development) keep teams thinking test-first. During planning meetings, they talk about the tests and the best way to do them. They design code to make the tests pass, so testability is never an issue. The automated test suite grows along with the code base, providing a safety net for constant refactoring. It’s important that the whole team practices TDD and consistently writes unit tests, or the safety net will have holes.

The bibliography contains an article by Jennitta Andrea [2008] on team etiquette for TDD.

The team also doesn’t accrue too much technical debt, and their velocity is bound to be stable or even increase over time. That’s one of the reasons why the business managers should be happy to let software teams take the time to implement good practices correctly.

Tests Are Great Documentation

In Part III, we explained how agile teams use examples and tests to guide development. When tests that illustrate examples of desired behavior are automated, they become “living” documentation of how the system actually works. It’s good to have narrative documentation about how a piece of functionality works, but nobody can argue with an executable test that shows in red and green how the code operates on a given set of inputs.

It’s hard to keep static documentation up to date, but if we don’t update our automated tests when the system changes, the tests fail. We need to fix them to keep our build process “green.” This means that automated tests are always an accurate picture of how our code works. That’s just one of the ways our investment in automation pays off.

ROI and Payback

All of the reasons just presented contribute to the bottom line and the payback of automation. Automation provides consistency to a project and gives the team opportunity to test differently and push the limits of the application. Automation means extra time for testers and team members to concentrate on getting the right product out to market in a timely manner.