Agile Testing: A Practical Guide for Testers and Agile Teams

An important component of test automation payback is the way defects are fixed. Teams that rely on manual tests tend to find bugs long after the code containing the bug is written. They get into the mode of fixing the “bug of the day,” instead of looking at the root cause of the bug and redesigning the code accordingly. When programmers run the automated test suite in their own sandbox, the automated regression tests find the bugs before the code is checked in, so there’s time to correct the design. That’s a much bigger payback, and it’s how you reduce technical debt and develop solid code.

Barriers to Automation—Things that Get in the Way

Back in 2001, Bret Pettichord [2001] listed seven problems that plague automation. They are still applicable, but are intended for teams that do not incorporate automation as part of their development. And of course, because you are doing agile, you are doing that, right?

We would like to think that everyone has included automation tasks as part of each story, but the reality is that you probably wouldn’t be reading this section if you had it all under control. We’ve included Bret’s list to show what problems you probably have if you don’t include automation as part of the everyday project deliverables.

Bret’s List

Bret’s list of automation problems looks like this:

Only using spare time for test automation doesn’t give it the focus it needs.

There is a lack of clear goals.

There is a lack of experience.

There is high turnover, because you lose any experience you may have.

A reaction to desperation is often the reason why automation is chosen, in which case it can be more of a wish than a realistic proposal.

There can be a reluctance to think about testing; the fun is in the automating, not in the testing.

Focusing on solving the technology problem can cause you to lose sight of whether the result meets the testing need.

We think there are some other problems that teams run into when trying to automate. Even if we do try to include automation in our project deliverables, there are other barriers to success. In the next section, we present our list of obstacles to successful test automation.

Our List

Our list of barriers to successful test automation is based on the experiences we’ve had with our own agile teams as well as that of the other teams we know.

Programmers’ attitude

The “Hump of Pain”

Initial investment

Code that’s always in flux

Legacy systems

Fear

Old habits

Programmers’ Attitude—“Why Automate?”

Programmers who are used to working in a traditional environment, where some separate, unseen QA team does all of the testing, may not even give functional test automation a lot of thought. Some programmers don’t bother to test much because they have the QA team as a safety net to catch bugs before release. Long waterfall development cycles make testing even more remote to programmers. By the time the unseen testers are doing their job, the programmers have moved on to the next release. Defects go into a queue to be fixed later at great expense, and nobody is accountable for having produced them. Even programmers who have adopted test-driven development and are used to automating tests at the unit level may not think about how acceptance tests beyond the unit level get done.

Lisa’s Story

I once joined an XP team of skilled programmers practicing test-driven development that had a reasonable suite of unit tests running in an automated build process. They had never automated any business-facing tests, so one day I started a discussion about what tools they might use to automate functional business-facing regression tests. The programmers wanted to know why we needed to automate these tests.

At the end of the first iteration, when everyone was executing the acceptance tests by hand, I pointed out that there would be all these tests to do again in the next iteration as regression tests, in addition to the tests for all of the new stories. In the third iteration, there would be three times as many tests. To a tester, it seems ridiculously obvious, but sometimes programmers need to do the manual tests before they understand the compulsion to automate them.

—Lisa

Education is the key to getting programmers and the rest of the team to understand the importance of automation.

The “Hump of Pain” (The Learning Curve)

It’s hard to learn test automation, especially to learn how to do it in a way that produces a good return on the resources invested in it. A term we’ve heard Brian Marick use to describe the initial phase of automation that developers (including testers) have to overcome is the “hump of pain” (see Figure 13-1). This phrase refers to the struggle that most teams go through when adopting automation.

Figure 13-1 Hump of pain of the automation learning curve