Agile Testing: A Practical Guide for Testers and Agile Teams

Many of our regression tests create their own data on top of the “seed” data. Our Watir tests create all of the data they need and include logic that makes them rerunnable no matter what data is present. For example, the script that tests an employee requesting a loan from his or her retirement plan first cancels any existing loans so a new one can be taken out.

FitNesse tests that test the database layer also create their own data. We use a special schema where we have removed most constraints, so we don’t have to add every column of every table. The tests only add the data that’s pertinent to the functionality being tested. Each test tears down the data it created, so subsequent tests aren’t affected, and each test is independent and rerunnable.

—Lisa

Cleaning up test data is as important as generating it. Your data creation toolkit should include ways to tear down the test data so it doesn’t affect a different test or prevent rerunning the same test.

We’ve looked at major areas where automation is required or at least useful. Our opinion is that whenever you need to do a test or some testing-related activity, first decide whether it can be aided by automation. In some cases, automation won’t be appropriate. Let’s look at some of those.

What Shouldn’t We Automate?

Some testing needs human eyes, ears, and intelligence. Usability and exploratory testing are two that fall into that category. Other tests that may not justify the automation investment are one-off tests and those that will never fail.

Usability Testing

Real usability testing requires someone to actually use the software. Automation might be helpful in setting up scenarios to subsequently examine for usability. Observing users in action, debriefing them on their experiences, and judging the results is a job for a person who understands that usability aspects of software cannot be automated. Logging user actions is helpful for usability testing.

We discuss some logging and monitoring tools in Chapter 10, “Business-Facing Tests that Critique the Product.”

Janet’s Story

We had evaluated several GUI tools but decided to use Ruby with Watir. We kept our tests limited to GUI functions only. One of our tests was checking to make sure that correct validation messages were displaying on the screen. I was running the tests and happened to be watching the screen because I hadn’t seen this particular test that one of the other testers created. My eyes caught something weird, but the test passed, so I replayed it again. One of the programmers had added a “$” to the screen, and the error message was displayed offset because of it. The correct message was displayed, just not in the right place. In this instance, the value in watching the tests run was huge because we were preparing to release fairly soon, and we probably wouldn’t have caught that particular problem.

—Janet

It is possible to automate tests that make sure the GUI never changes, but you need to ask yourself whether it’s worth the cost. Do you really care that a button has changed positions by one pixel? Do the results justify the effort? We don’t think you should automate “look and feel” testing, because an automated script can only look for what you tell it to see. Automation would miss visual problems that would jump out at a human.

Exploratory Testing

Similarly, exploratory testing may be speeded up with scripts to create test data and jump through some setup steps, but it requires a skilled tester to design and execute the tests. One major goal of exploratory testing is to learn more about the product by doing, and then use that information to improve future development. Automated scripts won’t do that for you. However, as we’ve said before, you won’t have time for exploratory testing without a lot of other automated tests.

See Chapter 10, “Business-Facing Tests that Critique the Product,” for more on exploratory testing and tools that can facilitate it.

Tests that Will Never Fail

We’ve heard an argument that tests that will never fail don’t need to be automated. If a requirement is so obvious that there’s only one way to implement it, and no programmer will ever look at that code later without knowing exactly what it should do, the chances of someone introducing a defect in that code are next to nothing. Let’s say we have a form with address fields. Do we need an automated regression test to verify that the second street address line is not required? After we’ve verified it manually, how likely is it that someone will accidentally change it to a required field later? Even if someone did, it wouldn’t be a catastrophic event. Someone else would notice it and people could work around it easily until it was fixed.