Agile Testing: A Practical Guide for Testers and Agile Teams

Janet’s Story

In one of the organizations I was working with, we used two different baseline test data schemes. For our individual test environments, we used Fit fixtures to load predefined data. We tried to make this data as close to production as possible, but we also seeded it with some very specific test data. Every time we checked out a new version of code, we were able to reload a base set of data. In this way, we also tested the database schema as well to see if anything had changed.

For our more stable test environment where we wanted data persisted, we used the data migration scripts that the programmers developed as they made database changes. These migration scripts were eventually used for the initial cut over from production and by then we were pretty certain they were correct.

—Janet

Enlist your customers’ support in obtaining meaningful test data. If you’re working on a story that involves sending a file to a third-party vendor, your business expert can find out what data the vendor expects in the file. Lisa’s team developed features to allow retirement plan brokers to offer their customers portfolios of mutual funds. They asked the product owner to provide samples of portfolios, including a name, description, and set of funds for each. This helped them test with realistic data.

Test data tends to get stale and out of date over time. Older data, even if it came from production, may no longer accurately reflect current production data. A “passing” test using data that’s no longer valid gives a misleading sense of confidence. Continually review your test data needs. Refresh data or create it using a new approach, as needed.

Test data requirements vary according to the type of testing. Regression tests can usually create their own data or run against a small representational set of data that can be refreshed to a known state quickly. Exploratory testing may need a complete replica of production type data.

Chapter 14, “An Agile Test Automation Strategy,” explores different approaches to obtaining test data

Test Results

Different teams have different requirements for test result reporting. Think about how you are going to report test results at this stage of the game so that you can do so effectively when the time comes to do the actual reporting. Your organization may have audit compliance requirements, or maybe your customer just wants to know how you tested. Understand your needs so that you can choose the approach that is right for your team.

There are many ways to report test results. There are vendor tools that will record both automated and manual results. Your team may find a way to persist the results from tools such as Fit, or you may just choose to keep a big visible manual chart.

The approach that a few teams have taken is to create home-grown test result applications. For example, a simple Ruby application written with Ruby on Rails for the database or a MySQL database with a PHP front end can make a very simple but easy-to-use test management system.

A tool such as this can be very simple or can include added complexity such as the capability to categorize your tests. The important thing is the test results. If your automated tests record their pass or fail result along with the error, you have some history to help determine fragility of the test.

Your team can configure your automated build process to provide test results from each build, by email, or a feedback utility or web interface that team members can view online. Results over time can be summarized in a variety of formats that make progress visible. One of Lisa’s teams produced a daily graph of tests written, run, and passing that was posted in the team’s work area. Another produced a daily calendar with the number of unit tests passing every day. Even simple visual results are effective.

We talk about some of the metrics you can use later in this chapter.

Test Plan Alternatives

We’ve talked about why to test plan and what you should consider. Now we talk about some of the alternatives to the heavy test plans you may be used to. Whatever type of test plan your organization uses, make it yours. Use it in a way that benefits your team, and make sure you meet your customer’s needs. As with any document your team produces, it should fulfill a purpose.

Lightweight Test Plans

If your organization or customer insists on a test plan for SOX compliance or other regulatory needs, consider a lightweight test plan that covers the necessities but not any extras. Do not repeat items that have already been included in the Project Plan or Project Charter. A sample Test Plan might look something like the one shown in Figure 15-6.

Figure 15-6 Sample Test Plan