Agile Testing: A Practical Guide for Testers and Agile Teams

Here’s an example where that comes in handy. We just found a problem in the code under development. Is it a new problem, or has it been lurking in the code for a while and somehow missed by the test? We can deploy the tag that’s in production, try to reproduce the problem, and investigate why the tests didn’t catch it. Lisa’s team recently had a situation where the regression suite missed a bug because a database constraint was missing in the test schema. That kind of problem is hard to pinpoint if you aren’t tying your test code versions to your production code versions.

Organizing Tests

Many tools come with their own means of organization. For example, FitNesse comes with its own wiki, with a hierarchical organization, and built-in version control. As of this writing, FitNesse is starting to provide support for source code control tools such as Subversion. Scripts written in other test tools, such as Watir and Canoo WebTest, can and should be maintained within the same source code control system as production code, just as the unit tests are.

Organizing Tests with the Project under Test

We asked some agile testing experts how they manage tests. Dierk König, founder and project manager of Canoo WebTest, explained how his teams have managed their automated tests to satisfy the needs of both the development and customer teams.

We always organize our tests alongside the project under test. That is, test sources are stored together with the project sources in the exact same repository, using the same mechanisms for revision control, tagging, and sharing the test base.

WebTest comes with a standard layout of how to organize tests and test data in directories. You can adapt this to any structure you fancy, but the “convention over configuration” shows its strength here. In large projects, every sub-project maintains its own test base in a “webtest” subdirectory that follows the convention.

Whenever a client did not follow this approach, the experience was very painful for all people involved. We have seen huge databases of test descriptions that did not even feature a proper revision control (i.e., where you could, for example, see diffs to old releases or who changed which test for what reason).

Keep in mind that tests are made up from modules so that you can eliminate duplication of test code; otherwise the maintenance will kill you. And before changing any module, you need to know where it is used.

In short: Make sure the master of your tests and your test data is in a text format that is versioned together with your code under test.

Nontechnical personnel (for example, management, QA) may require more high-level information about test coverage, latest test results, or even means of triggering a test run. Don’t let these valid requirements undermine the engineering approach to test automation. Instead, write little tools, for example, web-based report applications, that address these needs.

The ability of customers to access information about tests is as important as the ability to keep test and production code coordinated. As Dierk pointed out, you might not be able to do all this with the same tool.

Test management helps your team answer questions such as the following:

Which test cases have been automated?

Which still need automating?

Which tests are currently running as part of a regression suite?

Which tests cover what functional areas?

How is feature XYZ designed to work?

Who wrote this test case? When? Who changed it last?

How long has this test been part of the regression suite?

Because one of the primary reasons we write tests is to guide development, we need to organize tests so that everyone on the team can find the appropriate tests for each story and easily identify what functionality the tests cover. Because we use tests as documentation, it’s critical that anyone on either the development or customer team can find a particular test quickly when there’s a question about how the system should behave. We might need multiple tools to satisfy different test management goals.

It’s easy to lose control of test scripts. When a test fails, you need to pinpoint the problem quickly. You may need to know what changes have been made recently to the test script, which is easy with the history available in a source code control system. Your customer team also needs a way to keep track of project progress, to understand how much of the code is covered with tests, and possibly to run tests themselves. Test management systems, like the tests themselves, should promote communication and collaboration among team members and between different teams.

Test Transparency

Declan Whelan, a software developer and agile coach, uses a test management approach designed to keep tests visible to testers, developers, managers, and other teams.