Agile Testing: A Practical Guide for Testers and Agile Teams

If we know the requirements for performance, security, interaction with other systems, and other nonfunctional attributes before we start coding, it’s easier to design and code with that in mind. Some of these might be more important than actual functionality. For example, if an Internet retail website has a one-minute response time, the customers won’t wait to appreciate the fact that all of the features work properly. Technology-facing tests that critique the product should be considered at every step of the development cycle and not left until the very end. In many cases, such testing should even be done before functional testing.

In recent years we’ve seen many new lightweight tools appropriate to an agile development project become available to support tests. Automation tools can be used to create test data, set up test scenarios for manual testing, drive security tests, and help make sense of results. Automation is mandatory for some efforts such as load and performance testing.

Checking Nonfunctional Requirements

Alessandro Collino, a computer science and information engineer with Onion S.p.A., who works on agile projects, illustrates why executing tests that critique the product early in the development process is critical to project success.

Our Scrum/XP team used TDD to develop a Java application that would convert one form of XML to another. The application performed complex calculations on the data. For each simple story, we wrote a unit test to check the conversion of one element into the required format, implemented the code to make the test pass, and refactored as needed.

We also wrote acceptance tests that read subsets of the original XML files from disk, converted them, and wrote them back. The first time we ran the application on a real file to be converted, we got an out-of-memory error. The DOM parser we used for the XML conversion couldn’t handle such a large file. All of our tests used small subsets of the actual files; we hadn’t thought to write unit tests using large datasets.

Doing TDD gave us quick feedback on whether the code was working per the functional requirements, but the unit tests didn’t test any nonfunctional requirements such as capacity, performance, scalability, and usability. If you use TDD to also check nonfunctional requirements, in this case, capacity, you’ll have quick feedback and be able to avoid expensive mistakes.

Alessandro’s story is a good example of how the quadrant numbering doesn’t imply the order in which tests are done. When application performance is critical, plan to test with production-level loads as soon as testable code is available.

When you and your team plan a new release or project, discuss which types of tests from Quadrants 3 and 4 you need, and when they should be done. Don’t leave essential activities such as load or usability testing to the end, when it might be too late to rectify problems.

Using Tests that Critique the Product

The information produced during testing to review the product should be fed back into the left side of our matrix and used to create new tests to drive future development. For example, if the server fails under a normal load, new stories and tests to drive a more scalable architecture will be needed. Using the quadrants will help you plan tests that critique the product as well as tests that drive development. Think about why you are testing to make sure that the tests are performed at the optimum stage of development.

The short iterations of agile development give your team a chance to learn and experiment with the different testing quadrants. If you find out too late that your design doesn’t scale, start load testing earlier with the next story or project. If the iteration demo reveals that the team misunderstood the customer’s requirements, maybe you’re not doing a good enough job of writing customer tests to guide development. If the team puts off needed refactoring, maybe the unit and component tests aren’t providing enough coverage. Use the agile testing quadrants to help make sure all necessary testing is done at the right time.

Knowing When a Story Is Done

For most products, we need all four categories of testing to feel confident we’re delivering the right value. Not every story requires security testing, but you don’t want to omit it because you didn’t think of it.

Lisa’s Story