Agile Testing: A Practical Guide for Testers and Agile Teams

After the team introduced agile development, Janet went to France to facilitate the first UAT at the customer site. It worked fairly well, and the release was accepted after a few critical issues were fixed. The team learned a lot from that experience.

The second UAT sign-off was done in-house. To prepare, the team worked with the customer to develop a set of tests the customer could perform to verify new functionality. The customer was able to test the application throughout the development cycle, so UAT didn’t produce any issues. The customer came, ran through the tests, and signed off in a day.

We cannot stress the importance of working with the customer enough. Even though the product engineer was the proxy for the customer, it was crucial to get face time with the actual customer. The relationship that had been built over time was critical to the success of the project. Janet strongly believes that the UAT succeeded because the customer knew what the team was doing along the way.

Reliability

Reliability, one of the “ilities” addressed by Quadrant 4 tests, was a critical factor of the system because it was monitoring remote sites that were often inaccessible, especially in winter. The simulator that was developed for testing the embedded system was set up on a separate environment, and was run for weeks at a time measuring stability (yet another “ility”) of the whole system. Corrections to the system design could be planned and coded as needed. This is a good example of why you shouldn’t wait until the end of the project to do the technology-facing tests that critique the product.

See Chapter 10, “Business-Facing Tests that Critique the Product,” for more about Quadrant 4 tests such as reliability testing.

Documentation

The approach taken to documentation is presented in this section.

Documenting the Test Code

During development, it became clear that a formal documentation system was needed for the test code. The simplest solution was to use RDoc, similar to Javadoc, but for Ruby. RDoc extracted tagged comments from the source code and generated web pages with details of files, classes, and methods. The documents were generated every night using a batch file and were available to the complete team. It was easy to find what test fixtures were created.

The documentation of the test code helped to document the tests and make it easier to find what we were testing and what the tests did. It was very powerful and easy to use.

Reporting the Test Results

Although comprehensive testing was being performed, there was little evidence of this outside of the test team. The logs generated during automated tests provided good information to track down problems but were not suitable for a wider audience.

Chapter 16, “Hit the Ground Running,” gives more examples of ways teams report test results.

To raise the visibility of the tests being performed, the test team developed a logging and reporting system using Apache, PHP, and mySQL. When a test ran, it logged the result into the database. A web front end allowed project stakeholders to determine what tests were run, the pass/failure rate, and other information.

Chapter 18, “Coding and Testing,” also discusses uses of big visible charts.

We also believed in making our progress visible (good or bad) as much as possible. To this end we created charts and graphs along the way and posted them in common areas. Figure 12-4 shows some of the charts we created.

Figure 12-4 Big visible charts used by the remote monitoring system project team

Using the Agile Testing Quadrants

This example demonstrates how testing practices from all four agile testing quadrants are combined during the life of a complex development project to achieve successful delivery. The experience of this team illustrates many of the principles we have been emphasizing. The whole team, including programmers, testers, customer proxy, and the actual customer, contributed to efforts to solve automation problems. They experimented with different approaches. They combined their homegrown and open source tools in different ways to perform testing at all levels, from the unit level to end-to-end system testing and UAT. The success of the project demonstrates the success of the testing approach.

As you plan each epic, release, or iteration, work with your customer team to understand the business priorities and analyze risks. Use the quadrants to help identify all of the different types of testing that will be needed and when they should be performed. Is performance the most important criteria? Is the highest priority the ability to interface with other systems? Is usability perhaps the most important aspect?