Agile Testing: A Practical Guide for Testers and Agile Teams

In case you decide to use GUI test automation, here’s some advice: Stay away from recorded scripts, invest in maintainability, and minimize the required GUI testing with a good architecture of the application. It is my experience that investing in good GUI test automation practices will always pay off.

Pierre’s advice reflects well how good development practices, especially those followed in agile development projects, apply to automated test development as well as to production code development.

Built-In Record & Playback

Gerard Meszaros, agile coach and author of xUnit Test Patterns [2007], describes a situation where the simplest approach turned out to be record/playback. We’ve mentioned drawbacks to record/playback tools, but if you design your code to support them, they can be the best approach.

I was asked to help a team that was porting a “safety sensitive” application from OS2 to Windows. The business was very concerned about the amount of time it would take it to retest the ported system and the likelihood that the team would miss important bugs. The system was designed to only offer the user valid choices that would not compromise safety. They considered using a test recording tool to record tests on the old system and play them back on the new system, but there were no test recording tools available for both OS2 and Windows that could deal with windows drawn using ASCII characters. After reviewing the architecture of the system, we determined that writing xUnit tests would not be a cost-effective way to test the system because much of the business logic was embedded in the user interface logic, and refactoring the code to separate them would be too risky and time-consuming. Instead, we proposed building a Record & Playback test capability right into the system before we ported it.

Even though the rest of the project was milestone-driven, we developed the built-in test mechanism in a very agile way. Each screen required at least one new hook and sometimes several. We started with the most frequently used screens, adding the necessary hooks to record the user’s actions and the systems responses to them into an XML file. We also added the hooks to play back the XML and determine the test results. Initially, we focused our efforts on proving the concept by hooking only the screens we needed to record and then playing back a simple but realistic test. After everyone was convinced the approach would work, we prioritized the screens with respect to how much benefit it would provide. We implemented the hooks one by one until we could automate a significant portion of the tests. We also built an XSLT stylesheet that would format the XML in a Fit-like way, with green cells indicating acceptable results and red cells indicating a failed test step.

In the meantime, the client was identifying the test scenarios that needed test cases. As we finished enough screens to record a particular test, the client would “acceptance test” our hooks by recording and playing back (still on OS2) the test(s) that were waiting for those hooks. When all of the hooks were in place, we could go ahead and port the code, including the test hooks, from OS2 to Windows. After verifying successful playback on OS2, the client would move the XML test files over to Windows and run them against the ported version of the code. The client found this quite easy to do and was able to record a large number of tests in a relatively short period of time. Because the tests were recording actions and responses in business terms, the tests were fairly easy to understand. The client loved the capability, and still raves about how much effort it saved and how much more confidence it has in the product. “Not only did this save tens of man-years of testing effort, but it even uncovered hidden unknown bugs in the legacy system, which we had considered to be the gold standard.”

In Gerard’s story, the team worked together to retrofit testability onto a system that wasn’t designed for testability. They gave their customers a way to capture their test scenarios on one platform and play them back on both platforms to verify the successful port. This is a stellar example of the whole-team approach. When everyone on the team collaborates on a test automation solution, there’s a much better chance it’s going to succeed.