Agile Testing: A Practical Guide for Testers and Agile Teams

In one team I joined, the customers were very picky. In fact, the pickiest I had ever seen. They always asked for a full week of UAT just to be sure they had the time to test it all. They had prepared test cases and checked them all, including all the content, both in English and in French. Showstopper bugs included spelling errors such as a missing accent in the French content. Over time, as they gained more confidence in our releases and found fewer and fewer errors, they relaxed their demands but still wanted a week, just in case they couldn’t get to it right away. Their business group was very busy.

One release came that pushed the timeline. We were being held to the release date but couldn’t get all the functionality in and leave two weeks for the end game. We talked with the business users and we decided to decrease the end game to one week; the business users would perform their UAT while the project team finished up their system testing and cleanup. The only reason we were able to do this was because of the trust the customer had in our team and the consistency of our releases.

The good news was that, once again, the UAT found no issues that could not wait until the next release.

—Janet

Figure 20-1 shows an example timeline with a normal UAT at the end of the release cycle. The team starts working on the next release, doing release planning, and starts the first iteration with all team members ready to go.

Figure 20-1 Release timeline with UAT

Work with customers so that they understand the process, their role, and what is expected of them. If the UAT is not smooth, then the chances are there will be a high level of support needed. An experienced customer test team may have defined test cases, but most often its testing is ad hoc. Customers may approach their testing as if they were doing their daily job but will probably focus on the new functionality. This is an opportunity to observe how people use the system and to get feedback from them on what works well and what improvements would help them.

Testers can provide support to the customers who are doing the UAT by reviewing tests run and defects logged, and by tracking defects to completion. Both of us have found it helpful to provide customers involved in doing UAT with a report of all of the testing done during development, along with the results. That helps them decide where to focus their own testing.

Alpha/Beta Testing

If you are an organization that distributes software to a large customer base, you may not have a formal UAT. You are much more likely to incorporate alpha or beta testing. Your team will want to get feedback on new features from your real customers, and this is one mechanism for doing so. Alpha testing is early distribution of new versions of software. Because there are likely to be some major bugs, you need to pick your customers wisely. If you choose this method of customer feedback, make sure your customers understand their role. Alpha testing is to get feedback on the features—not to report bugs.

Beta testing is closer to UAT. It is expected that the release is fairly stable and can actually be used. It may not be “ready for prime time” for most customers, but many customers may feel the new features are worth the risk. Customers should understand that it is not a formal release and that you are asking them to test your product and report bugs.

As a tester, it is important to understand how customers view the product, because it may affect how you test. Alpha and beta testing may be the only time you get to interact with end users, so take advantage of the chance to learn how well the product meets their needs.

Post-Development Testing Cycles

If you work in a large organization or are developing a component of a large, complex system, you may need to budget time for testing after development is complete. Sometimes the UAT testing, or the test coordination, isn’t as smooth as it could be, so the timeline stretches out. Test environments that include test versions of all production systems may only be available for small, scheduled windows of time. You may need to coordinate test sessions with teams working on other applications that interact with yours. Whatever the reason, you need extra testing time that does not include the whole development team.

Lisa’s Story

I worked on a team developing components of both internal and external applications for a large telecom client. We could only get access to the complete test environment at scheduled intervals. Releases were also tightly scheduled.

The development team worked in two-week iterations. It could release to the test environment only after every third iteration. At that time, there was a two-week system integration and user acceptance test cycle, followed by the release.