Agile Testing: A Practical Guide for Testers and Agile Teams

Use this end-game time to do some final exploratory testing. Step back and look at the whole system and do some end-to-end scenarios. Such testing will confirm that the application is working correctly, give you added confidence in the product, and provide information for the next iteration or release.

Testing the Release Candidate

We recommend that the automated regression testing be done against every release candidate. If you’re following our recommendation to run automated regression tests continually on each new build, or at least daily, you’ve already done this. If some of your regression tests are manual, you’ll need to plan time for those or they might not get done. A risk assessment based on changes made to each build will determine what tests need to be run if there is more than one release candidate.

Test on a Staging Environment

Whether you are using traditional or agile development processes, a staging environment that mimics production is vital for final testing before release, as well as for testing the release process itself. As part of the end game, your application should be deployed to staging just like you would deploy it to production, or as your customers would on their environments. In many organizations that Janet has seen, the staging environment is usually shared among multiple projects, and the deployment must be scheduled as part of the release planning. Consider ahead of time how to handle dependencies, integrating with other teams using the staging environment, and working with external third parties. It might feel like “traditional” test planning, but you might be dealing with teams that haven’t embraced agile development.

Although agile promotes continuous integration, it is often difficult to integrate with third-party products or other applications outside your project’s control. Staging environments can have better controls so that external applications may connect and have access to third-party test environments. Staging environments can also be used for load and performance testing, mock deploys, fail-over testing, and manual regression tests and exploratory functional testing. There are always configuration differences between environments so your staging environment is a good place to test these.

Final Nonfunctional Testing

Load testing should be scheduled throughout the project on specific pieces of the application that you are developing. If your staging environment is in high demand, you may not be able to do full system load testing until the end game.

By this time, you should be able to do long-running reliability tests on all product functionality. Check for crashes and degradation of performance with normal load. When done at release time, it should be a final confirmation only.

Fault tolerance and recovery testing is best done on your staging environment as well, because test environments usually don’t have the necessary setup. For these same reasons, you may only be able to test certain aspects of security. One example is https, a secure http connection through encrypted secure sockets. Some organizations may choose to have the necessary certificates on their staging environment only. Other examples are clustering or data replication. Make sure you involve all parties who need to be included in this testing.

Integration with External Applications

Your team may be agile, but other product teams in your organization, or third parties your team works with, may not be.

Janet’s Story

In one organization that I worked with, the third-party partner that approved credit cards had a test account that could be used, but it was only accessible from the staging environment.

To test during development, test stubs were created to return specified results depending on the credit card number used. However, this wasn’t sufficient because the third party sometimes changed functionality on its end that we weren’t aware of. Testing with the actual third party was critical to the success of the project, and it is a key part of the end game.

—Janet

Coordinate well in advance with other product teams or outside partners that have products that need to integrate with your product. If you have identified these risks early and done as much up-front testing as possible, the testing done during the end game should be final verification only. However, there are always last-minute surprises, so you may need to be prepared to make changes to your application.

Tools like simulators and mock objects used for testing during development can help alleviate some of the risks, but the sooner you can test with external applications, the lower the risk.

Data Conversion and Database Updates