Agile Testing: A Practical Guide for Testers and Agile Teams

One problem we had, though, was getting this same level of clarity for “doneness” at a story level to extend beyond the individual stories. We found that often, when we approached the end of a Sprint or the end game of a release, we would have open expectations of what the team was supposed to accomplish within the sprint. For example, we would deliver stories that were thoroughly tested “in the small”; that is, the functionality of those stories was tested but the stories were not integrated into our staging environment for broader testing. That wasn’t part of our “understanding,” but external stakeholders had that expectation of the teams’ deliverables.

The way the teams solved this problem was to look at our criteria as a multitiered set of guiding goals that wrap each phase, if you will, of agile development. An example of this is shown in Table 20-1.

Table 20-1 Different Levels of Doneness

Defining doneness at these individual levels has proven to work for our teams and has significantly improved our ability to quantify and meet all of our various customer expectations. Keep in mind that there is a connection among all of the criteria, so defining at one level really helps define the others. We often start at the Release Criteria level and work our way “backwards.”

Agile development doesn’t work if stories, iterations, or releases aren’t “done.” “Doneness” includes testing, and testing is often the thing that gets postponed when time is tight. Make sure your success criteria at every level includes all of the necessary testing to guide and validate development.

Each project, each team, each business is unique. Agile teams work with the business experts to decide when they’re ready to deliver software to production. If the release deadline is set in stone, the business will have to modify scope. If there’s enough flexibility to release when the software has enough value, the teams can decide when the quality criteria have been met and the software can go to production.

Challenging Release Candidate Builds

Coni Tartaglia’s team uses a checklist to evaluate each release candidate build. The checklist might specify that the release candidate build:

• Includes all features that provide business value for the release, including artwork, logos, legal agreements, and documentation

• Meets all build acceptance criteria

• Has proof that all agreed-upon tests (acceptance, integration, regression, nonfunctional, UAT) have passed

• Has no open defect reports

Coni’s team challenges the software they might ship with a final set of inspections and agreed-upon “release acceptance tests,” or “RATS.” She explains:

The key phrase is “agreed-upon tests.” By agreeing on the tests in advance, the scope for the release checklist is well defined. Include system-level, end-to-end tests in the RATS, and select from the compatibility roster tests, which will really challenge the release candidate build. Performance tests can also be included in RATs. Agree in advance on the content of the automation suites as well as a subset of manual tests for each RAT.

Agree in advance which tests will be repeated if a RAT succeeds in causing the failure of a release candidate build. If the software has survived several iterations of continuously run automated regression tests, passing these final challenges should be a breeze.

Defining acceptance criteria is ultimately up to the customers. Testers are in a unique position to help the customer and development teams agree on the criteria that optimize product quality.

Traditional software development works in long time frames, with deadlines set far in advance and hurdles to clear from one phase to the next. Agile development lets us produce quality software in small increments and release as necessary. The development and customer teams can work closely to define and decide what to release and when. Testers can play a critical role in this goal-setting process.

Release Management

Many organizations have a release management team, but if you don’t, someone still does the work. Many times in a small organization it is the QA manager who fulfills this role. The person leading the release may hold a release readiness meeting with the stakeholders to evaluate readiness.

A release readiness checklist is a great tool to use to walk through what is important to your team. The intention of this checklist is to help the team objectively determine what was completed and identify the risks associated with not completing a task.

For example, if training is not required because the changes made to the product were transparent to the end user, then the risk is low. However, if there were significant changes to the process for how a new user is created in the system, the risk would be very high to the production support or help teams, and may warrant a delay. The needs of all stakeholders must be considered.