Agile Testing: A Practical Guide for Testers and Agile Teams

Coni Tartaglia, software test manager at Primavera Systems, Inc., reflects on what has worked for her team in delivering items that aren’t code but are necessary for a successful release.

Aside from the software, what is the team delivering? It is helpful to have a conversation with the people outside of the development team who may be concerned with this question. Groups such as Legal, Product Marketing, Training, and Customer Support will want to contribute to the list of deliverables.

After there is agreement on what is being delivered, assembly of the components can begin, and the Release Management function can provide confirmation of the delivery through execution of a release checklist. If the release is an update to an existing product, testers can check the deliverables from previous releases to ensure nothing critical is left out of the update package. Deliverables can include legal notices, documentation, translations, and third-party software that are provided as a courtesy to the customers.

Agile teams are delivering value, not just software. We work together with the customer team to improve all aspects of the product.

There are no hard and fast rules to what should be delivered with the product. Think of deliverables as something that adds value to your product. Who should be the recipient of the deliverable, and when does it make the most sense to deliver it?

Releasing the Product

When we talk about releasing the product, we mean making it available to the customer in whatever format that may take. Your organization might have a website that gets updated or a custom application that is delivered to a few large customers. Maybe the product is shrink-wrapped and delivered to millions of PCs around the world, or downloaded off the Internet.

Release Acceptance Criteria

How do you know when you’re done? Acceptance criteria are a traditional way of defining when to accept the product. Performance criteria may have to be met. We capture these for each story at the start of each iteration, and we may also specify them for larger feature sets when we begin a theme or epic. Customers may set quality criteria such as a certain percentage of code covered by automated tests, or that certain tests must pass. Line items such as having zero critical bugs, or zero bugs with serious impact to the system, are often part of the release criteria. The customers need to decide how they’ll know when there’s enough value in the product. Testers can help them define release criteria that accomplish their goals.

Agile teams work to attain the spirit of the quality goals, not just the letter. They don’t downgrade the severity of bugs to medium so they can say they achieved the criterion of no high-severity bugs. Instead, they frequently look at bug trends and think of ways to ensure that high-severity bugs don’t occur in production.

Your quality level should be negotiated with your customer up front so that there are no unpleasant surprises. The acceptance tests your team and your customers defined, using real examples, should serve as milestones for progress toward release. If your customer has a very low tolerance for bugs, and 100% of those acceptance tests must be passing, your iteration velocity should take that into consideration. If new features are more important than bug fixes, well, maybe you will be shipping with bugs.

A Tale of Multitiered “Doneness”

Bob Galen, agile coach and author of Software Endgames, explains how his teams define release acceptance criteria and evaluate whether they’ve been met.

I’ve joined several new agile teams over the past few years, and I’ve seen a common pattern within those teams. My current team does a wonderful job of establishing criteria at a user story or feature level—basically defining acceptance criteria. We’ve worked hard at refining our acceptance criteria. Initially they were developed from the Product Owners’ perspective, and often they were quite ambiguous and ill-defined. The testers decided they could really assist the customers in refining their tests to be much more relevant, clear, and testable. That collaboration proved to be a significant win at the story level, and the Product Owners really valued the engagement and help.

Quite often the testers would also automate the user story acceptance tests, running them during each sprint but also demonstrating overall acceptance during the sprint review.