Agile Testing: A Practical Guide for Testers and Agile Teams

As your team sets goals related to defects, use appropriate metrics to measure progress toward those goals. There are trends that you will want to monitor for the whole release, and there are ones that are iteration-specific. For example, if you’re trying to achieve zero defects, you may want to track the open bugs at the end of each iteration, or how many bugs were found after development but before release. Most of us are interested in knowing how many defects have been reported after the code is in production, which is something completely different altogether. These issues will tell you after the fact how well your team did on the last release, but not how well you are doing on the current release. They may give you some indication of what processes you need to change to reduce the number of defects. Lisa’s team is more concerned with production defects found in the “new” code that was rewritten in the new architecture. They’re working hard to produce this new code with zero defects, so they need to know how well they’re doing. They expect that bugs will be found fairly often in the legacy system, where only the most critical functionality is covered by automated GUI smoke tests, and there are few automated unit and behind-the-GUI tests.

Knowing the defect rate of legacy code might be good justification for refactoring or rewriting it, but the team’s top priority is doing a good job with the new code, so they group bugs by “new” and “old” code, and focus on the “new” bugs.

Make sure your bug database can track what you want to measure. You may have to make some changes in both the database and your process to get the data you need. For example, if you want to measure how many defects were found in production after a release, you have to make sure you have environment and version as mandatory fields, or make sure that people who enter bugs always fill them in.

More on defect tracking systems can be found in Chapter 5, “Transitioning Traditional Processes.”

Because defect tracking systems are often used for purposes besides tracking bugs, be sure not to muddle the numbers. A request for a manual update to the database doesn’t necessarily reflect an issue with the existing code. Use your defect tracking tool properly to ensure that your metrics are meaningful.

Lisa’s Story

Periodically evaluate the metrics you’re reporting and see if they’re still relevant. Figure 15-14 shows two defect reports that Lisa’s team used for years. When we first transitioned to agile, managers and others looked at these reports to see the progress that resulted from the new process. Four years later, our ScrumMaster found that nobody was reading these reports anymore, so we quit producing them. By that time, rates of new defects had reduced dramatically, and nobody really cared about the old defects still hanging about in the legacy code.

—Lisa

Figure 15-14 Sample defect reports used (and no longer used) by Lisa’s team

Release planning is a good time to evaluate the ROI of the metrics you’ve been tracking. How much effort are you spending to gather and report the metrics? Do they tell you what you need to know? Does the code you release meet your team’s standards for internal quality? Is the code coverage percentage going up? Is the team meeting its goals for reducing the number of defects that get out to production? If not, was there a good reason?

Metrics are just one piece of the puzzle. Use your release, theme, or project planning meetings to refocus on delivering business value when the business needs it. Take some time to learn about the features you’re about to develop. Don’t get caught up with committing to your plans—the situation is bound to change. Instead, prepare for doing the right activities and getting the right resources in time to meet the customers’ priorities.

Summary

As your team puts together its plan for a new theme or release, keep the main points of this chapter in mind.

When sizing a story, consider different viewpoints, including business value, risk, technical implementation, and how the feature will be used. Ask clarifying questions, but don’t get bogged down in details.

Testers can help identify the “thin slice” or “critical path” through a feature set to help prioritize stories. Schedule high-risk stories early if they might require extra testing early.

The size of testing effort for a story helps determine whether that story is in scope for the release.

Testers can help the team think about how new stories will impact the larger system.

Plan for extra testing time and resources when features may affect systems or subsystems developed by outside teams.

As the team identifies the scope of the release, evaluate the scope of testing and budget enough time and resources for it.

Spend some time during release planning to address infrastructure, test environment, and test data concerns.