Agile Testing: A Practical Guide for Testers and Agile Teams

You need some way to know how much work your team has completed at any point in the iteration and an idea of how much work is left to do. You need to know when it becomes obvious that some stories can’t be completed and the team needs a Plan B. Iteration burndown charts and estimated versus actual time for tasks are examples used to measure team progress. They may or may not provide value for your particular team.

Story or task boards are a good visual way to know the iteration’s status, especially if color coding is used. If too many test task cards are still in the “to do” column or not enough coding task cards have been moved to “Done” or “Tested,” it’s time for the team to think of ways to make sure all of the testing is completed. Maybe some team members need to stop coding and start taking on testing tasks, or maybe one story or a less critical part of a story needs to be put off until the next iteration so that testing for all the other stories can be finished.

This can be accomplished with virtual story boards as well as physical ones. Get creative with your visual effects so that problems are instantly visible. Remember that no story is “done” until it’s tested at all appropriate levels. Teams may have other criteria for when a story is “done,” such as whether it has been peer reviewed or the automated regression tests are completed. On the story board shown in Figure 18-6, the “Done” column for each story row is the rightmost column. The column just to the left of it is the “Verify” column. The story isn’t considered “done” until all the cards, including testing task cards, are in that “Done” column. A glance at the board is enough to know which stories are finished.

Figure 18-6 Story board showing iteration stories and tasks. Used with permission of Mike Thomas. Copyright 2008.

Even teams that don’t track burndown at the task level can do so at the story level. Knowing how much work the team can do each iteration (its velocity) helps with the overall release plan, and the reprioritizing for each iteration. It simply may be enough to know the number of stories completed in an iteration if they tend to average out to the same size. Although plans are tentative at best, it’s helpful to get an idea of about how many stories can be completed by a hard release date or what stories might get done in the upcoming quarter.

Defect Metrics

We talked about defect metrics in Chapter 15, “Tester Activities in Release or Theme Planning” giving you some high level ideas about what to track. Gathering metrics on defects can be very time consuming so always consider the goal before you start to measure. What is the purpose of the metrics you would like to gather? How long will you need to follow the trend before you know if they are useful?

Defect containment is always a favorite metric to capture. When was the defect found? In traditional projects, it is much easier as you have “hard” requirements and coding phases. When the whole team is responsible for quality, and everyone is working together throughout, it is much harder to determine “when” the defect was injected into the system.

We would like to challenge the idea of this type of metric as not necessary in agile development. However, if you find a lot of bugs are slipping through, you may want to start tracking what type of bugs they are so you can address the root cause. For example, if the bugs could have been caught with unit tests, then maybe the programmers need more training on writing unit tests. If the bugs are missed or misunderstood requirements, then maybe not enough time is spent in iteration planning, or acceptance tests aren’t detailed enough.

If you are practicing zero tolerance for defects, then you probably have no need to be tracking defects during coding and testing. A simple card on the story board will give you all the information you need.

Whatever metrics you choose to measure, go for simplicity.

Janet’s Story

In one organization I was with, we tracked the number of defects logged in the DTS over several releases. These were defects that escaped the iteration or were found in the legacy system.Figure 18-7 shows the trend over a year and a half.

Figure 18-7 Sample Defect Trend (but stopped after a while)

At the beginning, the number of issues found right after it was released to QA for final testing was high (33 issues found in one month). The customers found even more issues during UAT which lasted over two months because they were not confident in the quality of the release. In the month that zero defects were reported, we were just starting a new release so there was no new functionality to test. Over the next year, fewer and fewer defects were logged and it becomes impossible to tell where an actual release happened by just looking at the trend.