Agile Testing: A Practical Guide for Testers and Agile Teams

A test plan should not cover every eventuality or every story, and it is not meant to address traceability. It should be a tool to help you think about testing risks to your project. It should not replace face-to-face conversation with your customer or the rest of your team.

Using a Test Matrix

Janet uses release planning to work with the testers and customers to develop a high-level test matrix. A test matrix is a simple way to communicate the big picture concerning what functionality you want to test. It gives your team a quick overview of the testing required.

A test matrix is just a list of functionality down the side and test conditions across the top. When thinking about test conditions and functionality, consider the whole application and any impact the new or changed functionality might have on the rest of the application. Testers sitting with customers and thinking about test conditions is what is important.

It can also be a mechanism to track coverage and can be as detailed as you like. A high-level test matrix can be used by the team to show the customer team or management what has been tested already and what is left. A more detailed test matrix can be used by the team to show what is planned for testing and track the progress of the testing. After the matrix has been created, it becomes easy to fill in the squares when testing is done. Keep it simple. Because we like big visible charts that are easy to read, we recommend colors that mean something to your team. For example, green (G) means testing is done and the team is happy with it, while yellow (Y) might mean some testing has been done but more exploratory testing is needed if there is time. Red (R) means something is broken. A white square means it hasn’t been tested yet, and a gray (not applicable) square means it doesn’t need to be tested.

Let’s look at an example. We have a small release we want to put out that calculates shipping costs. In Figure 15-7, different pieces of functionality are represented on one axis, and properties of the shipment are represented on the other. Individual cells are color-coded to show which cases are tested and which need more attention. All of the cells for “<= 2 lbs” are finished, the top three cells for > 4 lbs are done but need more exploratory testing, and the “Ship to Alaska”/“>4 lbs” cell denotes a possible issue.

Figure 15-7 A sample test matrix

Janet’s Story

I had an unexpected side effect from using a test matrix in one project I was on. The customers and testers put the test matrix together, and had thought of all affected functionality for the project and the high-level test conditions they would need. As expected, the act of planning brought a lot of issues out that would have been missed until later.

When they hung the matrix on the wall in their team area, Dave, the developer team lead, expressed an interest. One of the testers explained the matrix to him, and I was surprised when he said it was very useful for them as well. Dave said “I didn’t know that this functionality would affect this area. We need to make sure our unit tests touch on this as well.”

Looking back on this, I shouldn’t have been surprised, but I had never had that experience with the programmers before.

—Janet

A test matrix is a very powerful tool and can be used to help address traceability issues if your team has those problems. Think about what makes sense for your team and adapt it for your team and what makes sense to you.

Test Spreadsheet

Janet has also seen a spreadsheet format used with some success. For example, at WestJet, the first tab in a workbook was a high-level list of functionality that existed in the application. For each row, the team determined if the project affected that piece of functionality. If so, they gave a rating of the expected impact. After the impact of the changes had been determined, decisions about test environments, test data, or UAT could then be made.

Tabs were used for risks and assumptions but could be used for anything your team may need. A flexible format such as a spreadsheet means you can tailor it to work for you.

This information can be used in a number of different ways. It can be used to determine where to concentrate your exploratory testing efforts, or maybe to help create a high-level test matrix to make sure you touch on all of the areas during your testing.

A Whiteboard

If your team is informal and has small releases, any kind of documentation may be too much. Sometimes it’s enough to list the risks and assumptions on a whiteboard or on index cards. Janet has used a whiteboard to manage risks, and it worked quite well. If a risk actually became an issue, the result was documented and crossed off. It was easy to add new risks and mitigation strategies, and the list was visible to the whole team. This could also be done on a wiki page.