Agile Testing: A Practical Guide for Testers and Agile Teams

The middle tier in the pyramid is the layer that includes most of the automated business-facing tests written to support the team. These are the functional tests that verify that we are “building the right thing.” The tests in this layer may include “story” tests, “acceptance” tests, and tests that cover larger sets of functionality than the unit test layer. These tests operate at the API level or “behind the GUI,” testing the functionality directly without going through the GUI. We write test cases that set up inputs and fixtures that feed the inputs into the production code, accept the outputs, and compare them to expected results. Because these tests bypass the presentation layer, they are less expensive to write and maintain than tests that use the interface.

See Chapter, 8, “Business-Facing Tests that Support the Team,” for more about business-facing tests that support the team.

We try to write them in a domain-specific language that the customers can understand, so they take more work than unit-level tests. They also generally run more slowly, because each test covers more ground than a unit test and may access the database or other components. The feedback they provide is not as quick as the unit-level tests, but it is still much faster than we could get operating through the user interface. Therefore, their ROI is not as high as the tests that form the base of the pyramid, but it’s higher than the top layer.

Fit and FitNesse are examples of tools used for the middle layer of the pyramid. Home-grown test harnesses that use spreadsheets or other business-friendly means for defining test cases are also common.

We have more about these tests in Chapter 8, “Business-Facing Tests that Support the Team,” and Chapter 9, “Toolkit for Business-Facing Tests that Support the Team,” where we discuss the business-facing tests that support the team and the tools that effectively capture these tests.

The top tier represents what should be the smallest automation effort, because the tests generally provide the lowest ROI. These tests are the ones done through the GUI, the ones that actually operate and manipulate the presentation layer. They are written after the code is completed, and so are usually written to critique the product and go directly to the regression suite.

These tests are traditionally more expensive to write, although there are new tools that help reduce the investment needed. Because components of the user interface tend to be changed often, these tests are much more brittle than tests that work at a functional or unit level. For example, just renaming HTML elements could cause a test script to fail. Operating through the user interface also slows these tests down, compared to tests in the lower levels of the pyramid that operate directly on production code. The tests in the top layer do provide important feedback, but a suite of GUI tests may take hours to run rather than the few minutes required for unit-level test suites. We want to minimize the number of tests at this layer, so they should only form the tip of the pyramid.

No matter how many automated tests they have, most systems also need manual testing activities, such as exploratory testing and user acceptance testing. We don’t want to forget these, so we’ve illustrated them with the little cloud at the tip of the pyramid. The bulk of our regression testing must be automated or our manual testing won’t give us a good return on investment either.

Patrick Wilson-Welsh [2008] adds a descriptive dimension to the test automation pyramid with a “three little pigs” metaphor. The bottom foundation layer is made of bricks. The tests are solid, and not vulnerable to the huffing and puffing of the Big Bad Wolf. The middle layer is made of sticks. They need rearranging more often than the brick layer to stay strong. The tests in the top layer are made of straw. It’s hard to get them to stay in place, and the wolf can easily blow them around. If we have too many tests made out of straw, we’re going to spend lots of time putting them back into shape.

Most new agile teams don’t start with this shape pyramid—it’s usually inverted, a left-over from previous projects. GUI test tools are often easier to learn, so teams start out with a lot of tests in their top “straw” layer. As we mentioned in the previous chapter, the “hump of pain” that most programmers have to overcome to master unit test automation means that the team may start out with only a few bricks. The fixtures that automate functional tests in the middle layer are easy to write if the system is designed with those tests in mind, so the sticks might pile up faster than the bricks. As teams master TDD and unit test automation, the bottom layer starts to grow. When they get traction, a team using TDD will quickly build out the brick foundation of the test pyramid.

See the bibliography for a link to Patrick Wilson-Welsh’s discussion of “flipping the test pyramid” right-side up.