Agile Testing: A Practical Guide for Testers and Agile Teams

The most important thing to remember about exploratory testing is that it’s not a test technique on its own. Instead, it’s an approach or a mind-set that can be applied to any test technique. The second thing to remember is that exploratory testing is not merely about test execution; testers can also take an exploratory approach when they’re designing new tests at the beginning of the iteration or analyzing the results of tests that have already been performed. A third important note is that exploratory testing isn’t sloppy or slapdash or unprepared testing. An exploratory approach might require very extensive and elaborate preparation for certain tests—and an exploratory tester’s knowledge and skill set, developed over years, is an often invisible yet important form of preparation. An exploratory test might be performed manually, or might employ extensive use of test automation—that is, any use of tools to support testing. So if exploratory testing isn’t a technique, nor test execution, nor spontaneous, nor manual, what is it that makes a test activity exploratory? The answer lies in the cognitive engagement of the tester—how the tester responds to a situation that is continuously changing.

Suppose that a tester is given the mission to test a configuration dialog for a text editor. A tester using an exploratory approach would use specifications and conversations about the desired behavior to inform test ideas, but would tend to record these ideas in less detail than a tester using a scripted approach. A skilled tester doesn’t generally need much explicit instruction unless the test ideas require some specific actions or data. If so, they might be written down or supplied to a program that could exercise them quickly. Upon seeing the dialog, the exploratory tester would interact with it, usually performing tests in accordance with the original test ideas—but she might also turn her attention to other ideas based on new problems or risks in the dialog as it appeared in front of her. Can two settings conflict in a way not covered by existing tests? The exploratory tester immediately investigates by performing a test on the spot. Does the dialog have a usability issue that could interfere with a user’s work flow? The exploratory tester quickly considers a variety of users and scenarios and evaluates the significance of the problem. Is there a delay upon pressing the OK button? The exploratory tester performs a few more tests to seek a general pattern. Is there a possibility that some configuration options might not be possible on another platform? The exploratory tester notes the need for additional testing and moves on. Upon receiving new builds, the exploratory tester would tend to deemphasize repetition and emphasize variation in order to discover problems missed by older tests that are no longer revealing interesting information. This approach, which has always been fruitful, is even more powerful in environments where the need for repeated testing is handled by the developers’ low-level, automated regression tests.

Exploratory testing is characterized by the degree to which the tester is under her own control, making informed choices about what he or she is going to do next, and where the last outcome of the last activity consciously informs the next choice. Exploratory and scripted approaches are at the opposite poles of a continuum. At the extreme end of the scripted mind-set, the decision as to what to do next comes exclusively from someone else, at some point in the past. In the exploratory mind-set, the decision to continue on the same line of inquiry or to choose a new path comes entirely from the individual tester, in the moment in which the activity occurs, The result of the last test strongly informs the tester’s choices for the next test. Other influences include the stakeholders for whom test information might be important, the quality criteria that are important to stakeholders, the test coverage that stakeholders seek, specific risks associated with the item being tested, the needs of the end user of the product, the skills of the tester, the skills of the developers, the state of the item under test, the schedule for the project, the equipment and tools that are available to the tester, and the extent to which she can use them effectively—and that’s only a partial list.

No test activity performed by a thinking human is entirely scripted. Humans have an extraordinary capacity to recognize things even when people are telling them not to, and as a result we can be distracted and diverted—but we can learn and adapt astonishingly quickly to new information and investigate its causes and effects. Machines only recognize what they’ve been programmed to recognize. When they’re confronted with a surprising test result, at best they ignore it; at worst, they crash or destroy data.