Agile Testing: A Practical Guide for Testers and Agile Teams

Some tests might be good candidates for automated regression suites. Some might just answer your exploratory charter and be “done.” The agile team must critically think about what they are learning and “evolve” tests accordingly. The most important aspect here is to be “brain on” while testing, where you are looking for the “funny,” unexpected, or new, which automated tests would miss. Use automation for what it is good at (repetitive tasks) and use agile humans for what we are good at (seeing, thinking, and dealing with the unexpected).

Several components are typically needed for useful exploratory testing:

• Test Design: An exploratory tester as a good test designer understands the many test methods. You should be able to call different methods into play on the fly during the exploration. This agility is a big advantage of exploratory testing over automated (scripted) procedures, where things must be thought out in advance.

• Careful Observation: Exploratory testers are good observers. They watch for the unusual and unexpected and are careful about assumptions of correctness. They might observe subtle software characteristics or patterns that drive them to change the test in real time.

• Critical Thinking: The ability to think openly and with agility is a key reason to have thinking humans doing nonautomated exploratory testing. Exploratory testers are able to review and redirect a test into unexpected directions on the fly. They should also be able to explain their logic of looking for defects and to provide clear status on testing. Critical thinking is a learned human skill.

• Diverse Ideas: Experienced testers and subject matter experts can produce more and better ideas. Exploratory testers can build on this diversity during testing. One of the key reasons for exploratory tests is to use critical thinking to drive the tests in unexpected directions and find errors.

• Rich Resources: Exploratory testers should develop a large set of tools, techniques, test data, friends, and information sources upon which they can draw. The agile test team members should grow their exploratory resources throughout a project and throughout their careers.

To help you understand a day in the life of an agile exploratory tester, here is a short tester’s story:

I arrived at 8:00 a.m. and reviewed what had happened the night before during automated testing. The previous night’s automated tests found some minor but interesting errors. A password field on a login form had accepted a special character, which should have been rejected by the validation. I created an outline as a starting point for my “attack” (a top-level plan and/or risk list).

As I thought about my “plan of attack,” I sketched a small state model of the problem on a flip chart and showed this to a developer and my team’s customer rep. I designed a test incorporating their suggestions, using some data stress inputs that I expected the validation to reject (a 1 MG file of special characters). I executed my test with my stress input, and, sure enough, the system rejected them as expected. I tried a different data set and the system failed with a buffer overflow in the database. I was learning, and we were on the trail of a potentially serious security bug. As the day went on, I explored different inputs to the password field and worked with the team to get the bug fixed.

You can learn from automated test results as well as from exploratory testing. Each type of testing feeds into the other. Develop a broad range of skills so you’ll be able to identify important issues and write tests to prevent them from reoccurring.

The term exploratory testing was popularized by the “context-driven school” of testing. It’s a highly disciplined activity, and it can be learned. Session-based test management is one method of testing that’s designed to make exploratory testing auditable and measurable [Bach, 2003].

Session-Based Testing

Session-based testing combines accountability and exploratory testing. It gives a framework to a tester’s exploratory testing experience so that they can report results in a consistent way.

Janet’s Story

James Bach [2003] compares exploratory testing to putting together a jigsaw puzzle. When I first read his article with the jigsaw puzzle analogy, exploratory testing made perfect sense to me.

I start a jigsaw puzzle by dumping out all of the pieces of the puzzle and then sorting them into the different colors and edge pieces. Next, I put the edge pieces together, which gives me a framework in which to start. The edge of the jigsaw is analogous both to the mission statement, which helps me focus, and to the time-boxing of a session, which keeps me within certain limits.