Agile Testing: A Practical Guide for Testers and Agile Teams

There is always a trade-off; your production database might be huge, so it is expensive and slow, but it provides the most accurate test data available. If your organization can afford hardware and software to store multiple copies of production data for testing purposes, this is ideal. Small companies may have resource constraints that might limit the amount of data that can be stored in test and staging environments. In this case, you’ll need to decide how much test data you can support and plan how to copy enough relevant data to make the test representative of what’s used in “real life.” Or you may consider making the investment in hardware, which is getting less expensive every day, to support a real production style environment. Otherwise, your test results might be misleading. As we mentioned with the canonical data, you may need to scrub the data before using it.

Data Migration

Data migration needs to be tested against a real database. The database upgrade scripts need to be run against real data and against the last known release of the database schema.

Testing a Database Migration

Paul Rogers, an automation test architect, tells this story of testing an eye-opening database migration [2008]:

Just yesterday, I ran a Rails migration against my test database. The developers had written it, tested it, and checked it using their development databases. My test database was probably 20,000 times larger. The migration for them took seconds. For me, well, I stopped it after three hours, at probably 10% complete. The programmers needed to redo their migration strategy.

I doubt this would have shown up on an in-memory database, so for me, a real database in this instance was definitely the right choice. In fact, this is likely to feed into things we need to consider before releasing, such as how long does a deployment take, or how long does the database update take. We can then use this to estimate how much down time we will need for the actual upgrade.

This is another example of how we must strike a balance between tests that deliver quick feedback and tests that realistically reflect events that might occur in production.

Understand Your Needs

If you understand the purpose of your tests, you can better evaluate your needs. For example, if you don’t need to test stored procedures or SQL queries directly for speed, consider tools such as in-memory databases, which work just like real databases but greatly speed up your tests. When you need to simulate the actual production environment, make a copy of the entire production database, if necessary. Quick feedback is the goal, so balance testing realistic scenarios with finding defects as efficiently as possible.

Evaluating Automation Tools

The first step in choosing an automation tool is to make a list of everything the tool needs to do for you. Let’s consider how you can decide on your test tool requirements.

Identifying Requirements for Your Automation Tool

After deciding on the next automation challenge to tackle, think about your tool needs. What tools do you already have? If you need additional ones, you probably want something that integrates well with your existing testing and development infrastructure. Do you need a tool to easily integrate into the continuous build process? Will your existing hardware support the automation you need to do? Setting up a second build process to run functional tests may require additional machinery.

Who’s going to use the test tool you’re hoping to implement? Will non-programmers be writing test cases? Do your programmers want a tool they feel comfortable with as well? Do you have distributed team members who need to collaborate?

Who will be automating and maintaining the tests? The skills already on your team are important. How much time do you have to get a tool installed and learn how to use it? If your application is written in Java, a tool that uses Java for scripting may be the most appropriate. Do team members have experience with particular tools? Is there a separate test team with expertise in a certain tool? If you’re starting the transition to agile development and you already have a team of test automators, it may make sense to leverage their expertise and keep using the tools they know.

Your tool requirements are dependent on your development environment. If you’re testing a web application, and the tool you choose doesn’t support SSL or AJAX, you may have a problem. Not every test tool can test web services applications. Embedded system testing can need different tools again. The case study in Chapter 12, “Summary of Testing Quadrants,” shows one way to use Ruby to test an embedded application.