Agile Testing: A Practical Guide for Testers and Agile Teams

Lisa’s coworker Mike Busse took on the task of obtaining performance baselines for their web application that manages retirement plans. He evaluated load test tools, implemented one (JMeter), and set about to get a baseline. He reported the results both in a high-level summary and a spreadsheet with detailed results.

The tests simulated slowly increasing the load up to 100 concurrent users. Three test scripts, each for a common user activity, were used, and they were run separately and all together. Data gathered included:

• Maximum time of a transaction

• Maximum number of busy connections.

• A plot of the max time of a transaction against the number of users (see Figure 11-2 for an example of a chart)

Figure 11-2 Max and average transaction times at different user loads.

• Number of users who were on the system when the max time of a transaction equaled eight seconds

An important aspect of reporting results was providing definitions of terms such as transaction and connection in order to make the results meaningful to everyone. For example, maximum time of a transaction is defined as the longest transaction of all transactions completed during the test.

Mike’s report also included assumptions made for the performance test:

• Eight seconds is a transaction threshold that we would not like to cross.

• The test web server is equivalent to either of the two web servers in production.

• The load the system can handle, as determined by these tests, can be doubled in production because the load is distributed between two web servers.

• The distribution of tasks in the test that combines all three tests is accurate to a reasonable degree.

Mike also identified shortcomings with the performance baseline. More than one transaction can contribute to loading a page, meaning that the max page load time could be longer than the max time of a transaction. The test machine doesn’t duplicate the production environment, which has two machines and load-balancing software to distribute the transactions.

The report ended with a conclusion about the number of concurrent users that the production system could support. This serves as a guideline to be aware of as the production load increases. The current load is less than half of this number, but there are unknowns, such as whether the production users are all active or have neglected to log out.

Make sure your performance tests adequately mimic production conditions. Make results meaningful by defining each test and metric, explaining how the results correlate to the production environment and what can be done with the results, and providing results in graphical form.

If there are specific performance criteria that have been defined for specific functionality, we suggest that performance testing be done as part of the iteration to ensure that issues are found before it is too late to fix them.

Benchmarking can be done at any time during a release. If new functionality is added that might affect the performance, such as complicated queries, rerun the tests to make sure there are no adverse effects. This way, you have time to optimize the query or code early in the cycle when the development team is still familiar with the feature.

Any performance, load, or stress test won’t be meaningful unless it’s run in an environment that mimics the production environment. Let’s talk more about environments.

Test Environments

Final runs of the performance tests will help customers make decisions about accepting their product. For accurate results, tests need to be run on equipment that is similar to that of production. Often teams will use smaller machines and extrapolate the results to decide if the performance is sufficient for the business needs. This should be clearly noted when reporting test results.

Stressing the application to see what load it can take before it crashes can also be done anytime during the release, but usually it is not considered high-priority by customers unless you have a mission-critical system with lots of load.

One resource that is affected by increasing load is memory. In the next section, we discuss memory management.

Memory Management

Memory is usually described in terms of the amount (normally the minimum or maximum) of memory to be used for RAM, ROM, hard drives, and so on. You should be aware of memory usage and watch for leaks, because they can cause catastrophic failures when the application is in production during peak usage. Some programming languages are more susceptible to memory issues, so understanding the strengths and weaknesses of the code will assist you in knowing what to watch for. Testing for memory issues can be done as part of performance, load, and stress testing.