Many tools and options are available today for performing load testing. When coupled with the abundance of features and technologies that must be evaluated and compared, the selection process can be quite confusing. In addition to technological parameters, you must also consider issues like technical support, cost of ownership, and the skills of your testing team.
Let’s navigate through these issues in a methodical way.
We begin by explaining how to get started with the evaluation process. Then, we continue to discuss technical features to be evaluated, and finally, provide tips on how to negotiate your best deal. The first step in choosing a load testing solution is evaluating your needs and resources. Here are some basic questions you should consider:
How often do you run load testing?
Start by evaluating the frequency of running performance tests. Will you be running tests once a quarter? Monthly? Daily? If you run load tests often, then you’ll want to look at a professional, higher-end type of solution. If you plan to load test infrequently, a “lightweight” or even open source solution may suffice.
On-Premise, cloud (SaaS) or hybrid?
Is your organization comfortable (from a security stand point) with a cloud solution, or do you need an on premise tool? Many solutions offer a hybrid approach where the load testing software runs on servers within your network, but also allow you to generate load via cloud based hardware (like Amazon AWS, Rackspace, etc.). Some lightweight cloud solutions are not capable of generating load on premise. Identifying your needs will eliminate many potential load testing solutions from your list.
Total Cost of Ownership
Total Cost of Ownership in load testing involves a few factors – from support to powerful features that should be built-in. You can get a free, open source tool. But you and your team won’t have anyone to call, will be spending a lot of time reading and posting questions on sites like StackOverflow (great tool – but can you afford to rely solely on the public domain?).
There are lightweight tools which may offer good support, but will lack correlation and advanced reporting. If your load testing tool doesn’t take you directly to the root cause and help you to identify bottlenecks, what good is it?
Do you need an open source, lightweight or powerful load testing tool? Consider these factors:
Support – You should expect easy access to tier 2 professionals during business hours. Support should help you and your team onboard and be very knowledgeable when it comes to scripting and scenario creation. When you can’t rely on support, you will lose time and money on your side.
Scripting capabilities vs. GUI – A complete solution should offer a combination of an easy-to-use GUI and nonproprietary scripting language like JavaScript. The intuitive user interface will allow non-technical individuals to quickly create scripts. Without it, you will spend more money on training your non-technical staff. Scripting will allow developers and advanced load testers to address more complex scenarios. Look for tools that use standard scripting languages that also match the expertise of your team.
Automatic correlation – An automatic correlation engine should identify and replace dynamic values which are unique for each run of the script such as session IDs, time, and others, and replace them automatically. Poor or no correlation is common in lightweight solutions. The money saved on a lightweight tool will be spent on labor-intensive script updates and manual correlation editing.
Integrations, frameworks and protocols supported – A huge number of frameworks and protocols are available today to your development team. Consequently, you must ensure that your load testing solution supports the widest range of frameworks and protocols. Down the road, it’s very likely that your development team will be using a different technology than today. A load testing tool that provides built-in integration with the technology used will make your testing much more efficient than tailoring an integration from scratch. Don’t get locked into a long-term maintenance contract with a load testing solution that doesn’t support a multitude of technologies.
Server side monitoring – The purpose of load testing is to evaluate how your system behaves under intense usage. An enterprise-scale load test solution will offer a wide range of server performance monitors and technologies to collect data during load testing. Without server side monitoring you are missing a key piece of load test information.
Comprehensive reporting – Reporting is often overlooked, but is often a weak area in many tools. However, extensive reporting capabilities will save you time and money identifying problems and performance bottlenecks. Ideally, reporting should provide a rich set of out-of-the box reports, but also the ability to easily customize reports so you can present data in any specific angle. Another important capability is a web interface to enable collaboration and the monitoring of results from any location in real-time.
And there’s more!
The next steps in choosing a load testing tool are:
- License Size & Cost
- Buying Strategies
- Product Demonstration
And finally, you need a detailed Features Matrix to compare the features of load testing tools that you are looking into.
This blog post is only the tip of the iceberg. For further reading, check out our entire Load Testing Buyer’s Guide eBook or learn about the Best Load Testing Tools in our updated blog for 2024.
