There is a growing recognition that performance testing needs to be an integral part of DevOps, as part of your CI/CD pipeline. It makes sense. Leaving performance to the end of your release cycle puts you right back in the days of the waterfall approach. Isolating the root cause becomes more difficult as more changes are made, and any fix can trigger a chain reaction of additional QA cycles. If performance is not up to speed at that point, it will very likely throw off the entire release schedule. With that in mind, what does it take to make performance testing part of DevOps? Here are some tips to get you started.
Related: Download the eBook
7 Performance and Load Testing Mistakes (and how to avoid them)
Walk before you run
Like any test, your performance tests will evolve over time throughout the development cycle. At the beginning of your project, performance testing may be limited to your APIs only. As additional services are developed and your flows become functional, you can add them to the scope of your tests.
Load testing all of your APIs or flows in a continuous manner is probably an unrealistic goal as these tests can take a long time and delay your pipeline. APIs and flows that are frequently used (e.g. login) and those that are part of critical user processes (e.g. completing a purchase) should probably be tested with each check-in, while others may be tested at a lower frequency.
Redefine success criteria as you go
There is no single pass/fail rule, as each test has its own success criteria. As your project progresses, performance success goals should be adjusted to reflect the evolution of the application. For example, a new activity added to the flow may require additional processing time, while improvements made to other functions may speed things up — changes that require a reevaluation of your success criteria.
Use your successful tests as a baseline
Once you have a successful test, save the results as a baseline for future tests. You may use these results to redefine your success criteria, or just add them as additional comparative metrics.
Compare apples to apples
Continuous changes to your test environment may result in performance variations. Failing to account for such changes could lead to misguided conclusions. For example, after installing a new server, your previous baseline results may no longer be relevant (which means it’s time to create a new baseline).
Don’t confuse load generation with performance measurement
When you run your load test you are simulating a load of hundreds or thousands of users accessing your system. You cannot, however, measure the response time of your application based on these users.
To measure your results correctly, you need two machines:
- One machine for generating the load
- A second machine with a probing client to measure the response of a single user which can be compared to your criteria
Test concurrently, but carefully
Running multiple tests in parallel can save time and help you accelerate your release cycles. However, concurrent-running tests may affect each other, and therefore skew the results. When running concurrent tests, make sure that your tests have no dependencies — not only using separate machines and load generator but also ensuring they are not calling the same APIs.
Closing thoughts
Treating your performance tests as an afterthought can be detrimental to your DevOps success. It will end up slowing down your release cycle, or even worse off — you may end up with a sub-par user experience that doesn’t meet market expectations and business requirements.
Integrating load testing into your DevOps doesn’t have to be difficult if you treat it like any part of your continuous development process — test often, fail fast, and iterate until you reach the desired results.
