Each holiday season has its e-commerce performance disaster story. This year, it was Macy’s turn, with Black Friday shoppers reporting massive credit card issues.
While the retailer names changes each year and the symptoms for disasters appear to be different (slow e-commerce website, stuck credit card system, bad CDN configuration), the root cause is constant: insufficient website performance testing.
Here are four common performance testing pitfalls and how they can be avoided:
“Our cloud machines can handle any load”
Using a powerful cloud infrastructure with large machines may lead to overconfidence and the erroneous assumption that the underlying architecture on its own provides immunity from performance disasters. The reality is that misconfigurations, application problems, and a host of other issues may lead to unexpected performance degradation – even when running the most robust enterprise infrastructure.
The case of Apple’s iPhone 6 launch is a well-documented example for such a failure, when a small CDN misconfiguration resulted in serious performance problems despite the massive infrastructure in place.
The lesson learned? Always include performance testing in you plans, even when you have the utmost confidence in the strength of your infrastructure.
Past performance is no guarantee of future results
The worst case scenario for your performance testing is the best case scenario for your business. You run your load tests, you feel all good about it, and now the big moment arrives and your site is painfully slow… What happened? You failed to plan for the best case scenario – turns out you have many more users hitting the site than you tested for.
If the load placed on the system fails to accurately simulate real-life load, then not much has been accomplished. Indeed, estimating the required load can be tricky. You may forecast the expected traffic based on past and current usage, but these are rearview mirror estimates that may fall well short of this year’s traffic.
The general rule of thumb to be on the safe side is to test 120-130% of your load estimate. When it comes to a significant event such as Black Friday, it’s better to be over conservative and plan your performance tests to generate a load that would be at least 200% of your traffic forecast. Even better – load test your system till it crashes, so you know its limits. The cost of increasing the load are negligible compared to the potential losses due to a dysfunctional e-commerce site.
Time heals everything, but it can break some things too
Running tests with high load for too short of a time is a common mistake. The test may show excellent response times during this short period of testing, but will fail to uncover issues that only surface after a longer period of stress placed on the system.
The graph below illustrates such as an example: during the first 20 minutes of the test the behavior is as expected, with memory consumption increasing as the load is increased. What happens next is not what we would expect, as memory consumption continues to increase even after the load size levels off. This can only be detected by letting the test run long enough and monitoring server-side events, not just response time.
Prior to peak e-commerce periods like Black Friday when you expect high loads to last several days, make sure to run performance tests for at least 5 hours to simulate extended stress on your website.
Even if the failure is not yours, the problem is
There are components in your system that you may not be able to control because they are run by third party providers; for example, the payment system that Macy’s used. But the fact that it is not your component does not mean that you don’t need to test it.
Just think how much money Macy’s lost when this 3rd party program failed to respond in as expected. An end to end test of the entire system is crucial to the delivery of the experience customers expect from your brand.
Don’t be the next headline
The annual recurrence of e-commerce disasters should serve as a stark reminder that when performance testing is compromised, the results are bound to be felt by customers and negatively impact the business.
The stakes are too high – don’t let the failure of performance testing be the downfall of your business.