In the financial world, investors are characterized by the risk they’re willing to take. There’s even a formula that attempts to correlate risk with age: 100 – age = % of risk. The younger your are, the higher the risk you can afford. But fact is – individuals have different risk preferences.
Testing, in a sense, also involves the art of risk management. Given the lack of resources, time, and infrastructure – software testing is always compromised to some degree. QA groups must evaluate the risks they’re willing to take, decide what will be tested first, where to cut corners and so on.
If you ask the typical QA manager about a test plan, she’ll probably answer something like “with the limited resources I have, I took educated risks, but I’m still confident about our ability to deliver.” But, are the risks actually ‘educated?’ Were risks taken based on a complete checklist? In many case, soak testing is missing from the common list of load tests, therefore compromising the risk.
Why is soak testing important?
While stress testing take the tested system to its limits, soak testing take the system to its limit over time. The most complex bugs – memory leaks, unreleased resource, stuck system – happen when a system runs for a long period of time. If you skip the long soak tests, your chances of detecting such bugs prior to deployment are quite slim.
To soak or not to soak?
The next question is ‘how critical it is to run soak tests and for how long?’ The answer depends on your system characteristics. Is it a 24*7 system? Is downtime acceptable? How many resource do you have to test your system for a longer duration of time? What happens if you discover a memory leak much later, when the system is in production?
The bottom line is – when taking educated risk decisions, make sure to include soak testing as part of your checklist.