At 3:14 AM, a payment processing system that had passed every load test in the build pipeline stopped accepting transactions. Throughput had been flawless at peak for weeks. The post-mortem traced it back to a single number: heap memory had climbed steadily to 3.8 GB with no plateau across an 18-hour window, and the JVM finally threw an OutOfMemoryError when there was nowhere left to allocate. Every short test had passed because the leak was slow – roughly 50 KB per request – and you simply cannot see a 50 KB-per-request leak in a 10-minute run.
That’s the problem soak testing exists to solve. It’s the test that catches what time hides: the slow heap creep, the connection that never gets returned to the pool, the file descriptor that quietly accumulates until you hit the OS limit. This guide turns soak testing from a vague best practice (“just run it overnight”) into an executable, measurable discipline. You’ll learn how to read the rising post-GC heap floor instead of watching a graph wiggle, how to quantify memory growth in MB/hour and project time-to-exhaustion, how to apply “circulation speed over pool size” reasoning to connection pools, how to pick tools by their long-duration readiness, and how to automate endurance runs in CI/CD without throttling your deploy velocity. Let’s get into the mechanics that actually matter.
- What Soak Testing Actually Is (and How It Differs from Load, Stress & Spike Testing)
- The Quantifiable Science of Time-Based Degradation
- How to Detect Memory Leaks During a Soak Test: A Step-by-Step Methodology
- How Long Should a Soak Test Run? A Duration Decision Matrix
- Choosing Soak Testing Software: A Long-Duration Readiness Framework
- Real-Time Monitoring & Trend-Based Alerting During Long Runs
- Automating Soak Tests in CI/CD Without Blocking Your Pipeline
- Soak Testing in Practice: Industry Use Cases & a Real Leak Discovery
- Frequently Asked Questions
- References & Authoritative Sources
What Soak Testing Actually Is (and How It Differs from Load, Stress & Spike Testing)
Soak testing – often called endurance testing – runs a system at sustained, realistic load over an extended period to surface failures that only emerge with time. Where a stress test asks “what’s the breaking point right now?”, a soak test asks “what slowly breaks if this keeps running for a day?” The failure modes are entirely different, and so is the diagnostic skill set. Understanding how soak testing can reduce your risk is the first step toward making it a routine discipline.
The Canonical Definition: Soak vs. Endurance Testing
Soak testing and endurance testing are the same discipline under two names; the ISTQB glossary treats endurance testing as the canonical term, defining it as testing to evaluate whether a system can sustain the required load continuously over a defined period [1]. The defining characteristic isn’t the magnitude of load – it’s the duration combined with realism. You’re typically running at 60 – 80% of measured peak capacity, the level your system genuinely sees during normal busy hours, and holding it there for hours or days. The point isn’t to break things with brute force; it’s to let small inefficiencies accumulate until they become visible. For the broader taxonomy of vendor-neutral testing terms, the ISTQB standard software testing glossary is the reference to bookmark.
The Comparison Matrix: Smoke, Load, Stress, Spike & Soak at a Glance
| Test type | Typical duration | Load pattern | Primary failure mode it catches |
|---|---|---|---|
| Smoke | Minutes | Minimal, single pass | Broken build / non-functional path |
| Load | 30 – 60 min | Steady at expected peak | SLA breach at expected volume |
| Stress | Minutes to ~1 hr | Ramping past capacity | Breaking point & recovery behavior |
| Spike | Seconds to minutes | Sudden surge then drop | Auto-scaling lag, queue overflow |
| Soak | 3 – 72 hours | Steady realistic load over time | Memory leaks, resource exhaustion, throughput decay |
The right sequence runs smoke first (does it work at all?), then load (does it meet SLAs at peak?), and only then soak (does it stay healthy over time?). Running soak before you’ve confirmed functional correctness wastes hours discovering a bug a 5-minute smoke test would have caught. A soak-capable execution engine such as WebLOAD handles the steady multi-hour plateau that this row demands. For a fuller breakdown of these categories, see RadView’s overview of the four types of load testing.
The Degradation Timeline: Which Failures Surface at Which Hour
Different failures have different incubation periods, and knowing the timeline tells you how long to run. Connection-pool leaks and unbounded log/disk growth often surface within the first 4 – 12 hours because requests cycle quickly and the leak rate is high relative to the resource ceiling. Garbage-collection pressure and cache-eviction inefficiencies tend to show up across the 12 – 24 hour band as caches fill and GC pauses lengthen. The slowest, most dangerous category – gradual heap leaks at a few KB per request – may need 24 – 72 hours before the post-GC floor rises far enough to confidently distinguish it from normal variance. That’s why a single overnight run isn’t a universal answer: the appropriate duration is dictated by the slowest failure mode you’re trying to expose.
The Quantifiable Science of Time-Based Degradation
This is where most guides stop at “watch the memory graph” and where you need to keep going. Every slow-burn failure soak testing catches has a measurable signature. Learn to read the numbers and you stop guessing.
Reading the Heap Floor: Why Post-GC Valleys Beat Raw Peaks
The single most important diagnostic skill in memory-leak detection is ignoring the noisy heap peaks and watching the post-GC minimum – what Oracle’s troubleshooting documentation calls the live set: the amount of heap still used after an old-generation collection has reclaimed everything that’s genuinely garbage. Oracle’s guidance is explicit: “your live set may go up and down, but if you see a steady increase over time, then you could have a memory leak” [2]. A healthy application produces a sawtooth – heap climbs as objects allocate, then drops back to a roughly constant floor after each GC. A leaking application produces a sawtooth whose floor is rising: each post-GC valley sits a little higher than the last (Oracle’s worked example tracks a live set climbing past 34 MB across collections). If your peaks look alarming but your post-GC floor is flat, you’re fine – that’s just allocation churn. If the floor trends upward, that’s your leak signature. For the underlying GC mechanics that produce these patterns, Oracle’s official guide to garbage collection tuning is the primary source.

Connection Pool Exhaustion: The Silent Killer
Connection-pool exhaustion is counterintuitive because raw pool size is the wrong thing to optimize. What governs effective capacity is circulation speed. Brett Wooldridge’s analysis of pool sizing puts it bluntly: “You want a small pool, saturated with threads waiting for connections” [3]. The math makes it concrete: a 100-connection pool where each connection returns in 10ms can service roughly 10,000 queries per second, while a 1,000-connection pool where each connection is held for 100ms is effectively more exhausted – it’s circulating ten times slower per connection. Wooldridge documents an Oracle Real-World Performance demonstration where simply reducing the pool dropped response times from ~100ms to ~2ms, a 50x improvement, and offers a starting-point formula of ((core_count * 2) + effective_spindle_count).

The reason this belongs in a soak-testing discussion is that pool leaks are slow killers. A single connection that fails to return per few thousand transactions won’t register in a short test, but over sustained hours the pool drains until acquisition blocks and the timeout cascade begins. LinkedIn famously suffered a multi-hour outage when a slow stored procedure held connections long enough to exhaust the pool. The metric to watch isn’t “active connections” – it’s connection acquisition time, the wait between requesting a connection and getting one. A rising acquisition-time trend over hours is your early warning. Connecting these symptoms to root cause is exactly the kind of work covered in RadView’s guide to identifying bottlenecks in performance testing.
The Resource-Leak Taxonomy: Beyond Memory
Memory gets all the attention, but the resources that quietly exhaust over time are broader. And a nuance worth internalizing: even a correct cleanup construct like Java’s try-with-resources can still leak if the underlying close() implementation silently fails to release the native handle – the language gives you the structure, not a guarantee.
| Resource | Typical cause | Diagnostic command | Exhaustion symptom |
|---|---|---|---|
| File descriptors | Unclosed streams/files | lsof -p <pid> |
“Too many open files” errors |
| Sockets | Connections not closed | ss -s / lsof -i |
Port/socket exhaustion, refused connects |
| Threads | Unbounded thread creation | thread dump (jstack) |
OOM “unable to create new native thread” |
| Native handles | Faulty close() in JNI/native libs |
/proc/<pid>/fd count |
Native memory growth outside heap |
Watch the trend of each count over the run, not the absolute value – a file-descriptor count climbing steadily across 24 hours against a ulimit ceiling is a leak even if you’re nowhere near the limit yet. The Linux lsof manual for diagnosing open file and socket handles and the Linux resource limits (getrlimit/setrlimit) reference are the authoritative references for the ceilings you’re testing against.
How to Detect Memory Leaks During a Soak Test: A Step-by-Step Methodology
Here’s the reproducible procedure you can execute mid-test, grounded in peer-reviewed heap-growth analysis [4] and Oracle’s documented dump-comparison workflow.
Step 1-2: Establish a Baseline and Apply Realistic Sustained Load
First, run your functional and load tests to confirm correctness – a soak test of broken behavior just wastes 24 hours. Record baseline heap, RSS, GC frequency, thread count, and connection-pool metrics at steady state. Then ramp gradually: a sudden load spike contaminates your early readings with allocation churn that looks like growth. A clean profile uses a short ramp, a long plateau, and a ramp-down:
ramp-up: 5 min → target (70% of measured peak)
plateau: 24 hr → hold target
ramp-down: 5 min → zero
A sustained execution engine such as WebLOAD holds this plateau steady for the full window without the generator itself drifting. Establish the baseline using the methodology in RadView’s guide on how to conduct a load test.
Step 3-4: Trend-Slope Analysis and Post-GC Valley Comparison
This is the differentiating step. Sample the post-GC heap floor at regular intervals and compute the slope in MB/hour. Worked example: if your floor grows 150 MB/hour and you have 4 GB (4,096 MB) of headroom above the current floor, projected time-to-exhaustion is roughly 4096 ÷ 150 ≈ 27 hours. That single number tells you whether you’ll survive a production deploy cycle. Critically, compare memory at equivalent post-GC valleys, never at peaks – peak-to-peak comparison produces false positives because peak height depends on allocation timing relative to the last collection. Valley-to-valley comparison isolates the genuinely retained set.
Step 5: Confirm with Heap-Dump Diffing
A rising slope is suspicion; a heap-dump diff is confirmation. Capture HPROF dumps at scheduled checkpoints – say hour 1, hour 12, and hour 24 – and compare them. Oracle’s guidance is to dump heap contents across recordings and identify which object class is accumulating. A positive result looks unambiguous: one object class (or a small set) shows steadily increasing retained size across the dumps while everything else stays flat. A dominator-tree or retention analysis pinpoints the reference chain holding those objects alive. Pair the heap dumps with thread dumps and connection-state snapshots at the same checkpoints so you can correlate a thread leak or connection leak against the same timeline.
How Long Should a Soak Test Run? A Duration Decision Matrix
Soak tests commonly run anywhere from 3 to 72 hours, with 24 hours the default value most teams reach for [5]. But “24 hours” without reasoning is cargo-cult testing. The duration should be derived, not borrowed.
The Slowest-Cycle Principle
Run your soak test for at least 2 – 3 full cycles of your system’s slowest periodic process. That principle gives you a defensible number tied to your actual system rather than a round figure. Identify the slowest cycle: it might be a log-rotation window, a cache TTL, a scheduled batch job, or a daily traffic peak-and-trough.
| System characteristic | Slowest cycle | Recommended duration |
|---|---|---|
| Daily-cycle web/API app | 24h traffic pattern | 24 – 48h (1 – 2 full daily cycles) |
| Batch / cron-heavy backend | Nightly scheduled jobs | 48 – 72h (2 – 3 nightly cycles) |
| Suspected slow heap leak | Leak slope confirmation | Multi-day until floor trend is unambiguous |
A nightly cron job is the classic trap: a 24-hour run captures it exactly once, so you can’t distinguish a real leak from a one-off allocation. Run 48 – 72 hours and the third nightly cycle either confirms the trend or clears it. Plenty of leaks that a 24-hour test waved through have been caught on the second night of a 72-hour run. For how this fits alongside other endurance approaches, see RadView’s endurance testing guide.
Cloud vs. On-Premises: The Cost of Multi-Day Tests
Multi-day runs cost real money on metered cloud infrastructure, and an aborted run is wasted spend. Two tactics keep costs sane: schedule extended 48 – 72h runs over weekend windows when both cloud spot capacity is cheaper and the test environment is otherwise idle, and run load generators on-prem or on reserved instances for predictable multi-day cost. Generator stability matters here too – a tool that destabilizes at hour 30 forces a costly re-run, so the stability of platforms like WebLOAD across 24+ hour windows directly protects the infrastructure budget you’ve committed to the test.
Choosing Soak Testing Software: A Long-Duration Readiness Framework
Most tool roundups score features that matter for a 30-minute test. For a 48-hour test, the criteria shift entirely – and one under-discussed risk dominates: the testing tool itself can leak. Open-source generators that accumulate full result sets in listener memory will themselves OOM long before they reveal a leak in your application. Score tools on what survives time, and lean on RadView’s guidance on how to choose a performance testing tool to structure the evaluation.
The Long-Duration Readiness Checklist
A transparent, weighted rubric for sustained tests:
| Criterion | Weight | What you’re scoring |
|---|---|---|
| Load-generator stability over time | 25% | Does the generator hold steady RSS for 24h+? |
| Distributed execution | 20% | Can load spread across nodes to avoid single-generator strain? |
| Result streaming vs. memory-bound accumulation | 20% | Are results streamed to storage or held in RAM? |
| Real-time monitoring depth | 20% | Live dashboards for heap, threads, connections, throughput? |
| CI/CD integration | 15% | Scriptable, headless, threshold-driven pass/fail? |
Score each candidate 1 – 5 per criterion, multiply by weight, and you get a defensible ranking you can adapt to your context.
Why WebLOAD by RadView Excels at Enterprise Endurance Testing
For enterprise endurance scenarios, RadView’s platform is built around the long-duration criteria above: stable generators across 24+ hour runs, real-time analytics with resource tracking, and intelligent correlation that ties client-side metrics to server-side resource trends as the test runs. Its protocol coverage – including WebSocket and MQTT – matters specifically for IoT and telecom soak tests, where the failure mode is sustained-connection leakage: thousands of persistent connections held open over hours, each one a potential resource leak. An MQTT soak scenario holding 10,000 persistent device connections for 48 hours, with live dashboards tracking open-connection count and broker memory, is exactly the workload where short-test tooling tells you nothing. An example soak scenario reuses your average-load script with an extended plateau stage and streams results to disk rather than buffering them in generator memory.
Choose-by-Use-Case Selector
- If you have a scriptable team, tight budget, and short-to-medium runs → open-source tools are a reasonable fit, provided you configure result streaming and distributed mode to avoid generator memory growth.
- If you need quick managed runs without infrastructure ownership → a SaaS-based platform handles provisioning, at the cost of less control over multi-day result retention.
- If you need low-code authoring, deep enterprise monitoring, sustained-connection protocol support, and proven 24h+ generator stability → an enterprise platform like WebLOAD by RadView fits the long-duration, high-observability profile.
For where these tool categories map onto load-testing types generally, see RadView’s four types of load testing overview.
Real-Time Monitoring & Trend-Based Alerting During Long Runs
The whole value of a soak test evaporates if you discover the failure by reading a crash log the next morning. The goal is to be alerted while there’s still time to act – and that requires alerting on trends, not absolutes.
Leading Indicators vs. Lagging Crash Alerts

Static threshold alerts fail during soak tests for a simple reason: absolute values look fine right up until they don’t. A heap at 60% utilization at hour 20 looks healthy, but if it’s climbing 50 MB/hour it’ll hit the wall at hour 28. The Google SRE practice of the Four Golden Signals – latency, traffic, errors, and saturation – captures this, noting that “latency increases are often a leading indicator of saturation” [6]. Even better, Google SRE explicitly endorses predictive, time-to-exhaustion alerting of the form “your database will fill its hard drive in 4 hours.” That’s the model: alert on the rate of change, not the level. Contrast a lagging alert (“fire when heap > 90%”) with a leading one (“fire when heap-floor slope > 50 MB/hour sustained over 2h”) – the second one buys you hours.
Building Trend-Based Alert Rules (with WebLOAD + APM)
A derivative-based alert expressed in a Prometheus/Grafana-style rule looks like:
# Fire when the post-GC heap floor grows > 50 MB/hour, sustained 2h
expr: deriv(jvm_memory_used_bytes{area="heap"}[2h]) > 52428800 / 3600
for: 2h
labels: { severity: warning }
annotations: { summary: "Heap floor rising ~{{ $value }} B/s — projected exhaustion" }
The three knobs to set: the metric (post-GC heap floor), the threshold (50 MB/hour here), and the evaluation window (2h). RadView’s platform metrics feed the same monitoring and alerting channels, so live test telemetry and your APM dashboards converge on one timeline during the run, an approach detailed in RadView’s guide to enhancing user experience with application monitoring.
Avoiding Alert Fatigue
The fastest way to make leading-indicator alerts useless is to make them noisy. The for: 2h duration condition above is the key noise-reduction technique – it requires the slope to hold before firing, so a momentary blip during a GC cycle or a transient load wobble won’t page anyone. Following Google SRE’s principle that alerts should be actionable, tune each rule so that when it fires, there’s a clear action to take. If an alert can’t be acted on, it’s a dashboard panel, not a page.
Automating Soak Tests in CI/CD Without Blocking Your Pipeline
Here’s the tension every DevOps team hits: soak tests run for hours or days, and CI/CD pipelines live and die by fast feedback. You cannot block a deploy for 48 hours. The resolution is placement, not compromise.
Pipeline-Placement Strategy: Fast Gates vs. Scheduled Soak Runs
Use the tiered model: lightweight smoke and short-load tests run as blocking gates after every commit, fuller intensive tests run in staging, and synthetic monitoring watches production. Full soak tests don’t belong in the per-commit path at all – they run on schedule. Nightly soak runs catch regressions within a day; weekend extended runs (48 – 72h) cover the slow-cycle leaks; and a pre-release validation soak gates the final promotion to production. None of these block a developer’s merge:
# .github/workflows/nightly-soak.yml
on:
schedule:
- cron: '0 22 * * 1-5' # 22:00 weeknights
jobs:
soak:
runs-on: self-hosted
steps:
- run: ./run-soak.sh --duration 8h --target 70pct --stream-results
The fast-gate side of this lives in RadView’s guide on how to conduct a load test.
Defining Trend-Based Automated Pass/Fail Criteria
A soak test’s pass/fail criteria can’t be a single-run absolute – they have to be regression-based, comparing this run’s trends against a stored baseline. This is SLO and error-budget thinking applied to endurance [6]. Two concrete thresholds:
- Fail if post-GC heap-floor slope exceeds baseline + 20 MB/hour.
- Fail if p99 latency drifts more than 15% from the run’s first hour to its last hour.
Both are trend deltas, not point values, which is what makes them trustworthy across runs with naturally varying absolute numbers. A WebLOAD CI integration runs headless, captures these trends, and returns a non-zero exit code when a threshold is breached – feeding straight into auto-ticketing.
Soak Testing in Practice: Industry Use Cases & a Real Leak Discovery
High-Stakes Industries: Finance, Healthcare, Telecom, E-commerce & SaaS
Some workloads can’t tolerate time-based degradation by their nature. Financial services trading platforms and payment gateways run 24/7, where a slow connection-pool leak over a full daily cycle cascades into the kind of multi-hour outage that makes headlines. Telecommunications VoIP and network-management systems hold sustained sessions for hours – precisely the WebSocket/MQTT persistent-connection pattern where socket and handle leaks accumulate invisibly until session establishment fails. E-commerce order-processing and inventory systems degrade through cache-eviction inefficiency and GC pressure that only manifests after a day of continuous transactions. Healthcare EMR and patient-monitoring platforms and SaaS multi-tenant background processors round out the set, where a thread leak in a job processor slowly starves the pool.
Worked Example: Catching a Leak Before It Reached Production

A team ran a 48-hour soak at 500 RPS against a release candidate. The baseline functional tests passed and the first hour looked clean. By hour 12, response times had drifted from 180ms to 400ms, and the post-GC heap floor was climbing a measured ~150 MB/hour – projecting heap saturation around 90% by hour 30. Slope analysis flagged it before the crash; a heap-dump diff between the hour-1 and hour-24 captures showed a single cache object class growing unbounded because entries were never evicted. The fix was a bounded LRU cache. The re-run produced a flat heap floor across the full 48 hours, response times steady at 180ms. The leak that would have surfaced as a 3 AM OOM in week two of production was caught and closed in the pipeline. For how a discovery like this connects to broader diagnosis, see RadView’s endurance testing guide.
Frequently Asked Questions
Is a 24-hour soak test always enough to catch memory leaks?
No – and assuming so is a common false confidence. Twenty-four hours covers one full daily traffic cycle, but it captures any nightly batch job or cron-driven process exactly once, which makes it impossible to distinguish a slow leak from a single-run allocation spike. For systems with slow periodic processes or very gradual leaks (a few KB per request), you need 48 – 72 hours to see 2 – 3 cycles and confirm the trend. The honest answer: run at least 2 – 3 full cycles of your slowest periodic process, which is often longer than 24 hours.
How do I tell a real memory leak apart from normal GC sawtooth behavior?
Stop looking at the peaks. The heap will always sawtooth up and down between collections – that’s healthy. Watch the post-GC floor (the live set after an old-gen collection). If that floor is flat or returns to roughly the same level after each GC, you don’t have a leak no matter how alarming the peaks look. If the floor trends steadily upward across collections, that’s the leak signature. Always compare valley-to-valley, never peak-to-peak.
Won’t my load testing tool itself run out of memory during a multi-day test?
It can, and many do – this is the most overlooked risk in long-duration testing. Open-source generators that accumulate full result sets in listener memory will OOM before your application does, producing a false failure or an aborted run. Mitigate it by configuring result streaming to disk rather than in-memory accumulation, distributing load across multiple generator nodes, and choosing a platform with documented generator stability across 24h+ runs. Verify generator RSS stays flat in a short dry run before committing to the multi-day test.
What single metric should I alert on for connection-pool problems?
Connection acquisition time – the wait between requesting a connection and receiving one – not active-connection count. Acquisition time rises before exhaustion fully hits, giving you a leading indicator. Set a trend-based alert on a sustained increase in acquisition time rather than a static “pool is full” threshold, since pool saturation is the lagging symptom.
Can soak test results be a reliable automated CI/CD gate, or do they need human review?
They can gate automatically using trend-based pass/fail criteria (e.g., heap-floor slope above baseline + 20 MB/hour, or p99 latency drift > 15% over the run). But automation flags the regression – it doesn’t diagnose the root cause. A human still reads the heap dump and decides whether the fix is a bounded cache, a lifecycle scope change, or a config tweak. Treat AI-assisted anomaly detection and automated thresholds as accelerators that surface the problem faster, not as a replacement for engineering judgment.
Configuration examples and metric thresholds are illustrative starting points; validate against your own production-like environment before relying on them. Tool capabilities described reflect documented behavior at time of writing.
References & Authoritative Sources
- ISTQB. (N.D.). Standard Glossary of Terms Used in Software Testing – endurance/soak testing. International Software Testing Qualifications Board. Retrieved from https://glossary.istqb.org/
- Oracle Corporation. (N.D.). 3.1 Debug a Memory Leak Using Java Flight Recorder – Java SE Troubleshooting Guide. Retrieved from https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/memleaks001.html
- Wooldridge, B. (N.D.). About Pool Sizing – HikariCP Wiki. GitHub. Retrieved from https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing
- Efficient Heap Monitoring Tool for Memory Leak Detection and Root Cause. (N.D.). IEEE Xplore (Document 9671473). Retrieved from https://ieeexplore.ieee.org/document/9671473
- Katalon. (N.D.). Soak Testing: Definition, Duration & Best Practices. Retrieved from https://katalon.com/resources-center/blog/soak-testing
- Ewaschuk, R. (ed. Beyer, B.). (2017). Chapter 6 – Monitoring Distributed Systems. Site Reliability Engineering, Google / O’Reilly Media. Retrieved from https://sre.google/sre-book/monitoring-distributed-systems/






