Phase 3: Reliability Engineering

Test patterns: soak, spike, stress & breakpoint testing

Intermediate ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you're building an amazing LEGO castle. You've spent hours making it perfect, with tall towers, strong walls, and even a working drawbridge. Before you let your friends play with it, or put it on display, you want to make absolutely sure it won't fall apart. That's a bit like what engineers do with big computer systems – they try to break them on purpose to make sure they're super strong and reliable for everyone who uses them. They use different "test patterns" to check for different kinds of problems.

One way to test your LEGO castle is to do a "soak test." This is like building your castle and then just leaving it on a shelf for a very long time – maybe days or even weeks. You're not actively playing with it much, but you're watching to see if any small problems appear over time. Does a wall slowly start to sag? Do two bricks that seemed tight at first slowly loosen up? This kind of test helps you find problems that only show up after a long time, like if a tiny piece slowly loses its grip, or if the weight of a top tower slowly squishes the pieces below it. It makes sure your castle is strong not just for a moment, but for the long haul.

Then there's "spike testing" and "stress testing." Imagine all your friends suddenly rush over to play with your LEGO castle at once! For spike testing, it's like a sudden burst: maybe five or ten friends all try to storm the castle, move the drawbridge, and play with the minifigures all at the exact same time, but only for a few minutes, then they all go back to playing calmly. Can your castle handle that sudden chaos without a tower toppling over or the drawbridge getting stuck? For stress testing, you push things even further. What if everyone you know, plus all their minifigures, tried to play with the castle at once? Or you tried to balance a super heavy book on its highest tower? You're trying to find out what happens when you push your castle past what it was ever designed for. This helps you discover its absolute weakest spots and where it will break first when things get really extreme.

So, by doing these different tests with your LEGO castle – letting it sit for ages, seeing how it handles a sudden rush, and pushing it to its breaking point – you learn all sorts of important things about its design. This means when you build something really important, like a computer system for a video game or a website, you can make sure it's strong enough to handle everything people throw at it, from normal everyday use to huge, unexpected crowds, without crashing or having problems.

As an SRE, understanding how your system behaves under various load conditions is paramount for ensuring reliability. Load testing isn't a one-size-fits-all activity; different "test patterns" help uncover distinct types of vulnerabilities. Soak testing, also known as endurance testing, involves subjecting your system to a typical, expected load for an extended period, often several hours or even days. The primary goal here is to identify long-term issues like memory leaks, resource exhaustion (e.g., database connection pools filling up), or slow degradation in performance over time that wouldn't surface in shorter tests. It's crucial for understanding system stability and resource management in the long run.

When dealing with sudden traffic surges, spike testing is your go-to. This pattern simulates an abrupt and significant increase in user load for a short duration, followed by a return to normal levels. Think viral events, flash sales, or news breaks. It tests your system's ability to handle sudden demand, scale quickly, and recover gracefully without cascading failures. Complementing this is stress testing, where you push your system beyond its expected maximum capacity, often until it visibly breaks or suffers severe performance degradation. The aim is to find bottlenecks, identify failure points, and understand how the system behaves under extreme, unsustainable pressure, giving insights into its resilience and error handling.

Finally, breakpoint testing (sometimes considered a form of stress testing) focuses on finding the precise maximum load your system can sustain before specific performance metrics (like response time or error rate) cross an unacceptable threshold defined by your Service Level Objectives (SLOs). Unlike pure stress testing which might aim to completely crash the system, breakpoint testing seeks to identify the "ceiling" of acceptable performance. By systematically increasing load until an SLO is violated, SREs can define concrete capacity limits and inform proactive scaling strategies. Together, these patterns provide a comprehensive view of your system's robustness, from everyday stability to catastrophic failure points.

Key Takeaways

  • Soak testing uncovers long-term stability issues like memory leaks and resource exhaustion over extended periods.
  • Spike testing evaluates system resilience and recovery from sudden, intense traffic bursts.
  • Stress testing reveals system breaking points and bottlenecks under extreme, sustained load beyond normal capacity.
  • Breakpoint testing identifies the maximum load capacity before violating defined performance SLOs.
  • Each pattern is critical for comprehensive capacity planning and proactive incident prevention.

Code Example

javascript
Preview

How this code works

This k6 script's primary job is to perform a "spike test," simulating a sudden, extreme surge in user traffic to observe how a website responds under stress and then recovers. It's a crucial test for identifying performance bottlenecks and ensuring system resilience. The script achieves this by defining a carefully orchestrated load profile within the options object.

The stages array within options dictates the test's user behavior over time. Initially, the script ramps up to target: 200 virtual users (VUs) over 30s for a baseline normal load. The core of the spike test then occurs: the VUs rapidly increase from 200 to target: 2000 in just 10s. Following this intense spike, the load ramps down to target: 200 VUs over 30s, and then maintains that level for 1m to monitor the system's recovery. The default function defines what each VU does: it makes an http.get request to the target URL, then sleep(1) for one second. This sleep is a subtle but critical detail; it simulates realistic user "think time" between actions. Without it, VUs would make requests continuously, creating an unrealistic, harsher load pattern that doesn't accurately reflect how users interact with a site.

Related Articles

Hand-picked reads to go deeper on load testing & performance.

Back to Site Reliability Engineer Roadmap