Phase 3: Reliability Engineering

Setting SLO targets aligned with user expectations

Intermediate ~2 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you're baking cookies for your friends. Your goal isn't to make every single cookie absolutely perfect – exactly the same size, with the same number of chocolate chips, and still warm from the oven for everyone. Trying to do that would be super stressful, take forever, and probably make you exhausted! A few cookies might break, some might cool down, and that's usually okay. What you really want is for your friends to be happy with the cookies you make, and for them to enjoy eating them.

When people build websites or apps, it's a lot like baking cookies. They're making something for other people to use. They can't make everything 100% perfect, 100% of the time – it would cost too much money and take too much time and effort. Instead, they need to figure out what makes their "users" (the people using the app) happy, just like you figure out what makes your friends happy with your cookies. For example, your friends might be upset if half the cookies are burnt, or if they take five hours to bake when you said an hour. But they probably don't mind if a cookie has 8 chips instead of 10. These "rules" about what makes a cookie good enough are how we set targets for how well a website should work.

How do you figure out those "cookie rules" for your friends? You might ask them directly, "Did you like these cookies?" or "What kind of cookies do you like best?" (That's like asking users in a survey). You might notice if they often leave a specific type of cookie uneaten (like checking problem reports or common complaints). You might even look at what cookies other friends' parents bake to get ideas. All this information helps you decide what's truly important. Is it more important that the cookies are warm, or that they have enough chocolate chips? Is it better to make a few perfect cookies, or a lot of good-enough ones?

So, when people build big computer programs or online services, they use this same idea. They think about what their users truly need and expect, not what's technically possible to make 100% perfect. This means they can spend their time and resources wisely, building a service that’s reliable enough for people to love, without trying to achieve an impossible level of perfection that no one even really needs. It means they can make sure the most important parts of their app work really well, so everyone stays happy, and they can keep building even more amazing things!

Setting SLO targets isn't about picking arbitrary high percentages; it's fundamentally about understanding and meeting your users' expectations for your service. Aiming for 100% availability or sub-millisecond latency for every component is almost always an overspend, leading to wasted resources and engineering effort. Instead, the goal is to find the "sweet spot" of reliability where users are generally satisfied, and the business impact of potential downtime is acceptable, without incurring excessive development and infrastructure costs. This requires a shift from an engineering-centric view to a user-centric perspective, asking not just "can we make it faster?" but "how fast does it need to be for our users to be happy?".

To practically align SLO targets with user expectations, begin by mapping critical user journeys. Identify the core actions users perform and which parts of your service support them. Gather data: analyze support tickets to see what users complain about most frequently (e.g., slow logins, failed checkouts), conduct user surveys, or even A/B test different reliability levels if feasible. Consider your competitors – what level of service do they provide? Equally important is understanding the business impact of downtime for these critical paths. If an e-commerce checkout failure costs thousands per minute, your reliability target for that flow will be much higher than for a rarely accessed administrative report.

Once you have a clear picture of user expectations and business impact, you can translate these qualitative insights into concrete, measurable SLO targets. For example, if users frequently complain about slow page loads on your main dashboard, you might set an SLO for "99% of dashboard requests to complete within 500ms over a 28-day window." This target directly addresses the user pain point and is achievable without chasing an unrealistic 100%. Remember, these targets aren't set in stone; they should be revisited and adjusted as user expectations evolve, your product changes, or your understanding of the service's performance deepens.

Key Takeaways

  • Don't chase 100% reliability; aim for the "sweet spot" that balances user satisfaction and cost.
  • Map critical user journeys and analyze user feedback (support tickets, surveys) to identify pain points.
  • Understand the business impact of unreliability to justify investment in higher SLOs.
  • Translate qualitative user expectations into specific, measurable SLO targets (e.g., latency, availability).
  • SLO targets are dynamic and should be re-evaluated as products and user expectations change.

Code Example

yaml
sli:
  name: "Login Success Rate"
  description: "Percentage of successful user login attempts."
  metric_source: "prometheus"
  query: "sum(rate(login_requests_total{status='success'}[5m])) / sum(rate(login_requests_total[5m]))"
slo:
  name: "User Login Availability"
  time_window: "28d"
  objective: "99.9%"
  sli_ref: "Login Success Rate"

How this code works

This code snippet defines a Service Level Indicator (SLI) and a Service Level Objective (SLO) for user logins. Its job is to establish how to measure the reliability of login attempts and then set a target for that reliability, ensuring a consistent user experience.

The sli block first defines "Login Success Rate" as the specific metric to track. Its query is a Prometheus expression that calculates the percentage of successful login requests out of all login attempts over a rolling five-minute period. This query precisely measures the proportion of 'good' login events. A subtle but important detail is the use of rate in the Prometheus query, which calculates the per-second average rate of increase for a counter. This ensures the SLI measures the success rate of recent events, not just potentially stale or cumulative counter values. Subsequently, the slo block sets the "User Login Availability" objective. It links to the "Login Success Rate" sli_ref and mandates an objective of "99.9%" success, measured over a time_window of "28d". This means the system aims for at least 99.9% successful logins over any given 28-day period.

Related Articles

Hand-picked reads to go deeper on slos, slis & error budgets.

Back to Site Reliability Engineer Roadmap