Phase 2: Observability

Trace analysis: critical path, span waterfall & error tagging

Intermediate ~2 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you're making a big, fancy meal for a party – maybe lasagna, a fresh salad, and some yummy garlic bread! It's not just one simple thing; there are lots of steps, and you want everything to be ready at the right time. First, you might chop veggies, then boil pasta, then assemble the lasagna, put it in the oven, mix the salad dressing, and finally, pop the garlic bread in. Each of these individual cooking tasks is like a "span" in the world of computers.

Now, imagine you write down every single step of your cooking adventure on a timeline. You note when you started chopping, how long the lasagna baked, and when the garlic bread came out of the oven. You can see which tasks were happening at the same time (like mixing the salad while the lasagna bakes) and which ones had to wait for another to finish (you can't assemble the lasagna until the pasta is cooked). This detailed timeline, showing all your cooking "spans" from start to finish, is like a "span waterfall." It’s your special x-ray vision to see exactly how your meal came together, step by step, and quickly spot if anything took too long or went wrong.

While your waterfall shows all the cooking tasks, some are more important for the total time it takes to get the whole meal ready. For example, if the lasagna absolutely must bake for 60 minutes, and you can make the salad and garlic bread in just 20 minutes each, then no matter how fast you make the salad, the whole meal won't be ready until the lasagna is done. The longest sequence of "must-do-this-then-that" steps that directly decides when the entire meal is ready is called the "critical path." In this case, baking the lasagna is probably on your critical path.

Knowing the critical path helps you decide what to focus on. If you want the party meal ready sooner, you'd try to make the lasagna step faster or start it earlier, rather than just rushing to make the salad, which finishes much sooner anyway. And if something goes wrong, like you accidentally burn the garlic bread, you can make a note, a "tag," on that specific step. This helps you remember what happened and quickly find out if there's a problem, so you can fix it for next time! So, when you're planning your next big cooking adventure, thinking about your span waterfall and critical path means you can make sure everything runs smoothly and your delicious meal is ready exactly when you want it.

As an SRE, understanding how a request flows through your distributed systems is crucial for debugging and performance optimization. Trace analysis begins with the "span waterfall," a visual representation of all spans within a single trace, ordered chronologically. This waterfall chart vividly displays each operation's start and end times, duration, and its hierarchical relationship to other operations. By inspecting the waterfall, you can immediately identify which services or functions run sequentially versus in parallel, and pinpoint individual operations that are contributing significant latency to the overall request. It's your x-ray vision into the timing of your system's interactions.

While the span waterfall shows all operations, the "critical path" helps you focus your optimization efforts. The critical path is the sequence of interdependent spans that directly determine the total end-to-end latency of a request. Imagine it as the longest path through your span dependencies; any delays on this path directly impact the user's waiting time. Identifying the critical path allows SREs to prioritize where to invest performance tuning. Optimizing a span not on the critical path, especially if it runs in parallel with longer operations, might yield minimal overall improvement. Focus on the critical path to make the most impactful changes to system responsiveness.

Beyond performance, traces are invaluable for understanding failures. "Error tagging" involves attaching specific, descriptive tags (key-value pairs) to spans that represent operations where an error occurred. Instead of just knowing "an error happened," error tags provide crucial context: the error type (e.g., TimeoutError, DatabaseError), a detailed message, relevant HTTP status codes, or even parts of a stack trace. This structured information allows SREs to quickly filter for problematic traces, diagnose root causes, and aggregate error metrics across services. Properly tagged errors drastically reduce Mean Time To Resolution (MTTR) by providing immediate insights into what went wrong and where.

Key Takeaways

  • The span waterfall visualizes timing and dependencies of operations in a request.
  • The critical path identifies the direct sequence of spans determining total request latency, guiding optimization efforts.
  • Error tagging provides rich, structured context about failures within spans, accelerating root cause analysis.
  • Together, these techniques empower SREs to diagnose performance bottlenecks and reliability issues effectively.

Code Example

python
from opentelemetry import trace, status

# Assume 'current_span' is the active span for the operation.
# Example: with tracer.start_as_current_span("payment_processing") as current_span:

def tag_payment_failure(current_span, error_message):
    current_span.set_status(status.StatusCode.ERROR,
                            description=f"Payment failed: {error_message}")
    current_span.set_attribute("error", True)
    current_span.set_attribute("error.type", "PaymentGatewayError")
    current_span.set_attribute("error.code", "4001")
    current_span.set_attribute("error.message", error_message)
    # If an exception object 'e' is available, you can also use:
    # current_span.record_exception(e)

How this code works

This code defines a function, tag_payment_failure, specifically designed to enrich a distributed trace with detailed error information when a payment operation encounters a problem. In distributed tracing, correctly identifying and detailing errors within a current_span is fundamental for critical path analysis and quickly debugging issues across microservices. The function's main job is to transform a generic operation span into a clear indicator of failure, making it easily discoverable and understandable within a trace waterfall visualization.

The function achieves this by first calling current_span.set_status with status.StatusCode.ERROR and a human-readable description. This immediately flags the span as erroneous. Crucially, it then uses current_span.set_attribute multiple times to add granular error information like error, error.type, error.code, and error.message. A subtle but important distinction is that while set_status provides an overall state, these dedicated error. attributes are essential. Many tracing platforms rely on these specific error. prefixes for advanced filtering and aggregation, allowing engineers to quickly find all payment gateway errors across thousands of traces, rather than just relying on the generic status description. The commented record_exception further illustrates how a Python exception object could be attached for deeper context.

Related Articles

Hand-picked reads to go deeper on distributed tracing.

Back to Site Reliability Engineer Roadmap