Phase 2: Observability

Trace backends: Jaeger, Tempo & Zipkin

Intermediate ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine a super busy restaurant kitchen, like one you see on a TV show. There are lots of chefs, each making a different part of the meal – one cooks the pasta, another bakes the bread, another prepares the salad, and someone else puts it all together. When a customer orders a pizza, that order ticket travels through many hands. If a customer complains their pizza took too long, how would the head chef know where the delay happened? Was it the person making the dough, the one adding toppings, or the oven operator? It would be really hard to find out!

That's where something called a "trace backend" comes in. Think of it like a special control room for the head chef. Every time a chef works on part of an order – chopping, cooking, plating – they quickly make a note: "Started making dough at 1:05 PM," "Finished baking at 1:15 PM," "Handed off to delivery at 1:20 PM." These little notes are like tiny pieces of information. The trace backend is a super smart system that collects all these notes from all the chefs for every single order. It organizes them perfectly, like putting all the notes for one pizza order into a single folder, with timestamps for each step.

With this amazing system, the head chef can look at any order ticket and see its entire journey, step by step. If a pizza was late, they can instantly see which chef held it up, or if a particular ingredient ran out, causing a delay. It's like having a superpower to see inside every part of the kitchen at all times. This helps the head chef (who we call a Site Reliability Engineer, or SRE, in the coding world) figure out exactly what went wrong and how to make the kitchen run much smoother and faster next time. They can pinpoint problems like a detective finding clues.

Just like different restaurants might use different ways to track orders – some might have a big digital screen, others a detailed paper system, and some a tablet app – there are different "trace backends" that do this job. Some popular ones are named Jaeger, Tempo, and Zipkin. They are all tools that help collect, store, and show you these notes, but they might have slightly different features or be easier to set up in certain situations. This means that when you build your own computer programs, you can use these special systems to understand exactly how your program is working, where it might be slow, or if anything is breaking, helping you make it the best it can be!

Trace backends are the central nervous system for your distributed tracing data. They are responsible for receiving, storing, indexing, and enabling the querying and visualization of the trace spans your applications generate. Think of them as specialized databases and analytical engines built specifically for understanding the flow and performance of requests across microservices. Without a backend, your applications would just be sending trace data into the void; the backend makes that data actionable, allowing SREs to pinpoint latency bottlenecks, errors, and optimize service interactions.

Jaeger, Zipkin, and Tempo are three prominent open-source trace backends, each with its strengths. Jaeger, a CNCF project, is widely adopted and known for its comprehensive UI and flexible storage options (Elasticsearch, Cassandra, BadgerDB). It's great for traditional tracing needs, allowing direct querying by various attributes like service name, operation, and tags. Zipkin, an earlier pioneer in distributed tracing, offers a simpler, lightweight setup and is excellent for getting started quickly, also supporting various storage backends. Both Jaeger and Zipkin typically index trace data, enabling granular searches directly within their interfaces.

Tempo, developed by Grafana Labs, takes a different approach, focusing on extremely cost-effective, high-scale trace ingestion. Unlike Jaeger or Zipkin, Tempo does not index every span attribute by default. Instead, it stores traces as blocks in inexpensive object storage (like S3, GCS). This design means you typically don't query Tempo directly by attributes. Instead, you'd find a trace ID through correlated logs (e.g., using Loki) or metrics (e.g., Prometheus), and then use that ID to fetch the full trace from Tempo. This makes Tempo ideal for environments with massive trace volumes where indexing every detail would be prohibitively expensive, fitting seamlessly into the Grafana observability stack.

Key Takeaways

  • Trace backends store, index, and visualize distributed trace data collected from your applications.
  • Jaeger and Zipkin offer direct querying by trace attributes and have built-in UIs for visualization.
  • Tempo focuses on high-scale, cost-effective ingestion without full indexing; trace IDs are typically found via correlated logs/metrics.
  • Choosing a backend depends on your scale, cost constraints, and how you intend to query and correlate trace data.

Code Example

yaml
version: '3.8'
services:
  jaeger-all-in-one:
    image: jaegertracing/all-in-one:1.51
    ports:
      - "6831:6831/udp"  # Jaeger Agent (UDP) for Thrift compact protocol
      - "16686:16686"    # Jaeger UI (HTTP)
      - "14268:14268"    # Jaeger Collector (HTTP) for Thrift binary protocol
      - "14250:14250"    # Jaeger Collector (gRPC) for Thrift binary protocol
      - "4317:4317"      # OTLP gRPC endpoint
      - "4318:4318"      # OTLP HTTP endpoint
    environment:
      COLLECTOR_ZIPKIN_HOST_PORT: 9411 # Enable Zipkin compatibility
      COLLECTOR_OTLP_ENABLED: true     # Explicitly enable OTLP support

How this code works

This docker-compose.yaml file's primary purpose is to quickly launch a Jaeger tracing server, making it ready to collect and visualize tracing data from various applications. It uses Docker Compose's services section to define a single container named jaeger-all-in-one, pulling a specific version of the jaegertracing/all-in-one image, which bundles all necessary Jaeger components.

The ports section maps specific internal container ports to the host machine, enabling external access. For instance, 16686 is for the Jaeger UI, 6831/udp is for the Jaeger Agent to receive traces, and 14268/14250 are for collectors. Crucially, 4317 and 4318 expose endpoints for the OpenTelemetry Protocol (OTLP), a modern, vendor-neutral standard for sending traces. Within the environment section, COLLECTOR_ZIPKIN_HOST_PORT: 9411 is a subtle but important configuration: it tells Jaeger to also listen for and process traces sent in the Zipkin format, demonstrating its compatibility with other tracing systems. COLLECTOR_OTLP_ENABLED: true explicitly turns on OpenTelemetry support, ensuring Jaeger is fully equipped to handle traces via OTLP.

Related Articles

Hand-picked reads to go deeper on distributed tracing.

Back to Site Reliability Engineer Roadmap