Phase 2: Observability

Structured logging with JSON & consistent field schemas

Intermediate ~3 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you’re building a huge, amazing Lego city, with bustling streets, tall buildings, and even little Lego people going about their day. Sometimes, a problem pops up – maybe a traffic jam, or a building falling over. In the old days, when computers wanted to remember what happened, it was like someone scribbling notes on random pieces of paper: “Traffic jam at 3 PM near the bakery,” or “Building 7 fell down, no idea why.” If you then wanted to figure out all the traffic jams, or every time Building 7 had an issue, you’d have to read through hundreds of messy, disorganized notes, which would take forever and you’d probably miss some important details.

That's where "structured logging" comes in, and it's like a superhero organizing system for your Lego city's events! Instead of messy notes, every time something happens, the computer makes a special, perfectly organized Lego brick. This brick isn't just any old piece; it's designed to hold specific information in specific spots. Think of JSON (which is just a fancy name for how these bricks are designed) as the blueprint for these special blocks. It says, "Okay, every event gets a block with a slot for 'time_it_happened', a slot for 'what_event_it_was', and a slot for 'who_was_involved'." So, for a traffic jam, the computer would fill in the slots: time_it_happened: "3:00 PM", what_event_it_was: "traffic_jam", location: "bakery_street". This makes it super easy for another computer to read and understand exactly what's going on.

Now, imagine your Lego city has grown even bigger, and you have thousands of these perfectly organized event blocks. The really clever part is making sure you use "consistent field schemas." This means that every single Lego block always uses the exact same names for its slots. If one block uses location to say where something happened, then every other block that talks about location must also use location. You wouldn't use place on one block and where_it_was on another if they mean the same thing. This is incredibly important because it lets your super-smart Lego sorting machine (which is like the big computer systems that help manage all these logs) instantly find exactly what you're looking for.

With consistent field schemas, you can tell your sorting machine, "Show me all the blocks where the location slot says 'bakery_street' and the what_event_it_was slot says 'traffic_jam'." It will zoom through millions of blocks and pull out exactly what you need in seconds, without getting confused. This means when you’re building your own cool computer programs or games, you’ll have a perfectly organized history of everything that happens, making it much easier to quickly find problems, understand why they occurred, and fix them so your creations run smoothly!

Traditional logging often involves writing human-readable plain text, making it incredibly difficult for machines to parse, query, and analyze effectively. Structured logging addresses this by outputting logs as machine-readable data, typically in JSON format. Instead of a free-form string, each log entry becomes a well-defined object with key-value pairs. JSON is preferred due to its ubiquitous adoption, flexibility, and easy parsing by virtually any programming language and log aggregation system. This transforms a log from a simple message into a rich data point, immediately unlocking capabilities for automated analysis.

However, simply using JSON isn't enough; the true power comes from employing consistent field schemas. This means always using the same field names (keys) for the same type of information across all your services and applications. For instance, if you log a request ID, always name it request_id, never requestId in one service and reqId in another. A consistent schema is crucial for log aggregation tools like Elasticsearch, Loki, or Splunk. These systems rely on predictable field names to index, search, filter, and aggregate data efficiently. Without it, queries become complex or impossible, as identical data spread across different field names cannot be easily correlated.

For an SRE, structured logging with consistent schemas is a game-changer for operational efficiency. It drastically speeds up incident response by allowing precise queries like "show me all errors for service_X where trace_id is ABC and user_id is 123." It enables proactive monitoring by letting you create dashboards and alerts based on specific log fields and values, such as error rates per service or latency percentiles. Furthermore, it facilitates comprehensive trend analysis, helping identify regressions, performance bottlenecks, or security incidents over time. Investing in proper structured logging is an essential foundation for any robust observability strategy.

Key Takeaways

  • Structured logging outputs logs as machine-readable data, typically JSON, rather than plain text.
  • JSON format allows logs to be easily parsed and processed by automated systems.
  • Consistent field schemas (e.g., always trace_id) are vital for effective search, filtering, and aggregation in log management tools.
  • Enables precise querying and faster incident response for SREs.
  • Supports proactive monitoring, dashboarding, and trend analysis based on log data.

Code Example

python
import json
import datetime

def log_event(level, message, **kwargs):
    log_entry = {
        "timestamp": datetime.datetime.utcnow().isoformat() + "Z",
        "level": level.upper(),
        "message": message,
    }
    log_entry.update(kwargs) # Add any extra fields
    print(json.dumps(log_entry))

# Example usage with consistent field names
log_event("info", "User logged in", user_id="123", service="auth", source_ip="203.0.113.45", trace_id="a1b2c3d4")
log_event("error", "Failed to connect to database", service="payment", db_host="prod-db-01", error_code="DB_CONN_FAIL", trace_id="e5f6g7h8", attempts=3)
log_event("warn", "High memory usage detected", host_id="app-server-02", metric_name="memory_percent", threshold=80, current_value=85.2)

How this code works

This Python code defines a reusable log_event function, designed to produce structured log messages in a consistent JSON format. This approach makes logs easy for machines to parse and analyze, a core principle of structured logging. The function ensures every log message always includes a timestamp, a standardized level (like INFO or ERROR), and a message.

Inside log_event, a base dictionary is created with these core fields. Importantly, datetime.datetime.utcnow().isoformat() + "Z" is used for the timestamp, ensuring all logs use Universal Coordinated Time (UTC) and explicitly mark it with "Z", preventing timezone confusion across different systems—a common pitfall for beginners. The function then leverages **kwargs to accept any number of additional, context-specific fields (like user_id or service). These extra fields are dynamically added to the log entry using log_entry.update(kwargs). Finally, json.dumps converts the complete Python dictionary into a JSON string, which print then outputs, ready for collection by a log aggregation system.

Related Articles

Hand-picked reads to go deeper on logging & log aggregation.

Back to Site Reliability Engineer Roadmap