Phase 2: Observability

Log aggregation with ELK stack, Loki or CloudWatch

Intermediate ~2 min read
Think of it this way A friendly analogy. Read this if the technical version feels dense. Show Hide

Imagine you're building an incredible, gigantic LEGO city with many connected sets. Each time you build a piece, or if a tiny part accidentally falls off, it's like a small instruction note is created in that specific set's manual. Now, if something goes wrong, like the space station's solar panels stop spinning, and you need to figure out why and how to fix it, where do you look?

You wouldn't want to dig through all the instruction manuals from all the LEGO sets in your city just to find out why one specific part isn't working, right? That would take forever! Our big computer programs and "digital cities" create lots of these "notes" – called "logs" – about everything they do. These logs are usually scattered everywhere, on different digital "servers." Trying to find a problem by looking at each one individually is like searching for a tiny red brick by emptying all your LEGO bins onto the floor.

This is where "log aggregation" comes in, which is just a fancy way of saying "gathering all the notes together." Instead of scattered manuals, imagine we have a super-smart LEGO library. When anything happens in your LEGO city – a new part is added, a minifigure gets stuck, or a light stops working – a special librarian instantly collects that note and brings it to our central library. This librarian carefully files each note, adding details like what kind of event it was and when it happened. This way, if your space station's solar panels stop, you can go to our super-smart library, ask "Show me all notes about power issues in the space station from the last hour," and boom – you get exactly what you need, instantly.

This amazing library helps you fix things super fast, understand exactly why something broke, and even spot cool patterns. Some popular "library systems" for these digital notes are called the ELK Stack, Loki, or CloudWatch Logs. The ELK Stack is like a huge, very detailed library with many ways to organize and see information. Loki is like a library that's good at quickly telling you where to look for a problem, by only noting general "labels" of the notes. CloudWatch Logs is like a special, ready-to-go library if you're building your digital LEGO city with Amazon's tools. So, when you build your own cool programs or digital cities someday, you'll know how to keep all your important notes perfectly organized to be a super-fixer!

In distributed systems, logs are scattered across many servers, containers, and services. Manually sifting through individual log files using grep and ssh is inefficient, slow, and unsustainable for an SRE. Log aggregation solves this by centralizing logs from all your sources into a single, searchable platform. This centralization is crucial for rapid incident response, root cause analysis, security auditing, and gaining a comprehensive understanding of your system's behavior, transforming raw log data into actionable insights.

Key aggregation solutions include the ELK Stack, Loki, and CloudWatch. The ELK Stack (Elasticsearch, Logstash, Kibana) is a powerful, mature, and highly customizable solution. Logstash collects and processes logs, Elasticsearch stores and indexes them for fast search, and Kibana provides rich visualization dashboards. Loki, from Grafana Labs, takes a different approach by indexing only log metadata (labels) rather than the full log content, making it incredibly cost-effective for high-volume logging and integrating seamlessly with Grafana for querying. CloudWatch Logs is Amazon Web Services' fully managed solution, offering deep integration with other AWS services, easy collection from various AWS resources, centralized storage, search, and alerting capabilities.

As an SRE, understanding these tools means knowing how to get logs into them, query them effectively during an outage, and build dashboards for proactive monitoring. Each platform has its strengths: ELK excels at complex log transformations and full-text search, Loki is ideal for cost-optimized, label-based querying alongside Grafana, and CloudWatch is the go-to for AWS-native environments due to its ease of use and deep integration. Choosing the right tool depends on your infrastructure, budget, existing tooling, and specific observability requirements, but all aim to give you a single pane of glass into your system's operational health.

Key Takeaways

  • Log aggregation centralizes scattered logs from distributed systems into a single searchable platform.
  • ELK (Elasticsearch, Logstash, Kibana) offers powerful full-text search and visualization, often self-hosted.
  • Loki prioritizes cost-efficiency by indexing only log metadata (labels) and integrates well with Grafana.
  • CloudWatch Logs provides a fully managed, deeply integrated logging solution for AWS environments.
  • These tools are essential for SREs to perform rapid troubleshooting, root cause analysis, and proactive monitoring.

Code Example

yaml
# Example Filebeat configuration to send Apache access logs to Elasticsearch
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/apache2/access.log
  fields:
    service: apache-web
    env: production
  tags: [apache, access]

output.elasticsearch:
  hosts: ["your-elasticsearch-host:9200"]
  username: "elastic"
  password: "changeme"

# Or, send to Logstash for further processing:
# output.logstash:
#   hosts: ["your-logstash-host:5044"]
#   loadbalance: true

How this code works

This Filebeat configuration defines how to collect logs from a server and send them to a central logging system. Its primary job is to watch a specific log file, enrich the log entries with useful information, and then forward them to either Elasticsearch or Logstash. The filebeat.inputs section specifies that Filebeat should monitor a log file using type: log. It points to /var/log/apache2/access.log using paths. Crucially, it adds custom metadata like service and env using fields, and descriptive tags like apache and access. These additions are vital for easily searching and filtering logs later in Kibana, making raw log data much more understandable.

The output.elasticsearch section then configures Filebeat to send the processed logs directly to an Elasticsearch cluster, specifying the hosts address and authentication username and password. Alternatively, the commented-out output.logstash section shows how logs could instead be sent to Logstash for additional processing before reaching Elasticsearch. A subtle point for beginners is that Filebeat expects only one active output configuration. While the example clearly comments out the Logstash option, uncommenting both output sections would likely lead to an error or unexpected behavior, as Filebeat attempts to send data to multiple destinations simultaneously.

Related Articles

Hand-picked reads to go deeper on logging & log aggregation.

Back to Site Reliability Engineer Roadmap