Log Aggregation

Context

Every service writes logs — that habit survived the move to microservices intact. What changed is where those logs live. In the monolith there was one process and one app.log; grep over that file was your debugger. Now the same logical request touches five services, each running several replicas, each writing to its own local file or standard output inside a container that the scheduler may destroy at any moment.

A customer reports that their checkout failed at 14:32. The relevant lines are scattered across the order replica that handled the click, the payment replica it called, and the inventory replica it called after that — three files on three machines, two of which have since been recycled and are gone.

Problem

There is no single place to look, and even if you could log into every box, the logs would not connect. The order service logged an error, the payment service logged a timeout, the inventory service logged nothing at all — but nothing in those lines tells you they belonged to the same request. You are left correlating by timestamp and guesswork, across files that may already have rotated out of existence.

So how do you make the log lines from a single request, spread across many short-lived replicas, both findable in one place and connected to each other?

Solution

Log aggregation has two halves, and you need both.

First, a pipeline ships logs off the box. Each service writes its logs to standard output (or a local agent tails them), and a collector forwards every line to a central, indexed store you can search across all services and all replicas at once. Because the logs leave the container immediately, they survive the replica that produced them.

Second, the logs are structured and correlated. Instead of free-form text, each line is a self-describing record — typically JSON — with consistent fields: a timestamp, a level, the service name, and crucially a correlation id (sometimes called a request id) that is generated once at the system edge and attached to every line emitted while handling that request, in every service it touches. Searching for that one id pulls back every line of the request’s journey, in order, regardless of which replica or service wrote it.

flowchart LR
  subgraph Replicas
    A[Order replica
stdout JSON]
    B[Payment replica
stdout JSON]
    C[Inventory replica
stdout JSON]
  end
  A --> COL[Log Collector / Agent]
  B --> COL
  C --> COL
  COL --> STORE[(Central Indexed
Log Store)]
  STORE --> SEARCH[Search UI
filter by correlationId]

Each replica emits structured logs to a collector, which forwards them to a central searchable store correlated by request id

Example

A structured log line emitted by the payment service while handling a checkout. It is a single JSON object per line, with stable field names and the correlation id that ties it to every other line of the same request:

{
  "timestamp": "2026-06-25T14:32:07.512Z",
  "level": "error",
  "service": "payment-service",
  "instance": "payment-7d9f-bk2qx",
  "correlationId": "c1f4e9a2-8b3d-4e77-9a01-2f6c5d8e1b04",
  "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
  "event": "charge_failed",
  "orderId": "ord_88213",
  "amount": 4999,
  "currency": "USD",
  "reason": "gateway_timeout",
  "durationMs": 3021,
  "message": "payment gateway did not respond within 3s"
}

Searching the central store for correlationId = "c1f4e9a2-8b3d-4e77-9a01-2f6c5d8e1b04" returns this line alongside the order service’s line that started the request and the inventory service’s lines that followed — the full story of one checkout, reassembled from three services.

Resulting context

What you gain:

One place to search, across everything. You query all services and replicas at once, and logs outlive the containers that wrote them, so a recycled pod no longer takes your evidence with it.
Requests read as stories. Filtering by correlation id reconstructs a single request’s path across service boundaries, in order — recovering the end-to-end view you had for free in the monolith.
Machine-readable fields. Structured logs can be filtered, aggregated, and alerted on by field (level, service, reason) instead of fragile text matching.

What it costs you:

You must propagate the correlation id. Generate it at the edge (gateway or first service) and pass it on every outbound call — usually via a header — or the chain breaks at the first hop that forgets.
Volume and cost. Centralizing every line from every replica produces a lot of data; you will need sensible log levels, sampling for high-volume debug logs, and retention policies, or the bill and the noise both grow without bound.
Discipline on fields. The value comes from consistent field names across services. Without a shared logging convention, userId here and user_id there quietly defeat your queries.

Distributed Tracing — a trace id is the same idea taken further, adding causal spans and timing on top of correlation.
Audit Logging — a separate, durable record for compliance, distinct from these operational debug logs.

Benefit	Trade-off
Every service’s logs live in one searchable place	Log volume in production gets very high — storage cost
Correlate logs across services with a trace ID	Log-shipping latency means logs aren’t 100% real-time
Find error patterns across deployments	Sensitive data in logs must be handled before aggregation
Audit and compliance from one central location	Setting up an ELK/Loki stack is non-trivial

Common mistakes

Logging everything at DEBUG level in production. Symptoms:

Log volume runs 100x higher than necessary.
Searches in Kibana are slow because the index is huge.
Unnecessarily high cost — use INFO in production, DEBUG only when actively debugging.

Logging with no correlation ID. Symptoms:

Tracing one request across 5 services means guessing which log lines belong together.
Incident investigation takes hours instead of minutes.
Inject a trace/request ID into every log line.

💡 Real-world examples

Elastic (ELK Stack): Elasticsearch + Logstash + Kibana is one of the most widely used log-aggregation stacks — Netflix uses ELK to monitor and analyze operational and security logs across hundreds of nodes, and Goldman Sachs uses Elasticsearch both to help clients track trades and to let engineering teams hunt bugs (Elastic: How the Elastic Stack Changed Goldman Sachs).

Grafana Loki: integrates with Prometheus/Grafana and doesn’t index log content, making it cheaper than Elasticsearch for high-volume logs — its official adopters list includes Grafana Labs itself and Dropbox (grafana/loki ADOPTERS.md).