Structured Logging: Turning Scattered Logs into Traceable Investigation Trails

In a small system, a free-form log line is often enough to guess what just happened. Once an application has queues, an API gateway, background workers and several services, that style quickly turns into noise. Structured logging records named fields so operators can filter by request, user, endpoint, tenant or trace instead of reading line after line by eye.

1. Good logs answer investigation questions

A line such as payment failed only says there was an error. A structured record can also show which request it belonged to, which service emitted it, which user was affected, which operation was running and which downstream dependency failed. The goal is not to log everything; it is to log enough context to reconstruct an event.

OpenTelemetry defines a log record with fields such as timestamp, severity, body, resource, attributes, TraceId and SpanId. Its official data model aims to create a common understanding of log records so storage and analysis systems can interpret them consistently. See the OpenTelemetry Logs Data Model.

2. Trace ID is the thread connecting many logs

When a request travels through multiple components, a trace ID groups events from the same flow. The API handler, database call, webhook worker and cache miss may live on different machines, but if they carry the same trace ID, engineers can find the full chain.

OpenTelemetry also emphasizes log correlation: logs can connect with traces and metrics by time, by execution context such as trace/span IDs, and by the resource that produced telemetry. See OpenTelemetry Logging.

{
  "timestamp": "2026-09-21T09:14:32.418Z",
  "level": "error",
  "service": "checkout-api",
  "event": "payment.authorization_failed",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "request_id": "req_8f4b1a",
  "order_id": "ord_20491",
  "payment_provider": "stripe",
  "error_code": "card_declined"
}

In Elastic Common Schema, common tracing fields include trace.id, span.id and transaction.id. Whatever field names you choose, keep the convention stable across services. See Elastic ECS tracing fields.

3. Separate trace ID, span ID and request ID

Trace ID represents the whole distributed flow. Span ID represents one unit of work inside that trace, such as a Redis call or email send. Request ID is usually generated by a gateway or application for one HTTP request. All three can appear together, but they should not be treated as synonyms.

Request IDs are useful before full tracing exists or when comparing application logs with reverse proxy logs. Trace IDs become stronger in multi-step systems because the context travels across services. For background jobs, propagate trace context or at least record business IDs such as order_id, invoice_id or message_id.

4. Choose a small, durable field set

A practical schema should start with the fields people use most during incidents:

• timestamp, level, service, environment
• event or message with a stable, searchable name
• trace_id, span_id, request_id when available
• Business IDs such as user_id, tenant_id, order_id
• Error details such as error.type, error.message, error.stack
• Latency and dependency fields such as duration_ms, db.statement_hash, http.status_code

Avoid hiding everything in one long string. If someone needs to find “all 500 responses for tenant A on endpoint B in the last 15 minutes,” each condition should be its own field. If a value is numeric, log it as a number; if it is boolean, log it as a boolean. Do not make the analysis system guess data types from strings.

5. Do not write secrets to logs

Structured logging makes logs easier to search, which means leaks become easier to spread too. Do not log passwords, tokens, session cookies, OTP codes, private keys, full card numbers or unnecessary personal data. When correlation is needed, use internal IDs, an appropriate one-way hash, or a redacted version.

OWASP recommends application logging for security events and emphasizes consistency, but logs also need to be designed so they can be managed, analyzed and protected correctly. See the OWASP Logging Cheat Sheet.

6. Design event names like an internal API

Do not let event names change with prose. user.login.failed is better than User failed to login because password is wrong because dashboards, alerts and queries can rely on it. Details can live in fields such as reason, error_code or message.

{
  "level": "warn",
  "service": "identity-api",
  "event": "user.login.failed",
  "trace_id": "2c5db8b7409f4e4d99c1a0fd0f2e3341",
  "request_id": "req_9d12c7",
  "user_id": "usr_183",
  "reason": "invalid_password",
  "ip_hash": "ip_7f61"
}

Once event names are stable, operations teams can count failures, detect spikes or drill down by user without parsing Vietnamese prose, English prose or library error strings.

7. Attach context at the system edge

The best place to create a request ID, read traceparent and attach context is often middleware at the edge of a service. From there, the logger can automatically include shared fields throughout the request. Developers do not have to pass IDs through every function, and logs from error branches still carry the right context.

For queues, put the required context in message metadata. The worker receives the job and restores the context before logging. Without that step, the original request and background processing split apart precisely when investigation needs them together.

8. A low-risk rollout path

1. Standardize JSON logger output in one real but low-risk service.
2. Add middleware that creates or accepts trace_id and request_id.
3. Pick 10 to 15 important event names: login, order creation, payment, provider calls and job processing.
4. Create dashboards that query by trace ID, user ID, status code and duration.
5. Add automated checks or code review rules to block logs containing tokens, cookies and secrets.
6. Expand gradually to workers and supporting services while keeping the same schema.

Done well, logs stop being only a place to find stack traces after an incident. They become an operational map: where a request went, where time was spent, where failure began and which users were affected.