Kubernetes Log Observation

Kubernetes Log Observation is the practice of retrieving and interpreting container log output, the primary mechanism by which application-level behavior — errors, request processing detail, internal state changes — becomes visible to operators, given that Kubernetes itself has no built-in mechanism to inspect application internals beyond whatever the container writes to its standard output and error streams.

How Kubernetes Captures Logs

stdout and stderr as the Log Source

Kubernetes expects containerized applications to write logs to stdout and stderr rather than to files inside the container, and the container runtime captures these streams, typically writing them to a log file on the node's filesystem that the kubelet then exposes through the Kubernetes API.

kubectl logs api-service-7d4f9

Log Rotation and Ephemeral Storage

Node-level log files are subject to rotation and size limits configured on the kubelet, and because they exist only on the node's local filesystem, they are lost when a pod is deleted or a node is removed — the reason centralized log aggregation is necessary for anything beyond very short-term, single-instance debugging.

Retrieving Logs Interactively

Basic and Filtered Retrieval

kubectl logs supports retrieving the most recent lines, following new output in real time, and filtering by time window or line count, covering the common interactive debugging needs without requiring a separate logging backend.

kubectl logs api-service-7d4f9 --tail=100
kubectl logs api-service-7d4f9 --follow
kubectl logs api-service-7d4f9 --since=10m

Multi-Container and Previous Instance Logs

For a pod with multiple containers, --container selects a specific one; --previous retrieves logs from a container's prior instance, essential for diagnosing why a container crashed and restarted, since the current instance's logs begin fresh after the crash and contain no information about what preceded it.

kubectl logs api-service-7d4f9 --container sidecar
kubectl logs api-service-7d4f9 --previous

Structured Versus Unstructured Logs

The Value of Structured Output

Applications emitting structured (typically JSON) log lines, with consistent fields for severity, timestamp, and contextual identifiers (trace ID, request ID, user ID), enable filtering and correlation in a centralized logging system far more effectively than free-text log lines that require regex parsing to extract equivalent information.

{"level":"error","timestamp":"2024-01-15T10:32:14Z","trace_id":"abc123","message":"database connection refused","service":"api-service"}

Correlating Logs With Other Signals

Including a trace or request identifier consistent with what appears in metrics labels and trace spans allows an operator to pivot directly from an elevated error-rate metric, to the relevant traces, to the exact log lines explaining the specific failure — a workflow that depends entirely on that identifier being present and consistent across all three signal types.

Centralized Log Aggregation

Why Aggregation Is Necessary

Because individual container logs are ephemeral and scattered across whichever node a pod happens to run on, any log retention or cross-pod search beyond the immediate present requires forwarding logs to a centralized system, typically via a node-level DaemonSet agent that tails log files and ships them to a log storage and indexing backend.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-agent
spec:
  template:
    spec:
      containers:
      - name: agent
        volumeMounts:
        - name: varlog
          mountPath: /var/log/containers

Enriching Logs With Kubernetes Metadata

A well-configured log pipeline enriches each log line with Kubernetes metadata — namespace, pod name, labels, node — automatically, allowing logs to be filtered by these dimensions in the aggregation system without the application itself needing to know or emit that context.

Practical Diagnostic Patterns

Starting From Recent Logs During an Incident

When investigating a reported problem, starting with kubectl logs --since scoped to the incident's approximate start time, combined with --previous if the container has restarted, narrows the log volume to the relevant window rather than scanning an entire log history.

Recognizing the Limits of Per-Pod Log Retrieval

kubectl logs retrieves logs from a single pod at a time; diagnosing an issue affecting many replicas of a workload requires either a centralized logging system's aggregate search capability, or scripting log retrieval across every matching pod, since there is no native kubectl logs equivalent for querying across a label selector's worth of pods simultaneously.