Kubernetes Container Log Handling

Kubernetes Container Log Handling covers how the container runtime itself — containerd, CRI-O, or another CRI-compliant runtime — captures a container's stdout and stderr streams and makes them available to the kubelet, a layer beneath the kubelet's own log file management and distinct from it, since the runtime is what actually intercepts the container process's output at the moment it is written.

The Container Runtime's Role in Logging

Capturing Streams at the Process Level

When a container process writes to stdout or stderr, the container runtime (via its shim process managing the container) is what actually captures those file descriptors, since Kubernetes itself has no direct visibility into a container's I/O — it depends entirely on the runtime correctly redirecting and persisting these streams.

# containerd's shim process holds the container's log pipe
ps aux | grep containerd-shim

CRI Logging Requirements

The Container Runtime Interface specifies that a compliant runtime must write logs to a file at a path the kubelet provides, in the standardized CRI log format (timestamp, stream tag, log content), ensuring the kubelet can consistently parse output regardless of which specific runtime is in use underneath.

2024-01-15T10:32:14.123456789Z stdout F {"level":"info","msg":"request handled"}

Streaming Logs on Demand

The CRI Streaming API

Beyond the persisted log file, the CRI streaming API supports live log streaming, exec, attach, and port-forward operations, each requiring the runtime to expose a streaming endpoint that the kubelet proxies requests to — this is the underlying mechanism kubectl logs --follow and kubectl exec rely on.

kubectl logs api-service-7d4f9 --follow

How kubectl Requests Reach the Container Runtime

A kubectl logs request travels from the client to the API server, which proxies it to the appropriate node's kubelet, which in turn either reads the persisted log file directly or, for follow requests, establishes a streaming connection through the container runtime's streaming server — a multi-hop path where a failure at any layer can prevent logs from being retrieved.

Runtime-Specific Logging Behavior

containerd's Logging Implementation

containerd uses its shim architecture to manage each container's I/O independently of the main containerd daemon process, meaning a containerd daemon restart does not necessarily interrupt already-running containers' logging, since the shim continues operating independently.

CRI-O's Logging Implementation

CRI-O similarly delegates container process supervision to a per-container conmon process, which handles log capture and forwarding to the configured log path, following the same general CRI logging contract but with its own specific process architecture underneath.

Consistency Across Runtimes From the Kubelet's Perspective

Because both runtimes conform to the same CRI logging contract, the kubelet's own log-serving logic (and by extension, kubectl logs behavior) remains consistent regardless of which specific runtime a node uses, which is precisely the abstraction the CRI is designed to provide.

Log Driver Considerations Beyond CRI Defaults

Alternative Logging Configurations

Some container runtime configurations support alternative log handling beyond the default CRI json-file-equivalent behavior, such as forwarding directly to a logging daemon rather than writing to a local file first, which can reduce disk I/O on the node at the cost of tighter coupling to a specific logging backend's availability.

Implications for Log Availability During Runtime Issues

Because the container runtime itself is what captures and persists container output, a malfunctioning or overloaded runtime component can itself become a source of log loss or delay, independent of anything happening in the application or the kubelet's own log-serving logic — worth considering during runtime upgrades or when diagnosing logging gaps that do not correlate with application-level issues.

Diagnosing Logging Problems at the Runtime Layer

Distinguishing Runtime-Level From Kubelet-Level Failures

When kubectl logs fails to return expected output, checking the container runtime's own health and logs (via crictl logs directly against the runtime, bypassing the kubelet) helps isolate whether the problem originates in the runtime's log capture itself versus in the kubelet's proxying or the API server's connection to it.

crictl logs <container-id>