Kubernetes Pod Log Handling

Kubernetes Pod Log Handling covers the node-level infrastructure and configuration governing how container log output is captured, stored, and rotated on disk, sitting beneath the interactive kubectl logs experience and largely invisible during normal operation, but directly determining how much log history is available, how quickly it consumes disk space, and how logs survive (or fail to survive) container restarts.

How the Kubelet and Container Runtime Capture Logs

The CRI Logging Path

The kubelet, communicating with the container runtime through the Container Runtime Interface, arranges for each container's stdout and stderr to be written to a log file on the node's filesystem, typically under /var/log/pods/, in a structured per-container directory layout that the kubelet uses to serve kubectl logs requests.

/var/log/pods/payments_api-service-7d4f9_<uid>/app/0.log

Log File Format

Container log files typically follow a CRI-standardized format, prefixing each line with a timestamp and stream indicator (stdout or stderr), allowing the kubelet and any log-shipping agent to parse timing and stream origin without needing application-level cooperation.

2024-01-15T10:32:14.123456789Z stdout F handling request for /api/v1/orders

Log Rotation Configuration

Size and Count-Based Rotation

The kubelet rotates container logs based on configurable maximum file size and maximum number of retained rotated files, preventing a verbose or misbehaving container from filling a node's disk indefinitely.

# kubelet configuration
containerLogMaxSize: "10Mi"
containerLogMaxFiles: 5

Consequences of Rotation for Log Retrieval

Because kubectl logs (without --previous) reads from the current, active log file, and rotation moves older content into separate rotated files that kubectl logs does not read by default, very verbose containers can lose easily-accessible log history to rotation well before a human operator gets a chance to review it — another reason centralized log shipping, which typically tails and forwards log content before rotation discards it, matters for anything beyond immediate live debugging.

Log Loss on Container and Pod Termination

Container Restarts and Previous Logs

When a container restarts (due to a crash or liveness probe failure), the kubelet retains the previous instance's log file long enough for kubectl logs --previous to retrieve it, but this retention is not indefinite and depends on the configured rotation and the node's own log retention behavior.

kubectl logs api-service-7d4f9 --previous

Pod Deletion Removes Node-Local Logs Entirely

Once a pod is deleted, its node-local log files are eventually cleaned up by the kubelet's garbage collection, meaning any log content not already forwarded to a centralized system before deletion is permanently lost — a critical reason log shipping infrastructure should be running continuously rather than only being consulted reactively during an incident.

Multi-Container and Sidecar Logging Considerations

Separate Log Streams Per Container

Each container within a pod, including sidecars and init containers, produces its own independent log stream and file, requiring kubectl logs calls (or log-shipping configuration) to explicitly account for every container that might contain diagnostically relevant output, not just the primary application container.

kubectl logs api-service-7d4f9 -c sidecar-proxy
kubectl logs api-service-7d4f9 -c init-migration

Init Container Logs

Init container logs are particularly easy to overlook since they only run once before the main containers start; kubectl logs <pod> -c <init-container-name> retrieves them, which is often the fastest way to diagnose why a pod is stuck before its main containers ever start.

Operational Considerations

Sizing Rotation for the Workload's Log Volume

A workload with high-volume logging benefits from larger containerLogMaxSize or a shorter effective retention if disk space on nodes is limited, while a workload with modest logging can use tighter defaults without losing meaningful history before a log-shipping agent has a chance to forward it.

Ensuring Log Agents Keep Pace With Rotation

A log-shipping DaemonSet that falls behind — due to backpressure from a downstream log storage system or its own resource constraints — risks missing rotated-away log content, making agent health and throughput monitoring a necessary complement to rotation configuration itself.