Kubernetes Node Log Observation

Kubernetes Node Log Observation is the practice of inspecting logs produced by node-level components — the kubelet, the container runtime, and the underlying operating system — as distinct from container and application logs, necessary for diagnosing problems that originate below the workload layer entirely: node registration failures, resource pressure, container runtime crashes, and kernel-level events affecting every pod scheduled on that node.

Kubelet Logs

Accessing Kubelet Logs Directly

On most node operating systems, the kubelet runs as a systemd-managed service, and its logs are retrieved through the systemd journal directly on the node rather than through any Kubernetes API, since the kubelet itself is what implements the Kubernetes API surface for that node and has no equivalent "meta" logging path through itself.

journalctl -u kubelet -f

What Kubelet Logs Reveal

Kubelet logs record pod admission decisions, volume mount attempts, probe execution results, and communication with the container runtime and API server, making them essential for diagnosing why a pod fails to start on a specific node even when the pod's own container logs show nothing, since a startup failure before the container even begins running would only appear here.

E0115 10:32:14.123456 kubelet.go:1523] Error syncing pod, skipping: failed to "StartContainer" for "app" with CrashLoopBackOff

Container Runtime Logs

Runtime Daemon Logs

The container runtime (containerd or CRI-O) itself, running as a separate systemd service from the kubelet, logs its own operational events — image pulls, container creation and deletion, and runtime-level errors — accessible through the same journal mechanism.

journalctl -u containerd -f

Diagnosing Runtime-Level Failures

Errors in pulling images, creating container sandboxes, or managing container namespaces surface in the runtime's own logs rather than the kubelet's, and distinguishing between the two is important when a pod fails to start with an ambiguous error, since the kubelet's log message may only indicate that the runtime call failed without full detail on why.

Operating System and Kernel Logs

System-Level Events Affecting Workloads

The node's own kernel and system logs can reveal out-of-memory killer activity, disk I/O errors, or network interface problems that manifest as workload instability without any corresponding error in the kubelet, runtime, or application logs, since these are lower-level events the higher layers may only indirectly observe as an unexplained container termination.

dmesg -T | grep -i "killed process"
journalctl -k --since "10 minutes ago"

OOM Killer Diagnosis

When a container is terminated with an OOMKilled reason visible in its pod status, cross-referencing the kernel's own OOM killer log entries confirms the specific process and memory pressure conditions that triggered the termination, providing detail beyond what the pod status alone reports.

Accessing Node Logs When Direct SSH Is Unavailable

kubectl debug for Node-Level Access

kubectl debug node/<node-name> creates a privileged debugging pod with access to the node's filesystem and namespaces, providing a way to inspect node-level logs and state through the Kubernetes API even when direct SSH access to nodes is restricted or unavailable, which is increasingly common in managed and hardened cluster environments.

kubectl debug node/worker-3 -it --image=busybox -- chroot /host bash
journalctl -u kubelet

Centralizing Node Logs Alongside Container Logs

Because node-level logs share the same diagnostic importance as container logs during many incidents, forwarding kubelet, runtime, and relevant kernel log output to the same centralized logging system used for application logs — via a node-level agent configured to also capture systemd journal output — avoids needing direct node access at all for most routine investigation.

When Node Logs Are the Right Starting Point

Symptoms That Point to the Node Layer

Widespread pod scheduling failures on a specific node, containers failing to start with no application-level log output at all, or multiple unrelated workloads on the same node experiencing simultaneous problems are all signals that point toward node-level logs as the appropriate starting point, rather than continuing to search individual application logs for a cause that lies below the application layer entirely.