Kubernetes Node Runtime Cleanup

Kubernetes Node Runtime Cleanup is the collection of ongoing housekeeping processes the kubelet and container runtime perform to reclaim disk space, remove stale objects, and prevent the gradual accumulation of dead containers, unused images, and orphaned resources that would otherwise degrade a node's health over time. Because Pods are constantly created, restarted, and deleted throughout a node's operational life, cleanup is not an occasional maintenance task but a continuous background responsibility, tightly coordinated with the resource pressure signals that drive node condition handling and eviction.

Container Garbage Collection

Why Dead Containers Accumulate

Every time a container restarts under its Pod's restartPolicy, the runtime does not necessarily destroy the previous, now-exited container instance immediately; retaining recently exited containers is deliberately useful, since it preserves logs and the exit state operators may need to inspect through commands like kubectl logs --previous, but left unchecked this retention would consume disk space indefinitely.

The kubelet's Container GC Manager

The kubelet runs a dedicated container garbage collection manager that periodically evaluates dead containers against configurable policies — a maximum number of dead containers to retain per Pod, a maximum age since a container terminated, and a maximum total number of dead containers across the node — removing containers that exceed these thresholds via the runtime's RemoveContainer CRI call.

Protecting Recently Failed Containers

Container GC deliberately avoids removing a dead container too soon after it exits, since doing so could interfere with the kubelet's own crash-loop backoff logic or with an operator actively investigating a recent failure, balancing space reclamation against the practical need to retain forensic evidence of recent crashes.

Image Garbage Collection

Disk-Pressure-Triggered Reclamation

As described under Image Runtime Handling, the kubelet's image garbage collector monitors node disk usage and, upon crossing a configured high-watermark threshold, begins removing unused images through the runtime's ImageService until usage falls back below a configured low-watermark threshold, prioritizing images that have gone longest without backing an active container.

Interaction with Pull Behavior

Because image garbage collection can remove images that a future Pod might reference again, aggressive image cleanup settings trade reduced disk pressure for a higher likelihood of needing to re-pull an image later, which is a tuning consideration operators weigh differently depending on whether nodes have ample disk or are frequently under pressure.

Sandbox and Pod-Level Cleanup

RemovePodSandbox After Full Teardown

Once every container belonging to a deleted Pod has stopped and been removed, and the Pod object itself is confirmed deleted from the API server, the kubelet issues RemovePodSandbox, allowing the runtime to tear down the sandbox's network namespace, release its CNI-allocated IP address, and clean up any sandbox-specific state.

Volume and Mount Cleanup

Paired with sandbox teardown, the kubelet's Volume Manager unmounts and, where applicable, requests detachment of volumes that were mounted for the deleted Pod, and a separate orphaned-mount scanning routine cleans up leftover mount points on disk that no longer correspond to any known Pod, which can otherwise result from a kubelet restart interrupting a teardown mid-sequence.

Log Retention and Rotation

Container Log Rotation

The kubelet manages rotation of container log files on the node according to configurable maximum size and maximum file count settings, preventing a single verbose container from filling the node's disk with unbounded log output while still preserving a bounded recent history for debugging.

Coordination with Log-Shipping Agents

On nodes running log-shipping agents that tail container log files, cleanup timing matters operationally, since aggressive rotation or container removal settings can cause log content to be deleted before a shipping agent has finished reading it, which is a common source of missing log data in clusters with tightly tuned cleanup thresholds.

Ephemeral Storage Cleanup

emptyDir and Writable Layer Reclamation

Space consumed by emptyDir volumes and by containers' writable filesystem layers is reclaimed automatically as part of Pod and container teardown, and the kubelet also tracks ephemeral storage usage against per-Pod limits, evicting Pods that exceed their ephemeral storage limit as part of the same eviction machinery that responds to memory and disk pressure more broadly.