4.17 Kubernetes Node Runtime Cleanup

Kubernetes Node Runtime Cleanup is the collection of garbage collection and reclamation processes the kubelet performs on a node to remove terminated containers, unused images, orphaned volumes, and stale cgroups, preventing the gradual accumulation of dead resources that would otherwise exhaust disk space, inode capacity, or other finite node resources over the operational lifetime of the node.

Why Cleanup Is Necessary

The Accumulation Problem

Every container that runs and exits leaves behind a writable filesystem layer, log files, and metadata unless explicitly removed. Left unmanaged, a long-running node executing thousands of pod restarts over weeks or months would steadily accumulate disk usage from dead containers and unreferenced images, eventually exhausting available storage.

Balancing Retention and Reclamation

Cleanup is not simply about deleting everything as soon as possible; recently terminated containers are often retained briefly to allow log inspection and debugging after a crash, so the kubelet's garbage collection policies balance operational usefulness against the risk of unbounded resource growth.

Container Garbage Collection

Per-Pod Retention Limits

The kubelet's container garbage collector enforces a maximum number of dead containers retained per pod, controlled by a configurable limit, removing the oldest terminated containers first once that limit is exceeded, while still preserving at least the most recent one for diagnostic purposes.

Global Dead Container Limits

In addition to a per-pod limit, a global maximum on the total number of dead containers across the entire node is enforced, ensuring that even a node running many distinct pods, each contributing a small number of dead containers, does not accumulate an unbounded total.

Minimum Age Before Collection

A minimum age threshold prevents garbage collection from removing containers that terminated only moments ago, giving operators or automated tooling a window of time to retrieve logs or exit details before the container's evidence is removed from the node.

Image Garbage Collection

High and Low Watermark Thresholds

Image garbage collection is driven by disk usage thresholds: when image filesystem usage crosses a configured high-water-mark percentage, the kubelet begins removing unused images, continuing until usage falls back below a configured low-water-mark percentage, at which point collection stops.

Least Recently Used Eviction Order

Among unused images eligible for removal, the kubelet generally removes the least recently used images first, based on tracked last-used timestamps, preserving images that are more likely to be needed again soon, such as those backing frequently restarted pods.

Interaction with Image Pull Behavior

Because image garbage collection can remove images between pod restarts, a subsequent pod creation referencing a previously removed image triggers a fresh pull, which is a deliberate tradeoff favoring disk space reclamation over avoiding redundant network transfer.

Sandbox and Pod Directory Cleanup

Terminated Sandbox Removal

Once a pod's containers and sandbox have fully terminated and the pod object itself has been deleted from the API server, the kubelet removes the corresponding sandbox through the runtime's RemovePodSandbox call, clearing associated metadata and any remaining sandbox-level storage.

Orphaned Pod Directory Scanning

The kubelet periodically scans its local pod directory tree, typically under a path like /var/lib/kubelet/pods, for directories that no longer correspond to any pod known to the kubelet, removing these orphaned directories along with any volume mounts or plugin state they contain.

cgroup Cleanup

Removing Empty cgroups

After a container's process has exited and been reaped, its cgroup becomes empty and eligible for removal. The kubelet, or the underlying runtime, deletes these empty cgroup paths to prevent the cgroup hierarchy from accumulating stale, unused entries over time.

Pod-Level cgroup Teardown

Once every container belonging to a pod has been cleaned up, the pod's own parent cgroup, which aggregated resource accounting and limits across its containers, is also removed, completing the resource isolation teardown that began when the pod was first admitted.

Log Rotation and Retention

Container Log File Rotation

The kubelet, or the container runtime depending on configuration, rotates container log files once they exceed a configured maximum size, retaining a bounded number of rotated log files per container to prevent an individual chatty container from consuming disk space unboundedly.

Coordination with External Log Collection

On nodes where a log collection agent runs as a DaemonSet to ship logs to a centralized system, log rotation and retention settings must be coordinated so that rotated files are not removed before the collection agent has had a chance to read and forward their contents.

Configuring Cleanup Behavior

kubelet Configuration Flags

Garbage collection thresholds and limits are exposed through kubelet configuration, either via command-line flags or the kubelet configuration file, allowing cluster operators to tune retention behavior according to node disk size, workload churn rate, and debugging needs specific to their environment.

Observability Into Cleanup Activity

Garbage collection actions are logged by the kubelet and, in some cases, surfaced as node-level events, giving operators visibility into when and why cleanup activity occurred, which is useful when diagnosing unexpected image re-pulls or missing historical container logs.