Kubernetes Node Volume Handling

Kubernetes Node Volume Handling is the collection of responsibilities the kubelet performs on a node to make the storage a Pod requests actually appear inside its containers as usable mounted directories, spanning everything from resolving a volume's underlying storage source, to attaching and mounting it at the node level, to eventually unmounting and detaching it when the Pod no longer needs it. This work happens largely outside the container runtime itself: while the runtime bind-mounts already-prepared paths into a container's mount namespace at container creation time, it is the kubelet's volume subsystem that does the preceding work of making those paths exist and be populated correctly on the host.

The Volume Manager

Role Within the kubelet

The kubelet runs a dedicated Volume Manager as an asynchronous reconciler operating alongside the main SyncLoop, whose job is to compare the volumes a Pod's spec declares it needs against the volumes actually mounted for that Pod on the node, and to drive node-local state toward matching the declared volumes before that Pod's containers are allowed to start.

Desired State and Actual State Tracking

The Volume Manager maintains two internal caches, a desired state of world built from Pods assigned to the node and an actual state of world reflecting what has genuinely been mounted, and its reconciliation loop continuously compares these two to decide whether a mount, unmount, attach, or detach operation is needed next.

The Attach/Mount Sequence

Controller-Managed Attach

For volume types backed by external storage systems, an attach operation — associating a storage volume with the node itself, such as attaching a cloud block storage volume to a VM — is typically performed by the Attach/Detach Controller running in the control plane rather than by the kubelet, since attach operations act on cluster-level infrastructure rather than purely local node state. The kubelet's Volume Manager waits for this controller-driven attach to complete before proceeding.

Node-Level Mount Operations

Once a volume is attached to the node (or immediately, for volume types that require no separate attach step), the kubelet performs the actual mount: formatting the block device if necessary, creating the mount point directory, and mounting the device or network share to a per-Pod, per-volume path under the kubelet's root directory.

Container Bind-Mount

The path the kubelet mounts the volume to is not, by itself, visible inside any container; it is the container runtime, acting on the ContainerConfig built from the Pod's volumeMounts, that bind-mounts this kubelet-managed path into each container's own mount namespace at the specific mount path the container spec requests.

The Container Storage Interface

Why CSI Exists

Similar to how CRI decoupled the kubelet from specific container runtimes, the Container Storage Interface (CSI) decouples Kubernetes from specific storage backends, allowing storage vendors to implement a standard gRPC interface rather than requiring in-tree, Kubernetes-release-coupled volume plugin code for every storage system.

CSI Node Plugin Responsibilities

A CSI driver's node-side component runs as a Pod (typically a DaemonSet) on every node and exposes NodeStageVolume and NodePublishVolume gRPC methods that the kubelet calls during its mount sequence, staging the volume at a global mount point and then publishing (bind-mounting) it to the Pod-specific target path the kubelet requests.

Volume Types and Their Handling Differences

Ephemeral Volumes

Volume types such as emptyDir require no attach step and no external storage system at all; the kubelet simply allocates space directly from the node's local filesystem or, when configured, from tmpfs-backed memory, and this space is created fresh when the Pod starts and removed entirely when the Pod is deleted.

Persistent Volumes and Claims

For Pods referencing a PersistentVolumeClaim, the kubelet's volume handling resolves the claim to its bound PersistentVolume and proceeds through the same attach/mount sequence appropriate to that volume's underlying type, whether that is a CSI-backed volume or an in-tree plugin, treating the resolved volume no differently at the mount level than a directly specified volume source.

ConfigMap and Secret Volumes

Volumes sourced from ConfigMaps and Secrets are populated by the kubelet itself, which materializes the referenced key-value data as files under a tmpfs-backed directory, and the kubelet is also responsible for detecting updates to the source object and propagating them into the mounted files during the Pod's lifetime, subject to the kubelet's sync period.

Unmounting and Cleanup

Teardown Ordering

When a Pod is deleted, the Volume Manager's reconciliation ensures containers are fully stopped before it unmounts and, if applicable, requests detachment of the Pod's volumes, and this ordering prevents a container from being left with a dangling mount reference mid-teardown.

Orphaned Volume Cleanup

The kubelet periodically scans for mount paths on disk that no longer correspond to any known Pod, cleaning up orphaned volume directories that can result from a kubelet restart occurring mid-teardown or from other interrupted lifecycle transitions, preventing gradual accumulation of stale mounts on long-running nodes.