4.10 Kubernetes Node Volume Handling

Kubernetes Node Volume Handling is the collection of processes on a worker node responsible for attaching, mounting, formatting, and exposing storage volumes so that they are ready for use by containers before those containers start, and for safely unmounting and detaching them once pods no longer need them. It is carried out primarily by the kubelet's volume manager in cooperation with CSI drivers, and it operates independently of, but in coordination with, container runtime execution.

The Volume Manager Component

Role Within the kubelet

The volume manager is an internal kubelet subsystem that tracks which volumes are required by pods scheduled to the node and drives them through a state machine of attach, mount, unmount, and detach operations. It runs as a continuous reconciliation loop, comparing the volumes currently mounted against the volumes that should be mounted given the pods present on the node.

Desired State and Actual State Tracking

The volume manager maintains two internal caches: a desired state of world, built from the volumes referenced by pods assigned to the node, and an actual state of world, reflecting volumes that have actually been successfully attached and mounted. Reconciliation continuously works to bring the actual state in line with the desired state.

Attach and Mount Stages

Volume Attachment

For volume types that require it, such as many cloud block storage volumes, an attach operation associates the underlying storage device with the node itself, typically performed through a call to a CSI controller plugin running outside the node, which interacts with the cloud provider's storage API.

Global and Pod-Level Mounts

Once attached, the volume is typically mounted first to a global mount point on the node, shared across any pods that might use the same volume, and then bind-mounted from that global path into the pod-specific directory that will ultimately be exposed to the container's mount namespace.

Filesystem Formatting

If a raw block device is being consumed as a filesystem volume for the first time, the node-level mount operation includes formatting the device with the requested filesystem type, such as ext4 or xfs, before the mount can succeed.

The Container Storage Interface

CSI Node Plugin Responsibilities

The Container Storage Interface, or CSI, defines a standard by which storage vendors implement pluggable drivers. On each node, a CSI node plugin runs as a DaemonSet pod and exposes a Unix domain socket that the kubelet communicates with to request NodeStageVolume and NodePublishVolume operations.

NodeStageVolume and NodePublishVolume

NodeStageVolume performs the global, node-wide mount preparation for a volume, while NodePublishVolume performs the pod-specific bind mount that makes the volume available at the exact path a container expects. Separating these two calls allows a single staged volume to be published to multiple pods when volume access modes permit it.

CSI Driver Registration

CSI drivers register themselves with the kubelet through a registration mechanism that informs the kubelet of the driver's socket location and supported capabilities, allowing the kubelet to route volume operations for a given storage class to the correct driver implementation.

In-Tree and Ephemeral Volume Types

ConfigMap and Secret Volumes

For ConfigMap and Secret volumes, the kubelet does not go through CSI at all. Instead, it directly renders the referenced data as files on a tmpfs-backed directory on the node, populating them with the current contents of the referenced API object before the pod's containers start.

emptyDir Volumes

An emptyDir volume is created directly on the node's local filesystem, or in memory if configured with a medium of Memory, and exists only for the lifetime of the pod on that node, being deleted entirely once the pod is removed from the node.

hostPath Volumes

hostPath volumes expose an existing path on the node's own filesystem directly into the container, bypassing most of the attach and mount machinery since no new storage needs to be provisioned, though the kubelet still validates the path according to the configured hostPath type.

Unmount, Detach, and Cleanup

Safe Unmounting

When a pod is deleted, the volume manager unmounts the pod-specific bind mount first, ensuring no process inside the container's former mount namespace still holds the volume open, before proceeding to unmount any shared global mount point if no other pod requires it.

Detachment from the Node

After unmounting completes, if the volume type requires attachment, a detach operation is issued, again generally through the CSI controller plugin, releasing the underlying storage device so that it can be attached to a different node if the associated pod is rescheduled elsewhere.

Orphaned Volume Cleanup

The volume manager periodically scans for volume directories on the node that no longer correspond to any known pod, cleaning up leftover mount points and directories that could otherwise accumulate from crashed or improperly terminated pods, preventing stale mounts from consuming node resources indefinitely.