12.14 Kubernetes StatefulSet Persistent Storage Behavior

Kubernetes StatefulSet Persistent Storage Behavior is the precise, underlying mechanical sequence by which a StatefulSet-managed Pod's storage is attached, detached, and reattached as that Pod moves through scheduling, rescheduling, and replacement, describing the actual runtime behavior of the storage subsystem rather than the higher-level configuration or operational practices surrounding it.

Initial Volume Binding

Claim Creation Preceding Pod Scheduling

When a new ordinal is first created, its corresponding PersistentVolumeClaim is generated before the Pod itself is scheduled, and depending on the storage class's binding mode, the underlying PersistentVolume may be provisioned and bound either immediately upon claim creation or deferred until a node has actually been selected for the Pod.

Volume Binding Mode Effects on Scheduling

Under immediate binding mode, the volume is bound to a specific location before scheduling occurs, which can constrain which nodes are eligible to run the Pod to only those with access to that location, whereas under wait-for-first-consumer binding mode, volume provisioning is deferred until after the scheduler has already chosen a node, avoiding this constraint on scheduling flexibility.

Behavior During Pod Rescheduling

Detachment From the Original Node

When a StatefulSet Pod is rescheduled, whether due to node failure or deliberate eviction, its volume must first be detached from the original node before it can be attached elsewhere, and this detachment process, particularly for network-attached block storage, can introduce a delay before the replacement Pod is able to proceed with attaching the same volume on its new node.

Reattachment to the Replacement Pod

Once detachment completes, the same underlying PersistentVolume, still bound to the same PersistentVolumeClaim, is attached to the node hosting the newly created replacement Pod for that same ordinal, preserving access to the previously written data without requiring any new provisioning step, since the claim itself was never deleted.

Cross-Zone Reattachment Constraints

Topology-Aware Volume Restrictions

Many storage backends provision volumes with an inherent zonal affinity, meaning a replacement Pod for a given ordinal can only be successfully scheduled and attached to its existing volume within the same zone the volume was originally provisioned in, a constraint that directly affects where the scheduler can place the replacement Pod following a node failure.

Consequences of a Zone-Wide Failure

If an entire availability zone experiences an outage, ordinals with volumes provisioned specifically within that zone cannot be successfully rescheduled elsewhere until the zone recovers or the underlying volume becomes accessible again, representing an inherent limitation of zonal storage behavior rather than a limitation of the StatefulSet controller itself.

Behavior During Node Termination Versus Pod Deletion

Graceful Detachment During Planned Pod Deletion

When a StatefulSet Pod is deleted through the ordinary graceful termination process, the volume detachment proceeds as part of the standard Pod teardown sequence, allowing the storage subsystem to cleanly release the attachment before the replacement Pod attempts to attach the same volume.

Forced Detachment After Node Failure

When the underlying node fails outright rather than being gracefully drained, the volume may remain marked as attached to the now-unreachable node until the cluster's storage attach and detach controller forcibly recognizes the failure and releases the attachment, introducing additional delay before the volume becomes available for reattachment to a replacement Pod elsewhere.

Behavior During Scale-Down and Scale-Up Cycles

Claim Persistence Across a Temporary Scale-Down

If a StatefulSet is scaled down and later scaled back up to include a previously removed ordinal, and the corresponding claim was retained rather than deleted, the same underlying volume and its previously written data become available again to the newly created Pod for that ordinal, exhibiting continuity across the scale-down and scale-up cycle despite the intervening period during which no Pod existed for that ordinal.