Kubernetes Stateful Reliability Basics

Kubernetes Stateful Reliability Basics is the set of reliability considerations specific to StatefulSet-managed workloads, where stable network identity, ordered pod lifecycle management, and per-replica persistent storage introduce constraints and failure modes that stateless Deployment reliability practices do not need to address.

Stable Network Identity

Predictable, Persistent Pod Names

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
spec:
  serviceName: db-headless
  replicas: 3

Each StatefulSet replica receives a stable, predictable name (db-0, db-1, db-2) and a corresponding stable DNS entry via a headless Service, that persists across pod restarts and rescheduling, unlike a Deployment's pods, which receive an arbitrary new name and IP on every replacement.

db-0.db-headless.default.svc.cluster.local

Why This Matters for Reliability

Distributed, quorum-based systems (a replicated database, a coordination service) frequently depend on knowing each member's stable identity to establish leader election, replication topology, or cluster membership; without stable identity, every pod replacement would require the entire cluster to rediscover and reconfigure its membership, a significantly more disruptive recovery process than what stable naming avoids.

Ordered Pod Lifecycle Management

Sequential Startup and Termination

spec:
  podManagementPolicy: OrderedReady

Under the default OrderedReady policy, StatefulSet pods are created, and each must become ready, before the next is created; pods are also terminated in strict reverse ordinal order during scale-down, guaranteeing that a quorum-sensitive application never loses more than one member's availability at a time during routine scaling operations.

Parallel Policy for Independent Replicas

spec:
  podManagementPolicy: Parallel

For workloads whose replicas do not have ordering dependencies on each other, Parallel creates and terminates pods simultaneously rather than sequentially, trading the ordering guarantee for faster scaling operations, appropriate specifically when the ordering constraint provides no actual reliability benefit for that particular workload.

Ordered Rolling Updates and Partitioning

Reverse-Ordinal Update Order

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0

StatefulSet rolling updates proceed in reverse ordinal order (highest-numbered pod first), and the partition field lets an update be applied only to pods with an ordinal greater than or equal to the partition value, providing a built-in canary mechanism: setting partition: 2 on a 3-replica StatefulSet updates only db-2, leaving db-0 and db-1 on the prior version until the partition is lowered.

Persistent Volume Reliability

volumeClaimTemplates and Per-Replica Storage

spec:
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 100Gi

Each StatefulSet replica receives its own PersistentVolumeClaim, generated from volumeClaimTemplates and named to match the replica's stable ordinal (data-db-0), and critically, this claim is not deleted when the pod is rescheduled, meaning a replaced db-0 pod reattaches to the exact same volume it was using before, preserving its data across the replacement.

PVC Retention on Scale-Down

spec:
  persistentVolumeClaimRetentionPolicy:
    whenScaled: Retain
    whenDeleted: Delete

The persistentVolumeClaimRetentionPolicy field explicitly controls whether PVCs are retained or deleted when the StatefulSet is scaled down versus when it is deleted entirely, an important reliability and cost consideration, since retaining volumes preserves data for a potential future scale-up but accumulates storage cost for volumes no longer backing any running pod.

Storage Reliability Considerations

Zone-Local Volumes and Rescheduling Constraints

A PersistentVolume backed by zone-local block storage can only be attached to a node within the same zone, meaning a StatefulSet pod's replacement is constrained to that zone even if capacity exists elsewhere in the cluster; this trade-off is what makes replicated, distributed storage backends, or an application-level replication strategy across zones, necessary for stateful workloads that must survive an entire zone outage rather than only a single node failure.

Relationship to Replica Reliability and the Reliability Model

Stateful reliability basics extend the replica-count reconciliation principles of replica reliability with the additional constraints stable identity and persistent per-replica storage introduce, and they represent a specialization of the broader reliability model's redundancy and fault-domain principles for workloads where a replacement pod's usefulness depends not only on it existing, but on it correctly reattaching to the same identity and data its predecessor held.