Kubernetes Node Failure Tolerance

Kubernetes Node Failure Tolerance is the specific mechanism by which Kubernetes detects an unresponsive node through heartbeat and lease tracking, marks it unhealthy, and eventually reschedules its pods elsewhere, along with the timing trade-offs and split-brain risks involved in doing so, addressing exactly the involuntary disruption category that disruption budgets deliberately do not cover.

Node Heartbeat and Lease Tracking

The Node Lease Mechanism

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  name: node-1
  namespace: kube-node-lease
spec:
  holderIdentity: node-1
  renewTime: "2024-06-01T10:00:05.000000Z"

Each node's kubelet renews a corresponding Lease object at a regular interval, a lightweight signal considerably cheaper for the control plane to track at scale than full node status updates, and the node lifecycle controller monitors this lease to determine whether a node is still responsive.

node-monitor-grace-period

kube-controller-manager --node-monitor-grace-period=40s

If a node's lease is not renewed within node-monitor-grace-period, the controller manager marks the node's Ready condition as Unknown, the first step in the failure detection sequence, distinct from and preceding any pod eviction decision.

Taint-Based Eviction Timing

Automatic Taints on Node Trouble

taints:
  - key: node.kubernetes.io/unreachable
    effect: NoExecute
  - key: node.kubernetes.io/not-ready
    effect: NoExecute

Once a node's condition transitions to Unknown or False, the node lifecycle controller automatically applies the corresponding NoExecute taint, and any pod without a matching toleration is scheduled for eviction after the toleration's default grace period, commonly 300 seconds unless explicitly overridden.

tolerations:
  - key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300

The Split-Brain Risk

Why a Delay Exists Before Rescheduling

The delay between detecting a node as unreachable and actually evicting and rescheduling its pods exists specifically because "unreachable" is ambiguous: the node might have genuinely crashed, or it might merely be network-partitioned from the control plane while its workloads continue running and holding whatever exclusive resources or leadership roles they had; rescheduling too aggressively risks two copies of the same logical workload running simultaneously, a serious correctness hazard for any workload assuming exclusive ownership of shared state.

StatefulSet Pods and Forceful Deletion

kubectl delete pod db-0 --grace-period=0 --force

For StatefulSet pods specifically, Kubernetes refuses to consider a pod on an unreachable node fully deleted, and therefore refuses to create its replacement, until the pod object is explicitly confirmed gone, since a StatefulSet's stable identity guarantee means creating a replacement while the original might still be running elsewhere risks two pods with the same identity active simultaneously; an operator manually force-deleting the pod object accepts this specific split-brain risk explicitly, which is why force deletion of StatefulSet pods requires deliberate, informed action rather than being automated by default.

Node Problem Detection

Surfacing Sub-Node-Failure Degradation

apiVersion: v1
kind: Event
reason: KernelDeadlock

The Node Problem Detector component (typically run as a DaemonSet) monitors for hardware and kernel-level issues, memory errors, filesystem corruption, kernel deadlocks, that fall short of full node unresponsiveness but still degrade reliability, surfacing them as node conditions or taints that scheduling and eviction logic can react to, extending failure tolerance to cover degraded-but-technically-responsive nodes, not only fully unreachable ones.

Tuning the Detection-to-Recovery Timeline

Balancing Fast Recovery Against False-Positive Risk

node-monitor-grace-period: 40s
pod-eviction-timeout: 300s (legacy) / tolerationSeconds

Shortening these timers speeds up recovery from genuine node failures but increases the risk of prematurely evicting and rescheduling pods from a node experiencing only a transient network blip, wasting resources on unnecessary rescheduling and, for stateful workloads, elevating split-brain risk; the appropriate values depend on the network reliability of the specific infrastructure the cluster runs on and the workload's own tolerance for either outcome.

Relationship to Disruption Budget Basics and the Reliability Model

Node failure tolerance is the involuntary-disruption counterpart to the voluntary-disruption mechanics covered under disruption budget basics, together forming the complete picture of how Kubernetes handles both planned and unplanned node-level failure; the deliberate delay and split-brain awareness built into this mechanism is a direct application of the reliability model's broader recognition that fast recovery must be balanced against the risk of acting on an ambiguous or incomplete failure signal.