4.14 Kubernetes Node Condition Handling

Kubernetes Node Condition Handling is the mechanism by which the kubelet evaluates the health of the node it runs on across several distinct dimensions, publishes the results as structured condition entries on the node object, and reacts internally to condition changes by altering pod admission and eviction behavior. It gives the control plane and cluster operators a standardized, machine-readable view of node health beyond a simple up-or-down signal.

The Node Conditions Array

Structure of a Condition

Each condition in a node's status is represented as an object with a type, a status of True, False, or Unknown, a last transition timestamp, a last heartbeat timestamp, and human-readable reason and message fields, providing both a machine-checkable status and a diagnostic explanation.

Standard Condition Types

Kubernetes defines several standard condition types, including Ready, MemoryPressure, DiskPressure, PIDPressure, and NetworkUnavailable, each reflecting a distinct aspect of node health that the kubelet independently evaluates on its own schedule.

The Ready Condition

Determining Overall Readiness

The Ready condition reflects whether the kubelet considers the node healthy and prepared to accept pods, computed as an aggregate signal that factors in whether the kubelet itself is functioning correctly, whether the container runtime is reachable, and whether critical node-level dependencies are satisfied.

Consequences of Not Ready

When Ready transitions to False or Unknown, the scheduler stops placing new pods on the node, and if the condition persists beyond a configured toleration period, the node controller can begin evicting existing pods so they can be rescheduled onto healthy nodes elsewhere in the cluster.

Resource Pressure Conditions

MemoryPressure

The kubelet sets MemoryPressure to True when available node memory falls below a configured eviction threshold, signaling that the node is at risk of memory exhaustion, which in turn causes the scheduler to avoid placing additional BestEffort pods on the node and triggers the kubelet's eviction manager to begin reclaiming memory from existing pods.

DiskPressure

DiskPressure reflects available disk space on either the node's root filesystem or its dedicated image filesystem falling below configured thresholds, prompting the kubelet to begin image garbage collection and, if pressure continues, evicting pods to free additional disk space.

PIDPressure

PIDPressure indicates the node is running low on available process IDs, a condition that can arise from processes leaking child processes or from a very high density of pods each spawning many threads or subprocesses, and it similarly influences scheduling and eviction decisions.

Network and Runtime Conditions

NetworkUnavailable

NetworkUnavailable is typically set by the cloud controller manager or a networking add-on rather than the kubelet itself, indicating that the node's networking has not yet been correctly configured, which prevents pod scheduling until the condition clears.

Runtime Health Detection

Although not always exposed as a distinct standard condition, the kubelet continuously verifies it can reach the configured container runtime through the CRI socket; sustained failure to do so degrades the Ready condition, since the kubelet cannot fulfill pod lifecycle operations without a functioning runtime connection.

Condition Evaluation and Reporting Loop

Independent Signal Evaluation

Each condition type is evaluated by its own internal check within the kubelet, typically on a periodic interval, comparing live measurements, such as available memory from cAdvisor-collected statistics, against configured threshold values before deciding whether to flip a condition's status.

Heartbeat and Lease Mechanism

Node conditions are refreshed and republished to the API server at a configured status update frequency, while a separate, more lightweight Lease object is renewed at a faster interval purely to signal that the kubelet process itself is alive, decoupling expensive full-status updates from cheap liveness heartbeats.

Node Controller Reaction

The node lifecycle controller in the control plane watches for condition changes and heartbeat staleness across all nodes, applying NoSchedule and NoExecute taints automatically based on condition status, which is the actual mechanism by which pod scheduling and eviction consequences are enforced cluster-wide.

Operator-Facing Visibility

Inspecting Conditions

Node conditions are visible through kubectl describe node, which surfaces the full condition list along with reasons and messages, making them a primary diagnostic starting point when investigating why pods are failing to schedule or why existing pods on a node are being evicted unexpectedly.

Custom Condition Extensions

Beyond the standard set, some node problem detection add-ons publish additional custom condition types to the node object, extending the same condition-based health signaling pattern to cover concerns such as kernel deadlocks or hardware errors detected outside the kubelet's own built-in checks.