✦ For everyone, free.

Practical knowledge for real and everyday life

Home

Kubernetes Node Condition Handling

Kubernetes Node Condition Handling ensures node reliability by monitoring and responding to node status changes in a cluster.

Kubernetes Node Condition Handling is the mechanism by which a node continuously evaluates and reports its own operational health as a structured set of named conditions on its Node object, giving the control plane, the scheduler, and human operators a standardized way to understand what state a node is in beyond a simple binary of up or down. Each condition captures a specific dimension of node health — such as whether the node is ready to accept new Pods, whether it is under memory pressure, or whether its network is functioning — and it is the kubelet running on that node that owns the responsibility for keeping these conditions accurate and current.


The Structure of a Node Condition

Condition Fields

Each entry in a Node's status.conditions list carries a type (such as Ready, MemoryPressure, DiskPressure, PIDPressure, or NetworkUnavailable), a status of True, False, or Unknown, a lastHeartbeatTime marking the last time the condition was checked, a lastTransitionTime marking the last time its status actually changed, and human-readable reason and message fields explaining the current state.

Distinguishing Heartbeat from Transition

The separation between heartbeat time and transition time matters operationally: a condition can be repeatedly reconfirmed as True for hours with the heartbeat updating regularly, while the transition time remains fixed at whatever moment it last actually flipped, which lets observers distinguish "still healthy, still being checked" from "just became healthy."


How the kubelet Determines Conditions

The Ready Condition

The kubelet computes the Ready condition from an aggregate of internal checks, including whether its own runtime and network are functioning, and reports True only when the node is genuinely prepared to run new Pods; a node reporting Ready=False or Ready=Unknown is excluded from scheduling consideration for new Pods by the default scheduler.

Pressure Conditions and the Eviction Manager

MemoryPressure, DiskPressure, and PIDPressure are driven directly by the kubelet's eviction manager, which compares live observed signals — available memory, available filesystem space and inodes, and available process IDs — against configured eviction thresholds, flipping the corresponding condition to True when a threshold is crossed and back to False once the underlying pressure resolves.

NetworkUnavailable

The NetworkUnavailable condition is typically set by a cloud provider integration or network plugin at node registration time and cleared once the node's networking (such as its CNI configuration and routes) is confirmed functional, preventing Pods from being scheduled to a node whose networking has not yet been fully provisioned.


Reporting Conditions to the Control Plane

Periodic Status Updates

The kubelet republishes its full node status, including all conditions, to the API server on a regular interval (its node status update frequency), and this periodic republishing is also what refreshes the lastHeartbeatTime on each condition even when the condition's actual value has not changed.

The Node Lifecycle Controller's Role

The control plane's node lifecycle controller independently monitors these heartbeats; if a node stops updating its status within a configured grace period, the controller marks the Ready condition as Unknown on the node's behalf (since the kubelet itself can no longer be trusted to report accurately) and, after a further grace period, begins evicting Pods from the unreachable node so that controllers can reschedule replacements elsewhere.


Effects of Conditions on Scheduling and Workloads

Taints Derived from Conditions

Several node conditions have corresponding taints that the node lifecycle controller automatically applies and removes in sync with the condition's value — such as a node.kubernetes.io/memory-pressure taint mirroring the MemoryPressure condition — which repels new Pods from scheduling onto a pressured node without requiring the scheduler itself to special-case condition values directly.

Pod Eviction Under Sustained Pressure

Sustained True status on a pressure condition drives the kubelet's own local eviction behavior independently of the control plane, meaning a single node under memory pressure will begin evicting its own BestEffort and over-limit Burstable Pods to relieve that pressure well before any cluster-level rescheduling response occurs.


Custom and Extended Conditions

Node Problem Detector and Custom Conditions

Beyond the built-in conditions the kubelet manages directly, tools such as the Node Problem Detector can report additional custom conditions — for example, reflecting kernel deadlocks or hardware errors detected through host-level monitoring — by writing directly to the Node object's condition list, extending the same structured mechanism to problems the kubelet itself has no native visibility into.