4.19 Kubernetes Node Scheduling State

Kubernetes Node Scheduling State is the set of attributes on a node object that determine whether the scheduler will consider that node eligible to receive new pods, encompassing the unschedulable flag, taints applied to the node, and the interplay between manual administrative actions and automated controller-driven changes that together gate a node's participation in future scheduling decisions.

The Unschedulable Flag

Purpose of Cordoning

A node's spec.unschedulable field, when set to true, marks the node as cordoned, instructing the scheduler to exclude it from consideration for any new pod placements, while explicitly leaving existing pods already running on the node untouched and still operational.

Administrative Use Cases

Cordoning is commonly used as a preparatory step before planned maintenance, allowing an operator to stop new workloads from landing on a node before manually or automatically draining the pods that are already there, giving a controlled, two-phase approach to node maintenance.

Automatic Cordoning

Beyond manual administrative action, certain automated processes, such as node problem detectors identifying hardware issues, or cluster autoscalers marking a node for scale-down, can also set the unschedulable flag programmatically as part of their own remediation or scaling workflows.

Taints as Scheduling State

Taint Structure

A taint consists of a key, a value, and an effect, one of NoSchedule, PreferNoSchedule, or NoExecute, attached to a node to repel pods that do not carry a matching toleration, forming a complementary mechanism to the unschedulable flag with finer-grained control.

NoSchedule Effect

A NoSchedule taint prevents the scheduler from placing new pods on the node unless those pods carry a toleration for that specific taint, while pods already running on the node before the taint was applied are left alone.

NoExecute Effect

A NoExecute taint goes further, actively evicting already-running pods that lack a matching toleration, in addition to blocking new pod placement, making it the more disruptive of the two primary scheduling-blocking effects.

PreferNoSchedule Effect

PreferNoSchedule expresses a soft preference rather than a hard constraint; the scheduler will attempt to avoid placing pods without a matching toleration on the tainted node but will do so anyway if no better-fitting node is available.

Automatically Applied Condition-Based Taints

Taint-Condition Coupling

The node lifecycle controller automatically applies specific taints in direct response to node conditions, such as applying a node.kubernetes.io/not-ready taint with NoExecute effect when the Ready condition becomes False, converting condition state into an actionable scheduling signal.

Memory, Disk, and PID Pressure Taints

Corresponding taints exist for MemoryPressure, DiskPressure, and PIDPressure conditions, each automatically applied and removed by the node lifecycle controller as the underlying condition changes, ensuring scheduling state stays synchronized with actual node health without requiring manual intervention.

Toleration Seconds for Graceful Handling

Pods can specify a tolerationSeconds value alongside a toleration for a NoExecute taint, allowing them to remain running on a node for a bounded grace period after the taint is applied rather than being evicted immediately, giving workloads a chance to react to transient node issues.

Interaction With the Scheduler

Filtering During Scheduling

During the filtering phase of scheduling, the scheduler checks each candidate node's taints and the node's unschedulable flag as part of its predicate evaluation, excluding any node that fails these checks from further consideration for the pod being scheduled.

DaemonSet Pod Exceptions

DaemonSet-managed pods are commonly configured with tolerations for many standard taints, including not-ready and unreachable, since DaemonSet pods are often expected to run on every node regardless of transient health issues, such as logging or monitoring agents.

Draining and Scheduling State Together

The Drain Workflow

A typical node drain operation first cordons the node by setting it unschedulable, then evicts existing pods using the eviction API, which respects PodDisruptionBudgets, ensuring pods are moved off gracefully rather than abruptly while new pods are simultaneously prevented from landing on the node mid-drain.

Uncordoning

Once maintenance is complete, an operator uncordons the node by clearing the unschedulable flag, restoring it as a normal candidate for scheduling, though any pods that were evicted during the drain do not automatically return to that specific node since scheduling decisions are made fresh for each new pod.

Observability of Scheduling State

Inspecting Current State

The kubectl get nodes output includes a SCHEDULING DISABLED indicator for cordoned nodes, while kubectl describe node surfaces the complete list of currently applied taints, giving operators a direct view into why a given node might not be receiving new pods.