4.4 Kubernetes kubelet Runtime Coordination

Kubernetes kubelet Runtime Coordination is the set of mechanisms by which the kubelet, the primary node agent in a Kubernetes cluster, communicates with and directs the container runtime to create, monitor, and terminate the containers that make up scheduled pods. It functions as the operational bridge between the declarative pod specifications stored in the API server and the actual containers running on a worker node, translating desired state into runtime actions and continuously reconciling observed state back to the control plane.

The Role of the kubelet in Node Operations

Node Agent Responsibilities

The kubelet runs as a persistent process on every worker node and is responsible for ensuring that containers described in PodSpecs are running and healthy. It watches the API server for pods assigned to its node, then works with the configured container runtime to bring those pods to the desired running state. Beyond container lifecycle management, the kubelet reports node conditions, resource capacity, and pod status back to the control plane, making it a critical feedback loop in cluster-wide scheduling decisions.

Position in the Node Architecture

The kubelet does not run containers directly. Instead, it delegates the actual container operations to a container runtime through a standardized interface, while retaining responsibility for higher-level orchestration tasks such as volume mounting, secret and configmap injection, network namespace setup coordination, and pod-level cgroup management.

The Container Runtime Interface

Purpose of CRI

The Container Runtime Interface, or CRI, is a plugin interface that decouples the kubelet from any specific container runtime implementation. Before CRI existed, the kubelet contained runtime-specific code paths, which made supporting multiple runtimes difficult and fragile. CRI defines a gRPC protocol with two primary services, RuntimeService and ImageService, that any conforming runtime must implement.

RuntimeService and ImageService

RuntimeService handles pod sandbox and container lifecycle operations, including creating and destroying pod sandboxes, starting and stopping containers, executing commands inside containers, and streaming logs. ImageService handles pulling, listing, and removing container images independently of runtime state.

Runtime Implementations

Modern clusters typically use containerd or CRI-O as the runtime implementation behind the CRI interface. Both expose the CRI gRPC socket, commonly located at a Unix domain socket path such as /run/containerd/containerd.sock or /var/run/crio/crio.sock, which the kubelet connects to on startup.

Pod Sandbox Coordination

The Sandbox Concept

Before any application container is started, the runtime creates a pod sandbox, an isolated environment that establishes the shared Linux namespaces, most notably the network namespace, that all containers in the pod will share. The kubelet requests sandbox creation first, then creates individual containers within that sandbox context.

Network Namespace Setup

Sandbox creation triggers invocation of a CNI plugin, which configures the pod's network interface, assigns an IP address, and sets up routing. The kubelet coordinates this indirectly by instructing the runtime to create the sandbox; the runtime, in turn, invokes CNI plugins according to the CNI configuration present on the node.

Sandbox Lifecycle States

A sandbox can exist in states such as ready, not ready, or unknown, and the kubelet tracks these states to determine whether containers can be safely started, restarted, or must be recreated from scratch if the sandbox itself has failed.

Container Lifecycle Management

Create, Start, Stop, Remove

For each container defined in a pod's spec, the kubelet issues CRI calls to create the container within the sandbox, start it, and later stop and remove it when the pod is deleted or the container needs to be replaced. These operations are idempotent from the kubelet's perspective, allowing it to safely retry after transient failures.

Restart Policies

The kubelet enforces the pod's restartPolicy, which can be Always, OnFailure, or Never. When a container exits, the kubelet consults this policy along with an internal exponential backoff mechanism to decide whether and when to restart the container, preventing crash-looping containers from overwhelming the node.

Init Containers and Ordering

Init containers are coordinated sequentially before any regular application container is started. The kubelet waits for each init container to complete successfully before proceeding to the next, and only starts the main containers once all init containers have finished.

Health Monitoring and Probes

Liveness, Readiness, and Startup Probes

The kubelet periodically executes liveness, readiness, and startup probes against running containers, using HTTP requests, TCP socket checks, gRPC calls, or exec commands inside the container. Liveness probe failures cause the kubelet to restart the container, readiness probe failures remove the pod from service endpoints without restarting it, and startup probes gate the execution of the other two probe types during slow-starting applications.

PLEG: Pod Lifecycle Event Generator

The kubelet runs an internal component historically known as PLEG, which periodically relists container states from the runtime and generates internal events when state changes are detected, rather than requiring a full sync of all pods on every iteration. This reduces the polling overhead of tracking potentially hundreds of containers per node while still keeping the kubelet's internal view of container state current.

Resource and cgroup Coordination

QoS Classes and cgroup Hierarchies

The kubelet organizes pods into cgroup hierarchies based on their Quality of Service class, Guaranteed, Burstable, or BestEffort, which is derived from the resource requests and limits specified in the pod spec. This hierarchy determines eviction priority and resource contention behavior under node pressure.

CPU and Memory Enforcement

Through the runtime, the kubelet ensures that CPU shares, CPU quotas, and memory limits declared in container resource specifications are translated into corresponding cgroup settings, whether using cgroup v1 or cgroup v2, depending on the host operating system configuration.

Node Pressure Eviction

When a node experiences resource pressure, such as low available memory or disk space, the kubelet coordinates with the runtime to evict lower-priority pods, reclaiming resources by terminating and removing containers according to QoS class and eviction thresholds configured on the node.

Status Reporting and Reconciliation

Syncing Pod Status

The kubelet continuously reconciles the actual state of containers, as reported by the runtime, with the desired state described in the PodSpec. Discrepancies trigger corrective actions, such as restarting a failed container or recreating a sandbox that has been destroyed unexpectedly.

Reporting to the API Server

Observed container and pod statuses are periodically written back to the API server as pod status updates, which include container states, restart counts, and condition transitions. This status information feeds into higher-level controllers, such as ReplicaSet and Deployment controllers, that rely on accurate node-level state to make scheduling and scaling decisions.

Node Status Heartbeats

Separately from pod status, the kubelet sends periodic node heartbeats to the API server, indicating that the node itself is healthy and reachable. Missed heartbeats beyond a configured threshold cause the node to be marked as unreachable, which can trigger pod eviction and rescheduling onto healthy nodes.