4.13 Kubernetes Node Resource Accounting

Kubernetes Node Resource Accounting is the process by which a node tracks, computes, and reports the compute resources it has available, the resources currently claimed by scheduled pods, and the resources actually being consumed at runtime, forming the basis for scheduling decisions, admission checks, and eviction behavior. It spans capacity discovery at node startup, ongoing allocation bookkeeping by the kubelet, and live usage measurement through cgroup and cAdvisor integration.

Capacity and Allocatable Resources

Node Capacity Discovery

When the kubelet starts, it inspects the underlying host to determine total capacity for CPU, memory, ephemeral storage, and any extended resources such as GPUs, reporting these values as the node's capacity in its status object.

Computing Allocatable Resources

From total capacity, the kubelet subtracts reservations for system daemons and Kubernetes components themselves, configured through --system-reserved and --kube-reserved flags, along with any configured eviction hard thresholds, to compute the allocatable value, which represents the resources actually available for scheduling pods.

Reservation Rationale

Reserving a portion of node capacity for the operating system and kubelet itself prevents pod workloads from starving critical node processes of CPU or memory, which could otherwise destabilize the entire node and every pod running on it.

Request and Limit Bookkeeping

Tracking Scheduled Requests

The kubelet maintains a running total of resource requests from all pods currently admitted to the node. This total is compared against allocatable capacity during node-level pod admission, ensuring the sum of requests never exceeds what the node has declared it can provide.

Requests Versus Limits

Requests represent the resource amount a container is guaranteed, used both for scheduling decisions and cgroup share configuration, while limits represent an upper bound a container cannot exceed, enforced through cgroup quota and OOM-related mechanisms; a container may be admitted based on its request even if its limit, if fully used by every pod on the node simultaneously, would exceed capacity.

Overcommit Implications

Because scheduling is based on requests rather than limits, nodes can be overcommitted with respect to limits, meaning actual peak usage across all pods, if every container used up to its limit simultaneously, could exceed the node's physical capacity, a deliberate tradeoff that improves average utilization at the cost of contention risk under peak load.

Extended Resources and Device Plugins

Advertising Extended Resources

Devices such as GPUs, FPGAs, or specialized NICs are advertised to the API server as extended resources through the device plugin framework, in which a device plugin running on the node reports available device counts to the kubelet, which then folds these into the node's capacity and allocatable fields.

Device Allocation Bookkeeping

When a pod requests an extended resource, the kubelet's device manager tracks which specific device instances have been allocated to which containers, ensuring the same physical device is not double-allocated to two different containers simultaneously.

Live Usage Measurement

cAdvisor Integration

The kubelet embeds cAdvisor, a resource usage and performance monitoring component, to collect real-time statistics on CPU, memory, filesystem, and network usage for every container and pod on the node, reading directly from cgroup accounting files and other kernel interfaces.

The Summary API

Collected usage statistics are exposed through the kubelet's Summary API, an HTTP endpoint that tools such as the Metrics Server and Vertical Pod Autoscaler query to obtain per-pod and per-node resource usage figures, distinct from the request and limit values declared in pod specs.

cgroup Statistics as the Source of Truth

Because cgroups are the kernel mechanism actually enforcing resource boundaries, the statistics exposed through cgroup interface files, such as memory current usage or CPU accumulated usage counters, serve as the authoritative source for live consumption figures reported by the kubelet.

Accounting Under Resource Pressure

Eviction Threshold Monitoring

The kubelet continuously compares live usage figures against configured eviction thresholds for signals such as available memory or available node filesystem space, using this accounting data to decide when to begin evicting pods to relieve pressure before the node becomes fully unusable.

QoS-Aware Reclamation Order

When accounting data indicates a need to reclaim resources, the kubelet uses QoS class and how far each pod's actual usage exceeds its requests to rank pods for eviction, generally reclaiming from BestEffort and over-limit Burstable pods before touching Guaranteed pods that are within their declared requests.

Node Status Reporting Cadence

Periodic Status Updates

The kubelet periodically publishes updated capacity, allocatable, and condition information to the node's status object in the API server, at an interval distinct from the more frequent node heartbeat lease mechanism used purely to signal node liveness.

Consumers of Accounting Data

Scheduler filtering plugins, cluster autoscalers, and monitoring dashboards all consume the accounting data published in node status and the Summary API, making accurate and timely resource accounting a foundational dependency for correct cluster-wide capacity management.