Kubernetes Node Metrics Observation

Kubernetes Node Metrics Observation covers retrieving and interpreting resource usage data at the level of an entire node, aggregating across every pod scheduled on it plus the node's own system overhead, distinct from per-pod or per-container metrics and essential for understanding whether a node itself — rather than any single workload — is the source of a performance or capacity problem.

The NodeMetrics API Object

Structure and Source

Analogous to PodMetrics, the NodeMetrics object served through the metrics.k8s.io API reports a node's current CPU and memory usage as a single aggregate figure, sourced from the kubelet's own resource accounting (via cAdvisor and the kubelet's Summary API) rather than summed independently from individual pod metrics.

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes/worker-3"

{
  "metadata": { "name": "worker-3" },
  "timestamp": "2024-01-15T10:32:00Z",
  "window": "30s",
  "usage": { "cpu": "3200m", "memory": "10240Mi" }
}

kubectl top node

kubectl top node renders this data as a human-readable table, additionally showing usage as a percentage of the node's allocatable capacity, giving an immediate sense of how close a node is to its practical resource ceiling.

kubectl top nodes
NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
worker-3   3200m        80%    10240Mi         64%

Node Usage Versus Capacity and Allocatable

Capacity, Allocatable, and Requested

A node's capacity is its total physical resource, allocatable subtracts reservations for system daemons and the kubelet itself, and current usage (from NodeMetrics) is what is actually being consumed at the moment of measurement — distinct again from the sum of pod resource requests, which represents reserved (not necessarily used) capacity.

kubectl describe node worker-3
# Capacity:    cpu: 4, memory: 16Gi
# Allocatable: cpu: 3800m, memory: 15Gi

Requested Versus Actual Usage Gaps

A node can show low actual usage via NodeMetrics while simultaneously having little remaining schedulable capacity, if pods on that node have requested resources far exceeding what they currently use — a common pattern that node usage metrics alone do not reveal without also examining aggregate pod requests against allocatable capacity.

kubectl describe node worker-3 | grep -A 5 "Allocated resources"

Diagnosing Node-Level Resource Pressure

Node Conditions as a Complementary Signal

MemoryPressure, DiskPressure, and PIDPressure node conditions indicate the kubelet itself has determined the node is under resource strain, a distinct and more authoritative signal than simply observing high usage percentages via NodeMetrics, since these conditions directly trigger eviction and scheduling behavior.

kubectl describe node worker-3 | grep -A 10 Conditions

Detailed Node-Level Metrics Beyond the Summary API

Prometheus-scraped node-exporter metrics (or the kubelet's own detailed metrics endpoint) provide far more granular node-level detail — per-CPU utilization, disk I/O latency, network interface statistics, filesystem usage per mount — than the simple aggregate NodeMetrics object, necessary for diagnosing node-level issues beyond a basic "is this node busy" check.

node_memory_MemAvailable_bytes{instance="worker-3"}
node_disk_io_time_seconds_total{instance="worker-3"}

Using Node Metrics for Cluster-Wide Analysis

Identifying Imbalanced Load Across Nodes

Comparing NodeMetrics across every node in a cluster surfaces load imbalance — some nodes running hot while others sit comparatively idle — which may point to suboptimal scheduling constraints (overly restrictive affinity rules, uneven taints) rather than genuine aggregate capacity shortage.

kubectl top nodes --sort-by=cpu

Informing Cluster Autoscaler and Capacity Decisions

Sustained high node-level usage across most or all nodes in a cluster, correlated with the Cluster Autoscaler's own scaling activity (or lack thereof), helps confirm whether current node group sizing and scaling thresholds are well matched to the cluster's actual demand pattern, complementing the pod-level signals that drive HPA and VPA decisions.