Kubernetes Resource Usage Observation

Kubernetes Resource Usage Observation is the practice of measuring how much CPU, memory, disk, and network capacity nodes and pods are actually consuming relative to what has been requested, limited, or is physically available, forming the basis for capacity planning, right-sizing resource requests, and diagnosing performance problems rooted in resource contention rather than application logic.

Sources of Resource Usage Data

cAdvisor and the Kubelet

Each node's kubelet embeds cAdvisor, which collects per-container resource usage directly from the kernel (via cgroups) and exposes it both through the kubelet's own metrics endpoint and, aggregated, through Metrics Server for cluster-wide queries.

kubectl top pod api-service-7d4f9 --containers

Container-Level Metrics for Deeper Analysis

Beyond the summarized values kubectl top shows, the kubelet's /metrics/cadvisor endpoint (typically scraped by Prometheus) exposes detailed container-level metrics — CPU throttling, memory working set versus RSS, network bytes sent and received — enabling analysis well beyond simple current usage snapshots.

container_memory_working_set_bytes{pod="api-service-7d4f9",container="app"}
container_cpu_cfs_throttled_periods_total{pod="api-service-7d4f9",container="app"}

Comparing Usage Against Requests and Limits

Requests as the Scheduling Baseline

A pod's resource requests determine how the scheduler allocates node capacity, and comparing actual usage against requested values reveals whether requests are set too high (wasting reservable capacity that could host other workloads) or too low (risking resource contention if actual usage regularly exceeds what was reserved).

container_memory_working_set_bytes / on(pod) kube_pod_container_resource_requests{resource="memory"}

Limits and Throttling

CPU limits, when set, cause the kernel's CFS scheduler to throttle a container's CPU access once it exceeds its allotted share within a scheduling period, which can degrade performance even when overall node CPU appears to have headroom — a distinct phenomenon from memory limits, which instead trigger termination (OOMKilled) rather than throttling when exceeded.

rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m])

Node-Level Resource Observation

Allocatable Versus Capacity

A node's capacity represents its total physical resources, while allocatable subtracts resources reserved for the operating system and kubelet itself, and comparing pod resource requests against allocatable (not raw capacity) determines actual available scheduling headroom on a given node.

kubectl describe node worker-3 | grep -A 5 Allocatable

Identifying Resource-Constrained Nodes

Nodes reporting MemoryPressure or DiskPressure conditions, or showing sustained high CPU utilization across all scheduled pods, indicate genuine node-level resource constraint that may require additional node capacity, workload rebalancing, or investigation into an unexpectedly resource-hungry workload.

kubectl top nodes --sort-by=memory

Using Resource Usage Data for Right-Sizing

Informing Manual Resource Request Adjustments

Reviewing sustained usage patterns (ideally over a period covering typical peak load, not just a single snapshot) against currently configured requests directly informs manual adjustments, or serves as validation data for VPA recommendations before trusting them in automatic mode.

Distinguishing Baseline From Peak Usage

Setting resource requests based only on average usage risks under-provisioning during legitimate peak periods, while sizing purely for peak usage can waste capacity during typical operation; reviewing the distribution of usage (not just a single average or maximum) supports a more informed tradeoff between efficiency and headroom.

Disk and Network Usage Observation

Disk I/O and Storage Consumption

Metrics covering persistent volume usage, disk I/O latency, and ephemeral storage consumption reveal storage-related bottlenecks that CPU and memory metrics alone would not surface, particularly relevant for stateful workloads with significant read/write activity.

Network Throughput and Errors

Per-pod and per-node network metrics (bytes transmitted and received, packet drops, connection errors) help distinguish application-level slowness from underlying network capacity or connectivity issues, especially in clusters running a service mesh or CNI plugin whose own overhead can itself become a resource consideration.