Kubernetes Container Metrics Observation

Kubernetes Container Metrics Observation covers the detailed, cgroup-derived metrics cAdvisor collects for every individual container — distinct from the summarized PodMetrics API used by kubectl top and the HPA — exposing a much richer set of CPU, memory, network, and filesystem measurements through the kubelet's Prometheus-format metrics endpoint, intended for deep diagnostic analysis rather than quick inspection.

Where Detailed Container Metrics Come From

cAdvisor and cgroups

cAdvisor, embedded in the kubelet, reads container resource usage directly from the Linux cgroup filesystem, exposing dozens of distinct metrics per container far beyond the simple current CPU and memory figures PodMetrics summarizes, scraped by Prometheus (or a compatible system) from the kubelet's /metrics/cadvisor endpoint.

curl -s https://node-ip:10250/metrics/cadvisor | grep container_memory

Memory Metric Semantics

Working Set Versus RSS Versus Cache

container_memory_working_set_bytes is the metric Kubernetes uses for OOM-related decisions and is what typically matters most for capacity planning, representing memory actively in use; container_memory_rss reflects resident set size excluding some cache pages; container_memory_cache reflects page cache that can usually be reclaimed under pressure without directly causing application failure — conflating these can lead to misinterpreting a container's genuine memory pressure.

container_memory_working_set_bytes{pod="api-service-7d4f9",container="app"}

Why Working Set Is the Relevant OOM Signal

Because the kernel's OOM killer and Kubernetes' own eviction logic act based on working set rather than raw RSS or cache, monitoring and alerting on working set (rather than a metric that includes reclaimable cache) more accurately predicts when a container is actually at risk of being killed for memory pressure.

CPU Metric Semantics

Usage Versus Throttling

container_cpu_usage_seconds_total is a cumulative counter of CPU time consumed, requiring a rate() calculation to derive a meaningful usage figure over a time window; container_cpu_cfs_throttled_periods_total and container_cpu_cfs_periods_total together reveal how often a container was throttled by the kernel's CFS scheduler due to hitting its configured CPU limit, a distinct phenomenon from raw usage that kubectl top does not surface at all.

rate(container_cpu_usage_seconds_total{container="app"}[5m])
rate(container_cpu_cfs_throttled_periods_total{container="app"}[5m])
  / rate(container_cpu_cfs_periods_total{container="app"}[5m])

Diagnosing Throttling-Induced Latency

A container showing moderate CPU usage but a high throttling ratio indicates its configured CPU limit is constraining performance during burst periods even though average utilization looks unremarkable, a pattern invisible to simpler usage-only observation and a common cause of intermittent latency that resource-usage dashboards alone might miss.

Network and Filesystem Metrics

Per-Container Network Statistics

container_network_receive_bytes_total, container_network_transmit_bytes_total, and associated error and packet-drop counters reveal per-container network activity, useful for identifying a specific container responsible for unexpected bandwidth consumption or connectivity errors within a multi-container pod.

rate(container_network_receive_bytes_total{pod="api-service-7d4f9"}[5m])

Filesystem Usage and I/O

container_fs_usage_bytes and related I/O metrics track ephemeral storage consumption and disk activity per container, relevant for workloads writing significant temporary data or logs to a container's writable layer rather than a dedicated volume.

Practical Use of Detailed Container Metrics

Deep-Dive Diagnosis Beyond Summary Views

While kubectl top and PodMetrics are sufficient for a quick check, genuinely understanding a performance problem's root cause — CPU throttling causing tail latency, a memory leak distinguishable from growing but reclaimable cache, a specific container's network errors within a multi-container pod — requires querying these detailed cAdvisor metrics directly rather than relying on the coarser summary the metrics API exposes.

Building Alerts on Detailed Signals

Alerting rules based on CPU throttling ratio or working-set-relative-to-limit trends catch degradation patterns that simple absolute usage thresholds would miss entirely, since a container can show unremarkable average usage while still suffering meaningfully from periodic throttling or approaching a memory limit.