Kubernetes Pod Metrics Observation

Kubernetes Pod Metrics Observation focuses specifically on the PodMetrics API object served through the metrics.k8s.io aggregated API, the structured data source underlying kubectl top pod and the resource-based Horizontal Pod Autoscaler calculations, distinct from node-level metrics and from richer custom or external metrics sourced through separate API extension points.

The PodMetrics API Object

Structure

A PodMetrics object reports current CPU and memory usage broken down per container within a pod, timestamped to indicate exactly when the underlying measurement was taken, giving both an aggregate pod-level view and the finer container-level detail needed to identify which specific container within a multi-container pod is responsible for a given resource footprint.

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/payments/pods/api-service-7d4f9"

{
  "metadata": { "name": "api-service-7d4f9" },
  "timestamp": "2024-01-15T10:32:00Z",
  "window": "30s",
  "containers": [
    { "name": "app", "usage": { "cpu": "120m", "memory": "256Mi" } },
    { "name": "sidecar-proxy", "usage": { "cpu": "15m", "memory": "48Mi" } }
  ]
}

The window Field

The window field indicates the time span the reported usage was averaged over, which matters when interpreting a value against a target — a short-lived CPU spike averaged over a 30-second window will show a lower peak value than the raw instantaneous usage during the spike itself.

Retrieving Pod Metrics Interactively

kubectl top pod

kubectl top pod queries the PodMetrics API and renders it in a human-readable table, with --containers breaking down the output per container rather than summing to a single pod-level figure.

kubectl top pod api-service-7d4f9 --containers
kubectl top pod -l app=api-service --sum

Sorting and Filtering Across Many Pods

--sort-by orders output by CPU or memory usage, useful for quickly identifying the highest-consuming pods in a namespace during a capacity investigation without needing to inspect each one individually.

kubectl top pods --all-namespaces --sort-by=memory

Pod Metrics as an HPA Data Source

Feeding Resource-Based Scaling Decisions

The same PodMetrics data that kubectl top pod displays is what the HorizontalPodAutoscaler controller queries for Resource and ContainerResource metric types, meaning any gap or staleness in pod metrics availability (a newly started pod not yet reporting data, for instance) directly affects HPA scaling calculations, not just interactive inspection.

Per-Container Attribution for ContainerResource Metrics

The container-level breakdown in PodMetrics is precisely what enables ContainerResource metric scaling to isolate a specific container's usage from the rest of the pod, making the API's per-container structure directly relevant to that HPA feature rather than merely a display convenience.

Limitations Specific to Pod Metrics

No Historical Retention

PodMetrics reflects only the most recent collection window; unlike a full metrics pipeline backed by a time-series database, there is no way to query pod-level resource usage from an hour ago through this API — historical analysis requires a separate monitoring system scraping and storing the same underlying cAdvisor data over time.

Dependency on Metrics Server Availability

Because Metrics Server is what actually implements the metrics.k8s.io API, any gap in Metrics Server's own health or its ability to reach a given node's kubelet directly produces missing or stale PodMetrics data for pods on that node, which should be the first thing checked when kubectl top pod unexpectedly returns no data for specific pods.

kubectl get apiservices v1beta1.metrics.k8s.io

Practical Usage Patterns

Quick Capacity Triage

During an incident or routine check, sorting pod metrics by memory or CPU usage across a namespace quickly surfaces any single pod consuming disproportionately more than its peers, which is often the fastest initial signal pointing toward a specific misbehaving instance worth investigating further through logs or traces.

Validating Resource Request Accuracy

Periodically comparing PodMetrics usage against each pod's configured resource requests, across a representative time sample rather than a single snapshot, is a lightweight way to spot-check whether requests remain reasonably calibrated without needing a full VPA deployment.