Kubernetes Metrics API Observation

Kubernetes Metrics API Observation refers to the practice and infrastructure of collecting, exposing, and querying real-time resource usage data — primarily CPU and memory consumption — for nodes and pods within a Kubernetes cluster, through the metrics.k8s.io aggregated API group. It is the mechanism by which lightweight, ephemeral, in-memory resource metrics become available to cluster components and operators without relying on a full monitoring stack.

Architectural Foundations

The Metrics API Group

The Metrics API is not part of the Kubernetes core API server. It is registered as an aggregated API under the group metrics.k8s.io, exposing two primary resource kinds:

• NodeMetrics, reachable via /apis/metrics.k8s.io/v1beta1/nodes
• PodMetrics, reachable via /apis/metrics.k8s.io/v1beta1/pods

This aggregation model means the API server does not compute or store these metrics itself. Instead, it forwards requests to a registered APIService that is backed by a separate component, most commonly metrics-server.

Metrics Server

metrics-server is a cluster add-on that collects resource usage data from the kubelet's /stats/summary (or /metrics/resource) endpoint on every node, aggregates it, and serves it through the Metrics API. Key characteristics:

• It is a cluster-wide singleton aggregator, typically deployed with one or two replicas for high availability.
• It stores metrics in memory only, with no persistence layer or long-term retention.
• It scrapes nodes on a fixed interval (default 60 seconds), meaning observed values are near-real-time snapshots, not continuous time series.
• It requires network connectivity from the metrics-server pod to the kubelet's secure port (10250) on every node, and correct TLS/certificate trust configuration, which is a common source of operational failure.

Data Flow

Collection Path

1. The kubelet on each node runs cAdvisor (or the equivalent CRI stats interface) to gather container-level CPU and memory usage.
2. metrics-server periodically queries each kubelet's summary API.
3. Raw values are aggregated per pod (summing container metrics) and per node.
4. Aggregated values are cached in memory and exposed through the metrics.k8s.io API.

Consumption Path

Once available through the Metrics API, resource usage data is consumed by multiple layers of the cluster:

• kubectl top nodes and kubectl top pods query the API directly for human-readable inspection.
• The Horizontal Pod Autoscaler (HPA) controller polls the Metrics API to compute current-to-target resource utilization ratios and adjust replica counts.
• The Vertical Pod Autoscaler (VPA), when installed, can use the same API as one of its recommendation input sources.
• Dashboards such as the Kubernetes Dashboard render live utilization figures sourced from this API.

Querying the Metrics API

Command-Line Observation

kubectl top nodes

kubectl top pods --all-namespaces

kubectl top pods --containers -n production

Direct API Access

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/production/pods"

The raw response is a JSON document listing usage.cpu (in nanocores, typically rendered as n) and usage.memory (in kibibytes, rendered as Ki) for each observed object, alongside a timestamp and window field indicating the measurement interval.

Deployment Considerations

Installation

apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      containers:
        - name: metrics-server
          image: registry.k8s.io/metrics-server/metrics-server:v0.7.2
          args:
            - --cert-dir=/tmp
            - --secure-port=4443
            - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
            - --kubelet-use-node-status-port
            - --metric-resolution=15s

Common Operational Failure Modes

• Untrusted kubelet certificates: managed or self-signed clusters frequently require --kubelet-insecure-tls during bootstrap, which trades certificate validation for connectivity and should be scoped narrowly.
• Network policy isolation: if NetworkPolicy resources restrict traffic to the kube-system namespace, metrics-server may be unable to reach kubelets, silently producing "metrics not available" errors.
• API service registration failures: if the APIService object for v1beta1.metrics.k8s.io is not marked Available, kubectl top returns an error rather than stale data, since there is no fallback path.
• Resource starvation: under CPU or memory pressure, metrics-server itself may be throttled, causing scrape timeouts and gaps in the exposed metrics window.

Relationship to the Broader Observability Stack

Kubernetes Metrics API Observation is intentionally minimal. It answers "how much CPU and memory is being used right now" but does not provide:

• Historical trends or long-term storage
• Custom application-level metrics
• Alerting or query languages

These capabilities are delegated to full monitoring systems such as Prometheus paired with kube-state-metrics and node exporters, which expose a richer custom.metrics.k8s.io or external.metrics.k8s.io API for advanced autoscaling scenarios. The core Metrics API remains the baseline layer that guarantees HPA and kubectl top function even in clusters without a dedicated monitoring stack installed.