Kubernetes Metric Observation

Kubernetes Metric Observation is the practice of retrieving and interpreting numeric, time-series measurements describing cluster, node, and workload behavior, spanning the built-in resource metrics served through Metrics Server, the richer metrics exposed by cluster components and applications through Prometheus-format endpoints, and the queries and dashboards built on top of a full metrics pipeline.

Immediate, Point-in-Time Observation

kubectl top for Quick Checks

kubectl top nodes and kubectl top pods provide an immediate snapshot of current CPU and memory usage sourced from Metrics Server, useful for a quick check of resource consumption without needing to open a dashboard or write a query.

kubectl top nodes
kubectl top pods -n payments --containers

Limitations of Point-in-Time Snapshots

Because kubectl top reflects only the current moment, it cannot reveal trends, historical spikes, or correlate a resource anomaly with a specific point in time unless the operator happens to be watching at exactly the right moment — for anything beyond an immediate spot-check, a proper metrics pipeline with historical storage is necessary.

Component and Workload Metrics Endpoints

The Prometheus Exposition Format

Kubernetes control plane components, kubelets, and instrumented applications commonly expose a /metrics HTTP endpoint in Prometheus's plain-text exposition format, listing named metrics with associated labels and current values, which a Prometheus server (or compatible scraper) polls at a configured interval.

# HELP apiserver_request_duration_seconds Request latency
apiserver_request_duration_seconds_bucket{verb="GET",resource="pods",le="0.1"} 4521

Discovering and Scraping Targets

In a cluster running Prometheus, ServiceMonitor or PodMonitor custom resources declare which endpoints to scrape and at what interval, automating target discovery so that newly deployed workloads with a correctly labeled metrics endpoint are picked up without manual Prometheus configuration changes.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: api-service
spec:
  selector:
    matchLabels:
      app: api-service
  endpoints:
  - port: metrics

Querying Historical Metrics

PromQL for Aggregation and Trend Analysis

Once metrics are stored in a time-series database, a query language like PromQL allows aggregating across labels, computing rates from counters, and comparing current values against historical baselines, forming the basis for both dashboards and alerting rules.

rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])

Building Dashboards From Queries

Dashboarding tools render saved queries as graphs and panels, giving a persistent, shareable view of key metrics that does not require re-typing a query every time — well-designed dashboards organize related metrics (latency, error rate, saturation) together to support rapid situational assessment during an incident.

Metric Observation for Specific Diagnostic Needs

Resource Saturation

CPU throttling metrics, memory usage approaching limits, and disk I/O wait times reveal whether a workload or node is resource-constrained, information that kubectl top's simple usage snapshot does not directly convey (CPU throttling in particular is invisible to kubectl top, which shows usage but not whether that usage was artificially capped).

rate(container_cpu_cfs_throttled_periods_total[5m])

Golden Signal Metrics for Services

Latency, traffic, errors, and saturation — commonly referred to as the four golden signals — form a standard, minimal set of metrics worth instrumenting for any service-oriented workload, giving a consistent baseline view applicable across many different applications regardless of their specific internal behavior.

Correlating Metrics With Other Signals

Pivoting From a Metric Anomaly to Root Cause

A metric observation revealing an anomaly (elevated latency, increased error rate) is typically the starting point of an investigation rather than its conclusion; the next step usually involves pivoting to traces covering the affected time window, then to the specific logs those traces point to, using shared labels or trace identifiers to connect the three signal types.

Avoiding Metric-Only Tunnel Vision

Because metrics aggregate across many individual events, they can obscure which specific requests or instances are actually responsible for an observed trend; treating a metric anomaly as requiring further investigation through logs or traces, rather than as a complete diagnosis on its own, avoids drawing conclusions the aggregate data alone cannot support.