Kubernetes HPA Pod Metric Scaling

Kubernetes HPA Pod Metric Scaling is the use of the Pods metric type within a HorizontalPodAutoscaler, which scales based on a custom, application-specific metric measured per pod and averaged across all pods belonging to the target workload, distinct from the built-in Resource type's reliance on CPU and memory alone. This metric type is the primary mechanism for scaling on signals that better represent actual application load — requests handled, messages processed, active connections — than generic resource consumption.

How Pods Metrics Work

Per-Pod Values Averaged Across the Target

A Pods metric expects each pod belonging to the HPA's target to expose a value for the named metric, which the controller retrieves through the custom metrics API, sums, and divides by the number of pods to compute an average — the target is always expressed as an AverageValue, since a per-pod value naturally scales with replica count in a way an absolute total would not.

metrics:
- type: Pods
  pods:
    metric:
      name: http_requests_per_second
    target:
      type: AverageValue
      averageValue: "50"

The Custom Metrics API Requirement

Unlike Resource metrics, which the metrics-server provides natively, Pods metrics require a custom metrics API adapter — commonly backed by Prometheus through the Prometheus Adapter — registered as an aggregated API server extension, translating metric queries into whatever monitoring system actually stores the underlying data.

kubectl get apiservices | grep custom.metrics.k8s.io

Selecting an Appropriate Application Metric

Choosing Metrics That Correlate With Actual Load

A useful Pods metric should scale roughly linearly with the resource cost of handling additional load — requests per second for a stateless web service, active worker threads for a job processor — so that averaging it across replicas and comparing against a target produces a meaningful scaling signal, unlike a metric that stays flat regardless of load (which would never trigger scaling) or one that is extremely noisy (which would cause erratic scaling).

Metric Label Selectors

The metric.selector field allows filtering which time series the adapter returns for a given metric name, useful when the same metric name is emitted with different label combinations across unrelated workloads and only a specific subset should feed a particular HPA's calculation.

metrics:
- type: Pods
  pods:
    metric:
      name: http_requests_per_second
      selector:
        matchLabels:
          endpoint: /api/v1/orders
    target:
      type: AverageValue
      averageValue: "100"

Implementing the Metrics Pipeline

Instrumenting the Application

The workload itself must expose the chosen metric in a form the monitoring system can scrape, typically a Prometheus-format /metrics endpoint incrementing a counter or reporting a gauge value that genuinely reflects current load rather than a cumulative total that only ever increases.

http_requests_per_second{pod="api-service-7d4f9-x2k1p"} 42.3

Configuring the Metrics Adapter

The Prometheus Adapter (or equivalent) requires explicit rule configuration mapping a Kubernetes metric name to a Prometheus query, associating the result with the correct pods through Kubernetes discovery labels so the custom metrics API can correctly attribute values back to individual pods belonging to the HPA's target.

rules:
- seriesQuery: 'http_requests_per_second{namespace!="",pod!=""}'
  resources:
    overrides:
      namespace: {resource: "namespace"}
      pod: {resource: "pod"}
  name:
    matches: "^(.*)_per_second"
    as: "${1}_per_second"
  metricsQuery: 'avg(rate(<<.Series>>[2m])) by (<<.GroupBy>>)'

Operational Considerations

Metric Freshness and Staleness

Because Pods metrics pass through an additional adapter layer querying a monitoring backend, the freshness of the underlying data (scrape interval, query aggregation window) adds to the overall control-loop delay beyond what a directly-collected resource metric would incur, which should be accounted for when tuning stabilization windows.

Validating the Full Pipeline Before Relying on It

Testing that kubectl get --raw against the custom metrics API returns sensible values for the target pods, independent of the HPA itself, isolates whether an unexpected scaling behavior originates from the metrics pipeline (missing data, incorrect aggregation) or from the HPA's own calculation and configuration.

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/payments/pods/*/http_requests_per_second"