Kubernetes Custom Metrics Usage

Kubernetes Custom Metrics Usage covers the Custom Metrics API itself — the aggregated API extension point, custom.metrics.k8s.io, that allows any application-specific or infrastructure-specific metric to be exposed in a form Kubernetes controllers can consume, distinct from the fixed CPU and memory metrics Metrics Server provides and from the External Metrics API used for signals with no association to Kubernetes objects.

The Custom Metrics API Extension Point

An Aggregated API, Not a Built-In Server

Unlike Metrics Server, which is a specific, well-known deployment, the Custom Metrics API is an extension point that any conforming implementation can register with the Kubernetes aggregation layer; the actual metric data and query logic come entirely from whichever adapter is deployed to fill that role.

kubectl get apiservices | grep custom.metrics.k8s.io

Object-Associated Metrics

Custom metrics are always associated with a Kubernetes object — most commonly pods, but potentially other resource types — distinguishing this API from the External Metrics API, which serves metrics with no inherent tie to any object in the cluster.

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/payments/pods/*/http_requests_per_second" | jq .

The Prometheus Adapter as a Common Implementation

Bridging Prometheus Queries to the Custom Metrics API

The most widely used implementation of this API translates Kubernetes custom metrics queries into PromQL queries against a Prometheus (or Prometheus-compatible) backend, using a rules configuration to map metric names and label selectors onto specific queries.

rules:
- seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
  resources:
    overrides:
      namespace: {resource: "namespace"}
      pod: {resource: "pod"}
  name:
    matches: "^(.*)_total"
    as: "${1}_per_second"
  metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

Rule Configuration as the Primary Maintenance Surface

Because the adapter's behavior is entirely determined by its rules configuration, maintaining and reviewing these rules — ensuring metric names remain stable, queries produce sensible aggregation windows, and label overrides correctly map Prometheus labels to Kubernetes object identifiers — is the primary ongoing operational task for teams relying on custom metrics.

Consuming Custom Metrics

Direct API Queries

Any client with appropriate RBAC permissions can query the Custom Metrics API directly, useful for verifying an adapter's configuration independently of any HPA that might consume the same data, or for building custom tooling that needs the same metric data outside the autoscaling context.

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/payments/pods/api-service-7d4f9-x2k1p/http_requests_per_second"

Consumption Through HPA

The most common consumer of custom metrics is the HorizontalPodAutoscaler's Pods metric type, which queries this API to retrieve per-pod values for whichever metric name the HPA spec references, averaging them across the target's pods as described in the HPA's own metric calculation.

metrics:
- type: Pods
  pods:
    metric:
      name: http_requests_per_second
    target:
      type: AverageValue
      averageValue: "50"

Access Control for the Custom Metrics API

RBAC Over Metric Access

Reading custom metrics requires RBAC permission against the custom.metrics.k8s.io API group's resources, and because these metrics can reveal operational details about workload behavior, scoping read access appropriately — typically to the HPA controller's own service account and to operators who genuinely need direct query access — follows the same least-privilege principle applied elsewhere in cluster RBAC.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: custom-metrics-reader
rules:
- apiGroups: ["custom.metrics.k8s.io"]
  resources: ["*"]
  verbs: ["get", "list"]

Operational Considerations

Verifying the Adapter Is Healthy Independently

Because a custom metric ultimately depends on the health of both the underlying data source (Prometheus, in the common case) and the adapter translating queries, diagnosing a missing or stale custom metric requires checking each layer separately — the underlying data source's own health, the adapter's logs, and the API response itself — rather than assuming any single component is at fault.

Metric Cardinality and Adapter Load

Because each distinct label combination for a metric name effectively becomes its own time series the adapter must track, high-cardinality metrics (one series per individual request ID, for instance) can overwhelm both the adapter and its underlying data source; designing custom metrics with cardinality appropriate to per-pod aggregation avoids this class of operational problem.