Kubernetes Observability Manifest Management

Kubernetes Observability Manifest Management is the practice of defining, versioning, and deploying the Kubernetes resources that constitute an observability stack — collector configurations, ServiceMonitor and PodMonitor objects, alerting rules, dashboards, and RBAC bindings — as declarative manifests subject to the same lifecycle discipline as application workloads, rather than as ad hoc, manually applied configuration.

Why Observability Configuration Needs Manifest Discipline

The Configuration Sprawl Problem

An observability stack accumulates a large surface of interrelated configuration: scrape targets, relabeling rules, alerting thresholds, dashboard JSON, retention settings, and access policies. Left unmanaged, this configuration drifts between environments, is edited directly in a running cluster without record, and becomes difficult to reproduce during a cluster rebuild or disaster recovery.

Declarative Resources as the Unit of Management

Kubernetes-native observability tooling (notably the Prometheus Operator) exposes its own configuration as Kubernetes Custom Resources, meaning that scrape configuration, alerting rules, and receiver routing become ordinary manifests that can be applied, diffed, and version-controlled with kubectl or GitOps tooling exactly like a Deployment or ConfigMap.

Core Custom Resources

ServiceMonitor and PodMonitor

These resources declare which services or pods Prometheus should scrape, replacing manually edited scrape_configs with a Kubernetes-native selector-based definition.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: checkout-service-monitor
  namespace: production
spec:
  selector:
    matchLabels:
      app: checkout
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics

PrometheusRule

Alerting and recording rules are defined as their own resource type, decoupled from the main Prometheus configuration file.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: checkout-alerts
  namespace: production
spec:
  groups:
    - name: checkout.rules
      rules:
        - alert: CheckoutErrorBudgetBurn
          expr: |
            sum(rate(http_requests_total{deployment="checkout", status=~"5.."}[5m]))
            / sum(rate(http_requests_total{deployment="checkout"}[5m])) > 0.05
          for: 5m
          labels:
            severity: page
          annotations:
            summary: "Checkout error rate exceeds budget"

Prometheus and Alertmanager Custom Resources

The Prometheus server and Alertmanager instances themselves are declared as top-level custom resources, specifying replica count, retention, and which ServiceMonitor/PrometheusRule resources to select via label matching.

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: cluster-monitoring
spec:
  replicas: 2
  retention: 15d
  serviceMonitorSelector:
    matchLabels:
      team: platform
  ruleSelector:
    matchLabels:
      team: platform

Managing Collector and Dashboard Configuration

OpenTelemetry Collector via Operator

The OpenTelemetry Operator introduces an OpenTelemetryCollector custom resource, allowing the entire receiver/processor/exporter pipeline to be expressed as a manifest and reconciled automatically, including automatic sidecar injection for application pods that request instrumentation via annotation.

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel-agent
spec:
  mode: daemonset
  config:
    receivers:
      otlp:
        protocols:
          grpc: {}
    exporters:
      otlp:
        endpoint: tempo:4317
    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [otlp]

Dashboards as ConfigMaps

Grafana dashboard JSON definitions are frequently stored as ConfigMap resources labeled for a sidecar discovery mechanism, allowing dashboards to be provisioned declaratively and version-controlled alongside the rest of the manifest set rather than created manually through the Grafana UI.

apiVersion: v1
kind: ConfigMap
metadata:
  name: checkout-dashboard
  labels:
    grafana_dashboard: "1"
data:
  checkout-dashboard.json: |
    { "title": "Checkout Service", "panels": [] }

GitOps Integration

Reconciliation Loops

When observability manifests are committed to a Git repository and reconciled into the cluster by a GitOps controller (Argo CD or Flux), any manual, out-of-band change to an alerting threshold or scrape configuration is automatically reverted on the next reconciliation pass, enforcing that the Git repository remains the single source of truth.

Environment Promotion

Manifest-based management allows the same ServiceMonitor and PrometheusRule templates to be parameterized (via Kustomize overlays or Helm values) and promoted consistently from a staging cluster to production, avoiding the configuration drift that accumulates when each environment's observability stack is configured independently.

Relationship to the Broader Observability Stack

Manifest management is the operational substrate that makes every other observability capability — metrics collection, telemetry pipelines, tracing, audit handling, and correlation — reproducible, auditable, and recoverable, since the definitions that determine what is observed and how alerts fire are themselves treated as version-controlled cluster state rather than tribal knowledge held only in a running system.