Kubernetes Telemetry Collection Pipeline

Kubernetes Telemetry Collection Pipeline is the end-to-end chain of components responsible for gathering, transforming, and routing observability signals — metrics, logs, and traces — from workloads and cluster infrastructure to storage and analysis backends. Unlike the minimal, in-memory Metrics API, a telemetry collection pipeline is designed for durability, correlation across signal types, and long-term retention.

Stages of the Pipeline

Instrumentation

Telemetry originates at the source: applications emit metrics through client libraries (such as Prometheus client libraries or the OpenTelemetry SDK), structured logs through stdout/stderr or logging frameworks, and traces through instrumentation libraries that propagate context across service boundaries via headers such as traceparent.

Collection

A collection layer runs as an agent, typically deployed as a DaemonSet so that one instance runs per node, or as sidecar containers within specific pods. Common collector implementations include:

• The OpenTelemetry Collector, which can receive, process, and export all three signal types through a single configurable pipeline.
• Fluent Bit or Fluentd, historically focused on log collection and forwarding.
• Prometheus exporters and scrape-based collection for metrics.

Processing and Enrichment

Before telemetry leaves the cluster, collectors commonly apply processing steps:

• Attaching Kubernetes metadata (namespace, pod name, labels, node) via the Kubernetes attributes processor.
• Batching records to reduce network overhead.
• Filtering or sampling high-volume signals, particularly traces, to control cost.
• Redacting sensitive fields before export.

Export

Processed telemetry is exported to one or more backends: a metrics time-series database (Prometheus, Mimir, Thanos), a log store (Loki, Elasticsearch), or a trace backend (Jaeger, Tempo). Many pipelines fan out to multiple backends simultaneously using the exporter stage of the collector configuration.

Reference Architecture with OpenTelemetry Collector

DaemonSet Deployment Pattern

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-collector-agent
  namespace: observability
spec:
  selector:
    matchLabels:
      app: otel-collector-agent
  template:
    metadata:
      labels:
        app: otel-collector-agent
    spec:
      containers:
        - name: otel-collector
          image: otel/opentelemetry-collector-contrib:0.104.0
          args:
            - --config=/etc/otel/config.yaml
          volumeMounts:
            - name: otel-config
              mountPath: /etc/otel
      volumes:
        - name: otel-config
          configMap:
            name: otel-collector-config

Pipeline Configuration

receivers:
  otlp:
    protocols:
      grpc:
      http:
  prometheus:
    config:
      scrape_configs:
        - job_name: kubernetes-pods

processors:
  batch: {}
  k8sattributes: {}

exporters:
  otlp:
    endpoint: tempo:4317
  prometheusremotewrite:
    endpoint: http://mimir:9009/api/v1/push

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [k8sattributes, batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp, prometheus]
      processors: [k8sattributes, batch]
      exporters: [prometheusremotewrite]

Signal Correlation

Trace-to-Log Correlation

A well-designed pipeline embeds trace identifiers into structured log lines emitted by the application, allowing logs and traces to be joined at query time. This requires consistent propagation of trace_id and span_id fields through the logging context.

Exemplars

Modern metrics pipelines support exemplars, which attach a sampled trace identifier to a specific metric data point. This allows an operator to jump directly from an anomalous latency histogram bucket to the exact trace that produced it.

Operational Concerns

Cardinality and Cost Control

Kubernetes metadata enrichment, while valuable, can dramatically increase metric cardinality when labels such as pod name (which changes on every restart) are attached without care. Pipelines commonly apply cardinality limits or drop high-cardinality labels before export.

Backpressure and Buffering

Collectors implement queuing and retry logic to absorb temporary backend unavailability. Persistent queues (backed by local disk) prevent data loss during backend outages longer than the in-memory buffer can hold.

Node-Level Resource Budgeting

Because collection agents run as DaemonSets, their CPU and memory requests must be sized conservatively across every node in the cluster; an undersized collector under load can itself become a source of dropped telemetry or node resource pressure.

Relationship to Cluster-Wide Observability

The Kubernetes Telemetry Collection Pipeline forms the ingestion layer beneath dashboards, alerting rules, and ad hoc querying. It complements narrower mechanisms such as the Kubernetes Metrics API by providing durable, queryable, cross-signal data rather than a transient snapshot of current resource usage.