Kubernetes Observability Signal Model

Kubernetes Observability Signal Model describes the structural properties shared and distinguished across the four core observability signal types — metrics, logs, traces, and events — covering how each is shaped, identified, correlated, and consumed, providing the conceptual foundation for reasoning about which signal type is appropriate for a given diagnostic or monitoring need.

Structural Properties of Each Signal Type

Metrics: Numeric, Aggregated, Time-Indexed

A metric is a named, numeric value associated with a set of labels and a timestamp, typically stored as a time series and queried through aggregation functions (rate, sum, percentile) rather than inspected individually; the model assumes high cardinality on labels is expensive and should be bounded deliberately.

http_request_duration_seconds{method="GET", path="/orders", status="200"} 0.042

Logs: Discrete, Textual, Context-Rich

A log entry is a discrete, timestamped record, typically textual (structured as JSON or unstructured free text), carrying rich contextual detail about a single occurrence; the model assumes logs are numerous and searched or filtered rather than aggregated numerically in the way metrics are.

{"timestamp": "2024-01-15T10:32:14Z", "level": "error", "trace_id": "abc123", "message": "database connection refused"}

Traces: Hierarchical, Causally Linked Spans

A trace is a tree or graph of spans, each representing a unit of work with a start time, duration, and parent-child relationship to other spans, modeling causality and timing across a distributed request rather than a single point-in-time measurement.

Span: api-gateway (duration: 245ms)
  └─ Span: auth-service (duration: 12ms)
  └─ Span: order-service (duration: 180ms)
        └─ Span: database-query (duration: 150ms)

Events and Status: State Transitions and Snapshots

Kubernetes events represent discrete state-transition notifications (a pod was scheduled, an image pull failed) with limited retention, while status conditions represent the current, continuously-updated state of a resource — together modeling both the history of significant occurrences and the present state of an object, distinct from the continuous numeric or textual streams metrics and logs represent.

Correlation Across Signal Types

Shared Identifiers as the Correlation Mechanism

The signal model depends on shared identifiers — most commonly a trace ID or request ID — propagated across metrics labels, log fields, and trace spans, enabling a single identifier to pivot from "this metric spiked" to "these are the specific traces during that window" to "these are the specific log lines for one of those traces."

# Metric label
http_requests_total{trace_id="abc123"}
# Log field
{"trace_id": "abc123", "message": "processing order"}
# Trace span attribute
span.trace_id = "abc123"

Correlating Kubernetes Objects Across Signals

Consistent labeling of metrics, logs, and traces with Kubernetes object identifiers (namespace, pod name, container name) allows correlation back to the specific workload instance responsible for a given signal, independent of any application-level trace identifier, which matters particularly for infrastructure-level diagnosis.

Cardinality and Volume Considerations in the Model

Metrics Cardinality Constraints

Because each unique label combination becomes its own time series, the signal model treats metrics cardinality as a resource to be managed deliberately — a label with unbounded values (a raw user ID, for instance) can produce cardinality explosion that overwhelms the storage backend, unlike logs or traces where high-cardinality fields are comparatively cheap to store as unindexed or selectively indexed text.

Sampling in the Tracing Model

Because capturing every span for every request at scale is often prohibitively expensive, the tracing model commonly incorporates sampling — recording only a subset of traces, or prioritizing traces exhibiting errors or high latency — meaning traces represent a statistically informative sample rather than a complete record of every request, unlike metrics (which aggregate over all requests) or logs (which, depending on configuration, may capture every event).

Choosing the Right Signal for a Given Question

Matching Question Type to Signal Type

"Is the error rate elevated right now" is best answered by metrics; "why did this specific request fail" is best answered by a trace, pivoting into the relevant log lines; "what happened to this specific pod an hour ago" is best answered by Kubernetes events and status history — recognizing which signal model fits a given diagnostic question avoids wasted effort searching in the wrong signal type first.

Designing Instrumentation With the Model in Mind

Instrumenting an application with awareness of each signal type's structural strengths — emitting metrics for aggregate trends, logs for detailed context, and trace spans for cross-service causality, all correlated through shared identifiers — produces an observability posture where each signal reinforces the others rather than existing as disconnected, siloed data.