Kubernetes Event Observation

Kubernetes Event Observation is the practice of reading and interpreting Event objects, the discrete, timestamped records the API server and cluster controllers emit to describe significant occurrences — scheduling decisions, image pulls, probe failures, resource creation and deletion — providing a chronological narrative of what happened to a resource distinct from its current status snapshot.

The Structure of an Event

Core Fields

An Event object records an involvedObject reference (what the event is about), a reason (a short, machine-readable code), a message (human-readable detail), a type (Normal or Warning), and timing information including firstTimestamp, lastTimestamp, and a count for repeated occurrences of the same event.

apiVersion: v1
kind: Event
involvedObject:
  kind: Pod
  name: api-service-7d4f9
reason: FailedScheduling
message: "0/5 nodes are available: insufficient memory"
type: Warning
count: 3
firstTimestamp: "2024-01-15T10:30:00Z"
lastTimestamp: "2024-01-15T10:32:15Z"

Normal Versus Warning Events

Normal events describe expected occurrences (a pod was scheduled, an image was pulled successfully) providing useful context without indicating a problem, while Warning events indicate something did not proceed as expected (a probe failed, a volume mount failed) and are typically the higher-priority signal when triaging an issue.

kubectl get events --field-selector type=Warning

Querying and Filtering Events

Scoping to a Specific Object

kubectl describe on any object includes its recent related events at the bottom of its output, giving an immediate combined view of current status and recent history without a separate query.

kubectl describe pod api-service-7d4f9

Filtering Across the Cluster

kubectl get events supports field selectors to filter by type, involved object kind, or reason, and sorting by timestamp gives a chronological view across many resources simultaneously, useful when investigating an incident that may span multiple related objects.

kubectl get events --all-namespaces --sort-by=.lastTimestamp --field-selector type=Warning

Common Event Reasons and Their Diagnostic Value

Scheduling-Related Events

FailedScheduling with a message detailing why no node was suitable (insufficient resources, unsatisfied affinity, taint tolerations) is typically the first place to look when a pod remains Pending unexpectedly.

Image and Container Events

Pulling, Pulled, Failed (with reasons like ErrImagePull or ImagePullBackOff), and container lifecycle events (Created, Started, Killing) trace a container's startup sequence and surface image registry authentication or availability problems distinctly from application-level failures.

Probe and Health Check Events

Unhealthy events with messages detailing a failed liveness or readiness probe (including the specific HTTP status or command exit code) directly point to why a container is being restarted or excluded from service endpoints, often faster to diagnose from than application logs alone.

Limitations of Event Observation

Limited Retention

Events are retained for a limited period by default (commonly one hour, configurable via --event-ttl on the API server), meaning they are not a durable historical record — an event describing an occurrence from several days ago will no longer be queryable unless it has been separately exported to a longer-retention logging or monitoring system.

Deduplication and Count Aggregation

Repeated identical events are deduplicated into a single Event object with an incrementing count and updated lastTimestamp, rather than creating a new object per occurrence, which is efficient but means the raw event count does not directly correspond to the number of separate Event objects visible in a kubectl get events listing.

Not a Substitute for Application Logs

Events describe orchestration-level occurrences the Kubernetes control plane and kubelet are aware of, not internal application behavior; diagnosing an application-level bug (as opposed to a scheduling, probe, or resource problem) still requires consulting the application's own logs.

Exporting Events for Durable Analysis

Forwarding to a Logging or Monitoring Pipeline

Because of limited default retention, many clusters run a component that watches the Events API and forwards them into the same centralized logging or metrics pipeline used for other observability data, preserving them for historical analysis, alerting, and correlation with metrics and logs well beyond the API server's own retention window.