2.7 Kubernetes Controller Architecture
Kubernetes Controller Architecture manages cluster state by continuously monitoring and enforcing desired conditions through control loops and reconciliation mechanisms.
Kubernetes Controller Architecture is the specific internal structure shared by controllers across Kubernetes, describing the common informer-and-work-queue pattern that nearly every controller, whether built into the controller manager or running as part of a custom Operator, is architected around to observe changes and drive reconciliation efficiently.
The Informer Pattern
Local Cache Synchronized Through Watches
A controller is architected to maintain a local, in-memory cache of the resources it cares about, kept synchronized with the API server through the watch mechanism, rather than issuing a fresh API request every time it needs to inspect current state.
Shared Informers to Avoid Redundant Watches
Where multiple controllers within the same process need to observe the same resource type, the architecture favors sharing a single informer among them, avoiding the cost of establishing redundant watch connections and redundant local caches for the same data.
The Work Queue Pattern
Decoupling Event Reception from Processing
Rather than reconciling a resource immediately and synchronously upon receiving a change notification, a controller is architected to enqueue a reference to the affected object onto a work queue, with a separate worker loop dequeuing and processing items independently of the rate at which notifications arrive.
Deduplication and Rate Limiting
The work queue is architected to deduplicate multiple pending notifications for the same object into a single queued entry, and to apply rate limiting with exponential backoff on repeated failures, preventing a single misbehaving object from overwhelming the controller with retries.
The Reconcile Loop
A Pure Function of Current State
At the center of a controller's architecture is a reconcile function, structured to read the current desired and observed state of a single object and drive the necessary changes, designed to be safely callable repeatedly and idempotently regardless of why it was triggered.
Level-Based, Not Edge-Based Triggering
Controllers are architected around level-based triggering, meaning a reconcile call is expected to bring the system fully into the correct state based on what it currently observes, rather than relying on precisely processing every individual historical change event in sequence.
Owner References and Garbage Collection
Structural Tracking of Ownership
Controllers that create subordinate objects are architected to record ownership through owner references, which a separate, generic garbage collection controller uses to automatically clean up dependent objects when their owner is deleted, without every controller needing to implement its own cleanup logic.