Kubernetes Control Plane Reconciliation Flow

Kubernetes Control Plane Reconciliation Flow is the exact sequence of events, from the moment a triggering change first occurs to the moment a controller's reconciliation of that change either completes successfully or is scheduled for retry, tracing one single pass of a control loop end to end rather than the broader multi-layer propagation pattern spanning several different controllers. Where state flow describes how desired and observed state move across a hierarchy of objects, reconciliation flow describes what happens inside and immediately around one specific controller each time it acts.

Trigger Sources

Watch Event as the Primary Trigger

The overwhelming majority of reconciliation passes are formally triggered by a watch event, an ADDED, MODIFIED, or DELETED notification delivered to the controller's informer as a direct consequence of some earlier write reaching etcd and propagating through the API server's watch cache.

Periodic Resync as a Secondary Trigger

Independent of any actual change, an informer's periodic resync formally re-enqueues every object it currently has cached, ensuring reconciliation still occurs for objects whose relevant watch event may have been missed due to a restart, network interruption, or other transient failure the watch mechanism did not fully guard against.

Requeue Following a Failed Attempt

A reconciliation pass that fails, or that determines further work remains, formally re-adds its own key to the work queue with a computed delay, becoming its own trigger for a subsequent pass rather than waiting for an external event.

kubectl get events --field-selector reason=SyncFailed -n codartium-team

The Single-Pass Sequence

Dequeue and Key Resolution

A worker formally pulls the next available key from the work queue and, using that key, looks up the corresponding object from the controller's local informer cache, formally never reading directly from the API server for this step, since the cache is expected to already reflect current state closely enough for reconciliation purposes.

Observation of Related State

The reconciliation function formally gathers whatever additional state its logic requires beyond the triggering object itself, related objects it owns, external system state it manages, again typically sourced from local caches for objects the controller already watches.

Comparison and Action Computation

The function formally compares the observed state it has gathered against the object's declared spec, computing the specific set of create, update, or delete operations, if any, required to close the gap between the two.

# Illustrative single-pass logic
function reconcile(key):
  obj = cache.get(key)
  if obj is None:
    return handle_deletion(key)
  owned = cache.list_owned(obj)
  actions = diff(obj.spec, owned)
  apply(actions)

Write-Back to the API Server

Any computed action formally results in a write request issued back to the API server, itself re-entering the ordinary request flow, authentication, authorization, admission, persistence, as an independent operation, meaning a single reconciliation pass may issue several separate API writes in sequence.

Pass Completion and Outcome

Successful Completion

A reconciliation pass that determines no further action is needed, or that successfully issues every required write, formally completes and is removed from the work queue's in-progress tracking, with no further action taken until a new trigger arrives.

Failure and Backoff

A pass that encounters an error, a failed write, an unreachable dependency, formally results in the key being re-added to the queue with an exponentially increasing delay, bounding how aggressively a persistently failing reconciliation retries while still eventually attempting again.

kubectl -n kube-system logs deployment/kube-controller-manager | grep -i "error syncing"

Abandonment After Excessive Retries

Some controllers formally impose a maximum retry count or duration, beyond which a persistently failing key is dropped from active retry and instead surfaced only through logs or Events, preventing an unrecoverable failure from consuming worker capacity indefinitely.

Why Tracing a Single Pass Matters Separately from State Flow

While state flow explains the overall directional pattern connecting many controllers across a hierarchy, understanding the internal sequence of one controller's single reconciliation pass, trigger, dequeue, observe, compare, act, complete or retry, is what makes it possible to reason about latency, idempotency, and failure recovery at the level of an individual control loop, the actual unit of behavior each controller author is responsible for getting correct.