Kubernetes Controller Coordination

Kubernetes Controller Coordination maintains cluster consistency by managing resources, enforcing policies, and responding to changes across the cluster.

Kubernetes Controller Coordination is the set of mechanisms that allow many independently written, independently running controllers, built-in and custom alike, to operate concurrently against the same shared cluster state without corrupting each other's work, despite none of them communicating with one another directly or being aware of each other's existence. Coordination in this context is almost entirely implicit, achieved through disciplined use of the API's own primitives, ownership, selectors, leader election, rather than through any explicit inter-controller messaging protocol.

Leader Election

Preventing Concurrent Active Instances

For controllers where multiple simultaneously active replicas would produce conflicting writes, most notably the built-in scheduler and controller manager, coordination is achieved through leader election: replicas race to acquire a lease object in etcd, with only the holder actively reconciling and the others idle, ready to take over if the lease is not renewed.

kubectl -n kube-system get lease kube-controller-manager -o yaml

active replicas = 1, enforced via lease acquisition

Custom Controllers Adopting the Same Pattern

Custom controllers facing the same risk, concurrent writes from multiple replicas of the same controller, formally adopt the identical leader election primitive available through the client libraries, coordinating in exactly the same way as built-in components without requiring any new mechanism.

# Illustrative leader election configuration for a custom controller
leaderElection:
  leaderElect: true
  resourceName: codartium-controller-lock
  resourceNamespace: codartium-system

Ownership as Coordination Between Different Controllers

Non-Overlapping Responsibility via ownerReferences

Where a Deployment, a ReplicaSet, and the Pods beneath them are each managed by a distinct controller, coordination between these different controllers is achieved through ownerReferences: each controller's reconciliation logic considers only objects that name it, directly or transitively, as owner, preventing two unrelated controllers from ever contending over the same object.

metadata:
  ownerReferences:
    - apiVersion: apps/v1
      kind: ReplicaSet
      name: codartium-api-7d9f8c
      controller: true

owned (c, o) \Rightarrow only c reconciles o

Cascading Deletion as Emergent Coordination

Because a controller deletes only objects it owns, deleting a top-level object formally triggers a cascade in which each layer's owning controller independently deletes its own directly owned children, with the aggregate cascading effect arising purely from each controller enforcing its own narrow scope, not from any centrally coordinated deletion routine.

kubectl delete deployment codartium-api --cascade=foreground

Selector Discipline as Implicit Coordination

Avoiding Selector Overlap

Multiple controllers of the same kind, two Deployments, for instance, formally coordinate by convention rather than by enforced mechanism: each is expected to use a selector matching only the Pods it created, and selector overlap between unrelated controllers is a misconfiguration the platform does not automatically prevent, relying instead on the practice of deriving each controller's selector from its own unique template labels.

selector (c_{1}) \cap selector (c_{2}) = \emptyset, expected but not enforced

Optimistic Concurrency Between Controllers Sharing an Object

Preventing Lost Updates on Shared Fields

Where two independent controllers might legitimately need to update different fields of the same object, such as a Deployment's spec (user-facing) and status (controller-written), coordination is achieved by scoping each controller's writes to the specific subresource it owns, combined with resourceVersion-based optimistic concurrency guarding against any accidental overlap.

kubectl get --raw /apis/apps/v1/namespaces/codartium-team/deployments/codartium-api/status

Sequencing Through Watch-Driven Reaction, Not Direct Calls

No Controller Calls Another Controller

At no point does one controller formally invoke another directly; a controller that depends on another's output (the ReplicaSet Controller depending on the Deployment Controller having created a ReplicaSet, for instance) achieves this dependency purely by watching for the resulting object to appear via the API, coordinating entirely through observed state rather than through any request-response interaction between the two.

controller A creates X \to controller B observes X via watch \to controller B reacts

kubectl get replicasets -n codartium-team -o jsonpath='{.items[*].metadata.ownerReferences[*].name}'

Why Coordination Remains Implicit

Relying on leader election, ownership, selector discipline, and watch-driven reaction, rather than a centralized coordination service or direct inter-controller communication, is what allows arbitrary new controllers, including ones written by third parties with no knowledge of the built-in controllers already running, to be introduced into a cluster safely: correct behavior follows automatically from each controller respecting the same shared API conventions, not from any explicit registration or handshake with the controllers already present.