Kubernetes Controller Watch Management

Kubernetes Controller Watch Management is the practice of configuring which resources a controller observes, how it maintains a consistent view of them through resourceVersion tracking and watch resumption, and how it filters and routes the resulting change events into reconcile requests, determining both the correctness and the efficiency of a controller's reaction to cluster state changes.

resourceVersion and Watch Continuity

Resuming a Watch Without Missing Events

Every watch is seeded by an initial LIST call returning a resourceVersion snapshot, and the subsequent WATCH request resumes exactly from that point, streaming every change since; if the connection drops, the client reconnects with the last known resourceVersion, allowing it to resume without missing intervening events, provided that version has not expired.

kubectl get postgresclusters --all-namespaces -o json | jq '.metadata.resourceVersion'

Handling 410 Gone and Compaction

etcd retains only a limited window of historical revisions; if a watch attempts to resume from a resourceVersion older than that window, the API server returns an HTTP 410 Gone, and the client must perform a fresh LIST to obtain a current snapshot and a new starting resourceVersion, effectively losing fine-grained visibility into exactly what changed during the gap and instead re-evaluating full current state.

if errors.IsResourceExpired(err) {
    return reflector.ListAndWatch(stopCh)
}

Informers and Local Caching

Shared Informers Reduce API Server Load

Rather than each reconcile loop issuing its own watch, controller-runtime and client-go maintain a shared informer per watched resource type, backed by a local, eventually-consistent cache; multiple controllers within the same process watching the same resource type share a single underlying watch connection and cache, avoiding redundant load on the API server.

mgr.GetCache().GetInformer(ctx, &databasesv1.PostgresCluster{})

Periodic Resync

ctrl.NewManager(cfg, ctrl.Options{
    Cache: cache.Options{SyncPeriod: &resyncPeriod},
})

In addition to event-driven notifications, informers periodically resync by re-delivering every cached object as if newly modified, at an interval typically measured in hours; this exists as a safety net against a missed or dropped event ever leaving the controller permanently unaware of an object's true state, at the cost of a periodic full reconcile pass across every watched object.

Watching Secondary Resources

Owns and Watches in Controller-Runtime

err = ctrl.NewControllerManagedBy(mgr).
    For(&databasesv1.PostgresCluster{}).
    Owns(&appsv1.StatefulSet{}).
    Owns(&corev1.Service{}).
    Complete(r)

A controller reconciling PostgresCluster objects must also react when a dependent StatefulSet it created is modified or deleted out of band (a manual kubectl edit, or an unrelated actor scaling it directly); Owns() registers a watch on the dependent type filtered to objects whose ownerReferences point back to the primary resource, automatically mapping any such change into a reconcile request for the correct owning PostgresCluster.

Watching Unrelated Resources

err = ctrl.NewControllerManagedBy(mgr).
    For(&databasesv1.PostgresCluster{}).
    Watches(
        &corev1.ConfigMap{},
        handler.EnqueueRequestsFromMapFunc(mapConfigMapToClusters),
    ).
    Complete(r)

For resources with no ownerReferences relationship to the primary resource, such as a shared ConfigMap that many PostgresCluster instances reference by name, an explicit Watches() registration paired with a custom mapping function determines which primary resource's reconcile request should be triggered by a change to that unrelated object.

Filtering Events with Predicates

Reducing Unnecessary Reconciles

err = ctrl.NewControllerManagedBy(mgr).
    For(&databasesv1.PostgresCluster{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})).
    Complete(r)

GenerationChangedPredicate filters out update events where only status changed (which does not increment metadata.generation), preventing a controller from re-triggering its own reconcile loop purely as a side effect of its own prior status write, a common source of otherwise-invisible reconcile churn.

Custom Predicates for Selective Watching

predicate.NewPredicateFuncs(func(obj client.Object) bool {
    cm, ok := obj.(*corev1.ConfigMap)
    return ok && cm.Labels["watched-by"] == "postgres-operator"
})

Label-based or field-based predicates narrow a broad watch (every ConfigMap cluster-wide) down to only the subset actually relevant to the controller, which is essential for controllers that watch widely-used built-in resource types without wanting to process every unrelated change across the entire cluster.

Relationship to the Extension Model and Custom Controller Management

Watch management is the concrete implementation of the list-watch mechanism introduced by the broader Kubernetes extension model, and it is the layer of custom controller management most directly responsible for reconcile efficiency: correctly scoped watches, well-chosen predicates, and properly configured owned-resource relationships determine whether a controller reacts promptly and precisely to relevant changes, or instead reconciles too rarely (missing real changes), too often (wasting API server and controller resources), or against the wrong set of objects entirely.