5.21 Kubernetes List and Watch Model

Kubernetes List and Watch Model is the client interaction pattern in which a controller or tool first retrieves a complete, point-in-time snapshot of a resource collection through a list operation, then transitions to a long-lived watch connection that streams incremental changes from that snapshot forward, avoiding the cost and staleness of repeated polling while still guaranteeing an eventually accurate, continuously updated local view of cluster state.

Why Polling Alone Is Insufficient

The Cost of Repeated Full Retrieval

Repeatedly listing an entire resource collection to detect changes wastes bandwidth and API server processing capacity proportional to collection size and poll frequency, a cost that becomes prohibitive at the scale of a cluster with thousands of Pods being monitored by dozens of independent controllers.

The Staleness Problem

Polling on any fixed interval introduces a window of staleness between when a change actually occurs and when a polling client next observes it, whereas a properly maintained watch connection delivers change notifications close to immediately, which matters for controllers that need to react quickly to cluster state changes.

The Initial List Phase

Establishing a Starting Snapshot

A client begins by issuing a list request against the target resource collection, receiving back every currently matching object along with a resourceVersion reflecting the etcd revision the snapshot corresponds to, which becomes the starting point for the subsequent watch phase.

Building the Local Cache

Client-side tooling, particularly the informer machinery in client-go, populates an in-memory cache, often called a local store, from this initial list response, giving the client an immediately usable, fully populated view of the resource collection before any watch events have even been processed.

Transitioning to Watch

Resuming From the List's resourceVersion

Immediately after the list completes, the client establishes a watch request specifying the resourceVersion obtained from the list, instructing the API server to begin streaming only the changes that have occurred since that exact snapshot point, avoiding any gap or duplication between the list and watch phases.

Streamed Event Types

The watch connection delivers a sequence of events, each typed ADDED, MODIFIED, or DELETED, carrying the affected object's full current representation, which the client applies directly to its local cache, keeping it synchronized with the authoritative state in etcd without needing to re-fetch unrelated objects.

Handling Watch Interruptions

Connections Do Not Last Forever

Watch connections are periodically terminated by the API server, deliberately, to bound resource consumption and to accommodate infrastructure such as load balancers that may not support arbitrarily long-lived connections, meaning clients must be prepared to reconnect and resume rather than assuming a single watch will persist indefinitely.

Resuming Versus Relisting

If a client's last known resourceVersion is still within etcd's retained history, it can resume the watch from that exact point using a fresh watch request; if too much time has passed and the requested version has been compacted away, the API server returns an expired error, forcing the client to perform a fresh list and rebuild its cache from scratch.

Bookmark Events

To help clients maintain a recent resourceVersion without requiring actual object changes, the API server can emit periodic BOOKMARK events carrying only an updated resourceVersion, reducing the chance that a client's last observed version becomes stale enough to force an expensive relist after a brief disconnection.

Informers as the Client-Side Implementation

Encapsulating List-Watch Logic

The informer pattern, implemented in client libraries such as client-go, encapsulates the entire list-then-watch lifecycle, including reconnection and relisting on expiration, behind a simple interface that exposes an always-current local cache and a mechanism for registering callbacks that fire on object add, update, or delete events.

Shared Informers and Reduced API Load

Because many independent pieces of controller logic often need to observe the same resource type, shared informer implementations allow a single underlying list-watch connection per resource type to be reused across multiple consumers within the same process, substantially reducing redundant API server load compared to each piece of logic maintaining its own independent watch.

Implications for API Server Scalability

Watch Cache

The API server itself maintains an in-memory watch cache per resource type, serving list and watch requests from this cache rather than hitting etcd directly for every request, a design choice that allows the API server to support a very large number of concurrent watching clients without linearly scaling etcd read load alongside client count.