Kubernetes Cluster State Architecture

Kubernetes Cluster State Architecture is the layered path a piece of cluster state travels from its durable, canonical home in etcd, through the API server's in-memory caching and watch-broadcasting machinery, out to every client and controller that depends on an up-to-date view of it, tracing how a single write becomes visible, consistently and efficiently, to potentially thousands of concurrent watchers without each one querying etcd directly.

The Canonical Layer: etcd

Single Source of Truth

Every object's authoritative representation formally exists in exactly one place: a key within etcd's multi-version key-value store, addressed by a fixed path derived from the object's resource type, namespace, and name. No other component in the architecture holds an independently authoritative copy.

Revision Ordering

Every write to etcd formally receives a strictly increasing global revision number; this revision ordering is what allows the API server to later ask etcd, and etcd to correctly answer, "give me everything that changed since revision N," the foundational operation the watch mechanism is built on.

etcdctl get /registry/pods/codartium-team/codartium-api-abc123 -w json

The Caching Layer: API Server Watch Cache

Why a Cache Is Necessary

Without an intermediate cache, every list or watch request from every client would require a direct round-trip to etcd, and every one of potentially thousands of simultaneous watchers, controllers, kubelets, kubectl sessions, would impose independent load on etcd proportional to watcher count rather than write rate.

Structure of the Watch Cache

The API server formally maintains one in-memory watch cache per resource type (and, in some implementations, per namespace scope), populated initially via a list against etcd and kept current via a single, shared etcd watch per resource type, from which it serves list and watch requests to all of its own clients.

kubectl get --raw /api/v1/namespaces/codartium-team/pods?watch=true

The Broadcast Layer: Client Watches

Fan-Out from One Cache to Many Clients

Once a change lands in the API server's watch cache, it is formally fanned out to every client currently holding an open watch against that resource type, each receiving the same ADDED, MODIFIED, or DELETED event derived from a single underlying etcd write, without any client-specific re-query of etcd.

resourceVersion as the Client's Bookmark

A client formally resumes a watch from a specific resourceVersion, allowing it to reconnect after a network interruption and receive exactly the events it missed, rather than needing to re-list the entire resource collection from scratch on every reconnection.

kubectl get pods -n codartium-team -o json | jq '.metadata.resourceVersion'

Consistency Guarantees Across the Layers

Linearizable Reads at the Source

etcd formally guarantees linearizable consistency for reads served directly against it, meaning any read reflects all writes that completed before it began; the API server's watch cache formally provides eventual, bounded-delay consistency with etcd, sufficient for controller reconciliation logic, which is inherently designed to tolerate brief staleness and correct for it on the next reconciliation pass.

Resync as a Correctness Backstop

Because the layered caching and broadcast architecture introduces the theoretical possibility of a missed event, informers formally perform periodic full resyncs against their cache regardless of whether any watch event was missed, an architectural safeguard that makes the overall system correct even if any individual layer briefly loses synchronization.

Multi-Replica API Server Consistency

Every Replica, Its Own Cache

In a highly available control plane, each API server replica formally maintains its own independent watch cache, populated and kept current from the same underlying etcd cluster; a client's specific view depends on which replica it happens to be connected to, though all replicas converge toward the same etcd-backed state.

kubectl get --raw /healthz

Why State Is Architected in Layers

Layering canonical storage (etcd), in-process caching (the API server's watch cache), and per-client broadcast (the watch protocol) is what formally allows cluster state to scale to large numbers of concurrent watchers without a corresponding increase in load on the single, consensus-based etcd cluster, while still preserving a clear, well-defined path back to one authoritative source of truth for every object in the system.