Kubernetes Control Plane High Availability

Kubernetes Control Plane High Availability is the specific, practical configuration required to run a control plane across multiple machines such that it continues serving the API and reconciling cluster state through the loss of any single control plane node, covering the concrete steps and components involved: a load-balanced API endpoint, a joined multi-member etcd cluster, and correctly configured leader election for the scheduler and controller manager, distinct from the broader conceptual case for redundancy already established elsewhere.

The Load-Balanced API Endpoint

A Single, Stable Address Fronting Multiple Replicas

Every client of the cluster, kubelets, kubectl, other control plane components, formally needs one stable address to reach the API server, regardless of how many replicas exist behind it; a load balancer, external or, in smaller setups, a simple VIP managed by keepalived, provides this address and distributes connections across the available API server replicas.

# Illustrative load balancer backend configuration
backend kube-apiserver
  balance roundrobin
  server cp-1 10.0.1.10:6443 check
  server cp-2 10.0.1.11:6443 check
  server cp-3 10.0.1.12:6443 check

Health-Checked Backend Selection

The load balancer formally excludes any API server replica failing its health check, /readyz, from receiving new connections, ensuring that a replica in the process of starting up or shutting down does not receive traffic it cannot yet, or can no longer, correctly serve.

curl -k https://cp-1:6443/readyz

Joining Additional Control Plane Nodes

The kubeadm join Procedure

Adding a control plane node to an existing highly available cluster, using kubeadm, formally requires joining both as a new etcd member and as a new control plane node, using a join command carrying a discovery token and certificate hash that allows the new node to securely retrieve the cluster's existing certificate authority.

kubeadm join <load-balancer-endpoint>:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane \
  --certificate-key <key>

Certificate Distribution

Each new control plane node formally requires its own set of serving and client certificates, signed by the cluster's shared certificate authority; kubeadm's join procedure handles this distribution automatically, but self-managed setups without such tooling must generate and distribute these certificates manually and correctly for the new node to be trusted.

Expanding the etcd Cluster

Formal Membership Addition

Adding an etcd member to an already-running cluster formally requires two coordinated steps: registering the new member's identity with the existing cluster via etcdctl member add, and starting the new etcd process itself configured to join as that registered member, with the order of these two steps mattering for the join to succeed correctly.

etcdctl member add etcd-4 --peer-urls=https://10.0.1.13:2380

Maintaining an Odd Member Count

Because etcd's quorum requirement is most efficiently satisfied with an odd number of members, expanding from three to five (rather than three to four) is formally the conventional choice when additional etcd redundancy is needed, since an even member count provides no additional fault tolerance over the next-lower odd count while adding operational overhead.

Verifying Leader Election Across Replicas

Confirming Standby Behavior

After adding scheduler and controller-manager replicas, their correct standby behavior is formally verified by inspecting the relevant Lease objects and confirming only one holderIdentity is active while the others remain idle, rather than assuming leader election is functioning correctly without direct confirmation.

kubectl -n kube-system get lease kube-scheduler kube-controller-manager -o wide

Draining and Removing a Control Plane Node

Graceful Removal Procedure

Removing a control plane node formally requires draining any workloads it hosts, removing it as an etcd member via etcdctl member remove, and finally removing the Node object itself, performed in this order to avoid leaving etcd in a state expecting a member that no longer exists.

kubectl drain cp-3 --ignore-daemonsets
etcdctl member remove <member-id>
kubectl delete node cp-3

Testing the Configuration

Simulated Failure Validation

A highly available control plane's actual guarantees are formally confirmed only by testing them directly: stopping services on one control plane node, or forcibly isolating it from the network, and verifying that the API endpoint, through the load balancer, remains reachable and that leader election correctly promotes a standby replica.

systemctl stop kubelet
kubectl get nodes  # verify continued API availability via remaining replicas

Why These Specific Steps Constitute Control Plane HA

Achieving control plane high availability in practice is not a single switch but the correct completion of each of these concrete steps together, a load-balanced endpoint, correctly joined etcd members maintaining quorum, and verified leader election, since omitting or misconfiguring any single one of them leaves the control plane only partially redundant despite appearing to run multiple replicas.