Kubernetes Custom Controller Management

Kubernetes Custom Controller Management is the operational practice of running a controller process safely and reliably in production, covering leader election for high availability, work queue tuning, error handling and requeue strategy, health and readiness probing, and the resource sizing and RBAC scoping required to deploy a controller as a well-behaved cluster citizen.

High Availability Through Leader Election

Why Leader Election Is Necessary

Running multiple replicas of a controller for availability creates a risk: if every replica reconciles the same objects simultaneously, conflicting or duplicated actions (two replicas both attempting to create the same dependent StatefulSet) can result; leader election ensures only one replica actively reconciles at a time, while the others remain on standby, ready to take over if the leader is lost.

mgr, err := ctrl.NewManager(cfg, ctrl.Options{
    LeaderElection:   true,
    LeaderElectionID: "postgres-operator-leader",
})

Lease-Based Coordination

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  name: postgres-operator-leader
  namespace: databases-system
spec:
  holderIdentity: postgres-operator-7d8f9c-abc12
  leaseDurationSeconds: 15
  renewTime: "2024-06-01T10:00:05Z"

Leader election is implemented atop a Lease object in the API server; the current leader periodically renews it, and if renewal lapses beyond leaseDurationSeconds, another replica acquires the lease and becomes the new leader, resuming reconciliation from the last observed state.

Work Queue Design

Rate-Limited Requeueing

return ctrl.Result{}, err

Returning an error from Reconcile causes controller-runtime to requeue the request automatically with exponential backoff via a rate-limiting work queue, preventing a persistently failing reconcile (an unreachable external dependency, for instance) from spinning in a tight retry loop that consumes CPU and floods logs.

workqueue.NewItemExponentialFailureRateLimiter(5*time.Millisecond, 1000*time.Second)

Explicit Requeue Intervals

return ctrl.Result{RequeueAfter: 30 * time.Second}, nil

For conditions that are not errors but require periodic re-checking regardless of any watched event (waiting for an external certificate to become valid, polling a cloud provider snapshot's completion status), an explicit RequeueAfter schedules a future reconcile pass independent of the rate limiter used for genuine failures.

Concurrency Control

MaxConcurrentReconciles

err = ctrl.NewControllerManagedBy(mgr).
    For(&databasesv1.PostgresCluster{}).
    WithOptions(controller.Options{MaxConcurrentReconciles: 4}).
    Complete(r)

Setting the number of concurrent reconcile workers balances reconciliation throughput against load placed on the API server and any external systems the controller touches; a value too low causes a backlog to accumulate under heavy churn, while a value too high can overwhelm a rate-limited external API the reconcile logic depends on.

Health, Readiness, and Metrics Endpoints

Liveness and Readiness Probes

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-operator
spec:
  template:
    spec:
      containers:
        - name: manager
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8081
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8081

A controller-runtime manager exposes /healthz and /readyz endpoints by default, letting the kubelet detect a hung or deadlocked controller process and restart it, and letting a Service route traffic (typically the metrics endpoint) only to a controller instance that has completed its initial cache sync.

Metrics Exposure

ports:
  - name: metrics
    containerPort: 8080

Controller-runtime exposes reconcile counts, error rates, and work queue depth as Prometheus metrics by default, which is the standard way to observe controller health at the fleet level (via the same metrics pipelines and query practices used for any other workload) rather than relying solely on log inspection.

RBAC Scoping via Code Generation

Marker-Driven RBAC Manifests

// +kubebuilder:rbac:groups=databases.example.com,resources=postgresclusters,verbs=get;list;watch;update;patch
// +kubebuilder:rbac:groups=apps,resources=statefulsets,verbs=get;list;watch;create;update;patch;delete

Kubebuilder markers embedded directly above the reconcile function generate the corresponding ClusterRole manifest automatically via controller-gen, keeping the controller's actual code-level API usage and its granted RBAC permissions mechanically synchronized rather than manually maintained and prone to drifting into either excessive or insufficient scope.

controller-gen rbac:roleName=postgres-operator-role paths="./controllers/..." output:rbac:artifacts:config=config/rbac

Resource Sizing

Requests and Limits for the Controller Pod

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    memory: 256Mi

A controller's memory usage scales primarily with the number of objects held in its informer caches; sizing requests and limits requires accounting for the total count of watched objects across every namespace the controller serves, not just the typical workload of a single reconcile pass, since an undersized memory limit causes cache-driven OOM kills under high object counts.

Relationship to Custom Resource Lifecycle and the Extension Model

Custom controller management is the operational substrate that makes the reconciliation loop described in the broader Kubernetes extension model actually run reliably in a live cluster, and it directly determines how faithfully the custom resource lifecycle's phases (admission through garbage collection) are honored in practice: leader election prevents duplicate action during those phases, work queue tuning determines how quickly a phase transition is observed and acted upon, and RBAC scoping determines whether the controller's authority matches the operator scope it was designed to occupy.