Kubernetes Daemon Reliability Basics

Kubernetes Daemon Reliability Basics is the set of reliability considerations specific to DaemonSet-managed workloads, where the one-pod-per-node guarantee, automatic scheduling onto newly joined nodes, taint tolerance for node-critical daemons, and the cumulative resource footprint across an entire fleet introduce concerns distinct from both stateless Deployment and stateful StatefulSet reliability.

The One-Pod-Per-Node Guarantee

Automatic Coverage of Every Matching Node

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-agent
spec:
  selector:
    matchLabels: { app: log-agent }
  template:
    metadata:
      labels: { app: log-agent }

A DaemonSet controller ensures exactly one matching pod runs on every node satisfying its node selector and tolerations, automatically scheduling a new pod the moment a new node joins the cluster, without requiring any manual intervention or explicit replica count, since "coverage" rather than "count" is the reliability property a DaemonSet maintains.

Tolerating Node-Level Taints

Running Despite Node Problems

spec:
  template:
    spec:
      tolerations:
        - key: node.kubernetes.io/not-ready
          operator: Exists
          effect: NoExecute
        - key: node.kubernetes.io/unschedulable
          operator: Exists
          effect: NoSchedule

Node-critical daemons, a log collector, a network plugin, a node monitoring agent, commonly need to keep running even on a node experiencing problems, precisely the situation ordinary workloads should be evicted from; explicit tolerations for taints like not-ready and unschedulable let a DaemonSet continue operating specifically during the conditions its own function may be needed to diagnose or remediate.

Priority for System-Critical Daemons

system-node-critical Priority Class

spec:
  template:
    spec:
      priorityClassName: system-node-critical

Assigning the built-in system-node-critical priority class to a genuinely critical daemon (a CNI plugin without which no pod on that node can obtain networking) ensures it is scheduled and, under resource pressure, preempts other workloads ahead of ordinary application pods, reflecting that the daemon's own failure would have consequences for every other workload on that node, not merely for itself.

Update Strategy Considerations

RollingUpdate vs. OnDelete

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1

RollingUpdate progressively replaces daemon pods node by node, bounded by maxUnavailable, while OnDelete requires a pod to be manually deleted before it is replaced with the new version, giving an operator direct control over exactly when each node's daemon is updated, appropriate for daemons where an uncoordinated automatic rollout across every node simultaneously carries unacceptable risk.

Why Update Pacing Matters More for Daemons

Because a DaemonSet's pods run on every node cluster-wide, an update that simultaneously affects every node risks a cluster-wide impact if the new version has a defect, a materially larger blast radius than a Deployment update affecting only the nodes hosting that specific workload's replicas, making conservative maxUnavailable values and careful update sequencing especially important for daemons.

Cumulative Resource Footprint

Per-Node Overhead Multiplied Across the Fleet

resources:
  requests: { cpu: 100m, memory: 128Mi }

A DaemonSet's resource requests are reserved on every single node in the cluster simultaneously, meaning even a modest per-pod request accumulates into a meaningful fleet-wide capacity reservation; sizing a daemon's resource requests conservatively is a reliability concern not just for the daemon itself but for every other workload competing for the capacity that daemon reserves on every node.

Interaction With Cluster Autoscaling

New Nodes Automatically Gain Daemon Coverage

When the cluster autoscaler adds a new node, the DaemonSet controller schedules its pod onto it automatically, but this scheduling itself takes a nonzero amount of time; a workload whose correct operation depends on the daemon (a network policy enforcer that must be running before other pods on that node can safely communicate) may need to account for a brief window during node startup where the daemon has not yet become ready.

Relationship to Deployment and Stateful Reliability Basics

Daemon reliability basics address a fundamentally different reliability property than the replica-count and rollout mechanisms covered under deployment reliability basics, coverage across the fleet rather than a fixed replica count, and they share with stateful reliability basics an emphasis on identity (each daemon pod tied to its specific node) but without the ordered lifecycle or persistent volume concerns of StatefulSets, reflecting the reliability model's general principle that different workload topologies require correspondingly different, purpose-built reliability mechanisms.