Kubernetes Daemon Workload Guidelines

Kubernetes Daemon Workload Guidelines describe the practices for running node-level agents on a cluster using the DaemonSet resource — workloads that must run exactly one instance per node (or per matching subset of nodes) rather than a controller-chosen number of replicas distributed arbitrarily. Log collectors, network plugins, storage drivers, and node-level monitoring agents are the canonical examples, and each depends on Kubernetes guaranteeing per-node presence rather than the aggregate replica count that a Deployment provides.

DaemonSet Scheduling Model

One Pod Per Matching Node

A DaemonSet does not use replicas; instead, the controller ensures exactly one Pod runs on every node matching the Pod template's node selection criteria, automatically creating a Pod when a new matching node joins the cluster and removing it when the node leaves. This inverts the normal scheduling model — instead of the scheduler deciding how many Pods and where, the DaemonSet controller decides where based on node membership, and each node effectively guarantees its own Pod's placement.

Node Selection Boundaries

nodeSelector, node affinity, and taints/tolerations determine which subset of nodes actually receive a Pod. A DaemonSet intended for all nodes, including control-plane nodes, needs an explicit toleration for the control-plane taint, since that taint is applied specifically to exclude ordinary workloads.

Interaction With Cluster Autoscaling

Because a DaemonSet Pod is created automatically on every qualifying node, its resource footprint must be included in capacity planning for cluster autoscaling — a DaemonSet with meaningful per-node resource requests effectively reduces the allocatable capacity available to other workloads on every node in the cluster, not just one.

Rollout Behavior

RollingUpdate Strategy

DaemonSets support a RollingUpdate strategy analogous to Deployments, replacing Pods node by node rather than all at once. maxUnavailable bounds how many nodes may be without a running instance of the daemon simultaneously during the rollout, which matters directly for daemons providing a capability other Pods on that node depend on (such as a CNI plugin or a local log shipper).

OnDelete Strategy

The OnDelete strategy requires manual deletion of each old Pod before the controller creates its replacement, giving an operator full control over rollout pacing. This is appropriate for daemons where an automatic, unattended rollout across every node in the cluster is considered too high-risk to run without manual gating at each step.

minReadySeconds for Daemon Pods

As with Deployments, minReadySeconds on a DaemonSet's update strategy adds a soak period before a newly rolled-out Pod is considered available, preventing a rollout from racing ahead across many nodes before a subtly broken new version has had time to reveal itself.

Design Considerations Specific to Daemons

Resource Discipline Is Amplified

A resource leak or oversized request in a daemon workload is multiplied by the node count of the entire cluster, unlike a similar issue in a service Deployment which affects only its own replica count. Resource requests and limits for daemon workloads deserve particularly careful sizing and monitoring given this multiplier effect.

Host-Level Access Patterns

Daemon workloads frequently require access to host resources unavailable to ordinary Pods — host networking (hostNetwork: true), host process ID namespace (hostPID: true), or privileged access to host devices and filesystems. Each of these should be scoped as narrowly as possible; a daemon that only needs to read a specific host path should mount that path specifically rather than requesting broad privileged access.

Priority to Prevent Eviction

Daemon workloads providing infrastructure-critical capability (networking, storage, security agents) should typically run with a high PriorityClass, often using the built-in system-node-critical or system-cluster-critical classes, ensuring the scheduler and eviction logic treat them as non-negotiable rather than allowing them to be preempted by ordinary application Pods under resource pressure.

Observability and Failure Handling

Per-Node Health Visibility

Because a DaemonSet's correctness depends on presence across every qualifying node, monitoring should track per-node Pod status explicitly (a node missing its daemon Pod is a distinct failure mode from a Pod that's present but unhealthy), rather than relying solely on an aggregate ready-Pod count the way a Deployment's health is typically summarized.

Startup Ordering With Node Readiness

Daemon Pods providing foundational capability (particularly CNI plugins) often need to become ready before other workloads can be scheduled onto that node at all, which requires coordination with node readiness conditions and taints that mark a node as not-ready-for-scheduling until its required daemons are running.

Example Configuration

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: codartium-log-agent
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: codartium-log-agent
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  template:
    metadata:
      labels:
        app: codartium-log-agent
    spec:
      priorityClassName: system-node-critical
      tolerations:
        - key: node-role.kubernetes.io/control-plane
          effect: NoSchedule
      containers:
        - name: log-agent
          image: registry.example.com/codartium-log-agent@sha256:bb55cc...
          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"
            limits:
              cpu: "200m"
              memory: "256Mi"
          volumeMounts:
            - name: varlog
              mountPath: /var/log
              readOnly: true
      volumes:
        - name: varlog
          hostPath:
            path: /var/log

Practical Consequences

Correctly applied daemon workload guidelines produce infrastructure agents that are present and healthy on every node they're meant to cover, roll out changes without leaving gaps in coverage, and don't silently degrade cluster capacity through unbounded resource use. Neglecting them commonly results in nodes silently missing critical agents after a rollout stalls partway, cluster-wide capacity loss from an oversized daemon's per-node footprint, or ordinary workloads preempting an infrastructure-critical daemon during resource contention because it was never assigned the priority its role actually requires.