18.2.1.5 Swarm Desired State

Swarm desired state is the underlying declarative model the entire orchestrator is built around: a service definition expresses what should be true, a specific image, a specific replica count, specific resource limits, and Swarm continuously, automatically works to reconcile the cluster's actual, current state toward that declared target, rather than executing imperative, one-time commands that simply happen once and stop.

Declarative versus imperative thinking

Creating a service declares an intended end state, not a sequence of one-time actions, and Swarm's own internal reconciliation loop continuously compares that declared state against actual, observed reality, taking whatever action is needed to close any gap between the two:

docker service create --replicas 3 my-api

If one of the three replicas crashes, nothing about the original create command runs again; instead, Swarm's continuous reconciliation process notices the actual running replica count has fallen below the declared desired count of three and automatically starts a new replica to close that gap, entirely on its own, without any further command being issued.

Updating desired state triggers reconciliation, not direct action

Modifying a service's definition, changing its image, its replica count, its resource limits, updates the declared desired state, and Swarm's reconciliation process then works out the specific actions needed to bring actual state into alignment with that new target:

docker service update --image my-api:1.5.0 my-api

This update does not directly, immediately replace every running task; it changes what the desired state now specifies, and the reconciliation process then carries out a rolling replacement of tasks, respecting whatever update parallelism and delay configuration is in effect, to actually converge toward that new declared state over time.

Observing the gap between desired and actual state

docker service ps and docker service inspect both expose the comparison between desired and actual state directly, which is the right place to look when investigating whether a service has actually reached its intended configuration or is still in the process of converging toward it:

docker service ps my-api

NAME         DESIRED STATE   CURRENT STATE
my-api.1     Running         Running 2 minutes ago
my-api.2     Running         Starting 4 seconds ago

A task whose current state has not yet caught up to its desired state, still starting rather than running, for instance, simply reflects that reconciliation is actively in progress, not necessarily a problem, though a task stuck in this gap for an unreasonably long time is worth investigating directly.

Why this model tolerates partial, transient failure gracefully

The continuous reconciliation model is specifically what allows Swarm to recover automatically from a wide range of transient failures, a crashed task, a failed node, without requiring any explicit failure-handling logic beyond simply declaring what the steady-state should look like; the reconciliation loop itself absorbs the work of detecting and correcting any deviation from that declared target.

docker node update --availability drain node-2

Draining a node removes its capacity from consideration, and the reconciliation process automatically redistributes that node's tasks onto other available capacity to maintain the declared desired replica count, again without any explicit, manually issued rescheduling command beyond the drain action itself.

Comparing this model to similar systems

This declarative, continuously reconciled model is the same fundamental philosophy underlying Kubernetes's own controller-based architecture, where a desired state declared in a manifest is continuously reconciled against actual cluster state by a controller loop; recognizing this shared philosophy is useful context for anyone moving between the two systems, since the specific mechanics differ but the underlying mental model, declare intent, let the system continuously work toward it, transfers directly.

spec:
  replicas: 3

Detecting drift between a stack definition and deployed state

Because docker stack deploy itself declares desired state from a Compose file, comparing the file's own declared configuration against the cluster's actual, currently deployed service definitions surfaces any drift, a manual docker service update applied directly against a running service without a corresponding update to the source stack file, which the declarative model does not automatically prevent on its own:

docker stack deploy -c docker-compose.yml my-stack

docker service inspect my-api --format '{{.Spec.TaskTemplate.ContainerSpec.Image}}'

A discrepancy between what the stack file declares and what is actually deployed indicates a manual change was applied directly, bypassing the file as the intended single source of truth, the same kind of drift concern relevant to any declarative infrastructure system regardless of the specific orchestrator in use.

Common mistakes

• Thinking of service operations as one-time, imperative commands rather than understanding them as updates to a continuously reconciled, declared target state.
• Treating a task's transitional current state, still starting or pending, as necessarily a problem rather than recognizing it as reconciliation actively in progress.
• Not recognizing the shared declarative, continuously-reconciled philosophy this model shares with other orchestration systems like Kubernetes, missing useful, transferable mental models when working across both.
• Applying manual docker service update changes directly against a running service without updating the corresponding stack file, introducing drift between the declared source of truth and actual deployed state.
• Assuming Swarm's automatic recovery from transient failure requires explicit, custom failure-handling logic, rather than understanding it as an inherent property of the underlying continuous reconciliation model.

Swarm desired state is the foundational concept underlying the entire orchestrator's behavior, every operation declares an intended target rather than executing a one-time action, and the continuous, automatic reconciliation between that declared target and actual, observed reality is what gives Swarm its self-healing properties without requiring explicit, custom recovery logic for every possible failure scenario.