18.2.2.5 Swarm Overlay Networking

Swarm overlay networking is the underlying mechanism that allows containers on entirely separate physical hosts within a cluster to communicate as though they were on the same local network, implemented through VXLAN encapsulation, with specific considerations around encryption, the special ingress network, and the performance cost this encapsulation layer introduces.

VXLAN encapsulation as the underlying mechanism

Overlay networks work by encapsulating container-to-container traffic inside VXLAN packets, which are then transmitted across the actual physical network connecting the cluster's hosts, allowing the containers' own network namespace to behave as though they share a local network regardless of which physical host each one actually runs on:

docker network create --driver overlay my-overlay

docker network inspect my-overlay --format '{{.Driver}} {{.Options}}'

This encapsulation happens transparently from the perspective of the containers themselves, which simply see ordinary network connectivity to other containers by name or address, with the actual cross-host VXLAN tunneling handled entirely beneath that abstraction.

The ingress network and the routing mesh

A special overlay network named ingress, created automatically when Swarm mode is initialized, specifically handles the routing mesh behavior covered in service discovery content, accepting published port traffic on any node and routing it to an appropriate replica regardless of where that replica actually runs:

docker network ls --filter name=ingress

docker network inspect ingress

This network should generally not be deleted or significantly reconfigured without understanding the routing mesh's dependency on it, since doing so directly affects how every published port across every service in the cluster is actually reached from outside.

Creating custom overlay networks for segmentation

Beyond the default ingress network, custom overlay networks can be created to segment communication between specific groups of services, the same network-based isolation principle covered for Compose generally, but extended across the entire multi-host cluster rather than confined to a single host:

docker network create --driver overlay --attachable backend-network
docker service create --network backend-network my-api

The --attachable flag specifically allows standalone containers, not just Swarm services, to join the overlay network directly, which is useful for attaching a one-off diagnostic container or a non-Swarm-managed process to the same network as Swarm services for troubleshooting or integration purposes.

Encrypting overlay network traffic

Overlay network traffic can be encrypted directly, protecting container-to-container communication crossing the underlying physical network from interception, which matters particularly when that physical network is shared with other tenants or traffic that should not be able to observe inter-service communication:

docker network create --driver overlay --opt encrypted my-secure-network

This encryption applies specifically to the VXLAN-encapsulated traffic between hosts; it does not replace application-level TLS for genuinely sensitive data in transit, but it does close a gap that would otherwise leave inter-service traffic readable to anything able to observe the underlying physical network between cluster hosts.

Performance overhead of overlay encapsulation

VXLAN encapsulation adds a measurable, if generally modest, per-packet overhead compared to a container's traffic traveling over an ordinary, unencapsulated bridge network on a single host, which is worth being aware of for latency-sensitive workloads where every additional layer of encapsulation has a real, measurable cost:

docker run --rm --network my-overlay alpine ping -c 10 other-service

docker run --rm --network bridge alpine ping -c 10 other-container

Comparing latency between overlay-networked and bridge-networked communication directly, for a workload genuinely sensitive to this difference, clarifies whether the encapsulation overhead is actually significant for that specific use case or negligible relative to everything else contributing to overall request latency.

Network isolation between overlay networks

Containers on separate overlay networks cannot reach each other by default, the same isolation principle that applies to bridge networks generally, which means a service genuinely needing to communicate across more than one overlay network segment needs to be explicitly attached to each network it requires.

docker service create --network frontend-network --network backend-network my-api

A service attached to multiple overlay networks this way can reach services on either network, while a service attached to only one has no path to anything on the other, which is the deliberate mechanism for enforcing genuine network-level segmentation across a multi-host cluster.

Common mistakes

• Deleting or significantly reconfiguring the default ingress network without understanding its specific role in the routing mesh's overall functioning.
• Not using the --attachable flag when a standalone, non-Swarm-managed container genuinely needs to join an overlay network alongside Swarm services.
• Leaving overlay network traffic unencrypted in an environment where the underlying physical network between hosts cannot be fully trusted.
• Assuming overlay network encapsulation has no performance cost at all, rather than measuring its actual impact for genuinely latency-sensitive workloads.
• Attaching a service to only one overlay network when it actually needs to communicate with services on a separate network segment, rather than explicitly attaching it to every network it genuinely requires.

Swarm overlay networking, built on VXLAN encapsulation, extends container-to-container connectivity transparently across an entire multi-host cluster, with the special ingress network handling the routing mesh specifically, encryption available for traffic crossing untrusted physical networks, and a real, generally modest performance cost worth measuring directly for any workload genuinely sensitive to encapsulation overhead.