Kubernetes Cluster Networking Architecture

Kubernetes Cluster Networking Architecture is the study of how a specific CNI implementation actually carries Pod-to-Pod traffic across the physical or virtual network connecting a cluster's nodes, encompassing the concrete design choices, encapsulation versus direct routing, per-node subnet allocation, integration with underlying cloud VPCs, that different networking implementations make while all satisfying the same abstract Kubernetes networking model. Where the networking model specifies required outcomes, networking architecture specifies how a particular implementation actually achieves them on the wire.

Overlay Networking Architecture

Encapsulation Between Nodes

An overlay network architecture formally wraps each Pod-to-Pod packet crossing node boundaries inside an outer packet addressed between the two nodes' own IP addresses, using an encapsulation protocol such as VXLAN, and unwraps it on arrival, allowing Pod IP addresses to exist in an address space independent of, and untouched by, the underlying physical network's own routing.

# Illustrative overlay configuration (Flannel VXLAN backend)
net-conf.json: |
  {
    "Network": "10.244.0.0/16",
    "Backend": {
      "Type": "vxlan"
    }
  }

Tradeoffs

Overlay architecture requires no cooperation from the underlying physical network, making it simple to deploy across arbitrary infrastructure, but formally imposes encapsulation and decapsulation overhead on every cross-node packet, along with a reduced effective MTU to accommodate the additional header space.

Routed (Native) Networking Architecture

Direct Routing Using Real Pod IPs

A routed networking architecture formally forwards Pod-to-Pod traffic using each Pod's actual IP address as both source and destination, with the underlying network's own routing tables, either configured statically, via BGP peering between nodes, or via a cloud provider's native route tables, directing traffic to the correct node without any encapsulation layer.

# Illustrative BGP peering configuration (Calico-style)
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: node-to-tor
spec:
  peerIP: 192.168.1.1
  asNumber: 65000

Tradeoffs

Routed architecture formally avoids encapsulation overhead and preserves full MTU, but requires the underlying network fabric to either run a routing protocol capable of learning per-node Pod subnets or be manually configured with static routes, a requirement not every environment can satisfy.

Cloud-Native VPC Integration Architecture

Pods as First-Class VPC Citizens

Some CNI implementations formally allocate Pod IP addresses directly from the cloud provider's own VPC address space, attaching each Pod's networking to native cloud networking primitives (such as secondary IP addresses on an instance's network interface), making Pods natively routable within the VPC without any overlay or additional routing protocol.

# Illustrative cloud-native IP allocation reference
podCIDR: 10.0.0.0/16   # allocated from the VPC's own address space

Tradeoffs

This architecture formally integrates most tightly with cloud-native tooling, security groups, VPC flow logs, native load balancers, but ties Pod IP allocation to the specific cloud provider's addressing model and available secondary IP capacity per instance, a constraint that can bound the number of Pods a given instance type can host.

Per-Node Subnet Allocation

CIDR Assignment as a Shared Architectural Element

Regardless of overlay, routed, or cloud-native architecture, most implementations formally allocate a distinct IP subnet (CIDR block) to each node, recorded in that node's spec.podCIDR, from which the node's local CNI plugin assigns individual addresses to Pods scheduled there.

kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'

eBPF-Based Architectures

Bypassing Traditional iptables/Routing Layers

Newer CNI implementations, most notably Cilium, formally use eBPF programs attached at various points in the kernel's networking stack to implement Pod networking, Service load balancing, and network policy enforcement directly, bypassing traditional iptables rule chains and, in some configurations, even traditional routing table lookups, for improved performance at scale.

cilium status
cilium monitor --type drop

Network Policy Enforcement Architecture

Data-Plane-Level Enforcement

Regardless of which of the above architectures a CNI plugin uses for basic connectivity, NetworkPolicy enforcement is formally implemented at the same data-plane layer, iptables rules, eBPF programs, or a dedicated policy engine, meaning policy enforcement capability and performance characteristics are tied directly to the specific CNI implementation's own architecture, not to the Kubernetes NetworkPolicy API itself.

kubectl get networkpolicies -A

Why Multiple Architectures Coexist

Because the Kubernetes networking model specifies required outcomes rather than a mandated mechanism, overlay, routed, cloud-native, and eBPF-based architectures can each satisfy it while making fundamentally different tradeoffs between deployment simplicity, performance, and dependency on underlying infrastructure capabilities, allowing operators to choose the architecture best matched to their specific environment rather than being locked into a single implementation strategy.