Kubernetes Pod Network Operation

Kubernetes Pod Network Operation is the sequence of low-level networking actions that occur whenever a Pod is created, scheduled, and started on a Node, resulting in the Pod acquiring an IP address, a network namespace, and connectivity to the rest of the cluster. This operation is orchestrated by the kubelet in cooperation with the container runtime and the CNI (Container Network Interface) plugin configured on the Node.

Lifecycle of Pod Network Setup

Pod Sandbox Creation

Before any application container starts, the container runtime (via CRI — Container Runtime Interface) creates a "pause" or "sandbox" container whose sole purpose is to hold the Pod's network namespace open. All other containers in the Pod join this same network namespace, which is why they share one IP address and can communicate over localhost.

Invoking the CNI Plugin

Once the sandbox exists, the kubelet calls the configured CNI plugin binary (e.g., calico, cilium-cni, flannel) with an ADD command, passing the container ID, network namespace path, and interface name. The CNI plugin is responsible for:

• Allocating an IP address from the Node's Pod CIDR block (IPAM).
• Creating a virtual ethernet (veth) pair, with one end inside the Pod's network namespace and the other on the Node's root namespace.
• Configuring routes so that Pod traffic reaches the Node's network stack.

# Example of inspecting CNI configuration on a Node
ls /etc/cni/net.d/
cat /etc/cni/net.d/10-calico.conflist

Interface Attachment and Route Programming

After IP allocation, the plugin attaches the veth pair, assigns the IP to the Pod-side interface, and installs routes on both the Node and (depending on topology) the broader network fabric so that other Nodes know how to reach this Pod's IP.

Data Path Once the Pod Is Running

Intra-Node Pod Communication

When two Pods on the same Node communicate, traffic flows from the source Pod's veth interface into the Node's root network namespace, through a bridge or routing table, and directly into the destination Pod's veth interface — without leaving the Node.

Inter-Node Pod Communication

When Pods reside on different Nodes, the CNI plugin's chosen data plane determines the path:

• Overlay mode: packets are encapsulated (e.g., VXLAN) with an outer header addressed to the destination Node, then decapsulated on arrival and delivered to the target Pod.
• Routed/underlay mode: the physical network (often via BGP peering, as in Calico's BGP mode) already knows how to route the Pod CIDR block directly, so packets travel unencapsulated.

kube-proxy and Service Traffic

If Pod traffic targets a Service ClusterIP rather than a direct Pod IP, kube-proxy's iptables, IPVS, or eBPF rules intercept the packet on the Node and perform destination NAT (DNAT) to rewrite it to one of the Service's backing Pod IPs before it is routed onward.

# Inspect kube-proxy mode and iptables rules
kubectl -n kube-system get configmap kube-proxy -o yaml | grep mode
iptables -t nat -L KUBE-SERVICES -n

Pod Teardown and Network Cleanup

CNI DEL Invocation

When a Pod is deleted, the kubelet invokes the CNI plugin with a DEL command. The plugin releases the allocated IP address back to the IPAM pool, removes the veth pair, and withdraws any routes that were specific to that Pod.

Garbage Collection of Stale Resources

If a Node crashes or a plugin fails mid-operation, orphaned veth interfaces or IP allocations can remain. Most CNI plugins include periodic reconciliation loops or garbage collectors that detect and clean up these stale artifacts to prevent IP exhaustion.

Failure Modes and Diagnostics

Pod Stuck in ContainerCreating

A Pod stuck in ContainerCreating frequently indicates a CNI plugin failure — commonly IP pool exhaustion, a misconfigured CNI config file, or the CNI daemon Pod not yet being ready on that Node.

kubectl describe pod <pod-name> -n <namespace>
kubectl logs -n kube-system <cni-daemonset-pod> -c <cni-container>

IP Address Exhaustion

Each Node is allocated a fixed-size Pod CIDR block. If the number of Pods scheduled on a Node exceeds the available addresses in that block, new Pods cannot be scheduled successfully until IPs are freed or the block size is increased.

MTU Mismatches

Overlay networks add encapsulation overhead, reducing the effective MTU available to Pod traffic. If the CNI plugin's MTU setting doesn't account for this overhead, large packets may be silently dropped, manifesting as intermittent connection failures for larger payloads.

Design Considerations for Operators

Choosing Overlay vs. Routed Networking

Overlay networks are simpler to deploy across arbitrary underlying infrastructure but add per-packet encapsulation overhead. Routed/BGP-based networking avoids that overhead and integrates more naturally with existing datacenter routing, but requires closer coordination with network operations teams.

Node Pod Capacity Planning

The size of each Node's Pod CIDR block (commonly a /24, yielding 254 usable addresses) directly limits how many Pods can be scheduled on that Node, independent of CPU/memory capacity — this must be accounted for when sizing Node pools for high pod-density workloads.

CNI Plugin Observability

Because Pod network operation spans kubelet, CRI, and CNI components, effective troubleshooting requires correlating kubelet logs, CNI plugin logs, and Node-level network state (ip addr, ip route, bridge fdb) rather than relying on kubectl output alone.