Container Networking Deep Dive: From Network Namespaces to Kubernetes

Note: This guide is based on the Linux kernel networking documentation, Docker networking documentation (v24+), Kubernetes networking model documentation (v1.28+), and CNI specification v1.0. All examples use documented networking primitives and follow production container networking patterns.

Container networking is fundamental to modern cloud-native applications. Understanding how packets flow from pod to pod, how services load-balance traffic, and how network policies enforce security requires knowledge of Linux networking primitives, Container Network Interface (CNI) plugins, and Kubernetes networking abstractions.

This guide demonstrates building container networks from first principles using network namespaces and veth pairs, explains Docker bridge networking and overlay networks, covers Kubernetes pod networking with CNI plugins (Calico, Cilium, Flannel), and shows how to troubleshoot networking issues with tcpdump and iptables analysis.

Prerequisites

Required Knowledge:

  • Linux command line basics
  • Basic networking concepts (IP addresses, routing, NAT)
  • Container basics (Docker or Kubernetes)
  • Understanding of TCP/IP stack

Required Tools:

# Install network utilities (Ubuntu/Debian)
sudo apt-get update
sudo apt-get install -y \
    iproute2 \          # ip command for network configuration
    bridge-utils \      # brctl for bridge management
    iptables \          # Firewall rules
    tcpdump \           # Packet capture
    netcat \            # Network testing
    iputils-ping \      # Ping utility
    curl                # HTTP testing

# Install Docker
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER

# Install Kubernetes (minikube for local testing)
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube

# Install kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install kubectl /usr/local/bin/

System Requirements:

  • Linux kernel 4.x+ (network namespaces support)
  • Root or sudo access for network configuration
  • 2+ CPU cores, 4GB RAM for Kubernetes

Linux Network Namespaces: Building Blocks

Understanding Network Isolation

Network namespaces provide isolated network stacks. Each namespace has its own:

  • Network interfaces (loopback, eth0, etc.)
  • IP addresses and routing tables
  • iptables rules
  • Network sockets

Creating Container-Like Network Isolation

# create_network_namespace.sh - Build network namespace from scratch

# 1. Create two network namespaces (simulating two containers)
sudo ip netns add container1
sudo ip netns add container2

# Verify namespaces
sudo ip netns list
# Output: container2, container1

# 2. Create veth pair (virtual ethernet cable)
# One end goes to container1, other to container2
sudo ip link add veth1 type veth peer name veth2

# 3. Move veth endpoints into namespaces
sudo ip link set veth1 netns container1
sudo ip link set veth2 netns container2

# 4. Assign IP addresses
sudo ip netns exec container1 ip addr add 10.0.1.2/24 dev veth1
sudo ip netns exec container2 ip addr add 10.0.1.3/24 dev veth2

# 5. Bring interfaces up
sudo ip netns exec container1 ip link set veth1 up
sudo ip netns exec container2 ip link set veth2 up
sudo ip netns exec container1 ip link set lo up  # Enable loopback
sudo ip netns exec container2 ip link set lo up

# 6. Test connectivity
sudo ip netns exec container1 ping -c 3 10.0.1.3
# PING 10.0.1.3 (10.0.1.3) 56(84) bytes of data.
# 64 bytes from 10.0.1.3: icmp_seq=1 ttl=64 time=0.045 ms

Packet Flow Analysis:

Container1 (10.0.1.2)  →  veth1  ←→  veth2  →  Container2 (10.0.1.3)
     [namespace]            ↓           ↓          [namespace]
                        Kernel network stack

Connecting Multiple Containers with a Bridge

# bridge_network.sh - Create bridge network (like Docker bridge)

# 1. Create bridge (acts as virtual switch)
sudo ip link add br0 type bridge
sudo ip addr add 172.18.0.1/24 dev br0
sudo ip link set br0 up

# 2. Create 3 container namespaces
for i in {1..3}; do
    sudo ip netns add container$i
done

# 3. Create veth pairs and connect to bridge
for i in {1..3}; do
    # Create veth pair
    sudo ip link add veth$i type veth peer name veth${i}-br

    # Move one end to container namespace
    sudo ip link set veth$i netns container$i

    # Connect other end to bridge
    sudo ip link set veth${i}-br master br0
    sudo ip link set veth${i}-br up

    # Configure container interface
    sudo ip netns exec container$i ip addr add 172.18.0.$((10+i))/24 dev veth$i
    sudo ip netns exec container$i ip link set veth$i up
    sudo ip netns exec container$i ip link set lo up

    # Add default route via bridge
    sudo ip netns exec container$i ip route add default via 172.18.0.1
done

# 4. Test inter-container communication
sudo ip netns exec container1 ping -c 2 172.18.0.12  # container1 → container2
sudo ip netns exec container2 ping -c 2 172.18.0.13  # container2 → container3

# 5. Enable NAT for internet access
sudo iptables -t nat -A POSTROUTING -s 172.18.0.0/24 ! -o br0 -j MASQUERADE
sudo sysctl -w net.ipv4.ip_forward=1

# Test internet access
sudo ip netns exec container1 ping -c 2 8.8.8.8

Network Topology:

Internet
   ↑
[iptables NAT]
   ↑
br0 (172.18.0.1) ← Virtual Switch
   ├─ veth1-br ←→ veth1 (172.18.0.11) [container1]
   ├─ veth2-br ←→ veth2 (172.18.0.12) [container2]
   └─ veth3-br ←→ veth3 (172.18.0.13) [container3]

Docker Networking

Docker Network Drivers

Driver Use Case Isolation Performance
bridge Single-host container communication Per-network isolation Good
host Direct host network access No isolation Excellent
overlay Multi-host communication (Swarm) Cross-host isolation Good (VXLAN overhead)
macvlan Containers with MAC addresses Physical network integration Excellent
none Complete isolation Full isolation N/A

Docker Bridge Network Internals

# docker_bridge_analysis.sh - Understanding Docker bridge networking

# 1. Inspect default Docker bridge
docker network inspect bridge

# Key fields:
# - Subnet: 172.17.0.0/16
# - Gateway: 172.17.0.1
# - Driver: bridge

# 2. See Linux bridge created by Docker
brctl show docker0
# bridge name    bridge id           STP enabled    interfaces
# docker0        8000.0242xxxxx      no             vethxxxxxxx

# 3. Create custom bridge network
docker network create \
    --driver bridge \
    --subnet 172.20.0.0/16 \
    --gateway 172.20.0.1 \
    --opt "com.docker.network.bridge.name"="br-custom" \
    custom-net

# 4. Run containers on custom network
docker run -d --name web --network custom-net nginx
docker run -d --name app --network custom-net alpine sleep 3600

# 5. Test connectivity by container name (Docker DNS)
docker exec app ping -c 2 web
# PING web (172.20.0.2): 56 data bytes
# 64 bytes from 172.20.0.2: seq=0 ttl=64 time=0.123 ms

# 6. Inspect iptables rules created by Docker
sudo iptables -t nat -L -n -v | grep 172.20.0.0
# Chain DOCKER (2 references)
# DNAT tcp -- !br-custom * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 to:172.20.0.2:80

# 7. Port mapping analysis
docker run -d -p 8080:80 --name nginx-exposed nginx

# Docker creates DNAT rule:
# Host:8080 → Container:80
sudo iptables -t nat -L DOCKER -n
# DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:8080 to:172.17.0.5:80

Docker Overlay Networks for Swarm

# overlay_network.sh - Multi-host container networking

# 1. Initialize Docker Swarm
docker swarm init --advertise-addr <node-ip>

# 2. Create overlay network
docker network create \
    --driver overlay \
    --subnet 10.0.9.0/24 \
    --attachable \
    my-overlay

# 3. Deploy service across multiple nodes
docker service create \
    --name web \
    --network my-overlay \
    --replicas 3 \
    nginx

# Overlay uses VXLAN encapsulation:
# Container IP → VXLAN header → Physical network → Destination host

# 4. Inspect VXLAN interface
ip -d link show
# vxlan interface: vx-xxxxxx
# VXLAN ID: 4097
# UDP Port: 4789

Overlay Network Packet Flow:

[Node1: Container A (10.0.9.2)]
         ↓
[VXLAN encapsulation]
         ↓
[Physical network: Node1 IP → Node2 IP]
         ↓
[VXLAN decapsulation]
         ↓
[Node2: Container B (10.0.9.3)]

Kubernetes Networking Model

Kubernetes Networking Requirements

Kubernetes imposes these requirements on any networking implementation:

  1. Pods can communicate with all other pods without NAT
  2. Nodes can communicate with all pods without NAT
  3. Pod’s IP is the same from inside and outside the pod

CNI (Container Network Interface)

CNI is a spec and libraries for configuring container network interfaces. Kubernetes uses CNI plugins to set up pod networking.

Common CNI Plugins:

Plugin Overlay Network Policy Performance Best For
Calico Optional (VXLAN/IPIP) Yes Excellent Production, security
Cilium Optional (VXLAN) Yes (eBPF) Excellent Advanced features, observability
Flannel Yes (VXLAN/host-gw) No Good Simple setups
Weave Yes Yes Good Ease of use
Canal Flannel + Calico Yes Good Hybrid approach

Installing Calico CNI

# install_calico.sh - Install Calico on Kubernetes

# 1. Start minikube without default CNI
minikube start --cni=false --network-plugin=cni

# 2. Install Calico operator and CRDs
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/tigera-operator.yaml

# 3. Install Calico custom resources
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/custom-resources.yaml

# 4. Wait for Calico pods to be ready
kubectl wait --for=condition=ready pod -l k8s-app=calico-node -n calico-system --timeout=300s

# 5. Verify Calico installation
kubectl get pods -n calico-system
# NAME                                      READY   STATUS    RESTARTS   AGE
# calico-kube-controllers-xxxxxxxxx-xxxxx   1/1     Running   0          2m
# calico-node-xxxxx                         1/1     Running   0          2m
# calico-typha-xxxxxxxxx-xxxxx              1/1     Running   0          2m

# 6. Check CNI configuration
cat /etc/cni/net.d/10-calico.conflist

Pod-to-Pod Communication

# pod_networking_test.yaml - Test pod networking

apiVersion: v1
kind: Pod
metadata:
  name: pod-a
  labels:
    app: test-a
spec:
  containers:
  - name: nginx
    image: nginx
    ports:
    - containerPort: 80
---
apiVersion: v1
kind: Pod
metadata:
  name: pod-b
  labels:
    app: test-b
spec:
  containers:
  - name: alpine
    image: alpine
    command: ["sleep", "3600"]

# Test pod-to-pod communication
kubectl apply -f pod_networking_test.yaml

# Get pod IPs
POD_A_IP=$(kubectl get pod pod-a -o jsonpath='{.status.podIP}')
POD_B_IP=$(kubectl get pod pod-b -o jsonpath='{.status.podIP}')

echo "Pod A IP: $POD_A_IP"
echo "Pod B IP: $POD_B_IP"

# Test connectivity from pod-b to pod-a
kubectl exec pod-b -- wget -qO- http://$POD_A_IP
# <!DOCTYPE html>
# <html>
# <head>
# <title>Welcome to nginx!</title>

# Trace route to see packet path
kubectl exec pod-b -- traceroute $POD_A_IP

Pod Communication Flow (Calico):

Pod A (10.244.1.5)
    ↓ eth0 (veth pair)
    ↓
Node's cali-xxxx interface
    ↓
Routing table (IP-IP tunnel or direct route)
    ↓
Node B's cali-yyyy interface
    ↓
Pod B (10.244.2.3)

Kubernetes Services and Load Balancing

# service_types.yaml - Different service types

# 1. ClusterIP - Internal load balancing
apiVersion: v1
kind: Service
metadata:
  name: web-service
spec:
  type: ClusterIP
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 80

---
# 2. NodePort - External access via node IP
apiVersion: v1
kind: Service
metadata:
  name: web-nodeport
spec:
  type: NodePort
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 80
    nodePort: 30080  # Accessible on <NodeIP>:30080

---
# 3. LoadBalancer - Cloud provider load balancer
apiVersion: v1
kind: Service
metadata:
  name: web-lb
spec:
  type: LoadBalancer
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 80

Service Implementation (kube-proxy):

kube-proxy implements services using iptables (default) or IPVS:

# Inspect service iptables rules
sudo iptables-save | grep web-service

# Example rules created by kube-proxy:
# 1. DNAT to redirect ClusterIP to pod IPs
# -A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m tcp --dport 80 -j KUBE-SVC-XXXXX

# 2. Load balancing to multiple pods
# -A KUBE-SVC-XXXXX -m statistic --mode random --probability 0.33 -j KUBE-SEP-POD1
# -A KUBE-SVC-XXXXX -m statistic --mode random --probability 0.50 -j KUBE-SEP-POD2
# -A KUBE-SVC-XXXXX -j KUBE-SEP-POD3

# 3. DNAT to actual pod IP
# -A KUBE-SEP-POD1 -p tcp -m tcp -j DNAT --to-destination 10.244.1.5:80

Network Policies

# network_policy.yaml - Restrict pod communication

# Allow only frontend pods to access backend
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-policy
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: database
    ports:
    - protocol: TCP
      port: 5432
  - to:  # Allow DNS
    - namespaceSelector: {}
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53

# Test network policy
kubectl apply -f network_policy.yaml

# Deploy test pods
kubectl run frontend --image=alpine --labels="app=frontend" -- sleep 3600
kubectl run backend --image=nginx --labels="app=backend"
kubectl run unauthorized --image=alpine --labels="app=unauthorized" -- sleep 3600

# Get backend IP
BACKEND_IP=$(kubectl get pod backend -o jsonpath='{.status.podIP}')

# Test from frontend (should succeed)
kubectl exec frontend -- wget -qO- http://$BACKEND_IP
# <!DOCTYPE html>...

# Test from unauthorized (should timeout)
kubectl exec unauthorized -- wget -qO- --timeout=5 http://$BACKEND_IP
# wget: download timed out

Network Policy Implementation (Calico): Calico translates NetworkPolicy to iptables rules:

# Inspect iptables rules for network policy
sudo iptables-save | grep cali-fw

# Calico creates chains:
# cali-from-<pod-name>  # Egress from pod
# cali-to-<pod-name>    # Ingress to pod

# Example rule:
# -A cali-to-backend -m set --match-set cali-frontend src -p tcp --dport 8080 -j ACCEPT
# -A cali-to-backend -j DROP  # Deny all other traffic

Troubleshooting Container Networking

Using tcpdump for Packet Analysis

# tcpdump_troubleshooting.sh - Capture and analyze traffic

# 1. Capture traffic on Docker bridge
sudo tcpdump -i docker0 -n
# tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
# listening on docker0, link-type EN10MB (Ethernet), capture size 262144 bytes
# 12:34:56.789012 IP 172.17.0.2.42356 > 172.17.0.3.80: Flags [S], seq 123456789

# 2. Capture traffic in specific network namespace
sudo ip netns exec container1 tcpdump -i veth1 -n

# 3. Filter specific pod traffic in Kubernetes
# Get pod's network namespace
POD_NAME=pod-a
POD_PID=$(kubectl get pod $POD_NAME -o jsonpath='{.status.containerStatuses[0].containerID}' | cut -d'/' -f3)
DOCKER_PID=$(docker inspect -f '{{.State.Pid}}' $POD_PID)

# Enter pod's network namespace and run tcpdump
sudo nsenter -t $DOCKER_PID -n tcpdump -i eth0 -n

# 4. Capture traffic to/from specific IP
sudo tcpdump -i any host 10.244.1.5 -w /tmp/pod-traffic.pcap

# Analyze captured traffic
tcpdump -r /tmp/pod-traffic.pcap -n

Debugging Service Resolution

# dns_troubleshooting.sh - Debug DNS resolution

# 1. Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns
# NAME                       READY   STATUS    RESTARTS   AGE
# coredns-xxxxxxxxx-xxxxx    1/1     Running   0          10m

# 2. Test DNS from pod
kubectl run test-dns --image=busybox --rm -it -- sh
/ # nslookup web-service
# Server:    10.96.0.10
# Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
#
# Name:      web-service
# Address 1: 10.96.0.50 web-service.default.svc.cluster.local

# 3. Check /etc/resolv.conf in pod
kubectl exec test-pod -- cat /etc/resolv.conf
# nameserver 10.96.0.10
# search default.svc.cluster.local svc.cluster.local cluster.local
# options ndots:5

# 4. Debug CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

Common Networking Issues

Issue 1: Pod cannot reach internet

# Check NAT rules
sudo iptables -t nat -L POSTROUTING -n -v

# Verify IP forwarding enabled
sysctl net.ipv4.ip_forward
# net.ipv4.ip_forward = 1

# Enable if disabled
sudo sysctl -w net.ipv4.ip_forward=1

# Test from pod
kubectl exec test-pod -- ping -c 2 8.8.8.8

Issue 2: Service not accessible

# Check service endpoints
kubectl get endpoints web-service
# NAME          ENDPOINTS                     AGE
# web-service   10.244.1.5:80,10.244.2.3:80   5m

# If no endpoints, check pod labels
kubectl get pods --show-labels

# Verify pod selector matches
kubectl get service web-service -o yaml | grep selector -A 2

# Test direct pod access
kubectl exec test-pod -- wget -qO- http://10.244.1.5

Issue 3: Network policy blocking traffic

# List network policies
kubectl get networkpolicies

# Describe policy
kubectl describe networkpolicy backend-policy

# Check Calico policy (if using Calico)
kubectl get globalnetworkpolicy -o yaml

# Temporarily disable policy for testing
kubectl delete networkpolicy backend-policy

Production Best Practices

Network Security Checklist

Pod Security:

  • ✅ Implement network policies for pod-to-pod communication
  • ✅ Use namespace isolation for multi-tenancy
  • ✅ Enable mTLS for inter-service communication (service mesh)
  • ✅ Scan container images for vulnerabilities
  • ✅ Run pods with least privilege (non-root user)

Service Exposure:

  • ✅ Use Ingress for HTTP/HTTPS traffic (not LoadBalancer for each service)
  • ✅ Implement TLS termination at ingress
  • ✅ Use authentication/authorization (OAuth2, JWT)
  • ✅ Rate limit external traffic
  • ✅ Enable WAF (Web Application Firewall)

Network Monitoring:

  • ✅ Monitor CNI plugin health
  • ✅ Track pod network latency
  • ✅ Alert on network policy denies
  • ✅ Log DNS queries for audit
  • ✅ Monitor iptables rule count (performance impact)

Performance Optimization

CNI Plugin Selection:

  • Calico: Best for large clusters (IP-IP or BGP routing)
  • Cilium: Best for eBPF-based performance and observability
  • Flannel: Best for simplicity, small clusters

kube-proxy Mode:

# Switch to IPVS mode for better performance (10,000+ services)
kubectl edit configmap kube-proxy -n kube-system

# Change mode: "" to mode: "ipvs"
# Restart kube-proxy pods
kubectl delete pods -n kube-system -l k8s-app=kube-proxy

Known Limitations

Limitation Impact Mitigation
iptables scalability Performance degrades with 10,000+ services Use IPVS mode for kube-proxy
Overlay network overhead 5-15% throughput reduction Use host-gw or Calico BGP for same-subnet nodes
CNI plugin crashes Pod networking fails Monitor CNI daemonsets, implement health checks
Network policy complexity Difficult to debug deny rules Use Calico policy board, log denied connections
MTU mismatches Packet fragmentation, performance issues Set correct MTU for overlay (typically 1450 for VXLAN)

Conclusion and Resources

Container networking builds on Linux primitives (namespaces, veth pairs, bridges) to provide isolated yet connected environments. Docker abstracts these with bridge and overlay networks, while Kubernetes adds services, ingress, and network policies on top of CNI plugins.

Key takeaways:

  • Network namespaces provide isolation at the kernel level
  • CNI plugins implement Kubernetes networking model (pod-to-pod without NAT)
  • Services provide stable endpoints with load balancing via kube-proxy
  • Network policies enforce security at L3/L4
  • Troubleshooting requires understanding packet flow and iptables rules

Further Resources: