Kubernetes Networking [Level-8: kube-proxy / iptables / IPVS / eBPF]

This is Level 8 of our Kubernetes networking series, and it's where we finally go inside the Service dataplane. Back in Level 3, we learned that a client Pod talks to a Service's ClusterIP, which somehow routes traffic to a backend Pod: Client Pod → Service (10.96.120.50:80) → Backend Pod (10.244.1.10:80) We deliberately left one question unanswered until now: how does traffic sent to that Service IP actually end up at a real Pod? That's exactly what this article unpacks. Table of Contents The Problem: A Service IP Isn't a Real Interface The Basic Idea: A Service Dataplane What Is kube-proxy? kube-proxy's Job kube-proxy Doesn't Create Pods kube-proxy Runs on Every Node The Classic Mental Model: iptables and DNAT What Is DNAT? Tracing the Packet Journey The Full Picture: DNS to Pod iptables in Kubernetes Control Plane vs Dataplane Why EndpointSlices Matter Here Load Balancing Across Endpoints Session Affinity The Scaling Problem with iptables IPVS: IP Virtual Server iptables vs IPVS An Important Modern Nuance eBPF Enters the Picture Why eBPF Is Interesting for Kubernetes Cilium and eBPF eBPF Can Replace Parts of kube-proxy Don't Confuse kube-proxy Replacement with CNI Three Generations to Remember What kube-proxy Actually Watches Why Services Survive Pod Churn Where CNI Fits Into All of This Service vs CNI vs kube-proxy: A Reference Table Following One Complete Packet What Is SNAT? Why SNAT Is Needed DNAT vs SNAT Why Return Traffic Works NodePort, Revisited LoadBalancer, Revisited A Layered Troubleshooting Model A Production Troubleshooting Tree What to Inspect on a Node How Do I Know Which Dataplane I'm Using? The Most Important Conceptual Separation Interview Questions Level 8 Mental Model What's Next: Kubernetes Networking Internals The Problem: A Service IP Isn't a Real Interface Suppose we have a frontend Pod at 10.244.1.5, and a Service named backend with ClusterIP 10.96.120.50:80, backed by three Pods: 10.244.1.10:80 10.244.2.10:80 10.244.3.10:80 The frontend runs curl http://backend:80. DNS resolves backend → 10.96.120.50, so the outgoing packet looks like: SOURCE: 10.244.1.5 DESTINATION: 10.96.120.50:80 Here's the strange part: 10.96.120.50 usually isn't a real, physical network interface attached to any Pod. So who transforms 10.96.120.50:80 into something like 10.244.2.10:80? That transformation is the job of the Service dataplane — and that's exactly what this article is about. The Basic Idea: A Service Dataplane Service (10.96.120.50:80) ↓ Service dataplane ↓ Backend Pod (10.244.2.10:80) Historically, Kubernetes clusters implemented this using kube-proxy. Modern clusters can implement it very differently. The core concepts we need to understand across this article are: kube-proxy, iptables, IPVS, and eBPF. What Is kube-proxy? Despite its name, kube-proxy isn't simply a traditional HTTP proxy — that's a genuinely useful interview point to remember. kube-proxy is a component that helps implement Kubernetes Service networking on each Node: Service ↓ kube-proxy / service dataplane ↓ backend Pods It watches Kubernetes objects — specifically Services and EndpointSlices — and programs the Node's networking rules/dataplane to match what it observes. kube-proxy's Job Given a Service backend at 10.96.120.50:80 with EndpointSlice entries pointing to 10.244.1.10:80, 10.244.2.10:80, and 10.244.3.10:80, kube-proxy's conceptual job is simple: "When traffic arrives for this Service IP and port, send it to one of these backend endpoints." 10.96.120.50:80 ↓ ┌──────┼──────┐ ↓ ↓ ↓ Pod 1 Pod 2 Pod 3 kube-proxy Doesn't Create Pods Another important distinction worth locking in: Deployment → creates/manages Pods Service → defines a stable endpoint kube-proxy/dataplane → implements Service traffic forwarding Three genuinely different responsibilities, each handled by a different component. kube-proxy Runs on Every Node kube-proxy typically runs as a DaemonSet, so every Node gets its own instance: Node 1 → Pods + kube-proxy Node 2 → Pods + kube-proxy Node 3 → Pods + kube-proxy Each Node needs the ability to handle Service traffic that arrives locally. The Classic Mental Model: iptables and DNAT Historically, kube-proxy commonly programmed iptables rules to do this work: Pod ↓ Service IP ↓ iptables rules ↓ DNAT ↓ Pod IP The key term here is DNAT. What Is DNAT? DNAT = Destination Network Address Translation. Suppose a packet starts out as: SOURCE: 10.244.1.5 DESTINATION: 10.96.120.50:80 The Service dataplane can rewrite the destination: SOURCE: 10.244.1.5 DESTINATION: 10.244.2.10:80 So 10.96.120.50 → 10.244.2.10 — that transformation is destination NAT in action. Tracing the Packet Journey Let's slow this down and walk through it explicitly. Frontend: 10.244.1.5 — Service: 10.96.120.50:80 — Backend: 10.244.2.10:80 The frontend sends a packet: SRC = 10.244.1.5, DST = 10.96.120.50:80. The Node's Service dataplane recognizes 10.96.120.50:80 and selects a backend, say 10.244.2.10:80. The packet effectively becomes: SRC = 10.244.1.5, DST = 10.244.2.10:80. From here, normal Pod networking (CNI, covered in Level 7) takes over and delivers it. The Full Picture: DNS to Pod Connecting this back to Level 4: Pod ↓ DNS ↓ Service name ↓ ClusterIP ↓ Service dataplane ↓ Endpoint ↓ Pod IP ↓ CNI networking ↓ destination Pod Or, with concrete values: frontend (10.244.1.5) | "backend" ↓ DNS ↓ 10.96.120.50 ↓ Service dataplane ↓ 10.244.2.10 ↓ CNI ↓ backend Pod iptables in Kubernetes Linux has a built-in packet-filtering and NAT framework called iptables. Kubernetes can program iptables rules so that Service traffic gets redirected to the correct backend endpoint: Packet ↓ iptables ↓ Service rule ↓ Backend selection ↓ DNAT ↓ Pod Conceptually, given a Service at 10.96.120.50:80 with endpoints 10.244.1.10:80, 10.244.2.10:80, and 10.244.3.10:80, the Node might carry rules that resemble: "traffic to 10.96.120.50:80 → choose an endpoint → DNAT to one of the three addresses above." Real Kubernetes iptables rule sets are considerably more complex than this simplification — don't worry about memorizing actual chain names yet. Control Plane vs Dataplane It helps to think of Kubernetes networking in two layers: Control plane decides what should happen: which Pods exist, which Services exist, which endpoints belong to a Service. Dataplane actually handles packets. Control plane → "Service backend Pods are A, B, C" ↓ Dataplane → "Send packets to A/B/C" Why EndpointSlices Matter Here Recall from Level 3: a Service like backend has an associated EndpointSlice listing its live backend IPs. The Service dataplane depends directly on this information: Service + EndpointSlice → Service dataplane → forwarding rules Without accurate, up-to-date EndpointSlice data, the dataplane has nothing correct to forward traffic to. Load Balancing Across Endpoints Given a Service with three backend Pods, a connection to the Service might land on any one of them: Request 1 → Pod A Request 2 → Pod C Request 3 → Pod B Request 4 → Pod A The exact algorithm and distribution behavior depends on the dataplane implementation and configuration. The key idea to hold onto: the Service is a stable front door, while the dataplane is what actually selects a backend endpoint for each connection. Session Affinity Kubernetes Services can optionally enable session affinity, e.g.: sessionAffinity: ClientIP With this enabled, the same client tends to keep landing on the same backend Pod: Client A → Pod A Client A → Pod A Client A → Pod A Again, the precise implementation details depend on which dataplane your cluster is running. The Scaling Problem with iptables The classic kube-proxy → iptables → DNAT → Pod flow works well for many clusters. But there's a scaling consideration: compare 10 Services with 100 total endpoints to 10,000 Services with 100,000 total endpoints. The volume of networking rules involved can grow enormous. This is exactly the pressure that led to IPVS, and eventually to eBPF. IPVS: IP Virtual Server IPVS stands for IP Virtual Server — a Linux kernel technology purpose-built for high-performance load balancing: Service IP (10.96.120.50:80) ↓ IPVS ↓ ┌───┼───┐ ↓ ↓ ↓ Pod A Pod B Pod C Instead of thinking in terms of a long chain of packet-filter rules, think of IPVS as a kernel-level virtual server / load-balancing mechanism purpose-built for exactly this job. iptables vs IPVS Simplified comparison: iptables: packet → rules → rules → rules → backend IPVS: packet → virtual service → backend selection IPVS can offer more specialized load-balancing behavior, and historically was attractive for clusters with large numbers of Services thanks to its performance characteristics. That said, modern Kubernetes networking has moved beyond simply choosing between these two — eBPF-based dataplanes are increasingly significant. An Important Modern Nuance You may hear the claim: "IPVS is always faster than iptables." Don't treat that as an absolute rule. Real-world performance depends on cluster size, traffic patterns, kernel version, configuration, the specific dataplane implementation, and even the underlying hardware. The important conceptual difference to remember is simpler: iptables → packet filtering/NAT rules IPVS → kernel virtual-server load balancing eBPF Enters the Picture Now we reach a genuinely important modern technology: eBPF. You don't need to master every detail yet — at a high level: eBPF = programmable logic that can run inside the Linux kernel. This lets networking systems perform operations directly in the kernel, rather than bouncing through separate userspace components or long rule chains. Why eBPF Is Interesting for Kubernetes The traditional path looks like: Packet → iptables rules → NAT → routing An eBPF-based networking implementation can inject custom packet-processing logic straight into the kernel: Packet → eBPF program → routing / load balancing / policy → destination This can avoid some of the overhead inherent in traditional networking paths and enables much more flexible dataplane behavior. Cilium and eBPF Cilium is one of the best-known Kubernetes networking implementations built on eBPF: Kubernetes → Cilium → eBPF → Linux kernel → networking + load balancing + NetworkPolicy + observability This is exactly why Cilium came up briefly back in Level 7 — and now the pieces should be clicking into place. eBPF Can Replace Parts of kube-proxy This is an important, genuinely modern Kubernetes concept. Some eBPF-based networking implementations can provide Service load balancing entirely on their own. Instead of: Service → kube-proxy → iptables → Pod you can have: Service → eBPF dataplane → Pod In such a setup, kube-proxy may not even be needed for Service handling at all. This is commonly referred to as "kube-proxy replacement," and Cilium is a well-known example of a system that offers it. Don't Confuse kube-proxy Replacement with CNI Keep these terms distinct in your head: CNI = networking interface/ecosystem Cilium = a networking implementation eBPF = a kernel technology kube-proxy replacement = one specific capability/configuration They're related, but not interchangeable. A cleaner way to see the relationship: Cilium → uses eBPF → can implement Kubernetes networking → can provide Service load balancing → can replace kube-proxy Three Generations to Remember A simple learning model for Service traffic handling: Service traffic Service IP | ┌──────────────┼──────────────┐ ↓ ↓ ↓ iptables IPVS eBPF ↓ ↓ ↓ kernel rules virtual server kernel programs ↓ ↓ ↓ Pod This isn't a strict historical replacement chain — real clusters can (and do) use any of these approaches — but it's a useful conceptual comparison to keep handy. What kube-proxy Actually Watches kube-proxy needs live information about Services and EndpointSlices. Whenever something changes — a Pod is added or removed, a Service is updated, an endpoint changes — the Service dataplane needs to be kept in sync: EndpointSlice changes ↓ kube-proxy/dataplane notices ↓ forwarding state updated ↓ new traffic uses current endpoints This is precisely how Kubernetes handles the constant churn of dynamic Pods without breaking client connectivity. Why Services Survive Pod Churn Suppose Service backend (10.96.120.50) initially has Pod A (10.244.1.10) and Pod B (10.244.2.10) as backends. Pod A dies, and a new Pod C (10.244.3.10) appears. The EndpointSlice updates: OLD: A, B NEW: B, C But the Service IP itself never changes — it stays 10.96.120.50. The dataplane simply updates its internal backend information, and clients keep using backend without ever needing to know that a Pod was replaced underneath them. This is one of the single most powerful abstractions in all of Kubernetes. Where CNI Fits Into All of This Combining Level 7 and Level 8: CNI provides the underlying Pod network: Pod → eth0 → veth → Node → Pod network The Service dataplane provides Service-level forwarding: Service IP → backend Pod Together: Kubernetes networking Pod | CNI ↓ Pod network ↓ Service dataplane ↓ another Pod The exact ordering within the Linux packet path can get more nuanced in practice, but this separation of responsibilities is the key mental model to carry forward. Service vs CNI vs kube-proxy: A Reference Table Component Main Job CNI/network implementation Pod networking IPAM Pod IP allocation Service Stable virtual endpoint EndpointSlice Backend endpoint information kube-proxy Traditional Service dataplane implementation iptables Linux packet filtering/NAT framework IPVS Linux virtual-server load balancing eBPF Programmable kernel dataplane technology Following One Complete Packet Let's trace one full request end-to-end. We have frontend (10.244.1.5), Service backend (10.96.120.50:80), and a backend Pod (10.244.2.10:80). The frontend runs curl http://backend. Step 1 — DNS: backend → 10.96.120.50. Step 2 — Application sends packet: SRC = 10.244.1.5, DST = 10.96.120.50:80. Step 3 — Service dataplane: either kube-proxy → iptables/IPVS, or an eBPF dataplane, takes over. Step 4 — Destination translation/selection: 10.96.120.50:80 → 10.244.2.10:80. Step 5 — Pod networking: CNI provides the actual route to 10.244.2.10. Step 6 — Backend receives traffic: Frontend → Service → Service dataplane → Backend Pod. What Is SNAT? We've covered DNAT — now meet its counterpart: SNAT = Source Network Address Translation. DNAT → destination changes SNAT → source changes For example, SRC = Pod IP, DST = external destination might become SRC = Node IP, DST = external destination. This matters specifically when Pods communicate outside the cluster. Why SNAT Is Needed Suppose Pod 10.244.1.10 tries to reach the internet. The external destination generally has no idea how to route return traffic back to 10.244.1.10 — that's a private, cluster-internal address. So the networking system may perform SNAT: 10.244.1.10 ↓ SNAT 192.168.1.10 (Node IP) Conceptually: Pod → SNAT → Node IP → Internet. The exact behavior depends on your specific cluster's networking architecture. DNAT vs SNAT Worth memorizing permanently: DNAT — D = Destination SNAT — S = Source Example (Service → Pod): 10.96.120.50 → DNAT → 10.244.2.10 Example (Pod → outside): 10.244.1.10 → SNAT → Node IP Why Return Traffic Works This is where NAT/connection tracking becomes essential. Given Client (10.244.1.5) → Service (10.96.120.50) → Backend (10.244.2.10), the forward path goes Client → Service IP → DNAT → Backend. For the response to correctly find its way back, the networking stack tracks the connection state so return traffic can be translated appropriately in reverse: Forward: Service → Backend Return: Backend → Client This bookkeeping is handled through connection/NAT state tracking within the Linux networking stack (often referred to as "conntrack"). NodePort, Revisited With a NodePort Service, given a Node IP of 192.168.1.10 and NodePort 30080, external traffic to 192.168.1.10:30080 can be forwarded toward the Service's backend Pods: External client ↓ NodeIP:30080 ↓ Service dataplane ↓ Backend Pod The exact mechanics again depend on which dataplane your cluster is running. LoadBalancer, Revisited With type: LoadBalancer, the flow typically looks like: Internet ↓ Cloud Load Balancer ↓ Node / Service ↓ Backend Pods Both the cloud load balancer and the Kubernetes Service dataplane participate in the overall traffic path — which is exactly why troubleshooting external traffic often requires checking multiple distinct layers. A Layered Troubleshooting Model Now that we understand the full stack, we can build a much better troubleshooting model. If curl http://backend fails, don't immediately blame DNS. Break it down layer by layer: 1. DNS ↓ 2. Service ↓ 3. EndpointSlice ↓ 4. Service dataplane ↓ 5. Pod networking ↓ 6. Application Step 1 — DNS: nslookup backend Does it resolve to the expected Service IP? If backend → 10.96.120.50, DNS is probably fine. Step 2 — Service: kubectl get svc backend Check the ClusterIP, port, and targetPort. Step 3 — EndpointSlice: kubectl get endpointslices kubectl describe svc backend Ask directly: does the Service actually have backend endpoints? If you see Endpoints: , don't waste time debugging iptables yet — the Service simply has no backends to send traffic to. Step 4 — Pod: kubectl get pods -o wide Confirm the backend Pods are Running, Ready, and have assigned IPs. Step 5 — Service dataplane: Ask whether the Service dataplane is correctly forwarding traffic — investigate kube-proxy, iptables, IPVS, or your eBPF/CNI dataplane as appropriate. This is exactly where Level 8 knowledge pays off. Step 6 — CNI: If Service forwarding looks correct but the destination Pod still isn't reachable, investigate CNI, routing, the Pod's interface, Node connectivity, and NetworkPolicy. A Production Troubleshooting Tree Request fails | ↓ DNS? / \ NO YES | | Fix DNS Service? / \ NO YES | | Fix Service EndpointSlice? / \ NO YES | | Fix selectors Dataplane? / \ NO YES | | kube-proxy/ CNI/ eBPF/etc. routing/ policy This structured approach beats randomly running commands and hoping something jumps out. What to Inspect on a Node For a traditional kube-proxy cluster using iptables, you can inspect: iptables-save # or iptables -t nat -S These outputs can be extremely large on real clusters — don't be surprised by the volume. For IPVS-based setups, if installed: ipvsadm -L For eBPF-based implementations, the relevant tooling depends heavily on the specific implementation — Cilium, for example, ships its own observability and status tooling. Don't run diagnostic commands blindly in production — first figure out which dataplane your cluster actually uses. How Do I Know Which Dataplane I'm Using? Start broad: kubectl get pods -A Look for names like kube-proxy, calico, cilium, flannel, or aws-node. Then: kubectl get daemonsets -A For example, seeing a kube-proxy DaemonSet suggests kube-proxy is deployed — but you should still inspect its configuration to determine whether it's actually running in iptables mode, IPVS mode, or something else entirely. The Most Important Conceptual Separation By now, you should be able to cleanly separate these layers in your head: Kubernetes networking Service | ↓ Service dataplane ┌─────────┼─────────┐ ↓ ↓ ↓ iptables IPVS eBPF \ | / └────────┼────────┘ ↓ Pod IP | ↓ CNI network | ↓ Pod Interview Questions Q1. What does kube-proxy do? It traditionally implements Kubernetes Service networking on Nodes by programming a Service dataplane. Q2. Is kube-proxy an HTTP proxy? No — the name is genuinely misleading. Q3. What is a ClusterIP? A stable virtual IP associated with a Kubernetes Service. Q4. How does a ClusterIP actually reach a Pod? Through the Service dataplane — traditionally via iptables/IPVS, or through an alternative dataplane such as eBPF. Q5. What is DNAT? Destination Network Address Translation — conceptually, Service IP → Pod IP. Q6. What is SNAT? Source Network Address Translation — conceptually, Pod IP → Node IP in some outbound scenarios. Q7. What is IPVS? Linux IP Virtual Server — a kernel-level load-balancing mechanism. Q8. What is eBPF? A Linux kernel technology that allows programmable logic to run in kernel contexts, used by modern networking systems for routing, load balancing, security, and observability. Q9. Can eBPF replace kube-proxy? Yes — some implementations, such as Cilium, can provide full Kubernetes Service functionality without needing kube-proxy at all. Q10. What does EndpointSlice provide? Information about the current backend endpoints associated with a Service. Q11. Is CNI the same as kube-proxy? No. CNI/network implementation handles Pod networking; kube-proxy/service dataplane handles Service forwarding. Q12. What happens if a Service has no EndpointSlices/endpoints? The Service has no backend Pods to send traffic to — regardless of whether the Service dataplane itself is working correctly. Level 8 Mental Model If you take away only one diagram from this entire article, make it this one: Service 10.96.120.50:80 | ↓ Service dataplane | ┌───────────┼───────────┐ ↓ ↓ ↓ iptables IPVS eBPF └───────────┼───────────┘ ↓ Pod endpoint 10.244.2.10 | ↓ CNI | ↓ Pod And the responsibility of each layer: CNI → "How do Pods communicate?" Service → "What stable endpoint do clients use?" EndpointSlice → "Which Pods are backends?" kube-proxy / dataplane → "How do Service packets reach those backends?" iptables / IPVS / eBPF → "How is that forwarding actually implemented?" What's Next: Kubernetes Networking Internals In Level 9, we'll connect everything into the actual Linux packet path — diving into network namespaces, veth pairs, bridges, routing tables, conntrack, NAT, the kube-proxy/CNI interaction, Node-to-Pod versus Pod-to-Pod traffic, and exactly what happens to a single packet as it moves through a Kubernetes Node. Conclusion The Service dataplane is where Kubernetes networking stops being abstract and becomes real packet manipulation. A ClusterIP is never a physical address a Pod actually owns — it's a rule, enforced by kube-proxy (or an eBPF-based alternative), that rewrites destinations via DNAT and tracks connections so return traffic finds its way home. Whether that enforcement happens through iptables rule chains, IPVS's kernel-level virtual server, or a fully custom eBPF dataplane, the underlying job is the same: turn a stable Service IP into real traffic to a real, currently-healthy Pod. The biggest practical takeaway: when Service traffic breaks, work the layers in order — DNS, Service, EndpointSlice, dataplane, then CNI. An empty EndpointSlice will waste hours of your time if you jump straight to debugging iptables rules instead of checking it first. Enjoyed this deep dive? Follow along for Level 9, where we trace a single packet's complete journey through the Linux networking stack inside a real Kubernetes Node. Drop your questions in the comments, and share this with a teammate who still thinks kube-proxy is a literal proxy server. Let's keep building this Kubernetes networking roadmap together. 🚀