I Replaced kube-proxy with eBPF in Production (And Why My Monitoring Went Blind for 6 Hours)

I Replaced kube-proxy with eBPF in Production (And Why My Monitoring Went Blind for 6 Hours) From a "simple" Cilium upgrade to a 6-hour outage where my SIEM couldn't see a single packet — the real story of deleting kube-proxy, the eBPF program that saved me, and why the docs never warned me about the one metric that matters. The Setup It started with a cilium upgrade command I ran on a Tuesday evening. My homelab cluster — the same 4-node bare-metal setup I wrote about last month (Dell OptiPlex + 3 Raspberry Pis, Talos Linux, Cilium, ArgoCD, Longhorn) — was running Cilium 1.15 in kube-proxy-replacement partial mode. Cilium was handling some traffic, but kube-proxy's iptables rules were still doing the heavy lifting for NodePort and ClusterIP services. I'd read the Cilium 1.16 release notes. The eBPF kube-proxy replacement was now "production-ready for all environments." The Talos Linux docs had a single bullet point: "Set kubeProxyReplacement: true in Cilium values." It looked harmless. It looked like flipping a switch. So I did. helm upgrade cilium cilium/cilium \ --namespace kube-system \ --set kubeProxyReplacement=true \ --set k8sServiceHost=auto \ --set k8sServicePort=6443 The Cilium pods rolled. The nodes stayed up. kubectl get pods -A showed everything running. I went to bed. At 2:47 AM, my phone buzzed. Not an alert from the cluster. An alert from my SIEM — or rather, the absence of data from it. The log ingestion rate for kubernetes-audit and network-flow had flatlined to zero. The cluster was "healthy." The monitoring was dead. This is not a tutorial. This is an autopsy of what happens when you delete the datapath that your security stack depends on, without knowing it. Why Not Just Keep kube-proxy? I run Kubernetes professionally at Siemens. I know what managed EKS gives you: kube-proxy in iptables mode, happily translating ClusterIP virtual IPs into pod endpoints, maintained by AWS, never thought about. But I was hitting a wall. 1. iptables doesn't scale In my homelab, 45 manifests and ~120 services meant ~3,500 iptables rules across all nodes. Every new service triggered a full iptables-restore that locked the kernel's netfilter table for hundreds of milliseconds. On a Raspberry Pi 4, that's a noticeable latency spike. 2. The gap between "cluster works" and "I understand the datapath" kube-proxy is a black box. You run it, it works, you ignore it. But if you can't explain how a packet flows from Service:80 to Pod:8080 at the kernel level, do you really own your cluster? 3. eBPF gives you superpowers — if you know the cost Cilium's eBPF programs can: Replace conntrack with BPF map-based connection tracking Provide socket-level load balancing (no NAT needed for pod-to-pod) Deliver Hubble flow logs with L3-L7 visibility Enforce network policies before the packet hits the host stack But eBPF isn't magic. It's a different universe. And the tools that read the old universe don't work in the new one. The Architecture (Before and After) Before: kube-proxy + Cilium (Hybrid) ┌─────────────────────────────────────────────────────────────────────┐ │ Worker Node │ │ │ │ ┌─────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ │ │ Pod A │───►│ ClusterIP │───►│ kube-proxy │ │ │ │ (app) │ │ Service │ │ (iptables) │ │ │ └─────────────┘ └──────────────────┘ └────────┬─────────┘ │ │ │ │ │ ┌────────────────▼─────────┐ │ │ │ iptables NAT/REDIRECT │ │ │ │ -A KUBE-SVC-XXX ... │ │ │ └────────────────┬─────────┘ │ │ │ │ │ ┌────────────────▼─────────┐ │ │ │ conntrack table │ │ │ │ /proc/net/nf_conntrack │ │ │ └────────────────┬─────────┘ │ │ │ │ │ ┌────────────────▼─────────┐ │ │ │ Pod B (target) │ │ │ └──────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────┘ ▲ │ ┌──────────────────┼──────────────────┐ │ SIEM reads │ Falco reads │ Security tools │ iptables rules │ conntrack │ depend on netfilter │ for flow logs │ for connections │ └──────────────────┘──────────────────┘ After: Cilium eBPF (kube-proxy-deleted) ┌─────────────────────────────────────────────────────────────────────┐ │ Worker Node │ │ │ │ ┌─────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ │ │ Pod A │───►│ ClusterIP │───►│ Cilium Agent │ │ │ │ (app) │ │ Service │ │ (eBPF) │ │ │ └─────────────┘ └──────────────────┘ └────────┬─────────┘ │ │ │ │ │ ┌────────────────▼─────────┐ │ │ │ eBPF Programs │ │ │ │ ├─ bpf_sock (socket LB) │ │ │ │ ├─ bpf_lxc (pod egress) │ │ │ │ ├─ bpf_host (NodePort) │ │ │ │ └─ bpf_network (ingress)│ │ │ └────────────────┬─────────┘ │ │ │ │ │ ┌────────────────▼─────────┐ │ │ │ BPF Maps (not iptables) │ │ │ │ cilium_lb4_services_v2 │ │ │ │ cilium_ct4_global │ │ │ └────────────────┬─────────┘ │ │ │ │ │ ┌────────────────▼─────────┐ │ │ │ Pod B (target) │ │ │ └──────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────┘ ▲ │ ┌──────────────────┘ │ SIEM: "WHERE ARE MY FLOW LOGS?!" │ Falco: "I CAN'T SEE CONNTRACK!" │ Legacy tools: broken by design Design Decision Component Before (kube-proxy) After (eBPF) Impact Service Load Balancing iptables DNAT eBPF socket-level redirect Latency: -30% Connection Tracking conntrack kernel module eBPF BPF map (cilium_ct4_global) No /proc/net/nf_conntrack NAT iptables MASQUERADE eBPF SNAT/DNAT No iptables -t nat -L output Network Policy Enforcement iptables + ipsets eBPF TC/XDP programs Earlier drop, better performance Flow Logs Conntrack events → Falco Hubble eBPF exporter Different schema, different source NodePort iptables REDIRECT eBPF bpf_host Same port, different kernel path The Failures (And What Each One Taught Me) Failure 1: I Upgraded and Went to Bed I ran the Cilium upgrade at 11 PM. The pods cycled green. I checked kubectl get svc and curled a NodePort service from my laptop. It worked. What I didn't check: whether Falco was still emitting network connection events. Falco's default rules include this: # Falco rule: outbound network connection - rule: Outbound Connection desc: Detect outbound connections from pods condition: > outbound and fd.name != "" and (fd.typechar = '4' or fd.typechar = '6') output: "Outbound connection from pod (command=%proc.cmdline connection=%fd.name)" priority: NOTICE Falco detects these connections by reading /proc//fd/ and correlating with conntrack state. When kube-proxy's iptables rules disappeared, Falco's fd.name resolution for ClusterIP destinations broke — the eBPF path doesn't leave the same /proc artifacts that Falco's syscall-based engine expects. Result: Falco emitted zero network alerts for 7 hours. Not because there was no traffic. Because it couldn't see the traffic the same way. Lesson 1: "Services work" is not "security monitoring works." Test both. Failure 2: My SIEM Couldn't Parse Hubble At 2:47 AM, my phone woke me up. Grafana showed a flat line for network_flow_events_per_minute. I checked the SIEM (a custom Fluent Bit → Kafka → ClickHouse pipeline I'd built). The Fluent Bit pods were running. The Kafka topics existed. The ClickHouse tables were healthy. But the kubernetes.flow_logs stream had zero rows since 23:12. Here's why: My SIEM consumed network flow logs from two sources: # /etc/fluent-bit/inputs.conf [INPUT] Name tail Path /var/log/audit/kube-apiserver-audit.log Parser json Tag k8s.audit [INPUT] Name tail Path /var/log/conntrack/flows.json Parser json Tag k8s.network-flows The second input — conntrack/flows.json — was populated by a small sidecar that read /proc/net/nf_conntrack every 30 seconds and emitted JSON. With eBPF replacing conntrack entirely, /proc/net/nf_conntrack was empty on every node. Cilium uses its own BPF map for connection tracking, not the kernel's conntrack module. So my sidecar emitted nothing. Fluent Bit had nothing to tail. Kafka got empty batches. ClickHouse had nothing to insert. The SIEM was blind. Not broken — blind by design, because I had built it around netfilter assumptions that eBPF invalidated. Lesson 2: eBPF isn't a drop-in replacement. It's a migration. Treat it like one. Failure 3: I Panic-Downgraded and Made It Worse At 3:30 AM, I decided to roll back. helm rollback cilium 1 Cilium reverted to the previous version. kube-proxy wasn't re-deployed automatically (I'd deleted the DaemonSet during the upgrade to "save resources"). Now I had no kube-proxy and no eBPF replacement. ClusterIP services stopped resolving entirely. DNS inside the cluster broke (CoreDNS is a ClusterIP service). My apps couldn't reach the API server. ArgoCD went into a crash loop. I had to: SSH into the Talos API (not SSH — talosctl) Re-apply the kube-proxy manifest manually Wait for iptables rules to populate Restart CoreDNS pods Fix ArgoCD's sync state Total downtime: 94 minutes. Lesson 3: Don't delete the old path until the new path is fully monitored. I had eBPF working, but I hadn't validated monitoring. Deleting kube-proxy killed my safety net. The Fix (And What Actually Works) Step 1: Enable Hubble (Not Optional) I had installed Cilium with Hubble disabled to "save resources on the Pis." That was stupid. Hubble is the only source of flow visibility in an eBPF-only cluster. # cilium-values.yaml hubble: enabled: true relay: enabled: true ui: enabled: true metrics: enabled: - dns:query - drop - tcp - flow - icmp - http # CRITICAL: Enable Hubble export for SIEM consumption export: static: enabled: true file: enabled: true path: /var/log/hubble/flows.json format: JSON fieldMask: [] This writes flow logs to /var/log/hubble/flows.json on each node in a structured JSON format. Step 2: Rewrite the SIEM Pipeline I replaced the conntrack sidecar with a Hubble reader: # fluent-bit-daemonset.yaml volumeMounts: - name: hubble-logs mountPath: /var/log/hubble readOnly: true volumes: - name: hubble-logs hostPath: path: /var/log/hubble type: DirectoryOrCreate And updated the Fluent Bit input: [INPUT] Name tail Path /var/log/hubble/flows.json Parser json Tag k8s.network-flows # Hubble JSON has nested objects; flatten them Mem_Buf_Limit 50MB Step 3: Schema Mapping (The Annoying Part) Conntrack JSON and Hubble JSON have completely different schemas. Old (conntrack): { "src_ip": "10.0.1.15", "dst_ip": "10.0.2.88", "src_port": 54321, "dst_port": 443, "protocol": "tcp", "state": "ESTABLISHED", "timestamp": "2026-08-05T23:12:00Z" } New (Hubble): { "flow": { "time": "2026-08-06T03:45:00Z", "verdict": "FORWARDED", "ethernet": {"source": "...", "destination": "..."}, "IP": {"source": "10.0.1.15", "destination": "10.0.2.88", "ipVersion": "IPv4"}, "l4": { "TCP": {"source_port": 54321, "destination_port": 443} }, "source": {"identity": 12345, "namespace": "frontend", "pod_name": "api-xxx"}, "destination": {"identity": 67890, "namespace": "backend", "pod_name": "db-yyy"}, "event_type": {"type": 4, "sub_type": 3}, "is_reply": false } } Hubble gives you way more — pod identity, namespace, Kubernetes labels, L7 protocol details, policy verdict. But you have to rewrite your SIEM parsers. I wrote a small Go transform (ironic, given how I learned it) to flatten Hubble JSON into the old ClickHouse schema, with extra columns for the new metadata: // hubble-flatten/main.go package main import ( "encoding/json" "fmt" "os" "time" ) type HubbleFlow struct { Flow struct { Time string `json:"time"` Verdict string `json:"verdict"` IP struct { Source string `json:"source"` Destination string `json:"destination"` } `json:"IP"` L4 struct { TCP struct { SourcePort uint32 `json:"source_port"` DestinationPort uint32 `json:"destination_port"` } `json:"TCP"` } `json:"l4"` Source HubIdentity `json:"source"` Destination HubIdentity `json:"destination"` } `json:"flow"` } type HubIdentity struct { Identity uint32 `json:"identity"` Namespace string `json:"namespace"` PodName string `json:"pod_name"` } func main() { // Reads Hubble JSON from stdin, emits flat JSON to stdout decoder := json.NewDecoder(os.Stdin) encoder := json.NewEncoder(os.Stdout) for decoder.More() { var h HubbleFlow if err := decoder.Decode(&h); err != nil { continue } flat := map[string]interface{}{ "timestamp": parseTime(h.Flow.Time), "src_ip": h.Flow.IP.Source, "dst_ip": h.Flow.IP.Destination, "src_port": h.Flow.L4.TCP.SourcePort, "dst_port": h.Flow.L4.TCP.DestinationPort, "protocol": "tcp", "verdict": h.Flow.Verdict, "src_namespace": h.Flow.Source.Namespace, "src_pod": h.Flow.Source.PodName, "dst_namespace": h.Flow.Destination.Namespace, "dst_pod": h.Flow.Destination.PodName, "src_identity": h.Flow.Source.Identity, "dst_identity": h.Flow.Destination.Identity, } encoder.Encode(flat) } } func parseTime(s string) int64 { t, _ := time.Parse(time.RFC3339Nano, s) return t.Unix() } Step 4: Replace Falco Network Rules with Hubble + Cilium Policy Falco's syscall-based network detection doesn't work well in eBPF-only clusters because: eBPF socket load balancing happens before the syscall layer Falco hooks ClusterIP connections never appear as "outbound to external IP" because they're redirected at the socket layer Solution: Use Cilium Network Policies for enforcement, Hubble for observability, and keep Falco for what it's good at (file integrity, process execution, privilege escalation). # Example: Instead of Falco detecting unauthorized DB connections, # use Cilium NetworkPolicy to deny them at the kernel level apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: backend-access-control namespace: frontend spec: endpointSelector: matchLabels: app: frontend-api egressDeny: - toEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: backend app: postgres toPorts: - ports: - port: "5432" rules: http: - method: "GET" # But allow the legitimate API service in backend egress: - toEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: backend app: backend-api This enforces at eBPF TC layer — before the packet even leaves the pod's network namespace. Falco could only tell you after it happened. The Real Numbers Performance Impact Metric kube-proxy (iptables) Cilium eBPF Delta Service Latency (p99) 2.8ms 1.1ms -61% NodePort Throughput 1.2 Gbps 3.8 Gbps +217% Connection Tracking Overhead 18% CPU (conntrack) 4% CPU (BPF) -78% New Service Propagation 30-60s (iptables sync)