Everything Is Green and Nothing Works: Tracing Kubernetes Network Reachability

Kubernetes says everything looks healthy: the Deployment is 2/2 Ready, the Service has endpoints, and the Ingress has an address. Traffic still does not arrive. A very ordinary configuration mistake can create exactly this situation. The workload may look like: containers: - name: nginx ports: - containerPort: 80 while the Service says: ports: - port: 80 targetPort: 8080 There is nothing especially mysterious about the bug once you see both sides of it. The annoying part is getting from several healthy-looking Kubernetes objects to the realization that the path between them does not make sense. That is the problem we built Reachability in Radar to help with. Start with the declared path Before sending traffic, Radar reconstructs the Kubernetes path: Ingress -> Service -> selected pods. It checks whether the Service selector matches workloads, whether endpoints exist, whether the selected pods are ready, and whether the ports declared by the Service line up with the workload. This analysis runs against Radar's informer cache. In the example above, that means the targetPort: 8080 versus container port 80 mismatch can be identified without first firing network probes at the Service. That separation is useful. Sometimes the configuration already contains enough information to explain why the path cannot work. Then test what is actually reachable Static analysis can tell you what Kubernetes declares. It cannot tell you whether the network behaves the way the configuration implies. For that, Reachability can run DNS, TCP, TLS, and HTTP checks where appropriate. The important part is that those probes can come from different vantages. For example, Radar may test from your machine, through the Kubernetes API-server proxy, or from a temporary workload inside the cluster. Those are different network paths, so their results should not be treated as interchangeable. Suppose a request through the API-server proxy returns 200. We learned something useful: the API server can reach the target. But that does not prove that a real application caller can reach it. NetworkPolicy or other dataplane behavior may make those paths behave differently. Radar therefore reports: Reached via API server - not live traffic rather than collapsing the result into a generic "reachable." The same idea applies to protocol checks. If Radar establishes a TCP connection to Redis, that proves TCP reachability. It does not prove that the Redis application protocol is working correctly, so the result says that explicitly. In-cluster probes require consent The in-cluster vantage is useful because it gets much closer to the path another workload would take. It also requires creating something in the cluster. Radar shows the probe before doing that. If approved, it runs as a temporary, self-deleting Job under the target namespace's ServiceAccount. If RBAC prevents the operation, Radar does not attempt to work around it. It gives you the equivalent kubectl command instead. There is no new CRD and no resident probe agent required for the test. Reproduce the result yourself Radar also exposes the kubectl commands behind its findings. That is useful for two reasons: you can inspect what Radar actually did, and you can rebuild the evidence manually without having to treat the diagnosis as a black box. Where the model stops Reachability still has important boundaries. It does not model every external cloud load-balancer hop. It cannot observe all CNI-specific enforcement behavior from Kubernetes configuration alone. It does not understand every routing CRD, and a successful TCP probe does not validate arbitrary application protocols. Those limitations matter when interpreting a result. The useful question is not "can this tool tell me with certainty that the entire network works?" It is "what did we actually verify, from where, and where does the path stop making sense?" Try it Reachability is available in the open-source version of Radar. Install Radar: brew install skyhook-io/tap/radar Point it at a cluster, open a Service, and select Reachability. If you have a Service that has been behaving strangely, that is probably the most interesting one to start with. Eyal Dulberg's full engineering write-up goes deeper into the implementation and current limitations: https://radarhq.io/blog/kubernetes-network-reachability Radar on GitHub: https://github.com/skyhook-io/radar