This blog post describes the troubleshooting approach I took, the root cause of the problem, and the lessons learned from the experience. If you’ve ever encountered similar issues in your Kubernetes environment, I hope this post will provide some insights into how to approach the problem.
A few months ago, in one of our staging environments used for pre-production testing, developers reported that their pods were intermittently unable to communicate with other services. A simple API call between two microservices would randomly fail with a connection timeout. The issue was unpredictable — sometimes it worked, and sometimes it didn’t.
The Problem
The issue was that pods in the staging environment were unable to communicate with other services on occasion. The problem seemed random, and sometimes an API call between microservices would succeed, but other times it would fail due to a timeout. This inconsistency made it hard to pin down the issue at first glance.
Debugging Approach
Here’s a step-by-step breakdown of the troubleshooting process:
1. Checking Events
My first step was to inspect the events in the cluster to see if any nodes or pods were in an abnormal state. Using the command:
kubectl get eventsI was able to confirm that there were no major issues with node health or obvious pod failures. This didn’t rule out networking problems, but it was an early indication that the issue might not be related to pod failures or node crashes.
2. Investigating Pod Network
Next, I began testing the network connectivity from within the affected pods. I executed ping, curl, and nc commands using kubectl exec.
During this step, I discovered that some pods couldn’t resolve DNS, while others had no issue. This was a crucial clue — it pointed to a problem in the networking layer of the cluster rather than an issue with the services or microservices themselves.
3. Checking CoreDNS
I immediately checked the logs of CoreDNS, as DNS resolution is a critical part of service discovery in Kubernetes. Using the following command, I inspected the logs for DNS errors:
kubectl logs -n kube-system -l k8s-app=kube-dnsI found errors such as “no route to host”, which indicated that the DNS server was inaccessible for some pods. This confirmed that DNS resolution issues were at the root of the problem.
4. Investigating the CNI Plugin (Calico)
Since we were using Calico as the CNI (Container Network Interface) plugin for networking, I checked the logs for the Calico node daemonset:
kubectl logs -n kube-system daemonset/calico-nodeHere, I discovered several errors related to BGP (Border Gateway Protocol) session downtimes and Felix errors. BGP is used by Calico to manage routing between nodes, so when BGP sessions drop, routing between nodes can be disrupted, leading to connectivity issues.
In our Kubernetes environment, BGP was used as part of the Calico CNI plugin to enable dynamic routing between nodes. BGP allows Calico to manage the network’s IP routes efficiently, ensuring that traffic can be routed between pods on different nodes. By using BGP, we were able to scale the cluster and maintain resilient networking across multiple nodes, as it provides a robust and flexible routing mechanism that adapts to changes in the network topology. However, when BGP sessions flapped due to a misconfiguration, it caused intermittent connectivity issues, affecting communication between pods and leading to random API timeouts.
5. Cross-Layer Debugging
At this point, I suspected an issue with the underlying network infrastructure — specifically, the physical switches. After some investigation, I discovered that a recent change had been made to the networking layer, which caused BGP flapping. Calico relies on BGP to route traffic between nodes, and when BGP peers drop, routing for certain IP ranges becomes unavailable.
Immediate Fixes
1. Fixing the BGP Sessions
The first step was to communicate with the network team to resolve the BGP configuration mismatch on the switches. The root cause of the BGP flapping was an MTU (Maximum Transmission Unit) configuration issue. Once this was fixed, BGP sessions stabilized, and the routing between the nodes was restored.
2. Temporary Workaround
While the network team worked on the fix, I implemented a temporary solution. We enabled IP-in-IP encapsulation in Calico to reduce the dependency on BGP until the physical network was fully stabilized. This allowed the pods to continue communicating while we resolved the issue with the switches.
calicoctl patch ipamconfig default - patch '{"spec":{"ipv4ipip":{"enabled":true}}}'Postmortems and Retro
After the issue was resolved, we implemented the following preventive measures to avoid similar problems in the future:
Monitoring and Alerting for BGP Sessions:
- We integrated Prometheus metrics and Alertmanager alerts for monitoring the health of BGP sessions. This allows us to proactively identify any BGP issues before they escalate.
Hardening the Calico Configuration:
- We tweaked the Calico configuration to allow faster failover for BGP sessions, ensuring that if a session goes down, traffic can still be routed with minimal disruption.
Network Change Policy:
- We introduced a network change policy to ensure that any changes to the underlying network infrastructure (including switches) would be coordinated with the DevOps/SRE team, even in the staging environment. This helped prevent unexpected network changes from causing issues.
Challenges and Lessons Learned
Challenges:
The main challenge in this incident was the randomness of the issue. Since some pods worked fine and others didn’t, it was easy to mistakenly assume the problem was with the application. However, my experience with Kubernetes networking and CNI plugins helped me quickly narrow down the problem to the networking layer. By ruling out the application as the cause, I was able to focus my efforts on fixing the networking issue without wasting time.
Lessons Learned:
- Random Network Failures Can Be Deceptive: When connectivity failures are intermittent, it’s easy to misattribute the issue to the application or pods themselves. Always start by testing network connectivity and examining logs from the networking layer (e.g., CoreDNS and CNI plugins).
- Collaborate with Network Teams: In a multi-layered environment like Kubernetes, network issues often lie outside the cluster itself. Collaboration with the networking team is essential for resolving issues related to the underlying physical infrastructure.
- Proactive Monitoring is Crucial: Setting up proper monitoring for BGP sessions, Calico health, and DNS resolution can help catch these issues early, before they affect application performance.
- Document Network Changes: Establishing a network change policy helps ensure that any changes made to the network infrastructure are communicated to the DevOps team, reducing the risk of accidental disruptions.
Conclusion
In Kubernetes, networking issues can sometimes be tricky to diagnose, especially when they appear randomly. By following a structured debugging approach, you can narrow down the cause and resolve the problem more effectively. This incident was a valuable reminder that while Kubernetes abstracts much of the complexity, issues at the networking layer still require careful investigation and collaboration with other teams.
I hope you found this post helpful! If you’ve encountered similar issues or have additional tips, feel free to share your experiences in the comments below.