calico: Calico + KIND pods unable to communicate externally

Expected Behavior

Deploy KIND Deploy Calico See pods come up successfully and have coreDNS pods be able to dig, ping successfully

Current Behavior

Any new pods deployed are not able to shift into ready state successfully and the coreDNS pods are not able to communicate externally via ping, or dig.

Steps to Reproduce (for bugs)

Deploy KIND
Deploy Calico
Deploy another pod

Logs

 [ERROR] plugin/errors: 2 7893400373289152203.6025455479212086695. HINFO: unreachable backend: read udp 10.244.1.3:38701->192.168.65.1:53: i/o timeout
 [ERROR] plugin/errors: 2 7893400373289152203.6025455479212086695. HINFO: unreachable backend: read udp 10.244.1.3:47810->192.168.65.1:53: i/o timeout
 [ERROR] plugin/errors: 2 7893400373289152203.6025455479212086695. HINFO: unreachable backend: read udp 10.244.1.3:54801->192.168.65.1:53: i/o timeout
 [ERROR] plugin/errors: 2 7893400373289152203.6025455479212086695. HINFO: unreachable backend: read udp 10.244.1.3:45085->192.168.65.1:53: i/o timeout
 [ERROR] plugin/errors: 2 7893400373289152203.6025455479212086695. HINFO: unreachable backend: read udp 10.244.1.3:34876->192.168.65.1:53: i/o timeout
 [ERROR] plugin/errors: 2 amazon.com. A: unreachable backend: read udp 10.244.1.3:58544->192.168.65.1:53: i/o timeout
 [ERROR] plugin/errors: 2 amazon.com. A: unreachable backend: read udp 10.244.1.3:45441->192.168.65.1:53: i/o timeout
 [ERROR] plugin/errors: 2 amazon.com. A: unreachable backend: read udp 10.244.1.3:51907->192.168.65.1:53: i/o timeout
 [ERROR] plugin/errors: 2 amazon.com. A: unreachable backend: read udp 10.244.1.3:36537->192.168.65.1:53: i/o timeout
 [ERROR] plugin/errors: 2 amazon.com. A: unreachable backend: read udp 10.244.1.3:49806->192.168.65.1:53: i/o timeout

; <<>> DiG 9.11.3-1ubuntu1.8-Ubuntu <<>> -t A +tries=5 +retry=5 +time=1 amazon.com
;; global options: +cmd
;; connection timed out; no servers could be reached

Your Environment

Calico version: Attempted with 3.0, 3.2, 3.3, 3.10, master
Orchestrator version (e.g. kubernetes, mesos, rkt): {Major:"1", Minor:"14", GitVersion:"v1.14.3"}
Operating System and version: darwin

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 16 (10 by maintainers)

Most upvoted comments

Well, some of your output above definitely shows 10.244 pod addresses. So perhaps KIND missed processing that config for some reason, or another field needs setting, or something; but I’m afraid I don’t know KIND that well yet.

Anyway, great that things seem to be working for you now.

nelljerram on Oct 30, 2019

Oh, I think the problem is that KIND’s default for the pod CIDR is 10.244.0.0/16, and Calico’s default is 192.168.0.0/16, and they don’t match.

Can you try again with something like this to modify the CIDR in the Calico YAML:

    wget -O - https://docs.projectcalico.org/v3.9/manifests/calico.yaml | \
	sed 's,192.168.0.0/16,10.244.0.0/16,' | \
	kubectl apply -f -

nelljerram on Oct 30, 2019

@sager-tech Also, stepping back to your reported problem…

Please try to distinguish between problems with name resolution (aka DNS) and IP reachability. If ping 8.8.8.8 works, but not ping google.com, it’s a name resolution problem. In that case, look at the /etc/resolv.conf in the place (i.e. host or pod) that you’re pinging from.

If you can ping 8.8.8.8 from the host, but not from a pod, that indicates missing SNAT/MASQUERADE, aka NatOutgoing - i.e. when the ping request reaches 8.8.8.8, the ping response can’t be routed back, because the source IP of the request is still that of the originating pod, which is a private IP that makes no sense to 8.8.8.8. (Actually in this case the request would have been dropped earlier because of an RPF check, but I hope you get the idea anyway.)

Hope that gives you a few ideas to look at…

nelljerram on Oct 29, 2019