microk8s: Pods don't have internet access

I have a single node cluster running on a Ubuntu 20.04 machine. None of the pods are able to ping ips or resolve dns requests. The coredns kube-system pod is able to receive requests but isn’t able to resolve them:

[INFO] 10.1.72.109:58333 - 12496 "AAAA IN google.com. udp 28 false 512" NOERROR - 0 2.000540708s
[ERROR] plugin/errors: 2 google.com. AAAA: read udp 10.1.72.108:38219->8.8.4.4:53: i/o timeout

inspection-report-20200813_094313.tar.gz

cat /proc/sys/net/ipv4/ip_forward is 1

iptables -P FORWARD ACCEPT has no effect

ufw status is inactive

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 16 (3 by maintainers)

Most upvoted comments

Hey everyone I know this is closed but wanted to alert you. This is not microk8s specific! I have a kubeadm initialized cluster and am seeing the exact same behavior. If anyone wants me to run commands or show logs etc… I’ll try to pay attention to this thread.

abonny on Nov 2, 2020

I have the same problem, however, the Windows 3.11 solution (uninstall/reboot/install) did not help in my case.

Since I had a fresh Ubuntu install, I went a bit further and also reinstalled Ubuntu, (server & desktop versions, both 20.04) and then tried installing microk8s, uninstall, reboot and install. At no point could any pod get internet connections.

I think reopening this issue would be justified.

mutech on Oct 21, 2020

Here is what might be happening.

When you microk8s enable dns you let kubernetes (the kubelet service) know of the existence of the dns service. When pods are created kubelet will inject the coredns endpoint in the containers so that coredns is used when resolving dns requests.

In the attached logs I see the coredns pod created 13 hours ago and all the pods before that. So, is it possible you enabled dns after you created the all pods? To test this theory you could delete one pod and let kubernetes reschedule it. The new pod will be created with resolv.conf pointing to the coredns service.

ktsakalozos on Aug 13, 2020

@yahmlevi if you have UFW enabled (check with sudo ufw status) try adding the following rules:

  ufw allow in on vxlan.calico
  ufw allow out on vxlan.calico
  ufw allow in on cali+
  ufw allow out on cali+

ktsakalozos on Aug 3, 2021

microk8s enable dns and recreate pod after this act resolves issue for most of people in latest versions. But please reopen the issue, I cant reinstall microk8s in production in case this arises. We run only 2 node cluster, reinstalling and recreating everything is a 20 minutes+ downtime.

velcrine on Aug 9, 2021

I snap removed, rebooted, snap installed.

mogwai on Sep 23, 2020

I disabled and enabled the dns, in an attempt to resolve the issues I was having. The dig requests from the pods are making it to the coredns service. The problem is that no request from the k8s cluster is able to reach any IP outside of the node. The core dns included it seems.

mogwai on Aug 14, 2020