kubeadm: kubedns container cannot connect to apiserver

systemctl stop kubelet systemctl stop docker iptables --flush iptables -tnat --flush systemctl start kubelet systemctl start docker

The route problem can be solved by flush iptables.

@jeffchanjunwei It is the problem of iptables.Please try the follow command

iptables -P FORWARD ACCEPT

If the command solve your problem,please tell me.

Running iptables -P FORWARD ACCEPT on master and nodes solved the problem for me. I was running flannel.

I have found the solution to my problem:

Client Version: version.Info{Major:“1”, Minor:“5”, GitVersion:“v1.5.2”, GitCommit:“a55267932d501b9fbd6d73e5ded47d79b5763ce5”, GitTreeState:“clean”, BuildDate:“2017-04-14T13:36:25Z”, GoVersion:“go1.7.4”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“5”, GitVersion:“v1.5.2”, GitCommit:“a55267932d501b9fbd6d73e5ded47d79b5763ce5”, GitTreeState:“clean”, BuildDate:“2017-04-14T13:36:25Z”, GoVersion:“go1.7.4”, Compiler:“gc”, Platform:“linux/amd64”}

1.First,we should make sure the ip-forward enabled on the linux kernel of every node.Just execute command: sysctl net.ipv4.conf.all.forwarding = 1

2.Secondly,if your docker’s version >=1.13,the default FORWARD chain policy was DROP,you should set default policy of the FORWARD chain to ACCEPT:$ sudo iptables -P FORWARD ACCEPT.

3.Then the configuration of the kube-proxy must be pass in : –cluster-cidr=<cluster-cidr>.

ps: --cluster-cidr string The CIDR range of pods in the cluster. It is used to bridge traffic coming from outside of the cluster. If not provided, no off-cluster bridging will be performed. Refer to this:https://github.com/kubernetes/kubernetes/issues/36835

systemctl stop kubelet systemctl stop docker iptables --flush iptables -tnat --flush systemctl start kubelet systemctl start docker

The route problem can be solved by flush iptables.

You may need to execute the below command to ensure that the default policy is ACCEPT， to avoiding you are kicked out of your machine when using ssh.

iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

And then you can safely flush your rules:

iptables -F

I suspect you’re hitting issue https://github.com/kubernetes/kubeadm/issues/196. You can verify that this is the root cause by manually editing /etc/kubernetes/manifests/kube-apiserver.yaml on the master and changing the liveness probe:

livenessProbe:
  failureThreshold: 8
  httpGet:
    host: 127.0.0.1
    path: /healthz
    port: 443           # was 6443
    scheme: HTTPS

@pineking yes. It is the cause of network that results into the problem.

Hi, How did you solve the problem? I encounter the same issue too.

kubectl delete svc kubernetes

@mhsabbagh, I have the exact version as yours, 1 master, 3 nodes, the dashboard was setup on node 2 automatically when apply dashboard.yaml. and dashboard error looks like the same as others.

Using HTTP port: 8443 Using in-cluster config to connect to apiserver Using service account token for csrf signing No request provided. Skipping authorization header Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout Refer to the troubleshooting guide for more information: https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md

I have been searching for an solution, but still cannot find a solution. I could telnet to 10.96.0.1 on port 443 from any of the master and nodes

Are we sure it has been fixed in v1.6?

This is still here on 1.7.3 with Ubuntu 16.04. Same exact problem. Have been trying all the possible solutions from disabling apparmor, changing the ports, making sure nothing blocks it… It still doesn’t work.

I tried it on a completely fresh droplet from DigitalOcean and it’s still the same. Doesn’t look like a configuration problem from my side. I just ran the commands as they are in https://medium.com/@SystemMining/setup-kubenetes-cluster-on-ubuntu-16-04-with-kubeadm-336f4061d929

@phagunbaya If you do try the above, I would also kill/restart kubelet for it to take effect faster. When I hit this problem myself, kubelet’s exponential backoff was making it take forever to try to restart the kube-apiserver pod.

I had this issue with Kubernetes 1.18 and docker 19 my cluster was working perfectly earlier. and there was nothing unusual like upgrading or else. restarting docker solved it.

我也遇到了相同的问题，通过刷新iptables解决了，但是不知道具体的原因。

systemctl stop kubelet systemctl stop docker iptables --flush iptables -t nat --flush systemctl start kubelet systemctl start docker

Same problem here with K8S 1.10.5 and weave 2.3.0.

The problem is solved temporarily thanks to lastboy1228 (https://github.com/kubernetes/kubeadm/issues/193#issuecomment-330060848)

I set k8s cluster using virtualbox, 1-kube-master, 2-kube-workers.

When google, there are lots of similar issue, although many ticket shows closed, I tried a lot, but no luck. I tried “$sudo iptables -P FORWARD ACCEPT”, “$ sudo iptables --flush”, this doesn’t work for me.

The root cause should be in kube-dns, flannel and kube-proxy, anyone can tell exactly what is wrong in them ? 😃

kube-dns has 3 components/container: kubedns, dnsmasq,sidecar

` kube-system kube-dns-598d7bf7d4-dzbn8 2/3 CrashLoopBackOff 43 10h kube-system kube-dns-598d7bf7d4-v99tk 2/3 CrashLoopBackOff 45 10h kube-system kube-flannel-ds-mvrt5 1/1 Running 8 20h kube-system kube-flannel-ds-vt2w6 1/1 Running 5 20h kube-system kube-flannel-ds-xrsq8 1/1 Running 5 20h kube-system kube-proxy-jrw6f 1/1 Running 5 21h kube-system kube-proxy-mt6mz 1/1 Running 8 21h kube-system kube-proxy-wwd95 1/1 Running 5 21h

`

try use kubectl exec to check each container

(1) kubedns = always down with error in log.

` Waiting for services and endpoints to be initialized from apiserver…

reflector.go:201] k8s.io/dns/pkg/dns/dns.go:147: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

reflector.go:201] k8s.io/dns/pkg/dns/dns.go:150: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

dns.go:173] Waiting for services and endpoints to be initialized from apiserver...

`

(2) dnsmasq = ok, but it seems the default /etc/resolv.conf might have issue, why it uses my HOST machine’s DNS setting? should it use “nameserver 10.96.0.10” ?

` dnsmasq[12]: using nameserver 127.0.0.1#10053 for domain ip6.arpa dnsmasq[12]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa dnsmasq[12]: using nameserver 127.0.0.1#10053 for domain cluster.local dnsmasq[12]: reading /etc/resolv.conf dnsmasq[12]: using nameserver 127.0.0.1#10053 for domain ip6.arpa dnsmasq[12]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa dnsmasq[12]: using nameserver 127.0.0.1#10053 for domain cluster.local dnsmasq[12]: using nameserver 10.158.54.11#53 dnsmasq[12]: using nameserver 10.158.54.12#53 dnsmasq[12]: using nameserver 10.158.57.11#53 dnsmasq[12]: read /etc/hosts - 7 addresses

/ # cat /etc/resolv.conf
nameserver 10.158.54.11
nameserver 10.158.54.12
nameserver 10.158.57.11
search nokia.com china.nsn-net.net


/ # netstat -nl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:53              0.0.0.0:*               LISTEN
tcp        0      0 :::10053                :::*                    LISTEN
tcp        0      0 :::10054                :::*                    LISTEN
tcp        0      0 :::10055                :::*                    LISTEN
tcp        0      0 :::53                   :::*                    LISTEN
udp        0      0 0.0.0.0:53              0.0.0.0:*
udp        0      0 0.0.0.0:14494           0.0.0.0:*
udp        0      0 0.0.0.0:42680           0.0.0.0:*
udp        0      0 0.0.0.0:61748           0.0.0.0:*
udp        0      0 :::10053                :::*
udp        0      0 :::53                   :::*

`

(3)sidecar = ok, with failure on dnsProbe, this seems NOT a big issue.

` dnsprobe.go:75] Starting dnsProbe {Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33}

dnsprobe.go:75] Starting dnsProbe {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33}

server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:45259->127.0.0.1:53: read: connection refused
server.go:64] Error getting metrics from dnsmasq: read udp 127.0.0.1:45932->127.0.0.1:53: read: connection refused


~ $ netstat -nl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:53              0.0.0.0:*               LISTEN
tcp        0      0 :::10054                :::*                    LISTEN
tcp        0      0 :::53                   :::*                    LISTEN
udp        0      0 0.0.0.0:39749           0.0.0.0:*
udp        0      0 0.0.0.0:45937           0.0.0.0:*
udp        0      0 0.0.0.0:44462           0.0.0.0:*
udp        0      0 0.0.0.0:18938           0.0.0.0:*
udp        0      0 0.0.0.0:53              0.0.0.0:*
udp        0      0 0.0.0.0:20040           0.0.0.0:*
udp        0      0 :::53                   :::*
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node Path

`

10.96.0.1:443 is the cluster ip of kubernetes service, this service is in “default” namespace, can kube-dns from namesapce “kube-system” able to access this in namespace “default” ? I suspect here might have problem ?

` $ kubectl describe service kubernetes

Name:              kubernetes
Namespace:         default
Labels:            component=apiserver
                   provider=kubernetes
Annotations:       <none>
Selector:          <none>
Type:              ClusterIP
IP:                10.96.0.1                            #-service-cidr 10.96.0.0/12 
Port:              https  443/TCP
TargetPort:        6443/TCP
Endpoints:         192.168.56.101:6443
Session Affinity:  ClientIP
Events:            <none>

`

@WanChengHu Does pod bind with ip? Most of reason is iptables and flannel network.

systemctl disable firewalld;systemctl stop firewalld;iptables -P FORWARD ACCEPT

I also have this problem in kubernetes v1.7.4, and after I restart docker, it fix.

I am not entierly sure this has to do with #196, I think there is a race condition elsewhere. I’ve just hit this in something I’m working on at the moment, I will update if I figure out what causes it, as seem to have a way of reproducing is reliably.

Also killing DNS pod seems to resolve this for me…