kubernetes: kubelet won't start - /etc/kubernetes/pki/ca.crt: no such file or directory

I found that kubeadm init will start the kubelet. Its working for me.

When you run “kubeadm init” , it does: [certificates] Generated ca certificate and key. [certificates] Generated apiserver certificate and key. [certificates] apiserver serving cert is signed for DNS names [****] [certificates] Generated apiserver-kubelet-client certificate and key. [certificates] Generated sa key and public key. [certificates] Generated front-proxy-ca certificate and key. [certificates] Generated front-proxy-client certificate and key. [certificates] Valid certificates and keys now exist in “/etc/kubernetes/pki”

@dennisjac is correct, in the instructions, this step is obviously missing

but the funny part is, when “kubeadm init” it requires “kubelet is running” already: Unfortunately, an error has occurred: timed out waiting for the condition

This error is likely caused by that: - The kubelet is not running - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

Now I am in the dead loop

https://kubernetes.io/docs/setup/independent/install-kubeadm/#installing-docker has specific docker install instructions and version recommendations:

On each of your machines, install Docker. Version v1.12 is recommended, but v1.11, v1.13 and 17.03 are known to work as well. Versions 17.06+ might work, but have not yet been tested and verified by the Kubernetes node team.

in my case I had to:

kubeadm reset

and:

apt remove --purge kube*

then reinstall

I’m able to get master up with kubeadm init --apiserver-advertise-address=<ip> but I’m unable to join the cluster.

kublete on the worker is getting kubelet[13291]: error: unable to load client CA file /etc/kubernetes/pki/ca.crt: open /etc/kubernetes/pki/ca.crt: no such file or directory.

Master node is in ready state:

[root@kubemaster1 ~]# kubectl get nodes
NAME          STATUS    ROLES     AGE       VERSION
kubemaster1   Ready     master    40m       v1.8.5

Master shows 6443 is open:

[root@kubemaster1 ~]# netstat -nltp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      1242/etcd           
tcp        0      0 127.0.0.1:10251         0.0.0.0:*               LISTEN      1137/kube-scheduler 
tcp        0      0 127.0.0.1:2380          0.0.0.0:*               LISTEN      1242/etcd           
tcp        0      0 127.0.0.1:10252         0.0.0.0:*               LISTEN      1179/kube-controlle 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      997/sshd            
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1243/master         
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      947/kubelet         
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      1321/kube-proxy     
tcp6       0      0 :::6443                 :::*                    LISTEN      1147/kube-apiserver 
tcp6       0      0 :::10255                :::*                    LISTEN      947/kubelet         
tcp6       0      0 :::10256                :::*                    LISTEN      1321/kube-proxy     
tcp6       0      0 :::22                   :::*                    LISTEN      997/sshd            
tcp6       0      0 ::1:25                  :::*                    LISTEN      1243/master         
tcp6       0      0 :::10250                :::*                    LISTEN      947/kubelet

Docker version of both master and worker is the same:

[root@kubemaster1 ~]# docker -v
Docker version 1.12.6, build ec8512b/1.12.6

[root@kubeworker1 ~]# docker -v
Docker version 1.12.6, build ec8512b/1.12.6

Nmap of master port 6443:

[root@kubeworker1 ~]# nmap <ip> -p 6443

Starting Nmap 6.40 ( http://nmap.org ) at 2017-12-13 18:22 EST
Nmap scan report for kubemaster1 (<ip>)
Host is up (0.00021s latency).
PORT     STATE    SERVICE
6443/tcp filtered unknown
MAC Address: <MAC>

Nmap done: 1 IP address (1 host up) scanned in 0.05 seconds

Is there something I am forgetting?

edited For record keeping or anyone with the same issue.

I flushed iptables on the master node using the following and this fixed the issue for me:

  iptables --flush
  iptables -tnat --flush

I noticed that kube-dns pod was failing and this flush allowed kube-dns to start up successfully and thereafter was able to join the cluster via kubeadm join

I just ran into the same problem and the source for this seems to be that the instructions at e.g. https://kubernetes.io/docs/setup/independent/install-kubeadm/ explicitly say to systemctl enable kubelet && systemctl start kubelet before anything is done with kubeadm which apparently cannot work.

Hi,

My problem was the version of Docker on the OS. It was using the latest from:

sudo wget -qO- https://get.docker.com/ | sh

I uninstalled and instead ran:

yum install -y docker
systemctl enable docker && systemctl start docker

And then kubeadm init worked kubelet was started successfully

Maybe extra prechecks are needed & the documentation could be improved. I cannot see the required version of Docker mentioned in the docs https://kubernetes.io/docs/setup/independent/install-kubeadm/ Or maybe document what changes are required if someone does install the latest Docker version.

Delete the current settings using following and restart the master settings kubeadm reset kubeadm init

@andrewhowdencom Had the same issue and resolved it with kubeadm init --ignore-preflight-errors CRI

kubeadm will generate the keys after the check