kubernetes: kubelet won't start - /etc/kubernetes/pki/ca.crt: no such file or directory

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug /sig node

What happened: Performed a clean install on CentOS Linux release 7.4.1708 and started kubelet

yum install -y kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet
systemctl status kubelet
tail -100f /var/log/messages | grep kubelet

Oct 13 12:25:26 master systemd: Started kubelet: The Kubernetes Node Agent.
Oct 13 12:25:26 master systemd: Starting kubelet: The Kubernetes Node Agent...
Oct 13 12:25:26 master kubelet: I1013 12:25:26.185686    1425 feature_gate.go:156] feature gates: map[]
Oct 13 12:25:26 master kubelet: I1013 12:25:26.185913    1425 controller.go:114] kubelet config controller: starting controller
Oct 13 12:25:26 master kubelet: I1013 12:25:26.185919    1425 controller.go:118] kubelet config controller: validating combination of defaults and flags
Oct 13 12:25:26 master kubelet: error: unable to load client CA file /etc/kubernetes/pki/ca.crt: open /etc/kubernetes/pki/ca.crt: no such file or directory
Oct 13 12:25:26 master systemd: kubelet.service: main process exited, code=exited, status=1/FAILURE
Oct 13 12:25:26 master systemd: Unit kubelet.service entered failed state.
Oct 13 12:25:26 master systemd: kubelet.service failed.
Oct 13 12:25:28 master systemd: Configuration file /etc/systemd/system/kubelet.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Oct 13 12:25:36 master systemd: kubelet.service holdoff time over, scheduling restart.

What you expected to happen: kubelet starts successfully

How to reproduce it (as minimally and precisely as possible):

yum install -y kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet
systemctl status kubelet
tail -100f /var/log/messages | grep kubelet

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-11T23:27:35Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration**: CentOS Linux release 7.4.1708 vm running on VMWare vCenter Server 6.0.0, 2776511
  • OS (e.g. from /etc/os-release):
cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
  • Kernel (e.g. uname -a):
uname -a
Linux master 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools: kubelet kubeadm kubectl
  • Others:

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 15 (2 by maintainers)

Most upvoted comments

I found that kubeadm init will start the kubelet. Its working for me.

When you run “kubeadm init” , it does: [certificates] Generated ca certificate and key. [certificates] Generated apiserver certificate and key. [certificates] apiserver serving cert is signed for DNS names [****] [certificates] Generated apiserver-kubelet-client certificate and key. [certificates] Generated sa key and public key. [certificates] Generated front-proxy-ca certificate and key. [certificates] Generated front-proxy-client certificate and key. [certificates] Valid certificates and keys now exist in “/etc/kubernetes/pki”

@dennisjac is correct, in the instructions, this step is obviously missing

but the funny part is, when “kubeadm init” it requires “kubelet is running” already: Unfortunately, an error has occurred: timed out waiting for the condition

This error is likely caused by that: - The kubelet is not running - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

Now I am in the dead loop

https://kubernetes.io/docs/setup/independent/install-kubeadm/#installing-docker has specific docker install instructions and version recommendations:

On each of your machines, install Docker. Version v1.12 is recommended, but v1.11, v1.13 and 17.03 are known to work as well. Versions 17.06+ might work, but have not yet been tested and verified by the Kubernetes node team.

in my case I had to:

kubeadm reset

and:

apt remove --purge kube*

then reinstall

I’m able to get master up with kubeadm init --apiserver-advertise-address=<ip> but I’m unable to join the cluster.

kublete on the worker is getting kubelet[13291]: error: unable to load client CA file /etc/kubernetes/pki/ca.crt: open /etc/kubernetes/pki/ca.crt: no such file or directory.

Master node is in ready state:

[root@kubemaster1 ~]# kubectl get nodes
NAME          STATUS    ROLES     AGE       VERSION
kubemaster1   Ready     master    40m       v1.8.5

Master shows 6443 is open:

[root@kubemaster1 ~]# netstat -nltp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      1242/etcd           
tcp        0      0 127.0.0.1:10251         0.0.0.0:*               LISTEN      1137/kube-scheduler 
tcp        0      0 127.0.0.1:2380          0.0.0.0:*               LISTEN      1242/etcd           
tcp        0      0 127.0.0.1:10252         0.0.0.0:*               LISTEN      1179/kube-controlle 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      997/sshd            
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1243/master         
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      947/kubelet         
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      1321/kube-proxy     
tcp6       0      0 :::6443                 :::*                    LISTEN      1147/kube-apiserver 
tcp6       0      0 :::10255                :::*                    LISTEN      947/kubelet         
tcp6       0      0 :::10256                :::*                    LISTEN      1321/kube-proxy     
tcp6       0      0 :::22                   :::*                    LISTEN      997/sshd            
tcp6       0      0 ::1:25                  :::*                    LISTEN      1243/master         
tcp6       0      0 :::10250                :::*                    LISTEN      947/kubelet

Docker version of both master and worker is the same:

[root@kubemaster1 ~]# docker -v
Docker version 1.12.6, build ec8512b/1.12.6

[root@kubeworker1 ~]# docker -v
Docker version 1.12.6, build ec8512b/1.12.6

Nmap of master port 6443:

[root@kubeworker1 ~]# nmap <ip> -p 6443

Starting Nmap 6.40 ( http://nmap.org ) at 2017-12-13 18:22 EST
Nmap scan report for kubemaster1 (<ip>)
Host is up (0.00021s latency).
PORT     STATE    SERVICE
6443/tcp filtered unknown
MAC Address: <MAC>

Nmap done: 1 IP address (1 host up) scanned in 0.05 seconds

Is there something I am forgetting?

edited For record keeping or anyone with the same issue.

I flushed iptables on the master node using the following and this fixed the issue for me:

  iptables --flush
  iptables -tnat --flush

I noticed that kube-dns pod was failing and this flush allowed kube-dns to start up successfully and thereafter was able to join the cluster via kubeadm join

I just ran into the same problem and the source for this seems to be that the instructions at e.g. https://kubernetes.io/docs/setup/independent/install-kubeadm/ explicitly say to systemctl enable kubelet && systemctl start kubelet before anything is done with kubeadm which apparently cannot work.

Hi,

My problem was the version of Docker on the OS. It was using the latest from:

sudo wget -qO- https://get.docker.com/ | sh

I uninstalled and instead ran:

yum install -y docker
systemctl enable docker && systemctl start docker

And then kubeadm init worked kubelet was started successfully

Maybe extra prechecks are needed & the documentation could be improved. I cannot see the required version of Docker mentioned in the docs https://kubernetes.io/docs/setup/independent/install-kubeadm/ Or maybe document what changes are required if someone does install the latest Docker version.

Delete the current settings using following and restart the master settings kubeadm reset kubeadm init

@andrewhowdencom Had the same issue and resolved it with kubeadm init --ignore-preflight-errors CRI

kubeadm will generate the keys after the check