kubernetes: kubelet won't start - /etc/kubernetes/pki/ca.crt: no such file or directory
Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug /sig node
What happened:
Performed a clean install on CentOS Linux release 7.4.1708
and started kubelet
yum install -y kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet
systemctl status kubelet
tail -100f /var/log/messages | grep kubelet
Oct 13 12:25:26 master systemd: Started kubelet: The Kubernetes Node Agent.
Oct 13 12:25:26 master systemd: Starting kubelet: The Kubernetes Node Agent...
Oct 13 12:25:26 master kubelet: I1013 12:25:26.185686 1425 feature_gate.go:156] feature gates: map[]
Oct 13 12:25:26 master kubelet: I1013 12:25:26.185913 1425 controller.go:114] kubelet config controller: starting controller
Oct 13 12:25:26 master kubelet: I1013 12:25:26.185919 1425 controller.go:118] kubelet config controller: validating combination of defaults and flags
Oct 13 12:25:26 master kubelet: error: unable to load client CA file /etc/kubernetes/pki/ca.crt: open /etc/kubernetes/pki/ca.crt: no such file or directory
Oct 13 12:25:26 master systemd: kubelet.service: main process exited, code=exited, status=1/FAILURE
Oct 13 12:25:26 master systemd: Unit kubelet.service entered failed state.
Oct 13 12:25:26 master systemd: kubelet.service failed.
Oct 13 12:25:28 master systemd: Configuration file /etc/systemd/system/kubelet.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Oct 13 12:25:36 master systemd: kubelet.service holdoff time over, scheduling restart.
What you expected to happen: kubelet starts successfully
How to reproduce it (as minimally and precisely as possible):
yum install -y kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet
systemctl status kubelet
tail -100f /var/log/messages | grep kubelet
Anything else we need to know?:
Environment:
- Kubernetes version (use
kubectl version
):
kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-11T23:27:35Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
- Cloud provider or hardware configuration**:
CentOS Linux release 7.4.1708
vm running on VMWare vCenter Server6.0.0, 2776511
- OS (e.g. from /etc/os-release):
cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
- Kernel (e.g.
uname -a
):
uname -a
Linux master 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
- Install tools: kubelet kubeadm kubectl
- Others:
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 15 (2 by maintainers)
I found that
kubeadm init
will start the kubelet. Its working for me.When you run “kubeadm init” , it does: [certificates] Generated ca certificate and key. [certificates] Generated apiserver certificate and key. [certificates] apiserver serving cert is signed for DNS names [****] [certificates] Generated apiserver-kubelet-client certificate and key. [certificates] Generated sa key and public key. [certificates] Generated front-proxy-ca certificate and key. [certificates] Generated front-proxy-client certificate and key. [certificates] Valid certificates and keys now exist in “/etc/kubernetes/pki”
@dennisjac is correct, in the instructions, this step is obviously missing
but the funny part is, when “kubeadm init” it requires “kubelet is running” already: Unfortunately, an error has occurred: timed out waiting for the condition
This error is likely caused by that: - The kubelet is not running - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
Now I am in the dead loop
https://kubernetes.io/docs/setup/independent/install-kubeadm/#installing-docker has specific docker install instructions and version recommendations:
in my case I had to:
and:
then reinstall
I’m able to get master up with
kubeadm init --apiserver-advertise-address=<ip>
but I’m unable to join the cluster.kublete on the worker is getting
kubelet[13291]: error: unable to load client CA file /etc/kubernetes/pki/ca.crt: open /etc/kubernetes/pki/ca.crt: no such file or directory
.Master node is in ready state:
Master shows 6443 is open:
Docker version of both master and worker is the same:
Nmap of master port 6443:
Is there something I am forgetting?
edited For record keeping or anyone with the same issue.
I flushed iptables on the master node using the following and this fixed the issue for me:
I noticed that kube-dns pod was failing and this flush allowed kube-dns to start up successfully and thereafter was able to join the cluster via
kubeadm join
I just ran into the same problem and the source for this seems to be that the instructions at e.g. https://kubernetes.io/docs/setup/independent/install-kubeadm/ explicitly say to
systemctl enable kubelet && systemctl start kubelet
before anything is done with kubeadm which apparently cannot work.Hi,
My problem was the version of Docker on the OS. It was using the latest from:
I uninstalled and instead ran:
And then
kubeadm init
workedkubelet
was started successfullyMaybe extra prechecks are needed & the documentation could be improved. I cannot see the required version of Docker mentioned in the docs https://kubernetes.io/docs/setup/independent/install-kubeadm/ Or maybe document what changes are required if someone does install the latest Docker version.
Delete the current settings using following and restart the master settings kubeadm reset kubeadm init
@andrewhowdencom Had the same issue and resolved it with
kubeadm init --ignore-preflight-errors CRI
kubeadm will generate the keys after the check