kubernetes: kubelet can't get node
When I Use kubeadm init --config /etc/kubernetes/kubeadm.yml
to install kubernetes, it hangs and reports:
# kubeadm init --config /etc/kubernetes/kubeadm.yml
[init] using Kubernetes version: v1.12.2
[preflight] running pre-flight checks
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [k8s-master-001 localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [k8s-master-001 localhost] and IPs [10.1.254.103 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [k8s-master-001 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.1.254.103]
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[certificates] Generated sa key and public key.
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
couldn't initialize a Kubernetes cluster
kubelet logs as follows:
Oct 28 09:32:14 k8s-master-001 kubelet[18343]: E1028 09:32:14.913402 18343 kubelet.go:2236] node "k8s-master-001" not found
Oct 28 09:32:15 k8s-master-001 kubelet[18343]: E1028 09:32:15.013584 18343 kubelet.go:2236] node "k8s-master-001" not found
Oct 28 09:32:15 k8s-master-001 kubelet[18343]: E1028 09:32:15.113773 18343 kubelet.go:2236] node "k8s-master-001" not found
Oct 28 09:32:15 k8s-master-001 kubelet[18343]: E1028 09:32:15.147608 18343 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Node: Get https://10.1.254.103:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-master-001&limit=500&resourceVersion=0: dial tcp 10.1.254.103:6443: connect: connection refused
Oct 28 09:32:15 k8s-master-001 kubelet[18343]: E1028 09:32:15.148489 18343 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.1.254.103:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dk8s-master-001&limit=500&resourceVersion=0: dial tcp 10.1.254.103:6443: connect: connection refused
Oct 28 09:32:15 k8s-master-001 kubelet[18343]: E1028 09:32:15.149459 18343 reflector.go:134] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v1.Service: Get https://10.1.254.103:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.1.254.103:6443: connect: connection refused
Oct 28 09:32:15 k8s-master-001 kubelet[18343]: E1028 09:32:15.213963 18343 kubelet.go:2236] node "k8s-master-001" not found
But in my /etc/hosts, it has the record:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.1.254.103 k8s-master-001
and I can ping k8s-master-001 successful, the uname -n
is also k8s-master-001.
docker ps -a | grep kube
get nothing.
Why kubelet can’t recognize my host, but apiserver and etcd can recognize it. it’s so strange, can somebody explain it, thanks!
Environment:
- Kubernetes version (use
kubectl version
): v1.12.2 - OS (e.g. from /etc/os-release): Red Hat Enterprise Linux Server release 7.5
- Kernel (e.g.
uname -a
): 3.10.0-862.14.4.el7.x86_64
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 28 (4 by maintainers)
Any updates on this yet? I’m also facing the same issue on Kubernetes v1.13.4
I have the same issue Docker version 18.09.7, kubernetes v1.16.2, Ubuntu 16.04
I was able to resolve this issue for my use-case by having the same cgroup driver for docker and kubelet. In my case on CentOS 7.6 I could fix the issue by adding
--exec-opt native.cgroupdriver=systemd
to docker systemd process and adding--cgroup-driver=systemd
to kubelet systemd process. This way both kubelet and docker are consuming the same cgroup-driver and both operate normally.Most likely these drivers can be set with any other driver types as well but that was not a part of my testing.