kubernetes: couldn't initialize a Kubernetes cluster
/sig component-standard
@kubernetes/sig-component-standard
What happened:
couldn’t initialize a Kubernetes cluster
What you expected to happen:
create the cluster normally
How to reproduce it (as minimally and precisely as possible):
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.0.80
Anything else we need to know?:
the details output are shown below.
kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.0.80 [init] Using Kubernetes version: v1.13.4 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using ‘kubeadm config images pull’ [kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env” [kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml” [kubelet-start] Activating the kubelet service [certs] Using certificateDir folder “/etc/kubernetes/pki” [certs] Generating “ca” certificate and key [certs] Generating “apiserver-kubelet-client” certificate and key [certs] Generating “apiserver” certificate and key [certs] apiserver serving cert is signed for DNS names [pimaster kubernetes kubernetes.default > kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.80] [certs] Generating “front-proxy-ca” certificate and key [certs] Generating “front-proxy-client” certificate and key [certs] Generating “etcd/ca” certificate and key [certs] Generating “etcd/server” certificate and key [certs] etcd/server serving cert is signed for DNS names [pimaster localhost] and IPs [192.168.0.80 127.0.0.1 ::1] [certs] Generating “etcd/peer” certificate and key [certs] etcd/peer serving cert is signed for DNS names [pimaster localhost] and IPs [192.168.0.80 127.0.0.1 ::1] [certs] Generating “apiserver-etcd-client” certificate and key [certs] Generating “etcd/healthcheck-client” certificate and key [certs] Generating “sa” key and public key [kubeconfig] Using kubeconfig folder “/etc/kubernetes” [kubeconfig] Writing “admin.conf” kubeconfig file [kubeconfig] Writing “kubelet.conf” kubeconfig file [kubeconfig] Writing “controller-manager.conf” kubeconfig file [kubeconfig] Writing “scheduler.conf” kubeconfig file [control-plane] Using manifest folder “/etc/kubernetes/manifests” [control-plane] Creating static Pod manifest for “kube-apiserver” [control-plane] Creating static Pod manifest for “kube-controller-manager” [control-plane] Creating static Pod manifest for “kube-scheduler” [etcd] Creating static Pod manifest for local etcd in “/etc/kubernetes/manifests” [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory “/etc/kubernetes/manifests”. This can take up to 4m0s [kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred: timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- ‘systemctl status kubelet’
- ‘journalctl -xeu kubelet’
Additionally, a control plane component may have crashed or exited when started by the container runtime. To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker. Here is one example how you may list all Kubernetes containers running in docker:
- ‘docker ps -a | grep kube | grep -v pause’ Once you have found the failing container, you can inspect its logs with:
- ‘docker logs CONTAINERID’ error execution phase wait-control-plane: couldn’t initialize a Kubernetes cluster
Environment:
- Kubernetes version (use kubectl version): v1.13.4
- Docker: 18.06.3-ce
- Cloud provider or hardware configuration: raspberrypi / 2018-11-13-raspbian-stretch-lite
- OS (e.g. from /etc/os-release): Raspbian (2018-11-13-raspbian-stretch-lite)
- Others: - I disable swap by cmd: dphys-swapfile swapoff && dphys-swapfile uninstall && update-rc.d dphys-swapfile remove - I add “cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1” in /boot/cmdline.txt - I have test add Environment=“KUBELET_EXTRA_ARGS=–fail-swap-on=false” without succes (from https://github.com/kubernetes/kubernetes/issues/61277) - return of cmd systemctl status kubelet
systemctl status kubelet● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: active (running) since Sat 2019-03-02 18:04:56 GMT; 6min ago Docs: https://kubernetes.io/docs/home/ Main PID: 29720 (kubelet) Memory: 25.9M CPU: 31.069s CGroup: /system.slice/kubelet.service ├─29720 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yam └─31626 iptables -w 5 -C KUBE-FIREWALL -t filter -m comment --comment kubernetes firewall for dropping marked packets -m mark --mark 0x00008000/0x00008000 -j DROP
Mar 02 18:10:43 pimaster kubelet[29720]: E0302 18:10:43.304176 29720 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady mes Mar 02 18:10:48 pimaster kubelet[29720]: W0302 18:10:48.308568 29720 cni.go:203] Unable to update cni config: No networks found in /etc/cni/net.d Mar 02 18:10:48 pimaster kubelet[29720]: E0302 18:10:48.310978 29720 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady mes Mar 02 18:10:53 pimaster kubelet[29720]: W0302 18:10:53.316838 29720 cni.go:203] Unable to update cni config: No networks found in /etc/cni/net.d Mar 02 18:10:53 pimaster kubelet[29720]: E0302 18:10:53.317876 29720 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady mes Mar 02 18:10:55 pimaster kubelet[29720]: E0302 18:10:55.234181 29720 event.go:203] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:“”, APIVersion:“”}, ObjectMeta:v Mar 02 18:10:58 pimaster kubelet[29720]: W0302 18:10:58.321466 29720 cni.go:203] Unable to update cni config: No networks found in /etc/cni/net.d Mar 02 18:10:58 pimaster kubelet[29720]: E0302 18:10:58.322096 29720 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady mes Mar 02 18:11:03 pimaster kubelet[29720]: W0302 18:11:03.326820 29720 cni.go:203] Unable to update cni config: No networks found in /etc/cni/net.d
Anyone can help me? Thanks a lot.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 18 (3 by maintainers)
I eventually did, but at this time I don’t recall the exact fix. I think it was this: KUBECONF=/etc/systemd/system/kubelet.service.d/10-kubeadm.conf sed -i ‘s/cgroupfs/systemd/g’ $KUBECONF sed -i ‘s|/etc/kubernetes/manifests|/etc/kubernetes/manifests --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice|’ $KUBECONF systemctl daemon-reload ; systemctl enable kubelet ; systemctl restart kubelet