kubeadm: kubeadm init fails

What keywords did you search in kubeadm issues before filing this one?

init

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version

&version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-10T23:33:30Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/arm"}

Environment:

  • Kubernetes version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.2", GitCommit:"cff46ab41ff0bb44d8584413b598ad8360ec1def", GitTreeState:"clean", BuildDate:"2019-01-10T23:35:51Z", GoVersion:"go1.11.4", Compiler:"gc", Platform:"linux/arm"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?
  • Cloud provider or hardware configuration:

RPi3 B+

  • OS:
PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"
NAME="Raspbian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
  • Kernel (e.g. uname -a):
Linux km01 4.14.34-hypriotos-v7+ #1 SMP Sun Apr 22 14:57:31 UTC 2018 armv7l GNU/Linux
  • Others:

What happened?

Fresh install of hypriotos-rpi-v1.9.0. Then apt-get install -y kubeadm. So far so good. As root kubeadm init --pod-network-cidr 10.244.0.0/16 fails with:

# kubeadm init --pod-network-cidr 10.244.0.0/16
[init] Using Kubernetes version: v1.13.2
[preflight] Running pre-flight checks
  [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [km01 localhost] and IPs [192.168.178.43 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [km01 localhost] and IPs [192.168.178.43 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [km01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.178.43]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.

Unfortunately, an error has occurred:
  timed out waiting for the condition

This error is likely caused by:
  - The kubelet is not running
  - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
  - 'systemctl status kubelet'
  - 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
  - 'docker ps -a | grep kube | grep -v pause'
  Once you have found the failing container, you can inspect its logs with:
  - 'docker logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster

# docker version
Client:
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:57:21 2018
 OS/Arch:           linux/arm
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:17:57 2018
  OS/Arch:          linux/arm
  Experimental:     false

# docker ps -a
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS                     PORTS               NAMES
72e8af203fcc        7a29ac9f3098           "kube-apiserver --au…"   2 minutes ago       Up 2 minutes                                   k8s_kube-apiserver_kube-apiserver-km01_kube-system_f4ce15d3927feb3b0957420d19d0c935_3
522449a6bc58        7a29ac9f3098           "kube-apiserver --au…"   3 minutes ago       Exited (0) 2 minutes ago                       k8s_kube-apiserver_kube-apiserver-km01_kube-system_f4ce15d3927feb3b0957420d19d0c935_2
fba7a40b03f3        2d981d285d92           "kube-scheduler --ad…"   7 minutes ago       Up 6 minutes                                   k8s_kube-scheduler_kube-scheduler-km01_kube-system_9729a196c4723b60ab401eaff722982d_0
d8671e640520        e7a8884c8443           "etcd --advertise-cl…"   7 minutes ago       Up 6 minutes                                   k8s_etcd_etcd-km01_kube-system_3e14953b2357b11169e86844e2c48a10_0
30cd3ecbdaea        a0e1a8b762a2           "kube-controller-man…"   7 minutes ago       Up 6 minutes                                   k8s_kube-controller-manager_kube-controller-manager-km01_kube-system_097345e297e344d595052996fbb45893_0
5162020dd593        k8s.gcr.io/pause:3.1   "/pause"                 7 minutes ago       Up 6 minutes                                   k8s_POD_etcd-km01_kube-system_3e14953b2357b11169e86844e2c48a10_0
ff429ccce911        k8s.gcr.io/pause:3.1   "/pause"                 7 minutes ago       Up 6 minutes                                   k8s_POD_kube-scheduler-km01_kube-system_9729a196c4723b60ab401eaff722982d_0
c154467b6f6d        k8s.gcr.io/pause:3.1   "/pause"                 7 minutes ago       Up 6 minutes                                   k8s_POD_kube-controller-manager-km01_kube-system_097345e297e344d595052996fbb45893_0
c79165dffa4b        k8s.gcr.io/pause:3.1   "/pause"                 7 minutes ago       Up 6 minutes                                   k8s_POD_kube-apiserver-km01_kube-system_f4ce15d3927feb3b0957420d19d0c935_0

# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Fri 2019-02-01 21:22:08 CET; 8min ago
     Docs: https://kubernetes.io/docs/home/
 Main PID: 28230 (kubelet)
    Tasks: 24 (limit: 4915)
   Memory: 27.6M
      CPU: 46.457s
   CGroup: /system.slice/kubelet.service
           └─28230 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cg

Feb 01 21:29:50 km01 kubelet[28230]: E0201 21:29:50.623818   28230 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plu
Feb 01 21:29:55 km01 kubelet[28230]: W0201 21:29:55.628723   28230 cni.go:203] Unable to update cni config: No networks found in /etc/cni/net.d
Feb 01 21:29:55 km01 kubelet[28230]: E0201 21:29:55.630138   28230 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plu
Feb 01 21:30:00 km01 kubelet[28230]: W0201 21:30:00.632774   28230 cni.go:203] Unable to update cni config: No networks found in /etc/cni/net.d

# journalctl -xeu kubelet
Feb 01 21:30:40 km01 kubelet[28230]: E0201 21:30:40.686723   28230 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plu
Feb 01 21:30:44 km01 kubelet[28230]: E0201 21:30:44.109239   28230 dns.go:132] Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 8.8.8.8 8.8.4.4 
Feb 01 21:30:45 km01 kubelet[28230]: W0201 21:30:45.691267   28230 cni.go:203] Unable to update cni config: No networks found in /etc/cni/net.d
Feb 01 21:30:45 km01 kubelet[28230]: E0201 21:30:45.692066   28230 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plu

What you expected to happen?

Finish of the init and then giving me the join command.

How to reproduce it (as minimally and precisely as possible)?

Fresh install of hypriotos-rpi-v1.9.0 then:

sudo bash <<EOF
curl -sSL https://packagecloud.io/Hypriot/rpi/gpgkey | apt-key add -
curl -sSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list
apt-get update && apt-get install -y kubeadm
kubeadm init --pod-network-cidr 10.244.0.0/16
EOF

Anything else we need to know?

This has worked before (as in yesterday). I don’t see anything that might have changed. Unless there was a new release like today or yesterday I am running out of ideas.

I am not sure the cni config is a problem yet. Installing flannel would be next on the list.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 35 (15 by maintainers)

Most upvoted comments

Holy smokes - that worked! 🎉

sudo kubeadm reset
sudo kubeadm init phase certs all
sudo kubeadm init phase kubeconfig all
sudo kubeadm init phase control-plane all --pod-network-cidr 10.244.0.0/16
sudo sed -i 's/initialDelaySeconds: [0-9][0-9]/initialDelaySeconds: 240/g' /etc/kubernetes/manifests/kube-apiserver.yaml
sudo sed -i 's/failureThreshold: [0-9]/failureThreshold: 18/g'             /etc/kubernetes/manifests/kube-apiserver.yaml
sudo sed -i 's/timeoutSeconds: [0-9][0-9]/timeoutSeconds: 20/g'            /etc/kubernetes/manifests/kube-apiserver.yaml
sudo kubeadm init --v=1 --skip-phases=certs,kubeconfig,control-plane --ignore-preflight-errors=all --pod-network-cidr 10.244.0.0/16
+13
tcurdt on Feb 2, 2019

I am so unbelievably grateful for this thread - thank you @neolit123 and everyone else involved - I finally got a Raspberry Pi 3 B to run a K8s master after three SOLID days of trying!

What’s weird is, the very first time I tried, I followed this tutorial and it worked great, I set up a 3 node cluster, no problem. I shut it down at night, the next day I started it up - and nothing worked!

So, I re-etchered my SD cards and started afresh. Then the pain began. I tried every permutation of Docker version, Raspiian version, all sorts of flags and kernel versions, even bought a network switch and started flailing at that, must have read over 100 web pages and Github issues, nothing at all worked until I used the steps that @tcurdt used above.

Anyway, THANK YOU

+1
gavD on Jul 25, 2019

Yes, the panic is really odd - and still leaves me a little puzzled. But well - for that discussion we got the other issue. Thanks for the help!

+1
tcurdt on Feb 2, 2019

I then installed flannel

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.11.0/Documentation/kube-flannel.yml

and after a while - yay!

$ kubectl get nodes
NAME   STATUS   ROLES    AGE     VERSION
km01   Ready    master   3m43s   v1.13.3
+1
tcurdt on Feb 2, 2019

posting in the panic ticket in a bit.

+1
neolit123 on Feb 2, 2019

also 1.13.3 was released as we speak: https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/kubernetes-dev/3V2-C5Z6HA0/OdwHVNABEgAJ

but please still file that panic bug report for 1.13.2.

+1
neolit123 on Feb 2, 2019

Understood - but is there an issue open for kubelet then?

AFAIK, no. because there is no proof it’s the kubelet.

was thinking I could init like this but apparently the --pod-network-cidr is not allowed as flag.

you can pass the podSubnet from the config (it’s the same as pod-network-cidr). https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1

Or is that maybe just not relevant for the control-plane phase?

it is relevant to the controller manager (so yes “control-plane” phase) and also kube-proxy.

[preflight] Running pre-flight checks [WARNING FileAvailable–etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists

try calling kubeadm reset first.

panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xaab708]

please file a separate bug report about the panic including the reproduction steps. might be only a case when you forgot to run reset.

+1
neolit123 on Feb 2, 2019
Read more comments on GitHub
← kubeadm: kubeadm init error marking master: timed out waiting for the condition
kubeadm: Kubeadm init fails with "Error writing Crisocket information for the control-plane node: timed out waiting for the condition" →