kubernetes: kubelet 1.11 won't start because /var/lib/kubelet/config.yaml is missing/empty on brand new install

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

What happened: no /var/lib/kubelet/config.yaml after brand new install of kubelet 1.11

How to reproduce it (as minimally and precisely as possible): did a simple install “yum install -y kubelet kubeadm kubectl”

when trying “systemctl start kubelet” the service doesn’t start and enters into an endless loop: Jul 12 03:17:40 116AV-4.dcoe.sio.emc.com systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a Jul 12 03:17:40 116AV-4.dcoe.sio.emc.com systemd[1]: Unit kubelet.service entered failed state. Jul 12 03:17:40 116AV-4.dcoe.sio.emc.com systemd[1]: kubelet.service failed. Jul 12 03:17:50 116AV-4.dcoe.sio.emc.com systemd[1]: kubelet.service holdoff time over, scheduling restart. Jul 12 03:17:50 116AV-4.dcoe.sio.emc.com systemd[1]: Started kubelet: The Kubernetes Node Agent. Jul 12 03:17:50 116AV-4.dcoe.sio.emc.com systemd[1]: Starting kubelet: The Kubernetes Node Agent… Jul 12 03:17:50 116AV-4.dcoe.sio.emc.com kubelet[25480]: I0712 03:17:50.970349 25480 feature_gate.go:230] feature gates: &{map[]} Jul 12 03:17:50 116AV-4.dcoe.sio.emc.com kubelet[25480]: F0712 03:17:50.970502 25480 server.go:190] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file “/var/lib/kubelet/config.yaml”, error: open /var/lib/kubelet/config.yaml: no such file or directory

What you expected to happen: kubelet should start with some minimal config.

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version): kubectl version Client Version: version.Info{Major:“1”, Minor:“11”, GitVersion:“v1.11.0”, GitCommit:“91e7b4fd31fcd3d5f436da26c980becec37ceefe”, GitTreeState:“clean”, BuildDate:“2018-06-27T20:17:28Z”, GoVersion:“go1.10.2”, Compiler:“gc”, Platform:“linux/amd64”} The connection to the server localhost:8080 was refused - did you specify the right host or port?

  • Cloud provider or hardware configuration: Hardware, bare-metal

  • OS (e.g. from /etc/os-release): cat /etc/redhat-release CentOS Linux release 7.3.1611 (Core)

  • Kernel (e.g. uname -a): Linux 116AV-4.dcoe.sio.emc.com 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

  • Install tools:

  • Others:

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 21 (3 by maintainers)

Most upvoted comments

you must run kubeadm init per the setup instructions to create the required configuration.

see https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#instructions

/close

I used the following command to create a default config (on a fresh installation):

kubeadm init phase kubelet-start

This solved my problem.

@liggitt Thanks for explaining that, I thought the kubelet service had to be running before kubeadm join.

I just got it working. The kubadm join command initially printed by my master during kubeadm init wasn’t working (token invalidated somehow?), so I generated a new token on the master using sudo kubeadm token create --print-join-command, then ran that join command instead, which worked as expected.

I’m running into the same problem with a fresh CentOS 7 install for a new worker node.

@sagyvolkov Did you find a solution for this?

@liggitt In my case, the master has already been initialized, and I’m trying to run kubeadm join on a worker node to join the cluster, but getting an Unauthorized error message - further inspection showed that kubelet service is not starting, reporting the same error shown above every 10 seconds as kubelet tries to restart:

kubelet: F0727 21:03:15.330129 2860 server.go:190] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file “/var/lib/kubelet/config.yaml”, error: open /var/lib/kubelet/config.yaml: no such file or directory

The entire /var/lib/kubelet directory is missing.

I ran thru the same steps on a different system a couple weeks ago (perhaps with v1.11.0) but it’s not working today with v1.11.1.

@greenlaw - yes, just run the init command. its confusing and of course not mentioned in any documentation 😃

By default, tokens expire after 24 hours. Check full installation guide here: https://zhuanlan.zhihu.com/p/40931670

I used the following command to create a default config (on a fresh installation):

kubeadm init phase kubelet-start

This solved my problem.

thanks it is work for me, but after that when i tried to restart then not started, enable, demon, restart…as per the doc…what could be the reason? how can i solve it.

kubeadm init phase kubelet-start

[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env” [kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml” [kubelet-start] Activating the kubelet service

The join command is responsible for pulling down cluster info and setting up the kubelet config. If that is failing, it is not surprising there is no config dir and the kubelet will not start.

Determining why the join command is failing would be the first step, but is a separate issue