kubernetes: kubelet.service fail to start up

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened: kubelet.service fail to start up

What you expected to happen: kubelet.service start up success

How to reproduce it (as minimally and precisely as possible):

[root@test-node1 system]# systemctl status kubelet -l
● kubelet.service - Kubernetes Kubelet Server
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; disabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since 三 2017-10-25 16:08:09 CST; 8s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
  Process: 18607 ExecStart=/usr/local/bin/kubelet $KUBE_LOGTOSTDERR $KUBE_LOG_LEVEL $KUBELET_API_SERVER $KUBELET_ADDRESS $KUBELET_PORT $KUBELET_HOSTNAME $KUBE_ALLOW_PRIV $KUBELET_POD_INFRA_CONTAINER $KUBELET_ARGS (code=exited, status=2)
 Main PID: 18607 (code=exited, status=2)

10月 25 16:08:09 test-node1 systemd[1]: Unit kubelet.service entered failed state.
10月 25 16:08:09 test-node1 kubelet[18607]: --vmodule moduleSpec                                       comma-separated list of pattern=N settings for file-filtered logging
10月 25 16:08:09 test-node1 kubelet[18607]: --volume-plugin-dir string                                 <Warning: Alpha feature> The full path of the directory in which to search for additional third party volume plugins (default "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/")
10月 25 16:08:09 test-node1 kubelet[18607]: --volume-stats-agg-period duration                         Specifies interval for kubelet to calculate and cache the volume disk usage for all pods and volumes.  To disable volume calculations, set to 0. (default 1m0s)
10月 25 16:08:09 test-node1 systemd[1]: kubelet.service failed.
10月 25 16:08:09 test-node1 systemd[1]: kubelet.service holdoff time over, scheduling restart.
10月 25 16:08:09 test-node1 systemd[1]: start request repeated too quickly for kubelet.service
10月 25 16:08:09 test-node1 systemd[1]: Failed to start Kubernetes Kubelet Server.
10月 25 16:08:09 test-node1 systemd[1]: Unit kubelet.service entered failed state.
10月 25 16:08:09 test-node1 systemd[1]: kubelet.service failed.

Anything else we need to know?: kubelet.service file

[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/local/bin/kubelet \
            $KUBE_LOGTOSTDERR \
            $KUBE_LOG_LEVEL \
            $KUBELET_API_SERVER \
            $KUBELET_ADDRESS \
            $KUBELET_PORT \
            $KUBELET_HOSTNAME \
            $KUBE_ALLOW_PRIV \
            $KUBELET_POD_INFRA_CONTAINER \
            $KUBELET_ARGS
Restart=on-failure

[Install]
WantedBy=multi-user.target

kubelet.service config file

###
## kubernetes kubelet (minion) config
#
## The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=192.168.8.67"
#
## The port for the info server to serve on
#KUBELET_PORT="--port=10250"
#
## You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=192.168.8.67"
#
## location of the api-server
KUBELET_API_SERVER="--api-servers=http://192.168.8.66:8080"
#
## pod infrastructure container
#KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image="
#
## Add your own!
KUBELET_ARGS="--cluster-dns=10.254.0.2 --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --require-kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local --hairpin-mode promiscuous-bridge --serialize-image-pulls=false"

Environment:

Kubernetes version: 1.8.0
Cloud provider or hardware configuration**: 1G 20G
OS (e.g. from /etc/os-release):CentOS7 1708
Kernel (e.g. uname -a): Linux test-node1 3.10.0-693.2.2.el7.x86_64 #1 SMP Tue Sep 12 22:26:13 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Install tools: binnary
Others:

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 26 (5 by maintainers)

Most upvoted comments

I also have the same issue when the kubelet always show error status 1/FAILURE

● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Wed 2017-10-25 15:20:06 UTC; 2s ago
     Docs: http://kubernetes.io/docs/
  Process: 32404 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
 Main PID: 32404 (code=exited, status=1/FAILURE)

is there any way to trace what makes this error? or maybe is there any way to fix it?

+56

darrylsepeda on Oct 25, 2017

please switchoff your swap memory i faced same problem and when i remove swap entry from my /etc/fstab file .then it worked in my case.also do swapoff

+28

awesomemayank007 on Nov 8, 2017

#yum install -y kubelet kubeadm kubectl docker Jst try installing docker 1.12…iam not getting solution with docker 1.17ce.( i think its not supported with kubelet) Make swap off by #swapoff -a Now reset kubeadm by #kubeadm reset Now try #kudeadm init after that check #systemctl status kubelet it will be working

+21

arunm8489 on Jan 11, 2018

It worked for me after changing the cgroup to systemd in docker. To do that I edited the /etc/docker/daemon.json file to

{
  "exec-opts": ["native.cgroupdriver=systemd"]
}

And restarted docker with sudo systemctl restart docker

+11

yogesh174 on May 29, 2021

I had the same issue, journalctl revealed:

Mar 25 10:22:53 ubuntu kubelet[3216]: error: failed to run Kubelet: Running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false. /proc/swaps contained: [

I ran:

swapoff -a

then restarted kubelet, all good.

MarounMaroun on Mar 25, 2018