kubernetes: kubelet failed start after config kubeReservedCgroup

What happened?

When I config kubeReservedCgroup and kubeReserved in config.yaml for kube daemon.Kubelet can not start with the following errors:

vim /var/lib/kubelet/config.yaml
......
kind: KubeletConfiguration
enforceNodeAllocatable:
  - pods
  - kube-reserved
kubeReserved:
  cpu: 1500m
  memory: 1Gi
kubeReservedCgroup: /system.slice/kubelet.service
systemReserved:
  cpu: 2000m
  memory: 2Gi
......
image

I create all the files follow the errors

mkdir -p /sys/fs/cgroup/cpuset/kubelet.service.slice
mkdir -p /sys/fs/cgroup/hugetlb/kubelet.service.slice
mkdir -p /sys/fs/cgroup/memory/kubelet.service.slice
mkdir -p /sys/fs/cgroup/cpu,cpuacct/kubelet.service.slice
mkdir -p /sys/fs/cgroup/cpu,cpuacct/kubelet.service.slice
mkdir -p /sys/fs/cgroup/systemd/kubelet.service.slice
mkdir -p /sys/fs/cgroup/pids/kubelet.service.slice

Then restart the kubelet. But the kubelet can not restart with the errors:Failed to enforce Kube Reserved Cgroup Limits on \"/system.slice/kubelet.service\": Unit kubelet.service.slice is not loaded.

5月 30 22:08:20 VM-0-116-centos kubelet[5949]: E0530 22:08:20.149768    5949 kubelet.go:1384] "Failed to start ContainerManager" err="Failed to enforce Kube Reserved Cgroup Limits on \"/system.slice/kubelet.service\": Unit kubelet.service.slice is not loaded."
5月 30 22:08:20 VM-0-116-centos kubelet[5949]: I0530 22:08:20.149788    5949 event.go:291] "Event occurred" object="10.0.0.116" kind="Node" apiVersion="" type="Warning" reason="FailedNodeAllocatableEnforcement" message="Failed to enforce Kube Reserved Cgroup Limits on \"/system.slice/kubelet.service\": Unit kubelet.service.slice is not loaded."
5月 30 22:08:30 VM-0-116-centos systemd[1]: kubelet.service holdoff time over, scheduling restart.
5月 30 22:08:30 VM-0-116-centos systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
5月 30 22:08:30 VM-0-116-centos systemd[1]: Started kubelet: The Kubernetes Node Agent.

kubelet try to start logs: image

What did you expect to happen?

The kubelet can restart successfully.

How can we reproduce it (as minimally and precisely as possible)?

Config the kubeReservedcgroup

Anything else we need to know?

kubectl version: v1.21.4 container engine:containerd kubelet cgroup driver:systemd OS: CentOS 7

Kubernetes version

$ kubectl version
v1.21.4

Cloud provider

/

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here
Linux VM-0-116-centos 5.4.119-19-0009.1 #1 SMP Sun Jan 23 23:20:30 CST 2022 x86_64 x86_64 x86_64 GNU/Linux
# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, …) and versions (if applicable)

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19 (10 by maintainers)

Most upvoted comments

We should clarify the difference configuration with cgroupfs and systemd in doc. I will create a pr to fix it.

You should probably use an actual systemd slice to create the directories, instead of manually creating them, then there won’t be missing paths and it will integrate properly. https://www.freedesktop.org/software/systemd/man/systemd.slice.html

When using the systemd cgroupDriver, kubelet will append .slice to the specified cgroup roots.