kubernetes: kubelet failed start after config kubeReservedCgroup
What happened?
When I config kubeReservedCgroup and kubeReserved in config.yaml for kube daemon.Kubelet can not start with the following errors:
vim /var/lib/kubelet/config.yaml
......
kind: KubeletConfiguration
enforceNodeAllocatable:
- pods
- kube-reserved
kubeReserved:
cpu: 1500m
memory: 1Gi
kubeReservedCgroup: /system.slice/kubelet.service
systemReserved:
cpu: 2000m
memory: 2Gi
......

I create all the files follow the errors
mkdir -p /sys/fs/cgroup/cpuset/kubelet.service.slice
mkdir -p /sys/fs/cgroup/hugetlb/kubelet.service.slice
mkdir -p /sys/fs/cgroup/memory/kubelet.service.slice
mkdir -p /sys/fs/cgroup/cpu,cpuacct/kubelet.service.slice
mkdir -p /sys/fs/cgroup/cpu,cpuacct/kubelet.service.slice
mkdir -p /sys/fs/cgroup/systemd/kubelet.service.slice
mkdir -p /sys/fs/cgroup/pids/kubelet.service.slice
Then restart the kubelet.
But the kubelet can not restart with the errors:Failed to enforce Kube Reserved Cgroup Limits on \"/system.slice/kubelet.service\": Unit kubelet.service.slice is not loaded
.
5月 30 22:08:20 VM-0-116-centos kubelet[5949]: E0530 22:08:20.149768 5949 kubelet.go:1384] "Failed to start ContainerManager" err="Failed to enforce Kube Reserved Cgroup Limits on \"/system.slice/kubelet.service\": Unit kubelet.service.slice is not loaded."
5月 30 22:08:20 VM-0-116-centos kubelet[5949]: I0530 22:08:20.149788 5949 event.go:291] "Event occurred" object="10.0.0.116" kind="Node" apiVersion="" type="Warning" reason="FailedNodeAllocatableEnforcement" message="Failed to enforce Kube Reserved Cgroup Limits on \"/system.slice/kubelet.service\": Unit kubelet.service.slice is not loaded."
5月 30 22:08:30 VM-0-116-centos systemd[1]: kubelet.service holdoff time over, scheduling restart.
5月 30 22:08:30 VM-0-116-centos systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
5月 30 22:08:30 VM-0-116-centos systemd[1]: Started kubelet: The Kubernetes Node Agent.
kubelet try to start logs:
What did you expect to happen?
The kubelet can restart successfully.
How can we reproduce it (as minimally and precisely as possible)?
Config the kubeReservedcgroup
Anything else we need to know?
kubectl version: v1.21.4 container engine:containerd kubelet cgroup driver:systemd OS: CentOS 7
Kubernetes version
$ kubectl version
v1.21.4
Cloud provider
/
OS version
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here
Linux VM-0-116-centos 5.4.119-19-0009.1 #1 SMP Sun Jan 23 23:20:30 CST 2022 x86_64 x86_64 x86_64 GNU/Linux
# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
Install tools
Container runtime (CRI) and version (if applicable)
Related plugins (CNI, CSI, …) and versions (if applicable)
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 19 (10 by maintainers)
We should clarify the difference configuration with cgroupfs and systemd in doc. I will create a pr to fix it.
You should probably use an actual systemd slice to create the directories, instead of manually creating them, then there won’t be missing paths and it will integrate properly. https://www.freedesktop.org/software/systemd/man/systemd.slice.html
When using the systemd cgroupDriver, kubelet will append
.slice
to the specified cgroup roots.