kubernetes: Failed to get system container stats for "/system.slice/kubelet.service": failed to get cgroup stats for "/system.slice/kubelet.service": failed to get container info for "/system.slice/kubelet.service": unknown container "/system.slice/kubelet.service"

Also on AWS default image for kops (k8s-1.8-debian-jessie-amd64-hvm-ebs-2017-12-02 (ami-bd229ec4))

sudo vim /etc/sysconfig/kubelet

add at the end of DAEMON_ARGS string:

 --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice

finally:

sudo systemctl restart kubelet

adding a file in : /etc/systemd/system/kubelet.service.d/11-cgroups.conf

with: [Service] CPUAccounting=true MemoryAccounting=true

and: systemctl daemon-reload systemctl restart kubelet

solve the issue for me .

Adding /etc/systemd/system/kubelet.service.d/12-after-docker.conf with the content below seems to help:

[Unit]
After=docker.service

Do systemctl daemon-reload && systemctl restart kubelet after adding this file.

I’ve got the same errors on

• CentOS 7.4.1708
• docker 1.12.6
• kubeadm v1.9.4

Mar 14 08:32:35 ksa-m1.blue kubelet[9322]: E0314 08:32:34.998853    9322 summary.go:92] Failed to get system container stats for "/system.slice/kubelet.service": failed to get cgroup stats for "/system.slice/kubelet.service": failed to get container info for "/system.slice/kubelet.service": unknown container "/system.slice/kubelet.service"
Mar 14 08:32:35 ksa-m1.blue kubelet[9322]: E0314 08:32:34.998879    9322 summary.go:92] Failed to get system container stats for "/system.slice/docker.service": failed to get cgroup stats for "/system.slice/docker.service": failed to get container info for "/system.slice/docker.service": unknown container "/system.slice/docker.service"

And the above fix works for me. On CentOS I can add these options in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf:

# egrep KUBELET_CGROUP_ARGS= /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice"

Should we have this fix added to the kubeadm RPM by default?

Environment:

# cat /etc/redhat-release 
CentOS Linux release 7.4.1708 (Core) 

# uname -a
Linux ksa-m1.blue 3.10.0-514.6.1.el7.x86_64 #1 SMP Wed Jan 18 13:06:36 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

# kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.4", GitCommit:"bee2d1505c4fe820744d26d41ecd3fdd4a3d6546", GitTreeState:"clean", BuildDate:"2018-03-12T16:29:47Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.4", GitCommit:"bee2d1505c4fe820744d26d41ecd3fdd4a3d6546", GitTreeState:"clean", BuildDate:"2018-03-12T16:21:35Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

# rpm -qa | egrep kube
kubernetes-cni-0.6.0-0.x86_64
kubelet-1.9.4-0.x86_64
kubectl-1.9.4-0.x86_64
kubeadm-1.9.4-0.x86_64

# docker version 
Client:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-71.git3e8e77d.el7.centos.1.x86_64
 Go version:      go1.8.3
 Git commit:      3e8e77d/1.12.6
 Built:           Tue Jan 30 09:17:00 2018
 OS/Arch:         linux/amd64

Server:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-1.12.6-71.git3e8e77d.el7.centos.1.x86_64
 Go version:      go1.8.3
 Git commit:      3e8e77d/1.12.6
 Built:           Tue Jan 30 09:17:00 2018
 OS/Arch:         linux/amd64

/kind bug

# ubuntu
cat << EOF | sudo tee /etc/systemd/system/kubelet.service.d/12-after-docker.conf
[Unit]
After=docker.service
EOF

# centos
cat << EOF | sudo tee /usr/lib/systemd/system/kubelet.service.d/12-after-docker.conf
[Unit]
After=docker.service
EOF

If run kubelet with “–docker-only” parameter，this problem will happen，just remove this paramter from your config file. It seems that kubelet will query cgroup info by cadvisor client，if you set “–docker-only”， cadvisor will not collect kubelet’s and runtime’s cgroups，so query will fail.

I have record this process but in Chinese：Kubernetes问题调查：failed to get cgroup stats for /systemd/system.slice .

by the way，i think --runtime-cgroups and --kubelet-cgroups should be set like this if you want：

–kubelet-cgroups=/system.slice/kubelet.service –runtime-cgroups=/system.slice/docker.service

Their values depend on your enviroment, the above setting example is for CentOS 7, kubelet and docker are managed by systemd.

I wonder if specifying the particular cgroup (as noted in https://github.com/kubernetes/kubernetes/issues/53960 ). Please see if the corresponding stackoverflow link ( https://stackoverflow.com/questions/46726216/kubelet-fails-to-get-cgroup-stats-for-docker-and-kubelet-services) fixes things?

@rdxmb the kubelet logs show --cgroup-driver has been deprecated, --kubelet-cgroups has been deprecated, but we also have to add the args to kubelet. My environment:

• CentOS 7.6 with kernal 3.10.0-957.el7.x86_64
• Docker version 1.13.1, build 7f2769b/1.13.1
• Kubernetes v1.16.3

My solution is: echo 'KUBELET_EXTRA_ARGS=--runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice' > /etc/sysconfig/kubelet

Why is this issue closed? This problem still occurs with kubernetes v1.16.2 (in my case with Ubuntu 16.04) and the workarounds with the systemd-includes above must be set by the user manually. Can somebody please reopen this? Is there a way to get these things to the apt-packages?

I’ve got the same errors on

• CentOS Linux release 7.2.1511 (Core)
• docker 1.13.1
• kubernetes v1.8.1

use

--kubelet-cgroups=/system.slice/kubelet.service
--runtime-cgroups=/system.slice/docker.service

in kubelet config still can’t solve.

This remains not fixed and it seems every distribution is patching around it. The fix needs to be added to the kubelet RPM as described at #56850 (comment)

/reopen

I solved this problem like this:

Edit /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf and add two lines below

CPUAccounting=true
MemoryAccounting=true

The whole config file:

# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
CPUAccounting=true
MemoryAccounting=true
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

systemctl daemon-reload
systemctl restart kubelet

Then this message gone.

No it doesn’t fix things, or no you don’t want to check.