kubernetes: cpuset.cpu_exclusive flag not set for Guaranteed Qos Pod Container

What happened:

I’ve deployed K8s cluster with kubelet’s CPU Manager set to static policy. While creating Guaranteed QoS POD with dedicated CPUs for the container, it assigns the dedicated CPUs, but without exclusive flag set in the container’s cgroup file system (/sys/fs/cgroup/cpuset/cpuset.cpu_exclusive)

worker

# kubectl get pods
NAME       READY   STATUS    RESTARTS   AGE
pod-test   1/1     Running   0          7m16s
# cat /var/lib/kubelet/cpu_manager_state
{"policyName":"static","defaultCpuSet":"0-1,6-17","entries":{"cdc2c5cc531dd48bceb064e5411259a46bd8b8e70974fce10a3620b5b5da2c05":"2-5"},"checksum":148873334}

pod container

# cat /proc/1/task/1/status | grep Cpus_allowed
Cpus_allowed:   0003c
Cpus_allowed_list:      2-5
# cat /sys/fs/cgroup/cpuset/cpuset.cpus
2-5
# cat /sys/fs/cgroup/cpuset/cpuset.cpu_exclusive
0

What you expected to happen:

The CPU cores 2-5 should get exclusively assigned to the pod container and cpuset.cpu_exclusive file is supposed to set with 1.

How to reproduce it (as minimally and precisely as possible):

Create Guaranteed QoS Pod.

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:20:10Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:12:17Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
  • OS (e.g: cat /etc/os-release):
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
  • Kernel (e.g. uname -a): Linux my-centos 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 29 (9 by maintainers)

Most upvoted comments

It appears that Exists() function in the cgroup manager logs the missing path on verbosity level 4 before it returns false, see https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cm/cgroup_manager_linux.go#L276.

Could you check what the log message says? Possibly you have some of the paths missing.