kubernetes: K8S 1.21 : Kubelet fails to start when setting memory-manager-policy as "static" or "none" in kubelet arguments

K8S 1.21 : Kubelet fails to start when setting memory-manager-policy as “static” or “none”.

May 17 11:53:07 my-node kubelet: E0517 11:53:07.195378    3261 container_manager_linux.go:358] "Failed to initialize memory manager" err="unknown policy: \"static\""
May 17 11:53:07 my-node kubelet: E0517 11:53:07.195440    3261 server.go:292] "Failed to run kubelet" err="failed to run Kubelet: unknown policy: \"static\""
May 17 11:53:07 my-node systemd: kubelet.service: main process exited, code=exited, status=1/FAILURE
May 17 11:53:07 my-node systemd: Unit kubelet.service entered failed state.
May 17 11:53:07 my-node systemd: kubelet.service failed.

Feature gate has also been enabled for apiserver/kube-controller-manager/scheduler/kube-proxy services

--feature-gates=VolumeSnapshotDataSource=true,EphemeralContainers=true,MemoryManager=true

Kubelet arguments

--reserved-cpus=0 --cpu-manager-policy=static --topology-manager-policy=restricted --topology-manager-scope=pod --memory-manager-policy=static --reserved-memory=0:memory=1Gi

What happened:

K8S 1.21 : Kubelet fails to start when setting memory-manager-policy as “static” or “none”

What you expected to happen:

No errors must be encountered during kubelet start and installation must succeed with setting MemoryManager feature gate.

How to reproduce it (as minimally and precisely as possible):

Try an kubeadm installation by setting MemoryManager feature gate using 1.21 kubeadm installer.

Following kubelet arguments will be set:

--enable-controller-attach-detach=true --cluster-dns=<cluster dns ip> --cluster-domain=cluster.local --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --feature-gates=StartupProbe=true,RotateKubeletServerCertificate=true,EphemeralContainers=true,MemoryManager=true --container-log-max-size=260M --read-only-port=0 --event-qps=0 --tls-cert-file=/var/lib/kubelet/pki/kubelet.crt --tls-private-key-file=/var/lib/kubelet/pki/kubelet.key --reserved-cpus=0 --cpu-manager-policy=static --topology-manager-policy=restricted --topology-manager-scope=pod --memory-manager-policy=static --reserved-memory=0:memory=1Gi

Anything else we need to know?:

Kubelet startup failure messages logs as follows

May 17 11:53:07 my-node kubelet: I0517 11:53:07.194940    3261 topology_manager.go:120] "Creating topology manager with policy per scope" topologyPolicyName="restricted" topologyScopeName="pod"
May 17 11:53:07 my-node kubelet: I0517 11:53:07.194976    3261 container_manager_linux.go:314] "Initializing Topology Manager" policy="restricted" scope="pod"
May 17 11:53:07 my-node kubelet: I0517 11:53:07.195002    3261 container_manager_linux.go:319] "Creating device plugin manager" devicePluginEnabled=true
May 17 11:53:07 my-node kubelet: I0517 11:53:07.195234    3261 cpu_manager.go:158] "Detected CPU topology" topology=&{NumCPUs:8 NumCores:4 NumSockets:1 CPUDetails:map[0:{NUMANodeID:0 SocketID:0 CoreID:0} 1:{NUMANodeID:0 SocketID:0 CoreID:0} 2:{NUMANodeID:0 SocketID:0 CoreID:2} 3:{NUMANodeID:0 SocketID:0 CoreID:2} 4:{NUMANodeID:0 SocketID:0 CoreID:4} 5:{NUMANodeID:0 SocketID:0 CoreID:4} 6:{NUMANodeID:0 SocketID:0 CoreID:6} 7:{NUMANodeID:0 SocketID:0 CoreID:6}]}
May 17 11:53:07 my-nodekubelet: I0517 11:53:07.195294    3261 policy_static.go:109] "Reserved CPUs not available for exclusive assignment" reservedSize=1 reserved="0"
May 17 11:53:07 my-node kubelet: E0517 11:53:07.195378    3261 container_manager_linux.go:358] "Failed to initialize memory manager" err="unknown policy: \"static\""
May 17 11:53:07 my-node kubelet: E0517 11:53:07.195440    3261 server.go:292] "Failed to run kubelet" err="failed to run Kubelet: unknown policy: \"static\""
May 17 11:53:07 my-node systemd: kubelet.service: main process exited, code=exited, status=1/FAILURE
May 17 11:53:07 my-node systemd: Unit kubelet.service entered failed state.
May 17 11:53:07 my-node systemd: kubelet.service failed.

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:25:06Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Kernel (e.g. uname -a):

Linux my-node 3.10.0-1127.el7.x86_64 #1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

• Install tools: kubeadm
• Network plugin and version (if this is a network-related bug):
• Others: