kubernetes: cgroup driver "systemd" does not work with custom CPU CFS quota period

What happened: I installed a cluster with kubeadm (1.14.1) and cgroup driver systemd with a custom cpu cfs quota period. Pods are unable to start with the following message:

Error: failed to start container "kube-flannel": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:424: container init caused \"process_linux.go:390: setting cgroup config for procHooks process caused \\\"failed to write 5000 to cpu.cfs_period_us: write /sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-pode7dac811_5d23_11e9_8ff8_005056935735.slice/docker-kube-flannel.scope/cpu.cfs_period_us: invalid argument\\\"\"": unknown

If I understand the cgroup-drivers documentation correct, it’s recommended to use the systemd driver on a systemd managed OS, but this currently blocks us to respect this recommendation.

Please note: cgroupfs driver is not affected of this particular problem. If we switch to use cgroupfs as driver everything works as expected.

What you expected to happen: Pods to start with a cfs quota period of 5ms set.

How to reproduce it (as minimally and precisely as possible):

1. Setup docker on a systemd managed system with the following config in /etc/docker/daemon.json

{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "bip": "10.127.1.1/24",
  "log-opts": {
    "max-size": "2g"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}

(relevant part is the exec-opts setting)

1. Setup a k8s cluster with kubeadm with at least the following special config

---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
  CustomCPUCFSQuotaPeriod: true
cpuCFSQuotaPeriod: 5ms

1. Try to run a pod.

Anything else we need to know?: docker info output:

Containers: 19
 Running: 16
 Paused: 0
 Stopped: 3
Images: 9
Server Version: 18.09.5
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-957.1.3.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.6 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.66GiB
Name: e1-k8s-mlsu105
ID: DUMM:WJD4:PCRQ:Z7QH:ZOO5:URCB:BWO4:K2BH:DB7D:7SFK:6CVU:5HYQ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

/var/lib/kubelet/kubeadm-flags.envcontent:

KUBELET_KUBEADM_ARGS=--cgroup-driver=systemd --network-plugin=cni 

related issue: https://github.com/kubernetes/kubernetes/issues/72878

Environment:

• Kubernetes version (use kubectl version): 1.14.1
• Cloud provider or hardware configuration: on-premises
• OS (e.g: cat /etc/os-release):

NAME="Red Hat Enterprise Linux Server"
VERSION="7.6 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.6"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.6 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.6:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.6"

• Kernel (e.g. uname -a):

Linux e1-k8s-mlsu105 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 15 17:36:42 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

• Install tools: kubeadm 1.14.1
• Others: -