kubernetes: Can't init cluster with containerd

What happened: kuebadm can’t create static pod, apiserver,etcd,schduler can’t be created by containerd What you expected to happen: Succeed to start cluster How to reproduce it (as minimally and precisely as possible): Please see my conf and command Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): kubeadm version: &version.Info{Major:“1”, Minor:“14”, GitVersion:“v1.14.9”, GitCommit:“500f5aba80d71253cc01ac6a8622b8377f4a7ef9”, GitTreeState:“clean”, BuildDate:“2019-11-13T11:19:05Z”, GoVersion:“go1.12.12”, Compiler:“gc”, Platform:“linux/amd64”}

• Cloud provider or hardware configuration:

• OS (e.g: cat /etc/os-release): CentOS Linux release 7.7.1908 (Core)

• Kernel (e.g. uname -a): Linux master-1 4.4.218-1.el7.elrepo.x86_64 #1 SMP Thu Apr 2 14:46:23 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux

• Install tools: root@master-1 ~]# containerd -v containerd containerd.io 1.2.13 7ad184331fa3e55e52b890ea95e65ba581ae3429

• Network plugin and version (if this is a network-related bug):

• Others: kubeadm.conf:

apiVersion: kubeadm.k8s.io/v1beta1
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.1.3
  bindPort: 5443
nodeRegistration:
  criSocket: /run/containerd/containerd.sock
  name: master-1
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: "192.168.1.8:8443"
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.14.0
networking:
  dnsDomain: cluster.local
  podSubnet: "10.5.0.0/16"
  serviceSubnet: 10.96.0.0/16
scheduler: {}
---

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
featureGates:
  SupportIPVSProxyMode: true
mode: ipvs

/etc/sysconfig/kubelet

[root@master-1 ~]# cat /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--fail-swap-on=false --container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
KUBELET_CGROUP_ARGS=-cgroup-driver=cgroupfs

cat /etc/containerd/config.toml

[root@master-1 ~]# cat /etc/containerd/config.toml
root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[debug]
  address = ""
  uid = 0
  gid = 0
  level = ""

[metrics]
  address = ""
  grpc_histogram = false

[cgroup]
  path = ""

[plugins]
  [plugins.cgroups]
    no_prometheus = false
  [plugins.cri]
    stream_server_address = "0.0.0.0"
    stream_server_port = "0"
    enable_selinux = false
    sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.1"
    stats_collect_period = 10
    systemd_cgroup = true
    enable_tls_streaming = false
    max_container_log_line_size = 16384
    disable_proc_mount = false
    [plugins.cri.containerd]
      snapshotter = "overlayfs"
      no_pivot = false
      [plugins.cri.containerd.default_runtime]
        runtime_type = "io.containerd.runtime.v1.linux"
        runtime_engine = ""
        runtime_root = ""
      [plugins.cri.containerd.untrusted_workload_runtime]
        runtime_type = ""
        runtime_engine = ""
        runtime_root = ""
    [plugins.cri.cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = ""
    [plugins.cri.registry]
      [plugins.cri.registry.mirrors]
        [plugins.cri.registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]
    [plugins.cri.x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""
  [plugins.diff-service]
    default = ["walking"]
  [plugins.linux]
    shim = "containerd-shim"
    runtime = "runc"
    runtime_root = ""
    no_shim = false
    shim_debug = false
  [plugins.opt]
    path = "/opt/containerd"
  [plugins.restart]
    interval = "10s"
  [plugins.scheduler]
    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = "0s"
    startup_delay = "100ms"

I run the kubeadm init

[root@master-1 ~]# kubeadm init --config=kube-config.yml --experimental-upload-certs
[init] Using Kubernetes version: v1.14.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.3 192.168.1.3]
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [master-1 localhost] and IPs [192.168.1.3 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [master-1 localhost] and IPs [192.168.1.3 127.0.0.1 ::1]
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.

Unfortunately, an error has occurred:
	timed out waiting for the condition

This error is likely caused by:
	- The kubelet is not running
	- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
	- 'systemctl status kubelet'
	- 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
	- 'docker ps -a | grep kube | grep -v pause'
	Once you have found the failing container, you can inspect its logs with:
	- 'docker logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster

containerd service logs

[root@master-1 ~]# journalctl  -xe -f -u containerd 
Apr 06 13:10:39 master-1 containerd[4251]: time="2020-04-06T13:10:39.628978353-04:00" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:etcd-master-1,Uid:323cc0a43b8b3fac4db38c0a60a6a354,Namespace:kube-system,Attempt:0,}"
Apr 06 13:10:39 master-1 containerd[4251]: time="2020-04-06T13:10:39.639642749-04:00" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/47c05971f984bb8f12a01c8d5188da1a139e2e1f50cc11b0f6ef1037359c59ed/shim.sock" debug=false pid=20040
Apr 06 13:10:39 master-1 containerd[4251]: time="2020-04-06T13:10:39.668734527-04:00" level=info msg="shim reaped" id=47c05971f984bb8f12a01c8d5188da1a139e2e1f50cc11b0f6ef1037359c59ed
Apr 06 13:10:39 master-1 containerd[4251]: time="2020-04-06T13:10:39.684922261-04:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-master-1,Uid:323cc0a43b8b3fac4db38c0a60a6a354,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:297: applying cgroup configuration for process caused \"Invalid slice name pod323cc0a43b8b3fac4db38c0a60a6a354\"": unknown"
Apr 06 13:10:41 master-1 containerd[4251]: time="2020-04-06T13:10:41.628136700-04:00" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:kube-apiserver-master-1,Uid:8b825f188edc32679577f2514660502d,Namespace:kube-system,Attempt:0,}"
Apr 06 13:10:41 master-1 containerd[4251]: time="2020-04-06T13:10:41.635274835-04:00" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/1fe45f1512e4afc0ac93845db8eff7e1be91f109962d600b607aa0b8953fccfe/shim.sock" debug=false pid=20059
Apr 06 13:10:41 master-1 containerd[4251]: time="2020-04-06T13:10:41.657913109-04:00" level=info msg="shim reaped" id=1fe45f1512e4afc0ac93845db8eff7e1be91f109962d600b607aa0b8953fccfe
Apr 06 13:10:41 master-1 containerd[4251]: time="2020-04-06T13:10:41.675787226-04:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-apiserver-master-1,Uid:8b825f188edc32679577f2514660502d,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:297: applying cgroup configuration for process caused \"Invalid slice name pod8b825f188edc32679577f2514660502d\"": unknown"
Apr 06 13:10:42 master-1 containerd[4251]: time="2020-04-06T13:10:42.914042893-04:00" level=error msg="Failed to load cni configuration" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
Apr 06 13:10:45 master-1 containerd[4251]: time="2020-04-06T13:10:45.629394976-04:00" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:kube-controller-manager-master-1,Uid:c9d1ac9723ac0e9577fede3c4bb32f2f,Namespace:kube-system,Attempt:0,}"
Apr 06 13:10:45 master-1 containerd[4251]: time="2020-04-06T13:10:45.636042015-04:00" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/6d369ca11731e3d444ef63859efba6f703bce736e1b07ea57812216e4cad7324/shim.sock" debug=false pid=20080
Apr 06 13:10:45 master-1 containerd[4251]: time="2020-04-06T13:10:45.679836432-04:00" level=info msg="shim reaped" id=6d369ca11731e3d444ef63859efba6f703bce736e1b07ea57812216e4cad7324
Apr 06 13:10:45 master-1 containerd[4251]: time="2020-04-06T13:10:45.701037059-04:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-controller-manager-master-1,Uid:c9d1ac9723ac0e9577fede3c4bb32f2f,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:297: applying cgroup configuration for process caused \"Invalid slice name podc9d1ac9723ac0e9577fede3c4bb32f2f\"": unknown"
Apr 06 13:10:47 master-1 containerd[4251]: time="2020-04-06T13:10:47.915245549-04:00" level=error msg="Failed to load cni configuration" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
Apr 06 13:10:49 master-1 containerd[4251]: time="2020-04-06T13:10:49.627563804-04:00" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:kube-scheduler-master-1,Uid:296424c5ca510b83fc0ff6b44e1f1bf1,Namespace:kube-system,Attempt:0,}"
Apr 06 13:10:49 master-1 containerd[4251]: time="2020-04-06T13:10:49.636776597-04:00" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/3b41812d4a3d5fc1b819eb45fcf8c7f291c0dabb59ac0cad84b1b83159d376c6/shim.sock" debug=false pid=20102
Apr 06 13:10:49 master-1 containerd[4251]: time="2020-04-06T13:10:49.680074967-04:00" level=info msg="shim reaped" id=3b41812d4a3d5fc1b819eb45fcf8c7f291c0dabb59ac0cad84b1b83159d376c6
Apr 06 13:10:49 master-1 containerd[4251]: time="2020-04-06T13:10:49.701294016-04:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-scheduler-master-1,Uid:296424c5ca510b83fc0ff6b44e1f1bf1,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:297: applying cgroup configuration for process caused \"Invalid slice name pod296424c5ca510b83fc0ff6b44e1f1bf1\"": unknown"
Apr 06 13:10:52 master-1 containerd[4251]: time="2020-04-06T13:10:52.916934553-04:00" level=error msg="Failed to load cni configuration" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
Apr 06 13:10:54 master-1 containerd[4251]: time="2020-04-06T13:10:54.626989234-04:00" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:kube-apiserver-master-1,Uid:8b825f188edc32679577f2514660502d,Namespace:kube-system,Attempt:0,}"
Apr 06 13:10:54 master-1 containerd[4251]: time="2020-04-06T13:10:54.629963711-04:00" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:etcd-master-1,Uid:323cc0a43b8b3fac4db38c0a60a6a354,Namespace:kube-system,Attempt:0,}"
Apr 06 13:10:54 master-1 containerd[4251]: time="2020-04-06T13:10:54.635081945-04:00" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/0c00035d57a91c9327143162ff3f4919c6d76655ddeba87e615345a970d7dd25/shim.sock" debug=false pid=20125
Apr 06 13:10:54 master-1 containerd[4251]: time="2020-04-06T13:10:54.637772522-04:00" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/1bdd5bfb4a83a563674d9102914a4f70f03535e6326c1e553c79df37909a5d69/shim.sock" debug=false pid=20129
Apr 06 13:10:54 master-1 containerd[4251]: time="2020-04-06T13:10:54.675337874-04:00" level=info msg="shim reaped" id=1bdd5bfb4a83a563674d9102914a4f70f03535e6326c1e553c79df37909a5d69
Apr 06 13:10:54 master-1 containerd[4251]: time="2020-04-06T13:10:54.696624705-04:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:etcd-master-1,Uid:323cc0a43b8b3fac4db38c0a60a6a354,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:297: applying cgroup configuration for process caused \"Invalid slice name pod323cc0a43b8b3fac4db38c0a60a6a354\"": unknown"
Apr 06 13:10:54 master-1 containerd[4251]: time="2020-04-06T13:10:54.708337090-04:00" level=info msg="shim reaped" id=0c00035d57a91c9327143162ff3f4919c6d76655ddeba87e615345a970d7dd25
Apr 06 13:10:54 master-1 containerd[4251]: time="2020-04-06T13:10:54.725221354-04:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-apiserver-master-1,Uid:8b825f188edc32679577f2514660502d,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:297: applying cgroup configuration for process caused \"Invalid slice name pod8b825f188edc32679577f2514660502d\"": unknown"
Apr 06 13:10:56 master-1 containerd[4251]: time="2020-04-06T13:10:56.627161825-04:00" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:kube-controller-manager-master-1,Uid:c9d1ac9723ac0e9577fede3c4bb32f2f,Namespace:kube-system,Attempt:0,}"
Apr 06 13:10:56 master-1 containerd[4251]: time="2020-04-06T13:10:56.637336147-04:00" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/c74d33903cbc5d517a210324e8649dcb256417dc7aea1e266719f481c2a87ced/shim.sock" debug=false pid=20161
Apr 06 13:10:56 master-1 containerd[4251]: time="2020-04-06T13:10:56.675754764-04:00" level=info msg="shim reaped" id=c74d33903cbc5d517a210324e8649dcb256417dc7aea1e266719f481c2a87ced
Apr 06 13:10:56 master-1 containerd[4251]: time="2020-04-06T13:10:56.692766888-04:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-controller-manager-master-1,Uid:c9d1ac9723ac0e9577fede3c4bb32f2f,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:297: applying cgroup configuration for process caused \"Invalid slice name podc9d1ac9723ac0e9577fede3c4bb32f2f\"": unknown"
Apr 06 13:10:57 master-1 containerd[4251]: time="2020-04-06T13:10:57.918673135-04:00" level=error msg="Failed to load cni configuration" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
Apr 06 13:11:00 master-1 containerd[4251]: time="2020-04-06T13:11:00.627147452-04:00" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:kube-scheduler-master-1,Uid:296424c5ca510b83fc0ff6b44e1f1bf1,Namespace:kube-system,Attempt:0,}"
Apr 06 13:11:00 master-1 containerd[4251]: time="2020-04-06T13:11:00.634893216-04:00" level=info msg="shim containerd-shim started" address="/containerd-shim/k8s.io/7a94c3cad911efbcc649557ece74e064d0754995562d5a4edfa3b43eaa4a74e4/shim.sock" debug=false pid=20183
Apr 06 13:11:00 master-1 containerd[4251]: time="2020-04-06T13:11:00.679811801-04:00" level=info msg="shim reaped" id=7a94c3cad911efbcc649557ece74e064d0754995562d5a4edfa3b43eaa4a74e4
Apr 06 13:11:00 master-1 containerd[4251]: time="2020-04-06T13:11:00.695752951-04:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:kube-scheduler-master-1,Uid:296424c5ca510b83fc0ff6b44e1f1bf1,Namespace:kube-system,Attempt:0,} failed, error" error="failed to create containerd task: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:297: applying cgroup configuration for process caused \"Invalid slice name pod296424c5ca510b83fc0ff6b44e1f1bf1\"": unknown"
Apr 06 13:11:02 master-1 containerd[4251]: time="2020-04-06T13:11:02.921111515-04:00" level=error msg="Failed to load cni configuration" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"