kubernetes: Kubelet fails when run inside container

There’s a problem with running kubelet in containers (k8s version v1.6.0-rc.1). I’m trying to start a cluster using kubeadm-dind-cluster and kubelet fails to start any containers. kubeadm-dind-cluster creates a cluster where the nodes are represented by containers. The problem occurs when using Mac docker of the following version:

Version 17.03.0-ce-mac2 (15657)
Channel: beta
32de842512

Kernel version: 4.9.12-moby

There’s the following diagnostics in kubelet log:

Mar 27 08:13:20 kube-master hyperkube[571]: I0327 08:13:20.817136     571 kubelet.go:1158] Container garbage collection succeeded
Mar 27 08:13:20 kube-master hyperkube[571]: E0327 08:13:20.823647     571 kubelet.go:1246] Failed to start ContainerManager failed to initialise top level QOS containers: root container /kubepods doesn't exist

kubeadm-dind-cluster works on Ubuntu 16.04.2 (kernel 4.4.0-57-generic) On Fedora 25 with kernel 4.8.6-300.fc25.x86_64 there are some problems with DIND script + k8s 1.6, but this “/kubepods” problem doesn’t occur either.

@errordeveloper has similar problems with kxd which doesn’t use DIND, he will provide more details in the comments shortly.

I suspected that #41234 was the culprit but turns out that kubelet starts just fine when I check out 9b4a8f746449aabc1b9083d7987fec55a89464f2

Currently bisecting.

In order to reproduce it on Mac:

wget https://cdn.rawgit.com/Mirantis/kubeadm-dind-cluster/master/fixed/dind-cluster-v1.6.sh
chmod +x dind-cluster-v1.6.sh
./dind-cluster-v1.6.sh up

After kubeadm init hangs, enter kube-master container and inspect kubelet log:

journalctl -xe -u kubelet.service

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 17 (15 by maintainers)

Commits related to this issue

Most upvoted comments

Update: #45515 solved this for me!

I am on Centos 7 with a kernel 4.11 running Kubernetes master branch on baremetal

My setup was working on kernel version 3.10 and I upgraded the Kernel and encountered the error - "Failed to start ContainerManager failed to initialise top level QOS containers: root container /kubepods doesn't exist"
  
Is there any other fix for this other than enabling the -cgroups-per-qos=false --enforce-node-allocatable= flags in kubelet? I am experimenting and need the cgroups-per-qos flag enabled.



@CallMeFoxie ok, I just never heard of anyone with such configuration before, but I’m sure you have all the good reasons for doing that. So in release#306 it’s been mentioned that --cgroup-driver=systemd is another working solution for RHEL, which is probably better and it’s just that the auto detection code is somehow not doing the right thing.