kubernetes: Kubelet fails when run inside container
There’s a problem with running kubelet in containers (k8s version v1.6.0-rc.1). I’m trying to start a cluster using kubeadm-dind-cluster and kubelet fails to start any containers. kubeadm-dind-cluster
creates a cluster where the nodes are represented by containers. The problem occurs when using Mac docker of the following version:
Version 17.03.0-ce-mac2 (15657)
Channel: beta
32de842512
Kernel version: 4.9.12-moby
There’s the following diagnostics in kubelet log:
Mar 27 08:13:20 kube-master hyperkube[571]: I0327 08:13:20.817136 571 kubelet.go:1158] Container garbage collection succeeded
Mar 27 08:13:20 kube-master hyperkube[571]: E0327 08:13:20.823647 571 kubelet.go:1246] Failed to start ContainerManager failed to initialise top level QOS containers: root container /kubepods doesn't exist
kubeadm-dind-cluster
works on Ubuntu 16.04.2 (kernel 4.4.0-57-generic)
On Fedora 25 with kernel 4.8.6-300.fc25.x86_64 there are some problems with DIND script + k8s 1.6, but this “/kubepods” problem doesn’t occur either.
@errordeveloper has similar problems with kxd which doesn’t use DIND, he will provide more details in the comments shortly.
I suspected that #41234 was the culprit but turns out that kubelet starts just fine when I check out 9b4a8f746449aabc1b9083d7987fec55a89464f2
Currently bisecting.
In order to reproduce it on Mac:
wget https://cdn.rawgit.com/Mirantis/kubeadm-dind-cluster/master/fixed/dind-cluster-v1.6.sh
chmod +x dind-cluster-v1.6.sh
./dind-cluster-v1.6.sh up
After kubeadm init
hangs, enter kube-master
container and inspect kubelet log:
journalctl -xe -u kubelet.service
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 17 (15 by maintainers)
Commits related to this issue
- Fix k8s 1.6 on Mac Docker Added workaround for kubernetes/kubernetes#43704 — committed to kubernetes-retired/kubeadm-dind-cluster by deleted user 7 years ago
- Fix k8s 1.6 on Mac Docker Added workaround for kubernetes/kubernetes#43704 — committed to kubernetes-retired/kubeadm-dind-cluster by deleted user 7 years ago
- Fix kubelet before 1.7 not starting on newer kernels * cf. https://github.com/kubernetes/kubernetes/issues/43704 — committed to simonswine/puppet-module-kubernetes by simonswine 7 years ago
- make /sys kubelet mount rw https://github.com/kubernetes/kubernetes/issues/43704#issuecomment-289479967 — committed to utilitywarehouse/tf_kube_ignition by deleted user 5 years ago
- bootkube/resources/charts/kubelet: make /sys RW To avoid issues mentioned in https://github.com/kubernetes/kubernetes/issues/43704. Signed-off-by: Mateusz Gozdek <mateusz@kinvolk.io> — committed to invidian/lokomotive-kubernetes by invidian 4 years ago
- bootkube/resources/charts/kubelet: make /sys RW To avoid issues mentioned in https://github.com/kubernetes/kubernetes/issues/43704. Signed-off-by: Mateusz Gozdek <mateusz@kinvolk.io> — committed to kinvolk-archives/lokomotive-kubernetes by invidian 4 years ago
- bootkube/resources/charts/kubelet: make /sys RW As stated as requirement in https://github.com/kubernetes/kubernetes/issues/43704#issuecomment-289479967. Also, bootstrap-kubelet also use writeable /... — committed to invidian/lokomotive-kubernetes by invidian 4 years ago
- Add more kubelet flags from https://github.com/kubernetes/kubernetes/issues/43704 — committed to ChameleonCloud/chi-edge-raspberrypi by deleted user 3 years ago
Update: #45515 solved this for me!
@CallMeFoxie ok, I just never heard of anyone with such configuration before, but I’m sure you have all the good reasons for doing that. So in release#306 it’s been mentioned that
--cgroup-driver=systemd
is another working solution for RHEL, which is probably better and it’s just that the auto detection code is somehow not doing the right thing.