kubernetes: kubeadm blocks waiting for 'control plane'
Hi @kubernetes/sig-cluster-lifecycle
I tried to follow the docs for kubeadm on centOS 7.1.
It seems that the kubeadm init blocks waiting for ‘control plane to become ready’ even though all containers are running.
# kubeadm init --token foobar.1234
<util/tokens> validating provided token
<master/tokens> accepted provided token
<master/pki> created keys and certificates in "/etc/kubernetes/pki"
<util/kubeconfig> created "/etc/kubernetes/kubelet.conf"
<util/kubeconfig> created "/etc/kubernetes/admin.conf"
<master/apiclient> created API client configuration
<master/apiclient> created API client, waiting for the control plane to become ready
here are the running containers on the same master machine:
$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
30aff4f98753 gcr.io/google_containers/kube-apiserver-amd64:v1.4.0 "/usr/local/bin/kube-" 3 minutes ago Up 3 minutes k8s_kube-apiserver.c44dda3f_kube-apiserver-k8ss-head_kube-system_6b83c87a9bf5c380c6f948f428b23dd1_408af885
8fd1842776ab gcr.io/google_containers/kube-controller-manager-amd64:v1.4.0 "/usr/local/bin/kube-" 3 minutes ago Up 3 minutes k8s_kube-controller-manager.a2978680_kube-controller-manager-k8ss-head_kube-system_5f805ed49f6fd9f0640be470e3dea2a2_7ac41d83
32b7bfb55dc0 gcr.io/google_containers/kube-scheduler-amd64:v1.4.0 "/usr/local/bin/kube-" 3 minutes ago Up 3 minutes k8s_kube-scheduler.1b5cde04_kube-scheduler-k8ss-head_kube-system_586d16be4ecaac95b0162c5d11921019_0ca14012
8a1797fdb1df gcr.io/google_containers/etcd-amd64:2.2.5 "etcd --listen-client" 8 minutes ago Up 8 minutes k8s_etcd.4ffa9846_etcd-k8ss-head_kube-system_42857e4bd57d261fc438bcb2a87572b9_f1b219d3
292bcafb3316 gcr.io/google_containers/pause-amd64:3.0 "/pause" 8 minutes ago Up 8 minutes k8s_POD.d8dbe16c_kube-controller-manager-k8ss-head_kube-system_5f805ed49f6fd9f0640be470e3dea2a2_fe9592ab
ab929dd920a2 gcr.io/google_containers/pause-amd64:3.0 "/pause" 8 minutes ago Up 8 minutes k8s_POD.d8dbe16c_kube-apiserver-k8ss-head_kube-system_6b83c87a9bf5c380c6f948f428b23dd1_c93e3a3b
71c28763aeab gcr.io/google_containers/pause-amd64:3.0 "/pause" 8 minutes ago Up 8 minutes k8s_POD.d8dbe16c_kube-scheduler-k8ss-head_kube-system_586d16be4ecaac95b0162c5d11921019_eb12a865
615cb42e0108 gcr.io/google_containers/pause-amd64:3.0 "/pause" 8 minutes ago Up 8 minutes k8s_POD.d8dbe16c_etcd-k8ss-head_kube-system_42857e4bd57d261fc438bcb2a87572b9_891fc5db
I tried to join a node but I get a connection refused error, even though there is no firewall…
# kubeadm join --token foobar.1234 <master_ip>
<util/tokens> validating provided token
<node/discovery> created cluster info discovery client, requesting info from "http://185.19.30.178:9898/cluster-info/v1/?token-id=foobar"
error: <node/discovery> failed to request cluster info [Get http://MASTER_IP:9898/cluster-info/v1/?token-id=foobar: dial tcp MASTER_IP:9898: getsockopt: connection refused]
and now I am actually wondering if the init is blocking waiting for nodes to join. According to the docs it is not blocking, but the logs of kubeadm seems to indicate that it is.
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Reactions: 8
- Comments: 66 (32 by maintainers)
Commits related to this issue
- document problem with SELINUX see https://github.com/kubernetes/kubernetes/issues/33544 — committed to oz123/kubernetes.github.io by oz123 8 years ago
There is a uninstall script referenced at http://deploy-preview-1321.kubernetes-io-vnext-staging.netlify.com/docs/getting-started-guides/kubeadm/. After running it, my init ran correctly again.
won’t work anymore on ubuntu. After using the updated manual on kubernetes.io the System hangs on the same point:
But the apiserver log changed:
@harsha544
Be careful the link and script which @kenzhaoyihui provided, that script tries to fake the google’s images with his own images.
You’d better to not run with it.
In fact the solution has been provided in this ticket, I fixed my issue already. It was provided by @benmathews commented on Sep 28, 2016. If yo missed that comment, you should take a try.
@harsha544 https://github.com/kenzhaoyihui/kubeadm-images-gcr.io/blob/master/pull_kubernetes_images.sh
The shell script is to pull the all docker images that needed, could you pull all the images and then execute “kubeadm init”.
I also hit this issues. have disabled SELINUX. <python> [certificates] Valid certificates and keys now exist in “/etc/kubernetes/pki” [kubeconfig] Wrote KubeConfig file to disk: “/etc/kubernetes/admin.conf” [kubeconfig] Wrote KubeConfig file to disk: “/etc/kubernetes/kubelet.conf” [kubeconfig] Wrote KubeConfig file to disk: “/etc/kubernetes/controller-manager.conf” [kubeconfig] Wrote KubeConfig file to disk: “/etc/kubernetes/scheduler.conf” [apiclient] Created API client, waiting for the control plane to become ready </python>
Hi Folk, problem solved for me just by: Stopping Apparmor :
# /etc/init.d/apparmor stopafter that, you shoud reset kubeadm# kubeadm resetand finally, rerun the Initialization of your master# kubeadm init@kamigerami you can generate token ownself and provide via attribute
kubeadm init --token="${k8s_token}"and use the same forkubeadm join --token=${k8s_token}ok so the discover port is using a hostPort on 9898.
logs on that pod return this:
I am following the docs
The DNS pod is not starting:
Some problems on centso7. Also just block on this steps: [apiclient] Created API client, waiting for the control plane to become ready.
1 Dec 29 07:10:29 master kubelet: E1229 07:10:29.744234 8891 pod_workers.go:184] Error syncing pod b4b25cab578f82fd99198c566860faf7, skipping: failed to "StartContainer " for “POD” with ImagePullBackOff: “Back-off pulling image "gcr.io/google_containers/pause-amd64:3.0"” 2 Dec 29 07:10:30 master kubelet: E1229 07:10:30.680786 8891 reflector.go:188] pkg/kubelet/config/apiserver.go:44: Failed to list *api.Pod: Get https://192.168.121.241: 6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster&resourceVersion=0: dial tcp 192.168.121.241:6443: getsockopt: connection refused 3 Dec 29 07:10:30 master kubelet: E1229 07:10:30.680797 8891 reflector.go:188] pkg/kubelet/kubelet.go:386: Failed to list *api.Node: Get https://192.168.121.241:6443/ap i/v1/nodes?fieldSelector=metadata.name%3Dmaster&resourceVersion=0: dial tcp 192.168.121.241:6443: getsockopt: connection refused 4 Dec 29 07:10:30 master kubelet: E1229 07:10:30.680835 8891 reflector.go:188] pkg/kubelet/kubelet.go:378: Failed to list *api.Service: Get https://192.168.121.241:6443 /api/v1/services?resourceVersion=0: dial tcp 192.168.121.241:6443: getsockopt: connection refused 5 Dec 29 07:10:31 master kubelet: I1229 07:10:31.144650 8891 kubelet_node_status.go:204] Setting node annotation to enable volume controller attach/detach 6 Dec 29 07:10:31 master kubelet: I1229 07:10:31.186977 8891 kubelet_node_status.go:74] Attempting to register node master 7 Dec 29 07:10:31 master kubelet: E1229 07:10:31.187254 8891 kubelet_node_status.go:98] Unable to register node “master” with API server: Post https://192.168.121.241:6 443/api/v1/nodes: dial tcp 192.168.121.241:6443: getsockopt: connection refused 8 Dec 29 07:10:31 master kubelet: I1229 07:10:31.397597 8891 kubelet_node_status.go:204] Setting node annotation to enable volume controller attach/detach 9 Dec 29 07:10:31 master kubelet: E1229 07:10:31.437996 8891 kubelet.go:1508] Failed creating a mirror pod for “kube-apiserver-master_kube-system(73c001656da6c2ae76abb7 d4879d2e36)”: Post https://192.168.121.241:6443/api/v1/namespaces/kube-system/pods: dial tcp 192.168.121.241:6443: getsockopt: connection refused 10 Dec 29 07:10:31 master kubelet: E1229 07:10:31.681357 8891 reflector.go:188] pkg/kubelet/config/apiserver.go:44: Failed to list *api.Pod: Get https://192.168.121.241: 6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster&resourceVersion=0: dial tcp 192.168.121.241:6443: getsockopt: connection refused 11 Dec 29 07:10:31 master kubelet: E1229 07:10:31.681376 8891 reflector.go:188] pkg/kubelet/kubelet.go:386: Failed to list *api.Node: Get https://192.168.121.241:6443/ap i/v1/nodes?fieldSelector=metadata.name%3Dmaster&resourceVersion=0: dial tcp 192.168.121.241:6443: getsockopt: connection refused 12 Dec 29 07:10:31 master kubelet: E1229 07:10:31.681424 8891 reflector.go:188] pkg/kubelet/kubelet.go:378: Failed to list *api.Service: Get https://192.168.121.241:6443 /api/v1/services?resourceVersion=0: dial tcp 192.168.121.241:6443: getsockopt: connection refused 13 Dec 29 07:10:31 master docker-current: time=“2016-12-29T07:10:31.738920860Z” level=error msg=“Handler for GET /v1.22/images/gcr.io/google_containers/pause-amd64:3.0/json returned error: No such image: gcr.io/google_containers/pause-amd64:3.0” 14 Dec 29 07:10:31 master kubelet: E1229 07:10:31.739387 8891 docker_manager.go:2188] Failed to create pod infra container: ImagePullBackOff; Skipping pod “kube-apiserve r-master_kube-system(73c001656da6c2ae76abb7d4879d2e36)”: Back-off pulling image “gcr.io/google_containers/pause-amd64:3.0” 15 Dec 29 07:10:31 master kubelet: E1229 07:10:31.739419 8891 pod_workers.go:184] Error syncing pod 73c001656da6c2ae76abb7d4879d2e36, skipping: failed to "StartContainer " for “POD” with ImagePullBackOff: “Back-off pulling image "gcr.io/google_containers/pause-amd64:3.0"” 16 Dec 29 07:10:32 master kubelet: E1229 07:10:32.301850 8891 cni.go:163] error updating cni config: No networks found in /etc/cni/net.d 17 Dec 29 07:10:32 master kubelet: E1229 07:10:32.681920 8891 reflector.go:188] pkg/kubelet/kubelet.go:378: Failed to list *api.Service: Get https://192.168.121.241:6443 /api/v1/services?resourceVersion=0: dial tcp 192.168.121.241:6443: getsockopt: connection refused 18 Dec 29 07:10:32 master kubelet: E1229 07:10:32.681920 8891 reflector.go:188] pkg/kubelet/config/apiserver.go:44: Failed to list *api.Pod: Get https://192.168.121.241: 6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster&resourceVersion=0: dial tcp 192.168.121.241:6443: getsockopt: connection refused 19 Dec 29 07:10:32 master kubelet: E1229 07:10:32.681957 8891 reflector.go:188] pkg/kubelet/kubelet.go:386: Failed to list *api.Node: Get https://192.168.121.241:6443/ap i/v1/nodes?fieldSelector=metadata.name%3Dmaster&resourceVersion=0: dial tcp 192.168.121.241:6443: getsockopt: connection refused 20 Dec 29 07:10:32 master kubelet: E1229 07:10:32.954775 8891 eviction_manager.go:202] eviction manager: unexpected err: failed GetNode: node ‘master’ not found
@miry Thanks, problem verified for weak access to
gcr.io/google_containers/kube*. It’s solved after I load necessary images downloaded independentely on other places.@miry : I tried your approach, seems to be working the same way in master node but this is the error message on the minion node (to be joined).
kubeadm join --token=p12345.12345p12345p12345 10.144.2.200 <util/tokens> validating provided token <node/discovery> created cluster info discovery client, requesting info from “http://10.144.2.200:9898/cluster-info/v1/?token-id=p12345” error: <node/discovery> failed to parse response as JWS object [square/go-jose: compact JWS format must have three parts]
Any clue how to solve it? Thanks!
my os is centos7,the command-kubeadm can’t start the container of kube-discovery!!!and bolck at the commad:“<master/apiclient> created API client, waiting for the control plane to become ready”. I had setenforce 0,who can help me?oh my god
Actually I just made it work. There is a small difference between CentOS and Ubuntu. In CentOS you have to manually start the kubelet:
systemctl enable kubelet && systemctl start kubelet. After that everything worked.I am still having this issue even after disabling SELinux. I am trying this on EC2 on CentOS 7.2 with ami:
ami-6d1c2007. Ubuntu 16.04 worked perfectly.Are the new pkgs available on:
http://yum.kubernetes.io/repos/kubernetes-el7-x86_64or should I be using a different repo?It does hang in the same step
waiting for 'control plane'but I don’t even get any docker containers running (fromdocker psi get 0 containers). Are there any logs that I should look, a quick search didn’t showed anything.Any help is appreciated 😃
Same issue for me on an aws installation,
but I can’t see any docker containers running. Some Informations are here: Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-38-generic x86_64) Using http_proxy and https_proxyLooked at the logs of the apiServer. It returns with an exception: