kubeadm: Kubeadm init blocks at "This might take a minute or longer if the control plane images have to be pulled

Versions

kubeadm version (use kubeadm version):

Environment:

• Kubernetes version (use kubectl version):v1.9.2
• Cloud provider or hardware configuration:Virtual Box
• OS (e.g. from /etc/os-release):Ubuntu 16.04.0 LTS (Xeniak Xerus) amd64
• Kernel (e.g. uname -a):linux 4.4.0-62-generic
• Others:kubeadm version :v1.9.2: amd64, kubelet version :v1.9.2 amd64, kubernetes-cni version :0.6.0-00 amd64 ,docker version:17.03.2-ce

What happened?

As I try to run kubeadm init,it hangs with xx@xx:~$ sudo kubeadm init --kubernetes-version=v1.9.2

[init] Using Kubernetes version: v1.9.2 [init] Using Authorization modes: [Node RBAC] [preflight] Running pre-flight checks. [WARNING FileExisting-crictl]: crictl not found in system path [preflight] Starting the kubelet service [certificates] Generated ca certificate and key. [certificates] Generated apiserver certificate and key. [certificates] apiserver serving cert is signed for DNS names [kickseed kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.17.41.15] [certificates] Generated apiserver-kubelet-client certificate and key. [certificates] Generated sa key and public key. [certificates] Generated front-proxy-ca certificate and key. [certificates] Generated front-proxy-client certificate and key. [certificates] Valid certificates and keys now exist in “/etc/kubernetes/pki” [kubeconfig] Wrote KubeConfig file to disk: “admin.conf” [kubeconfig] Wrote KubeConfig file to disk: “kubelet.conf” [kubeconfig] Wrote KubeConfig file to disk: “controller-manager.conf” [kubeconfig] Wrote KubeConfig file to disk: “scheduler.conf” [controlplane] Wrote Static Pod manifest for component kube-apiserver to “/etc/kubernetes/manifests/kube-apiserver.yaml” [controlplane] Wrote Static Pod manifest for component kube-controller-manager to “/etc/kubernetes/manifests/kube-controller-manager.yaml” [controlplane] Wrote Static Pod manifest for component kube-scheduler to “/etc/kubernetes/manifests/kube-scheduler.yaml” [etcd] Wrote Static Pod manifest for a local etcd instance to “/etc/kubernetes/manifests/etcd.yaml” [init] Waiting for the kubelet to boot up the control plane as Static Pods from directory “/etc/kubernetes/manifests”. [init] This might take a minute or longer if the control plane images have to be pulled.

Then I check the kubelet log: xx@xx:~$ sudo journalctl -xeu kubelet: Jan 31 14:45:03 kickseed kubelet[28516]: E0131 14:45:03.280984 28516 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image “gcr.io/google_containers/pause-amd64:3.0”: Error response from daemon: Get https://gcr.io/v1/_ping: dial tcp 172.217.6.127:443: i/o timeout Jan 31 14:45:03 kickseed kubelet[28516]: E0131 14:45:03.281317 28516 kuberuntime_sandbox.go:54] CreatePodSandbox for pod “kube-scheduler-kickseed_kube-system(69c12074e336b0dbbd0a1666ce05226a)” failed: rpc error: code = Unknown desc = failed pulling image “gcr.io/google_containers/pause-amd64:3.0”: Error response from daemon: Get https://gcr.io/v1/_ping: dial tcp 172.217.6.127:443: i/o timeout Jan 31 14:45:03 kickseed kubelet[28516]: E0131 14:45:03.281580 28516 kuberuntime_manager.go:647] createPodSandbox for pod “kube-scheduler-kickseed_kube-system(69c12074e336b0dbbd0a1666ce05226a)” failed: rpc error: code = Unknown desc = failed pulling image “gcr.io/google_containers/pause-amd64:3.0”: Error response from daemon: Get https://gcr.io/v1/_ping: dial tcp 172.217.6.127:443: i/o timeout Jan 31 14:45:03 kickseed kubelet[28516]: E0131 14:45:03.281875 28516 pod_workers.go:186] Error syncing pod 69c12074e336b0dbbd0a1666ce05226a (“kube-scheduler-kickseed_kube-system(69c12074e336b0dbbd0a1666ce05226a)”), skipping: failed to “CreatePodSandbox” for “kube-scheduler-kickseed_kube-system(69c12074e336b0dbbd0a1666ce05226a)” with CreatePodSandboxError: “CreatePodSandbox for pod "kube-scheduler-kickseed_kube-system(69c12074e336b0dbbd0a1666ce05226a)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v1/_ping: dial tcp 172.217.6.127:443: i/o timeout” Jan 31 14:45:03 kickseed kubelet[28516]: E0131 14:45:03.380290 28516 event.go:209] Unable to write event: ‘Patch https://172.17.41.15:6443/api/v1/namespaces/default/events/kickseed.150ecf46afb098b7: dial tcp 172.17.41.15:6443: getsockopt: connection refused’ (may retry after sleeping) Jan 31 14:45:03 kickseed kubelet[28516]: E0131 14:45:03.933783 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.17.41.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dkickseed&limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused Jan 31 14:45:03 kickseed kubelet[28516]: E0131 14:45:03.934707 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://172.17.41.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dkickseed&limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused Jan 31 14:45:03 kickseed kubelet[28516]: E0131 14:45:03.935921 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://172.17.41.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused Jan 31 14:45:04 kickseed kubelet[28516]: E0131 14:45:04.281024 28516 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed pulling image “gcr.io/google_containers/pause-amd64:3.0”: Error response from daemon: Get https://gcr.io/v1/_ping: dial tcp 172.217.6.127:443: i/o timeout Jan 31 14:45:04 kickseed kubelet[28516]: E0131 14:45:04.281352 28516 kuberuntime_sandbox.go:54] CreatePodSandbox for pod “kube-controller-manager-kickseed_kube-system(6546d6faf0b50c9fc6712ce25ee9b6cb)” failed: rpc error: code = Unknown desc = failed pulling image “gcr.io/google_containers/pause-amd64:3.0”: Error response from daemon: Get https://gcr.io/v1/_ping: dial tcp 172.217.6.127:443: i/o timeout Jan 31 14:45:04 kickseed kubelet[28516]: E0131 14:45:04.281634 28516 kuberuntime_manager.go:647] createPodSandbox for pod “kube-controller-manager-kickseed_kube-system(6546d6faf0b50c9fc6712ce25ee9b6cb)” failed: rpc error: code = Unknown desc = failed pulling image “gcr.io/google_containers/pause-amd64:3.0”: Error response from daemon: Get https://gcr.io/v1/_ping: dial tcp 172.217.6.127:443: i/o timeout Jan 31 14:45:04 kickseed kubelet[28516]: E0131 14:45:04.281938 28516 pod_workers.go:186] Error syncing pod 6546d6faf0b50c9fc6712ce25ee9b6cb (“kube-controller-manager-kickseed_kube-system(6546d6faf0b50c9fc6712ce25ee9b6cb)”), skipping: failed to “CreatePodSandbox” for “kube-controller-manager-kickseed_kube-system(6546d6faf0b50c9fc6712ce25ee9b6cb)” with CreatePodSandboxError: “CreatePodSandbox for pod "kube-controller-manager-kickseed_kube-system(6546d6faf0b50c9fc6712ce25ee9b6cb)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v1/_ping: dial tcp 172.217.6.127:443: i/o timeout” Jan 31 14:45:04 kickseed kubelet[28516]: E0131 14:45:04.934694 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.17.41.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dkickseed&limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused Jan 31 14:45:04 kickseed kubelet[28516]: E0131 14:45:04.935613 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://172.17.41.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dkickseed&limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused Jan 31 14:45:04 kickseed kubelet[28516]: E0131 14:45:04.936669 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://172.17.41.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused Jan 31 14:45:05 kickseed kubelet[28516]: W0131 14:45:05.073692 28516 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d Jan 31 14:45:05 kickseed kubelet[28516]: E0131 14:45:05.074106 28516 kubelet.go:2105] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized Jan 31 14:45:05 kickseed kubelet[28516]: E0131 14:45:05.935680 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.17.41.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dkickseed&limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused Jan 31 14:45:05 kickseed kubelet[28516]: E0131 14:45:05.937423 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://172.17.41.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dkickseed&limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused Jan 31 14:45:05 kickseed kubelet[28516]: E0131 14:45:05.937963 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://172.17.41.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused Jan 31 14:45:05 kickseed kubelet[28516]: I0131 14:45:05.974034 28516 kubelet_node_status.go:273] Setting node annotation to enable volume controller attach/detach Jan 31 14:45:06 kickseed kubelet[28516]: I0131 14:45:06.802447 28516 kubelet_node_status.go:273] Setting node annotation to enable volume controller attach/detach Jan 31 14:45:06 kickseed kubelet[28516]: I0131 14:45:06.804242 28516 kubelet_node_status.go:82] Attempting to register node kickseed Jan 31 14:45:06 kickseed kubelet[28516]: E0131 14:45:06.804778 28516 kubelet_node_status.go:106] Unable to register node “kickseed” with API server: Post https://172.17.41.15:6443/api/v1/nodes: dial tcp 172.17.41.15:6443: getsockopt: con

xx@xx:~$ sudo systemctl status kubelet:

kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/kubelet.service.d └─11-kubeadm.conf, 10-kubeadm1.conf, 90-local-extras.conf Active: active (running) since Wed 2018-01-31 13:53:46 CST; 49min ago Docs: http://kubernetes.io/docs/ Main PID: 28516 (kubelet) Tasks: 13 Memory: 37.8M CPU: 22.767s CGroup: /system.slice/kubelet.service └─28516 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --cgroup-driver=cgroupfs --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt --cadvisor-port=0 --rotate-certificates=true --cert-dir=/var/lib/kubelet/pki --fail-swap-on=false

Jan 31 14:43:17 kickseed kubelet[28516]: E0131 14:43:17.862590 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://172.17.41.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dkickseed&limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused Jan 31 14:43:17 kickseed kubelet[28516]: E0131 14:43:17.863474 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://172.17.41.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused Jan 31 14:43:18 kickseed kubelet[28516]: E0131 14:43:18.621818 28516 event.go:209] Unable to write event: ‘Patch https://172.17.41.15:6443/api/v1/namespaces/default/events/kickseed.150ecf46afb098b7: dial tcp 172.17.41.15:6443: getsockopt: connection refused’ (may retry after sleeping) Jan 31 14:43:18 kickseed kubelet[28516]: E0131 14:43:18.862440 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.17.41.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dkickseed&limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused Jan 31 14:43:18 kickseed kubelet[28516]: E0131 14:43:18.863379 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://172.17.41.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dkickseed&limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused Jan 31 14:43:18 kickseed kubelet[28516]: E0131 14:43:18.864424 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://172.17.41.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused Jan 31 14:43:19 kickseed kubelet[28516]: E0131 14:43:19.255460 28516 eviction_manager.go:238] eviction manager: unexpected err: failed to get node info: node “kickseed” not found Jan 31 14:43:19 kickseed kubelet[28516]: E0131 14:43:19.863266 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://172.17.41.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dkickseed&limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused Jan 31 14:43:19 kickseed kubelet[28516]: E0131 14:43:19.864238 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://172.17.41.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Dkickseed&limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused Jan 31 14:43:19 kickseed kubelet[28516]: E0131 14:43:19.865262 28516 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://172.17.41.15:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 172.17.41.15:6443: getsockopt: connection refused

Some docker images are listed as followed: gcr.io/google_containers/kube-apiserver-amd64:v1.9.2 gcr.io/google_containers/kube-controller-manager-amd64:v1.9.2 gcr.io/google_containers/kube-scheduler-amd64:v1.9.2 gcr.io/google_containers/kube-proxy-amd64:v1.9.2 gcr.io/google_containers/etcd-amd64:3.2.14 gcr.io/google_containers/pause-amd64:3.1 gcr.io/google_containers/kube-dnsmasq-amd64:1.4.1 gcr.io/google_containers/kubernetes-dashboard-amd64:v1.8.2 gcr.io/google_containers/kubedns-amd64:1.9 gcr.io/google_containers/kube-discovery-amd64:1.0 gcr.io/google_containers/exechealthz-amd64:v1.2.0 gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.8 gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.8 gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.8 gcr.io/google_containers/dnsmasq-metrics-amd64:1.0.1

What you expected to happen?

kubeadm init should complete

How to reproduce it (as minimally and precisely as possible)?

virtualbox with Ubuntu 16.04 and kubeadm 1.9.2

Anything else we need to know?