minikube: hyperkit: no internet connectivity, can't pull kube-addon-manager

The exact command to reproduce the issue:

minikube start

The full output of the command that failed:

😄  minikube v1.5.2 on Darwin 10.14.3
✨  Automatically selected the 'hyperkit' driver (alternates: [virtualbox])
💾  Downloading driver docker-machine-driver-hyperkit:
    > docker-machine-driver-hyperkit.sha256: 65 B / 65 B [---] 100.00% ? p/s 0s
    > docker-machine-driver-hyperkit: 10.79 MiB / 10.79 MiB  100.00% 636.16 KiB
🔑  The 'hyperkit' driver requires elevated permissions. The following commands will be executed:

    $ sudo chown root:wheel /Users/dcamata/.minikube/bin/docker-machine-driver-hyperkit
    $ sudo chmod u+s /Users/dcamata/.minikube/bin/docker-machine-driver-hyperkit

🔥  Creating hyperkit VM (CPUs=2, Memory=2000MB, Disk=20000MB) ...
⚠️  VM may be unable to resolve external DNS records
⚠️  VM is unable to access k8s.gcr.io, you may need to configure a proxy or set --image-repository
🐳  Preparing Kubernetes v1.16.2 on Docker '18.09.9' ...
🚜  Pulling images ...
❌  Unable to pull images, which may be OK: running cmd: "/bin/bash -c \"sudo env PATH=/var/lib/minikube/binaries/v1.16.2:$PATH kubeadm config images pull --config /var/tmp/minikube/kubeadm.yaml\"": Process exited with status 1
🚀  Launching Kubernetes ...
⌛  Waiting for: apiserverj
🏄  Done! kubectl is now configured to use "minikube"

The output of the minikube logs command:

Huge spam of this:

Nov 19 18:03:53 minikube kubelet[2721]: E1119 18:03:53.519954    2721 pod_workers.go:191] Error syncing pod c3e29047da86ce6690916750ab69c40b ("kube-addon-manager-minikube_kube-system(c3e29047da86ce6690916750ab69c40b)"), skipping: failed to "StartContainer" for "kube-addon-manager" with ImagePullBackOff: "Back-off pulling image \"k8s.gcr.io/kube-addon-manager:v9.0.2\""
Nov 19 18:04:07 minikube kubelet[2721]: E1119 18:04:07.531993    2721 remote_image.go:113] PullImage "k8s.gcr.io/kube-addon-manager:v9.0.2" from image service failed: rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.2:60592->192.168.64.1:53: read: connection refused
Nov 19 18:04:07 minikube kubelet[2721]: E1119 18:04:07.532050    2721 kuberuntime_image.go:50] Pull image "k8s.gcr.io/kube-addon-manager:v9.0.2" failed: rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.2:60592->192.168.64.1:53: read: connection refused
Nov 19 18:04:07 minikube kubelet[2721]: E1119 18:04:07.532087    2721 kuberuntime_manager.go:783] container start failed: ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.2:60592->192.168.64.1:53: read: connection refused
Nov 19 18:04:07 minikube kubelet[2721]: E1119 18:04:07.532111    2721 pod_workers.go:191] Error syncing pod c3e29047da86ce6690916750ab69c40b ("kube-addon-manager-minikube_kube-system(c3e29047da86ce6690916750ab69c40b)"), skipping: failed to "StartContainer" for "kube-addon-manager" with ErrImagePull: "rpc error: code = Unknown desc = Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.2:60592->192.168.64.1:53: read: connection refused"
Nov 19 18:04:18 minikube kubelet[2721]: E1119 18:04:18.518574    2721 pod_workers.go:191] Error syncing pod c3e29047da86ce6690916750ab69c40b ("kube-addon-manager-minikube_kube-system(c3e29047da86ce6690916750ab69c40b)"), skipping: failed to "StartContainer" for "kube-addon-manager" with ImagePullBackOff: "Back-off pulling image \"k8s.gcr.io/kube-addon-manager:v9.0.2\""

The operating system version: macOS 10.14.3

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 7
Comments: 16 (2 by maintainers)

Commits related to this issue

Add run-in-minikube make target that resolves all deps Only enabled on Darwin (a.k.a. macOS, a.k.a. OS X) because it's the only platform that we have tested. Contributions for other platforms are wel... — committed to rabbitmq/rabbitmq-peer-discovery-k8s by gerhard 5 years ago

Most upvoted comments

@douglascamata I haven’t heard from you I will close this issue, and I am sorry that you faced this isssue, please feel free to reopen this issue I also suggest to try our new driver “docker” with latest version of minikbue which might solve your issue.

minikube start --vm-driver=docker

medyagh on Mar 4, 2020

Quick note: changing the driver from hyperkit to virtualox makes it work.

douglascamata on Nov 19, 2019

I found this issue because I was experiencing the same problem. I also found the culprit:

When SSH’ing into the machine I found out it’s using the host machine’s IP as resolving IP. When using nslookup I first got connection refused, then connection timeout. This makes no sense, but it triggered me for something something else:

For local development I’ve been using dnsmasq, so in fact some other DNS server was running as opposed to the one hyperkit needs. When stopping dnsmasq, setting up a new minikube instance worked as intended. ✅

Jorijn on Dec 9, 2019

Hi @tstromberg!

Do you have any special VPN or Docker for Desktop configuration on this host by any chance?

I have a standard Docker for Mac installation, nothing special. Sometimes I use corporate VPNs (even a VPN over another VPN).

I always disconnect all the VPNs when using minikube, as I know they don’t place nice together.

If there are any commands you would like me to run to give you more information I’m happy to help.

douglascamata on Nov 20, 2019