minikube: hyperkit: VM is unable to access k8s.gcr.io (when VPN is in use)

Using --hyperkit-vpnkit-sock=auto tries to use the VPNKit included with Docker Desktop. But the whole reason to go to minikube/hyperkit was to remove the dependency on Docker Desktop. Is there any way to use VPNKit without Desktop?

We should update the error string to mention trying --driver=docker when a VPN is in use.

At last it seems this issue is with the hyperkitVM, because It works fine with the virtual box. Those who want their pod to be up and running, without worrying about the type of VM, Can follow these steps

1. Delete the existing minikube by running minikube delete
2. Install the virtual box brew cask install virtualbox , make sure you do the installation proper way, if you are using macos it will ask for permission to be enabled from firevault, once enabled re-run the command until you see virtual box is installed successfully.
3. once all is set up finally start the minikube using the virtual box minikube start --vm-driver=virtualbox
4. At last you can run the pod using your usual yaml file _kubectl create -f “yourfilename.yaml”
5. run kubectl get pods
6. in case of error you can check kubectl describe pods to check the detailed errors. If your yml is correct and path for the Image is correct, you should be good and will just require till step 5.

Using --hyperkit-vpnkit-sock=auto tries to use the VPNKit included with Docker Desktop. But the whole reason to go to minikube/hyperkit was to remove the dependency on Docker Desktop. Is there any way to use VPNKit without Desktop?

I’ve yet to find an answer to this as well. Why do people like macs again? =) I hope one day our company opens up other options other than forcing macs, but for now I still have my trusty linux desktop that just works.

For those of us who have to use a Mac and Cisco AnyConnect- are there any options to resolve this? Perhaps installing vpn kit from source somehow?

Using --hyperkit-vpnkit-sock=auto tries to use the VPNKit included with Docker Desktop. But the whole reason to go to minikube/hyperkit was to remove the dependency on Docker Desktop. Is there any way to use VPNKit without Desktop?

I’ve yet to find an answer to this as well. Why do people like macs again? =) I hope one day our company opens up other options other than forcing macs, but for now I still have my trusty linux desktop that just works.

Running minikube alongside vpnkit on Mac seems to work, with a couple of gotchas.

Building vpnkit from source fails on original’s repo (moby/vpnkit) and latest binaries are not available anywhere. In short, the Makefile for Mac build needs a bit of tweaking for opam dependencies. So I forked the original repo to build from source:

# install build dependencies
brew install opam gpatch pkg-config dune dylibbundler libtool automake

# build vpnkit
git clone git@github.com:ar2pi/vpnkit.git
cd vpnkit
make -f Makefile.darwin ocaml
make -f Makefile.darwin depends
make -f Makefile.darwin build
cp ~/.opam/4.12.0/bin/vpnkit /usr/local/bin/vpnkit

Then hyperkit Homebrew’s install also has a known issue, so we need to build that from source as well.

# build hyperkit
brew uninstall hyperkit
git clone git@github.com:moby/hyperkit.git
cd hyperkit
make
cp build/hyperkit /usr/local/bin/hyperkit

Once you have vpnkit and hyperkit, you can run:

# terminal 1
vpnkit --ethernet=/tmp/vpnkit.eth.sock
# terminal 2
minikube start --driver hyperkit --hyperkit-vpnkit-sock=/tmp/vpnkit.eth.sock --memory 8192 --cpus 4
eval $(minikube -p minikube docker-env)

# [...] your docker commands

And voilà! Docker will run within minikube’s hyperkit VM, through vpnkit.

But there’s still a couple of connection error messages that have been bugging me for a few days when starting / restarting a new VM:

[...]
✨  Using the hyperkit driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🔄  Restarting existing hyperkit VM for "minikube" ...
❗  This VM is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
[...]

And on vpnkit’s output we can see:

time="2021-11-27T02:53:59Z" level=warning msg="arp: ARP table has no entry for 192.168.64.26"
time="2021-11-27T02:53:59Z" level=info msg="ipv4: IP.output: could not determine link-layer address for local network (0.0.0.0/0) ip 192.168.64.26"
time="2021-11-27T02:53:59Z" level=warning msg="ipv4: Could not find 192.168.64.26 on the local network"
time="2021-11-27T02:53:59Z" level=warning msg="Wire: Error sending TCP packet via IP: no route to destination: no response for IP on local network"

Once VM is started everything appears to be fine though, could pull a few images and run basic docker commands. Haven’t yet tested container to container networking, nor file mounts.

I’m on mac and have to use the hyperkit driver for Ingress to work. I solved this VPN issue by adding the --hyperkit-vpnkit-sock=auto flag to minikube start. This way it uses the VPNKit that ships with Docker for Mac. I found it here: https://minikube.sigs.k8s.io/docs/drivers/hyperkit/#special-features

Location of the VPNKit socket used for networking. If empty, disables Hyperkit VPNKitSock, if ‘auto’ uses Docker for Mac VPNKit connection, otherwise uses the specified VSock

@vignesh-subburaj great. I’m not sure what the cause of this issue is, so for now it seems our recommended advice is to try to use the docker driver instead of the hyperkit driver.

If anyone would be interested in updating our warning message from:

This VM is having trouble accessing https://k8s.gcr.io

to something like

This VM is having trouble accessing https://k8s.gcr.io -- consider using the `docker` driver by running `minikube start --driver docker`

I just workaround by installing proxyman in my local laptop and add HTTP_PROXY settings when launching minikube

What are you setting your HTTP_PROXY to? The Proxyman “listening on” address and port?

After turning on VPN, I can’t pull image because can’t connect to the internet so I just run by

export HTTPS_PROXY=192.168.64.1:9090 export HTTP_PROXY=192.168.64.1:9090 export NO_PROXY=localhost,127.0.0.1,10.96.0.0/12,192.168.59.0/24,192.168.39.0/24,192.168.64.0/24 minikube start

and the postman will start a proxy and listen on port 9090

Reference to https://minikube.sigs.k8s.io/docs/reference/networking/proxy/

Still an issue for me:

😄  minikube v1.15.1 on Darwin 10.15.7
✨  Using the hyperkit driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating hyperkit VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
❗  This VM is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/

minikube version: v1.15.1
commit: 23f40a012abb52eff365ff99a709501a61ac5876

EDIT: I should add that no VPN is in use

Hi team I met the same error. Mac Mojave , version 10.14.6. run the command minikube start --vm-driver=hyperkit to start the minikube, below is the logs

😄  minikube v1.9.2 on Darwin 10.14.6
✨  Using the hyperkit driver based on existing profile
👍  Starting control plane node m01 in cluster minikube
🔄  Restarting existing hyperkit VM for "minikube" ...
🐳  Preparing Kubernetes v1.18.0 on Docker 19.03.8 ...
❗  This VM is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🌟  Enabling addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube"

I am using Ciso VPN, no matter i set the proxy or not, the result is the same.

kubectl -n kube-system get pods
NAME                               READY   STATUS    RESTARTS   AGE
coredns-66bff467f8-fnxht           1/1     Running   5          56m
coredns-66bff467f8-vxr5s           1/1     Running   5          56m
etcd-minikube                      1/1     Running   5          56m
kube-apiserver-minikube            1/1     Running   5          56m
kube-controller-manager-minikube   1/1     Running   5          56m
kube-proxy-pt5js                   1/1     Running   5          56m
kube-scheduler-minikube            1/1     Running   5          56m
storage-provisioner                1/1     Running   8          56m

Can get the dns log

kubectl -n kube-system logs coredns-66bff467f8-fnxht 
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
I0423 14:01:35.230402       1 trace.go:116] Trace[2019727887]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.226674406 +0000 UTC m=+0.085300319) (total time: 30.002647734s):
Trace[2019727887]: [30.002647734s] [30.002647734s] END
E0423 14:01:35.230463       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
I0423 14:01:35.230738       1 trace.go:116] Trace[1427131847]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.22650071 +0000 UTC m=+0.085126637) (total time: 30.004178291s):
Trace[1427131847]: [30.004178291s] [30.004178291s] END
E0423 14:01:35.230753       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
I0423 14:01:35.235236       1 trace.go:116] Trace[939984059]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.232689948 +0000 UTC m=+0.091315902) (total time: 30.002522331s):
Trace[939984059]: [30.002522331s] [30.002522331s] END
E0423 14:01:35.235287       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

I can get the some docker images inside the VM

minikube ssh
                         _             _
            _         _ ( )           ( )
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ docker images
REPOSITORY                                TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-proxy                     v1.18.0             43940c34f24f        4 weeks ago         117MB
k8s.gcr.io/kube-apiserver                 v1.18.0             74060cea7f70        4 weeks ago         173MB
k8s.gcr.io/kube-controller-manager        v1.18.0             d3e55153f52f        4 weeks ago         162MB
k8s.gcr.io/kube-scheduler                 v1.18.0             a31f78c7c8ce        4 weeks ago         95.3MB
kubernetesui/dashboard                    v2.0.0-rc6          cdc71b5a8a0e        5 weeks ago         221MB
k8s.gcr.io/pause                          3.2                 80d28bedfe5d        2 months ago        683kB
k8s.gcr.io/coredns                        1.6.7               67da37a9a360        2 months ago        43.8MB
kindest/kindnetd                          0.5.3               aa67fec7d7ef        5 months ago        78.5MB
k8s.gcr.io/etcd                           3.4.3-0             303ce5db0e90        6 months ago        288MB
kubernetesui/metrics-scraper              v1.0.2              3b08661dc379        6 months ago        40.1MB
gcr.io/k8s-minikube/storage-provisioner   v1.8.1              4689081edb10        2 years ago         80.8MB

But when i try to login the dockerhub, failed

$ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: dllbh
Password:
Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Some article said it has relationship with the DNS

sudo lsof -i4UDP:53 -P -n
COMMAND     PID           USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
mDNSRespo 93271 _mdnsresponder   33u  IPv4 0x1154dc77691251b5      0t0  UDP *:53

After kill the process, error still occurs. It seems a bug for minikube with hyperkit driver. How to fix this?

By the way,

minikube version
minikube version: v1.9.2
commit: 93af9c1e43cab9618e301bc9fa720c63d5efa393

dnsmasq was causing issues for me:

brew uninstall dnsmasq
sudo lsof -i4UDP:53 -P -n

COMMAND PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
dnsmasq 123 nobody    4u  IPv4 0xb150fdd17fe0cd81      0t0  UDP 127.0.0.1:53

sudo kill -9 123

Minikube then runs without issues.

If you want to keep dnsmasq installed then I would try these instructions: https://github.com/kubernetes/minikube/issues/3036#issuecomment-423150899

With minikube on virtualbox, all good till the deployment. My Problem is, pods running NodeJS service can’t connect to the external services (e.g. MongoDB Atlas). Note: I have deployed on DigitalOcen, the same docker build work perfectly fine.

2020-02-24 17:05:02 [ info ] : server started on port 3333 (local) Error: Error: querySrv ENOTFOUND _mongodb._tcp.twrdev-hokqm.mongodb.net at NativeConnection.<anonymous> (/usr/src/app/services/mongoose.js:57:19) at NativeConnection.emit (events.js:196:13) at NativeConnection.EventEmitter.emit (domain.js:471:20) at /usr/src/app/node_modules/mongoose/lib/connection.js:817:37 at processTicksAndRejections (internal/process/task_queues.js:82:9) 2020-02-24 17:05:02 [ error ] : Error connecting to DB : querySrv ENOTFOUND _mongodb._tcp.twrdev-hokqm.mongodb.net Error: querySrv ENOTFOUND _mongodb._tcp.twrdev-hokqm.mongodb.net at QueryReqWrap.onresolve [as oncomplete] (dns.js:203:19) { errno: ‘ENOTFOUND’, code: ‘ENOTFOUND’, syscall: ‘querySrv’, hostname: ‘_mongodb._tcp.twrdev-hokqm.mongodb.net’ } [Unhandled Rejection] Node NOT Exiting… Error: Could not connect to database at NativeConnection.<anonymous> (/usr/src/app/services/mongoose.js:70:19) at NativeConnection.emit (events.js:196:13) at NativeConnection.EventEmitter.emit (domain.js:471:20) at NativeConnection.set (/usr/src/app/node_modules/mongoose/lib/connection.js:125:12) at /usr/src/app/node_modules/mongoose/lib/connection.js:711:26 at executeCallback (/usr/src/app/node_modules/mongodb/lib/operations/execute_operation.js:74:5) at /usr/src/app/node_modules/mongodb/lib/operations/connect.js:209:23 at /usr/src/app/node_modules/mongodb/lib/operations/connect.js:299:21 at QueryReqWrap.callback (/usr/src/app/node_modules/mongodb/lib/core/uri_parser.js:56:21) at QueryReqWrap.onresolve [as oncomplete] (dns.js:203:10) 2020-02-24 17:05:02 [ error ] : Disconnected from Database! Any suggestion?? Thanks in advance

On my machine is some corporate software (cisco security, vpn) running and I got it working by starting vpnkit along with minikube:

vpnkit --ethernet /tmp/vpn.socket

minikube start --driver hyperkit --hyperkit-vpnkit-sock=/tmp/vpn.socket

BTW k8s.gcr.io is deprecated as a source of container images

I’m on mac and have to use the hyperkit driver for Ingress to work. I solved this VPN issue by adding the --hyperkit-vpnkit-sock=auto flag to minikube start. This way it uses the VPNKit that ships with Docker for Mac. I found it here: https://minikube.sigs.k8s.io/docs/drivers/hyperkit/#special-features

Location of the VPNKit socket used for networking. If empty, disables Hyperkit VPNKitSock, if ‘auto’ uses Docker for Mac VPNKit connection, otherwise uses the specified VSock

minikube start --hyperkit-vpnkit-sock=auto works for me. I found when close Cisco VPN client and minikube hyperkit VM can’t access internet. I try to turn off and then of Mac network, the VM can access internet again.

I’m also seeing this, trying to run on a Mac that has a “Cisco Anconnect” VPN software and hyperkit … the easiest workaround is to use the --vm-driver=virtualbox option. I’m happy to provide config information if anyone really wants the details however my gut feeling is that the corporate installed Cisco VPN software is the culprit, it futzes with DNS even when its not “turned on” to ensure I’m not accessing “inappropriate” websites like say … urban dictionary (I know, it seems unreasonable, but thats just collateral damage for a decent security posture so I put up with it).

For hyperkit, VM is unable to access k8s.gcr.io says to me that this is likely a DNS issue, and likely #3036

Do you mind sharing the output of the following two commands for me?

• sudo lsof -i4UDP:53 -P -n
• ps -afe | grep dns

Thank you!

Yes, Docker works just fine (with/without login) Yes, I use a VPN (hopefully everyone does, these days…) but the problem persists with/without VPN.

Yep, the docker login was the first thing I tried as I assumed that would have been the case; but really, the problem must be around the egress network for the hyperkit VM: as mentioned, when I use --vm-driver virtualbox it works just fine; VBox, by default, creates a bridge and allows egress to the Internet for the VM, I’m guessing Hyperkit doesn’t, and as the VM is on a separate subnet than the host network, then it doesn’t know how to reach the external network.

BTW - this seems to be a good place to remark how awesome Minikube is - I absolutely love it, it has helped me immensely in exploring K8s, developing a whole stack of services, and generally making my developer’s life so much easier: thank you deeply to the contributors!