minikube: hyperkit: external DNS resolution fails: conflict with DNS server running on host

I am using minikube using the hyperkit driver on macOS and experiencing the same problems with dnscrypt-proxy, which listens on 127.0.0.1:53 (udp+tcp). This binary is part of the Cisco Umbrella (corporate OpenDNS) roaming client. I cannot modify the configuration used by dnscrypt-proxy. The dnsmasq.conf changes linked are insufficient to resolve the issue for others.

Is there no way to apply a persistent DNS configuration change to minikube and the system pods?

~
❯ sudo lsof -i :53
Password:
COMMAND     PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
dnscrypt- 44992 nobody    7u  IPv4 0x94dd190be9fa26c5      0t0  UDP localhost:domain
dnscrypt- 44992 nobody    8u  IPv4 0x94dd190bff44e9ed      0t0  TCP localhost:domain (LISTEN)

~
❯ ps ax 44992
  PID   TT  STAT      TIME COMMAND
44992   ??  Ss     0:02.69 /Library/Application Support/OpenDNS Roaming Client/dnscrypt-proxy --user nobody --local-address=127.0.0.1:53 --plugin=/Library/Application Support/OpenDNS Roaming Client/libdcplugin_erc.so -d

~
❯ minikube ssh -- 'cat /etc/resolv.conf; echo "Ping by Name:" && ping -v -c 4 google.com; echo "Ping by IP:" && ping -c 4 8.8.8.8'
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 192.168.64.1
Ping by Name:
ping: bad address 'google.com'
Ping by IP:
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=51 time=23.489 ms
64 bytes from 8.8.8.8: seq=1 ttl=51 time=25.994 ms
64 bytes from 8.8.8.8: seq=2 ttl=51 time=27.537 ms
64 bytes from 8.8.8.8: seq=3 ttl=51 time=25.740 ms

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 23.489/25.690/27.537 ms

I’ve upped the priority based on the number of votes. Help wanted!

We’re hitting this issue and also have an Umbrella / dnscrypt-proxy setup we can’t change.

As a possible workaround, it appears that using Docker Desktop’s vpnkit, e.g. minikube start ... --hyperkit-vpnkit-sock=/Users/zarenner/Library/Containers/com.docker.docker/Data/vpnkit.eth.sock works, correctly forwarding DNS requests to dnscrypt-proxy. But it’s not really ideal since we’d rather not also have Docker Desktop installed and running in addition to minikube. I’m not sure if running a separate vpnkit instance would make more sense.

In my case, I have something listening on :53 and this process running: /usr/local/opt/dnsmasq/sbin/dnsmasq --keep-in-foreground

kubedebug.log

Update, fix to my issue: dnsmasq was running and ignoring the dns request, since it was not listening to the right local addresses

If you are using dnsmasq in your setup and cluster creation fails (stuck at kube-dns initialization) you might need to add listen-address=127.0.0.1,192.168.64.1 to dnsmasq.conf.

Note: If dnsmasq.conf contains listen-address=127.0.0.1 kubernetes discovers dns at 127.0.0.1:53 and tries to use it using bridge ip address, but dnsmasq replies only to reqests from 127.0.0.1

My /usr/local/etc/dnsmasq.conf looks like that:

address=/kube.local/192.168.64.1
listen-address=127.0.0.1,192.168.64.1

Also, please check if the host (Mac OS) has anything listening on port 53 already, or specifically, if dnsmasq is running.

I has run into this issue and it take me a while to fix that. This happends actually only with dnsmasq (or any other dns service) locally installed.

Here is my working dnsmasq.conf for anyone who may get into this error, too. Ofc you need to restart dnsmasq after adding the listen-address.

address=/.localhost/127.0.0.1
listen-address=127.0.0.1
listen-address=192.168.64.1

It seems there is progress ongoing … there is an open PR with a workaround: https://github.com/kubernetes/minikube/pull/3003/files

I was having the same issue on OS X running minikube with also having dnscrypt-proxy which config I can’t change. Adding this to my /etc/pf.conf seems to have fixed it (bridge100 is the hyperkit created bridge):

rdr on bridge100 proto {tcp,udp} from any to bridge100 port 53 -> 127.0.0.1 port 53

xxx:~ $ sudo lsof -i :53
COMMAND     PID           USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
mDNSRespo 70594 _mdnsresponder   52u  IPv6 0x33aa4c2d2a962605      0t0  UDP *:domain
mDNSRespo 70594 _mdnsresponder   53u  IPv4 0x33aa4c2d2a5abe4d      0t0  TCP *:domain (LISTEN)
mDNSRespo 70594 _mdnsresponder   54u  IPv6 0x33aa4c2d244e9b7d      0t0  TCP *:domain (LISTEN)

looks like in my case it’s mDNSResponder that’s using port 53?

@nikolay-te Discovered that pf is angry if you don’t put the rule in the right order:

sudo pfctl -f /etc/pf.conf
/etc/pf.conf:24: Rules must be in order: options, normalization, queueing, translation, filtering

So I: /etc/pf.conf

set limit {tables 10000, table-entries 400000}
scrub-anchor "cisco.anyconnect.vpn"
scrub-anchor "com.apple/*"
nat-anchor "com.apple/*"
rdr-anchor "com.apple/*"
rdr on bridge100 inet proto {tcp,udp} from any to bridge100 port 53 -> 127.0.0.1 port 53
dummynet-anchor "com.apple/*"
anchor "cisco.anyconnect.vpn"
anchor "com.apple/*"
load anchor "com.apple" from "/etc/pf.anchors/com.apple"

sudo pfctl -f /etc/pf.conf

I’m on a Mac that’s smothered with Cisco Umbrella sauce. BTW, I can’t run minikube AT ALL unless I disconnect from my corporate VPN

This allowed my minikube VM to hit my corporate private registry url via Umbrella’s DNS

I also have dnscrypt-proxy running:

ps aux | grep crypt
nobody           78236   0.0  0.0  5928120   4492   ??  SN   11:53AM   0:00.78 /opt/cisco/anyconnect/bin/dnscrypt-proxy --user nobody --local-address=127.0.0.1:53 --plugin=/opt/cisco/anyconnect/lib/libdcplugin_erc.so -d

Hey folks, you need to check the IP Adresses and the DNS resolution. This issue is mostly based on wrong proxy or dns settings. Just make sure, that the routing and dns is correct and everything will be fine.

If you have dnsmasq, you need to extend your configuration like here

Any other issue should be traced back.

Can someone on this thread that doesn’t have “Cisco Umbrella sauce” (sadly, that includes me) do a sudo lsof -i :53, and post the output?

In particular, I’m wondering if minikube’s DNS server, whatever that is, is trying to bind to all interfaces, instead of just the bridge. If I’m looking at this right, I’m wondering why there should be a conflict here: Umbrella only binds to localhost, so a bind to the bridge shouldn’t be affected by it. But, if minikube were instead trying to bind to all interfaces, then that would be a different story. The lsof from a normal setup should help answer that.

our current integration machines are out of work due to covid (they were in the office) we are looking into adding mac integration tests, once we have integration back, I would accept a PR that adds integration tests that fixes this issue

I was just investigating another DNS-related issue on my colleagues OS X High Sieera, and it turns out, it’s really terrible.

He has a mDNSresponder service occupying *:53/UDP. It does not actually respond to any of the requests made by the minikube VM. So, DNS is effectively broken for the VM, using HyperKit. We did not know about the DNSresponder thingie, since we installed dnsmasq, and began using it. Eventually, we realized it’s failing to start exactly because it cannot bind.

I think this is yet one more example of horrific engineering on Apple’s side. I am not saying everything they do is garbag, but WHY on earth would you have a service that (according to Google) is there to support Bonjour (kind of a discorery protocol) occupiying *:53???

If there was at least a way how to tell it not to occupy ALL the ip addresses, such that actually useful software could listen on them instead 😃

EDIT: You could possibly maybe tell Minikube to use a different IP for the DNS resolution, as routing seems to be working, but if you need that IP to be dnsmasq, you will have to host ti likely on a different VM, not the OS X 😦((

This seems to help in the meantime:

sudo socat UDP4-RECVFROM:53,bind=192.168.64.1,fork UDP4-SENDTO:127.0.0.1:53

I has run into this issue and it take me a while to fix that. This happends actually only with dnsmasq (or any other dns service) locally installed.

Here is my working dnsmasq.conf for anyone who may get into this error, too. Ofc you need to restart dnsmasq after adding the listen-address.

address=/.localhost/127.0.0.1
listen-address=127.0.0.1
listen-address=192.168.64.1

I did that and also had to restart dnsmasq to make it work (for macOS):

sudo launchctl stop homebrew.mxcl.dnsmasq
sudo launchctl start homebrew.mxcl.dnsmasq

I can confirm that @zarenner’s solution works with adding a line for listen-address for 192.168.64.1

Currently facing the same issue on linux with dnsmasq