minikube: Minikube rejects user connection, Localkube fails to start ("The connection to the server was refused - did you specify the right host or port?")

Metadata

BUG REPORT or FEATURE REQUEST? (choose one): Bug report Minikube version (use minikube version): v0.20.0 Environment:

• OS: Red Hat Enterprise Linux Workstation 7.3 (Maipo)
• VM Driver: virtualbox
• ISO version: v0.21.0

What happened

Localkube fails to start, preventing access to the Minikube VM.

In short, I start Minikube with minikube start and it starts up fine without error. However, any time that I try to interact with the local cluster, I receive this error:

$ kubectl cluster-info 
The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port?

Using kubectl, I am unable to do anything to interact with the local cluster at all. This is similar to #1381, #1498, #1546, etc. However, when I checked the logs, I saw that the localkube.service failed to start. I don’t understand how or why it failed, but it seems like this is partially to blame for the inability to interact with Minikube.

Disclaimer

I am behind a firewall and using a custom ISO file. However, my ISO only includes a custom resolv.conf and added SSL certificates to /etc/ssl/ and /etc/ssl/certs. The ISO was built with the last upstream commit being 6f5972f9b590c03241c986df5630d85e07a95813.

Expected behavior

1. minikube start
2. kubectl cluster-info

Kubernetes master is running at https://192.168.99.100:8443 To further debug and diagnose cluster problems, use ‘kubectl cluster-info dump’.

How to reproduce

1. minikube start
2. kubectl cluster-info

The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port?

1. minikube ssh
2. /usr/local/bin/localkube --dns-domain=cluster.local --node-ip=192.168.99.100 --generate-certs=false --logtostderr=true --enable-dns=false
3. Error stacktrace

Other information

This is a little shorter than the full stacktrace above.

$ systemctl status -l localkube
● localkube.service - Localkube
   Loaded: loaded (/lib/systemd/system/localkube.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Fri 2017-07-14 17:50:34 UTC; 1s ago
     Docs: https://github.com/kubernetes/minikube/tree/master/pkg/localkube
  Process: 20625 ExecStart=/usr/local/bin/localkube --dns-domain=cluster.local --node-ip=192.168.99.100 --generate-certs=false --logtostderr=true --enable-dns=false (code=exited, status=2)
 Main PID: 20625 (code=exited, status=2)
      CPU: 346ms

Jul 14 17:50:34 minikube localkube[20625]: net.lookupIP(0x68e6780, 0xc421a65a40, 0xc421aa73c8, 0x9, 0xc42000d740, 0x0, 0x0, 0xc42154b698, 0x6c5cae)
Jul 14 17:50:34 minikube localkube[20625]:         /usr/local/go/src/net/lookup_unix.go:70 +0xf9
Jul 14 17:50:34 minikube localkube[20625]: net.glob..func11(0x68e6780, 0xc421a65a40, 0x4b917c8, 0xc421aa73c8, 0x9, 0xc421c07900, 0x0, 0xc42000d740, 0xc4203eb900, 0x3b9aca00)
Jul 14 17:50:34 minikube localkube[20625]:         /usr/local/go/src/net/hook.go:19 +0x52
Jul 14 17:50:34 minikube localkube[20625]: net.lookupIPContext.func1(0xc42154b788, 0x627d88, 0xc420db3590, 0x68e6740)
Jul 14 17:50:34 minikube localkube[20625]:         /usr/local/go/src/net/lookup.go:119 +0x5c
Jul 14 17:50:34 minikube localkube[20625]: internal/singleflight.(*Group).doCall(0x6b0ece0, 0xc421a86f50, 0xc421aa73c8, 0x9, 0xc421a0f0b0)
Jul 14 17:50:34 minikube localkube[20625]:         /usr/local/go/src/internal/singleflight/singleflight.go:93 +0x3c
Jul 14 17:50:34 minikube localkube[20625]: created by internal/singleflight.(*Group).DoChan
Jul 14 17:50:34 minikube localkube[20625]:         /usr/local/go/src/internal/singleflight/singleflight.go:86 +0x339

…and…

$ journalctl -lu localkube
-- Logs begin at Fri 2017-07-14 17:45:04 UTC, end at Fri 2017-07-14 17:51:25 UTC. --
Jul 14 17:45:04 minikube localkube[15262]:         /usr/local/go/src/runtime/netpoll.go:160 +0x59
Jul 14 17:45:04 minikube localkube[15262]: net.(*pollDesc).wait(0xc4216e8ae0, 0x72, 0xc420e22e78, 0xc42001a130)
Jul 14 17:45:04 minikube localkube[15262]:         /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
Jul 14 17:45:04 minikube localkube[15262]: net.(*pollDesc).waitRead(0xc4216e8ae0, 0x68c2940, 0xc42001a130)
Jul 14 17:45:04 minikube localkube[15262]:         /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
Jul 14 17:45:04 minikube localkube[15262]: net.(*netFD).Read(0xc4216e8a80, 0xc4217833c2, 0x1, 0x1, 0x0, 0x68c2940, 0xc42001a130)
Jul 14 17:45:04 minikube localkube[15262]:         /usr/local/go/src/net/fd_unix.go:243 +0x1a1
Jul 14 17:45:04 minikube localkube[15262]: net.(*conn).Read(0xc420fef468, 0xc4217833c2, 0x1, 0x1, 0x0, 0x0, 0x0)
Jul 14 17:45:04 minikube localkube[15262]:         /usr/local/go/src/net/net.go:173 +0x70
Jul 14 17:45:04 minikube localkube[15262]: net/http.(*connReader).closeNotifyAwaitActivityRead(0xc420467040, 0xc42180f020, 0xc4218284c0)
Jul 14 17:45:04 minikube localkube[15262]:         /usr/local/go/src/net/http/server.go:602 +0x71
Jul 14 17:45:04 minikube localkube[15262]: created by net/http.(*connReader).startBackgroundRead
Jul 14 17:45:04 minikube localkube[15262]:         /usr/local/go/src/net/http/server.go:597 +0x98
Jul 14 17:45:04 minikube localkube[15262]: goroutine 623 [IO wait]:
Jul 14 17:45:04 minikube localkube[15262]: net.runtime_pollWait(0x7f440420cce0, 0x72, 0x43)
Jul 14 17:45:04 minikube localkube[15262]:         /usr/local/go/src/runtime/netpoll.go:160 +0x59
Jul 14 17:45:04 minikube localkube[15262]: net.(*pollDesc).wait(0xc4216e8920, 0x72, 0xc4213ef678, 0xc42001a130)
Jul 14 17:45:04 minikube localkube[15262]:         /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
Jul 14 17:45:04 minikube localkube[15262]: net.(*pollDesc).waitRead(0xc4216e8920, 0x68c2940, 0xc42001a130)
Jul 14 17:45:04 minikube localkube[15262]:         /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
Jul 14 17:45:04 minikube localkube[15262]: net.(*netFD).Read(0xc4216e88c0, 0xc421783662, 0x1, 0x1, 0x0, 0x68c2940, 0xc42001a130)
Jul 14 17:45:04 minikube localkube[15262]:         /usr/local/go/src/net/fd_unix.go:243 +0x1a1
Jul 14 17:45:04 minikube localkube[15262]: net.(*conn).Read(0xc420fef448, 0xc421783662, 0x1, 0x1, 0x0, 0x0, 0x0)
Jul 14 17:45:04 minikube localkube[15262]:         /usr/local/go/src/net/net.go:173 +0x70
Jul 14 17:45:04 minikube localkube[15262]: net/http.(*connReader).closeNotifyAwaitActivityRead(0xc420467bc0, 0xc42180f3e0, 0xc4218285a0)
Jul 14 17:45:04 minikube localkube[15262]:         /usr/local/go/src/net/http/server.go:602 +0x71
Jul 14 17:45:04 minikube localkube[15262]: created by net/http.(*connReader).startBackgroundRead
Jul 14 17:45:04 minikube localkube[15262]:         /usr/local/go/src/net/http/server.go:597 +0x98
Jul 14 17:45:04 minikube localkube[15262]: goroutine 624 [IO wait]:
Jul 14 17:45:04 minikube localkube[15262]: net.runtime_pollWait(0x7f440420cc20, 0x72, 0x44)
Jul 14 17:45:04 minikube localkube[15262]:         /usr/local/go/src/runtime/netpoll.go:160 +0x59
Jul 14 17:45:04 minikube localkube[15262]: net.(*pollDesc).wait(0xc4216e8990, 0x72, 0xc4203a8e78, 0xc42001a130)
Jul 14 17:45:04 minikube localkube[15262]:         /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
Jul 14 17:45:04 minikube localkube[15262]: net.(*pollDesc).waitRead(0xc4216e8990, 0x68c2940, 0xc42001a130)
Jul 14 17:45:04 minikube localkube[15262]:         /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
--More--