minikube: Pods unable to connect to connect to internet when using Minikube with --driver=none option

What Happened?

For some specific use cases, I want to start Minikube with the --driver=none option.

$ sudo minikube start --driver=none 
😄  minikube v1.31.2 on Ubuntu 22.04
✨  Using the none driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🤹  Running on localhost (CPUs=12, Memory=64249MB, Disk=233348MB) ...
ℹ  OS release is Ubuntu 22.04.3 LTS
🐳  Preparing Kubernetes v1.27.4 on Docker 24.0.5 ...
    ▪ kubelet.resolv-conf=/run/systemd/resolve/resolv.conf
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🤹  Configuring local host environment ...

❗  The 'none' driver is designed for experts who need to integrate with an existing VM
💡  Most users should use the newer 'docker' driver instead, which does not require root!
📘  For more information, see: https://minikube.sigs.k8s.io/docs/reference/drivers/none/

❗  kubectl and minikube configuration will be stored in /root
❗  To use kubectl or minikube commands as your own user, you may need to relocate them. For example, to overwrite your own settings, run:

    ▪ sudo mv /root/.kube /root/.minikube $HOME
    ▪ sudo chown -R $USER $HOME/.kube $HOME/.minikube

💡  This can also be done automatically by setting the env var CHANGE_MINIKUBE_NONE_USER=true
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner

❗  /usr/local/bin/kubectl is version 1.24.2, which may have incompatibilities with Kubernetes 1.27.4.
    ▪ Want kubectl v1.27.4? Try 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

All the pods are running fine:

$ sudo kubectl get pods --all-namespaces
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
default       ubuntu-pod                         1/1     Running   0          3m42s
kube-system   coredns-5d78c9869d-kf8h6           1/1     Running   0          5m55s
kube-system   etcd-abhishek                      1/1     Running   2          6m7s
kube-system   kube-apiserver-abhishek            1/1     Running   2          6m11s
kube-system   kube-controller-manager-abhishek   1/1     Running   2          6m8s
kube-system   kube-proxy-b4dn8                   1/1     Running   0          5m55s
kube-system   kube-scheduler-abhishek            1/1     Running   2          6m7s
kube-system   storage-provisioner                1/1     Running   0          6m7s

Then I wrote a simple ubuntu-pod to check internet access

ubuntu-pod.yml:

apiVersion: v1                                                                                                                                                                                 
kind: Pod                                                                                                                                                                                      
metadata:                                                                                                                                                                                      
  name: ubuntu-pod                                                                                                                                                                             
spec:                                                                                                                                                                                          
  containers:                                                                                                                                                                                  
  - name: ubuntu-container                                                                                                                                                                     
    image: ubuntu:latest
    command:
      - "sleep"
      - "3600"
    stdin: true
    tty: true

Then I try to launch the terminal of that pod and try to check internet connectivity:

$ sudo kubectl exec -it ubuntu-pod -- /bin/bash
root@ubuntu-pod:/# apt update
Ign:1 http://security.ubuntu.com/ubuntu jammy-security InRelease         
Ign:2 http://archive.ubuntu.com/ubuntu jammy InRelease                   
Ign:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease           
Ign:1 http://security.ubuntu.com/ubuntu jammy-security InRelease         
Ign:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease         
Ign:1 http://security.ubuntu.com/ubuntu jammy-security InRelease         
Ign:2 http://archive.ubuntu.com/ubuntu jammy InRelease                   
Err:1 http://security.ubuntu.com/ubuntu jammy-security InRelease         
  Temporary failure resolving 'security.ubuntu.com'
Ign:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease           
Ign:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Ign:2 http://archive.ubuntu.com/ubuntu jammy InRelease
Ign:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Ign:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Err:2 http://archive.ubuntu.com/ubuntu jammy InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Reading package lists... Done        
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-backports/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/jammy-security/InRelease  Temporary failure resolving 'security.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
root@ubuntu-pod:/#

I understand when I use the driver=none in minikube, it makes use of the host system. Is the issue in DNS resolution due to the host machine? I am not sure. But the internet works fine on my host machine.

When I remove the --driver=option and do $ minikube start and follow the above steps, the pods connects to internet just fine.

Attach the log file

log.txt Please consider the last start…

Operating System

Ubuntu

Driver

None

About this issue

  • Original URL
  • State: open
  • Created 8 months ago
  • Comments: 32 (15 by maintainers)

Most upvoted comments

So long as the minikube is up and running, and docker images are pulling and eventually running (the slowness is likely just as it needs to pull images off the internet) I don’t think there is necessarily anything wrong the error is what happens when there is no minikube network and if it doesn’t exist minikube will go on to create it. It does not reflect an actual error.

I see this on my ubuntu system I did minikube stop; minikube delete --purge --all then my starting point has no minikube network

docker network list
NETWORK ID     NAME                  DRIVER    SCOPE
62ae09eecd76   bridge                bridge    local
1e762a9bc9f3   host                  host      local
7da388b6d6cc   kubetracing_default   bridge    local
b207c08b8969   none                  null      local

If I then run minikube start --driver=docker and then minikube logs my logs contain the error you include but importantly if you examin below that section you can see it does the creation in the below snippetand you can check that with docker network ls | grep minikube

I1025 10:18:08.149077  190591 network_create.go:123] attempt to create docker network minikube 192.168.49.0/24 with gateway 192.168.49.1 and MTU of 1500 ...
I1025 10:18:08.149114  190591 cli_runner.go:164] Run: docker network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=1500 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=minikube minikube

Triage notes

minikube logs section with error than creation

``` I1025 10:18:08.120728 190591 cli_runner.go:164] Run: docker network inspect minikube --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}" W1025 10:18:08.131898 190591 cli_runner.go:211] docker network inspect minikube --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}" returned with exit code 1 I1025 10:18:08.131935 190591 network_create.go:281] running [docker network inspect minikube] to gather additional debugging logs... I1025 10:18:08.131940 190591 cli_runner.go:164] Run: docker network inspect minikube W1025 10:18:08.140259 190591 cli_runner.go:211] docker network inspect minikube returned with exit code 1 I1025 10:18:08.140269 190591 network_create.go:284] error running [docker network inspect minikube]: docker network inspect minikube: exit status 1 stdout: []

stderr: Error response from daemon: network minikube not found I1025 10:18:08.140275 190591 network_create.go:286] output of [docker network inspect minikube]: – stdout – []

– /stdout – ** stderr ** Error response from daemon: network minikube not found

** /stderr ** I1025 10:18:08.140309 190591 cli_runner.go:164] Run: docker network inspect bridge --format “{“Name”: “{{.Name}}”,“Driver”: “{{.Driver}}”,“Subnet”: “{{range .IPAM.Config}}{{.Subnet}}{{end}}”,“Gateway”: “{{range .IPAM.Config}}{{.Gateway}}{{end}}”,“MTU”: {{if (index .Options “com.docker.network.driver.mtu”)}}{{(index .Options “com.docker.network.driver.mtu”)}}{{else}}0{{end}}, “ContainerIPs”: [{{range $k,$v := .Containers }}”{{$v.IPv4Address}}“,{{end}}]}” I1025 10:18:08.149046 190591 network.go:209] using free private subnet 192.168.49.0/24: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:0xc001428b40} I1025 10:18:08.149077 190591 network_create.go:123] attempt to create docker network minikube 192.168.49.0/24 with gateway 192.168.49.1 and MTU of 1500 … I1025 10:18:08.149114 190591 cli_runner.go:164] Run: docker network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=1500 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=minikube minikube I1025 10:18:08.229518 190591 network_create.go:107] docker network minikube 192.168.49.0/24 created

</details>
+1
pnasrat on Oct 25, 2023
Read more comments on GitHub
← minikube: podman on ppc64le: Error: incompatible image architecture "amd64" on "ppc64le" host
minikube: preload causes /var conflicts with Docker Desktop File sharing →