colima: Enabling a `shared` or `bridged` network causes `lima kubectl cluster-info` to fail

Description

colima is unable to verify that the Kubernetes has started correctly when a shared or bridged network is used for the underlying VM.

Version

Colima Version: 0.4.4 Lima Version: 0.11.2 Qemu Version: 7.0.0

Operating System

  • macOS Intel
  • macOS M1
  • Linux

Reproduction Steps

This used to work, but started failing at some point.

I have set up networking for lima as discussed here:

https://github.com/lima-vm/lima/blob/master/docs/network.md

And then in this file: .lima/_config/override.yaml I have this:

networks:
  - lima: bridged

When I try to start colima with the network uncommented then I get:

INFO[0000] starting colima
INFO[0000] runtime: containerd+k3s
INFO[0000] preparing network ...                         context=vm
INFO[0000] starting ...                                  context=vm
INFO[0030] provisioning ...                              context=containerd
INFO[0030] starting ...                                  context=containerd
INFO[0036] provisioning ...                              context=kubernetes
INFO[0037] starting ...                                  context=kubernetes
FATA[0062] error starting kubernetes: error running [lima kubectl cluster-info], output: "The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?", err: "exit status 1"

Expected behaviour

I would expect the VM to come up with no errors and everything in a useable state.

Additional context

I am able to reproduce this even after recompiling and installing vde-2 and vde_vmnet.

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 2
  • Comments: 18 (4 by maintainers)

Most upvoted comments

If I attempt to change /etc/sudoers.d/colima, it just reverts itself when I stop/start colima. I don’t see how or where I can persist a change there for colima.

Yeah, true. It is not expected a user would modify the file directly.

Maybe an introduction of --network-type flag (and config) would help with the options being shared (default) and bridged.

One (unrelated) thing I find curious that you are using virtiofs with QEMU; I thought that would only work with VZ. Or maybe that is also some colima-specific extension.

It is a misinformation from Colima’s end, it is already corrected in main and would be part of the imminent next release.

I can’t get this to work. The Lima override in ~/.lima/_config/override.yaml is not picked up by Colima, as Colima seems to use a separate $LIMA_HOME in ~/.colima/_lima/.

Modifying ~/.colima/_lima/_config/override.yaml does “work”, in that the change is picked up, but since ~/.colima/_lima/_config/networks.yaml only contains a user-v2 network (https://github.com/abiosoft/colima/blob/9b0809d0ed9ad3ff1e57c405f27324e6298ca04f/embedded/network/networks.yaml) it fails with:

errors inspecting instance: [field networks[2].lima references network “bridged” which is not defined in networks.yaml]

And the networks.yaml can’t be edited, as it’s overwritten each time Colima starts.

So how is this supposed to work?

Yep I got exactly the same error when I was trying to start a colima VM with bridged network. Found this issue when I was googling for more information.

It looks like the code is always preparing the vmnet in “shared” mode rather than “bridged”. Finally, after some investigation, I got what I wanted with a local build after patching the code like this:

diff --git a/daemon/process/vmnet/vmnet.go b/daemon/process/vmnet/vmnet.go
index 12d4030..e66703d 100644
--- a/daemon/process/vmnet/vmnet.go
+++ b/daemon/process/vmnet/vmnet.go
@@ -75,10 +75,11 @@ func (*vmnetProcess) Start(ctx context.Context) error {
 	go func() {
 		// rootfully start the vmnet daemon
 		command := cli.CommandInteractive("sudo", BinaryPath,
-			"--vmnet-mode", "shared",
-			"--socket-group", "staff",
-			"--vmnet-gateway", NetGateway,
-			"--vmnet-dhcp-end", NetDHCPEnd,
+			"--vmnet-mode", "bridged",
+			"--vmnet-interface", "en0",
+//			"--socket-group", "staff",
+//			"--vmnet-gateway", NetGateway,
+//			"--vmnet-dhcp-end", NetDHCPEnd,
 			"--pidfile", pid,
 			socket,
 		)
diff --git a/embedded/network/networks.yaml b/embedded/network/networks.yaml
index abd37f1..20f1241 100644
--- a/embedded/network/networks.yaml
+++ b/embedded/network/networks.yaml
@@ -2,7 +2,6 @@
 # This file would be replaced by Colima on startup.
 
 networks:
-  user-v2:
-    mode: user-v2
-    gateway: 192.168.5.2
-    netmask: 255.255.255.0
+  bridged:
+    mode: bridged
+    interface: en0
diff --git a/embedded/network/sudo.txt b/embedded/network/sudo.txt
index 3fbd4ab..f8091fc 100644
--- a/embedded/network/sudo.txt
+++ b/embedded/network/sudo.txt
@@ -1,5 +1,5 @@
 # starting vmnet daemon
-%staff ALL=(root:wheel) NOPASSWD:NOSETENV: /opt/colima/bin/socket_vmnet --vmnet-mode shared --socket-group staff --vmnet-gateway 192.168.106.1 --vmnet-dhcp-end 192.168.106.254 *
+%staff ALL=(root:wheel) NOPASSWD:NOSETENV: /opt/colima/bin/socket_vmnet --vmnet-mode bridged *
 # terminating vmnet daemon
 %staff ALL=(root:wheel) NOPASSWD:NOSETENV: /usr/bin/pkill -F /opt/colima/run/*.pid
 # validating vmnet daemon

The patch is definitely hacky thus far from a mature PR. However, I suppose it’s a good start for proof-of-concept and further discussions regarding this feature that some people are looking for 😄 @abiosoft

This is the closest issue I can find related to what I’m trying to achieve setting up a bridged network mode.

I have yet to see how to configure colima to start up the VM using my own network dhcp so I can get a real IP off my network. When I do try to add the network override.yaml for lima, I can start colima fine, but my network override in lima does not setup a bridge the way I’d expect it to.

My override is simple:

networks:
   - lima: bridged

Is this even possible? I’ve googled this many times and have found little information on this topic. Any help is much appreciated.