colima: "FATA[0028] error starting kubernetes" after Mac OS 13.3 update
Description
When updating to the latest Mac OS 13.3 I cannot create a colima environment with k3s.
Getting:
TRAC[0101] cmd ["lima" "kubectl" "cluster-info"]
TRAC[0103] cmd ["lima" "kubectl" "cluster-info"]
TRAC[0105] cmd ["lima" "kubectl" "cluster-info"]
TRAC[0107] cmd ["lima" "kubectl" "cluster-info"]
TRAC[0109] cmd ["lima" "kubectl" "cluster-info"]
TRAC[0111] cmd ["lima" "kubectl" "cluster-info"]
FATA[0111] error starting kubernetes: error running [lima kubectl cluster-info], output: "The connection to the server localhost:8080 was refused - did you specify the right host or port?", err: "exit status 1"
Tested on another Mac M1 following upgrade to 13.3 and same issue.
It looks like the network “col0” is not created.
When ssh into colima profile I get:
colima-test:/var/log# tail -5 k3s.log
time="2023-04-03T00:13:01Z" level=fatal msg="invalid node-ip: invalid ip format ''"
time="2023-04-03T00:13:07Z" level=warning msg="unable to get global unicast ip from interface name: can't find ip for interface col0"
time="2023-04-03T00:13:07Z" level=fatal msg="invalid node-ip: invalid ip format ''"
time="2023-04-03T00:13:12Z" level=warning msg="unable to get global unicast ip from interface name: can't find ip for interface col0"
time="2023-04-03T00:13:12Z" level=fatal msg="invalid node-ip: invalid ip format ''"
ifconfig output:
colima-test:/var/log# ifconfig
col0 Link encap:Ethernet HWaddr 52:55:55:EA:D9:D8
inet6 addr: fe80::5055:55ff:feea:d9d8/64 Scope:Link
inet6 addr: fde2:fcb5:a448:d160:5055:55ff:feea:d9d8/64 Scope:Global
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:97 errors:0 dropped:0 overruns:0 frame:0
TX packets:182 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:15146 (14.7 KiB) TX bytes:58660 (57.2 KiB)
docker0 Link encap:Ethernet HWaddr 02:42:0E:70:BE:A6
inet addr:172.17.0.1 Bcast:172.17.255.255 Mask:255.255.0.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
eth0 Link encap:Ethernet HWaddr 52:55:55:B3:2A:E7
inet addr:192.168.5.15 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fec0::5055:55ff:feb3:2ae7/64 Scope:Site
inet6 addr: fe80::5055:55ff:feb3:2ae7/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1033 errors:0 dropped:0 overruns:0 frame:0
TX packets:883 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:104085 (101.6 KiB) TX bytes:162025 (158.2 KiB)
eth2 Link encap:Ethernet HWaddr 5A:94:EF:FA:43:FC
inet addr:192.168.107.2 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::5894:efff:fefa:43fc/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:1076 (1.0 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:460 errors:0 dropped:0 overruns:0 frame:0
TX packets:460 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:27600 (26.9 KiB) TX bytes:27600 (26.9 KiB)
Example create command specified:
$ colima --profile test start \
--cpu 4 \
--memory 8 \
--disk 10 \
--with-kubernetes \
--kubernetes-disable="traefik" \
--kubernetes-version="v1.24.3+k3s1" \
--network-address \
--ssh-agent \
--mount-type 9p \
--very-verbose
TRAC[0000] cmd ["limactl" "info"]
TRAC[0000] cmd ["limactl" "list" "colima-test" "--json"]
TRAC[0000] error retrieving running instance: instance 'colima [profile=test]' does not exist
INFO[0000] starting colima [profile=test]
INFO[0000] runtime: docker+k3s
INFO[0000] preparing network ... context=vm
TRAC[0000] cmd ["/opt/homebrew/bin/colima" "daemon" "status" "test"]
TRAC[0000] cmd ["/opt/homebrew/bin/colima" "daemon" "start" "test" "--vmnet" "--gvproxy" "--verbose"]
TRAC[0000] cmd ["/opt/homebrew/bin/colima" "daemon" "status" "test"]
TRAC[0003] cmd ["/opt/homebrew/bin/colima" "daemon" "status" "test"]
INFO[0004] creating and starting ... context=vm
TRAC[0004] cmd ["limactl" "start" "--tty=false" "/var/folders/_p/2854n4xs4nd7cdhv4llpj5dr0000gn/T/colima-test.yaml"]
> Terminal is not available, proceeding without opening an editor
> `mountType: 9p` is experimental
> "Attempting to download the image from \"https://github.com/abiosoft/alpine-lima/releases/download/colima-v0.5.0-2/alpine-lima-clm-3.16.2-aarch64.iso\"" digest="sha512:06abfa8c9fd954f8bfe4ce226bf282dd06e9dfbcd09f57566bf6c20809beb5a3367415b515e0a65d6a1638ecfd3a3bb3fb6d654dee3d72164bd0279370448507"
> Using cache "/Users/aelsnz/Library/Caches/lima/download/by-url-sha256/c37acb6308026b2fe12f6c0ef3371f690b3e33ee6b5d37d5dc68684f8fd5ee52/data"
> [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/aelsnz/.lima/colima-test/serial.log")
> SSH Local Port: 49956
> [hostagent] Waiting for the essential requirement 1 of 3: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 3: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 3: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 3: "ssh"
> [hostagent] The essential requirement 1 of 3 is satisfied
> [hostagent] Waiting for the essential requirement 2 of 3: "user session is ready for ssh"
> [hostagent] The essential requirement 2 of 3 is satisfied
> [hostagent] Waiting for the essential requirement 3 of 3: "the guest agent to be running"
> [hostagent] The essential requirement 3 of 3 is satisfied
> [hostagent] Waiting for the final requirement 1 of 1: "boot scripts must have finished"
> [hostagent] Forwarding "/var/run/docker.sock" (guest) to "/Users/aelsnz/.colima/test/docker.sock" (host)
> [hostagent] Forwarding "/run/lima-guestagent.sock" (guest) to "/Users/aelsnz/.lima/colima-test/ga.sock" (host)
> [hostagent] The final requirement 1 of 1 is satisfied
> READY. Run `limactl shell colima-test` to open the shell.
TRAC[0060] cmd ["lima" "sudo" "cat" "/etc/hosts"]
TRAC[0060] cmd ["lima" "sudo" "sh" "-c" "echo -e \"192.168.5.2\\thost.docker.internal\" >> /etc/hosts"]
TRAC[0060] cmd ["lima" "sudo" "cat" "/etc/hosts"]
TRAC[0060] cmd ["lima" "sudo" "sh" "-c" "echo -e \"127.0.0.1\\tcolima-test\" >> /etc/hosts"]
INFO[0060] provisioning ... context=docker
TRAC[0060] cmd ["lima" "sudo" "mkdir" "-p" "/etc/docker"]
TRAC[0060] cmd int ["lima" "sudo" "sh" "-c" "cat > /etc/docker/daemon.json"]
TRAC[0060] cmd ["docker" "context" "inspect" "colima-test"]
TRAC[0061] cmd ["docker" "context" "create" "colima-test" "--description" "colima [profile=test]" "--docker" "host=unix:///Users/aelsnz/.colima/test/docker.sock"]
> colima-test
> Successfully created context "colima-test"
TRAC[0061] cmd ["docker" "context" "use" "colima-test"]
> colima-test
> Current context is now "colima-test"
INFO[0061] starting ... context=docker
TRAC[0061] cmd ["lima" "sudo" "service" "docker" "start"]
> * /var/log/docker.log: creating file
> * /var/log/docker.log: correcting owner
> * Starting Docker Daemon ... [ ok ]
TRAC[0061] cmd ["lima" "sudo" "docker" "info"]
TRAC[0066] cmd ["lima" "sudo" "docker" "info"]
INFO[0066] provisioning ... context=kubernetes
TRAC[0066] cmd ["lima" "sudo" "service" "k3s" "status"]
TRAC[0066] cmd ["lima" "k3s" "--version"]
TRAC[0066] cmd ["lima" "command" "-v" "k3s-uninstall.sh"]
TRAC[0066] cmd ["lima" "uname" "-m"]
TRAC[0066] cmd ["lima" "uname" "-m"]
TRAC[0066] cmd ["limactl" "list" "colima-test" "--json"]
TRAC[0066] cmd ["limactl" "shell" "colima-test" "sh" "-c" "ifconfig col0 | grep \"inet addr:\" | awk -F' ' '{print $2}' | awk -F':' '{print $2}'"]
INFO[0066] downloading and installing ... context=kubernetes
TRAC[0066] cmd ["mkdir" "-p" "/Users/aelsnz/Library/Caches/colima/caches"]
TRAC[0067] cmd ["curl" "-Ls" "-o" "/dev/null" "-w" "%{url_effective}" "https://github.com/k3s-io/k3s/releases/download/v1.24.3+k3s1/k3s-arm64"]
TRAC[0070] cmd int ["curl" "-L" "-#" "-C" "-" "-o" "/Users/aelsnz/Library/Caches/colima/caches/eb2016fa10b36ab581c3d1d67992dd6d834d7af183891589917884f3154a8932.downloading" "https://objects.githubusercontent.com/github-production-release-asset-2e65be/135516270/05777c66-c6fd-483e-90c0-b67565483532?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230402%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230402T235117Z&X-Amz-Expires=300&X-Amz-Signature=6776d3cca2fca4b280e494145acc8e4aa54e6061120e91038150286618c2246e&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=135516270&response-content-disposition=attachment%3B%20filename%3Dk3s-arm64&response-content-type=application%2Foctet-stream"]
TRAC[0071] cmd ["mv" "/Users/aelsnz/Library/Caches/colima/caches/eb2016fa10b36ab581c3d1d67992dd6d834d7af183891589917884f3154a8932.downloading" "/Users/aelsnz/Library/Caches/colima/caches/eb2016fa10b36ab581c3d1d67992dd6d834d7af183891589917884f3154a8932"]
TRAC[0071] cmd ["lima" "cp" "/Users/aelsnz/Library/Caches/colima/caches/eb2016fa10b36ab581c3d1d67992dd6d834d7af183891589917884f3154a8932" "/tmp/k3s"]
TRAC[0072] cmd ["lima" "sudo" "install" "/tmp/k3s" "/usr/local/bin/k3s"]
TRAC[0072] cmd ["mkdir" "-p" "/Users/aelsnz/Library/Caches/colima/caches"]
TRAC[0072] cmd ["curl" "-Ls" "-o" "/dev/null" "-w" "%{url_effective}" "https://github.com/k3s-io/k3s/releases/download/v1.24.3+k3s1/k3s-airgap-images-arm64.tar.gz"]
TRAC[0077] cmd int ["curl" "-L" "-#" "-C" "-" "-o" "/Users/aelsnz/Library/Caches/colima/caches/135ef324027650ffc8b0ffad1ccfe1130afca2e32067ab1caed4b76152f4807e.downloading" "https://objects.githubusercontent.com/github-production-release-asset-2e65be/135516270/f6c169a8-26cf-4e1c-bd48-11aeddd59190?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230402%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230402T235122Z&X-Amz-Expires=300&X-Amz-Signature=21174a29bf38565594ac08e7a1c5e1210a44384a2a2b7edecb9b8f6a91823123&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=135516270&response-content-disposition=attachment%3B%20filename%3Dk3s-airgap-images-arm64.tar.gz&response-content-type=application%2Foctet-stream"]
TRAC[0080] cmd ["mv" "/Users/aelsnz/Library/Caches/colima/caches/135ef324027650ffc8b0ffad1ccfe1130afca2e32067ab1caed4b76152f4807e.downloading" "/Users/aelsnz/Library/Caches/colima/caches/135ef324027650ffc8b0ffad1ccfe1130afca2e32067ab1caed4b76152f4807e"]
TRAC[0080] cmd ["lima" "cp" "/Users/aelsnz/Library/Caches/colima/caches/135ef324027650ffc8b0ffad1ccfe1130afca2e32067ab1caed4b76152f4807e" "/tmp/k3s-airgap-images-arm64.tar.gz"]
TRAC[0080] cmd ["lima" "gzip" "-f" "-d" "/tmp/k3s-airgap-images-arm64.tar.gz"]
TRAC[0084] cmd ["lima" "sudo" "mkdir" "-p" "/var/lib/rancher/k3s/agent/images/"]
TRAC[0084] cmd ["lima" "sudo" "cp" "/tmp/k3s-airgap-images-arm64.tar" "/var/lib/rancher/k3s/agent/images/"]
INFO[0085] loading oci images ... context=kubernetes
TRAC[0085] cmd ["lima" "sudo" "docker" "load" "-i" "/tmp/k3s-airgap-images-arm64.tar"]
> Loaded image: rancher/local-path-provisioner:v0.0.21
> Loaded image: rancher/mirrored-coredns-coredns:1.9.1
> Loaded image: rancher/mirrored-library-busybox:1.34.1
> Loaded image: rancher/mirrored-library-traefik:2.6.2
> Loaded image: rancher/mirrored-metrics-server:v0.5.2
> Loaded image: rancher/mirrored-pause:3.6
> Loaded image: rancher/klipper-helm:v0.7.3-build20220613
> Loaded image: rancher/klipper-lb:v0.3.5
TRAC[0087] cmd ["mkdir" "-p" "/Users/aelsnz/Library/Caches/colima/caches"]
TRAC[0087] cmd ["curl" "-Ls" "-o" "/dev/null" "-w" "%{url_effective}" "https://raw.githubusercontent.com/k3s-io/k3s/v1.24.3+k3s1/install.sh"]
TRAC[0088] cmd int ["curl" "-L" "-#" "-C" "-" "-o" "/Users/aelsnz/Library/Caches/colima/caches/bcf7fa2c74a2bf7c4ac7d9e3b01762b677bb69d580bd277e46c5e28f729c28f7.downloading" "https://raw.githubusercontent.com/k3s-io/k3s/v1.24.3+k3s1/install.sh"]
TRAC[0088] cmd ["mv" "/Users/aelsnz/Library/Caches/colima/caches/bcf7fa2c74a2bf7c4ac7d9e3b01762b677bb69d580bd277e46c5e28f729c28f7.downloading" "/Users/aelsnz/Library/Caches/colima/caches/bcf7fa2c74a2bf7c4ac7d9e3b01762b677bb69d580bd277e46c5e28f729c28f7"]
TRAC[0088] cmd ["lima" "cp" "/Users/aelsnz/Library/Caches/colima/caches/bcf7fa2c74a2bf7c4ac7d9e3b01762b677bb69d580bd277e46c5e28f729c28f7" "/tmp/k3s-install.sh"]
TRAC[0088] cmd ["lima" "sudo" "install" "/tmp/k3s-install.sh" "/usr/local/bin/k3s-install.sh"]
TRAC[0088] cmd ["lima" "sh" "-c" "INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_SKIP_ENABLE=true k3s-install.sh --write-kubeconfig-mode 644 --resolv-conf /etc/resolv.conf --disable traefik --bind-address --advertise-address --flannel-iface col0 --container-runtime-endpoint unix:///run/cri-dockerd.sock"]
> [INFO] Skipping k3s download and verify
> [INFO] Skipping installation of SELinux RPM
> [INFO] Creating /usr/local/bin/kubectl symlink to k3s
> [INFO] Creating /usr/local/bin/crictl symlink to k3s
> [INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
> [INFO] Creating killall script /usr/local/bin/k3s-killall.sh
> [INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
> [INFO] env: Creating environment file /etc/rancher/k3s/k3s.env
> [INFO] openrc: Creating service file /etc/init.d/k3s
TRAC[0089] cmd ["lima" "sudo" "mkdir" "-p" "/etc/cni/net.d"]
TRAC[0089] cmd ["lima" "sudo" "mkdir" "-p" "/etc/cni/net.d"]
TRAC[0089] cmd int ["lima" "sudo" "sh" "-c" "cat > /etc/cni/net.d/10-flannel.conflist"]
TRAC[0089] cmd ["lima" "sudo" "service" "cri-dockerd" "start"]
> * Caching service dependencies ... [ ok ]
> * Starting cri-dockerd ... [ ok ]
TRAC[0089] cmd ["lima" "sudo" "cat" "/etc/colima/colima.json"]
TRAC[0089] cmd ["lima" "sudo" "mkdir" "-p" "/etc/colima"]
TRAC[0089] cmd ["lima" "sudo" "mkdir" "-p" "/etc/colima"]
TRAC[0089] cmd int ["lima" "sudo" "sh" "-c" "cat > /etc/colima/colima.json"]
INFO[0089] starting ... context=kubernetes
TRAC[0089] cmd ["lima" "sudo" "service" "k3s" "status"]
TRAC[0089] cmd ["lima" "sudo" "service" "k3s" "start"]
> * Starting k3s ... [ ok ]
TRAC[0089] cmd ["lima" "kubectl" "cluster-info"]
TRAC[0092] cmd ["lima" "kubectl" "cluster-info"]
TRAC[0094] cmd ["lima" "kubectl" "cluster-info"]
TRAC[0096] cmd ["lima" "kubectl" "cluster-info"]
TRAC[0098] cmd ["lima" "kubectl" "cluster-info"]
TRAC[0101] cmd ["lima" "kubectl" "cluster-info"]
TRAC[0103] cmd ["lima" "kubectl" "cluster-info"]
TRAC[0105] cmd ["lima" "kubectl" "cluster-info"]
TRAC[0107] cmd ["lima" "kubectl" "cluster-info"]
TRAC[0109] cmd ["lima" "kubectl" "cluster-info"]
TRAC[0111] cmd ["lima" "kubectl" "cluster-info"]
FATA[0111] error starting kubernetes: error running [lima kubectl cluster-info], output: "The connection to the server localhost:8080 was refused - did you specify the right host or port?", err: "exit status 1"
Version
Colima Version: 0.5.4 Lima Version: 0.15.0 Qemu Version: 7.2.0
Operating System
- macOS Intel <= 12 (Monterrey)
- macOS Intel >= 13 (Ventura)
- macOS M1 <= 12 (Monterrey)
- macOS M1 >= 13 (Ventura)
- Linux
Output of colima status
$ colima list
PROFILE STATUS ARCH CPUS MEMORY DISK RUNTIME ADDRESS
test Running aarch64 4 8GiB 10GiB docker+k3s
$ colima -p test status
FATA[0000] error retrieving current runtime: empty value
Reproduction Steps
- Update MacOS to 13.3
- Start new colima profile with kubernetes enabled
$ colima --profile test start \
--cpu 4 \
--memory 8 \
--disk 10 \
--with-kubernetes \
--kubernetes-disable="traefik" \
--kubernetes-version="v1.24.3+k3s1" \
--network-address \
--ssh-agent \
--mount-type 9p \
--very-verbose
- Error will be reported:
FATA[0111] error starting kubernetes: error running [lima kubectl cluster-info], output: "The connection to the server localhost:8080 was refused - did you specify the right host or port?", err: "exit status 1"
Expected behaviour
Should get a deployed k3s environment. Looks like issue with the col0 network not being created.
Additional context
No response
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 20 (17 by maintainers)
Quick update on last comment, I think this might help to have option to pass in static IP to be used. To test the concept I modified
https://github.com/abiosoft/colima/blob/main/embedded/network/ifaces.shscript to see if an environment variable COLIMA_IP as example is set, if it is, it will use it and update the network interfacecol0to use static ip using this environment variable. Not perfect way, but when I then run a test like this it works:Busy testing this more and will provide more feedback once done.
Ok, quick update, I saw this in the k3s log
which made me think that IPV6 is causing the issues here. If I look at ifconfig on the VM I saw:
Note there is the inet6 addr…
Under Mac, I went to my network, selected the interface, goto TCP/IP settings and set “Configure IPv6” to “Link-Local Only” - reboot and it is now working!
In my use case I ideally need the loadBalancer IP and using localhost is not ideal - so this is good to get the address.
I am not expert in IPv6, but feels like maybe the priority above ipv4 could have played a role here.
Now the question is can we adjust the underlying VM to only use IPv4 or to prioritise IPv4 or disable IPv6 maybe.
Would be interesting if others see this - I will get test on another setup and report back.