colima: Colima doesn't start after OS update, fails with Waiting for the essential requirement 1 of 5: "ssh"

Description

After upgrading my mac to Ventura, colima start fails with below error.

`INFO[0000] starting colima INFO[0000] runtime: docker INFO[0000] preparing network … context=vm INFO[0000] starting … context=vm

[hostagent] Waiting for the essential requirement 1 of 5: “ssh” [hostagent] Waiting for the essential requirement 1 of 5: “ssh” [hostagent] Waiting for the essential requirement 1 of 5: “ssh” [hostagent] Waiting for the essential requirement 1 of 5: “ssh” [hostagent] Waiting for the essential requirement 1 of 5: “ssh” did not receive an event with the “running” status FATA[0600] error starting vm: error at ‘starting’: exit status 1`

`colima list

PROFILE STATUS ARCH CPUS MEMORY DISK RUNTIME ADDRESS

default Running x86_64 2 4GiB 60GiB

delete Stopped x86_64 8 22GiB 60GiB

list Stopped x86_64 8 22GiB 60GiB`

Below is the error in stderr.log

{"level":"debug","msg":"Creating iso file /Users/test/.lima/colima/cidata.iso","time":"2023-02-21T16:48:52+01:00"} {"level":"debug","msg":"Using /var/folders/wt/f_t9xy6s0f173r1qyfw27m_c0000gr/T/diskfs_iso1691064567 as workspace","time":"2023-02-21T16:48:52+01:00"} {"level":"debug","msg":"OpenSSH version 9.0.1 detected","time":"2023-02-21T16:48:52+01:00"} {"level":"debug","msg":"AES accelerator seems available, prioritizing openssh.com","time":"2023-02-21T16:48:52+01:00"} {"level":"debug","msg":"firmware candidates = [/Users/test/.local/share/qemu/edk2-x86_64-code.fd /Users/test/.colima/_wrapper/***/share/qemu/edk2-x86_64-code.fd /usr/share/OVMF/OVMF_CODE.fd /usr/share/qemu/ovmf-x86_64-code.bin /usr/share/edk2-ovmf/x64/OVMF_CODE.fd]","time":"2023-02-21T16:48:53+01:00"} {"level":"info","msg":"Starting QEMU (hint: to watch the boot progress, see \"/Users/test/.lima/colima/serial.log\")","time":"2023-02-21T16:48:53+01:00"} {"level":"debug","msg":"qCmd.Args: [/Users/test/.colima/_wrapper/***/bin/qemu-system-x86_64 -m 4096 -cpu qemu64 -machine q35,vmport=off -accel tcg,thread=multi,tb-size=512 -global ICH9-LPC.disable_s3=1 -smp 2,sockets=1,cores=2,threads=1 -drive if=pflash,format=raw,readonly=on,file=/Users/test/.colima/_wrapper/***/edk2-x86_64-code.fd -boot order=d,splash-time=0,menu=on -drive file=/Users/test/.lima/colima/basedisk,media=cdrom,readonly=on -drive file=/Users/test/.lima/colima/diffdisk,if=virtio,discard=on -cdrom /Users/test/.lima/colima/cidata.iso -netdev user,id=net0,net=192.168.5.0/24,dhcpstart=192.168.5.15,hostfwd=tcp:127.0.0.1:51407-:22 -device virtio-net-pci,netdev=net0,mac=*** -device virtio-rng-pci -display none -device virtio-vga -device virtio-keyboard-pci -device virtio-mouse-pci -device qemu-xhci,id=usb-bus -parallel none -chardev socket,id=char-serial,path=/Users/test/.lima/colima/serial.sock,server=on,wait=off,logfile=/Users/test/.lima/colima/serial.log -serial chardev:char-serial -chardev socket,id=char-qmp,path=/Users/test/.lima/colima/qmp.sock,server=on,wait=off -qmp chardev:char-qmp -name lima-colima -pidfile /Users/test/.lima/colima/qemu.pid]","time":"2023-02-21T16:48:53+01:00"} {"level":"info","msg":"Waiting for the essential requirement 1 of 5: \"ssh\"","time":"2023-02-21T16:48:53+01:00"} {"level":"debug","msg":"executing script \"ssh\"","time":"2023-02-21T16:48:53+01:00"} {"level":"debug","msg":"executing ssh for script \"ssh\": /usr/bin/ssh [ssh -F /dev/null -o IdentityFile=\"/Users/test/.lima/_config/user\" -o IdentityFile=\"/Users/test/.ssh/id_rsa\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers=\"***" -o User=test -o ControlMaster=auto -o ControlPath=\"/Users/test/.lima/colima/ssh.sock\" -o ControlPersist=5m -p 51407 127.0.0.1 -- /bin/bash]","time":"2023-02-21T16:48:53+01:00"} {"level":"debug","msg":"stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 51407: Connection refused\\r\\n\", err=failed to execute script \"ssh\": stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 51407: Connection refused\\r\\n\": exit status 255","time":"2023-02-21T16:48:53+01:00"} {"level":"debug","msg":"qemu[stderr]: qemu-system-x86_64: -accel tcg,thread=multi,tb-size=512: warning: Guest expects a stronger memory ordering than the host provides","time":"2023-02-21T16:48:53+01:00"} {"level":"debug","msg":"qemu[stderr]: This may cause strange/hard to debug errors","time":"2023-02-21T16:48:53+01:00"} {"level":"info","msg":"Waiting for the essential requirement 1 of 5: \"ssh\"","time":"2023-02-21T16:49:03+01:00"} {"level":"debug","msg":"executing script \"ssh\"","time":"2023-02-21T16:49:03+01:00"} {"level":"debug","msg":"executing ssh for script \"ssh\": /usr/bin/ssh [ssh -F /dev/null -o IdentityFile=\"/Users/test/.lima/_config/user\" -o IdentityFile=\"/Users/test/.ssh/id_rsa\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers=\"^aes128-gcm@openssh.com,aes256-gcm@openssh.com\" -o User=staliyan001 -o ControlMaster=auto -o ControlPath=\"/Users/test/.lima/colima/ssh.sock\" -o ControlPersist=5m -p 51407 127.0.0.1 -- /bin/bash]","time":"2023-02-21T16:49:03+01:00"} {"level":"debug","msg":"stdout=\"\", stderr=\"kex_exchange_identification: read: Connection reset by peer\\r\\nConnection reset by 127.0.0.1 port 51407\\r\\n\", err=failed to execute script \"ssh\": stdout=\"\", stderr=\"kex_exchange_identification: read: Connection reset by peer\\r\\nConnection reset by 127.0.0.1 port 51407\\r\\n\": exit status 255","time":"2023-02-21T16:50:17+01:00"} {"level":"info","msg":"Waiting for the essential requirement 1 of 5: \"ssh\"","time":"2023-02-21T16:50:27+01:00"} {"level":"debug","msg":"executing script \"ssh\"","time":"2023-02-21T16:50:27+01:00"} {"level":"debug","msg":"executing ssh for script \"ssh\": /usr/bin/ssh [ssh -F /dev/null -o IdentityFile=\"/Users/test/.lima/_config/user\" -o IdentityFile=\"/Users/test/.ssh/id_rsa\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers=\"***" -o User=test -o ControlMaster=auto -o ControlPath=\"/Users/test/.lima/colima/ssh.sock\" -o ControlPersist=5m -p 51407 127.0.0.1 -- /bin/bash]","time":"2023-02-21T16:50:27+01:00"}

Version

Colima Version:0.5.2 Lima Version:0.14.2 Qemu Version:7.2.0

Operating System

  • macOS Intel <= 12 (Monterrey)
  • macOS Intel >= 13 (Ventura)
  • macOS M1 <= 12 (Monterrey)
  • macOS M1 >= 13 (Ventura)
  • Linux

Output of colima status

colima status FATA[0074] error retrieving current runtime: empty value

Reproduction Steps

1.colima start --cpu 8 --memory 22 --arch amd 2. 3.

Expected behaviour

Colima should start without errors and colima status should not throw any errors

Additional context

I tried reinstalling colima with --HEAD, still the same error. Also it starts with command colima start --arch aarch64, but obviously i get below error while starting my container. The requested image’s platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 3
  • Comments: 50 (9 by maintainers)

Most upvoted comments

I’ve also had the issue and found that it doesn’t happen if you delete any prior ‘profile’ in the ~/.lima folder. It should only contain the _config file, nothing else, and everything works smoothly. Here are the steps I followed: A. clearing the ‘profile’ folders in ~/.lima

  1. rm -rf ~/.colima (if folder exists)
  2. cd ~/.lima
  3. mv _config …/
  4. rm -rf *
  5. mv …/_config . B. Starting a fresh instance
  6. colima start --cpu 4 --memory 8 ( no need for the arch options )
  7. colima status (to check that it all works)

A new folder ~/.colima is created with configs and .sock.

Hope this help.

I had several issues setting up Colima.

Colima Version: 0.6.6 Docker Version: 24.0.7 OS: Mac M3 running Sonoma 14.1.1

Issue 1

Cannot run docker cli commands due to not being able to connect to Docker daemon. Error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Attempted Fix Which did NOT Work: Set environment variable DOCKER_HOST: export DOCKER_HOST="unix://$HOME/.colima/default/docker.sock"

Issue 2

Starting colima is stuck in Waiting for the essential requirement 1 of 4: "ssh" Error:

$ colima start
INFO[0000] starting colima
INFO[0000] runtime: docker
INFO[0001] creating and starting ...                     context=vm
> [hostagent] Starting QEMU (hint: to watch the boot progress, see "~/.colima/_lima/colima/serial*.log")
> SSH Local Port: 53987
> [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 5: "ssh"

The Fix

What eventually worked for me is:

  1. Delete default colima profile: colima delete default
  2. Delete ~/.colima: rm -rf ~/.colima
  3. Start colima with vm-type parameter: colima start --vm-type vz
  4. Colima status
    $ colima status
        INFO[0000] colima is running using macOS Virtualization.Framework
        INFO[0000] arch: aarch64
        INFO[0000] runtime: docker
        INFO[0000] mountType: virtiofs
        INFO[0000] socket: unix://$HOME/.colima/default/docker.sock
    

P.S.

I installed docker, docker-compose, docker-buildx using Homebrew BEFORE installing colima

This works for me:

Delete default colima profile: colima delete default Delete ~/.colima: rm -rf ~/.colima Start colima with vm-type parameter: colima start --vm-type vz

if you’d try to run amazon-linux-2023 docker image on arm64 macos you’ll get the error similar to this

$ docker run -i -t --rm --name al2023 --security-opt seccomp=unconfined public.ecr.aws/amazonlinux/amazonlinux:2023 /bin/bash
Unable to find image 'public.ecr.aws/amazonlinux/amazonlinux:2023' locally
2023: Pulling from amazonlinux/amazonlinux
bad342ce986c: Pull complete
Digest: sha256:6235e87e753667a84f0adb06e65ee19950d8335f4bf279932fe4f01aa4a4b3cf
Status: Downloaded newer image for public.ecr.aws/amazonlinux/amazonlinux:2023
Fatal glibc error: CPU does not support x86-64-v2

the only thing that helped me was this

$ export QEMU_SYSTEM_X86_64="qemu-system-x86_64 -cpu Haswell-v4"
$ colima start colima-intel --cpu 2 --memory 4 --disk 20 --arch x86_64 --vm-type=vz --vz-rosetta

I regularly experience the same issue on an M1 Mac (ARM arch.) with no emulation. Occassionally, deleting the colima profile colima delete default and then colima start is sometimes enough to get the SSH part to work. Typically though, I’ve found that removing ~/.lima and ~/.colima and then colima start again works but this might be a fluke. It’s a bit disappointing that I need to rebuild all my docker containers regularly. This problem also seems to happen every time I reboot my machine so might leave it running for a few days 😉.

Hopefully this helps someone in the meantime.

Edit: just to ask, do I need QEMU if I’m running m1 based images on colima? Curious to learn a bit more about how Colima works.

I had several issues setting up Colima.

Colima Version: 0.6.6 Docker Version: 24.0.7 OS: Mac M3 running Sonoma 14.1.1

Issue 1

Cannot run docker cli commands due to not being able to connect to Docker daemon. Error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Attempted Fix Which did NOT Work: Set environment variable DOCKER_HOST: export DOCKER_HOST="unix://$HOME/.colima/default/docker.sock"

Issue 2

Starting colima is stuck in Waiting for the essential requirement 1 of 4: "ssh" Error:

$ colima start
INFO[0000] starting colima
INFO[0000] runtime: docker
INFO[0001] creating and starting ...                     context=vm
> [hostagent] Starting QEMU (hint: to watch the boot progress, see "~/.colima/_lima/colima/serial*.log")
> SSH Local Port: 53987
> [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
> [hostagent] Waiting for the essential requirement 1 of 5: "ssh"

The Fix

What eventually worked for me is:

  1. Delete default colima profile: colima delete default
  2. Delete ~/.colima: rm -rf ~/.colima
  3. Start colima with vm-type parameter: colima start --vm-type vz
  4. Colima status
    $ colima status
        INFO[0000] colima is running using macOS Virtualization.Framework
        INFO[0000] arch: aarch64
        INFO[0000] runtime: docker
        INFO[0000] mountType: virtiofs
        INFO[0000] socket: unix://$HOME/.colima/default/docker.sock
    

P.S.

I installed docker, docker-compose, docker-buildx using Homebrew BEFORE installing colima

@ultimike have they tried stopping with the --force flag? Are they emulating x86 on m1 device?

colima stop --force

@tdy your issue seems to be something else, likely related to QEMU.

Can you try using the vz driver?

colima start --vm-type vz

Since Nov 22nd 2023, I have pinned v.0.6.0, and across all MacOS updates that version has worked perfectly.

I think it may be helpful for @abiosoft to take a look at what has changed after version 0.6.0, because it seems that’s when all of the problem started.

@tdy After the stop+delete, try also: rm -Rf ~/.colima ~/.lima folders, and then try starting again.

Hi @abiosoft I’m also affected by this.

I’m attempting to emulate a rockylinux:9.1-minimal container on x86_64 on an M1 laptop (aarch64). The container starts but fails on the first executable with:

Fatal glibc error: CPU does not support x86-64-v2

I didn’t dig into the details but some searching indicates --cpu-type=max fixes the issue. My full colima invocation is:

colima start --arch x86_64 --cpu-type max

As others have shared in this thread, tailing serial.log, the last few lines indicate 0 packages were installed due to apk throwing BAD signature followed by /sbin/init not found in new root. It then jumps into a root shell, and colima hangs forever waiting for ssh. If there are any logs or extra details I can share, let me know.

EDIT: The “x86-64-v2” error above is reproducible with the following command: docker run -it rockylinux:9.1-minimal /bin/bash