podman: Cannot start machine with podman version 4.1.0 in M1 Mac

In case it’s of any use, I’m finding that the max memory I’m able to set on a machine is 3072, otherwise I see the connection refused error.

✔️ The following works:

podman machine init # default memory is 2048
podman machine start

podman machine init -m 3072
podman machine start

podman machine init
podman machine set -m 3072
podman machine start

✖️ The following results in the connection refused error:

podman machine init -m 3073
podman machine start

podman machine init -m 4096
podman machine start

podman machine init
podman machine set -m 4096
podman machine start

I’m able to start the machine with memory greater than 3072 with the following changes to CmdLine in ~/.config/containers/podman/machine/qemu/<name>.json

• Set -cpu to host
• Set -M to virt (ie: remove highmem=off)

Tried a handful of other cpus listed by qemu-system-aarch64 -cpu help and of those I tried max also works.

This workaround is verified on macOS 12.4 but does result in a crash on macOS 12.3. Have not investigated what might work on these older versions.

I really don’t know what the implications of these changes are otherwise, any qemu savants in the house? =)

My workaround is to delete the VM and start from scratch

$ podman machine stop
$ podman machine rm --force
$ podman machine init --cpus 2 --memory 2048 --disk-size 20
$ podman machine start

version

$ podman --version
podman version 4.1.0
$ qemu-system-aarch64 --version 
QEMU emulator version 7.0.0

Highly recommend this solution if you are happy to throw away everything by the way… Although I do tend to do “–cpus $(sysctl -n hw.ncpu)”, to get all my CPUs in

For me the workaround by changing the memory size has no effect. I keep getting Error: dial unix /var/folders/bh/q062sd9d5yvcjqnv/T/podman/podman-machine-default_ready.sock: connect: connection refused I also tried to downgrade qemu to 6.2.0 Also tried to find the settings in the json but they are not there

I am on a MacBook Pro (16-inch, 2021) Apple M1 PRO

CPU setting of cortex-a57 is wrong. It should be host. Memory model should be himem=on and not highmem=off.

Podman managed machine JSON configuration file needs to be this:

 "-cpu",
  "host",
  "-M",
  "virt,highmem=on",

Instead of this:

  "-cpu",
  "cortex-a57",
  "-M",
  "virt,highmem=off",

Sorry, but it’s not QEMU that is the problem. It is the way that Podman is instructing QEMU what both the CPU architecture model and memory availability model looks like. Both of which are incorrect.

I can confirm that after downgrading the qemu to 6.2.0 I was able to start the machine and connect to it.

# =========================
# Versions

% podman --version
podman version 4.1.0

% qemu-system-aarch64 --version
QEMU emulator version 6.2.0

# =========================
# My VMs

% podman machine list           
NAME                     VM TYPE     CREATED      LAST UP      CPUS        MEMORY      DISK SIZE
opensearch               qemu        3 hours ago  3 hours ago  4           34.36GB     107.4GB
podman-machine-default*  qemu        3 hours ago  3 hours ago  1           2.147GB     10.74GB

# =========================
# Starting the machine

% podman machine start opensearch
Starting machine "opensearch"
Waiting for VM ...
Mounting volume... /Users/lukas.vlcek:/Users/lukas.vlcek

This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility
issues with non-podman clients, you can switch using the following command: 

	podman machine set --rootful opensearch

API forwarding listening on: /Users/lukas.vlcek/.local/share/containers/podman/machine/opensearch/podman.sock

The system helper service is not installed; the default Docker API socket
address can\'t be used by podman. If you would like to install it run the
following commands:

	sudo /opt/homebrew/Cellar/podman/4.1.0/bin/podman-mac-helper install
	podman machine stop opensearch; podman machine start opensearch

You can still connect Docker API clients by setting DOCKER_HOST using the
following command in your terminal session:

	export DOCKER_HOST='unix:///Users/lukas.vlcek/.local/share/containers/podman/machine/opensearch/podman.sock'

Machine "opensearch" started successfully

# =========================
# And finally, SSH into the machine

% podman machine ssh opensearch
Connecting to vm opensearch. To close connection, use `~.` or `exit`
Warning: Permanently added '[localhost]:56046' (ED25519) to the list of known hosts.
Fedora CoreOS 36.20220511.dev.0
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos

Note: But this is not to say that podman 4.1.0 will work correctly with qemu 6.2.0 !!!

Now, just a quick notes about how I downgraded the qemu. A great help was this article by @sandipb

# At this point I already have podman 4.1.0 installed with qemu 7.0.0

% brew tap-new lukas/local-qemu
% brew extract --version=6.2.0 qemu lukas/local-qemu
% brew uninstall --ignore-dependencies qemu
% brew install lukas/local-qemu/qemu@6.2.0

# Done.
# Notice that I did not use the HOMEBREW_NO_AUTO_UPDATE=1 parameter (see the article)
# so the qemu will probably be updated with the next `brew update` command...

My workaround is to delete the VM and start from scratch

$ podman machine stop
$ podman machine rm --force
$ podman machine init --cpus 2 --memory 2048 --disk-size 20
$ podman machine start

version

$ podman --version
podman version 4.1.0
$ qemu-system-aarch64 --version 
QEMU emulator version 7.0.0

I’ve tried running podman with increased memory: podman machine init -m 3072

I’ve tried to install Qemu 6.2.0 using:

curl -L -H "Authorization: Bearer QQ==" -o qemu-6.2.0_1.monterey.bottle.tar.gz https://ghcr.io/v2/homebrew/core/qemu/blobs/sha256:fcc3b1a8139f70dae57f5449f3856f9b3b67448ee0623e64da1e47dc255b46f6
brew install -f qemu-6.2.0_1.monterey.bottle.tar.gz

But both options did not resolve my errors.

➜  ~ podman machine start
Starting machine "podman-machine-default"
Waiting for VM ...
Error: dial unix /var/folders/l6/8c2s86493qx2w60cj0g846r00000gp/T/podman/podman-machine-default_ready.sock: connect: connection refused

 ~ podman info
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman. failed to create sshClient: connection to bastion host (ssh://core@localhost:50159/run/user/502/podman/podman.sock) failed: dial tcp 127.0.0.1:50159: connect: connection refused

Mac info: MacBook Pro (16-inch, 2021) Chip: Apple M1 Pro MacOS version: Monterey 12.4

Edit Okay, solved it. I had an x86 version of Homebrew installed on my Mac with a M1 chip, that caused a x86 version of Qemu to be installed. This happened because of me transferring my files from my old Mac (Intel chip) to my new one (M1 chip).

Run this bash script to migrate your Homebrew from Intel to ARM architecture: https://github.com/Homebrew/discussions/discussions/417#discussioncomment-2556937

Had the same issue with Qemu from homebrew. Downgraded to 6.2.0_1 and everything is ok. My workaround here https://github.com/Homebrew/homebrew-core/issues/102242#issuecomment-1137108389

EXCELLENT fix from Carlos Eduardo Gorges, on stackoverflow. Caused due to qemu 7.0.0 startup latency (3-5s). To fix … execute qemu, to initialize, before call “podman machine start”. Works for me … https://stackoverflow.com/questions/72404926/cant-get-podman-to-run-on-an-mac-m1-monterey

@shanesmith My hardware is an M1 Mac Mini with a mere 8GB of memory.

Total Cores:

$ sysctl -n hw.physicalcpu
8

Performance Cores:

sysctl -n hw.perflevel0.physicalcpu                   
4

Efficiency Cores:

sysctl -n hw.perflevel1.physicalcpu
4

It is seeming more like of an M1 Pro specific issue for some odd reason.

With just virt and no himem. Diff of ~/.config/containers/podman/machine/qemu/podman-machine-default.json:

28c28
<   "cortex-a57",
---
>   "host",
30c30
<   "virt,highmem=off",
---
>   "virt",
51c51
< }
\ No newline at end of file
---
> }

The VM starts up just fine:

$ podman machine start
Starting machine "podman-machine-default"
Waiting for VM ...
Mounting volume... /Users/cpolizzi:/Users/cpolizzi

This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility
issues with non-podman clients, you can switch using the following command: 

	podman machine set --rootful

API forwarding listening on: /Users/cpolizzi/.local/share/containers/podman/machine/podman-machine-default/podman.sock

The system helper service is not installed; the default Docker API socket
address can't be used by podman. If you would like to install it run the
following commands:

	sudo /opt/homebrew/Cellar/podman/4.1.0/bin/podman-mac-helper install
	podman machine stop; podman machine start

You can still connect Docker API clients by setting DOCKER_HOST using the
following command in your terminal session:

	export DOCKER_HOST='unix:///Users/cpolizzi/.local/share/containers/podman/machine/podman-machine-default/podman.sock'

Machine "podman-machine-default" started successfully

Listing verifies this:

$ podman machine list

podman machine list 
NAME                     VM TYPE     CREATED        LAST UP            CPUS        MEMORY      DISK SIZE
podman-machine-default*  qemu        5 minutes ago  Currently running  4           6.442GB     42.95GB

And further confirmed by connectivity:

$ podman machine ssh cat /etc/os-release | grep ^VERSION=
VERSION="36.20220511.dev.0 (CoreOS)"

Thanks @cpolizzi @shanesmith! I’ll open up a fix for this now.

@shanesmith legend!

editing ~/.config/containers/podman/machine/qemu/<podman machine name>.json upon machine creation, in the way described above, made things functional for me again on MacOS 12.4 with any desired amount of memory.

hope this gets addressed soon in a way which would make things work out of the box for everyone.

Hmm, maybe it looks like something changed in qemu 7.0? Taking a look…

Getting same issue. If I run in debug mode the following command is run which produces the error if run standalone:

/opt/homebrew/bin/qemu-system-aarch64 -m 27000 -smp 10 -fw_cfg name=opt/com.coreos/config,file=/Users/edwardnewman/.config/containers/podman/machine/qemu/podman-machine-default.ign -qmp unix://var/folders/tz/4n8rgr6n6wj5cgqzjx1dh0_80000gq/T/podman/qmp_podman-machine-default.sock,server=on,wait=off -netdev socket,id=vlan,fd=3 -device virtio-net-pci,netdev=vlan,mac=5a:94:ef:e4:0c:ee -device virtio-serial -chardev socket,path=/var/folders/tz/4n8rgr6n6wj5cgqzjx1dh0_80000gq/T/podman/podman-machine-default_ready.sock,server=on,wait=off,id=podman-machine-default_ready -device virtserialport,chardev=podman-machine-default_ready,name=org.fedoraproject.port.0 -accel hvf -accel tcg -cpu cortex-a57 -M virt,highmem=off -drive file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on -drive file=/Users/edwardnewman/.local/share/containers/podman/machine/qemu/podman-machine-default_ovmf_vars.fd,if=pflash,format=raw -virtfs local,path=/Users/edwardnewman,mount_tag=vol0,security_model=mapped-xattr -drive if=virtio,file=/Users/edwardnewman/.local/share/containers/podman/machine/qemu/podman-machine-default_fedora-coreos-36.20220511.dev.0-qemu.aarch64.qcow2 qemu-system-aarch64: -netdev socket,id=vlan,fd=3: can’t get socket option SO_TYPE

This is a problem with virtualizing memory in newer versions of Qemu (7.0+) on aarch64 machines. I resolved the issue by updating the ~/.config/containers/podman/machine/qemu/podman-machine-default.json file before starting the podman machine VM. Here are the steps that worked for me:

1. Create machine VM with podman machine init ( with -v $HOME:$HOME if you want to be able to mount volumes)
2. Update -M entry to virt,highmem=off in ~/.config/containers/podman/machine/qemu/podman-machine-default.json
3. Save file and run podman machine start

Today I had: podman/podman-machine-default_ready.sock: connect: connection refused

What worked for me:

1. podman machine stop
2. podman machine rm
3. podman machine init
4. podman machine start

Please upgrade your podman to 4.1.1 and make sure your macos version is 12.4

👋🏽 @ashley-cui and all, thanks for looking into this. I’m able to start the machine (albeit after a long time given I used 2 vCPUs, 2GB of memory and 20GB disk specs for the VM).

After some time debugging, I found the cause of this problem.

This problem is caused due to qemu 7.0.0 startup latency (3-5s) that occour in every first qemu execution after Mac Machine machine boots.

Podman has some bug that doesn’t expect that the creation of socks files, done by the qemu call, can be delayed some seconds, and when podman tries to access the socks files, the qemu is not created them yet, showing the error “Error: dial unix <TEMPDIR>/podman/podman-machine-default_ready.sock: connect: connection refused”.

To avoid this problem, just execute qemu, even with invalid options (just to initialize), before call “podman machine start”.

            echo "* Podman VM machine for MACOSX is stoped, starting..."
            
            # workaround - initialize qemu before machine start to avoid socket error
            /usr/local/bin/qemu-system-x86_64 -machine q35,accel=hvf:tcg -cpu host -display none INVALID_OPTION >> /dev/null 2>&1

            podman machine start podman-machine-default
            ECODE=$?;if [ $ECODE -ne 0 ];then echo "* Error starting podman linux vm machine: $ECODE";exit $ECODE;fi

I hope help. Carlos Eduardo Gorges.

Please upgrade your podman to 4.1.1 and make sure your macos version is 12.4

@e-minguez worked for me with json file editing

~ % sw_vers
ProductName:	macOS
ProductVersion:	12.4
BuildVersion:	21F79
~ % qemu-system-aarch64 --version
QEMU emulator version 7.0.0
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers
~ % podman --version
podman version 4.1.0
~ % podman machine list
NAME                     VM TYPE     CREATED      LAST UP            CPUS        MEMORY      DISK SIZE
podman-machine-default*  qemu        2 weeks ago  Currently running  8           6.442GB     68.72GB

Pointing to the source, where these defaults are defined https://github.com/containers/podman/blob/3c75c4a54cba6a1949e53c4386447120b7ddb5a9/pkg/machine/qemu/options_darwin_arm64.go#L18

@lukas-vlcek I followed the same steps to downgrade (used this article). I think your qemu will not update on the next brew update, because you have qemu@6.20 installed instead of qemu.

Also running into this, and I saw the same qemu-system-aarch64: -netdev socket,id=vlan,fd=3: can't get socket option SO_TYPE error message as @nycnewman. After downgrading qemu, I was able to run podman.

Experiencing similar(?) issue. M1 Mac.

a@M1-Air ~ $ podman machine start
Starting machine "podman-machine-default"
Waiting for VM ...
Error: dial unix /var/folders/1d/ts5cfq0d5sqfx5h9k7vkh2000000gn/T/podman/podman-machine-default_ready.sock: connect: connection refused

a@M1-Air ~ $ podman info --debug
Cannot connect to Podman....

a@M1-Air ~ $ brew info podman
podman: stable 4.1.0 (bottled), HEAD
Tool for managing OCI containers and pods
https://podman.io/
/usr/local/Cellar/podman/HEAD-948c5e9 (174 files, 47.8MB) *
  Built from source on 2022-05-19 at 14:15:13
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/podman.rb
License: Apache-2.0
==> Dependencies
Build: go ✔, go-md2man ✔
Required: qemu ✔
==> Options
--HEAD
	Install HEAD version