podman: [macos] `podman machine start` on Apple M1 uses qemu with hvf which is not yet supported by qemu on aarch64

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

podman machine start and the other podman machine commands use not supported qemu accelerator hvf on aarch64/darwin (MacOS 11.4 on M1).

podman/pkg/machine/qemu/options_darwin_arm64.go adds to the invocation of qemu --accel hvf.

The standard qemu built from brew formula did not contain hvf support, so I rebuilt qemu with configure option --enable-hvf which did not result in the expected outcome. Verifying the binary with qemu-system-aarch64 -accel help states only tcg supported. I reviewed the qemu source, and as it looks there is no implementation of hvf for aarch64 in qemu yet.

If I read it correctly the qemu code for target arm or aarch64 does not contain any calls to the Hypervisor Framework (hvf). See https://gitlab.com/qemu-project/qemu/-/tree/master/target/i386/hvf vs https://gitlab.com/qemu-project/qemu/-/tree/master/target/arm

Steps to reproduce the issue:

Create machine

# podman machine init                                                                              
Extracting compressed file

Verify the machine

# podman machine list                                                                                  
NAME                     VM TYPE     CREATED             LAST UP
podman-machine-default*  qemu        About a minute ago  About a minute ago

Start the machine

# podman machine start                                                                                         
Waiting for VM ...
qemu-system-aarch64: -accel hvf: invalid accelerator hvf
Error: dial unix /var/folders/cm/5dyqbf0s0ld37xyl34xyb_100000gn/T/podman/podman-machine-default_ready.sock: connect: connection refused

Describe the results you received:

qemu reports that hvf is an invalid accelerator.

Describe the results you expected:

Machine is started with the right qemu parameters.

Additional information you deem important (e.g. issue happens only occasionally):

Issue happens every time.

Output of podman version:

# podman --version                                                                                            
podman version 3.2.0

Output of podman info --debug:

# podman info --debug                                                                                          
Error: cannot connect to the Podman socket, please verify that Podman REST API service is running: Get "http://d/v3.2.0/libpod/_ping": dial unix ///var/folders/cm/5dyqbf0s0ld37xyl34xyb_100000gn/T/podman-run--1/podman/podman.sock: connect: no such file or directory

Package info (e.g. output of rpm -q podman or apt list podman):

Output of brew list podman

/opt/homebrew/Cellar/podman/3.2.0/bin/podman
/opt/homebrew/Cellar/podman/3.2.0/etc/bash_completion.d/podman
/opt/homebrew/Cellar/podman/3.2.0/share/fish/vendor_completions.d/podman.fish
/opt/homebrew/Cellar/podman/3.2.0/share/man/ (158 files)
/opt/homebrew/Cellar/podman/3.2.0/share/zsh/site-functions/_podman

Output of brew info podman

podman: stable 3.2.0 (bottled)
Tool for managing OCI containers and pods
https://podman.io/
/opt/homebrew/Cellar/podman/3.2.0 (167 files, 29.3MB) *
  Poured from bottle on 2021-06-06 at 09:38:46
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/podman.rb
License: Apache-2.0
==> Dependencies
Build: go ✘, go-md2man ✘
==> Caveats
fish completions have been installed to:
  /opt/homebrew/share/fish/vendor_completions.d
==> Analytics
install: 2,316 (30 days), 9,193 (90 days), 26,804 (365 days)
install-on-request: 2,315 (30 days), 9,190 (90 days), 26,365 (365 days)
build-error: 0 (30 days)

Output of brew list qemu

/opt/homebrew/Cellar/qemu/6.0.0/bin/elf2dmp
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-edid
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-img
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-io
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-nbd
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-storage-daemon
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-aarch64
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-alpha
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-arm
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-avr
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-cris
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-hppa
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-i386
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-m68k
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-microblaze
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-microblazeel
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-mips
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-mips64
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-mips64el
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-mipsel
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-moxie
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-nios2
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-or1k
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-ppc
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-ppc64
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-riscv32
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-riscv64
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-rx
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-s390x
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-sh4
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-sh4eb
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-sparc
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-sparc64
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-tricore
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-x86_64
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-xtensa
/opt/homebrew/Cellar/qemu/6.0.0/bin/qemu-system-xtensaeb
/opt/homebrew/Cellar/qemu/6.0.0/share/applications/qemu.desktop
/opt/homebrew/Cellar/qemu/6.0.0/share/icons/ (10 files)
/opt/homebrew/Cellar/qemu/6.0.0/share/qemu/ (107 files)

Output of brew info qemu

qemu: stable 6.0.0 (bottled), HEAD
Emulator for x86 and PowerPC
https://www.qemu.org/
/opt/homebrew/Cellar/qemu/6.0.0 (161 files, 569.4MB) *
  Built from source on 2021-06-06 at 11:15:13
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/qemu.rb
License: GPL-2.0-only
==> Dependencies
Build: libtool ✔, meson ✔, ninja ✔, pkg-config ✔
Required: glib ✔, gnutls ✔, jpeg ✔, libpng ✔, libslirp ✔, libssh ✔, libusb ✔, lzo ✔, ncurses ✔, nettle ✔, pixman ✔, snappy ✔, vde ✔
==> Options
--HEAD
	Install HEAD version
==> Analytics
install: 14,340 (30 days), 40,554 (90 days), 122,362 (365 days)
install-on-request: 14,262 (30 days), 40,376 (90 days), 121,152 (365 days)
build-error: 0 (30 days)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Hardware: Mac Mini M1 16G ProductName: macOS ProductVersion: 11.4 BuildVersion: 20F71

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 9
Comments: 29 (18 by maintainers)

Commits related to this issue

TCG Accel fallback for Apple Silicon. Iss #10577 Cause qemu to fall back to using TCG acceleration when HVP acceleration is not available on Darwin Aarch64. Qemu prints a warning which it is desirab... — committed to jonpspri/podman by jonpspri 3 years ago
TCG Accel fallback for Apple Silicon. Iss #10577 Cause qemu to fall back to using TCG acceleration when HVP acceleration is not available on Darwin Aarch64. Qemu prints a warning which it is desirab... — committed to jonpspri/podman by jonpspri 3 years ago
TCG Accel fallback for Apple Silicon. Iss #10577 Cause qemu to fall back to using TCG acceleration when HVP acceleration is not available on Darwin Aarch64. Qemu prints a warning which it is desirab... — committed to jonpspri/podman by jonpspri 3 years ago
TCG Accel fallback for Apple Silicon. Iss #10577 Cause qemu to fall back to using TCG acceleration when HVP acceleration is not available on Darwin Aarch64. Qemu prints a warning which it is desirab... — committed to jonpspri/podman by jonpspri 3 years ago
Merge pull request #11451 from jonpspri/qemu-apple-silicon-hvf-tcg Qemu TCG Accel fallback for Apple Silicon. Iss #10577 — committed to containers/podman by openshift-merge-robot 3 years ago
Merge pull request #11451 from jonpspri/qemu-apple-silicon-hvf-tcg Qemu TCG Accel fallback for Apple Silicon. Iss #10577 Signed-off-by: Aditya Rajan <arajan@redhat.com> — committed to flouthoc/podman by openshift-merge-robot 3 years ago
TCG Accel fallback for Apple Silicon. Iss #10577 Cause qemu to fall back to using TCG acceleration when HVP acceleration is not available on Darwin Aarch64. Qemu prints a warning which it is desirab... — committed to mheon/libpod by jonpspri 3 years ago
podman: `podman-machine` must have support for M1 See https://github.com/containers/podman/issues/10577 — committed to macports/macports-ports by judaew 3 years ago

Most upvoted comments

Before 3.3.1 I could start the machine with fixing the -accel tcg. Now any podman machine command says:

Error: due to missing upstream patches, Apple Silicon is not capable of running Podman machine yet

I think it would be better running with less optimal acceleration than tell you cannot do it at all.

+13

thomasw64 on Sep 2, 2021

With the change in Docker’s pricing today this feature becomes a lot more desired! It would be great to get this in the near future. Thank you

+13

lpil on Sep 1, 2021

FYI, I have patched and packed both qemu and podman due to this issue. You can check this out with the command below:

brew install simnalamburt/x/podman-apple-silicon

# Example
podman machine init
podman machine start
podman run -p 8080:80 docker.io/nginx

References

https://github.com/simnalamburt/homebrew-x/

+10

simnalamburt on Sep 5, 2021

Closing as we now have support for M1 via Homebrew’s Qemu

ashley-cui on Oct 12, 2021

@lpil unfortunately,we are subject to the merging of qemu patches upstream. once that is done, we will have it all set.

baude on Sep 1, 2021

@simnalamburt I looked at #11449 . I think we should add looking for the BIOS image in multiple locations into that PR as well (see my tree); however, to workaround this particular issue we may wish to add a separate PR for having qemu fallback to TCG acceleration if HVF acceleration is not available. That’s a one-liner I can set up in a separate PR.

[Note: I created PR #11453 to address the BIOS FD issue, since it’s very narrowly focused. The gvproxy conversation seems much more complicated.]

jonpspri on Sep 5, 2021

@ashley-cui not sure if this is in the scope of this issue, but should it be possible to run and build linux/amd64 with podman on MacOS via brew’s qemu, or is this a feature possibly coming at a later date?

fourstepper on Oct 13, 2021

Hi @simnalamburt . My podman patch is in https://github.com/jonpspri/podman/tree/homebrew.

I did what I could to make it backwards-compatible. I was working from head rather than 3.3.1, but I don’t think that’ll make much difference on a diff. Feel free to grab as much as you like and work it into your PR, since you’ve already started it. I’ll go pull and build your qemu and re-test on my hardware.

jonpspri on Sep 5, 2021

stay tuned, I will have instructions for you shortly.

baude on Jun 7, 2021

This is expected - we’re waiting on a few more patches to merge upstream in Qemu for this to be supported. podman machine is still in heavy development and very much experimental - we hope to have things more polished and ready in the next few releases.

The Podman issue may be my fault, I’ll bring it up at scrum on Monday. 3.1.2 and 3.2.0 should be able to talk to each other. We are needing to put out a Podman 3.2.1 already for another bug, so fixing this as the same time is probably not a big deal.

mheon on Jun 6, 2021

I was able to start the machine, by changing -accel hvf to -accel tcg in .config/containers/podman/machine/qemu/podman-machine-default.json

...
  "-device",
  "virtserialport,chardev=podman-machine-default_ready,name=org.fedoraproject.port.0",
  "-accel",
  "tcg",
  "-cpu",
  "cortex-a57",
...

output of podman machine start

Waiting for VM ...
qemu-system-aarch64: -accel tcg: qemu_mprotect__osdep: mprotect failed: Permission denied

output of podman machine list

NAME                     VM TYPE     CREATED        LAST UP
podman-machine-default*  qemu        2 minutes ago  Currently running

Now, it looks, I found another issue: The coreos image contains (34.20210601.dev.0) podman 3.1.2 instead of 3.2.0.

thomasw64 on Jun 6, 2021