quarkus: quarkus-integration-test-container-image-invoker tests do not pass on M1 mac/with a docker equivalent (podman, minikube, etc)

Describe the bug

The quarkus-integration-test-container-image-invoker fail in various ways when run with a docker subsitute on mac, or on M1 hardware. I don’t have a full matrix of behaviour, but we’ve observed:

• Mac M1, podman default: 15 failures, 1 error, 0 passed
  • Mac M1, podman patched to support multi-arch (see below): 9 failed, 7 passed
  • Mac M1, podman patched + ryuk disabled: 5-7 failed, 9-11 passed
  • Mac M1, podman patched + ryuk privileged (see below): 3 failed
• Mac M1, docker: Frozen at Building container-build-jib-with-mssql 0.1-SNAPSHOT, further failures after workaround
• Mac x86, podman: Failed in container-build-jib-with-db2
• Mac x86, minikube: 12 failures, 4 passed
• Mac x86, docker : clean

This is the invocation: ./mvnw -Dquickly -DskipTests=false -Dstart-containers=true -f integration-tests/container-image/maven-invoker-way

(There was originally a guard which (on some maven versions) only ran the tests on linux. The guard was removed as part of #25231 since it had some issues).

Ideally, the tests will pass on mac, and ideally without a hard requirement on docker as the container runtime. If we can’t get them to pass, we should reintroduce a guard, but make the guard as focussed as possible, perhaps just on the tests which are failing, and for the exact conditions which aren’t supported.

Causes of failures

I suspect there are several different issues.

Issues I was able to work around:

• x86 arch on images. Docker can cope with this, and podman can’t. I was able to update podman to cope by following the instructions on https://edofic.com/posts/2021-09-12-podman-m1-amd64/ (installing qemu inside the podman vm)
• docker socket. Working through https://xphyr.net/post/podman_on_osx/ resolved this.
• missing docker binary. IsDockerWorking looks for a docker binary or DOCKER_HOST. I didn’t need a DOCKER_HOST because of the podman helper, so duplicated the podman executable and called it docker and put ti on my path. I don’t think setting quarkus.docker.executable-name would be enough because the groovy scripts in this project do a straight exec of docker so need the script on the path

Issues I haven’t worked around/investigated

• expectations of localhost AFAIK Docker Desktop forwards from localhost to the VM, whereas using Minikube or similar exposes container ports on the VM. If tests assume everything is on localhost instead of allowing for containers being on another host and ask TestContainers what the address was it can cause issues
• …?

Expected behavior

The tests should run cleanly.

Actual behavior

Here are some example failures, from the platforms above.

M1, with patched podman

TESTCONTAINERS_RYUK_DISABLED="true" ./mvnw -Dquickly -DskipTests=false -Dstart-containers=true -f integration-tests/container-image/maven-invoker-way

[INFO] -------------------------------------------------
[INFO] Build Summary:
[INFO]   Passed: 7, Failed: 9, Errors: 0, Skipped: 0
[INFO] -------------------------------------------------
[ERROR] The following builds failed:
[ERROR] *  container-build-docker/pom.xml
[ERROR] *  container-build-jib-with-mysql/pom.xml
[ERROR] *  container-build-jib-with-db2/pom.xml
[ERROR] *  container-build-jib-with-mongo/pom.xml
[ERROR] *  container-build-multiple-tags-jib/pom.xml
[ERROR] *  container-build-jib-with-mssql/pom.xml
[ERROR] *  container-build-multiple-tags-docker/pom.xml
[ERROR] *  container-build-jib-appcds/pom.xml
[ERROR] *  container-image-push/pom.xml
[INFO] -------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  11:21 min
[INFO] Finished at: 2022-04-27T17:45:09+01:00
[INFO] ------------------------------------------------------------------------

This one looks like a podman compatibility issue but podman should support the java API?!

[INFO] Caused by: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=docker.io/ibmcom/db2:11.5.7.0a, imagePullPolicy=DefaultPullPolicy(), imageNameSubstitutor=org.testcontainers.utility.ImageNameSubstitutor$LogWrappedImageNameSubstitutor@7b8f4190)
[INFO] Caused by: java.lang.NullPointerException: Cannot invoke "String.matches(String)" because the return value of "com.github.dockerjava.api.model.PullResponseItem.getStatus()" is null

Unsure of this one

[INFO] [ERROR] Caused by: java.util.concurrent.ExecutionException: java.io.IOException: 'docker load' command failed with error: Error: unable to load image: payload does not match any of the supported image formats:
[INFO] [ERROR]  * oci: initializing source oci:/var/tmp/libpod-images-load.tar3361710228:: open /var/tmp/libpod-images-load.tar3361710228/index.json: not a directory
[INFO] [ERROR]  * oci-archive: creating temp directory: untarring file "/var/tmp/oci2061944008": unexpected EOF
[INFO] [ERROR]  * docker-archive: loading tar component manifest.json: unexpected EOF
[INFO] [ERROR]  * dir: open /var/tmp/libpod-images-load.tar3361710228/manifest.json: not a directory
[INFO] [ERROR] 
[INFO] [ERROR] 	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:588)

x86 mac with minikube

[INFO] -------------------------------------------------
[INFO] Build Summary:
[INFO]   Passed: 4, Failed: 12, Errors: 0, Skipped: 0
[INFO] -------------------------------------------------
[ERROR] The following builds failed:
[ERROR] *  container-build-with-keycloak/pom.xml
[ERROR] *  container-build-jib-with-mysql/pom.xml
[ERROR] *  container-build-jib-with-db2/pom.xml
[ERROR] *  container-build-jib-with-kafka/pom.xml
[ERROR] *  container-build-with-keycloak-default-realm/pom.xml
[ERROR] *  container-build-jib-with-mongo/pom.xml
[ERROR] *  container-build-jib-with-postgresql/pom.xml
[ERROR] *  container-build-jib-with-mssql/pom.xml
[ERROR] *  container-build-jib-with-mariadb/pom.xml
[ERROR] *  container-build-jib-appcds/pom.xml
[ERROR] *  container-image-jib-with-redis/pom.xml
[ERROR] *  container-image-push/pom.xml
[INFO] -------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  17:29 min
[INFO] Finished at: 2022-04-27T18:57:24+02:00
[INFO] ------------------------------------------------------------------------

M1 mac with Docker

[INFO] Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
[INFO]  at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
[INFO]  at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:338)
[INFO]  ... 15 more
[INFO] Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
[INFO]  at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:537)
[INFO]  at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:340)
[INFO]  at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
[INFO]  ... 16 more
[INFO] Caused by: java.lang.IllegalStateException: Container exited with code 1
[INFO]  at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:509)
[INFO]  ... 18 more

And some Caused by: java.net.ConnectException: Connection refused too

Frozen at Building container-build-jib-with-mssql 0.1-SNAPSHOT but -Dmssql.image=mcr.microsoft.com/azure-sql-edge seemed to work around that.

How to Reproduce?

1. ./mvnw -Dquickly -DskipTests=false -Dstart-containers=true -f integration-tests/container-image/maven-invoker-way

Output of uname -a or ver

We’re seeing a range of issues on various mac systems.

Output of java -version

A range of levels.

GraalVM version (if different from Java)

No response

Quarkus version or git rev

HEAD

Build tool (ie. output of mvnw --version or gradlew --version)

Range of levels.