docker-maven-plugin: Multi-Architecture Builds Succeed But Fail to Push

Description

I’m able to run the build goal successfully, which is able to auth and pull down images.

[INFO] --- docker:0.43.4:build (build-docker-image) @ application ---
[INFO] DOCKER> GET unix://127.0.0.1:1/version
[INFO] Building tar: /Users/toconnor/Development/application/target/docker/application-img/tmp/docker-build.tar
[INFO] DOCKER> [application-img:latest] "application": Created docker-build.tar in 570 milliseconds
[INFO] DOCKER> Credentials helper reply for "docker-credential-osxkeychain" is docker-credential-osxkeychain (github.com/docker/docker-credential-helpers) v0.7.0
[INFO] DOCKER> Using Docker CLI 24.0.6
[INFO] DOCKER> docker --config /Users/toconnor/Development/application/target/docker/application-img/docker buildx create --driver docker-container --name multiarch
[INFO] DOCKER> multiarch
[INFO] Expanding: /Users/toconnor/Development/application/target/docker/application-img/tmp/docker-build.tar into /Users/toconnor/Development/application/target/docker/application-img/tmp/docker-build
[INFO] DOCKER> docker --config /Users/toconnor/Development/application/target/docker/application-img/docker buildx build --progress=plain --builder multiarch --platform linux/arm64 --tag application-img:latest --file=/Users/toconnor/Development/application/target/docker/application-img/tmp/docker-build/Dockerfile /Users/toconnor/Development/application/target/docker/application-img/tmp/docker-build --load
[INFO] DOCKER> #0 building with "multiarch" instance using docker-container driver
[INFO] DOCKER>
[INFO] DOCKER> #1 [internal] booting buildkit
[INFO] DOCKER> #1 pulling image moby/buildkit:buildx-stable-1
[INFO] DOCKER> #1 pulling image moby/buildkit:buildx-stable-1 1.4s done
[INFO] DOCKER> #1 creating container buildx_buildkit_multiarch0
[INFO] DOCKER> #1 creating container buildx_buildkit_multiarch0 0.6s done
[INFO] DOCKER> #1 DONE 2.0s
[INFO] DOCKER>
[INFO] DOCKER> #2 [internal] load build definition from Dockerfile
[INFO] DOCKER> #2 transferring dockerfile: 618B done
[INFO] DOCKER> #2 DONE 0.0s
[INFO] DOCKER>
[INFO] DOCKER> #3 [internal] load metadata for docker-repo/asr/java_v17_oracle_build_debug:3
[INFO] DOCKER> #3 ...
[INFO] DOCKER>
[INFO] DOCKER> #4 [auth] asr/java_v17_oracle_build_debug:pull token for docker-repo
[INFO] DOCKER> #4 DONE 0.0s
[INFO] DOCKER>
[INFO] DOCKER> #5 [auth] asr/java_v17_oracle_runtime:pull token for docker-repo
[INFO] DOCKER> #5 DONE 0.0s
[INFO] DOCKER>
[INFO] DOCKER> #6 [internal] load metadata for docker-repo/asr/java_v17_oracle_runtime:2.8.0
[INFO] DOCKER> #6 DONE 1.4s
[INFO] DOCKER>
[INFO] DOCKER> #3 [internal] load metadata for docker-repo/asr/java_v17_oracle_build_debug:3
[INFO] DOCKER> #3 DONE 1.5s
[INFO] DOCKER>
[INFO] DOCKER> #7 [internal] load .dockerignore
[INFO] DOCKER> #7 transferring context: 2B done
[INFO] DOCKER> #7 DONE 0.0s
[INFO] DOCKER>

... 

[INFO] DOCKER>
[INFO] DOCKER> #16 exporting to docker image format
[INFO] DOCKER> #16 exporting layers
[INFO] DOCKER> #16 exporting layers 2.2s done
[INFO] DOCKER> #16 exporting manifest sha256:8e266ee3684ab9ccac16475ed59998e358e2af6f506376bd9618c9542e376d7d 0.0s done
[INFO] DOCKER> #16 exporting config sha256:e8c5bb4c156e81f0d0b110cb66fc1eca0864974eee9a3f8f2e03b63952414a3e
[INFO] DOCKER> #16 exporting config sha256:e8c5bb4c156e81f0d0b110cb66fc1eca0864974eee9a3f8f2e03b63952414a3e 0.0s done
[INFO] DOCKER> #16 sending tarball
[INFO] DOCKER> #16 ...
[INFO] DOCKER>
[INFO] DOCKER> #17 importing to docker
[INFO] DOCKER> #17 DONE 0.7s
[INFO] DOCKER>
[INFO] DOCKER> #16 exporting to docker image format
[INFO] DOCKER> #16 sending tarball 3.6s done
[INFO] DOCKER> #16 DONE 5.8s

However, the push goal fails with a 403.

[INFO] --- docker:0.43.4:push (push-docker-image) @ application ---
[INFO] DOCKER> GET unix://127.0.0.1:1/version
[INFO] DOCKER> Using Docker CLI 24.0.6
[INFO] Expanding: /Users/toconnor/Development/application/target/docker/application-img/tmp/docker-build.tar into /Users/toconnor/Development/application/target/docker/application-img/tmp/docker-build
[INFO] DOCKER> docker --config /Users/toconnor/Development/application/target/docker/application-img/docker buildx build --progress=plain --builder multiarch --platform linux/amd64,linux/arm64 --tag remote-repo/application-img:latest --file=/Users/toconnor/Development/application/target/docker/application-img/tmp/docker-build/Dockerfile /Users/toconnor/Development/application/target/docker/application-img/tmp/docker-build --push
[INFO] DOCKER> #0 building with "multiarch" instance using docker-container driver
[INFO] DOCKER>
[INFO] DOCKER> #1 [internal] load build definition from Dockerfile
[INFO] DOCKER> #1 transferring dockerfile: 618B done
[INFO] DOCKER> #1 DONE 0.0s
[INFO] DOCKER>
[INFO] DOCKER> #2 [linux/arm64 internal] load metadata for docker-repo/asr/java_v17_oracle_build_debug:3
[INFO] DOCKER> #2 ERROR: unexpected status from HEAD request to https://docker-repo/v2/asr/java_v17_oracle_build_debug/manifests/3: 403 Forbidden
[INFO] DOCKER>
[INFO] DOCKER> #3 [linux/amd64 internal] load metadata for docker-repo/asr/java_v17_oracle_build_debug:3
[INFO] DOCKER> #3 ERROR: unexpected status from HEAD request to https://docker-repo/v2/asr/java_v17_oracle_build_debug/manifests/3: 403 Forbidden
[INFO] DOCKER>
[INFO] DOCKER> #4 [linux/amd64 internal] load metadata for docker-repo/asr/java_v17_oracle_runtime:2.8.0
[INFO] DOCKER> #4 CANCELED
[INFO] DOCKER>
[INFO] DOCKER> #5 [linux/arm64 internal] load metadata for docker-repo/asr/java_v17_oracle_runtime:2.8.0
[INFO] DOCKER> #5 CANCELED
[INFO] DOCKER> ------
[INFO] DOCKER>  > [linux/amd64 internal] load metadata for docker-repo/asr/java_v17_oracle_build_debug:3:
[INFO] DOCKER> ------
[INFO] DOCKER> ------
[INFO] DOCKER>  > [linux/arm64 internal] load metadata for docker-repo/asr/java_v17_oracle_build_debug:3:
[INFO] DOCKER> ------
[INFO] DOCKER> Dockerfile:5
[INFO] DOCKER> --------------------
[INFO] DOCKER>    3 |     ARG jacocoVersion
[INFO] DOCKER>    4 |
[INFO] DOCKER>    5 | >>> FROM ${buildImage} as builder
[INFO] DOCKER>    6 |     ARG jacocoVersion
[INFO] DOCKER> --------------------
[INFO] DOCKER> ERROR: failed to solve: docker-repo/asr/java_v17_oracle_build_debug:3: unexpected status from HEAD request to https://docker-repo/v2/asr/java_v17_oracle_build_debug/manifests/3: 403 Forbidden
[ERROR] DOCKER> Error status (1) when building

To make things even more confusing, if I copy the command and execute it locally in the shell the push command works as expected. Specifically this command which was copied verbatim from the debug output from DMP.

docker --config /Users/toconnor/Development/application/target/docker/application-img/docker buildx build --progress=plain --builder multiarch --platform linux/amd64,linux/arm64 --tag remote-repo/application-img:latest --file=/Users/toconnor/Development/application/target/docker/application-img/tmp/docker-build/Dockerfile /Users/toconnor/Development/application/target/docker/application-img/tmp/docker-build --push

I’ve tried every authentication mechanism specified in the documentation and they all result in the same behavior.

Both the source registry and destination registry are internally hosted on Artifactory and are v2 Docker registries. They require auth for all requests. The fact that I’m able to build successfully proves that I have authentication configured correctly. On the surface it appears that the HEAD request may not be sending authentication credentials but all the other requests do?

I’m running on an M1 Macbook with the latest Docker Desktop. As far as I can tell, this seems to be an issue only on Mac or with recent versions of Docker. This actually works from our Linux AMD64 based CI infrastructure. However, that infra is using Docker 20.x.x. I know from other issues reported that there is some variation in behavior between older Docker clients and newer.

At this point, I’m not sure how to debug this any further. I’m open to debug builds or even some hints on where to dig into the code. I spent some time looking at it but there was nothing obvious.

Info

  • docker-maven-plugin version : 0.43.4
  • Maven version (mvn -v) : 3.9.2

  • Docker version :
Client:
 Cloud integration: v1.0.35+desktop.4
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:28:49 2023
 OS/Arch:           darwin/arm64
 Context:           default

Server: Docker Desktop 4.23.0 (120376)
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:31:36 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
  • If it’s a bug, how to reproduce :

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 34 (16 by maintainers)

Commits related to this issue

Most upvoted comments

@rohanKanojia 0.44.0 works for me. Thanks.

@rohanKanojia you got it! This is working now!

@rohanKanojia BuildMojo doesn’t appear to be used during the buildx push operation. When I added logging to the BuildMojo code I can see the source registry is created as expected in the config.json but since this code path doesn’t execute in the buildx push path the authconfig doesn’t contain the source registry credentials. I think this is the source of the issue. Does this make sense?