buildx: Default image output from buildx v0.10 cannot run on Google Cloud Run or AWS Lambda
As of Buildx 0.10, images are built with provenance – this behavior is opt-out, and can be disabled with --provenance=false if using the buildx build cli, or provenance: false if using docker/build-push-action.
Images built with these default attestations that use the attestation storage from buildkit cannot be run on Google Cloud Run or AWS Lambda. While both Cloud Run and Lambda support the OCI format, they do not support multi-platform images.
From Google Cloud Run docs (emphasis mine):
Cloud Run accepts container images in the Docker Image Manifest V2, Schema 1, Schema 2, and OCI image formats.
Manifest lists used for Multi-Architecture Images are not supported.
Note
Cloud Run has now added support for processing image indexes: https://github.com/docker/buildx/issues/1533#issuecomment-1416814739.
From Lambda docs (emphasis mine):
Lambda provides multi-architecture base images. However, the image you build for your function must target only one of the architectures. Lambda does not support functions that use multi-architecture container images.
Ideally, GCR and Lambda should support multi-platform images, and detect the current platform from the Docker manifest list / OCI index.
As a temporary workaround (in order of preference):
- Users should set
--provenance=falseon buildx build, or setprovenance: falseon docker/build-push-action - Users can force an explicit buildx version to v0.9.1
- Users can force an explicit buildkit version to v0.10
About this issue
- Original URL
- State: closed
- Created a year ago
- Reactions: 71
- Comments: 28 (2 by maintainers)
Commits related to this issue
- Set provenance to false on docker/build-push-action Builds were failing with the error: > OCI index found, but Accept header does not support OCI indexes The suggested fix is to temporarily disable... — committed to DFE-Digital/early-careers-framework by peteryates a year ago
- Set provenance to false on docker/build-push-action Builds were failing with the error: > OCI index found, but Accept header does not support OCI indexes The suggested fix is to temporarily disable... — committed to DFE-Digital/early-careers-framework by peteryates a year ago
- Set provenance to false on docker/build-push-action Builds were failing with the error: > OCI index found, but Accept header does not support OCI indexes The suggested fix is to temporarily disable... — committed to DFE-Digital/early-careers-framework by peteryates a year ago
- Set provenance to false on docker/build-push-action Builds were failing with the error: > OCI index found, but Accept header does not support OCI indexes The suggested fix is to temporarily disable... — committed to DFE-Digital/early-careers-framework by peteryates a year ago
- Set provenance to false on docker/build-push-action Builds were failing with the error: > OCI index found, but Accept header does not support OCI indexes The suggested fix is to temporarily disable... — committed to DFE-Digital/early-careers-framework by peteryates a year ago
- Set provenance to false on docker/build-push-action Builds were failing with the error: > OCI index found, but Accept header does not support OCI indexes The suggested fix is to temporarily disable... — committed to DFE-Digital/early-careers-framework by peteryates a year ago
- Add prvenance: false to docker/build-push-action Ref: docker/buildx#1533 — committed to Tautulli/Tautulli by JonnyWong16 a year ago
- Disable buildx provenance You gotta be kidding me. See: https://github.com/docker/buildx/issues/1533 — committed to jgarber623/indieweb-endpoints.cc by jgarber623 a year ago
- Disable buildx provenance You gotta be kidding me. See: https://github.com/docker/buildx/issues/1533 — committed to jgarber623/micromicro.cc by jgarber623 a year ago
- Disable buildx provenance You gotta be kidding me. See: https://github.com/docker/buildx/issues/1533 — committed to jgarber623/rel-me.cc by jgarber623 a year ago
- Update publish-docker.yml compatibility issues with google cloud and provenance. https://github.com/docker/buildx/issues/1533 — committed to ftrackhq/ftrack-actions by torsdag a year ago
- Add provenance parameter https://github.com/docker/buildx/issues/1533 — committed to vimeda/shared-actions by kieranajp a year ago
- Fix official docker images nto being compatible with some cloud providers See #23333 and https://github.com/docker/buildx/issues/1533 — committed to ClearlyClaire/mastodon by ClearlyClaire a year ago
- Fix official docker images nto being compatible with some cloud providers (#23354) See #23333 and https://github.com/docker/buildx/issues/1533 — committed to mastodon/mastodon by ClearlyClaire a year ago
- ci: upgra build-push-action v4.0.0 Disable provenance attestation functionality until issue https://github.com/docker/buildx/issues/1533 fix https://github.com/docker/build-push-action/releases/tag/... — committed to alex1989hu/kubelet-serving-cert-approver by alex1989hu a year ago
- Fix official docker images nto being compatible with some cloud providers (#23354) See #23333 and https://github.com/docker/buildx/issues/1533 — committed to btrd/mastodon by ClearlyClaire a year ago
- ci: Disable provenance attestation on docker images to work around docker/buildx#1533 Fixes https://github.com/n8n-io/n8n/issues/5394 — committed to netroy/n8n by netroy a year ago
- ci: Disable provenance attestation on docker images to work around docker/buildx#1533 Fixes https://github.com/n8n-io/n8n/issues/5394 — committed to n8n-io/n8n by netroy a year ago
@steren If you are Cloud Run PM then I have a request that can help save few hours of debugging for everyone. Would be nice to return proper error message when we try to use multi-platform images with cloud run instead of returning
It clearly exist in artifact registry since I choose it from select list when creating cloud run service.
Message like:
Would probably reduce debug time to 15 min instead of 6h.
Thanks
Cloud Run PM here, we received many reports about this. At the moment, Cloud Run doesn’t support multi architecture images.
We were tracking this work item, but we do not have an delivery date to communicate yet.
Thanks for your patience
Cloud Run now supports multi-architecture images as long as the manifest list includes
amd64/linux.We now expect buildx images built with default settings to deploy to Cloud Run, please let us know here if you still experience issues.
I agree, Cloud Run is now prioritizing this. And we hope to be able to deploy these images soon after.
Because this release was known to break Cloud Run and Lambda, it would have been in the best interest of users to give a heads up to these impacted runtimes ahead of time.
Agreed. We actually unfortunately didn’t know that this would break Cloud Run and Lambda at the time of release - we only realized late last week when GitHub updated buildx on GitHub actions. We later then amended the release notes to reflect this, to try and point users towards this issue, to make it easier to debug and track.
@tonistiigi understand the Docker site isn’t the place where the spec lives any more, but nevertheless this situation is incredibly confusing and there’s nowhere on the Docker site to find out about this. We burned about 8 hours of eng time before finally finding this issue, I imagine an awful lot of teams are affected by this. Could do well by making the change easy to find
We did some testing and found that we only had this issue on clusters running on version 1.21. All other versions (1.22, 1.23 and 1.24) all seemed to work fine for us.
@majidakbaridh you can use
-o oci-mediatypes=falsebut if you want to avoid attestations then disable attestations directly with--provenance=false. You can’t have the attestations without the OCI mediatypes as well.Try pushing to Artifact Registry instead of Container Registry