buildkit: Cache manifest lists can't be exported to gcr
Related to https://github.com/moby/buildkit/issues/720 that currently has a gcr specific workaround in the pull code.
When exporting a manifest list that contains the cache metadata for a build (eg. with mode max), the upload fails in gcr with 400 error.
There was a report in slack that it also failed with 404 https://dockercommunity.slack.com/archives/C7S7A40MP/p1566224768282100 (hence similarities with #720) but couldn’t repro that.
#7 ERROR: error writing manifest blob: failed commit on ref "sha256:813b455d58cf597f96c8f20d04ae670127a94cd4786f14da09fef88e97bab090": unexpected status: 400 Bad Request
------
> exporting cache:
------
error: failed to solve: rpc error: code = Unknown desc = error writing manifest blob: failed commit on ref "sha256:813b455d58cf597f96c8f20d04ae670127a94cd4786f14da09fef88e97bab090": unexpected status: 400 Bad Request
After pushing all the blobs manifest list push fails with:
request:
PUT https://gcr.io/v2/.../hello/manifests/cache HTTP/2.0
← 400 application/json 190b 197ms
:authority: gcr.io
content-type: application/vnd.docker.distribution.manifest.list.v2+json
content-length: 935
accept-encoding: gzip
user-agent: Go-http-client/2.0
authorization: Bearer ....
{"schemaVersion":2,"mediaType":"application/vnd.docker.distribution.manifest.list.v2+json","manifests":[{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","digest":"sha256:0503825856099e6adb39c8297af09547f69684b7016b7f3680ed801aa310baaa","size":2789742,"annotations":{"buildkit/createdat":"2019-08-14T17:52:26.789506223-07:00","containerd.io/uncompressed":"sha256:1bfeebd65323b8ddf5bd6a51cc7097b72788bc982e9ab3280d53d3c613adffa7"}},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","digest":"sha256:a8ce4bba72fb43cbaeb05eca5d6d682696b915c53a3519d0f00f5aec063c0ae9","size":219,"annotations":{"buildkit/createdat":"2019-08-19T14:39:20.018002388-07:00","containerd.io/uncompressed":"sha256:72e39bafb519d4c8b9d12597a538667085627a6b8c3c3f085d319d3ef6955b4f"}},{"mediaType":"application/vnd.buildkit.cacheconfig.v0","digest":"sha256:02656190e9941165d11c1ac96d683c3da0cec10714e01ed4de4b6ded7e8b7c49","size":565}]}
response:
docker-distribution-api-versio registry/2.0
n:
content-type: application/json
content-length: 190
content-encoding: gzip
date: Mon, 19 Aug 2019 21:39:36 GMT
server: Docker Registry
cache-control: private
x-xss-protection: 0
x-frame-options: SAMEORIGIN
alt-svc: quic=":443"; ma=2592000; v="46,43,39"
[decoded gzip] JSON
{
"errors": [
{
"code": "MANIFEST_INVALID",
"message": "Failed to parse manifest for request \"/v2/.../hello/manifests/cache\": Failed to deserialize application/vnd.docker.distribution.manifest.list.v2+json."
}
]
}
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 3
- Comments: 34 (8 by maintainers)
Any update on this? I can’t
--export-cache type=registry,ref=to GCR neithermode=minnormaxand neither OCI worker nor containerd.I’m running into the same issue. Was anyone able to get it to work for caching with ECR?
I know this is closed, but leaving it here for people experiencing similar errors when porting GCR to GAR:
It’s worth noting that there is now an additional element in the repository path… You can experience this 400 BAD_REQUEST error when you don’t add that in. For example:
Works fine, but creating a GAR repository called
myappand simply switching the base address…Will give a BAD_REQUEST that looks a lot like this error. You need to specify the container name within the repository:
Happy artifacting 😎
@gajus I can confirm cache export does work with Google Artifact Registry.
I had the same issue with GitLab’s registry too, so I doubt it is GAR issue.
So the fix here is basically to not use GCR and use GAR instead?
Edit: I can confirm that after switching to GAR the error is gone.
gcr.io: not working, probably won’t
us.gcr.ioanddocker buildx create --useus.gcr.ioanddocker buildx create --name builder --driver docker-container --driver-opt image=moby/buildkit:v0.8-beta --usepkg.dev: working
ℹ️ Requires format: `{projectid}/{docker_repository}/{image_repository}
us-east1-docker.pkg.dev/proj/repo/cacheanddocker buildx create --useus-east1-docker.pkg.dev/proj/repo/cacheanddocker buildx create --name builder --driver docker-container --driver-opt image=moby/buildkit:v0.8-beta --use@crazy-max similar to this comment about the missing third path segment, that fixed it for me as well on
pkg.dev. Interesting! Thanks for the help 🙂 Overall I think this means pkg.dev is what one should use if expecting remote caching to work with GCP, right? Caveat: it is in beta and all.Do not close this issue, it still happening
Thanks!
This is really frustrating, because we return a structured error describing exactly the issue, but containerd swallows it: https://github.com/docker/build-push-action/issues/171#issuecomment-714644376
Hang tight.
This changed recently! https://cloud.google.com/blog/products/devops-sre/artifact-registry-is-ga
GitHub Container Registry doesn’t work either, see #1865.
@jbielick
AFAIK Google suggests to switch to GAR:
cc. @jonjohnsonjr
Depends on what you mean by most. Every registry based on https://github.com/docker/distribution allows it without any updates. Docker hub didn’t require any updates. I know Azure Container Registry works fine as well. ECR otoh still doesn’t support any variant of manifest lists afaik. The same pattern is used for example in containerd checkpoints definition and cnab-oci (that is used by docker app for example).