buildkit: Get error: content digest sha256: ***: not found when exporting image
Would help check below issue and advise if any update? Thank you.
When exporting image, I found below error
#18 exporting to image
#18 pushing layers 0.1s done
#18 ERROR: content digest sha256:97bac3dab075a8e745a60a2e05e9f678053d6bca7ad1d109867220704b154443: not found
Then I checked the dictionary: ‘~/.local/share/buildkit/runc-native/content/blobs/sha256’ and file 97bac3dab075a8e745a60a2e05e9f678053d6bca7ad1d109867220704b154443 is missing, but could be found in one of the image manifest files. buildkit version:0.9.0, I didn’t config gc in Buildkitd configure file. I suspect there is an issue, which deletes a shared cache record in default gc.
some debug logs: time=“2022-02-23T05:59:48Z” level=debug msg=push time=“2022-02-23T05:59:48Z” level=debug msg=“fetch response received” response.header.accept-ranges=bytes response.header.cache-control=“max-age=31536000” response.header.connection=keep-alive response.header.content-length=306 response.header.content-type=application/octet-stream response.header.date=“Wed, 23 Feb 2022 05:59:48 GMT” response.header.docker-content-digest=“sha256:32678decbeb81d3211ddd542bd383f7ff304d63af7a78321e7b01b4021f65614” response.header.docker-distribution-api-version=registry/2.0 response.header.etag=“"sha256:32678decbeb81d3211ddd542bd383f7ff304d63af7a78321e7b01b4021f65614"” response.header.server=nginx response.header.set-cookie=“sid=6bd09de24143b204fc15afffd55b05a0; Path=/; HttpOnly” response.status=“200 OK” time=“2022-02-23T05:59:48Z” level=debug msg=“checking and pushing to” url=“http://harbor.jd.com/v2/jpipe-test/prod/jimidatalhwebservices/blobs/sha256:5e44ff2aeae6efe1449c178bf8edd1e7ced6fa5510fda8a3edc8d99c5fd64cc0” time=“2022-02-23T05:59:48Z” level=debug msg=“do request” request.header.content-type=application/octet-stream request.header.user-agent=containerd/1.6.0-beta.1+unknown request.method=PUT time=“2022-02-23T05:59:48Z” level=debug msg=push time=“2022-02-23T05:59:48Z” level=debug msg=“do request” request.header.accept=“application/vnd.docker.image.rootfs.diff.tar.gzip, /” request.header.user-agent=containerd/1.6.0-beta.1+unknown request.method=HEAD time=“2022-02-23T05:59:48Z” level=debug msg=“checking and pushing to” url=“http://harbor.jd.com/v2/jpipe-test/prod/jimidatalhwebservices/blobs/sha256:d9a2c8ccae4221b6b060b63bf56e53dad2295c1ce9d2cf1eb047ebd4eba1b297” time=“2022-02-23T05:59:48Z” level=warning msg=“failed to update distribution source for layer sha256:32678decbeb81d3211ddd542bd383f7ff304d63af7a78321e7b01b4021f65614: content digest sha256:32678decbeb81d3211ddd542bd383f7ff304d63af7a78321e7b01b4021f65614: not found” … time=“2022-02-23T06:08:09Z” level=debug msg=“checking and pushing to” url=“http://harbor.jd.com/v2/jpipe-test/prod/dbbakmasterlb/blobs/sha256:ce68c6bbd0a17f0742f673fb01cf19b88b575861ea0b45533ddbd82a068d1246” time=“2022-02-23T06:08:09Z” level=debug msg=“do request” request.header.accept=“application/vnd.docker.image.rootfs.diff.tar.gzip, /” request.header.user-agent=containerd/1.6.0-beta.1+unknown request.method=HEAD time=“2022-02-23T06:08:09Z” level=debug msg=“fetch response received” response.header.connection=keep-alive response.header.content-length=0 response.header.content-type=“text/plain; charset=utf-8” response.header.date=“Wed, 23 Feb 2022 06:08:09 GMT” response.header.docker-distribution-api-version=registry/2.0 response.header.docker-upload-uuid=94aee7cd-b8fa-4b04-a973-767fa2e87226 response.header.location=“http://harbor.jd.com/v2/jpipe-test/prod/dbbakmasterlb/blobs/uploads/94aee7cd-b8fa-4b04-a973-767fa2e87226?_state=O1cQSYNegKctbLxK1DNYrkatUhKemITsws5DR0VRnpt7Ik5hbWUiOiJqcGlwZS10ZXN0L3Byb2QvZGJiYWttYXN0ZXJsYiIsIlVVSUQiOiI5NGFlZTdjZC1iOGZhLTRiMDQtYTk3My03NjdmYTJlODcyMjYiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjItMDItMjNUMDY6MDg6MDkuNDc0NTc4NDkxWiJ9” response.header.range=0-0 response.header.server=nginx response.header.set-cookie=“sid=d63550dbee60e074afbebdfa4c37ba99; Path=/; HttpOnly” response.status=“202 Accepted” time=“2022-02-23T06:08:09Z” level=debug msg=“do request” request.header.content-type=application/octet-stream request.header.user-agent=containerd/1.6.0-beta.1+unknown request.method=PUT time=“2022-02-23T06:08:09Z” level=error msg=“/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = content digest sha256:32678decbeb81d3211ddd542bd383f7ff304d63af7a78321e7b01b4021f65614: not found\n” time=“2022-02-23T06:08:09Z” level=debug msg=“session finished: <nil>”
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 5
- Comments: 25 (7 by maintainers)
@jbguerraz Will try to submit a PR.
Thank you very much for your contribution, I have tested 3k images build for two weeks in our build cluster and the issue didn’t reproduce.
It should have been fixed in https://github.com/moby/buildkit/pull/3566.
@tonistiigi Was able to test your patch. Here is the method and result.
I built buildkit with
docker buildx build --tag=moby/buildkit:local --output=type=docker .. Using an older Docker Desktop, image was not exported to Docker withmake imageswhich runsdocker buildx build --tag=moby/buildkit:local --output=type=docker,buildinfo-attrs=true --attest=type=sbom --attest=type=provenance,mode=max ..Is there anything else I should try? Does it help?
Same issue here. If the patch helps why not merging it ?