buildkit: Get error: content digest sha256: ***: not found when exporting image

Would help check below issue and advise if any update? Thank you.

When exporting image, I found below error

#18 exporting to image
#18 pushing layers 0.1s done
#18 ERROR: content digest sha256:97bac3dab075a8e745a60a2e05e9f678053d6bca7ad1d109867220704b154443: not found

Then I checked the dictionary: ‘~/.local/share/buildkit/runc-native/content/blobs/sha256’ and file 97bac3dab075a8e745a60a2e05e9f678053d6bca7ad1d109867220704b154443 is missing, but could be found in one of the image manifest files. buildkit version:0.9.0, I didn’t config gc in Buildkitd configure file. I suspect there is an issue, which deletes a shared cache record in default gc.

some debug logs: time=“2022-02-23T05:59:48Z” level=debug msg=push time=“2022-02-23T05:59:48Z” level=debug msg=“fetch response received” response.header.accept-ranges=bytes response.header.cache-control=“max-age=31536000” response.header.connection=keep-alive response.header.content-length=306 response.header.content-type=application/octet-stream response.header.date=“Wed, 23 Feb 2022 05:59:48 GMT” response.header.docker-content-digest=“sha256:32678decbeb81d3211ddd542bd383f7ff304d63af7a78321e7b01b4021f65614” response.header.docker-distribution-api-version=registry/2.0 response.header.etag=“"sha256:32678decbeb81d3211ddd542bd383f7ff304d63af7a78321e7b01b4021f65614"” response.header.server=nginx response.header.set-cookie=“sid=6bd09de24143b204fc15afffd55b05a0; Path=/; HttpOnly” response.status=“200 OK” time=“2022-02-23T05:59:48Z” level=debug msg=“checking and pushing to” url=“http://harbor.jd.com/v2/jpipe-test/prod/jimidatalhwebservices/blobs/sha256:5e44ff2aeae6efe1449c178bf8edd1e7ced6fa5510fda8a3edc8d99c5fd64cc0” time=“2022-02-23T05:59:48Z” level=debug msg=“do request” request.header.content-type=application/octet-stream request.header.user-agent=containerd/1.6.0-beta.1+unknown request.method=PUT time=“2022-02-23T05:59:48Z” level=debug msg=push time=“2022-02-23T05:59:48Z” level=debug msg=“do request” request.header.accept=“application/vnd.docker.image.rootfs.diff.tar.gzip, /” request.header.user-agent=containerd/1.6.0-beta.1+unknown request.method=HEAD time=“2022-02-23T05:59:48Z” level=debug msg=“checking and pushing to” url=“http://harbor.jd.com/v2/jpipe-test/prod/jimidatalhwebservices/blobs/sha256:d9a2c8ccae4221b6b060b63bf56e53dad2295c1ce9d2cf1eb047ebd4eba1b297” time=“2022-02-23T05:59:48Z” level=warning msg=“failed to update distribution source for layer sha256:32678decbeb81d3211ddd542bd383f7ff304d63af7a78321e7b01b4021f65614: content digest sha256:32678decbeb81d3211ddd542bd383f7ff304d63af7a78321e7b01b4021f65614: not found” … time=“2022-02-23T06:08:09Z” level=debug msg=“checking and pushing to” url=“http://harbor.jd.com/v2/jpipe-test/prod/dbbakmasterlb/blobs/sha256:ce68c6bbd0a17f0742f673fb01cf19b88b575861ea0b45533ddbd82a068d1246” time=“2022-02-23T06:08:09Z” level=debug msg=“do request” request.header.accept=“application/vnd.docker.image.rootfs.diff.tar.gzip, /” request.header.user-agent=containerd/1.6.0-beta.1+unknown request.method=HEAD time=“2022-02-23T06:08:09Z” level=debug msg=“fetch response received” response.header.connection=keep-alive response.header.content-length=0 response.header.content-type=“text/plain; charset=utf-8” response.header.date=“Wed, 23 Feb 2022 06:08:09 GMT” response.header.docker-distribution-api-version=registry/2.0 response.header.docker-upload-uuid=94aee7cd-b8fa-4b04-a973-767fa2e87226 response.header.location=“http://harbor.jd.com/v2/jpipe-test/prod/dbbakmasterlb/blobs/uploads/94aee7cd-b8fa-4b04-a973-767fa2e87226?_state=O1cQSYNegKctbLxK1DNYrkatUhKemITsws5DR0VRnpt7Ik5hbWUiOiJqcGlwZS10ZXN0L3Byb2QvZGJiYWttYXN0ZXJsYiIsIlVVSUQiOiI5NGFlZTdjZC1iOGZhLTRiMDQtYTk3My03NjdmYTJlODcyMjYiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjItMDItMjNUMDY6MDg6MDkuNDc0NTc4NDkxWiJ9” response.header.range=0-0 response.header.server=nginx response.header.set-cookie=“sid=d63550dbee60e074afbebdfa4c37ba99; Path=/; HttpOnly” response.status=“202 Accepted” time=“2022-02-23T06:08:09Z” level=debug msg=“do request” request.header.content-type=application/octet-stream request.header.user-agent=containerd/1.6.0-beta.1+unknown request.method=PUT time=“2022-02-23T06:08:09Z” level=error msg=“/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = content digest sha256:32678decbeb81d3211ddd542bd383f7ff304d63af7a78321e7b01b4021f65614: not found\n” time=“2022-02-23T06:08:09Z” level=debug msg=“session finished: <nil>”

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 5
  • Comments: 25 (7 by maintainers)

Most upvoted comments

@jbguerraz Will try to submit a PR.

Hi @sunchunming @Shaked @lugeng @okgolove @dweomer

We meet the same issue in our buildkit production environment (buildkitd v0.10.3), there is about a 1% chance of error.

This issue can be fixed by this patch, we have tested 5k image builds for some days.

We have observed that this problem always occurs during the push layers phase, and that the base image layer is not found.

By adding some logs we found that in the https://github.com/imeoer/buildkit/blob/26c11880022774bc6eca6376aef5e698ecf629c5/cache/refs.go#L276, !cr. getBlobOnly() always true, causing the push image process to not download the lazy layer into the content store first, then throw the error in https://github.com/imeoer/buildkit/blob/26c11880022774bc6eca6376aef5e698ecf629c5/cache/remote.go#L304.

This issue seems difficult to reproduce, perhaps we can take a look deeper. cc @tonistiigi @sipsma

Thank you very much for your contribution, I have tested 3k images build for two weeks in our build cluster and the issue didn’t reproduce.

It should have been fixed in https://github.com/moby/buildkit/pull/3566.

@tonistiigi Was able to test your patch. Here is the method and result.

  1. Created a public repo: https://github.com/sharesight/buildkit-debug
  2. Added a simple workflow using well known actions and basic Dockerfile. Did not reproduce the problem, v0.11.2 used.
  3. Create a first PR (https://github.com/sharesight/buildkit-debug/pull/1) triggering a complete rebuild of the image. Still did not reproduce the problem
  4. Create a second PR (https://github.com/sharesight/buildkit-debug/pull/2) adding a more complicated Dockerfile. Workflow stil using v0.11.2. Problem reproduced. Workflow ran twice and failed each time: https://github.com/sharesight/buildkit-debug/actions/runs/4062589459/attempts/1, https://github.com/sharesight/buildkit-debug/actions/runs/4062589459
  5. Added a commit to second PR, to use patch above against revision v0.11.2. Ran the workflow twice and succeeded each time: https://github.com/sharesight/buildkit-debug/actions/runs/4118068370/attempts/1, https://github.com/sharesight/buildkit-debug/actions/runs/4118068370. The buildkit image I built is public accessible: public.ecr.aws/m4n2v6l4/buildkit:blobonly-debug.

I built buildkit with docker buildx build --tag=moby/buildkit:local --output=type=docker .. Using an older Docker Desktop, image was not exported to Docker with make images which runs docker buildx build --tag=moby/buildkit:local --output=type=docker,buildinfo-attrs=true --attest=type=sbom --attest=type=provenance,mode=max ..

Is there anything else I should try? Does it help?

Same issue here. If the patch helps why not merging it ?