buildx: GitHub Actions cache fails with cache key: blob not found

Copying from one stage to another failed in one of my repositories today in a Github Actions workflow after a minor change (https://github.com/jojomatik/blockcluster/commit/cdeb83a457f28692cc162e2453a8fe4a36be4ec8) in a workflow file, which shouldn’t affect docker at all: https://github.com/jojomatik/blockcluster/runs/3116972956

Dockerfile:62
--------------------
  60 |     RUN cd backend && npm install --only=prod
  61 |     
  62 | >>> COPY --from=builder /usr/games/blockcluster/dist dist
  63 |     COPY --from=builder /usr/games/blockcluster/backend/dist backend/dist
  64 |     COPY --from=builder /usr/games/blockcluster/.env ./
--------------------
error: failed to solve: failed to compute cache key: blob not found
Error: buildx failed with: error: failed to solve: failed to compute cache key: blob not found

I’m not sure where exactly this error message comes from and I wasn’t able to pin it down with a google search either. So I’m not entirely sure if this is a buildx issue or an issue with one of its dependencies?

This is the relevant section in the workflow file: https://github.com/jojomatik/blockcluster/blob/cdeb83a457f28692cc162e2453a8fe4a36be4ec8/.github/workflows/publish.yml#L92-L111

I’ve since tried to update it to use the non-rc versions (https://github.com/jojomatik/blockcluster/commit/d4488bae0dcd91ea99e85d6f05f839bf7ff536ec), but it has the same issues. Yesterday those workflows worked like a charm.

If I open up this issue in the wrong repository, I’m sorry. I’m not yet fully comprehending how all these docker parts work together 😃

I’d be grateful if you’d find some time to look into this, Thank you! If you need additional information, feel free to ask. I couldn’t find a PR template though.

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 9
Comments: 19 (8 by maintainers)

Commits related to this issue

ci: enable buildkit debug Enable debug output to provide more information for cache restore issue (docker/buildx#681). — committed to jojomatik/blockcluster by jojomatik 3 years ago
ci: use blockcluster scope for caching docker layers Use blockcluster scope for caching docker layers to check if it helps with the caching issue docker/buildx#681. — committed to jojomatik/blockcluster by jojomatik 3 years ago
Remove caching from docker builds There are still some bugs in the `gha` cache backend that are causing builds to fail: - docker/buildx#681 - docker/build-push-action#422 Removing caching until ups... — committed to artichoke/docker-artichoke-nightly by lopopolo 3 years ago
ci(web-server): limit workflow concurrency to avoid GHA rate limits There seems to be an issue currently where concurrent builds cause the Docker GHA cache exporter to trip over GitHub throttling lim... — committed to ntls-io/nautilus-wallet by PiDelport 3 years ago
ci(web-server): limit workflow concurrency to avoid GHA rate limits There seems to be an issue currently where concurrent builds cause the Docker GHA cache exporter to trip over GitHub throttling lim... — committed to ntls-io/nautilus-wallet by PiDelport 3 years ago
ci(web-server): limit workflow concurrency to avoid GHA rate limits There seems to be an issue currently where concurrent builds cause the Docker GHA cache exporter to trip over GitHub throttling lim... — committed to ntls-io/nautilus-wallet by PiDelport 3 years ago
ci(web-server): limit workflow concurrency to avoid GHA rate limits There seems to be an issue currently where concurrent builds cause the Docker GHA cache exporter to trip over GitHub throttling lim... — committed to ntls-io/nautilus-wallet by PiDelport 3 years ago
ci(web-server): limit workflow concurrency to avoid GHA rate limits There seems to be an issue currently where concurrent builds cause the Docker GHA cache exporter to trip over GitHub throttling lim... — committed to ntls-io/nautilus-wallet by PiDelport 3 years ago
replace gha cache to local cache docker build using gha cache failed as described here: https://github.com/docker/buildx/issues/681 using local cache as a workaround until it is fixed — committed to faldez/tanoshi by deleted user 3 years ago
cache locally due to docker/buildx#681 — committed to chrispyles/otter-grader by chrispyles 3 years ago
Revert caching temporarily Caching is currently causing the build to fail See https://github.com/docker/buildx/issues/681 — committed to Podcastindex-org/podping-hivewriter by agates 3 years ago
replace gha cache to local cache docker build using gha cache failed as described here: https://github.com/docker/buildx/issues/681 using local cache as a workaround until it is fixed — committed to faldez/tanoshi by faldez 3 years ago
replace gha cache to local cache docker build using gha cache failed as described here: https://github.com/docker/buildx/issues/681 using local cache as a workaround until it is fixed — committed to faldez/tanoshi by faldez 3 years ago

Most upvoted comments

Should be fixed since BuildKit 0.9.1.

crazy-max on Oct 5, 2021

Same issue here (private project). We have been using the gha cache for about 10 days and started seeing the same issue on all builds across branches, always failing on the step that has changed from build to build. The image with all layers is around 200MB (.NET Core app) with a couple of builds per day. We also cache node_modules for frontend — in a different workflow — same way as @jojomatik does.

We have reverted to using manual buildx cache and it works but isn’t really much of an improvement compared to not caching at all.

#30 [frontend  9/10] COPY js ./js
#30 ERROR: blob not found
------
 > [frontend  9/10] COPY js ./js:
------
Dockerfile:21
--------------------
  19 |     COPY css ./css
  20 |     COPY less ./less
  21 | >>> COPY js ./js
  22 |     RUN yarn build
  23 |     
--------------------
error: failed to solve: failed to compute cache key: blob not found
Error: buildx failed with: error: failed to solve: failed to compute cache key: blob not found

runs-on: ubuntu-20.04
steps:
  - uses: actions/checkout@v2
  - uses: docker/setup-buildx-action@v1
  - uses: docker/login-action@v1
      with:
        registry: ghcr.io
  - uses: docker/build-push-action@v2
     with:
       context: .
       push: true
       tags: ghcr.io/…
       cache-from: type=gha
       cache-to: type=gha,mode=max

patrikholcak on Sep 2, 2021

Thanks for your feedback everyone.

Seems like this worked as well. Does this confirm that my cache was somehow corrupted?

@jojomatik Blobs are somehow missing but it should not fail in this case actually. We are going to investigate on this issue.

crazy-max on Jul 29, 2021

@Niek

Do we need to specify this in the docker/setup-buildx-action step until a new version is released?

Not needed the default tag has been updated.

crazy-max on Oct 5, 2021

Run into probably the same issue – with type=gha (Buildx action with v0.6.0) and using scope. For some time the builds are OK and then suddenly they start to fail with:

error: failed to solve: failed to compute cache key: blob not found
Error: buildx failed with: error: failed to solve: failed to compute cache key: blob not found

It fails on COPY . . in the first stage of multi-stage build. There is almost nothing before that:

FROM node:14-buster-slim as packages
WORKDIR /build
COPY . .

And we use:

context: .
file: api/Dockerfile

JanJakes on Aug 9, 2021