moby: Building with BUILDKIT and cache-from fails in Google Cloud Build

I’m trying to migrate my docker build in Google Cloud Build CI to use DOCKER_BUILDKIT.

The build step is now using docker:19.03-rc-dind:

  - name: 'docker:19.03-rc-dind'
    id: 'Build and Push'
    entrypoint: 'sh'
    args:
      - '-c'
      - |
        export DOCKER_BUILDKIT=1 && \
        docker build -f service.Dockerfile -t gcr.io/acme/service:latest --build-arg BUILDKIT_INLINE_CACHE=1 . && \
        docker push gcr.io/acme/service:latest

And this works great. However, when I tried to integrate cache-from, I’m getting errors. Switching the build command above to

docker build -f service.Dockerfile -t gcr.io/acme/service:latest --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from gcr.io/acme/service:latest .

Fails with the following:

error on cache query: invalid build cache from {MediaType:application/vnd.docker.distribution.manifest.v2+json Digest:sha256:xxx Size:3685 URLs:[] Annotations:map[] Platform:<nil>}

When I run the same command using the same docker:19.03-rc-dind image in my local macOS, build completes successfuly with --cache-from and even actually loads the cache context.

Is this an issue with Google Cloud environment? This gitlab issue suggests it may be an issue with gcr (Google Container Registry), but I don’t think so, as I can pull from gcr fine locally.

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 3
Comments: 29 (11 by maintainers)

Commits related to this issue

Reland "Use single source of truth for building images for Local and GCB build systems" (#653)" Has bug fixes - Fix for buildkit failures, see https://github.com/moby/moby/issues/40262 Use latest... — committed to google/fuzzbench by inferno-chromium 4 years ago
Reland "Use single source of truth for building images for Local and … (#729) * Reland "Use single source of truth for building images for Local and GCB build systems" (#653)" Has bug fixes - Fix... — committed to google/fuzzbench by inferno-chromium 4 years ago

Most upvoted comments

Ah, looks like it’s indeed the mirror issue; https://github.com/moby/buildkit/issues/1271. From https://github.com/kubernetes/kubernetes/issues/92030, it appears that Google Cloud configures a mirror (https://mirror.gcr.io) by default, which would likely explain the issue

thaJeztah on Jul 9, 2020

This is going to get worse with many people, me included, jumping to the gcr mirror since the Docker Hub rate limits.

hansbogert on Nov 10, 2020

@tonistiigi is there a solution now ? I’m not using cloud build but similarly I pull image from gcr and gcr doesn’t contain a image specify in my Dockerfile and i get error image not found.

ishallbethat on Aug 28, 2020

I meant just a mirror that you didn’t configure yourself but is automatically added to your build by the infra. Yes, that is the problem with grc mirror that is doesn’t fallback itself if it doesn’t have contents, like the mirror mode in docker/distribution registry does for example.

tonistiigi on Jul 9, 2020

Like I said, this issue happens only in google cloud build which was just updated to 19.03.8 (https://issuetracker.google.com/issues/157501467) and now it supports BuildKit.

It might be that google cloud build internally has a registry mirror, don’t know. Here’s the output from failing google cloud build.

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 380B 0.0s done
#1 DONE 0.0s

#3 [internal] load metadata for docker.io/library/golang:1.14.4-alpine3.12
#3 ERROR: docker.io/library/golang:1.14.4-alpine3.12 not found

#13 importing cache manifest from eu.gcr.io/svc-vodka/k8s-node-labeler:lates...
#13 DONE 0.0s

#6 [internal] load build context
#6 DONE 0.0s

#14 [builder 1/8] FROM docker.io/library/golang:1.14.4-alpine3.12
#14 resolve docker.io/library/golang:1.14.4-alpine3.12 0.1s done
#14 ERROR: docker.io/library/golang:1.14.4-alpine3.12 not found
------
 > [internal] load metadata for docker.io/library/golang:1.14.4-alpine3.12:
------
------
 > [builder 1/8] FROM docker.io/library/golang:1.14.4-alpine3.12:
------
failed to solve with frontend dockerfile.v0: failed to build LLB: failed to load cache key: docker.io/library/golang:1.14.4-alpine3.12 not found

matti on Jul 9, 2020

now it works, but for some reason all FROM images need to be pulled - created a helper script for this: https://github.com/matti/gbuilder

matti on Jul 8, 2020