moby: Using --cache-from doesn't skip base image pull

Ok so i’m not the only one experiencing this. I’ve spent the last 4 hours disabling overlay driver and creating clean runners in my Gitlab CI server, testing if that was the cause with no success.

On my machine the base-image is skipped and on Gitlab CI is always pulled.

I’ve opened an issue also on Gitlab #29184 until is not clear where this bug comes from.

Confirmed

I do this work-around to avoid pulling the base image and therefore invalidating the cache layers:

docker pull $(grep -ioP '(?<=^from)\s+\S+' Dockerfile)

In the end my pipeline looks about like this:

latest=$CI_REGISTRY/$CI_PROJECT_PATH:latest
this=$CI_REGISTRY/$CI_PROJECT_PATH:$CI_COMMIT_TAG

docker pull $(grep -ioP '(?<=^from)\s+\S+' Dockerfile)
docker pull $CI_REGISTRY/$CI_PROJECT_PATH:latest

docker build -t $this --cache-from $latest .
docker tag $this $latest

docker push $this
docker push $latest

/Cc @tonistiigi

Confirmed

I do this work-around to avoid pulling the base image and therefore invalidating the cache layers:

docker pull $(grep -ioP '(?<=^from)\s+\S+' Dockerfile)

For when on busybox (no -P for grep):

$(grep -oE 'FROM .+$' Dockerfile | head -n 1 | cut -d ' ' -f 2)

I think I bumped into the same issue with Travis CI. After some digging, to reproduce the issue locally, I had to remove /var/lib/docker/image/overlay2/distribution/diffid-by-digest. It looks like this directory contains mappings to the layer IDs. When doing docker save it is not preserved and loading the image does not populate it. Subsequently, even though the layers for the base image are loaded, the verification of their existence fails during docker pull.

As a workaround, I save that directory and restore it, as part of saving the images. Here is our .travis.yml file that does it and an example build.

If it helps anyone, I was able to use cache from previous builds in the shared GitLab Runner using the docker in docker service using Buildkit:

docker:build:
  stage: build
  services:
    - docker:dind
  image: docker
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_BUILDKIT: 1
  before_script:
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
    - docker pull $CI_REGISTRY_IMAGE:latest
  script:
    - docker build
      --build-arg BUILDKIT_INLINE_CACHE=1
      --cache-from $CI_REGISTRY_IMAGE:latest
      --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
      --tag $CI_REGISTRY_IMAGE:latest
      --file ./Dockerfile
      "."
  after_script:
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
    - if [[ "$CI_DEFAULT_BRANCH" = "$CI_COMMIT_BRANCH" ]]; then docker push $CI_REGISTRY_IMAGE:latest; fi

Couple things to note:

• This only seemed to work when I had the --build-arg BUILDKIT_INLINE_CACHE=1 as the first flag on docker build (I might be crazy)
• Or what made it work from me was that the latest tag was actually pushed from within the dind service itself

References:

• https://github.com/moby/moby/issues/38815
• https://testdriven.io/blog/faster-ci-builds-with-docker-cache/

Update:

I didn’t have to actually pull the latest tag either (saves a little bit more time in the runner):

docker:build:
  stage: build
  services:
    - docker:dind
  image: docker
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_BUILDKIT: 1
  before_script:
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
  script:
    - docker build
      --build-arg BUILDKIT_INLINE_CACHE=1
      --cache-from $CI_REGISTRY_IMAGE:latest
      --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
      --tag $CI_REGISTRY_IMAGE:latest
      --file ./docker/Dockerfile
      "."
  after_script:
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
    - if [[ "$CI_DEFAULT_BRANCH" = "$CI_COMMIT_BRANCH" ]]; then docker push $CI_REGISTRY_IMAGE:latest; fi

+1

Maybe i’m wrong and this is not the way it should work, but from my understanding inside myimage:latest i’ve all the layers from the base image so even if i don’t pull mybaseimage:latest it should use that layers to compare with the version on the registry and reuse them if the checksum is the same.