moby: Cache not used if docker build is run on different hosts

What is the current status on using cache for docker builds on different hosts?

Using GitLab CI I noticed that if caching is used in multi-stage builds and functions correctly on one host (e.g. by using a runner tag) caching from the images does not work if the built is run by a runner on a different host.

Fore example this built (job) used a custom runner, leveraging build caches as expected: https://gitlab.com/romangrothausmann/dind-sikuli/-/jobs/169875266 whereas this built job (for the same git-commit) used a shared runner from GL and for some reason did not use the cache provided by exactly the same images (according to the SHAs): https://gitlab.com/romangrothausmann/dind-sikuli/-/jobs/169882202

In both jobs the same SHAs are reported for the cache images:

docker:stable: 683afd3e5d5f8904fb8aa7874a84325970ac82b720d02336a380a77f874bc51e
registry.gitlab.com/romangrothausmann/dind-sikuli/master:system-latest: ff7fca3a1f3e6587807d41848fc4c3510181c28fbca64b1066fceffdc88a5f26
registry.gitlab.com/romangrothausmann/dind-sikuli/master:builder-latest 091c8818a462570a844721cd61bcbb9260821a57c5e392589436f918a491b41d
registry.gitlab.com/romangrothausmann/dind-sikuli/master:test f565c61fedd23957dafe8ee864e252240166bede9a0c6a070d04964b8b3b1913

In the custom runner job the building starts with: https://gitlab.com/romangrothausmann/dind-sikuli/-/jobs/169875266/raw

Status: Downloaded newer image for docker:stable
 ---> 8044a7b7add7
Step 2/2 : RUN apk add --update --no-cache     curl
 ---> Using cache
 ---> 8df41a120c91

however in the GL runner building starts with: https://gitlab.com/romangrothausmann/dind-sikuli/-/jobs/169882202/raw

Status: Downloaded newer image for docker:stable
 ---> 8044a7b7add7
Step 2/2 : RUN apk add --update --no-cache     curl
 ---> Running in 34ba059a4a77
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
(1/4) Installing nghttp2-libs (1.35.1-r0)
(2/4) Installing libssh2 (1.8.0-r4)
(3/4) Installing libcurl (7.64.0-r1)
(4/4) Installing curl (7.64.0-r1)
Executing busybox-1.29.3-r10.trigger
OK: 7 MiB in 19 packages
Removing intermediate container 34ba059a4a77
 ---> 7f034b1d017d

These are some issues that seem related but are either quite old or the final reason discussed seems not to be related to different hosts: https://github.com/moby/moby/issues/35286 https://github.com/moby/moby/issues/22600 https://github.com/moby/moby/issues/20316 https://github.com/moby/moby/issues/31613

It is said here: https://stackoverflow.com/questions/45238429/docker-caching-how-does-it-really-work#45246760

The build command needs to be run against the same docker host where the previous image’s cache exists.

but I could not find any info on that in the docker docs. If that is actually the case it would be great to have some info on that and the reasons for it in the docker docs. Is there any way to make use of cache images across different hosts?

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 4
Comments: 22 (8 by maintainers)

Most upvoted comments

--build-arg BUILDKIT_INLINE_CACHE=1 (requires either Docker 19.03 or buildx)

AkihiroSuda on Sep 14, 2019

--cache-from doesn’t work when image that docker pulls is different from the one in the cache.

Case study

If an image wasn’t updated in dockerhub since the tagged build:

Step 1/14 : FROM centos:7
7: Pulling from library/centos
8ba884070f61: Already exists
Digest: sha256:b5e66c4651870a1ad435cd75922fe2cb943c9e973a9673822d1414824a1d0475
Status: Downloaded newer image for centos:7
Step 2/14 : RUN yum update -y &&     yum groupinstall -y 'Development Tools' &&     yum install -y wget
 ---> Using cache
 ---> 071bec3016bb

Cache is used.

If the image in dockerhub is newer:

Step 1/21 : FROM ubuntu:16.04
 ---> 13c9f1285025
Step 2/21 : ENV DEBIAN_FRONTEND noninteractive
 ---> Running in df769f231930

Cache is not used.

Dr-Desty-Nova on Jun 21, 2019

If this question is about cache-from then this does not work for multi-stage builds in the legacy builder, or more precisely it only works if you provide cache for all the stages in the multi-stage build.

tonistiigi on Mar 1, 2019