buildkit: Cannot build from local image with buildctl (OCI Worker)

Description

Cannot build from a local image with buildctl which always tries to resolve to docker.io. This applies to a multi-stage setup or simply from a local image in a CI setup. The base image should not be expected to be pushed to any repository (either docker.io or even private). It is only available locally (docker images).

This works with docker build and even DOCKER_BUILDKIT=1 using image names (but not SHA ID in the last case). See also https://github.com/moby/moby/issues/39769.

This looks similar to the problems with local cache described here: https://github.com/moby/moby/issues/39003.

Steps to reproduce the issue:

Demo project available here: https://github.com/virgile-hogman/docker-buildkit
Run ./buildctl-build.sh from bash
See logs, fails to resolve FROM docker.io/library/bk1:latest

Describe the results you received:

+ buildctl build --frontend dockerfile.v0 --progress=plain --local context=src2 --local dockerfile=src2 --output type=docker,name=bk2
+ docker load
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 66B done
#2 DONE 0.1s

#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.1s

#3 [internal] load metadata for docker.io/library/bk1:latest
#3 ERROR: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

#4 [1/2] FROM docker.io/library/bk1:latest
#4 resolve docker.io/library/bk1:latest
#4 resolve docker.io/library/bk1:latest 0.4s done
#4 ERROR: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
------
 > [internal] load metadata for docker.io/library/bk1:latest:
------
------
 > [1/2] FROM docker.io/library/bk1:latest:
------
error: failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to build LLB: failed to load cache key: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
open /var/lib/docker/tmp/docker-import-259291410/repositories: no such file or directory

Describe the results you expected:

This should work as for docker build when using image names (with or without DOCKER_BUILDKIT=1. Note with SHA-ID it does not work either in the last case as described in https://github.com/moby/moby/issues/39769).

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client:
 Version:           18.09.6
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        481bc77
 Built:             Sat May  4 02:35:57 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Output of docker info:

Native install with Ubuntu 18.04.
More info does not seem relevant for now, can provide if really needed.

Additional environment details (AWS, VirtualBox, physical, etc.):

Connected to a private repository on a different host but this should not have effect here since it works with standard docker build.

About this issue

Original URL
State: open
Created 5 years ago
Reactions: 9
Comments: 15 (5 by maintainers)

Most upvoted comments

Is there any simple tutorial or documentation for setting up such a configuration? I have never used containerd and it sounds quite obscure to an end-user not familiar with terms like rund, containerd, workers or OCI, this all sounds like internal stuff for container developers which i’m not. The nice thing with Docker is its convenience and accessibility, it can be used out of the box from a simple apt installation. Here i don’t even know where to look.

virgile-hogman on Aug 27, 2019

I’m sorry I don’t understand all this but just curious if this is the same root issue as I’m seeing in: https://github.com/docker/for-mac/issues/5873 where building FROM a local image, which itself was built FROM a debian image, fails with “pull access denied” only when using buildkit on an M1 Mac.

jamshid on Aug 3, 2021

I face the same problem, and tried to switch from oci to containerd worker but fails to boot the buildkit deamon.

$ docker run --name buildkit --privileged -p 1234:1234 moby/buildkit --addr tcp://0.0.0.0:1234 --oci-worker=false --containerd-worker=true
buildkitd: failed to connect client to "/run/containerd/containerd.sock" . make sure containerd is running: failed to dial "/run/containerd/containerd.sock": context deadline exceeded

The image tag for moby/buildkit is 8ce8d3df040f. I am running on Mac Docker desktop (2.1.03) with Engine version 19.03.2

Is that I must build the buildkitd myself if I want to test with the containerd worker? Thanks for your help.

rickmak on Sep 17, 2019

moby/moby#39144 has been merged in yet it looks like I got the same problem on circleci:

The dockerfile is a multi-stage dockerfile.

partial docker build script:

export DOCKER_BUILDKIT=1

###############################################

BACKEND_TAG=${CIRCLE_BRANCH}-backend-v2

echo "Pull old image for layer caching"
# Return true to ensure branches without images can still build
docker pull ${AWS_ACCOUNT_ID}.dkr.ecr.us-east-1.amazonaws.com/15five/fifteen5:${BACKEND_TAG} || true

echo "Build new python backend docker image"
docker build -f Dockerfile.production --target python_backend \
    --tag 15five/fifteen5:${BACKEND_TAG} \
    --build-arg dynatrace_subdomain=${DT_ENDPOINT} \
    --build-arg use_dynatrace=1 \
    .

      - setup_remote_docker:
          version: 19.03.12

Output of the above is:

Allocating a remote Docker Engine
Requesting version: 19.03.12

Assigned Docker Engine request id: 216311031
  provisioning: .
Remote Docker engine created. Using VM 'default-61b1abb0-61a6-4349-9ccb-5ba12832ffab'
Created container accessible with:
  DOCKER_CERT_PATH=/tmp/docker-certs181859510
  DOCKER_HOST=tcp://35.227.29.140:2376
  DOCKER_MACHINE_NAME=216311031
  DOCKER_TLS_VERIFY=1
  NO_PROXY=127.0.0.1,localhost,circleci-internal-outer-build-agent,35.227.29.140:2376

Server Engine Details:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       48a66213fe
  Built:            2020-06-22T15:44:20.000000000+00:00
  OS/Arch:          linux/amd64
  Experimental:     false

docker build script log:

Pull old image for layer caching
Error response from daemon: manifest for 482205614324.dkr.ecr.us-east-1.amazonaws.com/15five/fifteen5:add_node_image-backend-v2 not found: manifest unknown: Requested image not found
Build new python backend docker image
[+] Building 0.0s (0/2)                                                         
[+] Building 0.1s (1/2)                                                         
 => [internal] load build definition from Dockerfile.production            0.1s
 => => transferring dockerfile: 1.09kB                                     0.0s
 => [internal] load .dockerignore                                          0.1s
 => => transferring context: 1.61kB                                        0.0s
[+] Building 0.3s (2/3)                                                         
 => [internal] load build definition from Dockerfile.production            0.1s
 => => transferring dockerfile: 1.09kB                                     0.0s
 => [internal] load .dockerignore                                          0.1s
 => => transferring context: 1.61kB                                        0.0s
 => resolve image config for docker.io/docker/dockerfile:experimental      0.1s
[+] Building 0.4s (2/3)                                                         
 => [internal] load build definition from Dockerfile.production            0.1s
 => => transferring dockerfile: 1.09kB                                     0.0s
 => [internal] load .dockerignore                                          0.1s
 => => transferring context: 1.61kB                                        0.0s
 => resolve image config for docker.io/docker/dockerfile:experimental      0.3s
[+] Building 0.6s (2/3)                                                         
 => [internal] load build definition from Dockerfile.production            0.1s
 => => transferring dockerfile: 1.09kB                                     0.0s
 => [internal] load .dockerignore                                          0.1s
 => => transferring context: 1.61kB                                        0.0s
 => resolve image config for docker.io/docker/dockerfile:experimental      0.4s
[+] Building 0.7s (3/4)                                                         
 => => transferring context: 1.61kB                                        0.0s
 => resolve image config for docker.io/docker/dockerfile:experimental      0.4s
 => docker-image://docker.io/docker/dockerfile:experimental@sha256:de85b2  0.1s
 => => resolve docker.io/docker/dockerfile:experimental@sha256:de85b2f3a3  0.0s
 => => sha256:61261561661960014533790f5d6c42f5b88362db7e005f3 0B / 8.88MB  0.1s
 => => sha256:de85b2f3a3e8a2f7fe48e8e84a65f6fdd5cd5183afa 1.69kB / 1.69kB  0.0s
 => => sha256:8c69d118cfcd040a222bea7f7d57c6156faa938cb61b476 521B / 521B  0.0s
 => => sha256:08fb2b2ca3d58e19d791e73dea16126df37608115532e74 897B / 897B  0.0s
[+] Building 0.8s (3/4)                                                         
 => => transferring context: 1.61kB                                        0.0s
 => resolve image config for docker.io/docker/dockerfile:experimental      0.4s
 => docker-image://docker.io/docker/dockerfile:experimental@sha256:de85b2  0.3s
 => => resolve docker.io/docker/dockerfile:experimental@sha256:de85b2f3a3  0.0s
 => => sha256:61261561661960014533790f5d6c42f5b88362db7e0 5.01MB / 8.88MB  0.2s
 => => sha256:de85b2f3a3e8a2f7fe48e8e84a65f6fdd5cd5183afa 1.69kB / 1.69kB  0.0s
 => => sha256:8c69d118cfcd040a222bea7f7d57c6156faa938cb61b476 521B / 521B  0.0s
 => => sha256:08fb2b2ca3d58e19d791e73dea16126df37608115532e74 897B / 897B  0.0s
[+] Building 0.9s (3/4)                                                         
 => resolve image config for docker.io/docker/dockerfile:experimental      0.4s
 => docker-image://docker.io/docker/dockerfile:experimental@sha256:de85b2  0.4s
 => => resolve docker.io/docker/dockerfile:experimental@sha256:de85b2f3a3  0.0s
 => => sha256:61261561661960014533790f5d6c42f5b88362db7e0 8.88MB / 8.88MB  0.2s
 => => sha256:de85b2f3a3e8a2f7fe48e8e84a65f6fdd5cd5183afa 1.69kB / 1.69kB  0.0s
 => => sha256:8c69d118cfcd040a222bea7f7d57c6156faa938cb61b476 521B / 521B  0.0s
 => => sha256:08fb2b2ca3d58e19d791e73dea16126df37608115532e74 897B / 897B  0.0s
 => => extracting sha256:61261561661960014533790f5d6c42f5b88362db7e005f35  0.1s
[+] Building 1.0s (3/4)                                                         
 => resolve image config for docker.io/docker/dockerfile:experimental      0.4s
 => docker-image://docker.io/docker/dockerfile:experimental@sha256:de85b2  0.5s
 => => resolve docker.io/docker/dockerfile:experimental@sha256:de85b2f3a3  0.0s
 => => sha256:61261561661960014533790f5d6c42f5b88362db7e0 8.88MB / 8.88MB  0.2s
 => => sha256:de85b2f3a3e8a2f7fe48e8e84a65f6fdd5cd5183afa 1.69kB / 1.69kB  0.0s
 => => sha256:8c69d118cfcd040a222bea7f7d57c6156faa938cb61b476 521B / 521B  0.0s
 => => sha256:08fb2b2ca3d58e19d791e73dea16126df37608115532e74 897B / 897B  0.0s
 => => extracting sha256:61261561661960014533790f5d6c42f5b88362db7e005f35  0.2s
[+] Building 1.2s (4/4)                                                         
 => resolve image config for docker.io/docker/dockerfile:experimental      0.4s
 => docker-image://docker.io/docker/dockerfile:experimental@sha256:de85b2  0.6s
 => => resolve docker.io/docker/dockerfile:experimental@sha256:de85b2f3a3  0.0s
 => => sha256:61261561661960014533790f5d6c42f5b88362db7e0 8.88MB / 8.88MB  0.2s
 => => sha256:de85b2f3a3e8a2f7fe48e8e84a65f6fdd5cd5183afa 1.69kB / 1.69kB  0.0s
 => => sha256:8c69d118cfcd040a222bea7f7d57c6156faa938cb61b476 521B / 521B  0.0s
 => => sha256:08fb2b2ca3d58e19d791e73dea16126df37608115532e74 897B / 897B  0.0s
 => => extracting sha256:61261561661960014533790f5d6c42f5b88362db7e005f35  0.3s
[+] Building 1.3s (4/4)                                                         
 => resolve image config for docker.io/docker/dockerfile:experimental      0.4s
 => docker-image://docker.io/docker/dockerfile:experimental@sha256:de85b2  0.6s
 => => resolve docker.io/docker/dockerfile:experimental@sha256:de85b2f3a3  0.0s
 => => sha256:61261561661960014533790f5d6c42f5b88362db7e0 8.88MB / 8.88MB  0.2s
 => => sha256:de85b2f3a3e8a2f7fe48e8e84a65f6fdd5cd5183afa 1.69kB / 1.69kB  0.0s
 => => sha256:8c69d118cfcd040a222bea7f7d57c6156faa938cb61b476 521B / 521B  0.0s
 => => sha256:08fb2b2ca3d58e19d791e73dea16126df37608115532e74 897B / 897B  0.0s
 => => extracting sha256:61261561661960014533790f5d6c42f5b88362db7e005f35  0.3s
[+] Building 1.5s (4/4)                                                         
 => resolve image config for docker.io/docker/dockerfile:experimental      0.4s
 => docker-image://docker.io/docker/dockerfile:experimental@sha256:de85b2  0.6s
 => => resolve docker.io/docker/dockerfile:experimental@sha256:de85b2f3a3  0.0s
 => => sha256:61261561661960014533790f5d6c42f5b88362db7e0 8.88MB / 8.88MB  0.2s
 => => sha256:de85b2f3a3e8a2f7fe48e8e84a65f6fdd5cd5183afa 1.69kB / 1.69kB  0.0s
 => => sha256:8c69d118cfcd040a222bea7f7d57c6156faa938cb61b476 521B / 521B  0.0s
 => => sha256:08fb2b2ca3d58e19d791e73dea16126df37608115532e74 897B / 897B  0.0s
 => => extracting sha256:61261561661960014533790f5d6c42f5b88362db7e005f35  0.3s
[+] Building 1.6s (4/5)                                                         
 => docker-image://docker.io/docker/dockerfile:experimental@sha256:de85b2  0.6s
 => => resolve docker.io/docker/dockerfile:experimental@sha256:de85b2f3a3  0.0s
 => => sha256:61261561661960014533790f5d6c42f5b88362db7e0 8.88MB / 8.88MB  0.2s
 => => sha256:de85b2f3a3e8a2f7fe48e8e84a65f6fdd5cd5183afa 1.69kB / 1.69kB  0.0s
 => => sha256:8c69d118cfcd040a222bea7f7d57c6156faa938cb61b476 521B / 521B  0.0s
 => => sha256:08fb2b2ca3d58e19d791e73dea16126df37608115532e74 897B / 897B  0.0s
 => => extracting sha256:61261561661960014533790f5d6c42f5b88362db7e005f35  0.3s
 => [internal] load metadata for docker.io/15five/fifteen5:base            0.1s
[+] Building 1.8s (7/10)                                                        
 => => sha256:8c69d118cfcd040a222bea7f7d57c6156faa938cb61b476 521B / 521B  0.0s
 => => sha256:08fb2b2ca3d58e19d791e73dea16126df37608115532e74 897B / 897B  0.0s
 => => extracting sha256:61261561661960014533790f5d6c42f5b88362db7e005f35  0.3s
 => ERROR [internal] load metadata for docker.io/15five/fifteen5:base      0.1s
 => ERROR [python_backend 1/4] FROM docker.io/15five/fifteen5:base         0.1s
 => => resolve docker.io/15five/fifteen5:base                              0.1s
 => CANCELED [internal] load build context                                 0.1s
 => => transferring context: 144.35kB                                      0.1s
[+] Building 1.9s (7/10)                                                        
 => [internal] load build definition from Dockerfile.production            0.1s
 => => transferring dockerfile: 1.09kB                                     0.0s
 => [internal] load .dockerignore                                          0.1s
 => => transferring context: 1.61kB                                        0.0s
 => resolve image config for docker.io/docker/dockerfile:experimental      0.4s
 => docker-image://docker.io/docker/dockerfile:experimental@sha256:de85b2  0.6s
 => => resolve docker.io/docker/dockerfile:experimental@sha256:de85b2f3a3  0.0s
 => => sha256:61261561661960014533790f5d6c42f5b88362db7e0 8.88MB / 8.88MB  0.2s
 => => sha256:de85b2f3a3e8a2f7fe48e8e84a65f6fdd5cd5183afa 1.69kB / 1.69kB  0.0s
 => => sha256:8c69d118cfcd040a222bea7f7d57c6156faa938cb61b476 521B / 521B  0.0s
 => => sha256:08fb2b2ca3d58e19d791e73dea16126df37608115532e74 897B / 897B  0.0s
 => => extracting sha256:61261561661960014533790f5d6c42f5b88362db7e005f35  0.3s
 => ERROR [internal] load metadata for docker.io/15five/fifteen5:base      0.1s
 => ERROR [python_backend 1/4] FROM docker.io/15five/fifteen5:base         0.1s
 => => resolve docker.io/15five/fifteen5:base                              0.1s
 => CANCELED [internal] load build context                                 0.1s
 => => transferring context: 144.35kB                                      0.1s
------
 > [internal] load metadata for docker.io/15five/fifteen5:base:
------
------
 > [python_backend 1/4] FROM docker.io/15five/fifteen5:base:
------
failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: rpc error: code = Unknown desc = failed to build LLB: failed to load cache key: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Return code from docker build 1

Note that base image on circleci worked fine with buildkit enabled, so buildkit can work with circleci - it’s just fails in the situation above for some reason 🤔

caleb15 on Sep 22, 2020

As I wrote in https://github.com/moby/buildkit/issues/1142#issuecomment-524394554 moby/buildkit is made for oci worker. It does not come with containerd binaries. If you want to run containerd you can refer to their getting started guide https://github.com/containerd/containerd/blob/master/docs/getting-started.md#getting-started-with-containerd . If you have containerd running you can start buildkitd binary with configuration pointing to your containerd daemon. As containerd runs in host and requires clients to share the filesystem, normally you would run buildkitd in host as well in that case. There’s an old example of how to run them both in containers in https://github.com/moby/buildkit/pull/192#issue-156873378 but that’s probably not what you want.

I’m also freshly started with docker, but from the official documentation buildx is described as an improved version of build, but I’m a bit puzzled on why it can fail the very simple (and common I think) use case of creating an image starting from a base image already available locally under docker images.

buildx doesn’t have anything to do with the question in here that is about buildctl. in buildx results are automatically get/put to docker images with docker driver, and with container driver --load loads the build results into docker. buildx never interacts with containerd daemon, nor does running a containerd daemon have any influence on the output of docker images.

tonistiigi on Sep 23, 2019