buildx: Unable to pull from insecure registry

I think I am still missing something.

One the machine that hosts the registry I have this buildkit.toml (192.168.189.102 is that machines IP address – should be roughly equivalent to 127.0.1.1)

debug = true
insecure-entitlements = [ "network.host", "security.insecure" ]

[registry."192.168.189.102:5000"]
  http = true
  insecure = true

I think create a buildx instance as

docker buildx rm buildx_instance && docker buildx create --name buildx_instance --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=-1 --driver-opt env.BUILDKIT_STEP_LOG_MAX_SPEED=-1 --config docker/buildkitd.toml && docker buildx use buildx_instance

and docker buildx inpsect buildx_instance shows

Name:   buildx_instance
Driver: docker-container

Nodes:
Name:           buildx_instance0
Endpoint:       unix:///var/run/docker.sock
Driver Options: env.BUILDKIT_STEP_LOG_MAX_SIZE="-1" env.BUILDKIT_STEP_LOG_MAX_SPEED="-1"
Status:         running
Buildkit:       v0.10.5
Platforms:      linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/386

and then building the image with this command

docker buildx build --output type=image,\"name=192.168.189.102:5000/image:test,192.168.189.102:5000/image:test_cache\",push=true -t 192.168.189.102:5000/image:test --file docker/Dockerfile --pull --build-arg platform=generic --cache-from type=registry,ref=192.168.189.102:5000/image:test_cache --cache-to type=registry,ref=192.168.189.102:5000/image:test_cache,mode=max docker

shows

 => [internal] load .dockerignore                                                                                                                                                                                                        0.0s
 => => transferring context: 2B                                                                                                                                                                                                          0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                     0.0s
 => => transferring dockerfile: 17.51kB                                                                                                                                                                                                  0.0s
 => resolve image config for docker.io/docker/dockerfile:1.3                                                                                                                                                                             6.0s
 => CACHED docker-image://docker.io/docker/dockerfile:1.3@sha256:42399d4635eddd7a9b8a24be879d2f9a930d0ed040a61324cfdf59ef1357b3b2                                                                                                        0.0s
 => => resolve docker.io/docker/dockerfile:1.3@sha256:42399d4635eddd7a9b8a24be879d2f9a930d0ed040a61324cfdf59ef1357b3b2                                                                                                                   0.0s
 => [internal] load metadata for docker.io/library/archlinux:base-devel-20220710.0.67642                                                                                                                                                 0.3s
 => importing cache manifest from 192.168.189.102:5000/image:test_cache                                                                                                                                                             0.0s
 => [internal] load build context                                                                                                                                                                                                        0.0s
 => => transferring context: 1.33kB  
=========== SNIP (all layers cached) ===========
 => exporting to image                                                                                                                                                                                                                   0.1s
 => => exporting layers                                                                                                                                                                                                                  0.0s
 => => exporting manifest sha256:2f6d120f45bd4fd1b5123a0e039e036112dcbf72377352739f9b4c7ee97bdd5b                                                                                                                                        0.0s
 => => exporting config sha256:f0d3b939ab9782a6f877ca7ad7a2c9d2134ba9da036e340f40e7b8d58ffa26d1                                                                                                                                          0.0s
 => => pushing layers                                                                                                                                                                                                                    0.1s
 => => pushing manifest for 192.168.189.102:5000/image:test@sha256:2f6d120f45bd4fd1b5123a0e039e036112dcbf72377352739f9b4c7ee97bdd5b                                                                                                 0.0s
 => exporting cache                                                                                                                                                                                                                     12.7s
 => => preparing build cache for export         
=========== SNIP (=> => writing layer sha256:{many shas}) ===========        
 => => writing config sha256:922f75492f9fce1124446fd27e4fee2f93f05022001fa150614e9a8bb57dc47e                                                                                                                                            0.0s
 => => writing manifest sha256:b9341065e8df4dbe07b712101bea55ec3ab6bed200f07bf48db18b3e14a4c2c6   

However, when I try to build the same image on a different machine on the same network (192.168.189.102 is accessible from this machine), I get

 => ERROR importing cache manifest from 192.168.189.102:5000/image:test_cache

and the logs for the buildx container shows

time="2022-10-24T22:41:46Z" level=debug msg=resolving host="192.168.189.102:5000"
time="2022-10-24T22:41:46Z" level=debug msg="do request" host="192.168.189.102:5000" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=buildkit/v0.10 request.method=HEAD url="http://192.168.189.102:5000/v2/image/manifests/test_cache"
time="2022-10-24T22:41:46Z" level=debug msg="fetch response received" host="192.168.189.102:5000" response.header.content-length=100 response.header.content-type="application/json; charset=utf-8" response.header.date="Mon, 24 Oct 2022 22:41:46 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.x-content-type-options=nosniff response.status="404 Not Found" url="http://192.168.189.102:5000/v2/image/manifests/test_cache"
time="2022-10-24T22:41:46Z" level=info msg="trying next host - response was http.StatusNotFound" host="192.168.189.102:5000"
time="2022-10-24T22:41:46Z" level=debug msg=resolving host="192.168.189.102:5000"
time="2022-10-24T22:41:46Z" level=debug msg="do request" host="192.168.189.102:5000" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=buildkit/v0.10 request.method=HEAD url="https://192.168.189.102:5000/v2/image/manifests/test_cache"
time="2022-10-24T22:41:46Z" level=info msg="trying next host" error="failed to do request: Head \"https://192.168.189.102:5000/v2/image/manifests/test_cache\": http: server gave HTTP response to HTTPS client" host="192.168.189.102:5000"
time="2022-10-24T22:41:46Z" level=debug msg="error while importing cache manifest from cmId=192.168.189.102:5000/image:test_cache: failed to do request: Head \"https://192.168.189.102:5000/v2/image/manifests/test_cache\": http: server gave HTTP response to HTTPS client"

So it appears that the buildkit config worked, but it can’t find the manifest for test_cache even though the registry machine could find it? I also tried to docker push 192.168.189.102:5000/image:test_cache (in case it was built on the registry machine but not pushed to the registry) but it says

The push refers to repository [192.168.189.102:5000/image]
An image does not exist locally with the tag: 192.168.189.102:5000/image

---

On another note, does docker/buildx look at /etc/hosts on the host machine? I have entries in there so I can use a name rather than the IP address, but whenever I use the name in the docker buildx build command line I get errors saying that a lookup on 1.1.1.1 for the name failed.

When using a docker-container builder you have to set the registry configuration for the BuildKit daemon: https://github.com/docker/buildx/blob/master/docs/guides/custom-registry-config.md

In your case the configuration will look like this:

[registry."192.168.189.102:5000"]
http = true
insecure = true

@tonistiigi @jedevc I wonder if could read DockerAPI.Info(ctx).RegistryConfig.InsecureRegistryCIDRs and automatically set the registry config (if not already populated) in the container when creating a docker-container builder?