moby: Buildkit in docker 18.09 won't allow insecure registry with self-signed TLS

same issue with docker 20.10 if registry work on http://repo.mycompany.com,but buildkit pull https://repo.mycompany.com, notice the insecure-registries

{
  "insecure-registries" : ["http://repo.mycompany.com"],
  "features":{
    "buildkit": true
  }
}

It works for me.

We are using private registries and a registry mirror on private domains with plain http. Will there be a fix for buildkit supporting plain http registries in the future?

Initially the registry was using a self-signed certificate, so I had to declare it as insecure. Then I switched to letsencrypt and it wouldn’t work without that setting. Then I followed the documentation to add letsencrypt certificates in /etc/docker/certs.d/, and I am finally able to remove this setting and use my “secure” registry with the “default” builder. Still, it doesn’t solve the issues with buildkit (see below).

Our “private mirror” is an instance of Sonatype Nexus 3 configured as a proxy repository of docker.io. We also have a hosted private repository and both are in a single group. URL https://repo.mycompany.com/ points to this group. It is used in daemon.json in the registry-mirrors entry.

This morning I’m testing with the following configuration in daemon.json :

{
  "registry-mirrors": ["https://repo.mycompany.com"],
  "features":{
    "buildkit": true
  }
}

This is the output I’m getting for docker build . (same Dockerfile as above):

[+] Building 0.3s (3/3) FINISHED
 => [internal] load build definition from Dockerfile
 => => transferring dockerfile: 238B 
 => [internal] load .dockerignore 
 => => transferring context: 2B 
 => ERROR resolve image config for docker.io/docker/dockerfile:1.0-experimental
------
 > resolve image config for docker.io/docker/dockerfile:1.0-experimental:
------
docker.io/docker/dockerfile:1.0-experimental not found

(I’ve also tried with docker/dockerfile:experimental, same result)

However, docker pull docker/dockerfile:1.0-experimental succeeds… So after pulling the “experimental” image, I retried and the build goes further, but fails with a similar error:

------
 > [internal] load metadata for docker.io/library/maven:3.5.3-alpine:
------
------
 > [1/4] FROM docker.io/library/maven:3.5.3-alpine:
------
rpc error: code = Unknown desc = docker.io/library/maven:3.5.3-alpine not found

If I understand correctly:

• buildkit has a problem when the insecure-registry setting is used, the workaround is to configure the certificates properly to use the registry as a secure one
• even with a secure registry, buildkit fails to fetch the images when the registry-mirrors setting is used (in my case, Sonatype Nexus). The strange thing is that a docker pull of these same images succeeds : it seems that buildkit is not pulling images the same way the cli does ??

[EDIT] I also tested removing the registry-mirrors parameter. In this case, building an image referencing images from docker.io only works (ex FROM maven). But if I am referencing my private registry (FROM repo.mycompany.com/whatever/image) I am getting the same rpc error: code = Unknown desc = repo.mycompany.com/whatever/image not found

When using docker buildkit to build images, if you don’t want to configure the CA certificate, you can modify the docker configuration file (/etc/docker/daemon.json) to make it work properly. You can follow these steps: 1)：edit /etc/docker/daemon.json (#Add the prefix ‘http://’ in front of ‘myregistry.domain.com’)

{
  "insecure-registries": ["http://myregistry.domain.com"]
}

2): save json file, and restart docker service:

service docker restart 
#or 
systemctl restart docker

Updated workaround: adding the cert used on my Artifactory server to the docker host works with buildkit. That is cp <cert> /usr/local/share/ca-certificates/ update-ca-certificates systemctl restart docker works just fine.

I’ve made progress.

My problem seems to be caused by two simultaneous issues:

1. Buildkit has issues with insecure registries, as stated above by @tonistiigi . Solution: secure the registry
2. Buildkit can’t fetch images from private registry hosted on Sonatype Nexus. I’ve eventually found that Buildkit issues a HEAD request to the registry before fetching the images, which a regular “docker pull” doesn’t do. This triggers this Sonatype Nexus bug: https://issues.sonatype.org/browse/NEXUS-12684. Solution : upgrade Nexus to version > 3.15

With these two changes in place, I am now able to use buildkit. 😃

Maybe it’d be worth to mention the Nexus issue somewhere in Docker’s documentation ? (how about here : https://docs.docker.com/develop/develop-images/build_enhancements/ ? I’ll prepare a PR)

Then, if there is another ticket for the issue with insecure registries, I think we could close this issue.

@thaJeztah Nope, it doesn’t work either. The error is slightly different though 😉 - see in the log how it is using https despite the mirror address is configured as HTTP ?

me@pic:~/test$ docker build .
[+] Building 0.1s (3/3) FINISHED
 => [internal] load .dockerignore
 => => transferring context: 2B
 => [internal] load build definition from Dockerfile
 => => transferring dockerfile: 238B
 => ERROR resolve image config for docker.io/docker/dockerfile:1.0-experimental
------
 > resolve image config for docker.io/docker/dockerfile:1.0-experimental:
------
failed to do request: Head https://repo.mycompany.com:80/v2/docker/dockerfile/manifests/1.0-experimental: http: server gave HTTP response to HTTPS client
me@pic:~/test$ sudo cat /etc/docker/daemon.json
{
  "registry-mirrors": ["http://repo.mycompany.com:80"],
  "insecure-registries" : ["repo.mycompany.com"],
  "features":{
    "buildkit": true
  }
} 

(if I remove the “insecure-registries” entry, restart docker, and try again, I’m getting the same error)