containerd: registry mirroring using auth on incorrect registry

Description

We have configured registry-mirroring to first attempt to pull an image from dockerhub and if its not there pull from our private registry which requires credentials.

We now observe that containerd appears to be using the credentials for dockerhub as well on the initial pull.

Steps to reproduce the issue:

Update the config.toml to include

    [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
        endpoint = ["https://registry-1.docker.io", "https://appconnect-docker-local.artifactory.swg-devops.com"] 
    [plugins."io.containerd.grpc.v1.cri".registry.configs]
      [plugins."io.containerd.grpc.v1.cri".registry.configs."appconnect-docker-local.artifactory.swg-devops.com".auth]
        auth = "removed"

Attempt to start a container which requires pulling from dockerhub 3.

Describe the results you received:

5s          Warning   Failed                    pod/nsenter-kyo8lv                    Failed to pull image "docker.io/library/alpine": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/alpine:latest": failed to resolve reference "docker.io/library/alpine:latest": failed to authorize: failed to fetch oauth token: unexpected status: 401 Unauthorized

Describe the results you expected:

I expect it to be able to pull images from dockerhub without a password and only fall back to the secured registry if its not on dockerhub.

Output of containerd --version:

containerd github.com/containerd/containerd 1.4.3+azure 269548fa27e0089a8b8278fc4fc781d7f65a939b

Any other relevant information:

About this issue

Original URL
State: open
Created 3 years ago
Comments: 16 (4 by maintainers)

Most upvoted comments

@ericazhao-china hi,is there so comfig in new version like this

  [plugins."io.containerd.grpc.v1.cri".registry.configs]
    [plugins."io.containerd.grpc.v1.cri".registry.configs."registry-1.docker.io".auth]
      username = "<username>"
      password = "<password>"

i used containerd 1.6.4 and config like this https://github.com/containerd/containerd/blob/main/docs/hosts.md,but don’t found how to config username & password to auth dockerhub.

lixd on Sep 1, 2022

I have a similar problem, it’s reproducible even if the private registry is not accessible. Tested on containerd 1.4.4

$ containerd -v
containerd containerd.io 1.4.4 05f951a3781f4f2c1911b05e61c160e9c30eaa8e

[plugins.cri.registry]
[plugins.cri.registry.mirrors]
[plugins.cri.registry.mirrors."docker.io"]
  endpoint = ["http://127.0.0.1:5000","https://registry-1.docker.io"]

[plugins.cri.registry.configs]
[plugins.cri.registry.configs."127.0.0.1:5000".auth]
  auth = "foo"

$ crictl pull docker.io/library/busybox
FATA[0002] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/busybox:latest": failed to resolve reference "docker.io/library/busybox:latest": failed to do request: Head http://127.0.0.1:5000/v2/library/busybox/manifests/latest?ns=docker.io: dial tcp 127.0.0.1:5000: connect: connection refused

Remove entire [plugins.cri.registry.configs] or change config.toml to, then it works:

[plugins.cri.registry]
[plugins.cri.registry.mirrors]
[plugins.cri.registry.mirrors."docker.io"]
  endpoint = ["http://127.0.0.1:5000","https://registry-1.docker.io"]

[plugins.cri.registry.configs]
[plugins.cri.registry.configs."127.0.0.1:5000".auth]
  auth = ""

$ crictl pull docker.io/library/busybox
Image is up to date for sha256:388056c9a6838deea3792e8f00705b35b439cf57b3c9c2634fb4e95cfc896de6

azuwis on Apr 8, 2021