containerd: Image pull fail unauthorized 401 with working credentials.

Hi! I noticed the “help-wanted” label so I digged a bit into that. I found multiple references to “old syntaxes”:

Syntax 1 (oldest? https://github.com/containerd/cri/issues/835#issuecomment-403652902):

[plugins.cri.registry.auths]
  [plugins.cri.registry.auths."https://my.private.repo:8444"]
    auth = "omitted"

Syntax 2 (mentioned by OP, https://github.com/containerd/containerd/commit/050e8c63c0501dfc712ec8fec389212c14c852f9):

[plugins."io.containerd.grpc.v1.cri".registry.configs]
  [plugins."io.containerd.grpc.v1.cri".registry.configs.auths]
    [plugins."io.containerd.grpc.v1.cri".registry.configs.auths."https://my.private.repo:8444"]
      auth = "omitted"

Syntax 3 (currently used):

[plugins."io.containerd.grpc.v1.cri".registry.configs]
  [plugins."io.containerd.grpc.v1.cri".registry.configs."my.private.repo:8444".auth]
    auth = "omitted"

I replicated locally the initial problem and the chunk of code you mentioned handles syntax 1 but not syntax 2. That explains why the warning has not been triggered.

Now I don’t know if the “syntax 2” is something we wish to handle (i.e. fire a warning/convert automatically to the “new” syntax) but I looked a bit into how we could:

https://github.com/containerd/containerd/blob/a72fe7da21237815731386d6b73a0e93700112f9/pkg/cri/config/config.go#L147-L160

configs.auths is not mapped to anything because the current type specifications map configs to a map[string]RegistryConfig. A dedicated struct containing a map[string]AuthConfig would be needed for syntax 2. It looks painful, but I believe we could achieve that by mapping configs to a toml primitive: then, we could try to map it consecutively agains syntaxes 2 and 3 and do our conversion things.

If that’s something we wish to do, I would be glad to provide a patch @mikebrow 🙂

@cpuguy83 Ok, so I’ve configured a bare-metal kubeadm Kubernetes cluster with containerd as the runtime. I’ve configured this cluster to to pull it’s control plane images from a private repo - I did this by updating the kubeadm config with the following imageRepository: my.private.repo.

Then, I configured the containerd registry and authentication (provided in my initial post) in the /etc/containerd/config.toml to make sure when the cluster is initialized or upgraded, it will leverage the credentials in the config.toml along side the registry in the kubeadm-config, to connect to my private repo and pull down the images as needed.

Now, if I attempt to do any of the following: initialize a new cluster, upgrade an existing cluster, or to migrate an existing cluster from docker-ce runtime to the containerd runetime, I see errors in containerd telling me it’s not able to pull down the control plane images.

Specifically speaking, the errors containerd gives are:

Dec 17 12:21:09 master1 containerd[2629]: time="2020-12-17T12:21:09.724485713-05:00" level=error msg="PullImage \"my.private.repo:8444/kube-apiserver:v1.18.8\" failed" error="failed to pull and unpack image \"my.private.repo:8444/kube-apiserver:v1.18.8\": failed to resolve reference \"my.private.repo:8444/kube-apiserver:v1.18.8\": unexpected status code [manifests v1.18.8]: 401 Unauthorized"
Dec 17 12:26:10 master1 containerd[2629]: time="2020-12-17T12:26:10.682165723-05:00" level=info msg="PullImage \"my.private.repo:8444/etcd:3.4.3-0\""
Dec 17 12:26:10 master1 containerd[2629]: time="2020-12-17T12:26:10.734521436-05:00" level=error msg="PullImage \"my.private.repo:8444/etcd:3.4.3-0\" failed" error="failed to pull and unpack image \"my.private.repo:8444/etcd:3.4.3-0\": failed to resolve reference \"my.private.repo:8444/etcd:3.4.3-0\": unexpected status code [manifests 3.4.3-0]: 401 Unauthorized"

My question is why? I’ve configured the config.toml with the proper credentials and I know for a fact that the credentials I’ve provided work.