containerd: Image pull fail unauthorized 401 with working credentials.
Description Containerd is failing to pull k8s control plane images from a private repo even though crictl is working fine. Our clusters are all on bare-metal and point to a private repo as part of the kubeadm-config we’re using. After configuring the auth plugin for the cri registry to point to our private repo, containerd complains that it isn’t able to pull down any of those images with error 401.
I’ve tried using username/password and auth, neither works.
Pulling down the image using crictl pull
with both --creds and --auth function perfect however.
Steps to reproduce the issue: Config.toml
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."my.private.repo.com:8444"]
endpoint = ["https://my.private.repo.com:8444"]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs.auths]
[plugins."io.containerd.grpc.v1.cri".registry.configs.auths."https://my.private.repo.com:8444"]
auth = "omitted"
Describe the results you received:
Dec 17 12:21:09 master1 containerd[2629]: time="2020-12-17T12:21:09.724485713-05:00" level=error msg="PullImage \"my.private.repo:8444/kube-apiserver:v1.18.8\" failed" error="failed to pull and unpack image \"my.private.repo:8444/kube-apiserver:v1.18.8\": failed to resolve reference \"my.private.repo:8444/kube-apiserver:v1.18.8\": unexpected status code [manifests v1.18.8]: 401 Unauthorized"
Dec 17 12:26:10 master1 containerd[2629]: time="2020-12-17T12:26:10.682165723-05:00" level=info msg="PullImage \"my.private.repo:8444/etcd:3.4.3-0\""
Dec 17 12:26:10 master1 containerd[2629]: time="2020-12-17T12:26:10.734521436-05:00" level=error msg="PullImage \"my.private.repo:8444/etcd:3.4.3-0\" failed" error="failed to pull and unpack image \"my.private.repo:8444/etcd:3.4.3-0\": failed to resolve reference \"my.private.repo:8444/etcd:3.4.3-0\": unexpected status code [manifests 3.4.3-0]: 401 Unauthorized"
Describe the results you expected:
Output of containerd --version
:
containerd github.com/containerd/containerd v1.4.2 b321d358e6eef9c82fa3f3bb8826dca3724c58c6
Any other relevant information:
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 4
- Comments: 21 (10 by maintainers)
Hi! I noticed the “help-wanted” label so I digged a bit into that. I found multiple references to “old syntaxes”:
Syntax 1 (oldest? https://github.com/containerd/cri/issues/835#issuecomment-403652902):
Syntax 2 (mentioned by OP, https://github.com/containerd/containerd/commit/050e8c63c0501dfc712ec8fec389212c14c852f9):
Syntax 3 (currently used):
I replicated locally the initial problem and the chunk of code you mentioned handles syntax 1 but not syntax 2. That explains why the warning has not been triggered.
Now I don’t know if the “syntax 2” is something we wish to handle (i.e. fire a warning/convert automatically to the “new” syntax) but I looked a bit into how we could:
https://github.com/containerd/containerd/blob/a72fe7da21237815731386d6b73a0e93700112f9/pkg/cri/config/config.go#L147-L160
configs.auths
is not mapped to anything because the current type specifications mapconfigs
to amap[string]RegistryConfig
. A dedicated struct containing amap[string]AuthConfig
would be needed for syntax 2. It looks painful, but I believe we could achieve that by mappingconfigs
to a toml primitive: then, we could try to map it consecutively agains syntaxes 2 and 3 and do our conversion things.If that’s something we wish to do, I would be glad to provide a patch @mikebrow 🙂
@cpuguy83 Ok, so I’ve configured a bare-metal kubeadm Kubernetes cluster with containerd as the runtime. I’ve configured this cluster to to pull it’s control plane images from a private repo - I did this by updating the kubeadm config with the following
imageRepository: my.private.repo
.Then, I configured the containerd registry and authentication (provided in my initial post) in the
/etc/containerd/config.toml
to make sure when the cluster is initialized or upgraded, it will leverage the credentials in theconfig.toml
along side the registry in thekubeadm-config
, to connect to my private repo and pull down the images as needed.Now, if I attempt to do any of the following: initialize a new cluster, upgrade an existing cluster, or to migrate an existing cluster from docker-ce runtime to the containerd runetime, I see errors in containerd telling me it’s not able to pull down the control plane images.
Specifically speaking, the errors containerd gives are:
My question is why? I’ve configured the
config.toml
with the proper credentials and I know for a fact that the credentials I’ve provided work.