spegel: Removal of certs in containerd configuration causing 500 error (mirror resolve retries exhausted for key)
Spegel version
v0.0.19
Kubernetes distribution
EKS
Kubernetes version
v1.27.0
CNI
VPC CNI
Describe the bug
We are using jfrog artifactory registry as our registry. we are using tls certificate authentication for resolving and accessing our registry endpoints. We were facing 500 error with mirror resolve retries exhausted for key while running a pod, it gave this error
│ Type Reason Age From Message │
│ ---- ------ ---- ---- ------- │
│ Normal Scheduled 29m default-scheduler Successfully assigned large-pod-1 to ip-10-190-18-201.eu-west-1.compute.internal │
│ Normal Pulling 27m (x4 over 29m) kubelet Pulling image "xxx" │
│ Warning Failed 27m (x4 over 29m) kubelet Failed to pull image "xxx": rpc error: code = Unknown desc = failed to pull and unpack image "xxx": failed to resolve reference "xxx": unexpected status from HEAD request to http://10.190.18.201:30020/v2/xxx 500 Internal Server Error │
│ Warning Failed 27m (x4 over 29m) kubelet Error: ErrImagePull │
│ Warning Failed 27m (x6 over 29m) kubelet Error: ImagePullBackOff │
│ Normal BackOff 4m6s (x109 over 29m) kubelet Back-off pulling image xxx" │
│
On diving deep we saw that our client.cert and client.key files are not loaded in the configuration
configuration {"level":"info","ts":1712209973.4201605,"caller":"oci/containerd.go:511","msg":"backing up Containerd host configuration","path":"/etc/containerd/certs.d/xxx"} │
│ configuration {"level":"info","ts":1712209973.420308,"caller":"oci/containerd.go:564","msg":"added containerd mirror configuration","registry":"https://xxx","path":"/etc/containerd/certs.d/xxx/hosts.toml" │
│ configuration {"level":"info","ts":1712209973.4203854,"caller":"oci/containerd.go:564","msg":"added containerd mirror configuration","registry":"xxx","path":"/etc/containerd/certs.d/xxx/hosts.toml" │
│ configuration {"level":"info","ts":1712209973.4204602,"caller":"oci/containerd.go:564","msg":"added containerd mirror configuration","registry":"xxx","path":"/etc/containerd/certs.d/xxx/hosts.toml"} │
│ configuration {"level":"info","ts":1712209973.4205267,"caller":"build/main.go:87","msg":"gracefully shutdown"}
only host.toml seems to be loaded to spegel. Therefore not being able to access artifactory and pull.
Sharing the spegel log file for reference. Can you please ensure that the all the files in cert configuration is copied in the spegel configuration?
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 34 (15 by maintainers)
@badaldavda8 @IndhumithaR I have cut v0.0.21 now which contains an option to enable merging. As long as you are not placing certificates in certs.d directory I think this would work for you.