spegel: Removal of certs in containerd configuration causing 500 error (mirror resolve retries exhausted for key)

Spegel version

v0.0.19

Kubernetes distribution

EKS

Kubernetes version

v1.27.0

CNI

VPC CNI

Describe the bug

We are using jfrog artifactory registry as our registry. we are using tls certificate authentication for resolving and accessing our registry endpoints. We were facing 500 error with mirror resolve retries exhausted for key while running a pod, it gave this error

│   Type     Reason     Age                   From               Message                                                                                                                                                                                                 │
│   ----     ------     ----                  ----               -------                                                                                                                                                                                                 │
│   Normal   Scheduled  29m                   default-scheduler  Successfully assigned large-pod-1 to ip-10-190-18-201.eu-west-1.compute.internal                                                                                                                    │
│   Normal   Pulling    27m (x4 over 29m)     kubelet            Pulling image "xxx"                                                                                                  │
│   Warning  Failed     27m (x4 over 29m)     kubelet            Failed to pull image "xxx": rpc error: code = Unknown desc = failed to pull and unpack image "xxx": failed to resolve reference "xxx": unexpected status from HEAD request to http://10.190.18.201:30020/v2/xxx 500 Internal Server Error                                                                                                                                                 │
│   Warning  Failed     27m (x4 over 29m)     kubelet            Error: ErrImagePull                                                                                                                                                                                     │
│   Warning  Failed     27m (x6 over 29m)     kubelet            Error: ImagePullBackOff                                                                                                                                                                                 │
│   Normal   BackOff    4m6s (x109 over 29m)  kubelet            Back-off pulling image xxx"                                                                                         │
│                                                                                                                                                                  

On diving deep we saw that our client.cert and client.key files are not loaded in the configuration

configuration {"level":"info","ts":1712209973.4201605,"caller":"oci/containerd.go:511","msg":"backing up Containerd host configuration","path":"/etc/containerd/certs.d/xxx"}                                                            │
│ configuration {"level":"info","ts":1712209973.420308,"caller":"oci/containerd.go:564","msg":"added containerd mirror configuration","registry":"https://xxx","path":"/etc/containerd/certs.d/xxx/hosts.toml" │
│ configuration {"level":"info","ts":1712209973.4203854,"caller":"oci/containerd.go:564","msg":"added containerd mirror configuration","registry":"xxx","path":"/etc/containerd/certs.d/xxx/hosts.toml" │
│ configuration {"level":"info","ts":1712209973.4204602,"caller":"oci/containerd.go:564","msg":"added containerd mirror configuration","registry":"xxx","path":"/etc/containerd/certs.d/xxx/hosts.toml"}               │
│ configuration {"level":"info","ts":1712209973.4205267,"caller":"build/main.go:87","msg":"gracefully shutdown"}                     

only host.toml seems to be loaded to spegel. Therefore not being able to access artifactory and pull.

first.log

Sharing the spegel log file for reference. Can you please ensure that the all the files in cert configuration is copied in the spegel configuration?

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 34 (15 by maintainers)

Most upvoted comments

@badaldavda8 @IndhumithaR I have cut v0.0.21 now which contains an option to enable merging. As long as you are not placing certificates in certs.d directory I think this would work for you.