spegel: `verifyStatusResponse` does not work with containerd v2

Spegel version

v0.0.24

Kubernetes distribution

Talos

Kubernetes version

v1.31.0

CNI

Cilium

Describe the bug

After updating my homelab cluster to Talos v1.8.0-beta0, I noticed that the spegel pod no longer launched. The logs just say

{"time":"2024-09-10T07:29:25.425786168Z","level":"ERROR","source":{"function":"main.main","file":"/build/main.go","line":89},"msg":"run exit with error","err":"Containerd registry config path needs to be set for mirror configuration to take effect"}

before the program terminates. Looking at the code, that’s pkg.oci.containerd verifyStatusResponse() failing its check for ConfigPath in the received containerd config.

https://github.com/spegel-org/spegel/blob/main/pkg/oci/containerd.go#L126

I have a feeling Talos is not configuring containerd correctly, but the config file reads:

version = 3

[plugins]
  [plugins.'io.containerd.cri.v1.images']
    discard_unpacked_layers = false

    [plugins.'io.containerd.cri.v1.images'.registry]
      config_path = '/etc/cri/conf.d/hosts'

which according to https://github.com/containerd/containerd/blob/main/docs/cri/config.md is the correct way in 2.x.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 4
  • Comments: 15 (9 by maintainers)

Commits related to this issue

Most upvoted comments

@phillebaba any chance that PR can be looked at? Talos 1.8.0 has packaged containerd 2.0.0-rc.4.

I have been busy with work lately so have not had time for Spegel. I would rather we fix the underlying problem rather than just ignore verification if the version is Containerd v2. I will have a look and see if this can be solved some other way.

I have spent some time looking at alternative methods of verifying the configuration without much luck. I have created an issue in Containerd https://github.com/containerd/containerd/issues/10780 for long term work. In the meantime I will work to merge #581 to disable verification for Containerd v2.

It would be nice to start testing Containerd v2 so Talos may be a good candidate to run e2e tests.

@aki263 for now I am using this image in the helm values

image:
  repository: ghcr.io/deedee-ops/spegel
  tag: 0.0.24
  digest: ""

I have cherry-picked the patch of PR #581 into the v0.0.24 release tag and built the container image locally. Pushed it to my own registry and configured the helm chart to use my custom image instead. With the patch applied, spegel works perfectly again on Talos 1.8.0.