spegel: Getting high volumes of 404 and 500 errors. Any insight?

Spegel version

v0.0.22

Kubernetes distribution

EKS

Kubernetes version

1.29.0

CNI

AWS VPC CNI

Describe the bug

UPDATE 1: After some further investigation, it seems that adding my team’s internal ECR registry caused a rise in errors and performance to degrade a bit (below). Seems like things aren’t going as they should be… wondering what’s happening and how to fix…🤔


Hello, apologies if this is very simple. I’m new to this tool. I just deployed Spegel after modifying the containerd settings in my EKS cluster according to the compatibility guide here: https://github.com/spegel-org/spegel/blob/main/docs/COMPATIBILITY.md.

The pods are running so I think on some levels it is working according to https://github.com/spegel-org/spegel/blob/main/docs/FAQ.md but I noticed that when a new node comes up I am getting a lot of 404/500 errors which leads to believe the mirroring isn’t working as expected. I’m not familiar with what these errors indicate. Is this normal?

I did find https://github.com/spegel-org/spegel/issues/217 but not sure if it’s related. Using containerd config dump, confirmed the settings should be ok.

sh-4.2$ containerd config dump | grep config_path -B 5

    [plugins."io.containerd.grpc.v1.cri".image_decryption]
      key_model = "node"

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = "/etc/containerd/certs.d:/etc/docker/certs.d"
sh-4.2$ containerd config dump | grep discard_unpacked_layers -B 5
      setup_serially = false

    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"
      disable_snapshot_annotations = true
      discard_unpacked_layers = false

Any insight is much appreciated! Thank you. 😄

Error 1

{"time":"2024-05-08T20:28:40.362320827Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":135},"msg":"","err":"content digest sha256:1e3d9b7d145208fa8fa3ee1c9612d0adaac7255f1bbc9ddea7e461e0b317805c: not found","path":"/v2/jimmidyson/configmap-reload/blobs/sha256:1e3d9b7d145208fa8fa3ee1c9612d0adaac7255f1bbc9ddea7e461e0b317805c","status":500,"method":"GET","latency":"574.095µs","ip":"<some-ip>"}

Error 2

{"time":"2024-05-08T20:28:39.25413767Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handleMirror.func1","file":"/build/pkg/registry/registry.go","line":272},"msg":"proxy failed attempting next","key":"sha256:fe5ca62666f04366c8e7f605aa82997d71320183e99962fa76b3209fdfbb8b58","path":"/v2/jimmidyson/configmap-reload/blobs/sha256:fe5ca62666f04366c8e7f605aa82997d71320183e99962fa76b3209fdfbb8b58","ip":"<some-ip>","err":"expected mirror to respond with 200 OK but received: 500 Internal Server Error"}

Error 3

{"time":"2024-05-08T20:28:36.85743246Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).handle.func1","file":"/build/pkg/registry/registry.go","line":135},"msg":"","err":"request closed for key: sha256:566294b429823a40b102147b6742390ede89f1f30f9dd6c1f742aae4defe4504","path":"/v2/jetstack/cert-manager-webhook/blobs/sha256:566294b429823a40b102147b6742390ede89f1f30f9dd6c1f742aae4defe4504","status":404,"method":"GET","latency":"5.021653989s","ip":"<some-ip>"}

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 17 (5 by maintainers)

Most upvoted comments

registry.k8s.io does not offer a fixed list of backend - if you need to filter aggressively, you need to pull from a mirror you control.

I will start off by clarifying something about the Spegel. It is a best effort cache, meaning that images may or may not exist on other nodes when and image is pulled. On top of that images may not be discovered due to the distributed nature of the DHT.

Having said that I do not think that this is an issue that you are facing. I made some changes before v0.0.22 to make sure that 404 are returned when an image is not found and 500 errors are returned for other types of errors. The error content digest sha256:ce9698c36317f0d048a2e692511e8a1ec371fa047493b809f71efe3af526b366: not found that you are seeing is interesting however. This means that traffic is being routed to the node but the node is not able to find the layer. It is probably an indication of a larger problem as the layer should not have been advertised if it is not present on the node.

I saw that you wrote that the errors seems to come specifically from images pulled from ECR. I am not sure why this would be the case, but there might be something specific that is done when EKS pulls images from ECR?

Could you share your values configuration used to deploy Spegel?