spegel: "could not get peer id" and timeouts since 0.0.29

Spegel version

v0.0.30

Kubernetes distribution

Talos 1.9.1

Kubernetes version

v1.31.4

CNI

Cilium

Describe the bug

Since v0.0.29 with the new peer discovery, I am seeing errors in spegel logs and 60s delays whenever a container needs to fetch an image (basically a timeout, after which the image is quickly fetched from upstream). I am assuming this means spegel is basically in a bad state.

I haven’t changed my CNI, Kubernetes, or Talos versions in-between those spegel versions, and I haven’t changed my spegel settings.

My cluster is dual-stack (v4 and v6).

I use the helm chart for installation using the following values:

serviceMonitor:
  enabled: true
grafanaDashboard:
  enabled: true
spegel:
  containerdRegistryConfigPath: /etc/cri/conf.d/hosts

Logs:

{"time":"2025-01-22T06:39:54.884189668Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/oci.(*Containerd).Verify","file":"/build/pkg/oci/containerd.go","line":118},"msg":"unable to verify status response","runtime_version":"2.0.1"}
{"time":"2025-01-22T06:39:54.901603359Z","level":"INFO","source":{"function":"main.registryCommand","file":"/build/main.go","line":212},"msg":"running Spegel","registry":":5000","router":":5001"}
{"time":"2025-01-22T06:39:54.90170104Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/routing.(*P2PRouter).Run","file":"/build/pkg/routing/p2p.go","line":111},"msg":"starting p2p router","logger":"p2p","id":"/ip6/2001:5a8:42a6:e2fb::f95/tcp/5001/p2p/12D3KooWLPhmVFbJie7ua7juDgqrcH123CTRSeFV2iMNJhM27dYt"}
{"time":"2025-01-22T06:39:54.902058451Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":34},"msg":"running scheduled image state update"}
{"time":"2025-01-22T06:39:59.90690095Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/routing.bootstrapFunc.func1","file":"/build/pkg/routing/p2p.go","line":286},"msg":"could not get peer id","logger":"p2p","err":"failed to dial: failed to dial 92B: all dials failed\n  * [/ip4/10.11.1.204/tcp/5001] dial tcp4 0.0.0.0:5001->10.11.1.204:5001: i/o timeout"}
{"time":"2025-01-22T06:39:59.912454683Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/routing.bootstrapFunc.func1","file":"/build/pkg/routing/p2p.go","line":286},"msg":"could not get peer id","logger":"p2p","err":"failed to dial: failed to dial 92B: all dials failed\n  * [/ip4/10.11.3.186/tcp/5001] dial to self attempted\n  * [/ip4/10.11.1.204/tcp/5001] dial backoff\n  * [/ip4/10.11.2.154/tcp/5001] dial backoff"}
{"time":"2025-01-22T06:39:59.920541642Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/routing.bootstrapFunc.func1","file":"/build/pkg/routing/p2p.go","line":286},"msg":"could not get peer id","logger":"p2p","err":"failed to dial: failed to dial 92B: all dials failed\n  * [/ip4/10.11.3.186/tcp/5001] dial to self attempted\n  * [/ip4/10.11.1.204/tcp/5001] dial backoff\n  * [/ip4/10.11.2.154/tcp/5001] dial backoff"}
{"time":"2025-01-22T06:39:59.920706382Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/routing.bootstrapFunc.func1","file":"/build/pkg/routing/p2p.go","line":286},"msg":"could not get peer id","logger":"p2p","err":"failed to dial: failed to dial 92B: all dials failed\n  * [/ip4/10.11.3.186/tcp/5001] dial to self attempted\n  * [/ip4/10.11.1.204/tcp/5001] dial backoff\n  * [/ip4/10.11.2.154/tcp/5001] dial backoff"}
{"time":"2025-01-22T06:39:59.920869343Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/routing.bootstrapFunc.func1","file":"/build/pkg/routing/p2p.go","line":286},"msg":"could not get peer id","logger":"p2p","err":"failed to dial: failed to dial 92B: all dials failed\n  * [/ip4/10.11.3.186/tcp/5001] dial to self attempted\n  * [/ip4/10.11.1.204/tcp/5001] dial backoff\n  * [/ip4/10.11.2.154/tcp/5001] dial backoff"}
{"time":"2025-01-22T06:39:59.920891363Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/routing.bootstrapFunc.func1","file":"/build/pkg/routing/p2p.go","line":293},"msg":"no bootstrap nodes found","logger":"p2p"}
{"time":"2025-01-22T06:48:54.903045168Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":34},"msg":"running scheduled image state update"}
{"time":"2025-01-22T06:57:54.902146115Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":34},"msg":"running scheduled image state update"}
{"time":"2025-01-22T07:06:54.902113162Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":34},"msg":"running scheduled image state update"}

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 3
  • Comments: 15 (11 by maintainers)

Most upvoted comments

@jfroy I have confirmed that dual stack networking will not function properly with the current version of Spegel. It worked before mostly by mistake, rather than due to any effort by me. I do not have that much experience with dual stack clusters so I am trying to understand the purpose and best practices.

I think that in theory both IPv4 and IPv6 could be run at the same time with some modifications. The reality however is that the Containerd configuration will only support a either IPv4 or IPv6 addresses, so I see little value in supporting both at the same time. Same goes for Kubernetes services which need to be explicitly opted in to run dual stack, otherwise it defaults to IPv4. Either we default to one or the other, or we make it configurable. A similar problem is described in #619.

Seems reasonable to me to default to V4 only and have a setting to prefer V6-only. Maybe also document that both can’t be used at the same time.

I can imagine more complex clusters where some nodes are V4-only and some are V6-only, but that feels broken to me and I wouldn’t spend time supporting that.

@phillebaba can this be implemented soon please? I can probably contribute if you give some pointers on where to look We’re having more troubles with spegel, but want to stick with it.

Calico 3.30 now supports a cni.projectcalico.org/ipFamilies annotation that can be set to '["IPv4"]' (JSON array wrapped in a string) so that certain pods like spegel only get an IPv4 address. In our cluster, this does seem to work as expected and spegel started working.