spegel: "could not get peer id" and timeouts since 0.0.29
Spegel version
v0.0.30
Kubernetes distribution
Talos 1.9.1
Kubernetes version
v1.31.4
CNI
Cilium
Describe the bug
Since v0.0.29 with the new peer discovery, I am seeing errors in spegel logs and 60s delays whenever a container needs to fetch an image (basically a timeout, after which the image is quickly fetched from upstream). I am assuming this means spegel is basically in a bad state.
I haven’t changed my CNI, Kubernetes, or Talos versions in-between those spegel versions, and I haven’t changed my spegel settings.
My cluster is dual-stack (v4 and v6).
I use the helm chart for installation using the following values:
serviceMonitor:
enabled: true
grafanaDashboard:
enabled: true
spegel:
containerdRegistryConfigPath: /etc/cri/conf.d/hosts
Logs:
{"time":"2025-01-22T06:39:54.884189668Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/oci.(*Containerd).Verify","file":"/build/pkg/oci/containerd.go","line":118},"msg":"unable to verify status response","runtime_version":"2.0.1"}
{"time":"2025-01-22T06:39:54.901603359Z","level":"INFO","source":{"function":"main.registryCommand","file":"/build/main.go","line":212},"msg":"running Spegel","registry":":5000","router":":5001"}
{"time":"2025-01-22T06:39:54.90170104Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/routing.(*P2PRouter).Run","file":"/build/pkg/routing/p2p.go","line":111},"msg":"starting p2p router","logger":"p2p","id":"/ip6/2001:5a8:42a6:e2fb::f95/tcp/5001/p2p/12D3KooWLPhmVFbJie7ua7juDgqrcH123CTRSeFV2iMNJhM27dYt"}
{"time":"2025-01-22T06:39:54.902058451Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":34},"msg":"running scheduled image state update"}
{"time":"2025-01-22T06:39:59.90690095Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/routing.bootstrapFunc.func1","file":"/build/pkg/routing/p2p.go","line":286},"msg":"could not get peer id","logger":"p2p","err":"failed to dial: failed to dial 92B: all dials failed\n * [/ip4/10.11.1.204/tcp/5001] dial tcp4 0.0.0.0:5001->10.11.1.204:5001: i/o timeout"}
{"time":"2025-01-22T06:39:59.912454683Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/routing.bootstrapFunc.func1","file":"/build/pkg/routing/p2p.go","line":286},"msg":"could not get peer id","logger":"p2p","err":"failed to dial: failed to dial 92B: all dials failed\n * [/ip4/10.11.3.186/tcp/5001] dial to self attempted\n * [/ip4/10.11.1.204/tcp/5001] dial backoff\n * [/ip4/10.11.2.154/tcp/5001] dial backoff"}
{"time":"2025-01-22T06:39:59.920541642Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/routing.bootstrapFunc.func1","file":"/build/pkg/routing/p2p.go","line":286},"msg":"could not get peer id","logger":"p2p","err":"failed to dial: failed to dial 92B: all dials failed\n * [/ip4/10.11.3.186/tcp/5001] dial to self attempted\n * [/ip4/10.11.1.204/tcp/5001] dial backoff\n * [/ip4/10.11.2.154/tcp/5001] dial backoff"}
{"time":"2025-01-22T06:39:59.920706382Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/routing.bootstrapFunc.func1","file":"/build/pkg/routing/p2p.go","line":286},"msg":"could not get peer id","logger":"p2p","err":"failed to dial: failed to dial 92B: all dials failed\n * [/ip4/10.11.3.186/tcp/5001] dial to self attempted\n * [/ip4/10.11.1.204/tcp/5001] dial backoff\n * [/ip4/10.11.2.154/tcp/5001] dial backoff"}
{"time":"2025-01-22T06:39:59.920869343Z","level":"ERROR","source":{"function":"github.com/spegel-org/spegel/pkg/routing.bootstrapFunc.func1","file":"/build/pkg/routing/p2p.go","line":286},"msg":"could not get peer id","logger":"p2p","err":"failed to dial: failed to dial 92B: all dials failed\n * [/ip4/10.11.3.186/tcp/5001] dial to self attempted\n * [/ip4/10.11.1.204/tcp/5001] dial backoff\n * [/ip4/10.11.2.154/tcp/5001] dial backoff"}
{"time":"2025-01-22T06:39:59.920891363Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/routing.bootstrapFunc.func1","file":"/build/pkg/routing/p2p.go","line":293},"msg":"no bootstrap nodes found","logger":"p2p"}
{"time":"2025-01-22T06:48:54.903045168Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":34},"msg":"running scheduled image state update"}
{"time":"2025-01-22T06:57:54.902146115Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":34},"msg":"running scheduled image state update"}
{"time":"2025-01-22T07:06:54.902113162Z","level":"INFO","source":{"function":"github.com/spegel-org/spegel/pkg/state.Track","file":"/build/pkg/state/state.go","line":34},"msg":"running scheduled image state update"}
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 3
- Comments: 15 (11 by maintainers)
Seems reasonable to me to default to V4 only and have a setting to prefer V6-only. Maybe also document that both can’t be used at the same time.
I can imagine more complex clusters where some nodes are V4-only and some are V6-only, but that feels broken to me and I wouldn’t spend time supporting that.
@phillebaba can this be implemented soon please? I can probably contribute if you give some pointers on where to look We’re having more troubles with spegel, but want to stick with it.
Calico 3.30 now supports a
cni.projectcalico.org/ipFamiliesannotation that can be set to'["IPv4"]'(JSON array wrapped in a string) so that certain pods like spegel only get an IPv4 address. In our cluster, this does seem to work as expected and spegel started working.