talos: Running cilium causes kernel error: dead loop on virtual device cilium_vxlan, fix it urgently
Bug Report
I wanted to run Talos 1.7.5 and with Cilium, but it causes an error of:
Description
I tried with Cilium 1.15.7 and 1.16.0.
Cilium values are:
cluster:
name: "prod-buero"
id: 1
socketLB:
enabled: true
bgpControlPlane:
enabled: true
hubble:
enabled: true
relay:
enabled: true
prometheus:
enabled: true
serviceMonitor:
enabled: true
metrics:
enabled:
- dns:query;ignoreAAAA
- drop
- tcp
- flow
- icmp
- http
serviceMonitor:
enabled: true
prometheus:
enabled: true
serviceMonitor:
enabled: true
operator:
prometheus:
enabled: true
serviceMonitor:
enabled: true
kubeProxyReplacement: true
securityContext:
capabilities:
ciliumAgent:
- CHOWN
- KILL
- NET_ADMIN
- NET_RAW
- IPC_LOCK
- SYS_ADMIN
- SYS_RESOURCE
- DAC_OVERRIDE
- FOWNER
- SETGID
- SETUID
cleanCiliumState:
- SYS_ADMIN
- SYS_RESOURCE
cgroup:
autoMount:
enabled: false
hostRoot: /sys/fs/cgroup
k8sServiceHost: localhost
k8sServicePort: 7445
envoy:
enabled: true
prometheus:
enabled: true
serviceMonitor:
enabled: true
extraConfig:
external-dns-proxy: "true"
These are VMs running on Proxmox with the following specs:
The cluster runs for 24 hours without any issues, but then after ~ 24 hours, the errors start to appear.
Logs
Environment
- Talos version:
Client:
Tag: v1.7.5
SHA: 47731624
Built:
Go version: go1.22.4
OS/Arch: darwin/arm64
Server:
NODE: 192.168.20.11
Tag: v1.7.5
SHA: 47731624
Built:
Go version: go1.22.4
OS/Arch: linux/amd64
Enabled: RBAC
- Kubernetes version:
Server Version: v1.29.0
- Platform: Promox
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (8 by maintainers)
Talos runs with a vanilla Linux LTS kernel, so there might be an issue in the Linux kernel as well, but even figuring out why it happens will be way easier starting with Cilium. It might be a bug either way, or a misconfiguration, or incompatibility.
We test Cilium on Talos in default configurations, but Cilium itself is really big, so testing every possible feature is not possible.
We are pretty sure that Cilium itself works, but we can’t verify every possible configuration.
I’m also running Talos 1.7.6, Cilium 1.16 on top of Proxmox and I’m not having this issue because I’m running nativeRouting:
I’ll see if I can run a cluster with tunnelling enabled to replicate the issue