talos: Running cilium causes kernel error: dead loop on virtual device cilium_vxlan, fix it urgently

Bug Report

I wanted to run Talos 1.7.5 and with Cilium, but it causes an error of: image

Description

I tried with Cilium 1.15.7 and 1.16.0.

Cilium values are:

cluster:
  name: "prod-buero"
  id: 1
socketLB:
  enabled: true
bgpControlPlane:
  enabled: true
hubble:
  enabled: true
  relay:
    enabled: true
    prometheus:
      enabled: true
      serviceMonitor:
        enabled: true
  metrics:
    enabled: 
      - dns:query;ignoreAAAA
      - drop
      - tcp
      - flow
      - icmp
      - http
    serviceMonitor:
      enabled: true
prometheus:
  enabled: true
  serviceMonitor:
    enabled: true
operator:
  prometheus:
    enabled: true
    serviceMonitor:
      enabled: true
kubeProxyReplacement: true
securityContext:
  capabilities:
    ciliumAgent: 
      - CHOWN
      - KILL
      - NET_ADMIN
      - NET_RAW
      - IPC_LOCK
      - SYS_ADMIN
      - SYS_RESOURCE
      - DAC_OVERRIDE
      - FOWNER
      - SETGID
      - SETUID
    cleanCiliumState: 
      - SYS_ADMIN
      - SYS_RESOURCE
cgroup:
  autoMount:
    enabled: false
  hostRoot: /sys/fs/cgroup
k8sServiceHost: localhost
k8sServicePort: 7445
envoy:
  enabled: true
  prometheus:
    enabled: true
    serviceMonitor:
      enabled: true
extraConfig:
  external-dns-proxy: "true"

These are VMs running on Proxmox with the following specs: image

The cluster runs for 24 hours without any issues, but then after ~ 24 hours, the errors start to appear.

Logs

support.zip

Environment

  • Talos version:
Client:
        Tag:         v1.7.5
        SHA:         47731624
        Built:       
        Go version:  go1.22.4
        OS/Arch:     darwin/arm64
Server:
        NODE:        192.168.20.11
        Tag:         v1.7.5
        SHA:         47731624
        Built:       
        Go version:  go1.22.4
        OS/Arch:     linux/amd64
        Enabled:     RBAC
  • Kubernetes version:
Server Version: v1.29.0
  • Platform: Promox

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (8 by maintainers)

Most upvoted comments

Talos runs with a vanilla Linux LTS kernel, so there might be an issue in the Linux kernel as well, but even figuring out why it happens will be way easier starting with Cilium. It might be a bug either way, or a misconfiguration, or incompatibility.

We test Cilium on Talos in default configurations, but Cilium itself is really big, so testing every possible feature is not possible.

We are pretty sure that Cilium itself works, but we can’t verify every possible configuration.

@Syntax3rror404 I already tested on the latest Talos and it also happened there. Are you also running on Proxmox?

I’m also running Talos 1.7.6, Cilium 1.16 on top of Proxmox and I’m not having this issue because I’m running nativeRouting:


kubeProxyReplacement: true
bpf:
  masquerade: true
ipam:
  mode: kubernetes
  operator:
    clusterPoolIPv4PodCIDR: "<pod network>"
ipv4NativeRoutingCIDR: "<larger network>"
autoDirectNodeRoutes: true
routingMode: native
endpointRoutes:
  enabled: true
installNoConntrackIptablesRules: true
bandwidthManager:
  enabled: true
  bbr: true
enableIPv4BIGTCP: true
l2announcements:
  enabled: true
ipv4:
  enabled: true
securityContext:
  capabilities:
    ciliumAgent:
      - CHOWN
      - KILL
      - NET_ADMIN
      - NET_RAW
      - IPC_LOCK
      - SYS_ADMIN
      - SYS_RESOURCE
      - DAC_OVERRIDE
      - FOWNER
      - SETUID
      - SETGID
    cleanCiliumState:
      - NET_ADMIN
      - SYS_ADMIN
      - SYS_RESOURCE
cgroup:
  hostRoot: /sys/fs/cgroup
  autoMount:
    enabled: false
# Enable Cilium Ingress Controller
ingressController:
  enabled: true
# Use KubePrism to access cluster API
k8sServiceHost: localhost
k8sServicePort: 7445
# Enable Hubble
hubble:
  metrics:
    enabled:
    - dns
    - drop
    - tcp
    - flow
    - icmp
    - http
  relay:
    enabled: true
  ui:
    enabled: true

I’ll see if I can run a cluster with tunnelling enabled to replicate the issue