talos: Tailscale Kubernetes operator - is CONFIG_TUN enabled in your kernel? `modprobe tun` failed with: modprobe: can't change directory to '/lib/modules': No such file or directory

On my test cluster (running on Proxmox) I started testing 1.8.0-beta.0 and noticed that the Tailscale Kubernetes Operator run into a timeout and than the pod restarts.

Bug Report

Tailscale Kubernetes operator restarts after timeout on Talos 1.8.0-beta.0, but worked with 1.7.6. Maybe similar related to https://github.com/siderolabs/talos/issues/9309

Description

It seems like this is the issue, but I am not sure what exactly changed between 1.8.0 and 1.7.6 or how I could fix this. I am not using the Tailscale Extension, only the Kubernetes Operator.

tailscale 2024/09/13 19:51:19 is CONFIG_TUN enabled in your kernel? `modprobe tun` failed with: modprobe: can't change directory to '/lib/modules': No such file or directory                                                 │

Logs

| sysctler net.ipv4.ip_forward = 1                                                                                                                                                          │
│ sysctler net.ipv6.conf.all.forwarding = 0                                                                                                                                                 │
│ sysctler net.ipv6.conf.all.forwarding = 1                                                                                                                                                 │
│ tailscale boot: 2024/09/13 20:00:51 skipping file "..data" in tailscaled config directory "/etc/tsconfig": input does not match format                                                    │
│ tailscale boot: 2024/09/13 20:00:51 skipping file "_current-profile" in tailscaled config directory "/etc/tsconfig": input does not match format                                          │
│ tailscale boot: 2024/09/13 20:00:51 skipping file "_machinekey" in tailscaled config directory "/etc/tsconfig": input does not match format                                               │
│ tailscale boot: 2024/09/13 20:00:51 skipping file "_profiles" in tailscaled config directory "/etc/tsconfig": input does not match format                                                 │
│ tailscale boot: 2024/09/13 20:00:51 skipping file "device_fqdn" in tailscaled config directory "/etc/tsconfig": input does not match format                                               │
│ tailscale boot: 2024/09/13 20:00:51 skipping file "device_id" in tailscaled config directory "/etc/tsconfig": input does not match format                                                 │
│ tailscale boot: 2024/09/13 20:00:51 skipping file "device_ips" in tailscaled config directory "/etc/tsconfig": input does not match format                                                │
│ tailscale boot: 2024/09/13 20:00:51 skipping file "profile-3a0d" in tailscaled config directory "/etc/tsconfig": input does not match format                                              │
│ tailscale boot: 2024/09/13 20:00:51 Using tailscaled config file '_' for capability version 'h'                                                                                           │
│ tailscale boot: 2024/09/13 20:00:51 Starting tailscaled                                                                                                                                   │
│ tailscale boot: 2024/09/13 20:00:51 Waiting for tailscaled socket                                                                                                                         │
│ tailscale 2024/09/13 20:00:51 logtail started                                                                                                                                             │
│ tailscale 2024/09/13 20:00:51 Program starting: v1.72.1-tf4a95663c, Go 1.22.5: []string{"tailscaled", "--socket=/tmp/tailscaled.sock", "--state=kube:ts-ingress-nginx-controller-sprbw-0" │
│ tailscale 2024/09/13 20:00:51 LogID: 7cf1c32b65cc6b1b80aed64d374870ad3b1f15e9aa3bd52df8b04c0bf31daf02                                                                                     │
│ tailscale 2024/09/13 20:00:51 logpolicy: using system state directory "/var/lib/tailscale"                                                                                                │
│ tailscale logpolicy.ConfigFromFile /var/lib/tailscale/tailscaled.log.conf: open /var/lib/tailscale/tailscaled.log.conf: no such file or directory                                         │
│ tailscale logpolicy.Config.Validate for /var/lib/tailscale/tailscaled.log.conf: config is nil                                                                                             │
│ tailscale 2024/09/13 20:00:51 dns: [rc=unknown ret=direct]                                                                                                                                │
│ tailscale 2024/09/13 20:00:51 dns: using "direct" mode                                                                                                                                    │
│ tailscale 2024/09/13 20:00:51 dns: using *dns.directManager                                                                                                                               │
│ tailscale 2024/09/13 20:00:51 wgengine.NewUserspaceEngine(tun "tailscale0") ...                                                                                                           │
│ tailscale 2024/09/13 20:00:51 Linux kernel version: 6.6.49-talos                                                                                                                          │
│ tailscale 2024/09/13 20:00:51 is CONFIG_TUN enabled in your kernel? `modprobe tun` failed with: modprobe: can't change directory to '/lib/modules': No such file or directory             │
│ tailscale 2024/09/13 20:00:51 wgengine.NewUserspaceEngine(tun "tailscale0") error: tstun.New("tailscale0"): operation not permitted                                                       │
│ tailscale 2024/09/13 20:00:51 flushing log.                                                                                                                                               │
│ tailscale 2024/09/13 20:00:51 logger closing down                                                                                                                                         │
│ tailscale 2024/09/13 20:00:51 getLocalBackend error: createEngine: tstun.New("tailscale0"): operation not permitted                                                                       │
│ tailscale boot: 2024/09/13 20:01:51 Timed out waiting for tailscaled socket 

Environment

  • Talos version: [1.8.0-beta.0]
  • Kubernetes version: [v1.31.0]
  • Platform: macOS

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 22 (8 by maintainers)

Most upvoted comments

I use Smarter Device Manager for accessing my USB and TUN devices in Talos. The advantage it has over the generic device manager is that it doesn’t require running as a privileged pod.

I created a proxyClass in my tailscale namespace. I’m using Smarter device manager, because it doesn’t require host privileges, but this should work the same with squat.ai/tun:

apiVersion: tailscale.com/v1alpha1
kind: ProxyClass
metadata:
  name: tun-access
spec:
  statefulSet:
    pod:
      tailscaleContainer:
        resources:
          requests:
          limits:
            smarter-devices/net_tun: "1"

Then in my service, just add a label referencing the proxyClass:

labels:
  tailscale.com/proxy-class: "tun-access"

@phiilu did you ever sort this out? I encountered this on Talos 1.8 and am also using the Tailscale operator.

I figured out how to fix it via Tailscale proxyClass, if you’re still looking for a solution.

Thanks for the details, I was able to get it working as well. As I combined multiple comments/documentation links, I pasted my install summary notes in https://github.com/tailscale/tailscale/issues/10814#issuecomment-2479977752 if anyone is interested in a full end-to-end walkthru.

Trying to setup a subnet router using custom Connector CRD but with no options to specify resources i’m guessing i’m stuck for the moment. My best bet here should be creating a tailscale pod and specifying the tun device. Don’t know if anyone has a workaround/solution for this or not.

You can define a proxy class and use it from the connector. See https://github.com/jfroy/flatops/tree/main/kubernetes/apps/tailscale/tailscale-operator/connector for an example.

This was the second issue that I’ve hit when upgrading Talos from v1.7 to v1.8.

I appreciate the fixes that you all shared, and also for documenting https://www.talos.dev/v1.8/kubernetes-guides/configuration/device-plugins/ @smira - thank you!

I am going to stick with Talos v1.7 for now since I am not keen on depending on a random, unversioned image - i.e. squat/generic-device-plugin - for such an important functionality.

Are there any plans to ship this part of Talos v1.8?


FWIW, this was my first issue:

The documentation is published https://www.talos.dev/v1.8/kubernetes-guides/configuration/device-plugins/

That should be enough to get you unblocked