talos: Tailscale Kubernetes operator - is CONFIG_TUN enabled in your kernel? `modprobe tun` failed with: modprobe: can't change directory to '/lib/modules': No such file or directory
On my test cluster (running on Proxmox) I started testing 1.8.0-beta.0 and noticed that the Tailscale Kubernetes Operator run into a timeout and than the pod restarts.
Bug Report
Tailscale Kubernetes operator restarts after timeout on Talos 1.8.0-beta.0, but worked with 1.7.6.
Maybe similar related to https://github.com/siderolabs/talos/issues/9309
Description
It seems like this is the issue, but I am not sure what exactly changed between 1.8.0 and 1.7.6 or how I could fix this. I am not using the Tailscale Extension, only the Kubernetes Operator.
tailscale 2024/09/13 19:51:19 is CONFIG_TUN enabled in your kernel? `modprobe tun` failed with: modprobe: can't change directory to '/lib/modules': No such file or directory │
Logs
| sysctler net.ipv4.ip_forward = 1 │
│ sysctler net.ipv6.conf.all.forwarding = 0 │
│ sysctler net.ipv6.conf.all.forwarding = 1 │
│ tailscale boot: 2024/09/13 20:00:51 skipping file "..data" in tailscaled config directory "/etc/tsconfig": input does not match format │
│ tailscale boot: 2024/09/13 20:00:51 skipping file "_current-profile" in tailscaled config directory "/etc/tsconfig": input does not match format │
│ tailscale boot: 2024/09/13 20:00:51 skipping file "_machinekey" in tailscaled config directory "/etc/tsconfig": input does not match format │
│ tailscale boot: 2024/09/13 20:00:51 skipping file "_profiles" in tailscaled config directory "/etc/tsconfig": input does not match format │
│ tailscale boot: 2024/09/13 20:00:51 skipping file "device_fqdn" in tailscaled config directory "/etc/tsconfig": input does not match format │
│ tailscale boot: 2024/09/13 20:00:51 skipping file "device_id" in tailscaled config directory "/etc/tsconfig": input does not match format │
│ tailscale boot: 2024/09/13 20:00:51 skipping file "device_ips" in tailscaled config directory "/etc/tsconfig": input does not match format │
│ tailscale boot: 2024/09/13 20:00:51 skipping file "profile-3a0d" in tailscaled config directory "/etc/tsconfig": input does not match format │
│ tailscale boot: 2024/09/13 20:00:51 Using tailscaled config file '_' for capability version 'h' │
│ tailscale boot: 2024/09/13 20:00:51 Starting tailscaled │
│ tailscale boot: 2024/09/13 20:00:51 Waiting for tailscaled socket │
│ tailscale 2024/09/13 20:00:51 logtail started │
│ tailscale 2024/09/13 20:00:51 Program starting: v1.72.1-tf4a95663c, Go 1.22.5: []string{"tailscaled", "--socket=/tmp/tailscaled.sock", "--state=kube:ts-ingress-nginx-controller-sprbw-0" │
│ tailscale 2024/09/13 20:00:51 LogID: 7cf1c32b65cc6b1b80aed64d374870ad3b1f15e9aa3bd52df8b04c0bf31daf02 │
│ tailscale 2024/09/13 20:00:51 logpolicy: using system state directory "/var/lib/tailscale" │
│ tailscale logpolicy.ConfigFromFile /var/lib/tailscale/tailscaled.log.conf: open /var/lib/tailscale/tailscaled.log.conf: no such file or directory │
│ tailscale logpolicy.Config.Validate for /var/lib/tailscale/tailscaled.log.conf: config is nil │
│ tailscale 2024/09/13 20:00:51 dns: [rc=unknown ret=direct] │
│ tailscale 2024/09/13 20:00:51 dns: using "direct" mode │
│ tailscale 2024/09/13 20:00:51 dns: using *dns.directManager │
│ tailscale 2024/09/13 20:00:51 wgengine.NewUserspaceEngine(tun "tailscale0") ... │
│ tailscale 2024/09/13 20:00:51 Linux kernel version: 6.6.49-talos │
│ tailscale 2024/09/13 20:00:51 is CONFIG_TUN enabled in your kernel? `modprobe tun` failed with: modprobe: can't change directory to '/lib/modules': No such file or directory │
│ tailscale 2024/09/13 20:00:51 wgengine.NewUserspaceEngine(tun "tailscale0") error: tstun.New("tailscale0"): operation not permitted │
│ tailscale 2024/09/13 20:00:51 flushing log. │
│ tailscale 2024/09/13 20:00:51 logger closing down │
│ tailscale 2024/09/13 20:00:51 getLocalBackend error: createEngine: tstun.New("tailscale0"): operation not permitted │
│ tailscale boot: 2024/09/13 20:01:51 Timed out waiting for tailscaled socket
Environment
- Talos version: [
1.8.0-beta.0] - Kubernetes version: [
v1.31.0] - Platform: macOS
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 22 (8 by maintainers)
I use Smarter Device Manager for accessing my USB and TUN devices in Talos. The advantage it has over the generic device manager is that it doesn’t require running as a privileged pod.
I created a proxyClass in my tailscale namespace. I’m using Smarter device manager, because it doesn’t require host privileges, but this should work the same with squat.ai/tun:
Then in my service, just add a label referencing the proxyClass:
@phiilu did you ever sort this out? I encountered this on Talos 1.8 and am also using the Tailscale operator.
I figured out how to fix it via Tailscale proxyClass, if you’re still looking for a solution.
Thanks for the details, I was able to get it working as well. As I combined multiple comments/documentation links, I pasted my install summary notes in https://github.com/tailscale/tailscale/issues/10814#issuecomment-2479977752 if anyone is interested in a full end-to-end walkthru.
You can define a proxy class and use it from the connector. See https://github.com/jfroy/flatops/tree/main/kubernetes/apps/tailscale/tailscale-operator/connector for an example.
This was the second issue that I’ve hit when upgrading Talos from v1.7 to v1.8.
I appreciate the fixes that you all shared, and also for documenting https://www.talos.dev/v1.8/kubernetes-guides/configuration/device-plugins/ @smira - thank you!
I am going to stick with Talos v1.7 for now since I am not keen on depending on a random, unversioned image - i.e.
squat/generic-device-plugin- for such an important functionality.Are there any plans to ship this part of Talos v1.8?
FWIW, this was my first issue:
The documentation is published https://www.talos.dev/v1.8/kubernetes-guides/configuration/device-plugins/
That should be enough to get you unblocked