tailscale: Kubernetes operator can't open tuntap device in proxy

What is the issue?

When trying to use the LoadBalancer or ExternalName services with the Kubernetes operator, the proxy container that gets created fails to start and prints out the following:

boot: 2024/01/11 01:36:41 Unable to create tuntap device file: operation not permitted

It seems like for some reason the securityContext the operator gives the pod with NET_ADMIN isn’t enough for my setup.

I modified this file to look like this

# This file is not a complete manifest, it's a skeleton that the operator embeds
# at build time and then uses to construct Tailscale proxy pods.
apiVersion: apps/v1
kind: StatefulSet
metadata: {}
spec:
  replicas: 1
  template:
    metadata:
      deletionGracePeriodSeconds: 10
    spec:
      serviceAccountName: proxies
      initContainers:
        - name: sysctler
          securityContext:
            privileged: true
          command: ["/bin/sh"]
          args:
            - -c
            - sysctl -w net.ipv4.ip_forward=1 net.ipv6.conf.all.forwarding=1
      resources:
        requests:
          cpu: 1m
          memory: 1Mi
      containers:
        - name: tailscale
          imagePullPolicy: Always
          env:
            - name: TS_USERSPACE
              value: "false"
            - name: TS_AUTH_ONCE
              value: "true"
          securityContext:
            privileged: true

and after building an image and using it I was able to use the LoadBalancer and ExternalName the way the documentation describes.

Steps to reproduce

  1. Install the tailscale operator helm chart
  2. Create an ExternalName or Loadbalancer:
apiVersion: v1
kind: Service
metadata:
  annotations:
    tailscale.com/tailnet-fqdn: service-name.example-name.ts.net
  name: service-name
spec:
  externalName: placeholder
  type: ExternalName

or

apiVersion: v1
kind: Service
metadata:
  name: example-service
spec:
  type: LoadBalancer
  loadBalancerClass: tailscale
  ports:
  - name: http
    port: 8888
    protocol: TCP
    targetPort: http
  selector:
    app.kubernetes.io/instance: example
    app.kubernetes.io/name:  example

Are there any recent changes that introduced the issue?

I don’t know. I just recently started using tailscale.

OS

Linux, Other

OS version

Fedora 38

Tailscale version

1.56.1

Other software

Kubernetes: v1.26.5 Crio: 1.26.3

Bug report

BUG-42e65eedd3327119eb9fd14521e340d737025b2302499169c670a093e8bc4b98-20240111011724Z-21eb645710405b28

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 2
  • Comments: 23 (6 by maintainers)

Commits related to this issue

Most upvoted comments

@gerhard that’s right, I tried it out end-to-end on a single node talos cluster on my homelab proxmox node today. Let me share my notes here, I followed different docs, I link them as I go.

To give a full overview of steps, I’ll start with how I created the talos cluster, you can jump over this section, if you have a cluster already, but I added it for completeness.

Installing test Talos single node cluster

I created a new single node Talos cluster on proxmox following these instructions: Creating Talos Kubernetes cluster using Proxmox

On Talos Linux Image Factory I chose Cloud Server > 1.8.3 > Nocloud > amd64 and picked the system extension siderolabs/qemu-guest-agent (9.1.0). I skipped over customizations and ended up with the iso:

nocloud-amd64.iso

Initial Installation
For the initial installation of Talos Linux (not applicable for disk image boot), add the following installer image to the machine configuration:
factory.talos.dev/installer/ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515:v1.8.3

Upgrading Talos Linux
To upgrade Talos Linux on the machine, use the following image:
factory.talos.dev/installer/ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515:v1.8.3

In proxmox I navigated to my local storage pool > ISO images and hit Download from URL where I pasted the ISO URL and fetched the image. I created a new VM with these details:

  • talos ISO as boot media
  • activated the Qemu Agent support
  • 40 GB disk with SSD emulation
  • set CPU type to host
  • assigned 2 sockets and 2 cores
  • assigned 4096MB RAM
  • default bridge network

Waited for the VM to boot and note the IP of the new node which got assigned via DHCP.

# update talosctl
$ brew install siderolabs/tap/talosctl

$ CONTROL_PLANE_IP=192.168.9.59
$ talosctl gen config talos-proxmox-cluster https://$CONTROL_PLANE_IP:6443 --output-dir _out --install-image factory.talos.dev/installer/ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515:v1.8.3
generating PKI and tokens
Created _out/controlplane.yaml
Created _out/worker.yaml
Created _out/talosconfig

Uncomment allowSchedulingOnControlPlanes: true in _out/controlplane.yaml, I only want a single node, so I must allow workloads to be scheduled on the controlplane.

Apply the controlplane config:

talosctl apply-config --insecure --nodes $CONTROL_PLANE_IP --file _out/controlplane.yaml

Afterwards, the VM console shows some initialization logs on the screen, wait until the kubelet is shown as healthy, etcd error logs are still expected in that phase.

If you wanted to have dedicated worker nodes, this would be the point in time to add them, I skip over them for now.

Configure the new cluster as target in talosctl:

export TALOSCONFIG="_out/talosconfig"
talosctl config endpoint $CONTROL_PLANE_IP
talosctl config node $CONTROL_PLANE_IP

Bootstrap etcd:

talosctl bootstrap

This should trigger the initialization of etcd in the VM logs, give it a moment to get healthy and retrieve the kubeconfig in the meantime.

$ talosctl kubeconfig .
$ export KUBECONFIG=$(pwd)/kubeconfig
konrad@fedora ~/git/homelab/kubernetes (main)$ kubectl get nodes -o wide
NAME            STATUS   ROLES           AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION   CONTAINER-RUNTIME
talos-6bu-n6b   Ready    control-plane   48s   v1.31.2   192.168.9.59   <none>        Talos (v1.8.3)   6.6.60-talos     containerd://2.0.0

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                    READY   STATUS    RESTARTS        AGE
kube-system   coredns-68d75fd545-ld5qc                1/1     Running   0               116s
kube-system   coredns-68d75fd545-zpd92                1/1     Running   0               116s
kube-system   kube-apiserver-talos-6bu-n6b            1/1     Running   0               14s
kube-system   kube-controller-manager-talos-6bu-n6b   1/1     Running   2 (2m25s ago)   58s
kube-system   kube-flannel-fc8bz                      1/1     Running   0               103s
kube-system   kube-proxy-7w4qb                        1/1     Running   0               103s
kube-system   kube-scheduler-talos-6bu-n6b            1/1     Running   3 (2m14s ago)   94s

Our cluster looks good so far.

Installing prerequisites and Tailscale

As noted in the GitHub issue https://github.com/siderolabs/talos/issues/9322, there are problems to access the tun device with a default installation of Talos 1.8.3. But there is a documented way to install a device plugin which can expose the /dev/net/tun device: https://www.talos.dev/v1.8/kubernetes-guides/configuration/device-plugins/#deploying-the-device-plugin

# generic-device-plugin.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: generic-device-plugin
  namespace: kube-system
  labels:
    app.kubernetes.io/name: generic-device-plugin
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: generic-device-plugin
  template:
    metadata:
      labels:
        app.kubernetes.io/name: generic-device-plugin
    spec:
      priorityClassName: system-node-critical
      tolerations:
      - operator: "Exists"
        effect: "NoExecute"
      - operator: "Exists"
        effect: "NoSchedule"
      containers:
      - image: squat/generic-device-plugin
        args:
        - --device
        - |
          name: tun
          groups:
            - count: 1000
              paths:
                - path: /dev/net/tun          
        name: generic-device-plugin
        resources:
          requests:
            cpu: 50m
            memory: 10Mi
          limits:
            cpu: 50m
            memory: 20Mi
        ports:
        - containerPort: 8080
          name: http
        securityContext:
          privileged: true
        volumeMounts:
        - name: device-plugin
          mountPath: /var/lib/kubelet/device-plugins
        - name: dev
          mountPath: /dev
      volumes:
      - name: device-plugin
        hostPath:
          path: /var/lib/kubelet/device-plugins
      - name: dev
        hostPath:
          path: /dev
  updateStrategy:
    type: RollingUpdate

Apply that to your cluster: kubectl apply -f generic-device-plugin.yaml and watch the pods in the kube-system namespace to see, if the DeamonSet could deploy correctly:

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                    READY   STATUS    RESTARTS        AGE
...
kube-system   generic-device-plugin-sx646             1/1     Running   0               19s
...

Next, install the tailscale operator in your cluster. As prerequisite adjust your Access Controls file at https://login.tailscale.com/admin/acls/file. I added the following as outlined in https://tailscale.com/kb/1236/kubernetes-operator#prerequisites.

...
	"tagOwners": {
		"tag:k8s-operator": [],
		"tag:k8s":          ["tag:k8s-operator"],
	},
...

Create an OAuth client at https://login.tailscale.com/admin/settings/oauth and choose:

  • Devices Core and Auth Keys write scopes
  • tag tag:k8s-operator.

Install the operator, e.g. via helm:

# prepare a namespace and permissions
$ kubectl create namespace tailscale
namespace/tailscale created
$ kubectl label namespace tailscale pod-security.kubernetes.io/enforce=privileged
namespace/tailscale labeled
$ helm repo add tailscale https://pkgs.tailscale.com/helmcharts
"tailscale" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "tailscale" chart repository
Update Complete. ⎈Happy Helming!⎈
$ helm upgrade \
  --install \
  tailscale-operator \
  tailscale/tailscale-operator \
  --namespace=tailscale \
  --set-string oauth.clientId="xxx" \
  --set-string oauth.clientSecret="tskey-client-xxx-xxx"
Release "tailscale-operator" does not exist. Installing it now.
W1115 21:54:03.203012   71786 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "operator" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "operator" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "operator" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "operator" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
NAME: tailscale-operator
LAST DEPLOYED: Fri Nov 15 21:54:02 2024
NAMESPACE: tailscale
STATUS: deployed
REVISION: 1
TEST SUITE: None

Ignore the warning, you about the PodSecurity for now and ensure that the operator pod is not crashing:

$ kubectl get pods -n tailscale
NAME                        READY   STATUS    RESTARTS   AGE
operator-75484c7945-7cjmb   1/1     Running   0          75s

If the pod is crashing, check the logs, they might indicate something like a missing scope in the oauth secrets. The example below used an oauth client which lacked permissions to create authkeys, once I defined the right set of permissions the pod did not crash anymore:

$ kubectl logs -f operator-75484c7945-89slc -n tailscale
2024/11/15 20:30:25 tsnet starting with hostname "tailscale-operator", varRoot "/root/.config/tsnet-operator"
2024/11/15 20:30:25 LocalBackend state is NeedsLogin; running StartLoginInteractive...
{"level":"fatal","ts":"2024-11-15T20:30:25Z","logger":"startup","msg":"creating operator authkey: Status: 403, Message: \"calling actor does not have enough permissions to perform this function\"","stacktrace":"main.initTSNet\n\ttailscale.com/cmd/k8s-operator/operator.go:195\nmain.main\n\ttailscale.com/cmd/k8s-operator/operator.go:97\nruntime.main\n\truntime/proc.go:272"}

Once the operator is running and all CustomResourceDefinitions (CRDs) are installed, a new ProxyClass can be created, like mentioned by @Santhin in https://github.com/siderolabs/talos/issues/9322#issuecomment-2414685070. I changed it slightly:

# proxy-class.yaml
apiVersion: tailscale.com/v1alpha1
kind: ProxyClass
metadata:
  name: tun-access-proxy-class
spec:
  statefulSet:
    pod:
      tailscaleContainer:
        resources:
          limits:
            squat.ai/tun: "1"
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
              - ALL
            add:
              - NET_ADMIN

Deploy the ProxyClass with kubectl apply -f proxy-class.yaml -n tailscale.

Deploy the workload

Finally, test if you can expose some service to your tailnet.

# demo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.27.2
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    tailscale.com/proxy-class: "tun-access-proxy-class"
  annotations:
    tailscale.com/expose: "true"
spec:
  selector:
    app: nginx
  ports:
    - port: 80
      targetPort: 80

Deploy the resource in a new namespace:

$ kubectl create namespace demo
namespace/demo created
$ kubectl apply -f demo.yaml -n demo
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "nginx" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "nginx" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "nginx" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "nginx" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
deployment.apps/nginx created
service/nginx created

$ kubectl get pods -n demo
NAME                     READY   STATUS    RESTARTS   AGE
nginx-7f8746677b-n9f62   1/1     Running   0          57s

$ kubectl get pods -n tailscale
NAME                        READY   STATUS    RESTARTS   AGE
operator-75484c7945-7cjmb   1/1     Running   0          14m
ts-nginx-m5js6-0            1/1     Running   0          52s

As shown above, a new pod was deployed into the tailscale namespace and nginx is running.

In the machines overview you will see a new machine called demo-nginx afterwards: https://login.tailscale.com/admin/machines It carries a tag k8s and gets a proper tailscale IP.

You can connect to that IP as usual in tailscale:

$ curl http://demo-nginx
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

$ curl 100.85.225.42
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

🥳

We have the same problem. We think it’s because the proxy pod is lacking:

   securityContext:
      privileged: true

The init pod does have this security flag, but the non-init container does not. Is there any way to inject this security flag into the running pod?

Also, thank you so much for moving this to a helm chart! We were doing “bleeding edge” things with some hand-crafted terraform, but now we’re on the helm chart. We’re using the operator for both ingresses and service endpoints and we couldn’t be happier with how well it works! Thanks for all the hard work on this project!

Thank you for all the contributions and useful information on this issue folks. Especially thanks to @konrad-ohms for the amazing device plugin description.

As folks already mentioned, with the change in containerd this is now going to be an issue for all Kubernetes distributions.

I’ve tested the device plugin installation and it works, but we would like to avoid a hard dependency on third party tool, so making the proxies that need to use tun device run in privileged containers seem a better solution and we are thinking of making that change, see https://github.com/tailscale/tailscale/pull/14262. Folks can then choose to install the device plugin and drop the privileged security context (and we should add docs for that). Note that we already run a privileged init container for all proxies that need tun device, so as far as I can tell this should not be a breaking change.

Longer term, we are wanting to look into running these proxies in netstack mode to avoid needing a tun device altogether.

Thank you for sharing such detailed notes @konrad-ohms, I can imagine how anyone that hits this coming from Talos will find them most helpful.

I am sticking to Talos 1.7.x for now, which doesn’t require squat/generic-device-plugin for the Tailscale operator to work. I’m pretty much at 1. from this list: https://github.com/siderolabs/talos/issues/9322#issuecomment-2426833819

Containerd is still at 2.0.0 in the latest Talos 1.8.x available today (1.8.3 as of this comment). I would like to see containerd hit 2.1 before I pick it up. Based on https://containerd.io/releases/, I am guessing spring 2025. Hopefully by then we will have a versioned squat/generic-device-plugin (note to self: search / open an issue for this).

Thank you all for making this a great example of community effort!

Thank you @irbekrm. The tailscale.com/proxy-class label works.

A temporary workaround would be to use ProxyClass to make the tailscale container privileged like

apiVersion: tailscale.com/v1alpha1
kind: ProxyClass
metadata:
  name: prod
spec:
  statefulSet:
    pod:
       tailscaleContainer:
         securityContext:
           privileged: true

https://tailscale.com/kb/1236/kubernetes-operator#cluster-resource-customization-using-proxyclass-custom-resource

We need to take a look whether the tailscale container needs extra permissions to fix this by default.