tailscale: Kubernetes operator can't open tuntap device in proxy
What is the issue?
When trying to use the LoadBalancer or ExternalName services with the Kubernetes operator, the proxy container that gets created fails to start and prints out the following:
boot: 2024/01/11 01:36:41 Unable to create tuntap device file: operation not permitted
It seems like for some reason the securityContext the operator gives the pod with NET_ADMIN isn’t enough for my setup.
I modified this file to look like this
# This file is not a complete manifest, it's a skeleton that the operator embeds
# at build time and then uses to construct Tailscale proxy pods.
apiVersion: apps/v1
kind: StatefulSet
metadata: {}
spec:
replicas: 1
template:
metadata:
deletionGracePeriodSeconds: 10
spec:
serviceAccountName: proxies
initContainers:
- name: sysctler
securityContext:
privileged: true
command: ["/bin/sh"]
args:
- -c
- sysctl -w net.ipv4.ip_forward=1 net.ipv6.conf.all.forwarding=1
resources:
requests:
cpu: 1m
memory: 1Mi
containers:
- name: tailscale
imagePullPolicy: Always
env:
- name: TS_USERSPACE
value: "false"
- name: TS_AUTH_ONCE
value: "true"
securityContext:
privileged: true
and after building an image and using it I was able to use the LoadBalancer and ExternalName the way the documentation describes.
Steps to reproduce
- Install the tailscale operator helm chart
- Create an ExternalName or Loadbalancer:
apiVersion: v1
kind: Service
metadata:
annotations:
tailscale.com/tailnet-fqdn: service-name.example-name.ts.net
name: service-name
spec:
externalName: placeholder
type: ExternalName
or
apiVersion: v1
kind: Service
metadata:
name: example-service
spec:
type: LoadBalancer
loadBalancerClass: tailscale
ports:
- name: http
port: 8888
protocol: TCP
targetPort: http
selector:
app.kubernetes.io/instance: example
app.kubernetes.io/name: example
Are there any recent changes that introduced the issue?
I don’t know. I just recently started using tailscale.
OS
Linux, Other
OS version
Fedora 38
Tailscale version
1.56.1
Other software
Kubernetes: v1.26.5 Crio: 1.26.3
Bug report
BUG-42e65eedd3327119eb9fd14521e340d737025b2302499169c670a093e8bc4b98-20240111011724Z-21eb645710405b28
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 2
- Comments: 23 (6 by maintainers)
Commits related to this issue
- cmd/k8s-operator,docs/k8s: run tun mode proxies in privileged containers We were previously relying on unintended behaviour by runc where all containers where by default given read/write/mknod permis... — committed to tailscale/tailscale by irbekrm 2 years ago
- cmd/k8s-operator,docs/k8s: run tun mode proxies in privileged containers We were previously relying on unintended behaviour by runc where all containers where by default given read/write/mknod permis... — committed to tailscale/tailscale by irbekrm 2 years ago
- cmd/k8s-operator,docs/k8s: run tun mode proxies in privileged containers (#14262) We were previously relying on unintended behaviour by runc where all containers where by default given read/write/mk... — committed to tailscale/tailscale by irbekrm 2 years ago
- cmd/k8s-operator,docs/k8s: run tun mode proxies in privileged containers (#14262) We were previously relying on unintended behaviour by runc where all containers where by default given read/write/mk... — committed to ninech/tailscale by irbekrm 2 years ago
@gerhard that’s right, I tried it out end-to-end on a single node talos cluster on my homelab proxmox node today. Let me share my notes here, I followed different docs, I link them as I go.
To give a full overview of steps, I’ll start with how I created the talos cluster, you can jump over this section, if you have a cluster already, but I added it for completeness.
Installing test Talos single node cluster
I created a new single node Talos cluster on proxmox following these instructions: Creating Talos Kubernetes cluster using Proxmox
On Talos Linux Image Factory I chose
Cloud Server > 1.8.3 > Nocloud > amd64and picked the system extensionsiderolabs/qemu-guest-agent (9.1.0). I skipped over customizations and ended up with the iso:nocloud-amd64.iso
In proxmox I navigated to my local storage pool > ISO images and hit
Download from URLwhere I pasted the ISO URL and fetched the image. I created a new VM with these details:hostWaited for the VM to boot and note the IP of the new node which got assigned via DHCP.
Uncomment
allowSchedulingOnControlPlanes: truein_out/controlplane.yaml, I only want a single node, so I must allow workloads to be scheduled on the controlplane.Apply the controlplane config:
Afterwards, the VM console shows some initialization logs on the screen, wait until the kubelet is shown as healthy, etcd error logs are still expected in that phase.
If you wanted to have dedicated worker nodes, this would be the point in time to add them, I skip over them for now.
Configure the new cluster as target in talosctl:
Bootstrap etcd:
This should trigger the initialization of etcd in the VM logs, give it a moment to get healthy and retrieve the kubeconfig in the meantime.
Our cluster looks good so far.
Installing prerequisites and Tailscale
As noted in the GitHub issue https://github.com/siderolabs/talos/issues/9322, there are problems to access the tun device with a default installation of Talos 1.8.3. But there is a documented way to install a device plugin which can expose the
/dev/net/tundevice: https://www.talos.dev/v1.8/kubernetes-guides/configuration/device-plugins/#deploying-the-device-pluginApply that to your cluster:
kubectl apply -f generic-device-plugin.yamland watch the pods in thekube-systemnamespace to see, if the DeamonSet could deploy correctly:Next, install the tailscale operator in your cluster. As prerequisite adjust your Access Controls file at https://login.tailscale.com/admin/acls/file. I added the following as outlined in https://tailscale.com/kb/1236/kubernetes-operator#prerequisites.
Create an OAuth client at https://login.tailscale.com/admin/settings/oauth and choose:
Devices CoreandAuth Keyswrite scopestag:k8s-operator.Install the operator, e.g. via helm:
Ignore the warning, you about the PodSecurity for now and ensure that the operator pod is not crashing:
If the pod is crashing, check the logs, they might indicate something like a missing scope in the oauth secrets. The example below used an oauth client which lacked permissions to create authkeys, once I defined the right set of permissions the pod did not crash anymore:
Once the operator is running and all CustomResourceDefinitions (CRDs) are installed, a new ProxyClass can be created, like mentioned by @Santhin in https://github.com/siderolabs/talos/issues/9322#issuecomment-2414685070. I changed it slightly:
Deploy the ProxyClass with
kubectl apply -f proxy-class.yaml -n tailscale.Deploy the workload
Finally, test if you can expose some service to your tailnet.
Deploy the resource in a new namespace:
As shown above, a new pod was deployed into the tailscale namespace and nginx is running.
In the machines overview you will see a new machine called
demo-nginxafterwards: https://login.tailscale.com/admin/machines It carries a tag k8s and gets a proper tailscale IP.You can connect to that IP as usual in tailscale:
🥳
We have the same problem. We think it’s because the proxy pod is lacking:
The init pod does have this security flag, but the non-init container does not. Is there any way to inject this security flag into the running pod?
Also, thank you so much for moving this to a helm chart! We were doing “bleeding edge” things with some hand-crafted terraform, but now we’re on the helm chart. We’re using the operator for both ingresses and service endpoints and we couldn’t be happier with how well it works! Thanks for all the hard work on this project!
Thank you for all the contributions and useful information on this issue folks. Especially thanks to @konrad-ohms for the amazing device plugin description.
As folks already mentioned, with the change in containerd this is now going to be an issue for all Kubernetes distributions.
I’ve tested the device plugin installation and it works, but we would like to avoid a hard dependency on third party tool, so making the proxies that need to use tun device run in privileged containers seem a better solution and we are thinking of making that change, see https://github.com/tailscale/tailscale/pull/14262. Folks can then choose to install the device plugin and drop the privileged security context (and we should add docs for that). Note that we already run a privileged init container for all proxies that need tun device, so as far as I can tell this should not be a breaking change.
Longer term, we are wanting to look into running these proxies in netstack mode to avoid needing a tun device altogether.
Thank you for sharing such detailed notes @konrad-ohms, I can imagine how anyone that hits this coming from Talos will find them most helpful.
I am sticking to Talos 1.7.x for now, which doesn’t require
squat/generic-device-pluginfor the Tailscale operator to work. I’m pretty much at1.from this list: https://github.com/siderolabs/talos/issues/9322#issuecomment-2426833819Containerd is still at
2.0.0in the latest Talos 1.8.x available today (1.8.3 as of this comment). I would like to see containerd hit 2.1 before I pick it up. Based on https://containerd.io/releases/, I am guessing spring 2025. Hopefully by then we will have a versionedsquat/generic-device-plugin(note to self: search / open an issue for this).Thank you all for making this a great example of community effort!
Thank you @irbekrm. The
tailscale.com/proxy-classlabel works.A temporary workaround would be to use
ProxyClassto make thetailscalecontainer privileged likehttps://tailscale.com/kb/1236/kubernetes-operator#cluster-resource-customization-using-proxyclass-custom-resource
We need to take a look whether the tailscale container needs extra permissions to fix this by default.