tailscale: Tailscale Operator does not pass TS_DEBUG_FIREWALL_MODE to pods that it creates

What is the issue?

Relates to #391 Relates to #7895

Tailscale Operator does not pass TS_DEBUG_FIREWALL_MODE to pods that it creates

In the Oracle Kubernetes (OKE) environment used by QuickNode, I recently tested the latest Tailscale Operator. I used this:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: operator
  namespace: tailscale
spec:
  replicas: 1
  selector:
    matchLabels:
      app: operator
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: operator
    spec:
      containers:
      - env:
        - name: PROXY_IMAGE
          value: docker.io/tailscale/tailscale:unstable-v1.49.197
        - name: TS_DEBUG_FIREWALL_MODE
          value: nftables
        - name: AUTH_PROXY
          value: "true"
        - name: OPERATOR_LOGGING
          value: info
        - name: OPERATOR_HOSTNAME
          value: tailscale-operator
        - name: OPERATOR_SECRET
          value: operator
        - name: OPERATOR_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: CLIENT_ID_FILE
          value: /oauth/client_id
        - name: CLIENT_SECRET_FILE
          value: /oauth/client_secret
        - name: PROXY_TAGS
          value: tag:k8s
        image: docker.io/tailscale/k8s-operator:unstable-v1.49.197
        name: operator
        resources:
          requests:
            cpu: 500m
            memory: 100Mi
        volumeMounts:
        - mountPath: /oauth
          name: oauth
          readOnly: true
      serviceAccountName: operator
      volumes:
      - name: oauth
        secret:
          secretName: operator-oauth

The operator successfully deployed. I was also able to successfully creates an ingress which uses funnel, as I mentioned here:

However, I also have a service which uses LoadBalancer, and is annotated like this:

---
apiVersion: v1
kind: Service
metadata:
  annotations:
    tailscale.com/hostname: myargocdserver
  finalizers:
  - tailscale.com/finalizer
  labels:
    app.kubernetes.io/component: server
    app.kubernetes.io/instance: argocd
    app.kubernetes.io/name: argocd-server
    app.kubernetes.io/part-of: argocd
  name: argocd-server
  namespace: argocd
spec:
  allocateLoadBalancerNodePorts: true
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  loadBalancerClass: tailscale
  ports:
  - name: http
    nodePort: 31514
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: https
    nodePort: 30761
    port: 443
    protocol: TCP
    targetPort: 8080
  selector:
    app.kubernetes.io/name: argocd-server
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - hostname: myargocdserver.xxx.ts.net

In this case the pod that the operator creates comes up but it does not use TS_DEBUG_FIREWALL_MODE , and instead tries to use the “old” iptables logic.

This results in errors that look like this:

boot: 2023/09/08 19:00:58 No authkey found in kube secret and TS_AUTHKEY not provided, login will be interactive if needed.
boot: 2023/09/08 19:00:58 Starting tailscaled
boot: 2023/09/08 19:00:58 Waiting for tailscaled socket
2023/09/08 19:00:58 logtail started
2023/09/08 19:00:58 Program starting: v1.49.197-t3280c81c9, Go 1.21.0: []string{"tailscaled", "--socket=/tmp/tailscaled.sock", "--state=kube:ts-victoria-metrics-cluster-vminsert-7f5ml-0", "--statedir=/tmp"}
2023/09/08 19:00:58 LogID: 45f47924847605de89fb765b2db56e0d12b46189b3de4fc8d08193541f0ae37e
2023/09/08 19:00:58 logpolicy: using system state directory "/var/lib/tailscale"
logpolicy.ConfigFromFile /var/lib/tailscale/tailscaled.log.conf: open /var/lib/tailscale/tailscaled.log.conf: no such file or directory
logpolicy.Config.Validate for /var/lib/tailscale/tailscaled.log.conf: config is nil
2023/09/08 19:00:58 wgengine.NewUserspaceEngine(tun "tailscale0") ...
2023/09/08 19:00:58 router: default choosing iptables
2023/09/08 19:00:58 router: using iptables
2023/09/08 19:00:58 router: v6nat = false
2023/09/08 19:00:58 dns: [rc=unknown ret=direct]
2023/09/08 19:00:58 dns: using "direct" mode
2023/09/08 19:00:58 dns: using *dns.directManager
2023/09/08 19:00:58 link state: interfaces.State{defaultRoute=eth0 ifs={eth0:[XXXX llu6]} v4=true v6=false}
2023/09/08 19:00:58 magicsock: disco key = XXXXX
2023/09/08 19:00:58 Creating WireGuard device...
2023/09/08 19:00:58 Bringing WireGuard device up...
2023/09/08 19:00:58 Bringing router up...
2023/09/08 19:00:58 external route: up
2023/09/08 19:00:58 Clearing router settings...
2023/09/08 19:00:58 Starting network monitor...
2023/09/08 19:00:58 Engine created.
2023/09/08 19:00:58 monitor: ip rule deleted: {Family:2 DstLength:0 SrcLength:0 Tos:0 Table:254 Protocol:0 Scope:0 Type:1 Flags:0 Attributes:{Dst:<nil> Src:<nil> Gateway:<nil> OutIface:0 Priority:5210 Table:254 Mark:16711680 Pref:<nil> Expires:<nil> Metrics:<nil> Multipath:[]}}
2023/09/08 19:00:58 monitor: ip rule deleted: {Family:2 DstLength:0 SrcLength:0 Tos:0 Table:253 Protocol:0 Scope:0 Type:1 Flags:0 Attributes:{Dst:<nil> Src:<nil> Gateway:<nil> OutIface:0 Priority:5230 Table:253 Mark:16711680 Pref:<nil> Expires:<nil> Metrics:<nil> Multipath:[]}}
2023/09/08 19:00:58 monitor: ip rule deleted: {Family:2 DstLength:0 SrcLength:0 Tos:0 Table:0 Protocol:0 Scope:0 Type:7 Flags:0 Attributes:{Dst:<nil> Src:<nil> Gateway:<nil> OutIface:0 Priority:5250 Table:0 Mark:16711680 Pref:<nil> Expires:<nil> Metrics:<nil> Multipath:[]}}
2023/09/08 19:00:58 monitor: ip rule deleted: {Family:2 DstLength:0 SrcLength:0 Tos:0 Table:52 Protocol:0 Scope:0 Type:1 Flags:0 Attributes:{Dst:<nil> Src:<nil> Gateway:<nil> OutIface:0 Priority:5270 Table:52 Mark:0 Pref:<nil> Expires:<nil> Metrics:<nil> Multipath:[]}}
2023/09/08 19:00:58 pm: using backend prefs for "profile-c02b": Prefs{ra=false dns=false want=true routes=[] nf=on host="monitoring-vminsert-dev-k8s" update=check Persist{lm=, o=, n=[0lmAA] u="monitoring-vminsert-dev-k8s.taild857f.ts.net"}}
2023/09/08 19:00:58 logpolicy: using system state directory "/var/lib/tailscale"
2023/09/08 19:00:58 got LocalBackend in 68ms
2023/09/08 19:00:58 Start
2023/09/08 19:00:58 Backend: logs: be:45f47924847605de89fb765b2db56e0d12b46189b3de4fc8d08193541f0ae37e fe:
2023/09/08 19:00:58 control: client.Login(false, 0)
2023/09/08 19:00:58 health("overall"): error: not in map poll
2023/09/08 19:00:58 control: doLogin(regen=false, hasUrl=false)
boot: 2023/09/08 19:00:58 tailscaled in state "NoState", waiting
2023/09/08 19:00:59 router: somebody (likely systemd-networkd) deleted ip rules; restoring Tailscale's
2023/09/08 19:00:59 control: control server key from https://controlplane.tailscale.com: ts2021=[fSeS+], legacy=[nlFWp]
2023/09/08 19:00:59 control: RegisterReq: onode= node=[0lmAA] fup=false nks=false
2023/09/08 19:00:59 control: creating new noise client
2023/09/08 19:00:59 control: RegisterReq: got response; nodeKeyExpired=false, machineAuthorized=true; authURL=false
2023/09/08 19:01:02 control: netmap: got new dial plan from control
2023/09/08 19:01:02 active login: yyy.xxx.ts.net
2023/09/08 19:01:02 Switching ipn state NoState -> Starting (WantRunning=true, nm=true)
2023/09/08 19:01:02 magicsock: SetPrivateKey called (init)
2023/09/08 19:01:02 wgengine: Reconfig: configuring userspace WireGuard config (with 0/1569 peers)
2023/09/08 19:01:02 wgengine: Reconfig: starting up network logger (node:xxx tailnet:xxx)
2023/09/08 19:01:02 wgengine: Reconfig: configuring router
2023/09/08 19:01:02 health("router"): error: setting up filter/ts-input: running [/sbin/iptables -t filter -N ts-input --wait]: exit status 3: modprobe: can't change directory to '/lib/modules': No such file or directory
iptables v1.8.8 (legacy): can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
2023/09/08 19:01:02 peerapi: serving on http://xxx.xxx.xxx
2023/09/08 19:01:02 peerapi: serving on http://[xxx]:33847
boot: 2023/09/08 19:01:02 tailscaled in state "Starting", waiting
2023/09/08 19:01:02 magicsock: home is now derp-1 (nyc)
2023/09/08 19:01:02 magicsock: adding connection to derp-1 for home-keep-alive
2023/09/08 19:01:02 magicsock: 1 active derp conns: derp-1=cr0s,wr0s
2023/09/08 19:01:02 magicsock: endpoints changed: xxx:42829 (stun), 10.244.6.179:60681 (local)
2023/09/08 19:01:02 control: NetInfo: NetInfo{varies=true hairpin=false ipv6=false ipv6os=true udp=true icmpv4=false derp=#1 portmap= link="" firewallmode="ipt-default"}
2023/09/08 19:01:02 derphttp.Client.Connect: connecting to derp-1 (nyc)
2023/09/08 19:01:02 Switching ipn state Starting -> Running (WantRunning=true, nm=true)
boot: 2023/09/08 19:01:02 Running 'tailscale set'
2023/09/08 19:01:02 magicsock: derp-1 connected; connGen=1
boot: 2023/09/08 19:01:02 Deleting authkey from kube secret
boot: 2023/09/08 19:01:02 Installing proxy rules
modprobe: can't change directory to '/lib/modules': No such file or directory
modprobe: can't change directory to '/lib/modules': No such file or directory
modprobe: can't change directory to '/lib/modules': No such file or directory
iptables v1.8.8 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
boot: 2023/09/08 19:01:02 installing ingress proxy rules: executing iptables failed: exit status 3

The operator needs to be able to pass TS_DEBUG_FIREWALL_MODE to the pods that are spun up in this case, so that we can force using nftables.

Steps to reproduce

Deploy Tailscale operator using config I listed above in an Oracle Kubernetes environment

Are there any recent changes that introduced the issue?

No response

OS

Linux

OS version

Oracle Linux Server 8.8 , https://docs.oracle.com/en-us/iaas/images/image/7e68ad9b-2109-4cc2-a7af-2f47e8900817/

Tailscale version

unstable-v1.49.197

Other software

No response

Bug report

BUG-3f7fc157b0949c712ce2ee441e83d2869cdf7282bc253adc827ec04eefc02a02-20230908205306Z-e7929ed9cfaa8cff

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (14 by maintainers)

Commits related to this issue

Most upvoted comments

Yes this fix will be tricky since the combinations of iptables (legacy) vs. nftables are plentiful. In addition to Oracle Kubernetes using nftables which I have found,

@clarkezone found that Azure AKS uses nftables here: https://github.com/tailscale/tailscale/issues/8111#issuecomment-1657235953

@zyclonite found that CoreOS uses nftables here: https://github.com/tailscale/tailscale/issues/5621#issuecomment-1707599073