tailscale: Tailscale Operator does not pass TS_DEBUG_FIREWALL_MODE to pods that it creates
What is the issue?
Relates to #391 Relates to #7895
Tailscale Operator does not pass TS_DEBUG_FIREWALL_MODE to pods that it creates
In the Oracle Kubernetes (OKE) environment used by QuickNode, I recently tested the latest Tailscale Operator. I used this:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: operator
namespace: tailscale
spec:
replicas: 1
selector:
matchLabels:
app: operator
strategy:
type: Recreate
template:
metadata:
labels:
app: operator
spec:
containers:
- env:
- name: PROXY_IMAGE
value: docker.io/tailscale/tailscale:unstable-v1.49.197
- name: TS_DEBUG_FIREWALL_MODE
value: nftables
- name: AUTH_PROXY
value: "true"
- name: OPERATOR_LOGGING
value: info
- name: OPERATOR_HOSTNAME
value: tailscale-operator
- name: OPERATOR_SECRET
value: operator
- name: OPERATOR_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: CLIENT_ID_FILE
value: /oauth/client_id
- name: CLIENT_SECRET_FILE
value: /oauth/client_secret
- name: PROXY_TAGS
value: tag:k8s
image: docker.io/tailscale/k8s-operator:unstable-v1.49.197
name: operator
resources:
requests:
cpu: 500m
memory: 100Mi
volumeMounts:
- mountPath: /oauth
name: oauth
readOnly: true
serviceAccountName: operator
volumes:
- name: oauth
secret:
secretName: operator-oauth
The operator successfully deployed. I was also able to successfully creates an ingress which uses funnel, as I mentioned here:
However, I also have a service which uses LoadBalancer, and is annotated like this:
---
apiVersion: v1
kind: Service
metadata:
annotations:
tailscale.com/hostname: myargocdserver
finalizers:
- tailscale.com/finalizer
labels:
app.kubernetes.io/component: server
app.kubernetes.io/instance: argocd
app.kubernetes.io/name: argocd-server
app.kubernetes.io/part-of: argocd
name: argocd-server
namespace: argocd
spec:
allocateLoadBalancerNodePorts: true
externalTrafficPolicy: Cluster
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
loadBalancerClass: tailscale
ports:
- name: http
nodePort: 31514
port: 80
protocol: TCP
targetPort: 8080
- name: https
nodePort: 30761
port: 443
protocol: TCP
targetPort: 8080
selector:
app.kubernetes.io/name: argocd-server
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer:
ingress:
- hostname: myargocdserver.xxx.ts.net
In this case the pod that the operator creates comes up but it does not use TS_DEBUG_FIREWALL_MODE , and instead tries to use the “old” iptables logic.
This results in errors that look like this:
boot: 2023/09/08 19:00:58 No authkey found in kube secret and TS_AUTHKEY not provided, login will be interactive if needed.
boot: 2023/09/08 19:00:58 Starting tailscaled
boot: 2023/09/08 19:00:58 Waiting for tailscaled socket
2023/09/08 19:00:58 logtail started
2023/09/08 19:00:58 Program starting: v1.49.197-t3280c81c9, Go 1.21.0: []string{"tailscaled", "--socket=/tmp/tailscaled.sock", "--state=kube:ts-victoria-metrics-cluster-vminsert-7f5ml-0", "--statedir=/tmp"}
2023/09/08 19:00:58 LogID: 45f47924847605de89fb765b2db56e0d12b46189b3de4fc8d08193541f0ae37e
2023/09/08 19:00:58 logpolicy: using system state directory "/var/lib/tailscale"
logpolicy.ConfigFromFile /var/lib/tailscale/tailscaled.log.conf: open /var/lib/tailscale/tailscaled.log.conf: no such file or directory
logpolicy.Config.Validate for /var/lib/tailscale/tailscaled.log.conf: config is nil
2023/09/08 19:00:58 wgengine.NewUserspaceEngine(tun "tailscale0") ...
2023/09/08 19:00:58 router: default choosing iptables
2023/09/08 19:00:58 router: using iptables
2023/09/08 19:00:58 router: v6nat = false
2023/09/08 19:00:58 dns: [rc=unknown ret=direct]
2023/09/08 19:00:58 dns: using "direct" mode
2023/09/08 19:00:58 dns: using *dns.directManager
2023/09/08 19:00:58 link state: interfaces.State{defaultRoute=eth0 ifs={eth0:[XXXX llu6]} v4=true v6=false}
2023/09/08 19:00:58 magicsock: disco key = XXXXX
2023/09/08 19:00:58 Creating WireGuard device...
2023/09/08 19:00:58 Bringing WireGuard device up...
2023/09/08 19:00:58 Bringing router up...
2023/09/08 19:00:58 external route: up
2023/09/08 19:00:58 Clearing router settings...
2023/09/08 19:00:58 Starting network monitor...
2023/09/08 19:00:58 Engine created.
2023/09/08 19:00:58 monitor: ip rule deleted: {Family:2 DstLength:0 SrcLength:0 Tos:0 Table:254 Protocol:0 Scope:0 Type:1 Flags:0 Attributes:{Dst:<nil> Src:<nil> Gateway:<nil> OutIface:0 Priority:5210 Table:254 Mark:16711680 Pref:<nil> Expires:<nil> Metrics:<nil> Multipath:[]}}
2023/09/08 19:00:58 monitor: ip rule deleted: {Family:2 DstLength:0 SrcLength:0 Tos:0 Table:253 Protocol:0 Scope:0 Type:1 Flags:0 Attributes:{Dst:<nil> Src:<nil> Gateway:<nil> OutIface:0 Priority:5230 Table:253 Mark:16711680 Pref:<nil> Expires:<nil> Metrics:<nil> Multipath:[]}}
2023/09/08 19:00:58 monitor: ip rule deleted: {Family:2 DstLength:0 SrcLength:0 Tos:0 Table:0 Protocol:0 Scope:0 Type:7 Flags:0 Attributes:{Dst:<nil> Src:<nil> Gateway:<nil> OutIface:0 Priority:5250 Table:0 Mark:16711680 Pref:<nil> Expires:<nil> Metrics:<nil> Multipath:[]}}
2023/09/08 19:00:58 monitor: ip rule deleted: {Family:2 DstLength:0 SrcLength:0 Tos:0 Table:52 Protocol:0 Scope:0 Type:1 Flags:0 Attributes:{Dst:<nil> Src:<nil> Gateway:<nil> OutIface:0 Priority:5270 Table:52 Mark:0 Pref:<nil> Expires:<nil> Metrics:<nil> Multipath:[]}}
2023/09/08 19:00:58 pm: using backend prefs for "profile-c02b": Prefs{ra=false dns=false want=true routes=[] nf=on host="monitoring-vminsert-dev-k8s" update=check Persist{lm=, o=, n=[0lmAA] u="monitoring-vminsert-dev-k8s.taild857f.ts.net"}}
2023/09/08 19:00:58 logpolicy: using system state directory "/var/lib/tailscale"
2023/09/08 19:00:58 got LocalBackend in 68ms
2023/09/08 19:00:58 Start
2023/09/08 19:00:58 Backend: logs: be:45f47924847605de89fb765b2db56e0d12b46189b3de4fc8d08193541f0ae37e fe:
2023/09/08 19:00:58 control: client.Login(false, 0)
2023/09/08 19:00:58 health("overall"): error: not in map poll
2023/09/08 19:00:58 control: doLogin(regen=false, hasUrl=false)
boot: 2023/09/08 19:00:58 tailscaled in state "NoState", waiting
2023/09/08 19:00:59 router: somebody (likely systemd-networkd) deleted ip rules; restoring Tailscale's
2023/09/08 19:00:59 control: control server key from https://controlplane.tailscale.com: ts2021=[fSeS+], legacy=[nlFWp]
2023/09/08 19:00:59 control: RegisterReq: onode= node=[0lmAA] fup=false nks=false
2023/09/08 19:00:59 control: creating new noise client
2023/09/08 19:00:59 control: RegisterReq: got response; nodeKeyExpired=false, machineAuthorized=true; authURL=false
2023/09/08 19:01:02 control: netmap: got new dial plan from control
2023/09/08 19:01:02 active login: yyy.xxx.ts.net
2023/09/08 19:01:02 Switching ipn state NoState -> Starting (WantRunning=true, nm=true)
2023/09/08 19:01:02 magicsock: SetPrivateKey called (init)
2023/09/08 19:01:02 wgengine: Reconfig: configuring userspace WireGuard config (with 0/1569 peers)
2023/09/08 19:01:02 wgengine: Reconfig: starting up network logger (node:xxx tailnet:xxx)
2023/09/08 19:01:02 wgengine: Reconfig: configuring router
2023/09/08 19:01:02 health("router"): error: setting up filter/ts-input: running [/sbin/iptables -t filter -N ts-input --wait]: exit status 3: modprobe: can't change directory to '/lib/modules': No such file or directory
iptables v1.8.8 (legacy): can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
2023/09/08 19:01:02 peerapi: serving on http://xxx.xxx.xxx
2023/09/08 19:01:02 peerapi: serving on http://[xxx]:33847
boot: 2023/09/08 19:01:02 tailscaled in state "Starting", waiting
2023/09/08 19:01:02 magicsock: home is now derp-1 (nyc)
2023/09/08 19:01:02 magicsock: adding connection to derp-1 for home-keep-alive
2023/09/08 19:01:02 magicsock: 1 active derp conns: derp-1=cr0s,wr0s
2023/09/08 19:01:02 magicsock: endpoints changed: xxx:42829 (stun), 10.244.6.179:60681 (local)
2023/09/08 19:01:02 control: NetInfo: NetInfo{varies=true hairpin=false ipv6=false ipv6os=true udp=true icmpv4=false derp=#1 portmap= link="" firewallmode="ipt-default"}
2023/09/08 19:01:02 derphttp.Client.Connect: connecting to derp-1 (nyc)
2023/09/08 19:01:02 Switching ipn state Starting -> Running (WantRunning=true, nm=true)
boot: 2023/09/08 19:01:02 Running 'tailscale set'
2023/09/08 19:01:02 magicsock: derp-1 connected; connGen=1
boot: 2023/09/08 19:01:02 Deleting authkey from kube secret
boot: 2023/09/08 19:01:02 Installing proxy rules
modprobe: can't change directory to '/lib/modules': No such file or directory
modprobe: can't change directory to '/lib/modules': No such file or directory
modprobe: can't change directory to '/lib/modules': No such file or directory
iptables v1.8.8 (legacy): can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
boot: 2023/09/08 19:01:02 installing ingress proxy rules: executing iptables failed: exit status 3
The operator needs to be able to pass TS_DEBUG_FIREWALL_MODE
to the pods that are spun up in this case, so that we can force using nftables.
Steps to reproduce
Deploy Tailscale operator using config I listed above in an Oracle Kubernetes environment
Are there any recent changes that introduced the issue?
No response
OS
Linux
OS version
Oracle Linux Server 8.8 , https://docs.oracle.com/en-us/iaas/images/image/7e68ad9b-2109-4cc2-a7af-2f47e8900817/
Tailscale version
unstable-v1.49.197
Other software
No response
Bug report
BUG-3f7fc157b0949c712ce2ee441e83d2869cdf7282bc253adc827ec04eefc02a02-20230908205306Z-e7929ed9cfaa8cff
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 15 (14 by maintainers)
Commits related to this issue
- cmd/containerboot,cmd/k8s-operator: ingress/egress proxy can optionally use nftables Makes it possible for users to make the ingress/egress proxies created by the operator to use nftables by setting ... — committed to tailscale/tailscale by irbekrm 3 years ago
- cmd/containerboot,cmd/k8s-operator: ingress/egress proxy can optionally use nftables Makes it possible for users to make the ingress/egress proxies created by the operator to use nftables by setting ... — committed to tailscale/tailscale by irbekrm 3 years ago
- cmd/containerboot,cmd/k8s-operator: ingress/egress proxy can optionally use nftables Makes it possible for users to make the ingress/egress proxies created by the operator to use nftables by setting ... — committed to tailscale/tailscale by irbekrm 3 years ago
- cmd/containerboot,cmd/k8s-operator: ingress/egress proxy can optionally use nftables Makes it possible for users to make the ingress/egress proxies created by the operator to use nftables by setting ... — committed to tailscale/tailscale by irbekrm 3 years ago
- cmd/containerboot,cmd/k8s-operator: ingress/egress proxy can optionally use nftables Makes it possible for users to make the ingress/egress proxies created by the operator to use nftables by setting ... — committed to tailscale/tailscale by irbekrm 3 years ago
- util/linuxfw: move detection logic Just a refactor to consolidate this to be used by a later commit in containerboot. Updates #9310 Signed-off-by: Maisem Ali <maisem@tailscale.com> — committed to tailscale/tailscale by deleted user 3 years ago
- util/linuxfw: move detection logic Just a refactor to consolidate the firewall detection logic in a single pacakage so that it can be reused in a later commit by containerboot. Updates #9310 Signed... — committed to tailscale/tailscale by deleted user 3 years ago
- util/linuxfw: move detection logic Just a refactor to consolidate the firewall detection logic in a single package so that it can be reused in a later commit by containerboot. Updates #9310 Signed-... — committed to tailscale/tailscale by deleted user 3 years ago
- util/linuxfw: move detection logic Just a refactor to consolidate the firewall detection logic in a single package so that it can be reused in a later commit by containerboot. Updates #9310 Signed-... — committed to tailscale/tailscale by deleted user 3 years ago
- util/linuxfw: move detection logic Just a refactor to consolidate the firewall detection logic in a single package so that it can be reused in a later commit by containerboot. Updates #9310 Signed-... — committed to tailscale/tailscale by deleted user 3 years ago
- cmd/containerboot: use linuxfw.NetfilterRunner This migrates containerboot to reuse the NetfilterRunner used by tailscaled instead of manipulating iptables rule itself. This has the added advantage o... — committed to tailscale/tailscale by deleted user 3 years ago
- cmd/containerboot: use linuxfw.NetfilterRunner This migrates containerboot to reuse the NetfilterRunner used by tailscaled instead of manipulating iptables rule itself. This has the added advantage o... — committed to tailscale/tailscale by deleted user 3 years ago
- cmd/containerboot: use linuxfw.NetfilterRunner This migrates containerboot to reuse the NetfilterRunner used by tailscaled instead of manipulating iptables rule itself. This has the added advantage o... — committed to tailscale/tailscale by deleted user 3 years ago
- cmd/containerboot: use linuxfw.NetfilterRunner This migrates containerboot to reuse the NetfilterRunner used by tailscaled instead of manipulating iptables rule itself. This has the added advantage o... — committed to tailscale/tailscale by deleted user 3 years ago
- cmd/containerboot: use linuxfw.NetfilterRunner This migrates containerboot to reuse the NetfilterRunner used by tailscaled instead of manipulating iptables rule itself. This has the added advantage o... — committed to tailscale/tailscale by deleted user 3 years ago
- cmd/containerboot: use linuxfw.NetfilterRunner This migrates containerboot to reuse the NetfilterRunner used by tailscaled instead of manipulating iptables rule itself. This has the added advantage o... — committed to tailscale/tailscale by deleted user 3 years ago
- cmd/k8s-operator: add simple validation for firewall mode Make sure that if an invalid firewall mode gets configured, the operator will not start provisioning proxy resources, but will instead log an... — committed to tailscale/tailscale by irbekrm 3 years ago
- cmd/k8s-operator: add simple validation for firewall mode Make sure that if an invalid firewall mode gets configured, the operator will not start provisioning proxy resources, but will instead log an... — committed to tailscale/tailscale by irbekrm 3 years ago
- cmd/k8s-operator: add simple validation for firewall mode Make sure that if an invalid firewall mode gets configured, the operator will not start provisioning proxy resources, but will instead log an... — committed to tailscale/tailscale by irbekrm 3 years ago
- cmd/k8s-operator: add simple validation for firewall mode Make sure that if an invalid firewall mode gets configured, the operator will not start provisioning proxy resources, but will instead log an... — committed to tailscale/tailscale by irbekrm 3 years ago
Yes this fix will be tricky since the combinations of iptables (legacy) vs. nftables are plentiful. In addition to Oracle Kubernetes using nftables which I have found,
@clarkezone found that Azure AKS uses nftables here: https://github.com/tailscale/tailscale/issues/8111#issuecomment-1657235953
@zyclonite found that CoreOS uses nftables here: https://github.com/tailscale/tailscale/issues/5621#issuecomment-1707599073