tailscale: Tailscale Operator v1.80.0 cannot finalize LoadBalancer service; proxy pod loses connectivity to K8s API
What is the issue?
I’m trying to expose a MongoDB service on my Kubernetes cluster via Tailscale using the Tailscale Operator v1.80.0. Unfortunately, the service never becomes accessible, and after about 18 minutes, the Tailscale proxy pod restarts. Before the restart, it successfully connects to Kubernets API, Then it fails to reach the Kubernetes API (10.43.0.1).
Below are all the details, logs, and configurations (with any sensitive project name replaced by myProject).
Cluster & Environment
- Cluster: RKE2 on Ubuntu 22.04 (kernel 5.19), running on Proxmox VMs in Hetzner
- Kubernetes version: 1.27
- Tailscale Operator: v1.80.0
- CNI: Cilium with the following config:
kubeProxyReplacement: strict socketLb: enabled: true hostNamespaceOnly: true k8sServiceHost: 172.17.100.250 k8sServicePort: 6443 cni: chainingMode: "none" - Both the Proxmox hosts and the VMs are in the same Tailnet.
- Networking pods (operator & proxy) run in the tailscale namespace. The service is in developers namespace.
Steps to reproduce
I created a LoadBalancer service to expose MongoDB via Tailscale:
apiVersion: v1
kind: Service
metadata:
name: mongodb
namespace: developers
annotations:
tailscale.com/hostname: "mongodb-dev"
spec:
type: LoadBalancer
loadBalancerClass: "tailscale"
ports:
- name: mongodb
port: 27017
protocol: TCP
targetPort: mongodb
selector:
app.kubernetes.io/component: mongodb
app.kubernetes.io/instance: mongodb3
app.kubernetes.io/name: mongodb
sessionAffinity: None
Observed behavior
- The service shows
<pending>in theEXTERNAL-IPfield:kubectl get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE mongodb LoadBalancer 10.43.165.86 <pending> 27017:32317/TCP ... - The Tailscale network assigns an IP (
100.79.156.63) and I can ping that Tailscale IP (mongodb-dev), but port 27017 is always “connection refused.”$ tailscale status | grep mongodb-dev 100.79.156.63 mongodb-dev tagged-devices linux - $ tailscale ping mongodb-dev pong from mongodb-dev (100.79.156.63) via DERP(nue) in 77ms pong from mongodb-dev (100.79.156.63) via DERP(nue) in 51ms ... direct connection not established $ ping mongodb-dev PING mongodb-dev.halibut-lime.ts.net (100.79.156.63) 56(84) bytes of data. 64 bytes from mongodb-dev.halibut-lime.ts.net (100.79.156.63): icmp_seq=1 ttl=64 time=47.9 ms 64 bytes from mongodb-dev.halibut-lime.ts.net (100.79.156.63): icmp_seq=2 ttl=64 time=55.6 ms 64 bytes from mongodb-dev.halibut-lime.ts.net (100.79.156.63): icmp_seq=3 ttl=64 time=55.1 ms 64 bytes from mongodb-dev.halibut-lime.ts.net (100.79.156.63): icmp_seq=4 ttl=64 time=57.3 ms ^C --- mongodb-dev.halibut-lime.ts.net ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3003ms rtt min/avg/max/mdev = 47.900/53.953/57.266/3.587 ms $ nc -v mongodb-dev 27017 nc: connect to mongodb-dev (100.79.156.63) port 27017 (tcp) failed: Connection refused - Service show status:
no Tailscale hostname known yet, waiting for proxy pod to finish auth:apiVersion: v1 kind: Service metadata: annotations: tailscale.com/hostname: mongodb-dev creationTimestamp: "2025-03-03T19:56:54Z" finalizers: - tailscale.com/finalizer name: mongodb namespace: developers resourceVersion: "362541238" uid: c080652d-222e-43e0-b5b8-72a719fe014b spec: allocateLoadBalancerNodePorts: true clusterIP: 10.43.165.86 clusterIPs: - 10.43.165.86 externalTrafficPolicy: Cluster internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack loadBalancerClass: tailscale ports: - name: mongodb nodePort: 32317 port: 27017 protocol: TCP targetPort: mongodb selector: app.kubernetes.io/component: mongodb app.kubernetes.io/instance: mongodb3 app.kubernetes.io/name: mongodb sessionAffinity: None type: LoadBalancer status: conditions: - lastTransitionTime: "2025-03-03T19:56:54Z" message: no Tailscale hostname known yet, waiting for proxy pod to finish auth reason: ProxyPending status: "False" type: TailscaleProxyReady loadBalancer: {}
Logs from the Tailscale proxy pod
-
After about 18 minutes, the Tailscale proxy pod restarts. Before the restart, it seems to connect successfully. Then it fails to reach the Kubernetes API (
10.43.0.1) afterward. Here are some relevant lines from the proxy pod logs:ts-mongodb-z6hfm-0 tailscale 2025/03/03 19:57:21 health(warnable=no-derp-connection): ok ts-mongodb-z6hfm-0 tailscale 2025/03/03 19:58:24 wgengine: idle peer [/1F3u] now active, reconfiguring WireGuard ts-mongodb-z6hfm-0 tailscale 2025/03/03 19:58:24 wgengine: Reconfig: configuring userspace WireGuard config (with 2/13 peers) ts-mongodb-z6hfm-0 tailscale 2025/03/03 19:59:17 magicsock: disco: node [blykt] d:961278e2a5b4abfa now using 172.17.100.250:41641 mtu=1360 tx=bd2c029079d1 ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:03:08 wgengine: Reconfig: configuring userspace WireGuard config (with 3/13 peers) ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:03:08 magicsock: disco: node [pRUdh] d:a19cc6b42dc1f70b now using 172.17.100.208:41641 mtu=1360 tx=2b687936ed06 ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:03:08 wgengine: Reconfig: configuring userspace WireGuard config (with 4/13 peers) ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:03:08 magicsock: disco: node [Fv/PR] d:51121954c1b2acf2 now using 172.17.100.237:41641 mtu=1360 tx=7e4f6355bebb ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:09:58 Received error: PollNetMap: unexpected EOF ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:09:58 control: controlhttp: forcing port 443 dial due to recent noise dial ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:09:58 control: netmap: got new dial plan from control ts-mongodb-z6hfm-0 tailscale boot: 2025/03/03 20:12:59 Checking for consistent state ts-mongodb-z6hfm-0 tailscale boot: 2025/03/03 20:12:59 Sending SIGTERM to tailscaled ts-mongodb-z6hfm-0 tailscale boot: 2025/03/03 20:12:59 storing capability version and UID: Patch "https://kubernetes.default.svc/api/v1/namespaces/tailscale/secrets/ts-mongodb-z6hfm-0?fieldManager=tailscale-container": read tcp 10.42.1.119:51082->10.43.0.1:443: read: connection timed out ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 tailscaled got signal terminated; shutting down ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 canceling captive portal context ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 control: client.Shutdown ... ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 control: authRoutine: exiting ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 control: updateRoutine: exiting ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 control: mapRoutine: exiting ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 control: Client.Shutdown done. ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 magicsock: closing connection to derp-26 (conn-close), age 15m38s ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 magicsock: 0 active derp conns ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 monitor: RTM_DELROUTE: src=, dst=fe80::/64, gw=, outif=2, table=254 ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 monitor: RTM_DELROUTE: src=, dst=fe80::/128, gw=, outif=2, table=255 ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 monitor: RTM_DELROUTE: src=, dst=fe80::9aa4:819f:87ec:f691/128, gw=, outif=2, table=255 ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 monitor: RTM_DELROUTE: src=, dst=ff00::/8, gw=, outif=2, table=255 - ts-mongodb-z6hfm-0 › tailscale- After this point, the Tailscale container is unable to contact
10.43.0.1, and it continues to fail with timeouts.
+ ts-mongodb-z6hfm-0 › tailscale ts-mongodb-z6hfm-0 tailscale boot: 2025/03/03 20:13:01 Using tailscaled config file "/etc/tsconfig/cap-107.hujson" to match current capability version 113 ts-mongodb-z6hfm-0 tailscale boot: 2025/03/03 20:15:10 kubeclient: error checking Events permissions: Post "https://kubernetes.default.svc/apis/authorization.k8s.io/v1/selfsubjectaccessreviews": dial tcp 10.43.0.1:443: connect: connection timed out ts-mongodb-z6hfm-0 tailscale boot: 2025/03/03 20:15:10 kubeclient: this client is not able to write tailscaled Events to the Pod in which it is running. ts-mongodb-z6hfm-0 tailscale To help with future debugging you can make it able write Events by giving it get,create,patch permissions for Events in the Pod namespace ts-mongodb-z6hfm-0 tailscale and setting POD_NAME, POD_UID env vars for the Pod. ts-mongodb-z6hfm-0 tailscale boot: 2025/03/03 20:15:10 error checking get permission on secret ts-mongodb-z6hfm-0: Post "https://kubernetes.default.svc/apis/authorization.k8s.io/v1/selfsubjectaccessreviews": context deadline exceeded ts-mongodb-z6hfm-0 tailscale boot: 2025/03/03 20:15:10 error checking update permission on secret ts-mongodb-z6hfm-0: Post "https://kubernetes.default.svc/apis/authorization.k8s.io/v1/selfsubjectaccessreviews": context deadline exceeded ts-mongodb-z6hfm-0 tailscale boot: 2025/03/03 20:15:10 error checking patch permission on secret ts-mongodb-z6hfm-0: Post "https://kubernetes.default.svc/apis/authorization.k8s.io/v1/selfsubjectaccessreviews": context deadline exceeded ts-mongodb-z6hfm-0 tailscale boot: 2025/03/03 20:15:10 error setting up for running on Kubernetes: getting Tailscale state Secret ts-mongodb-z6hfm-0: Get "https://kubernetes.default.svc/api/v1/namespaces/tailscale/secrets/ts-mongodb-z6hfm-0": context deadline exceeded - ts-mongodb-z6hfm-0 › tailscale - After this point, the Tailscale container is unable to contact
Diagnostic inside the proxy pod
After the first 18 minutes, the Tailscale proxy pod restarts. Before that, it connects and authenticates successfully. Once it restarts, it can no longer reach 10.43.0.1. To demonstrate this, I run:
kubectl exec -ti statefulset.apps/ts-mongodb-z6hfm -- \
sh -c "apk add curl && while true; do \
date=\$(date -u +\"%Y-%m-%dT%H:%M:%SZ\"); \
http=\$(curl -s -o /dev/null -w \"%{http_code}\" -k -m 3 https://10.43.0.1); \
echo \"\$date: \$http\"; \
sleep 1; \
done"
Initially, I see 401 (unauthenticated request), but eventually it becomes 000 or times out:
# Repeatedly hitting Kubernetes API
2025-03-03T20:12:40Z: 401
2025-03-03T20:12:41Z: 401
...
2025-03-03T20:12:58Z: 401
<container terminates>
# After the restart:
2025-03-03T20:14:05Z: 000
2025-03-03T20:14:09Z: 000
- Checking routes inside the pod:
# ping 10.43.0.1
PING 10.43.0.1 (10.43.0.1) 56(84) bytes of data.
--- 10.43.0.1 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4092ms
# ip r g 10.43.0.1
10.43.0.1 via 10.42.1.86 dev eth0 src 10.42.1.119 uid 0
cache mtu 1230
Operator Logs
Similarly, here are excerpts from the Operator pod (operator-5fbcb7bb9b-dpgf8), which show repeated attempts to reconcile the service and patch the proxy Secret:
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T19:57:21Z","logger":"tailscaled","msg":"[v1] netmap diff: (none)"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T19:57:21Z","logger":"tailscaled","msg":"[v1] magicsock: got updated network map; 13 peers"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T19:57:21Z","logger":"tailscaled","msg":"[v1] magicsock: peermtu: peer MTU status is false"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T19:57:21Z","logger":"tailscaled","msg":"[v1] wgengine: Reconfig done"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T19:57:21Z","logger":"tailscaled","msg":"[v1] authReconfig: ra=true dns=true 0x02: <nil>"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: mapRoutine: [v1] backoff: 14 msec"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v1] PollNetMap: unexpected EOF"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v1] sendStatus: mapRoutine1: state:authenticated"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v1] mapRoutine: state:authenticated"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v1] PollNetMap: stream=true ep=[188.40.51.175:56618 10.42.1.158:56618]"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"Received error: PollNetMap: unexpected EOF"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 3.30 seconds before dialing \"controlplane.tailscale.com\" @ 18.197.242.41"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 1.70 seconds before dialing \"controlplane.tailscale.com\" @ 3.78.132.46"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 1.80 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:4277:1574:2e7b:d812"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 1.60 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:2697:66bf:c24a:a674"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 2.00 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:4844:1954:f211:a255"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 2.10 seconds before dialing \"controlplane.tailscale.com\" @ 3.74.105.242"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 2.30 seconds before dialing \"controlplane.tailscale.com\" @ 35.156.244.91"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 0.80 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:789e:cd8d:3b4b:df4d"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 1.40 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:b1ee:6525:7c95:d0f3"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 1.50 seconds before dialing \"controlplane.tailscale.com\" @ 3.73.239.57"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 2.20 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:1e36:2966:5bec:a911"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 2.40 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:4d:f871:20f3:6602"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: trying to dial \"controlplane.tailscale.com\" @ 2a05:d014:386:202:f041:44b6:9559:668"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 2.50 seconds before dialing \"controlplane.tailscale.com\" @ 3.124.108.117"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 1.90 seconds before dialing \"controlplane.tailscale.com\" @ 52.28.255.255"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 1.20 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:c5e9:4c3f:9808:f2b0"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 0.40 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:30f5:de62:8eed:5c47"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 3.10 seconds before dialing \"controlplane.tailscale.com\" @ 3.125.194.96"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 3.20 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:9501:5da3:6d45:8ca0"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 2.80 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:6089:6a93:59a4:920a"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 2.60 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:9a01:a6fe:4251:8db8"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 0.30 seconds before dialing \"controlplane.tailscale.com\" @ 3.125.149.81"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 1.10 seconds before dialing \"controlplane.tailscale.com\" @ 3.69.76.60"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 1.30 seconds before dialing \"controlplane.tailscale.com\" @ 3.70.110.84"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 2.70 seconds before dialing \"controlplane.tailscale.com\" @ 18.159.140.118"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 2.90 seconds before dialing \"controlplane.tailscale.com\" @ 18.199.123.246"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 0.60 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:b88b:3cda:3907:3f2b"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 0.90 seconds before dialing \"controlplane.tailscale.com\" @ 18.159.235.209"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 3.00 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:eee7:5001:722d:1156"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 0.50 seconds before dialing \"controlplane.tailscale.com\" @ 18.197.213.150"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 1.00 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:256c:6ede:f4f9:a51b"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 0.70 seconds before dialing \"controlplane.tailscale.com\" @ 18.194.180.142"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: trying to dial \"controlplane.tailscale.com\" @ 3.125.149.81"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: controlhttp: forcing port 443 dial due to recent noise dial"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v1] controlhttp: high-priority success dialing \"controlplane.tailscale.com\" @ 3.125.149.81 from dial plan"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v\u0000JSON]1{\"controltime\":\"2025-03-03T20:11:15.693057573Z\"}"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: netmap: got new dial plan from control"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v1] new network map (periodic):\nnetmap: self: [+4lHm] auth=machine-authorized u=tailscale-operator.halibut-lime.ts.net [100.72.240.23/32 fd7a:115c:a1e0::6001:f017/128]\n"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v1] mapRoutine: netmap received: state:synchronized"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v1] sendStatus: mapRoutine-got-netmap: state:synchronized"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"[v1] netmap diff:\n- [TT0cx] d:0ec0b1a9b595075e D0 100.79.156.63 fd7a:115c:a1e0::4a01:9c40/128 : \n+ [TT0cx] d:0ec0b1a9b595075e D26 100.79.156.63 fd7a:115c:a1e0::4a01:9c40/128 : 188.40.51.175:41355 10.42.1.119:41355\n- [Oo7ax] d:4a5c46417601f67f D26 100.68.9.2 fd7a:115c:a1e0::301:902/128 : 23.88.46.174:42955 23.88.46.174:41641 23.88.46.174:41641 10.0.0.58:41641\n+ [Oo7ax] d:4a5c46417601f67f D26 100.68.9.2 fd7a:115c:a1e0::301:902/128 : 23.88.46.174:42955 23.88.46.174:41641 10.0.0.58:41641\n- [Fv/PR] d:51121954c1b2acf2 D26 100.80.155.103 fd7a:115c:a1e0::8501:9b6a/128 : 88.198.12.2:33390 88.198.12.2:55132 88.198.12.2:41641 10.0.0.54:41641 10.42.2.20:41641 172.17.100.237:41641\n+ [Fv/PR] d:51121954c1b2acf2 D26 100.80.155.103 fd7a:115c:a1e0::8501:9b6a/128 : 88.198.12.2:33390 88.198.12.2:41641 88.198.12.2:12118 10.0.0.54:41641 10.42.2.20:41641 172.17.100.237:41641\n"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"[v1] magicsock: got updated network map; 13 peers"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"[v1] magicsock: peermtu: peer MTU status is false"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"[v1] wgengine: Reconfig done"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"[v1] authReconfig: ra=true dns=true 0x02: <nil>"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:00Z","logger":"service-reconciler","msg":"starting reconcile","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:00Z","logger":"service-reconciler","msg":"reconciling headless service for StatefulSet","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:00Z","logger":"dns-records-reconciler","msg":"starting reconcile","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:00Z","logger":"dns-records-reconciler","msg":"DNSConfig does not exist, not creating DNS records","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:00Z","logger":"dns-records-reconciler","msg":"reconcile finished","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:00Z","logger":"service-reconciler","msg":"secret tailscale/ts-mongodb-z6hfm-0 already exists","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:00Z","logger":"service-reconciler","msg":"patching the existing proxy Secret with tailscaled config {\"Version\":\"alpha0\",\"Locked\":false,\"AuthKey\":\"**redacted**\",\"Hostname\":\"mongodb-dev\",\"acceptDNS\":false,\"acceptRoutes\":false,\"AppConnector\":{\"Advertise\":false},\"NoStatefulFiltering\":true}","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:00Z","logger":"service-reconciler","msg":"reconciling statefulset tailscale/ts-mongodb-z6hfm","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:00Z","logger":"service-reconciler","msg":"no Tailscale hostname known yet, waiting for proxy pod to finish auth","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:00Z","logger":"service-reconciler","msg":"reconcile finished","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"dns-records-reconciler","msg":"starting reconcile","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"dns-records-reconciler","msg":"DNSConfig does not exist, not creating DNS records","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"dns-records-reconciler","msg":"reconcile finished","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"service-reconciler","msg":"starting reconcile","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"service-reconciler","msg":"reconciling headless service for StatefulSet","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"dns-records-reconciler","msg":"starting reconcile","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"dns-records-reconciler","msg":"DNSConfig does not exist, not creating DNS records","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"dns-records-reconciler","msg":"reconcile finished","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"service-reconciler","msg":"secret tailscale/ts-mongodb-z6hfm-0 already exists","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"service-reconciler","msg":"patching the existing proxy Secret with tailscaled config {\"Version\":\"alpha0\",\"Locked\":false,\"AuthKey\":\"**redacted**\",\"Hostname\":\"mongodb-dev\",\"acceptDNS\":false,\"acceptRoutes\":false,\"AppConnector\":{\"Advertise\":false},\"NoStatefulFiltering\":true}","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"service-reconciler","msg":"reconciling statefulset tailscale/ts-mongodb-z6hfm","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"service-reconciler","msg":"no Tailscale hostname known yet, waiting for proxy pod to finish auth","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"service-reconciler","msg":"reconcile finished","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:11Z","logger":"service-reconciler","msg":"starting reconcile","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:11Z","logger":"service-reconciler","msg":"reconciling headless service for StatefulSet","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:11Z","logger":"dns-records-reconciler","msg":"starting reconcile","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:11Z","logger":"dns-records-reconciler","msg":"DNSConfig does not exist, not creating DNS records","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:11Z","logger":"dns-records-reconciler","msg":"reconcile finished","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:11Z","logger":"service-reconciler","msg":"secret tailscale/ts-mongodb-z6hfm-0 already exists","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:11Z","logger":"service-reconciler","msg":"patching the existing proxy Secret with tailscaled config {\"Version\":\"alpha0\",\"Locked\":false,\"AuthKey\":\"**redacted**\",\"Hostname\":\"mongodb-dev\",\"acceptDNS\":false,\"acceptRoutes\":false,\"AppConnector\":{\"Advertise\":false},\"NoStatefulFiltering\":true}","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:11Z","logger":"service-reconciler","msg":"reconciling statefulset tailscale/ts-mongodb-z6hfm","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:11Z","logger":"service-reconciler","msg":"no Tailscale hostname known yet, waiting for proxy pod to finish auth","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:11Z","logger":"service-reconciler","msg":"reconcile finished","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:12Z","logger":"dns-records-reconciler","msg":"starting reconcile","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:12Z","logger":"dns-records-reconciler","msg":"DNSConfig does not exist, not creating DNS records","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:12Z","logger":"dns-records-reconciler","msg":"reconcile finished","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:24Z","logger":"dns-records-reconciler","msg":"starting reconcile","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:24Z","logger":"dns-records-reconciler","msg":"DNSConfig does not exist, not creating DNS records","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:24Z","logger":"dns-records-reconciler","msg":"reconcile finished","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:24Z","logger":"service-reconciler","msg":"starting reconcile","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:24Z","logger":"service-reconciler","msg":"reconciling headless service for StatefulSet","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:24Z","logger":"service-reconciler","msg":"secret tailscale/ts-mongodb-z6hfm-0 already exists","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:24Z","logger":"service-reconciler","msg":"patching the existing proxy Secret with tailscaled config {\"Version\":\"alpha0\",\"Locked\":false,\"AuthKey\":\"**redacted**\",\"Hostname\":\"mongodb-dev\",\"acceptDNS\":false,\"acceptRoutes\":false,\"AppConnector\":{\"Advertise\":false},\"NoStatefulFiltering\":true}","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:24Z","logger":"service-reconciler","msg":"reconciling statefulset tailscale/ts-mongodb-z6hfm","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:24Z","logger":"service-reconciler","msg":"no Tailscale hostname known yet, waiting for proxy pod to finish auth","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:24Z","logger":"service-reconciler","msg":"reconcile finished","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:25Z","logger":"dns-records-reconciler","msg":"starting reconcile","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:25Z","logger":"dns-records-reconciler","msg":"DNSConfig does not exist, not creating DNS records","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:25Z","logger":"dns-records-reconciler","msg":"reconcile finished","Service":"tailscale/ts-mongodb-z6hfm"}
Are there any recent changes that introduced the issue?
No response
OS
Linux
OS version
No response
Tailscale version
1.80
Other software
No response
Bug report
No response
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 4
- Comments: 17
Fixed for me by setting the following in my Cilium configmap and no additional changes:
This is a requirement for usage with Cilium as per the docs here: https://tailscale.com/kb/1236/kubernetes-operator#cilium-in-kube-proxy-replacement-mode
Edit: Seems like @WMP already has this relevant configuration set
also seeing this issue with
1.82.0and k8s1.32my CNI is also cilium