tailscale: Tailscale Operator v1.80.0 cannot finalize LoadBalancer service; proxy pod loses connectivity to K8s API

What is the issue?

I’m trying to expose a MongoDB service on my Kubernetes cluster via Tailscale using the Tailscale Operator v1.80.0. Unfortunately, the service never becomes accessible, and after about 18 minutes, the Tailscale proxy pod restarts. Before the restart, it successfully connects to Kubernets API, Then it fails to reach the Kubernetes API (10.43.0.1).

Below are all the details, logs, and configurations (with any sensitive project name replaced by myProject).


Cluster & Environment

  • Cluster: RKE2 on Ubuntu 22.04 (kernel 5.19), running on Proxmox VMs in Hetzner
  • Kubernetes version: 1.27
  • Tailscale Operator: v1.80.0
  • CNI: Cilium with the following config:
    kubeProxyReplacement: strict
    socketLb:
      enabled: true
      hostNamespaceOnly: true
    k8sServiceHost: 172.17.100.250
    k8sServicePort: 6443
    cni:
      chainingMode: "none"
    
  • Both the Proxmox hosts and the VMs are in the same Tailnet.
  • Networking pods (operator & proxy) run in the tailscale namespace. The service is in developers namespace.

Steps to reproduce

I created a LoadBalancer service to expose MongoDB via Tailscale:

apiVersion: v1
kind: Service
metadata:
  name: mongodb
  namespace: developers
  annotations:
    tailscale.com/hostname: "mongodb-dev"
spec:
  type: LoadBalancer
  loadBalancerClass: "tailscale"
  ports:
    - name: mongodb
      port: 27017
      protocol: TCP
      targetPort: mongodb
  selector:
    app.kubernetes.io/component: mongodb
    app.kubernetes.io/instance: mongodb3
    app.kubernetes.io/name: mongodb
  sessionAffinity: None

Observed behavior

  • The service shows <pending> in the EXTERNAL-IP field:
    kubectl get service
    NAME      TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
    mongodb   LoadBalancer   10.43.165.86    <pending>     27017:32317/TCP  ...
    
  • The Tailscale network assigns an IP (100.79.156.63) and I can ping that Tailscale IP (mongodb-dev), but port 27017 is always “connection refused.”
    $ tailscale status | grep mongodb-dev
    100.79.156.63   mongodb-dev  tagged-devices linux   -
    
    $ tailscale ping mongodb-dev
    pong from mongodb-dev (100.79.156.63) via DERP(nue) in 77ms
    pong from mongodb-dev (100.79.156.63) via DERP(nue) in 51ms
    ...
    direct connection not established
    $ ping mongodb-dev
    PING mongodb-dev.halibut-lime.ts.net (100.79.156.63) 56(84) bytes of data.
    64 bytes from mongodb-dev.halibut-lime.ts.net (100.79.156.63): icmp_seq=1 ttl=64 time=47.9 ms
    64 bytes from mongodb-dev.halibut-lime.ts.net (100.79.156.63): icmp_seq=2 ttl=64 time=55.6 ms
    64 bytes from mongodb-dev.halibut-lime.ts.net (100.79.156.63): icmp_seq=3 ttl=64 time=55.1 ms
    64 bytes from mongodb-dev.halibut-lime.ts.net (100.79.156.63): icmp_seq=4 ttl=64 time=57.3 ms
    ^C
    --- mongodb-dev.halibut-lime.ts.net ping statistics ---
    4 packets transmitted, 4 received, 0% packet loss, time 3003ms
    rtt min/avg/max/mdev = 47.900/53.953/57.266/3.587 ms
    
    $ nc -v mongodb-dev 27017
    nc: connect to mongodb-dev (100.79.156.63) port 27017 (tcp) failed: Connection refused
    
  • Service show status: no Tailscale hostname known yet, waiting for proxy pod to finish auth:
    apiVersion: v1
    kind: Service
    metadata:
      annotations:
        tailscale.com/hostname: mongodb-dev
      creationTimestamp: "2025-03-03T19:56:54Z"
      finalizers:
      - tailscale.com/finalizer
      name: mongodb
      namespace: developers
      resourceVersion: "362541238"
      uid: c080652d-222e-43e0-b5b8-72a719fe014b
    spec:
      allocateLoadBalancerNodePorts: true
      clusterIP: 10.43.165.86
      clusterIPs:
      - 10.43.165.86
      externalTrafficPolicy: Cluster
      internalTrafficPolicy: Cluster
      ipFamilies:
      - IPv4
      ipFamilyPolicy: SingleStack
      loadBalancerClass: tailscale
      ports:
      - name: mongodb
        nodePort: 32317
        port: 27017
        protocol: TCP
        targetPort: mongodb
      selector:
        app.kubernetes.io/component: mongodb
        app.kubernetes.io/instance: mongodb3
        app.kubernetes.io/name: mongodb
      sessionAffinity: None
      type: LoadBalancer
    status:
      conditions:
      - lastTransitionTime: "2025-03-03T19:56:54Z"
        message: no Tailscale hostname known yet, waiting for proxy pod to finish auth
        reason: ProxyPending
        status: "False"
        type: TailscaleProxyReady
      loadBalancer: {}
    

Logs from the Tailscale proxy pod

  • After about 18 minutes, the Tailscale proxy pod restarts. Before the restart, it seems to connect successfully. Then it fails to reach the Kubernetes API (10.43.0.1) afterward. Here are some relevant lines from the proxy pod logs:

    ts-mongodb-z6hfm-0 tailscale 2025/03/03 19:57:21 health(warnable=no-derp-connection): ok
    ts-mongodb-z6hfm-0 tailscale 2025/03/03 19:58:24 wgengine: idle peer [/1F3u] now active, reconfiguring WireGuard
    ts-mongodb-z6hfm-0 tailscale 2025/03/03 19:58:24 wgengine: Reconfig: configuring userspace WireGuard config (with 2/13 peers)
    ts-mongodb-z6hfm-0 tailscale 2025/03/03 19:59:17 magicsock: disco: node [blykt] d:961278e2a5b4abfa now using 172.17.100.250:41641 mtu=1360 tx=bd2c029079d1
    ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:03:08 wgengine: Reconfig: configuring userspace WireGuard config (with 3/13 peers)
    ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:03:08 magicsock: disco: node [pRUdh] d:a19cc6b42dc1f70b now using 172.17.100.208:41641 mtu=1360 tx=2b687936ed06
    ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:03:08 wgengine: Reconfig: configuring userspace WireGuard config (with 4/13 peers)
    ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:03:08 magicsock: disco: node [Fv/PR] d:51121954c1b2acf2 now using 172.17.100.237:41641 mtu=1360 tx=7e4f6355bebb
    
    
    ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:09:58 Received error: PollNetMap: unexpected EOF
    ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:09:58 control: controlhttp: forcing port 443 dial due to recent noise dial
    ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:09:58 control: netmap: got new dial plan from control
    ts-mongodb-z6hfm-0 tailscale boot: 2025/03/03 20:12:59 Checking for consistent state
    ts-mongodb-z6hfm-0 tailscale boot: 2025/03/03 20:12:59 Sending SIGTERM to tailscaled
    ts-mongodb-z6hfm-0 tailscale boot: 2025/03/03 20:12:59 storing capability version and UID: Patch "https://kubernetes.default.svc/api/v1/namespaces/tailscale/secrets/ts-mongodb-z6hfm-0?fieldManager=tailscale-container": read tcp 10.42.1.119:51082->10.43.0.1:443: read: connection timed out
    ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 tailscaled got signal terminated; shutting down
    ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 canceling captive portal context
    ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 control: client.Shutdown ...
    ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 control: authRoutine: exiting
    ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 control: updateRoutine: exiting
    ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 control: mapRoutine: exiting
    ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 control: Client.Shutdown done.
    ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 magicsock: closing connection to derp-26 (conn-close), age 15m38s
    ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 magicsock: 0 active derp conns
    ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 monitor: RTM_DELROUTE: src=, dst=fe80::/64, gw=, outif=2, table=254
    ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 monitor: RTM_DELROUTE: src=, dst=fe80::/128, gw=, outif=2, table=255
    ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 monitor: RTM_DELROUTE: src=, dst=fe80::9aa4:819f:87ec:f691/128, gw=, outif=2, table=255
    ts-mongodb-z6hfm-0 tailscale 2025/03/03 20:12:59 monitor: RTM_DELROUTE: src=, dst=ff00::/8, gw=, outif=2, table=255
    - ts-mongodb-z6hfm-0 › tailscale
    
    • After this point, the Tailscale container is unable to contact 10.43.0.1, and it continues to fail with timeouts.
    + ts-mongodb-z6hfm-0 › tailscale
    ts-mongodb-z6hfm-0 tailscale boot: 2025/03/03 20:13:01 Using tailscaled config file "/etc/tsconfig/cap-107.hujson" to match current capability version 113
    ts-mongodb-z6hfm-0 tailscale boot: 2025/03/03 20:15:10 kubeclient: error checking Events permissions: Post "https://kubernetes.default.svc/apis/authorization.k8s.io/v1/selfsubjectaccessreviews": dial tcp 10.43.0.1:443: connect: connection timed out
    ts-mongodb-z6hfm-0 tailscale boot: 2025/03/03 20:15:10 kubeclient: this client is not able to write tailscaled Events to the Pod in which it is running.
    ts-mongodb-z6hfm-0 tailscale                    To help with future debugging you can make it able write Events by giving it get,create,patch permissions for Events in the Pod namespace
    ts-mongodb-z6hfm-0 tailscale                    and setting POD_NAME, POD_UID env vars for the Pod.
    ts-mongodb-z6hfm-0 tailscale boot: 2025/03/03 20:15:10 error checking get permission on secret ts-mongodb-z6hfm-0: Post "https://kubernetes.default.svc/apis/authorization.k8s.io/v1/selfsubjectaccessreviews": context deadline exceeded
    ts-mongodb-z6hfm-0 tailscale boot: 2025/03/03 20:15:10 error checking update permission on secret ts-mongodb-z6hfm-0: Post "https://kubernetes.default.svc/apis/authorization.k8s.io/v1/selfsubjectaccessreviews": context deadline exceeded
    ts-mongodb-z6hfm-0 tailscale boot: 2025/03/03 20:15:10 error checking patch permission on secret ts-mongodb-z6hfm-0: Post "https://kubernetes.default.svc/apis/authorization.k8s.io/v1/selfsubjectaccessreviews": context deadline exceeded
    ts-mongodb-z6hfm-0 tailscale boot: 2025/03/03 20:15:10 error setting up for running on Kubernetes: getting Tailscale state Secret ts-mongodb-z6hfm-0: Get "https://kubernetes.default.svc/api/v1/namespaces/tailscale/secrets/ts-mongodb-z6hfm-0": context deadline exceeded
    - ts-mongodb-z6hfm-0 › tailscale
    

Diagnostic inside the proxy pod

After the first 18 minutes, the Tailscale proxy pod restarts. Before that, it connects and authenticates successfully. Once it restarts, it can no longer reach 10.43.0.1. To demonstrate this, I run:

kubectl exec -ti statefulset.apps/ts-mongodb-z6hfm -- \
  sh -c "apk add curl && while true; do \
    date=\$(date -u +\"%Y-%m-%dT%H:%M:%SZ\"); \
    http=\$(curl -s -o /dev/null -w \"%{http_code}\" -k -m 3 https://10.43.0.1); \
    echo \"\$date: \$http\"; \
    sleep 1; \
  done"

Initially, I see 401 (unauthenticated request), but eventually it becomes 000 or times out:

# Repeatedly hitting Kubernetes API
2025-03-03T20:12:40Z: 401
2025-03-03T20:12:41Z: 401
...
2025-03-03T20:12:58Z: 401
<container terminates>

# After the restart:
2025-03-03T20:14:05Z: 000
2025-03-03T20:14:09Z: 000
  • Checking routes inside the pod:
# ping 10.43.0.1
PING 10.43.0.1 (10.43.0.1) 56(84) bytes of data.
--- 10.43.0.1 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4092ms

# ip r g 10.43.0.1
10.43.0.1 via 10.42.1.86 dev eth0 src 10.42.1.119 uid 0
    cache mtu 1230

Operator Logs

Similarly, here are excerpts from the Operator pod (operator-5fbcb7bb9b-dpgf8), which show repeated attempts to reconcile the service and patch the proxy Secret:

operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T19:57:21Z","logger":"tailscaled","msg":"[v1] netmap diff: (none)"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T19:57:21Z","logger":"tailscaled","msg":"[v1] magicsock: got updated network map; 13 peers"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T19:57:21Z","logger":"tailscaled","msg":"[v1] magicsock: peermtu: peer MTU status is false"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T19:57:21Z","logger":"tailscaled","msg":"[v1] wgengine: Reconfig done"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T19:57:21Z","logger":"tailscaled","msg":"[v1] authReconfig: ra=true dns=true 0x02: <nil>"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: mapRoutine: [v1] backoff: 14 msec"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v1] PollNetMap: unexpected EOF"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v1] sendStatus: mapRoutine1: state:authenticated"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v1] mapRoutine: state:authenticated"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v1] PollNetMap: stream=true ep=[188.40.51.175:56618 10.42.1.158:56618]"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"Received error: PollNetMap: unexpected EOF"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 3.30 seconds before dialing \"controlplane.tailscale.com\" @ 18.197.242.41"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 1.70 seconds before dialing \"controlplane.tailscale.com\" @ 3.78.132.46"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 1.80 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:4277:1574:2e7b:d812"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 1.60 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:2697:66bf:c24a:a674"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 2.00 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:4844:1954:f211:a255"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 2.10 seconds before dialing \"controlplane.tailscale.com\" @ 3.74.105.242"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 2.30 seconds before dialing \"controlplane.tailscale.com\" @ 35.156.244.91"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 0.80 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:789e:cd8d:3b4b:df4d"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 1.40 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:b1ee:6525:7c95:d0f3"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 1.50 seconds before dialing \"controlplane.tailscale.com\" @ 3.73.239.57"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 2.20 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:1e36:2966:5bec:a911"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 2.40 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:4d:f871:20f3:6602"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: trying to dial \"controlplane.tailscale.com\" @ 2a05:d014:386:202:f041:44b6:9559:668"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 2.50 seconds before dialing \"controlplane.tailscale.com\" @ 3.124.108.117"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 1.90 seconds before dialing \"controlplane.tailscale.com\" @ 52.28.255.255"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 1.20 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:c5e9:4c3f:9808:f2b0"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 0.40 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:30f5:de62:8eed:5c47"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 3.10 seconds before dialing \"controlplane.tailscale.com\" @ 3.125.194.96"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 3.20 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:9501:5da3:6d45:8ca0"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 2.80 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:6089:6a93:59a4:920a"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 2.60 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:9a01:a6fe:4251:8db8"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 0.30 seconds before dialing \"controlplane.tailscale.com\" @ 3.125.149.81"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 1.10 seconds before dialing \"controlplane.tailscale.com\" @ 3.69.76.60"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 1.30 seconds before dialing \"controlplane.tailscale.com\" @ 3.70.110.84"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 2.70 seconds before dialing \"controlplane.tailscale.com\" @ 18.159.140.118"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 2.90 seconds before dialing \"controlplane.tailscale.com\" @ 18.199.123.246"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 0.60 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:b88b:3cda:3907:3f2b"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 0.90 seconds before dialing \"controlplane.tailscale.com\" @ 18.159.235.209"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 3.00 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:eee7:5001:722d:1156"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 0.50 seconds before dialing \"controlplane.tailscale.com\" @ 18.197.213.150"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 1.00 seconds before dialing \"controlplane.tailscale.com\" @ 2a05:d014:386:202:256c:6ede:f4f9:a51b"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: waiting 0.70 seconds before dialing \"controlplane.tailscale.com\" @ 18.194.180.142"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v2] controlhttp: trying to dial \"controlplane.tailscale.com\" @ 3.125.149.81"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: controlhttp: forcing port 443 dial due to recent noise dial"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v1] controlhttp: high-priority success dialing \"controlplane.tailscale.com\" @ 3.125.149.81 from dial plan"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v\u0000JSON]1{\"controltime\":\"2025-03-03T20:11:15.693057573Z\"}"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: netmap: got new dial plan from control"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v1] new network map (periodic):\nnetmap: self: [+4lHm] auth=machine-authorized u=tailscale-operator.halibut-lime.ts.net [100.72.240.23/32 fd7a:115c:a1e0::6001:f017/128]\n"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v1] mapRoutine: netmap received: state:synchronized"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"control: [v1] sendStatus: mapRoutine-got-netmap: state:synchronized"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"[v1] netmap diff:\n- [TT0cx] d:0ec0b1a9b595075e D0 100.79.156.63 fd7a:115c:a1e0::4a01:9c40/128 : \n+ [TT0cx] d:0ec0b1a9b595075e D26 100.79.156.63 fd7a:115c:a1e0::4a01:9c40/128 :   188.40.51.175:41355     10.42.1.119:41355\n- [Oo7ax] d:4a5c46417601f67f D26 100.68.9.2 fd7a:115c:a1e0::301:902/128 :    23.88.46.174:42955    23.88.46.174:41641    23.88.46.174:41641       10.0.0.58:41641\n+ [Oo7ax] d:4a5c46417601f67f D26 100.68.9.2 fd7a:115c:a1e0::301:902/128 :    23.88.46.174:42955    23.88.46.174:41641       10.0.0.58:41641\n- [Fv/PR] d:51121954c1b2acf2 D26 100.80.155.103 fd7a:115c:a1e0::8501:9b6a/128 :     88.198.12.2:33390     88.198.12.2:55132     88.198.12.2:41641       10.0.0.54:41641      10.42.2.20:41641  172.17.100.237:41641\n+ [Fv/PR] d:51121954c1b2acf2 D26 100.80.155.103 fd7a:115c:a1e0::8501:9b6a/128 :     88.198.12.2:33390     88.198.12.2:41641     88.198.12.2:12118       10.0.0.54:41641      10.42.2.20:41641  172.17.100.237:41641\n"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"[v1] magicsock: got updated network map; 13 peers"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"[v1] magicsock: peermtu: peer MTU status is false"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"[v1] wgengine: Reconfig done"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:11:15Z","logger":"tailscaled","msg":"[v1] authReconfig: ra=true dns=true 0x02: <nil>"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:00Z","logger":"service-reconciler","msg":"starting reconcile","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:00Z","logger":"service-reconciler","msg":"reconciling headless service for StatefulSet","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:00Z","logger":"dns-records-reconciler","msg":"starting reconcile","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:00Z","logger":"dns-records-reconciler","msg":"DNSConfig does not exist, not creating DNS records","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:00Z","logger":"dns-records-reconciler","msg":"reconcile finished","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:00Z","logger":"service-reconciler","msg":"secret tailscale/ts-mongodb-z6hfm-0 already exists","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:00Z","logger":"service-reconciler","msg":"patching the existing proxy Secret with tailscaled config {\"Version\":\"alpha0\",\"Locked\":false,\"AuthKey\":\"**redacted**\",\"Hostname\":\"mongodb-dev\",\"acceptDNS\":false,\"acceptRoutes\":false,\"AppConnector\":{\"Advertise\":false},\"NoStatefulFiltering\":true}","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:00Z","logger":"service-reconciler","msg":"reconciling statefulset tailscale/ts-mongodb-z6hfm","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:00Z","logger":"service-reconciler","msg":"no Tailscale hostname known yet, waiting for proxy pod to finish auth","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:00Z","logger":"service-reconciler","msg":"reconcile finished","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"dns-records-reconciler","msg":"starting reconcile","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"dns-records-reconciler","msg":"DNSConfig does not exist, not creating DNS records","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"dns-records-reconciler","msg":"reconcile finished","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"service-reconciler","msg":"starting reconcile","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"service-reconciler","msg":"reconciling headless service for StatefulSet","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"dns-records-reconciler","msg":"starting reconcile","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"dns-records-reconciler","msg":"DNSConfig does not exist, not creating DNS records","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"dns-records-reconciler","msg":"reconcile finished","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"service-reconciler","msg":"secret tailscale/ts-mongodb-z6hfm-0 already exists","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"service-reconciler","msg":"patching the existing proxy Secret with tailscaled config {\"Version\":\"alpha0\",\"Locked\":false,\"AuthKey\":\"**redacted**\",\"Hostname\":\"mongodb-dev\",\"acceptDNS\":false,\"acceptRoutes\":false,\"AppConnector\":{\"Advertise\":false},\"NoStatefulFiltering\":true}","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"service-reconciler","msg":"reconciling statefulset tailscale/ts-mongodb-z6hfm","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"service-reconciler","msg":"no Tailscale hostname known yet, waiting for proxy pod to finish auth","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:13:01Z","logger":"service-reconciler","msg":"reconcile finished","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:11Z","logger":"service-reconciler","msg":"starting reconcile","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:11Z","logger":"service-reconciler","msg":"reconciling headless service for StatefulSet","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:11Z","logger":"dns-records-reconciler","msg":"starting reconcile","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:11Z","logger":"dns-records-reconciler","msg":"DNSConfig does not exist, not creating DNS records","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:11Z","logger":"dns-records-reconciler","msg":"reconcile finished","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:11Z","logger":"service-reconciler","msg":"secret tailscale/ts-mongodb-z6hfm-0 already exists","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:11Z","logger":"service-reconciler","msg":"patching the existing proxy Secret with tailscaled config {\"Version\":\"alpha0\",\"Locked\":false,\"AuthKey\":\"**redacted**\",\"Hostname\":\"mongodb-dev\",\"acceptDNS\":false,\"acceptRoutes\":false,\"AppConnector\":{\"Advertise\":false},\"NoStatefulFiltering\":true}","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:11Z","logger":"service-reconciler","msg":"reconciling statefulset tailscale/ts-mongodb-z6hfm","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:11Z","logger":"service-reconciler","msg":"no Tailscale hostname known yet, waiting for proxy pod to finish auth","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:11Z","logger":"service-reconciler","msg":"reconcile finished","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:12Z","logger":"dns-records-reconciler","msg":"starting reconcile","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:12Z","logger":"dns-records-reconciler","msg":"DNSConfig does not exist, not creating DNS records","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:12Z","logger":"dns-records-reconciler","msg":"reconcile finished","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:24Z","logger":"dns-records-reconciler","msg":"starting reconcile","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:24Z","logger":"dns-records-reconciler","msg":"DNSConfig does not exist, not creating DNS records","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:24Z","logger":"dns-records-reconciler","msg":"reconcile finished","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:24Z","logger":"service-reconciler","msg":"starting reconcile","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:24Z","logger":"service-reconciler","msg":"reconciling headless service for StatefulSet","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:24Z","logger":"service-reconciler","msg":"secret tailscale/ts-mongodb-z6hfm-0 already exists","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:24Z","logger":"service-reconciler","msg":"patching the existing proxy Secret with tailscaled config {\"Version\":\"alpha0\",\"Locked\":false,\"AuthKey\":\"**redacted**\",\"Hostname\":\"mongodb-dev\",\"acceptDNS\":false,\"acceptRoutes\":false,\"AppConnector\":{\"Advertise\":false},\"NoStatefulFiltering\":true}","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:24Z","logger":"service-reconciler","msg":"reconciling statefulset tailscale/ts-mongodb-z6hfm","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:24Z","logger":"service-reconciler","msg":"no Tailscale hostname known yet, waiting for proxy pod to finish auth","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:24Z","logger":"service-reconciler","msg":"reconcile finished","service-ns":"developers","service-name":"mongodb"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:25Z","logger":"dns-records-reconciler","msg":"starting reconcile","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:25Z","logger":"dns-records-reconciler","msg":"DNSConfig does not exist, not creating DNS records","Service":"tailscale/ts-mongodb-z6hfm"}
operator-5fbcb7bb9b-dpgf8 operator {"level":"debug","ts":"2025-03-03T20:15:25Z","logger":"dns-records-reconciler","msg":"reconcile finished","Service":"tailscale/ts-mongodb-z6hfm"}

Are there any recent changes that introduced the issue?

No response

OS

Linux

OS version

No response

Tailscale version

1.80

Other software

No response

Bug report

No response

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 4
  • Comments: 17

Most upvoted comments

Fixed for me by setting the following in my Cilium configmap and no additional changes:

data:
  bpf-lb-sock-hostns-only: "true"

This is a requirement for usage with Cilium as per the docs here: https://tailscale.com/kb/1236/kubernetes-operator#cilium-in-kube-proxy-replacement-mode

Edit: Seems like @WMP already has this relevant configuration set

also seeing this issue with 1.82.0 and k8s 1.32 my CNI is also cilium