tailscale: Tailscale disconnectivity with unexpected error "log derp-1 does not know about peer"

What is the issue?

  • it spreads across two of our clusters with very different sizes (~10 vs ~100+)
  • when it happens, the problematic node can’t access some of the peers and can access some others
  • when the log stops, connections are resumed
  • it suddenly started from yesterday (with no change on our end) and become quite frequent today
Mar 29 21:53:32 146-235-225-157 tailscaled[1754]: [RATELIMIT] format("open-conn-track: timeout opening %v to node %v; online=%v, lastRecv=%v")
Mar 29 21:53:33 146-235-225-157 tailscaled[1754]: [unexpected] magicsock: derp-1 does not know about peer [R68An], removing route
Mar 29 21:53:34 146-235-225-157 tailscaled[1754]: Accept: TCP{100.93.234.70:54682 > 10.0.80.89:10250} 92 tcp non-syn
Mar 29 21:53:38 146-235-225-157 tailscaled[1754]: [unexpected] magicsock: derp-1 does not know about peer [R68An], removing route
Mar 29 21:53:43 146-235-225-157 tailscaled[1754]: [RATELIMIT] format("open-conn-track: timeout opening %v to node %v; online=%v, lastRecv=%v") (31 dropped)
Mar 29 21:53:43 146-235-225-157 tailscaled[1754]: open-conn-track: timeout opening (TCP 100.79.193.119:46489 => 10.0.42.131:443) to node [R68An]; online=no, lastseen=1248h33m43s, lastRecv=never
Mar 29 21:53:43 146-235-225-157 tailscaled[1754]: [RATELIMIT] format("open-conn-track: timeout opening %v to node %v; online=%v, lastRecv=%v")
Mar 29 21:53:43 146-235-225-157 tailscaled[1754]: [unexpected] magicsock: derp-1 does not know about peer [R68An], removing route
Mar 29 21:53:48 146-235-225-157 tailscaled[1754]: [unexpected] magicsock: derp-1 does not know about peer [R68An], removing route

Steps to reproduce

No response

Are there any recent changes that introduced the issue?

No response

OS

Linux

OS version

linux 5.15.0-101-generic and linux v6.2

Tailscale version

1.52.1

Other software

No response

Bug report

BUG-3f60dfef69f7d82d9ac990fd92d287e5f2acec980793e7013d6ce3c4ef698139-20240329215413Z-48a3a7403f53d0e8

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 10
  • Comments: 40 (1 by maintainers)

Most upvoted comments

Having the same issue here, tailscale connected to headscale. It causes periodic disconnects where the exit node goes offline, but then comes back after 1-2 minutes. Syslog is spammed with these errors.

Seeing this issue a ton. Refrained from opening an issue because we are using Headscale, but it seems like this is not a Headscale bug, since people here are having it with regular Tailscale.

I’m seeing the bug with both Tailscale DERP and our own custom DERP server (which I set up thinking there was something wrong with Tailscale’s DERP servers due to this bug). It’s making some of our nodes completely inaccessible until I restart tailscaled, and often that only helps for a little.

I also have a few improvements on the systemd unit and script that @expoli posted above. Namely, it no longer Requires= tailscaled, but just uses After= so that the monitor stays up even while tailscaled restarts. It also uses a different looping mechanism that avoids catching rapid multiples of the same error message - triggering multiple restarts:

tailscaled-monitor.service:

[Unit]
Description=Handle various bugs in tailscale (such as DERP bug here https://github.com/tailscale/tailscale/issues/11565) by monitoring tailscaled logs, looking for failures, and restarting the appropriate service
After=network.target tailscaled.service

[Service]
ExecStart=/bin/bash /usr/bin/tailscaled-monitor.sh
Restart=always
User=root
StandardOutput=journal
StandardError=journal
TimeoutSec=0
RestartSec=60s

[Install]
WantedBy=multi-user.target

tailscaled-monitor.sh:

#!/bin/bash

logger -t tailscaled-monitor -p user.info "Monitoring tailscaled logs..."

while true; do

    # Follow journal for tailscaled, ignoring any old lines
    journalctl --follow --lines=0 -u tailscaled.service | while read line; do

        if echo "$line" | grep -q "magicsock: derp-[0-9]* does not know about peer"; then

            logger -t tailscaled-monitor -p user.info "DERP Bug detected. Restarting tailscaled..."

            echo "restarted tailscaled"
            systemctl restart tailscaled

            # give the service a chance to restart
            sleep 10

            # break out of inner loop to discard any additional logs and start listening fresh
            break
        fi
    done
done

Same here

root@fnos-nas:~# tailscale --version
+ tailscale --version
1.76.6
  tailscale commit: 1edcf9d466ceafedd2816db1a24d5ba4b0b18a5b
  other commit: d0a6cd8b27eb46f6dec31425499159f7949be7f9
  go version: go1.23.1
  • error output
root@fnos-nas:~# journalctl -u tailscaled.service -r -n 6000 | grep 'magicsock: derp-900 does not know about peer'
+ grep --color=auto 'magicsock: derp-900 does not know about peer'
+ journalctl -u tailscaled.service -r -n 6000
Dec 02 09:46:17 fnos-nas tailscaled[55172]: [unexpected] magicsock: derp-900 does not know about peer [DkPTz], removing route
Dec 02 09:46:11 fnos-nas tailscaled[55172]: [unexpected] magicsock: derp-900 does not know about peer [DkPTz], removing route
Dec 02 09:46:05 fnos-nas tailscaled[55172]: [unexpected] magicsock: derp-900 does not know about peer [DkPTz], removing route
Dec 02 09:45:59 fnos-nas tailscaled[55172]: [unexpected] magicsock: derp-900 does not know about peer [DkPTz], removing route
Dec 02 09:45:53 fnos-nas tailscaled[55172]: [unexpected] magicsock: derp-900 does not know about peer [DkPTz], removing route
Dec 02 09:45:47 fnos-nas tailscaled[55172]: [unexpected] magicsock: derp-900 does not know about peer [DkPTz], removing route
Dec 02 09:45:41 fnos-nas tailscaled[55172]: [unexpected] magicsock: derp-900 does not know about peer [DkPTz], removing route
Dec 02 09:45:37 fnos-nas tailscaled[55172]: [unexpected] magicsock: derp-900 does not know about peer [DkPTz], removing route
Dec 02 09:45:35 fnos-nas tailscaled[55172]: [unexpected] magicsock: derp-900 does not know about peer [DkPTz], removing route
Dec 02 09:45:32 fnos-nas tailscaled[55172]: [unexpected] magicsock: derp-900 does not know about peer [DkPTz], removing route
Dec 02 09:44:57 fnos-nas tailscaled[55172]: [unexpected] magicsock: derp-900 does not know about peer [RxjY9], removing route

The specific behavior is that after tailscale runs for a period of time, this node cannot be connected by other nodes through the derp relay node (the relay node built by itself). Other nodes in the same network can connect normally. After restarting tailscaled service with systemd, the connection behavior of nodes returned to normal. I used a bash script to monitor the log output of tailscaled service to restart tailscaled service.

  • /usr/local/bin/tailscaled-log-monitor.sh
+ cat /usr/local/bin/tailscaled-log-monitor.sh
#!/bin/bash
# 监听 tailscaled 日志,匹配特定错误并重启服务

journalctl -f -u tailscaled.service | while read line; do
  logger -t tailscaled-log-monitor -p user.info "$line" | grep -q "magicsock: derp-900 does not know about peer"
  if [ $? -eq 0 ]; then
    logger -t tailscaled-log-monitor -p user.info "Match found: $line"
    logger -t tailscaled-log-monitor -p user.info "Restarting tailscaled..."
    systemctl restart tailscaled.service
    # 增加冷却时间,避免频繁重启
    sleep 60  # 等待 60 秒后再继续检测
  fi
done
  • /etc/systemd/system/tailscaled-log-monitor.service
+ cat /etc/systemd/system/tailscaled-log-monitor.service
[Unit]
Description=Monitor tailscaled logs for specific errors and restart tailscaled service
After=network.target tailscaled.service
Requires=tailscaled.service

[Service]
ExecStart=/usr/local/bin/tailscaled-log-monitor.sh
Restart=always
User=root
StandardOutput=journal
StandardError=journal
TimeoutSec=0
RestartSec=5s

[Install]
WantedBy=multi-user.target

We are also having same behavior with 1.66.4 version

May 28 08:52:12 staging-pri-app-01 tailscaled[10744]: [unexpected] magicsock: derp-4 does not know about peer [LM9CM], removing route
May 28 08:52:17 staging-pri-app-01 tailscaled[10744]: [unexpected] magicsock: derp-4 does not know about peer [LM9CM], removing route

when connection makes direct again connection restores.

cc: @santiagoalvarez68

@lukehsiao to provide a potential explanation from Tailscale support team sometime ago, although i haven’t found a way to verify this:

The [redacted] node may have been terminated on the AWS side, but the node is still listed in Tailscale, with it's approved routes still in place, which is why it is still showing up in logs and having an effect on devices. Once the node is offline, it should be removed from the Tailscale configuration as well to prevent issues like you are seeing here.

Taking a look at the logs, there was a moment where the coordination server appears to have lost contact with the new subnet router for the [redacted], and swapped over to the other node that has that subnet approved, the old offline node. The coordination server chooses the active subnet router for a subnet based on several factors, one of which is the age of the node. The older a node, the higher priority it is, so when the new subnet router went offline, it swapped back to the old subnet router as all nodes advertising the subnet were offline and it defaults to the oldest node. When the new node came back online after a couple seconds, it swapped back and connections were restored.

To prevent this from happening in the future, removing disconnected nodes, or at least un-approving their advertised routes, will ensure that the online subnet routers are not swapped out for known bad nodes when connectivity makes them appear offline temporarily.

not sure if it’s applicable to your case. what makes the suggestion a bit hard to automate is, tailscale cli cannot remove itself from the telnet and probably need to setup some automation job to scan the nodes, which feels not as safe.

reason i am asking is i used to annoy @bradfitz with issue #9525 and eventually gave up (same happens in docker btw) but what i recall is random nodes dropping out, being reachable by some nodes but not others and completely random stuff and just coming back etc. same like you AND i had the same “does not know about peer removing route” error msgs related to exactly these nodes. these were nodes that had direct connections, no firewall etc. one thing though they had ipv6 enabled on the system but no ipv6 connectivity / ips setup. in case any of this helps