tailscale: Tailscale disconnectivity with unexpected error "log derp-1 does not know about peer"
What is the issue?
- it spreads across two of our clusters with very different sizes (~10 vs ~100+)
- when it happens, the problematic node can’t access some of the peers and can access some others
- when the log stops, connections are resumed
- it suddenly started from yesterday (with no change on our end) and become quite frequent today
Mar 29 21:53:32 146-235-225-157 tailscaled[1754]: [RATELIMIT] format("open-conn-track: timeout opening %v to node %v; online=%v, lastRecv=%v")
Mar 29 21:53:33 146-235-225-157 tailscaled[1754]: [unexpected] magicsock: derp-1 does not know about peer [R68An], removing route
Mar 29 21:53:34 146-235-225-157 tailscaled[1754]: Accept: TCP{100.93.234.70:54682 > 10.0.80.89:10250} 92 tcp non-syn
Mar 29 21:53:38 146-235-225-157 tailscaled[1754]: [unexpected] magicsock: derp-1 does not know about peer [R68An], removing route
Mar 29 21:53:43 146-235-225-157 tailscaled[1754]: [RATELIMIT] format("open-conn-track: timeout opening %v to node %v; online=%v, lastRecv=%v") (31 dropped)
Mar 29 21:53:43 146-235-225-157 tailscaled[1754]: open-conn-track: timeout opening (TCP 100.79.193.119:46489 => 10.0.42.131:443) to node [R68An]; online=no, lastseen=1248h33m43s, lastRecv=never
Mar 29 21:53:43 146-235-225-157 tailscaled[1754]: [RATELIMIT] format("open-conn-track: timeout opening %v to node %v; online=%v, lastRecv=%v")
Mar 29 21:53:43 146-235-225-157 tailscaled[1754]: [unexpected] magicsock: derp-1 does not know about peer [R68An], removing route
Mar 29 21:53:48 146-235-225-157 tailscaled[1754]: [unexpected] magicsock: derp-1 does not know about peer [R68An], removing route
Steps to reproduce
No response
Are there any recent changes that introduced the issue?
No response
OS
Linux
OS version
linux 5.15.0-101-generic and linux v6.2
Tailscale version
1.52.1
Other software
No response
Bug report
BUG-3f60dfef69f7d82d9ac990fd92d287e5f2acec980793e7013d6ce3c4ef698139-20240329215413Z-48a3a7403f53d0e8
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 10
- Comments: 40 (1 by maintainers)
Having the same issue here, tailscale connected to headscale. It causes periodic disconnects where the exit node goes offline, but then comes back after 1-2 minutes. Syslog is spammed with these errors.
Seeing this issue a ton. Refrained from opening an issue because we are using Headscale, but it seems like this is not a Headscale bug, since people here are having it with regular Tailscale.
I’m seeing the bug with both Tailscale DERP and our own custom DERP server (which I set up thinking there was something wrong with Tailscale’s DERP servers due to this bug). It’s making some of our nodes completely inaccessible until I restart tailscaled, and often that only helps for a little.
I also have a few improvements on the systemd unit and script that @expoli posted above. Namely, it no longer Requires= tailscaled, but just uses After= so that the monitor stays up even while tailscaled restarts. It also uses a different looping mechanism that avoids catching rapid multiples of the same error message - triggering multiple restarts:
tailscaled-monitor.service:
tailscaled-monitor.sh:
Same here
The specific behavior is that after tailscale runs for a period of time, this node cannot be connected by other nodes through the derp relay node (the relay node built by itself). Other nodes in the same network can connect normally. After restarting tailscaled service with systemd, the connection behavior of nodes returned to normal. I used a bash script to monitor the log output of tailscaled service to restart tailscaled service.
/etc/systemd/system/tailscaled-log-monitor.serviceWe are also having same behavior with
1.66.4versionwhen connection makes direct again connection restores.
cc: @santiagoalvarez68
@lukehsiao to provide a potential explanation from Tailscale support team sometime ago, although i haven’t found a way to verify this:
not sure if it’s applicable to your case. what makes the suggestion a bit hard to automate is, tailscale cli cannot remove itself from the telnet and probably need to setup some automation job to scan the nodes, which feels not as safe.
reason i am asking is i used to annoy @bradfitz with issue #9525 and eventually gave up (same happens in docker btw) but what i recall is random nodes dropping out, being reachable by some nodes but not others and completely random stuff and just coming back etc. same like you AND i had the same “does not know about peer removing route” error msgs related to exactly these nodes. these were nodes that had direct connections, no firewall etc. one thing though they had ipv6 enabled on the system but no ipv6 connectivity / ips setup. in case any of this helps