tailscale: Custom derp not working after tailscale upgraded to 1.84
What is the issue?
I’m running derp & tailscale on Ubuntu 24.10. Both of them are with version 1.82.5 and derp is started with -verify-clients. They were working fine.
After upgrading both to 1.84, the derp stopped working with the following error:
nodekey not found in local tailscaled
Tailscale 1.82.5 + Derper 1.82.5 = OK
Tailscale 1.82.5 + Derper 1.84.0 = OK
Tailscale 1.84.0 + Derper 1.82.5 = ERROR
Tailscale 1.84.0 + Derper 1.84.0 = ERROR
I tried the combination and finally found the issue is not in derp, but in tailscale. After downgrade tailscale to 1.82.5 and keep derp at 1.84, the issue went away.
Steps to reproduce
Install version 1.84 of tailscale and derp on Ubuntu 24.10, sign in tailscale and start derp with -verify-clients. The error will show in both https://yourderpdomain/debug/check and logs on server.
Are there any recent changes that introduced the issue?
Upgrading both tailscale and derper from 1.82.5 to 1.84.
OS
Linux
OS version
24.10
Tailscale version
1.84.0
Other software
No response
Bug report
No response
About this issue
- Original URL
- State: closed
- Created a year ago
- Reactions: 8
- Comments: 19 (3 by maintainers)
Commits related to this issue
- ipn/ipnlocal: fix `PeerByID` can not find `SelfNode` Like NodeByKey, add an if stmt for checking the NodeId is SelfNode. This commit Fixes #16052 Signed-off-by: Jerry Yan <792602257@qq.com> — committed to q792602257/tailscale by q792602257 a year ago
- ipn/ipnlocal: fix `PeerByID` can not find `SelfNode` Like NodeByKey, add an if stmt for checking the NodeId is SelfNode. This commit Updates #16052 Signed-off-by: Jerry Yan <792602257@qq.com> — committed to q792602257/tailscale by q792602257 a year ago
- ipn/ipnlocal: fix `PeerByID` can not find `SelfNode` Like NodeByKey, add an if stmt for checking the NodeId is SelfNode. Updates #16052 Signed-off-by: Jerry Yan <792602257@qq.com> — committed to q792602257/tailscale by q792602257 a year ago
- Rename `PeerById` to `NodeByID` and change all related calling. Updates #16052 Signed-off-by: Jerry Yan <792602257@qq.com> — committed to q792602257/tailscale by q792602257 a year ago
- Remove redundancy logic checking `nid` is `selfNode` after `NodeByID`, because `NodeByID` already checked `nid` is `selfNode`. Updates #16052 Signed-off-by: Jerry Yan <792602257@qq.com> — committed to q792602257/tailscale by q792602257 a year ago
- ipn/ipnlocal: update PeerByID to return SelfNode and rename it to NodeByID (#16096) Like NodeByKey, add an if stmt for checking the NodeId is SelfNode. Updates #16052 Signed-off-by: Jerry Yan <7926... — committed to tailscale/tailscale by q792602257 a year ago
- ipn/ipnlocal: update PeerByID to return SelfNode and rename it to NodeByID (#16096) Like NodeByKey, add an if stmt for checking the NodeId is SelfNode. Updates #16052 Signed-off-by: Jerry Yan <7926... — committed to tailscale/tailscale by q792602257 a year ago
Maybe I find it. There is wrong at
WhoIsNodeKeyfunction when you passed the node itself, self is not in “peer”, and will return a false, cause derper check request get a 404 response. Eg. MyNodeIdis5,NodeKeyisnodekey:2a283f0fe6de******49b360373, when trying to callWhoIsNodeKeywithnodekey:2a283f0fe6de******49b360373atserveWhoIsWithBackend(which is custom derper service will call via unix socket/var/run/tailscale/tailscaled.sockand api ishttp://local-tailscaled.sock/localapi/v0/whois?addr=), the function try getNodeIdby thenodekey, but in this line , it will findnb.peers, but self node is inselfand not inpeers.My Setup
Headscale as server, and tailscale(1.84.0) as client, with latest build of derper(with
go install tailscale.com/cmd/derper@latest) tailscale client and derper server is in same machine.How I Find This
/localapi/v0/whois?addr=api and assert it will get a 200 response.addris exist and startswithnodekey:, it will callWhoIsNodeKeyand I guess it might be this function.http://local-tailscaled.sock/localapi/v0/statusvia unix socket, check the nodekey isSelfand it not appear atPeer.statusapi, it will runs at here, it only display client inpeersinfo.This change is currently live in v1.85.140 on the unstable track (
tailscale update --track=unstable). Based on previous releases, a stable version should be later this monthI’ve already tested it to be working as expected. Just have to make sure you compile and install
tailscaledwith the changes.derperjust uses the api fromtailscaledso that doesn’t need to be updated你可以参考这个PR,在
ipn/ipnlocal/node_backend.go这个文件中进行修改。修改后,需要构建
tailscaled和tailscale。根据官方的打包教程或者直接运行
go build tailscale.com/cmd/tailscaled(tailscaled)和go build tailscale.com/cmd/tailscale(tailscale)。得到tailscaled和tailscale两个二进制文件,如果你是在linux上运行,替换官方安装的两个二进制文件即可。如果你需要跨平台编译,请参考golang的跨平台编译教程。
You can change the
ipn/ipnlocal/node_backend.gofile like this pr.Then do an build of tailscaled and tailscale(optional).
Refer to official build guide or just run
go build tailscale.com/cmd/tailscaledfortailscaledandgo build tailscale.com/cmd/tailscalefor tailscale.And you get the custom build of tailscaled and tailscale, If you running tailscale in linux machine, you can directly replace the offical binary. If you need build for other platform, you need follow the golang cross compile guide.
@bradfitz It’s been a month, just mentioned you in case you didn’t see this issue. May we merge or review the related PR #16096? This affects all custom derp relay server with
--verify-clientson with tailscaled v1.84.0I don’t think it will work, you need to wait for the version with this PR merged. https://github.com/tailscale/tailscale/pull/16096