tailscale: Custom derp not working after tailscale upgraded to 1.84

What is the issue?

I’m running derp & tailscale on Ubuntu 24.10. Both of them are with version 1.82.5 and derp is started with -verify-clients. They were working fine.

After upgrading both to 1.84, the derp stopped working with the following error: nodekey not found in local tailscaled Tailscale 1.82.5 + Derper 1.82.5 = OK Tailscale 1.82.5 + Derper 1.84.0 = OK Tailscale 1.84.0 + Derper 1.82.5 = ERROR Tailscale 1.84.0 + Derper 1.84.0 = ERROR

I tried the combination and finally found the issue is not in derp, but in tailscale. After downgrade tailscale to 1.82.5 and keep derp at 1.84, the issue went away.

Steps to reproduce

Install version 1.84 of tailscale and derp on Ubuntu 24.10, sign in tailscale and start derp with -verify-clients. The error will show in both https://yourderpdomain/debug/check and logs on server.

Are there any recent changes that introduced the issue?

Upgrading both tailscale and derper from 1.82.5 to 1.84.

OS

Linux

OS version

24.10

Tailscale version

1.84.0

Other software

No response

Bug report

No response

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 8
  • Comments: 19 (3 by maintainers)

Commits related to this issue

Most upvoted comments

Maybe I find it. There is wrong at WhoIsNodeKey function when you passed the node itself, self is not in “peer”, and will return a false, cause derper check request get a 404 response. Eg. My NodeId is 5, NodeKey is nodekey:2a283f0fe6de******49b360373, when trying to call WhoIsNodeKey with nodekey:2a283f0fe6de******49b360373 at serveWhoIsWithBackend (which is custom derper service will call via unix socket /var/run/tailscale/tailscaled.sock and api is http://local-tailscaled.sock/localapi/v0/whois?addr=), the function try get NodeId by the nodekey, but in this line , it will find nb.peers, but self node is in self and not in peers.

My Setup

Headscale as server, and tailscale(1.84.0) as client, with latest build of derper(with go install tailscale.com/cmd/derper@latest) tailscale client and derper server is in same machine.

How I Find This

  1. From derper service, I find the log is came from here, it try to call /localapi/v0/whois?addr= api and assert it will get a 200 response.
  2. And I Find this function handle the request, when addr is exist and startswith nodekey:, it will call WhoIsNodeKey and I guess it might be this function.
  3. Add a ugly debug, build and test on a linux machine. Image
  4. And got it, fails at this line. Image
  5. Call http://local-tailscaled.sock/localapi/v0/status via unix socket, check the nodekey is Self and it not appear at Peer.
  6. Find this function handle status api, it will runs at here, it only display client in peers info.
  7. Do a ugly hack, test it, and the bug fixed.

Image

Image

When will this be released?

This change is currently live in v1.85.140 on the unstable track (tailscale update --track=unstable). Based on previous releases, a stable version should be later this month

I’ve already tested it to be working as expected. Just have to make sure you compile and install tailscaled with the changes. derper just uses the api from tailscaled so that doesn’t need to be updated

Great, could you please write a simple tutorial so that we can learn how to do it. thank you.

你可以参考这个PR,在ipn/ipnlocal/node_backend.go这个文件中进行修改。

修改后,需要构建tailscaled和tailscale。

根据官方的打包教程或者直接运行go build tailscale.com/cmd/tailscaled(tailscaled)和go build tailscale.com/cmd/tailscale(tailscale)。

得到tailscaled和tailscale两个二进制文件,如果你是在linux上运行,替换官方安装的两个二进制文件即可。如果你需要跨平台编译,请参考golang的跨平台编译教程。


You can change the ipn/ipnlocal/node_backend.go file like this pr.

Then do an build of tailscaled and tailscale(optional).

Refer to official build guide or just run go build tailscale.com/cmd/tailscaled for tailscaled and go build tailscale.com/cmd/tailscale for tailscale.

And you get the custom build of tailscaled and tailscale, If you running tailscale in linux machine, you can directly replace the offical binary. If you need build for other platform, you need follow the golang cross compile guide.

@bradfitz It’s been a month, just mentioned you in case you didn’t see this issue. May we merge or review the related PR #16096? This affects all custom derp relay server with --verify-clients on with tailscaled v1.84.0

Tailscale just drop an update to 1.84.1. Anyone tested if it’s working ?

I don’t think it will work, you need to wait for the version with this PR merged. https://github.com/tailscale/tailscale/pull/16096