tailscale: Tailscale SSH failing to forward SSH Agent on 1.68.0
What is the issue?
I think there might be a regression in behaviour between Tailscale v1.66.0 and v1.68.0 using SSH Agent Forwarding via Tailscale SSH.
We use Capistrano to deploy one of our services onto EC2 instances, and bringing up a new test instance with Tailscale 1.68.0 installed this morning fails to run any git commands against GitHub.com with permission denied errors. Capistrano makes a non-tty SSH connection with agent forwarding enabled to allow the remote instance to talk to GitHub using my local key. This is what I see in the Capistrano log after it fails:
INFO [19dd61c1] Running /usr/bin/env git ls-remote git@github.com:relayplatform/sekrit.git HEAD as ubuntu@job-relay-staging
DEBUG [19dd61c1] Command: ( export RBENV_ROOT="/usr/lib/rbenv" RBENV_VERSION="3.2.4" GIT_ASKPASS="/bin/echo" GIT_SSH="/tmp/git-ssh-c768630fd7962370543c.sh" ; /usr/bin/env git ls-remote git@github.com:relayplatform/sekrit.git HEAD )
DEBUG [19dd61c1] git@github.com: Permission denied (publickey).
DEBUG [19dd61c1] fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
DEBUG [19dd61c1] exit status 128
The test nodes are running Ubuntu 22.04.4 LTS with all updates applied as of yesterday, and tailscale installed via the apt repository. We bring tailscale up in the initial boot with tailscale up --authkey ${TAILSCALE_AUTHKEY} --accept-dns=false --accept-routes=false --ssh --advertise-tags=${TAILSCALE_TAGS}.
My local machine is a MBP M1 running Tailscale 1.68.0 installed from the App Store.
If we downgrade the server to Tailscale v1.66.0 via apt and restart tailscaled.service to ensure it’s running from the correct binary, we see:
basic-1:~$ tailscale --version
1.66.0
tailscale commit: 2b2f4268ad16fb3330ef3dea8e4df92443ade52e
other commit: 4c86ccb9d46a4f0782acb35d8302520dc8ad5c58
go version: go1.22.3
basic-1:~$ tailscaled --version
1.66.0
tailscale commit: 2b2f4268ad16fb3330ef3dea8e4df92443ade52e
other commit: 4c86ccb9d46a4f0782acb35d8302520dc8ad5c58
go version: go1.22.3
basic-1:~$ tailscale bugreport
BUG-809d160926d1eddaf89a96f4b7c7ae6cf3c8c6faba74c0b9c5fb6525dfda77ce-20240614141433Z-f4b09c373272d195
Then running ssh -T git@github.com on the remote machine via SSH from my local machine without agent forwarding we observe the expected permission denied response:
$ ssh ubuntu@basic-1 "ssh -T git@github.com"
git@github.com: Permission denied (publickey).
Forwarding the agent from local machine to remote returns the expected result:
$ ssh -A ubuntu@basic-1 "ssh -T git@github.com"
Hi caius! You've successfully authenticated, but GitHub does not provide shell access.
For the second command with agent forward enabled the relevant logs on the server for tailscaled.service are:
Jun 14 14:36:06 basic-1 tailscaled[10575]: ssh-conn-20240614T143605-b5417dccfb: handling conn: 100.84.214.123:51719->ubuntu@100.101.177.108:22
Jun 14 14:36:06 basic-1 tailscaled[10575]: ssh-conn-20240614T143605-b5417dccfb: starting session: sess-20240614T143606-b2b369c9fe
Jun 14 14:36:06 basic-1 tailscaled[10575]: ssh-session(sess-20240614T143606-b2b369c9fe): handling new SSH connection from caius@relayplatform (100.84.214.123) to ssh-user "ubuntu"
Jun 14 14:36:06 basic-1 tailscaled[10575]: ssh-session(sess-20240614T143606-b2b369c9fe): access granted to caius@relayplatform as ssh-user "ubuntu"
Jun 14 14:36:06 basic-1 tailscaled[10575]: ssh-session(sess-20240614T143606-b2b369c9fe): ssh: agent forwarding requested
Jun 14 14:36:06 basic-1 tailscaled[10575]: ssh-session(sess-20240614T143606-b2b369c9fe): starting non-pty command: [/usr/sbin/tailscaled be-child ssh --uid=1000 --gid=1000 --groups=1000,4,20,24,25,27,29,30,44,46,119,120 --local-user=ubuntu --remote-user=caius@relayplatform --remote-ip=100.84.214.123 --has-tty=false --tty-name= --cmd=/bin/bash -- -c ssh -T git@github.com]
Jun 14 14:36:07 basic-1 tailscaled[10575]: ssh-session(sess-20240614T143606-b2b369c9fe): Wait: code=1
Upgrading the server to Tailscale 1.68.0 via apt and explicitly restarting tailscaled.service results in:
basic-1:~$ tailscale --version
1.68.0
tailscale commit: c03256c2bea444722cb84f8aebc065e689c0293c
other commit: c69ba9b2858480e2ddd85d45eab7c3025d3b393e
go version: go1.22.4
basic-1:~$ tailscaled --version
1.68.0
tailscale commit: c03256c2bea444722cb84f8aebc065e689c0293c
other commit: c69ba9b2858480e2ddd85d45eab7c3025d3b393e
go version: go1.22.4
basic-1:~$ tailscale bugreport
BUG-809d160926d1eddaf89a96f4b7c7ae6cf3c8c6faba74c0b9c5fb6525dfda77ce-20240614141646Z-f82063ec4910503a
Running the same commands again, without agent forwarding we get the expected permission denied error:
$ ssh ubuntu@100.101.177.108 "ssh -T git@github.com"
git@github.com: Permission denied (publickey).
exit status 255
Interestingly this logs exit status 255 in the output now and my local shell shows the exit code from ssh as 1, where previously it was exiting with code 255 and no output.
Re-running with the agent forwarded, we observe a change in behaviour however which is what’s causing cap deploy to fail:
$ ssh -A ubuntu@100.101.177.108 "ssh -T git@github.com"
git@github.com: Permission denied (publickey).
exit status 255
The tailscaled logs on the server for the second connection with agent forwarded are:
Jun 14 14:29:16 basic-1 tailscaled[9119]: ssh-session(sess-20240614T142916-6ddfe0f069): handling new SSH connection from caius@relayplatform (100.84.214.123) to ssh-user "ubuntu"
Jun 14 14:29:16 basic-1 tailscaled[9119]: ssh-session(sess-20240614T142916-6ddfe0f069): access granted to caius@relayplatform as ssh-user "ubuntu"
Jun 14 14:29:16 basic-1 tailscaled[9119]: ssh-session(sess-20240614T142916-6ddfe0f069): ssh: agent forwarding requested
Jun 14 14:29:16 basic-1 tailscaled[9119]: ssh-session(sess-20240614T142916-6ddfe0f069): starting non-pty command: [/usr/sbin/tailscaled be-child ssh --login-shell=/bin/bash --uid=1000 --gid=1000 --groups=1000,4,20,24,25,27,29,30,44,46,119,120 --local-user=ubuntu --remote-user=caius@relayplatform --remote-ip=100.84.214.123 --has-tty=false --tty-name= --cmd=ssh -T git@github.com]
Jun 14 14:29:16 basic-1 su[9385]: (to ubuntu) root on none
Jun 14 14:29:16 basic-1 su[9385]: pam_unix(su-l:session): session opened for user ubuntu(uid=1000) by (uid=0)
Jun 14 14:29:16 basic-1 su[9385]: pam_unix(su-l:session): session closed for user ubuntu
Jun 14 14:29:16 basic-1 tailscaled[9119]: ssh-session(sess-20240614T142916-6ddfe0f069): Wait: code=1
Forcing a TTY out of interest doesn’t change the behaviour, the forwarded agent doesn’t appear to be available in the remote shell:
$ ssh -tA ubuntu@100.101.177.108 "ssh -T git@github.com"
git@github.com: Permission denied (publickey).
exit status 255
Connection to 100.101.177.108 closed.
If I SSH in with agent forwarding enabled and drop into an interactive shell it does forward my agent correctly, but I think this is taking a different code path from looking at the logs, no mention of su and it starts bash as the cmd.
Steps to reproduce
local $ ssh user@remote "ssh -T git@github.com"
Expect it to respond with “Hi $GitHub_User”, get “permission denied”.
Are there any recent changes that introduced the issue?
I suspect https://github.com/tailscale/tailscale/pull/11910 might be relevant here, because of the mention of su in the logs from 1.68.0.
OS
Local machine macOS, remote machine Ubuntu.
OS version
- macOS 14.5
- Ubuntu 22.04.4 LTS
Tailscale version
- 1.66.0 working
- 1.68.0 appears broken
Other software
No response
Bug report
- v1.66.0: BUG-809d160926d1eddaf89a96f4b7c7ae6cf3c8c6faba74c0b9c5fb6525dfda77ce-20240614141433Z-f4b09c373272d195
- v.168.0: BUG-809d160926d1eddaf89a96f4b7c7ae6cf3c8c6faba74c0b9c5fb6525dfda77ce-20240614141646Z-f82063ec4910503a
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 2
- Comments: 18 (6 by maintainers)
Commits related to this issue
- ssh/tailssh: replace incubator process with su instead of running su as child This allows the SSH_AUTH_SOCK environment variable to work inside of su and agent forwarding to succeed. Fixes #12467" ... — committed to tailscale/tailscale by oxtoacart 2 years ago
- ssh/tailssh: replace incubator process with su instead of running su as child This allows the SSH_AUTH_SOCK environment variable to work inside of su and agent forwarding to succeed. Fixes #12467 S... — committed to tailscale/tailscale by oxtoacart 2 years ago
- ssh/tailssh: replace incubator process with su instead of running su as child This allows the SSH_AUTH_SOCK environment variable to work inside of su and agent forwarding to succeed. Fixes #12467 S... — committed to stepbrobd/tailscale by oxtoacart 2 years ago
- tailcfg: bump CurrentCapabilityVersion to capture SSH agent forwarding fix Updates #12467 Signed-off-by: Percy Wegmann <percy@tailscale.com> — committed to tailscale/tailscale by oxtoacart 2 years ago
- tailcfg: bump CurrentCapabilityVersion to capture SSH agent forwarding fix Updates #12467 Signed-off-by: Percy Wegmann <percy@tailscale.com> — committed to tailscale/tailscale by oxtoacart 2 years ago
- ssh/tailssh: replace incubator process with su instead of running su as child This allows the SSH_AUTH_SOCK environment variable to work inside of su and agent forwarding to succeed. Fixes #12467 S... — committed to Ckid-Home/tailscale by oxtoacart 2 years ago
- tailcfg: bump CurrentCapabilityVersion to capture SSH agent forwarding fix Updates #12467 Signed-off-by: Percy Wegmann <percy@tailscale.com> — committed to Ckid-Home/tailscale by oxtoacart 2 years ago
- ssh/tailssh: replace incubator process with su instead of running su as child This allows the SSH_AUTH_SOCK environment variable to work inside of su and agent forwarding to succeed. Fixes #12467 S... — committed to Asutorufa/tailscale by oxtoacart 2 years ago
- tailcfg: bump CurrentCapabilityVersion to capture SSH agent forwarding fix Updates #12467 Signed-off-by: Percy Wegmann <percy@tailscale.com> — committed to Asutorufa/tailscale by oxtoacart 2 years ago
We’ve had to reactivate the new behavior. I’ll report back when it’s fixed.
@oxtoacart More information from debugging:
Here
machine2is the name of the machine on the Tailnet. But, when I do the following, the behavior is what I would expect:Here
100.109.34.111is the IP address ofmachine2on the Tailnet.NOTE: In both examples above, I’m not using the
-Aflag. So I would expect that SSH forwarding will be disabled on both.Edit: Ignore. False alarm. With my recent switch to Tailscale I did not check if patterns in my
~/.ssh/configmatched the machines name. Commenting out the rules fixed my issue. Sorry for the confusion! 🤦Still not working for me in 1.68.1. Switching to 1.66.4 still works. Hopefully the new release will solve it.
Fix has been merged to main, but I’m leaving the ticket open until it’s released.
We are again instructing clients to use the V1 behavior, so you should be able to use 1.68.1 now if you’d like.
Absolutely! Thanks for the quick response and fixes.
Thanks @shubang93 . So to confirm, 1.70 works as expected for you and we can close the ticket?
Thanks for getting both a mitigation and a fix in place for this @oxtoacart 👍🏻
Just built a new image with v1.68.1 in and can confirm it’s all working as expected here:
Many thanks 😄
+1, ran into this same issue today with ansible.
Downgrading to 1.66.0 on the remote server also solved it for me, even when the local machine was on 1.68.1
Thanks so much for the very detailed report! We are looking into this.