tailscale: mosh fails to start correctly after upgrade to Tailscale 1.26.1 + ssh

What is the issue?

After turning on “tailscale ssh” on a server, mosh fails to start with the error message

/opt/homebrew/bin/mosh: Did not find mosh server startup message. (Have you installed mosh on your server?)

Steps to reproduce

Upgrade Tailscale on a server to v1.26.1, turn on “tailscale ssh”, try to connect to server with “mosh user@tailscale-node”

Are there any recent changes that introduced the issue?

It looks like #4752 addresses this very issue, which is one thing that confuses me - I think that 1.26.1 has this patch in it.

OS

Linux, macOS

OS version

Debian 10 (buster)

Tailscale version

1.26.1

Bug report

BUG-739f04a08adef6a9fb63d380f766a3fd2ab3e4d7a2c586f9b21e720db5b4fe72-20220624030033Z-34749df6e9fcfebd

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 21 (10 by maintainers)

Commits related to this issue

ssh/tailssh: fix mosh over Tailscale SSH WIP for #4919 Change-Id: I8adbd957d8fe67f35ce72379da97951d37e02efb Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com> — committed to tailscale/tailscale by bradfitz 2 years ago
ssh/tailssh: fix mosh over Tailscale SSH Updates #4919 Co-authored-by: James Tucker <raggi@tailscale.com> Co-authored-by: Brad Fitzpatrick <bradfitz@tailscale.com> Signed-off-by: Maisem Ali <maisem@... — committed to tailscale/tailscale by maisem 2 years ago
ssh/tailssh: fix mosh over Tailscale SSH Updates #4919 Co-authored-by: James Tucker <raggi@tailscale.com> Co-authored-by: Brad Fitzpatrick <bradfitz@tailscale.com> Signed-off-by: Maisem Ali <maisem@... — committed to tailscale/tailscale by maisem 2 years ago
ssh/tailssh: fix mosh over Tailscale SSH Updates #4919 Co-authored-by: James Tucker <raggi@tailscale.com> Co-authored-by: Brad Fitzpatrick <bradfitz@tailscale.com> Signed-off-by: Maisem Ali <maisem@... — committed to tailscale/tailscale by maisem 2 years ago
ssh/tailssh: fix mosh over Tailscale SSH Updates #4919 Co-authored-by: James Tucker <raggi@tailscale.com> Co-authored-by: Brad Fitzpatrick <bradfitz@tailscale.com> Signed-off-by: Maisem Ali <maisem@... — committed to tailscale/tailscale by maisem 2 years ago
ssh/tailssh: better handling of signals and exits We were not handling errors occurred while copying data between the subprocess and the connection. This makes it so that we pass the appropriate sign... — committed to tailscale/tailscale by maisem 2 years ago
ssh/tailssh: better handling of signals and exits We were not handling errors occurred while copying data between the subprocess and the connection. This makes it so that we pass the appropriate sign... — committed to tailscale/tailscale by maisem 2 years ago
ssh/tailssh: better handling of signals and exits We were not handling errors occurred while copying data between the subprocess and the connection. This makes it so that we pass the appropriate sign... — committed to tailscale/tailscale by maisem 2 years ago
ssh/tailssh: better handling of signals and exits We were not handling errors occurred while copying data between the subprocess and the connection. This makes it so that we pass the appropriate sign... — committed to tailscale/tailscale by maisem 2 years ago
ssh/tailssh: better handling of signals and exits We were not handling errors occurred while copying data between the subprocess and the connection. This makes it so that we pass the appropriate sign... — committed to tailscale/tailscale by maisem 2 years ago

Most upvoted comments

This should’ve been fixed by af412e8874e94dc3ac57c37c3ec5e0606aa08fbb in 1.28.0.

Please file a new bug report against 1.28.0 if it’s not fixed and we’ll take a look.

bradfitz on Jul 23, 2022

I can repro the failure.

From macOS (homebrew mosh 1.3.2) to Linux Tailscale SSH mosh 1.3.2, using tailscaled at main as of Friday July 8th (d6b8a18b09d2c24111777d51f79fd63680023438).

Works with mosh --no-ssh-pty, but in PTY mode it’s sporadic. It usually fails but works every few tries. Sometimes every 3 or 4 tries. Sometimes after 10 tries.

tailscaled logs says:

ssh-session(sess-20220711T032257-1b468504ce): starting pty command: [/home/bradfitz/go/bin/tailscaled be-child ssh --uid=1001 --gid=1001 --groups=1001,27,105,115 --local-user=bradfitz --remote-user=bradfitz@tailscale.com --remote-ip=100.120.208.62 --has-tty=true --tty-name=pts/25 --cmd=/bin/bash -- -c mosh-server 'new' '-c' '256' '-s' '-l' 'LANG=en_US.UTF-8']
ssh-session(sess-20220711T032257-1b468504ce): stdout copy: read /dev/ptmx: file already closed
ssh-session(sess-20220711T032257-1b468504ce): terminating SSH session from 100.120.208.62: read /dev/ptmx: file already closed
ssh-session(sess-20220711T032257-1b468504ce): Wait: code=-1

So I suspect it’s a stdout/stderr close order/dup/race issue.

/cc @maisem

bradfitz on Jul 11, 2022

I just tested on two client devices (Blink Shell’s mosh on iOS, and mosh 1.3.2 from homebrew on macOS) connecting to an ubuntu linux machine running tailscale 1.25.98 and then 1.27.57, both times with tailscale --ssh enabled. mosh on the server is also 1.3.2.

I didn’t have to do anything special and it worked for me. just mosh <hostname> from both Blink and macOS.

apenwarr on Jun 30, 2022

I was able to get a successful connection with mosh --no-ssh-pty user@tailscale-host, where mosh user@tailscale-host is failing.

Once I saw this error message

Bad MOSH CONNECT string: MOSH CONNECT 60001 3Fyuy71RtW45pFpfxmVBHgConnection to tailscale-host closed.

but it’s not consistent.

My version of mosh is mosh 1.3.2 [build mosh 1.3.2] on Apple MacBook Air M1, installed with homebrew. It doesn’t have a mosh --verbose option.

vielmetti on Jun 24, 2022