WSL: Unix domain sockets return spurious data from getsockopt(SO_PEERCRED) if the peer has already departed, which breaks ssh-agent forwarding for Ubuntu 18.04

Since upgrading from Ubuntu 16.04 to 18.04, I’ve noticed that my ssh-agent setup no longer works reliably. For the purposes of debugging, I ran ssh-agent as:

/usr/bin/ssh-agent -s -d -a /tmp/.sshagent.sock

After doing this, I can add keys to the agent, and make exactly one SSH connection. Later, trying to connect to the agent fails with Error connecting to agent: Connection refused. Before the SSH connection (when everything still works), getsockopt appears to behave as expected, returning the pid/uid/gid of the process connecting to the socket:

accept(3</unknown>, {sa_family=AF_UNIX}, [110->2]) = 4</unknown>
getsockopt(4</unknown>, SOL_SOCKET, SO_PEERCRED, {pid=369, uid=1000, gid=1000}, [12]) = 0
getuid()                                = 1000

However later, getsockopt returns -1 as uid/gid which obviously makes ssh-agent unhappy:

accept(3</unknown>, {sa_family=AF_UNIX}, [110->2]) = 4</unknown>
getsockopt(4</unknown>, SOL_SOCKET, SO_PEERCRED, {pid=0, uid=-1, gid=-1}, [12]) = 0
getuid()                                = 1000
getuid()                                = 1000
write(2</dev/pts/1>, "uid mismatch: peer euid 42949672"..., 48) = 48
close(4</unknown>)                      = 0
close(3</unknown>)                      = 0

I’m on RS4 build 17134. /tmp is on an lxfs mountpoint.

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Reactions: 24
  • Comments: 49 (1 by maintainers)

Most upvoted comments

@0xabu from the thread above I was under the impression this was also broken in other distros, but yes, that worked, thanks!

Not that I’m happy about it, but it is a semi-acceptable workaround for WSL. Which should probably have off-cycle updates (hint hint).

Here’s a quick HOWTO for a fix (and for the non-squeamish):

If you’re reading this after August 2018, go here to check you’re getting the latest version from xenial-updates :

cd /tmp/
wget http://mirrors.kernel.org/ubuntu/pool/main/o/openssh/openssh-client_7.2p2-4ubuntu2.4_amd64.deb
dpkg -x openssh-client_7.2p2-4ubuntu2.4_amd64.deb /tmp/deb
sudo mv /usr/bin/ssh-agent /usr/bin/ssh-agent.18.04
# for safekeeping in case of bionic updates
sudo mv /tmp/deb/usr/bin/ssh-agent /usr/bin/ssh-agent.16.04
sudo cp /usr/bin/ssh-agent.16.04 /usr/bin/ssh-agent
sudo chown root:ssh /usr/bin/ssh-agent

(Edits for readability.)

Just updating the script again for those who need it.

cd /tmp/
wget http://mirrors.kernel.org/ubuntu/pool/main/o/openssh/openssh-client_7.2p2-4ubuntu2.8_amd64.deb 
dpkg -x openssh-client_7.2p2-4ubuntu2.8_amd64.deb /tmp/deb
sudo mv /usr/bin/ssh-agent /usr/bin/ssh-agent.18.04
# for safekeeping in case of bionic updates
sudo mv /tmp/deb/usr/bin/ssh-agent /usr/bin/ssh-agent.16.04
sudo cp /usr/bin/ssh-agent.16.04 /usr/bin/ssh-agent
sudo chown root:ssh /usr/bin/ssh-agent

add these code to ~/.bashrc

[ -x /usr/bin/ssh-agent ] && eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_rsa

You create a new process each time you open bash. We can reuse one ssh-agent instance:

if ! pgrep ssh-agent > /dev/null; then
  rm -f /tmp/ssh-auth-sock
  eval "$(ssh-agent -s -a /tmp/ssh-auth-sock)"
else
  export SSH_AUTH_SOCK=/tmp/ssh-auth-sock
fi
ssh-add

add these code to ~/.bashrc

[ -x /usr/bin/ssh-agent ] && eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_rsa

I can confirm that Windows 10 October 2018 Update Version 1809 (OS Build 17763.1) works fine with Ubuntu Bionic’s ssh-agent from the openssh-client apt package version 1:7.6p1-4.

https://blogs.windows.com/windowsexperience/2018/10/02/how-to-get-the-windows-10-october-2018-update/

ssh-agent is still broken in the latest ubuntu 18 from the windows store.

$ echo $SSH_AUTH_SOCK

the environment variable is not being set when you close a Ubuntu terminal and reopen it. I tried the script @adotout posted but the environment variable is still not being set. This is annoying for a lot of reasons, especially because you cannot use ssh-keys to log into servers anymore.

If anyone having problems with BSOD (lxcore.sys), try to run sudo apt install --reinstall openssh-client

Just posting @rcarmo workaround with fixed package url

cd /tmp/
wget http://mirrors.kernel.org/ubuntu/pool/main/o/openssh/openssh-client_7.2p2-4ubuntu2.6_amd64.deb
dpkg -x openssh-client_7.2p2-4ubuntu2.6_amd64.deb /tmp/deb
sudo mv /usr/bin/ssh-agent /usr/bin/ssh-agent.18.04
# for safekeeping in case of bionic updates
sudo mv /tmp/deb/usr/bin/ssh-agent /usr/bin/ssh-agent.16.04
sudo cp /usr/bin/ssh-agent.16.04 /usr/bin/ssh-agent
sudo chown root:ssh /usr/bin/ssh-agent

Package is now openssh-client_7.2p2-4ubuntu2.7_amd64.deb

cd /tmp/
wget http://mirrors.kernel.org/ubuntu/pool/main/o/openssh/openssh-client_7.2p2-4ubuntu2.7_amd64.deb
dpkg -x openssh-client_7.2p2-4ubuntu2.7_amd64.deb /tmp/deb
sudo mv /usr/bin/ssh-agent /usr/bin/ssh-agent.18.04
# for safekeeping in case of bionic updates
sudo mv /tmp/deb/usr/bin/ssh-agent /usr/bin/ssh-agent.16.04
sudo cp /usr/bin/ssh-agent.16.04 /usr/bin/ssh-agent
sudo chown root:ssh /usr/bin/ssh-agent

I know this is an old issue, but some companies are still rolling out Build 1803 so the fix is still relevant

will this get fixed for non insider-builds? or do we have to wait for the fall-update of Windows 10? wsl is kind of useless without working ssh-agent.

this snippet I currently use in my .bashrc.local (as I often copy .bashrc and other dot files to servers, splitting into separate .bashrc.local ensures desktop specific settings doesn’t hit remote servers)

#setting ssh agent
ssh-add -l &>/dev/null
if [ "$?" == 2 ]; then
  test -r ~/.ssh-agent && \
    eval "$(<~/.ssh-agent)" >/dev/null
fi
ssh-add -l &>/dev/null
if [ "$?" == 2 ]; then
    (umask 066; ssh-agent > ~/.ssh-agent)
    eval "$(<~/.ssh-agent)" >/dev/null
fi
ssh-add -l &>/dev/null
if [ "$?" == 1 ]; then
    ssh-add .ssh/mykey.id_rsa.key
fi

.bashrc part to include local file:

if [ -f ~/.bashrc.local ]; then
    . ~/.bashrc.local
fi

.ssh/config to learn new keys once used:

Host *
  ControlMaster auto
  ControlPath /tmp/%r@%h:%p
  ServerAliveInterval 30
  ServerAliveCountMax 10
  AddKeysToAgent yes

all that works fine for me on WSL2 with Ubuntu 20.04

Actually non of the above scripts are working in WSL2 Ubuntu 18.04. The only good solution for me so far was when I simply added eval "$(ssh-agent -s)" to my .bash_profile.

@rcarmo this worked for me, thanks!

There’s by the way some “official” script in the docs:

Linux:

To start the SSH Agent in the background, run:

eval "$(ssh-agent -s)"

To start the SSH Agent automatically on login, add these lines to your ~/.bash_profile:

if [ -z "$SSH_AUTH_SOCK" ]
then
   # Check for a currently running instance of the agent
   RUNNING_AGENT="`ps -ax | grep 'ssh-agent -s' | grep -v grep | wc -l | tr -d '[:space:]'`"
   if [ "$RUNNING_AGENT" = "0" ]
   then
        # Launch a new instance of the agent
        ssh-agent -s &> .ssh/ssh-agent
   fi
   eval `cat .ssh/ssh-agent`
fi

Source: https://code.visualstudio.com/docs/remote/troubleshooting#_setting-up-the-ssh-agent

I can approve that the script is working fine to reuse one ssh-agent. But you are calling ssh-add even if the ssh-agent exists. I would only call it only once:

if ! pgrep ssh-agent > /dev/null; then
  rm -f /tmp/ssh-auth-sock
  eval "$(ssh-agent -s -a /tmp/ssh-auth-sock)"
  ssh-add
else
  export SSH_AUTH_SOCK=/tmp/ssh-auth-sock
fi

There is still the issue that you have to enter the password after a restart. Any idea how to solve that?

@rcarmo have you tried grabbing the ssh-agent binary from 16.04 and using that as a workaround?

* ssh(1): Add a ProxyJump option and corresponding -J command-line
   flag to allow simplified indirection through a one or more SSH
   bastions or "jump hosts".

was introduced in version 7.3 ( http://www.openssh.com/txt/release-7.3 ) which is not the case for ubuntu 16.04:

Version: 1:7.2p2-4ubuntu2.2
coolcold@x230-coolcold:~$ dpkg -s openssh-client|egrep ^Version:
Version: 1:7.2p2-4ubuntu2.2

and is limiting factor for me, for example.

@thegass it’s not useless, just install Ubuntu 16 instead of 18 from the windows store, it works fine

There were changes in Insider Build 17704 that may address this.

@Morgy93 I updated script above and replace socket file lookup with a process.

@HarikrishnanBalagopal :

If you’re still having this issue, double-check that ssh-agent is actually running and listening to the socket specified by $SSH_AUTH_SOCK. In my .bashrc I have:

if [ ! -S ~/.ssh/ssh_auth_sock ]; then
  eval `ssh-agent`
  ln -sf "$SSH_AUTH_SOCK" ~/.ssh/ssh_auth_sock
fi
export SSH_AUTH_SOCK=~/.ssh/ssh_auth_sock
[ -z "`ssh-add -l | grep 'somekey'`" ] && ssh-add ~/.ssh/somekey

I started getting “Error connecting to agent: Connection refused” after an automatic system reboot, so I checked, and sure enough, the socket in /tmp had somehow persisted across a reboot. Obviously the listening ssh-agent instance didn’t, hence the error. I don’t yet know if this is a reproducible issue, but if you have similar logic in your .bashrc, it might be the cause of your problem.

Got same issue and the workaround really save my day. thanks a lot.

It’s not practical to maintain two copies of Ansible, Terraform, az, my custom pyenvs and $DIVINITY knows what else and switch over to another distro “just” for SSH. I just reinstalled everything today after realizing 18.04 had new GCC and “good enough” Go, and having a critical function like agent forwarding conk out on me is a major issue (I pretty much live inside the terminal other than a browser and corp e-mail).

I can confirm that SSH agent forwarding works on 17711.