vscode: SSH Agent Forwarding not working with Remote SSH on MacOS (still a problem)

Just chiming in, I have encountered this a few times now (including today, right now lol). Works for months, and suddenly the stale SSH behavior pops up. On 1.82.0 locally. My steps to reproduce:

1. Open remote with “Remote-SSH: Connect to Host…” to a server with fresh code server
2. In new window, try ssh -T git@github.com, successful auth
3. Add seemingly any new folder to the workspace
4. Try authentication command again, fails this time
5. Delete the code server ~/.vscode-server
6. Go to 1.

Edit: to be clear, using Agent forwarding from the host that has worked previously

I was able to get agent forwarding working again by setting remote.SSH.useExecServer: false. It looks like the issue you and I are experiencing was reported here: https://github.com/microsoft/vscode-remote-release/issues/8926

Same thing happing to me. I found that disabling, reloading, and enabling Remote.SSH: Use Local Server works for me but, given this looks like a race condition issue, I may just have been lucky.

I kept doing this for a few days after that post, and the issue still occurred but was a lot less frequent. So I then disabled Remote.SSH: Use Local Server altogether and haven’t had issues since.

Same issue here, closing and recreating the terminal fixes it, but terminals cerated when vscode starts have a stale SSH_AUTH_SOCK

This is now happening from my coworker (@gamegamer43) and myself since the latest update. It seems like this command is not actually running:

[14:36:09.935] Updating $SSH_AUTH_SOCK: ln -f -s "/tmp/ssh-XXXXwXtrPD/agent.4194" "/run/user/0/vscode-ssh-auth-sock-934342336"

When I run the command above manually the key is correctly passed and the symlink exists.

If I then reload the window (Developer: Reload Window). The same thing happens again:

root@development:/usr/src/iris# ls -l $SSH_AUTH_SOCK
lrwxrwxrwx 1 root root 30 Feb  6 22:42 /run/user/0/vscode-ssh-auth-sock-588170625 -> /tmp/ssh-XXXX0bQDaU/agent.5891
root@development:/usr/src/iris# cat  /tmp/ssh-XXXX0bQDaU/agent.5891
cat: /tmp/ssh-XXXX0bQDaU/agent.5891: No such file or directory

Output window shows:

[14:43:44.488] Updating $SSH_AUTH_SOCK: ln -f -s "/tmp/ssh-XXXXFjuLtA/agent.6780" "/run/user/0/vscode-ssh-auth-sock-588170625"

I run that same command and it now works.

This issue seems to have started again today. vscode can’t seem to fix this issue. Keeps coming back every year. see https://github.com/microsoft/vscode/issues/175806 This suggestion worked for me, https://github.com/microsoft/vscode-remote-release/issues/9662#issuecomment-2010515888 no idea why.

Ok, I’m able to reproduce now. These are all run from within the VSCode terminal. Here’s the initial situation:

$ echo $SSH_AUTH_SOCK

/tmp/ssh-7G9TqKKzBp/agent.30503

$ ls -la /tmp/ssh*/**

srwxr-xr-x 1 xxx xxx 0 Feb  9 00:27 /tmp/ssh-9L8JuutTmn/agent.14198
srwxr-xr-x 1 xxx xxx 0 Feb  8 20:34 /tmp/ssh-jFxWY0hBUn/agent.16442

$ ssh -T git@github.com

git@github.com: Permission denied (publickey).

We can see that the SSH_AUTH_SOCK variable is not pointed at an extant ssh agent. Now we apply the workaround:

$ export SSH_AUTH_SOCK=$(ls -t /tmp/ssh-**/* | head -1)

$ ssh -T git@github.com

Hi mbmccoy! You've successfully authenticated, but GitHub does not provide shell access.

$ echo $SSH_AUTH_SOCK

/tmp/ssh-9L8JuutTmn/agent.14198

Was experiencing this issue today. Can confirm that "remote.SSH.useExecServer": false, fixes it.

Just chiming in, I have encountered this a few times now (including today, right now lol). Works for months, and suddenly the stale SSH behavior pops up. On 1.82.0 locally. My steps to reproduce:

1. Open remote with “Remote-SSH: Connect to Host…” to a server with fresh code server
2. In new window, try ssh -T git@github.com, successful auth
3. Add seemingly any new folder to the workspace
4. Try authentication command again, fails this time
5. Delete the code server ~/.vscode-server
6. Go to 1.

Edit: to be clear, using Agent forwarding from the host that has worked previously

I was able to get agent forwarding working again by setting remote.SSH.useExecServer: false. It looks like the issue you and I are experiencing was reported here: microsoft/vscode-remote-release#8926

You are awesome!

+1 on this In my case, agent is forwarding with ssh in cmd (this means config file is set correctly). But in VS Code, agent not forwarding, until I changed this setting remote.SSH.useExecServer to false.

Update: unfortunately, it seems this works randomly. Now, ssh-add -l shows the key, but git pull fails with error “Permission denied (publickey).”

I hit this again, apparently triggered by logging from a terminal outside of VSCode. Previous workarounds failed, including restarting the terminal.

Current workaround

1. SSH into machine outside of VSCode, but forwarding remote agent: ssh -A me@my-machine.

2. In this terminal, echo the $SSH_AUTH_SOCK variable:

   echo $SSH_AUTH_SOCK
   

   which will produce something like

   /tmp/ssh-dHPZ4apwPl/agent.1100176
   

3. In the VSCode integrated terminal, set the environment variable:

    export SSH_AUTH_SOCK=<from-above>
   

This seems to work for me as a workaround:

ln -f -s $(ls -t /tmp/ssh-*/agent.* | head -n1) $SSH_AUTH_SOCK

(link the latest ssh agent socket to $SSH_AUTH_SOCK)

(I have this problem with a linux -> linux ssh remote.)

Following this thread. I am also experiencing the same issue, trying to perform git operations in a remote connection vscode shell, with the ForwardAgent option set to True on my ssh_config file (MacOS 13.4.1). In a separate shell, not on vscode, I can perfectly execute this command ssh -T git@github.com with correct authn but not on the plugin vscode terminal, where I receive a Permission denied (publickey) error. From the log files, it seems that the vscode terminal is looking for the wrong pubkey (the login ssh key, not the github forwarded key).

I can confirm that this still an issue, and that it is not specific to Mac, as we have both Mac and Linux users affected by it. On the working servers it shows the "Update SSH_AUTH_SOCK" terminal command done as usual, and on the non-working servers it does not. We are loading from workspace files on the remote hosts.

I’ve commented on https://github.com/microsoft/vscode-remote-release/issues/7995 too but this looks like a race condition in the loading of the remote-ssh extension where the ssh socket creation code requires a terminal backend to be registered, but it is run before registerTerminalBackend is called

Hmm… I don’t think that’s the source of the problem. In particular, when I use ssh from the command line (using the same user/host that I use in VSC), I don’t have this issue. I don’t have any special settings for IdentityAgent in either my local or remote machine.

I’m often able to trigger the issue this way:

1. Open remote connection to server.
   • I can see that my identity is not appropriately forwarded via ssh -T git@github.com in VSCode terminal.
2. Run workaround in VSCode terminal: export SSH_AUTH_SOCK=$(ls -t /tmp/ssh-**/* | head -1).
   • Now ssh -T git@github.com succeeds.
3. Reload window (Developer: Reload Window).
   • Now ssh -T git@github.com fails.

Even weirder, after repeating steps 2 and 3 a few times, it’s started using the correct socket after reload.

Any news on this? This problem is still happening and I am not able to get a persistent agent with any of the newer stable or insiders versions of VSCode.

Just chiming in, I have encountered this a few times now (including today, right now lol). Works for months, and suddenly the stale SSH behavior pops up. On 1.82.0 locally. My steps to reproduce:

1. Open remote with “Remote-SSH: Connect to Host…” to a server with fresh code server
2. In new window, try ssh -T git@github.com, successful auth
3. Add seemingly any new folder to the workspace
4. Try authentication command again, fails this time
5. Delete the code server ~/.vscode-server
6. Go to 1.

Edit: to be clear, using Agent forwarding from the host that has worked previously

Any progress on this? Another VS code update and git is still broken

restarting the host system fixes this for me, so it is most likely a vscode server side issue

Workaround that finally worked for me:

SSH_AUTH_SOCK=$(ls -td /tmp/ssh**/* | head -n1)
ssh-add -l

I need to use the local server, so this turnaround was not a solution for me

I made a shim script that I source in my .zshrc to reduce the pain around that issue. It is just a variation of other solutions posted above, I only added a check that the last vscode ssh sock is a valid symlink, otherwise I need to open a new vscode window thru the Remote SSH GUI (Connect to Host…).

#!/usr/bin/env bash

fix_vscode_ssh() {
  if [ -e "$SSH_AUTH_SOCK" ] && [ -h "$SSH_AUTH_SOCK" ]; then # Valid symlink
    echo Valid symlink for SSH_AUTH_SOCK $SSH_AUTH_SOCK
    return 0;
  fi

  echo Invalid symlink for SSH_AUTH_SOCK: $SSH_AUTH_SOCK

  local redefine_alias=false
  if alias ls >/dev/null 2>&1; then # Existing alias
    unalias ls # exa is printing where the symlink redirects while I just want the path of the symlink
    redefine_alias=true 
  fi

  local new_ssh=$(ls -t  /run/user/${UID}/vscode-ssh-auth-*  | head -1)

  if [ redefine_alias ]; then
    alias ls="exa --color=auto --icons" # TBC depending on your own alias settings
  fi

  if ! ([ -e "$new_ssh" ] && [ -h "$new_ssh" ]); then # Invalid symlink
    printf "Latest vscode symlink [%s] for ssh is also dead. Open a new vsode workspace manually.\n" $new_ssh;
    return 1
  fi

  export SSH_AUTH_SOCK=$new_ssh

  echo SSH_AUTH_SOCK set to $SSH_AUTH_SOCK.

  return 0
}

if [ "$TERM_PROGRAM" = "vscode" ]; then
  fix_vscode_ssh
fi

and then in my .zshrc

source <path_of_the_above_script.sh>

I kept doing this for a few days after that post, and the issue still occurred but was a lot less frequent. So I then disabled Remote.SSH: Use Local Server altogether and haven’t had issues since.

This also worked for me! 🤷

Fresh installation of MacOS Ventura, added keys to ssh-agent, installed VSCode, reusing my synced ssh config file (identical same one from my Debian Linux laptop) and ssh forwarding not working on the Mac. Works on my Linux laptop although. Nothing special on that machine, as reinstalled about 3 weeks ago, with all working, no issues.

Killing vscode server on the remote machine and reconnecting fixed the issue. I think that vscode server running on the remote machine is expecting the old agent connection.

Is there a way to make the running process exit when vscode disconnects?

Great tip @madalinignisca , thanks! I had to remove the binaries rm -rf ~/.vscode-server/bin for this to work

Deleting ~/.vscode-server fixed it for me

At the remote, presumably? @micahcarroll

Just in case anyone else has this issue, and "remote.SSH.useExecServer": false (or other things that were mentioned above) does not fix it for them:

Deleting ~/.vscode-server fixed it for me

UPDATE: it only fixed it temporarily 😢 Currently tried the "remote.SSH.useExecServer": false approach, and will report back if it doesn’t work for me long-term

its annoying having to run export SSH_AUTH_SOCK=$(ls -t /tmp/ssh-**/* | head -1) all the time

Just encountered the same problem, it seems you can add export SSH_AUTH_SOCK=$(ls -t /tmp/ssh-**/* | head -1) to your .bashrc/.zshrc/whatever to avoid being annoyed 😄

(in addition to the details above)

With workaround it worked on Windows with Ubuntu Server SSH host.

Extra touches:

• remote.SSH.useLocalServer left TRUE, as it hadn’t influenced the issue
• remote.SSH.remoteServerListenOnSocket set to TRUE

Steps (in addition to details above):

1. Updated OpenSSH on the Windows host (from v8.6 to v9.4): winget install Microsoft.OpenSSH.Beta
2. Exported $SSH_AUTH_SOCK in the integrated terminal within remote SSH session: export SSH_AUTH_SOCK=$(ls -td /tmp/ssh**/* | head -n1)

Just for the reference: $SSH_AUTH_SOCK was /run/user/1000/vscode-ssh-auth-sock-* upon connection to remote host. $SSH_AUTH_SOCK became /tmp/ssh-XXXXBHX4WS/agent.* after step 2 applied.

I can confirm this is not a Mac-specific problem:

• VSCode Remote SSH from Windows to Ubuntu Server host (fresh installations).

Regular terminal (Windows Terminal PS session):

• ssh -vT git@github.com is authenticated successfully.

Integrated terminal (VSCode Remote SSH session):

• ssh-add -l shows forwarded key fine.
• ssh -vT git@github.com ends with dreaded “git@github.com: Permission denied (publickey)”.

I tried lots of permutations at this point.

Steps:

1. $SSH_AUTH_SOCK didn’t match to the current agent at the remote session
2. made it straight with the suggested workaround: export SSH_AUTH_SOCK=$(ls -t /tmp/ssh-**/* | head -1)
3. however, the same Permission denied

Context:

• terminal.integrated.inheritEnv and remote.SSH.enableAgentForwarding are both enabled in VSCode.
• .ssh config is trivial

  Host 192.168.1.100
      HostName 192.168.1.100
      ForwardAgent yes
  

Version: 1.83.1 (user setup)
Commit: f1b07bd25dfad64b0167beb15359ae573aecd2cc
Date: 2023-10-10T23:48:05.904Z
Electron: 25.8.4
ElectronBuildId: 24154031
Chromium: 114.0.5735.289
Node.js: 18.15.0
V8: 11.4.183.29-electron.0
OS: Windows_NT x64 10.0.22621

Also facing the same issue, workaround is also working. Just wondering why VSCode server creates a symlink in /run/user/$uid/... to /tmp/ssh*

Sounds the link is not created. From Output console: Updating $SSH_AUTH_SOCK: ln -f -s "/tmp/ssh-zP7cLxc6yg/agent.282444" "/run/user/1001/vscode-ssh-auth-sock-956100014"

Then :

$ ls "/run/user/1001/vscode-ssh-auth-sock-956100014"
: ... '/run/user/1001/vscode-ssh-auth-sock-956100014': No such file or directory

If I create the link manually, then the magic occurs and all works well.

Why not define $SSH_AUTH_SOCK to /tmp/…, or fix the link creation issue ?

in my current setup, if I have ssh local server enabled, I connect, and it doesn’t work, I can run: “Restart extension host”, and then it starts working. I might need to re-open a terminal if it was already open.

Disabling “use local server” did not work for Windows client (1.76.2) and Rhel8 host. I ended up crafting the below code and injecting it into my .bashrc

if [[ "$SSH_AUTH_SOCK" =~ "vscode" ]]; then
  # echo "This is a vscode-terminal. SSH_AUTH_SOCK=$SSH_AUTH_SOCK"
  ssh-add -l 1>/dev/null 2>/dev/null
  if [[ "$?" != "0" ]]; then
    export SSH_AUTH_SOCK=$(ls -lt /tmp/ssh-**/* | awk '{print $9}' | head -n1 | grep -Po '.*\d+');
    # echo "Replaced SSH_AUTH_SOCK to $SSH_AUTH_SOCK"
  fi
fi

This worked for me. I had to stop disabling useLocalServer because of what @madalinignisca stated.

Can we get an update on this? I’ve never had an issue before with remote-ssh and for the last month it has been very flakey. Sometimes killing the remote server or local server works. Sometimes I need to restart my computer. I have ssh forwarding configured properly and it works from a regular terminal, just not vscode. I noticed this started happening for me when I upgrade to 1.76. I downgraded to 1.75 and that fixed it but this morning it started happening again (still on 1.75).

edit: Looks like killing every vscode window and relaunching the app has solved my issue for the time being.

Disabling “use local server” did not work for Windows client (1.76.2) and Rhel8 host. I ended up crafting the below code and injecting it into my .bashrc

if [[ "$SSH_AUTH_SOCK" =~ "vscode" ]]; then
  # echo "This is a vscode-terminal. SSH_AUTH_SOCK=$SSH_AUTH_SOCK"
  ssh-add -l 1>/dev/null 2>/dev/null
  if [[ "$?" != "0" ]]; then
    export SSH_AUTH_SOCK=$(ls -lt /tmp/ssh-**/* | awk '{print $9}' | head -n1 | grep -Po '.*\d+');
    # echo "Replaced SSH_AUTH_SOCK to $SSH_AUTH_SOCK"
  fi
fi

Fresh installation of MacOS Ventura, added keys to ssh-agent, installed VSCode, reusing my synced ssh config file (identical same one from my Debian Linux laptop) and ssh forwarding not working on the Mac. Works on my Linux laptop although. Nothing special on that machine, as reinstalled about 3 weeks ago, with all working, no issues.

Killing vscode server on the remote machine and reconnecting fixed the issue. I think that vscode server running on the remote machine is expecting the old agent connection.

Is there a way to make the running process exit when vscode disconnects?

This is also happening in windows 10.

As refered as previous comment, below command was logged but not executed.

[10:27:28.317] Updating $SSH_AUTH_SOCK: ln -f -s "/tmp/ssh-TQUU6UJGJL/agent.11754" "/run/user/1000/vscode-ssh-auth-sock-952306072"

The symbolic link was not created on the destination.

remote$ ls -a /run/user/1000/vscode-ssh-auth-sock-952306072
ls: cannot access '/run/user/1000/vscode-ssh-auth-sock-952306072': No such file or directory

And ssh-agent in local is not forwarded to remote.

remote$ ssh-add -l
Error connecting to agent: No such file or directory

After run the below command, symbolic link was created and ssh-agent was forwarded successfully.

remote$ ln -f -s "/tmp/ssh-TQUU6UJGJL/agent.11754" "/run/user/1000/vscode-ssh-auth-sock-952306072"
remote$ ls -a /run/user/1000/vscode-ssh-auth-sock-952306072
/run/user/1000/vscode-ssh-auth-sock-952306072
remote$ ssh-add -l
2048 SHA256:L7TcPUCaZ************* \****\***\.ssh\*** (RSA)

Below is My VSCode version

Version: 1.75.0 (user setup)
Commit: e2816fe719a4026ffa1ee0189dc89bdfdbafb164
Date: 2023-02-01T15:23:45.584Z
Electron: 19.1.9
Chromium: 102.0.5005.194
Node.js: 16.14.2
V8: 10.2.154.23-electron.0
OS: Windows_NT x64 10.0.19045
Sandboxed: No