dvc: DVC push hangs when pushing via SSH
DVC version: dvc[ssh]==2.3.0 (I have also tried to downgrade to 2.2.0 and 2.1.0, same result unfortunately) OS: Windows 10 Python version: 3.9.5
I am trying to set up a data registry on a SFTP server. I have dvc added all my files, set up the remote:
dvc remote add -d myremote ssh://user@example.com
and set up the password of this user on this server:
dvc remote modify --local myremote password mypassword
However, when I push, I see it’s checking my local cache (this works), but eventually it gets stuck here:
0% Querying cache in ssh://user@example.com| |0.00/180k [00:00<?, ?file/s]
I let it run for about 10 minutes, but still no progress. I then tried to ssh to the server using PowerShell, which also kept hanging after I entered the password. But when I stopped the terminal that dvc push was running in, the ssh in PowerShell stopped hanging and I was in the server.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 21 (16 by maintainers)
@gcoter this seems like something related to the server itself. Do you know if there are any changed limits in regards to maximum number of sessions? Or can you try to run the following script with filling the options necessary to authenticate and paste the output?
Did some further tests. One of them was to let a colleague set up part of the data registry on their laptop and try to push it to the same server. He failed at the same point.
Something that I also find strange is that DVC opens up 4 connections when executing
dvc push -j 1. Shouldn’t this mean that there should only be one connection?We also found out that when I close the terminal window (the only way I can stop the hanging process), 3 connections are closed on the server, but 1 always stays alive for some reason. Even though I don’t kill the push command in the nicest way, I think it’s strange that this 1 process is not cleaned up properly.
It is really weird since I can’t reproduce this locally. Config parameters always take precedence over the remote url ones (e.g
ssh://username@host.comwill be overriden if the config specifies another username). Can you a new issue @sjawhar?I’m running into the same “Can’t create any SFTP connections” error. Ran the debug script from @isidentical and get the same output as @gcoter (error after 10). Tried
--jobs 1, still error.Update I’ve been using a local config that sets the remote user in config.local. I switched that to overriding the entire SSH url with the user included (e.g. ssh://username@host.com) and now it works. @gcoter are you also using any kind of config override to set the SSH username?
Hi @isidentical, thanks for your answer 🙂 I ran your script and here is the output:
I had to replace “/tmp” with another folder name (because tmp does not exist on this sftp server). Apart from that, I did not change your script.
Regarding the server, I did not make any change myself for a while. Of course, it had some updates, but I don’t remember that anything related to the number of sessions was modified.
When we look at the output, it seems like there is a limit of 10 sessions apparently?
@efiop I tried version 2.7.4, I still have the same error:
@efiop Yes, SSH and SFTP CLI work. I tried with
-j 1, I have the same errorHi! Since we couldnt get the SSH remote working, we switched to another remote type, so I’m not sure if I still have access to that particular server. I’m willing to give it a shot though. How can I install it with the ssh dependency from master branch on Windows?
Something like this?
pip install git+git://github.com/iterative/dvc[ssh]