dvc: Secsh channel 10 open FAILED: open failed: Connect failed

Please provide information about your setup DVC version(i.e. dvc --version), Platform and method of installation (pip, homebrew, pkg Mac, exe (Windows), DEB(Linux), RPM(Linux))

version 0.88.0 installed via pip on Linux and Mac. Syncing to a Ubuntu host via ssh.

Everything seems to be working but I get a cryptic warning message every time I do anything. For example:

dvc push 0% Querying cache in ssh://bror@zotac.bror.us/media/sda2/dvc/first_day| |0/42 Secsh channel 10 open FAILED: open failed: Connect failed Secsh channel 10 open FAILED: open failed: Connect failed Secsh channel 10 open FAILED: open failed: Connect failed 2% /media/sda2/dvc/first_day/4e/4b31f0c5784a2e185d88a3120cac19| |1/42 [00:02<0Secsh channel 10 open FAILED: open failed: Connect failed Everything is up to date.

This is probably an edge case due to my setup but I’m not sure how to quiet the message or resolve the issue.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 20 (11 by maintainers)

Most upvoted comments

I’ll look into this, although out of the recent patches, I don’t think any of the relevant changes should have affected ssh remotes, since RemoteSSH still has its own overridden cache_exists implementation.

pmrowla on Apr 24, 2020

The message in itself is benign I think. You poll paramiko for max number of channels by open them one by one until max number is reached and paramiko raises an exception. I still don’t understand why I’m the only one that get’s the warning message though. I’ve tried two different setups (my office linux machine and my Mac laptop) and both generates it. Could it be because I’ve installed dvc in a conda environment? I tried to create a minimal environment with only pip paramiko and dvc installed and got the same warnings…

brorfred on Mar 17, 2020

We load paramiko lazily, so, I guess, the logger is not active at the point when we try to disable it. Probably, we need to bring back the logger dict configs that were before for paramiko.

skshetry on Mar 17, 2020

I think the issue is just that we are not suppressing the paramiko logger. We hit the channel limit when determining the max number of channels and trigger the message, even though it’s expected and not an error for DVC.

pmrowla on Apr 24, 2020

@efiop

Retried for a few more times and the rest was uploaded. I also found when running dvc push without any actual changes (everything up-to-date), I still get these errors.

hg-zt on Apr 24, 2020

@gurunath-p Thanks for the report! A workaround is to run dvc push a few more times, it will upload what is missing.

But since we are getting more reports it makes me think whether some of our recent patches have made dvc go over the reasonable limit. CC @pmrowla

efiop on Apr 23, 2020

Our MaxSession was set to a large number, and I found hundreds of sessions are opened for dvc push -j 1 and failed.

After reverting to v0.41.3, which is before #2131 was merged, we could now use dvc push

hg-zt on Apr 21, 2020

@brorfred, My google-fu returned this answer from Unix StackExchange: https://unix.stackexchange.com/a/459051

It basically says that this happens if you get over resource limit, eg: no. of open file descriptors. Please give that link a try, and also increase the ulimit.

And, please respond if it solved the issue as it will help other users as well.

skshetry on Mar 16, 2020