compose: Secsh channel 10 open FAILED: open failed: Connect failed

Description of the issue

I don’t know exactly if this is a docker-compose bug or docker bug.

Connection faild on docker-compose -H 'ssh://xxx@192.168.1.1' up Without the host parameter, the docker-compose will start without problems.

Context information (for bug reports)

Output of docker-compose version

docker-compose version 1.24.0-rc1, build 0f3d4dda
docker-py version: 3.7.0
CPython version: 3.6.6
OpenSSL version: OpenSSL 1.1.0h  27 Mar 2018

Output of docker version Local:

Client: Docker Engine - Community
 Version:           18.09.1
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        4c52b90
 Built:             Wed Jan  9 19:33:12 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       4c52b90
  Built:            Wed Jan  9 19:41:49 2019
  OS/Arch:          linux/amd64
  Experimental:     true

Remote:

Client:
 Version:           18.09.1
 API version:       1.38 (downgraded from 1.39)
 Go version:        go1.10.6
 Git commit:        4c52b90
 Built:             Wed Jan  9 19:35:31 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       e68fc7a
  Built:            Mon Oct  1 14:25:33 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker-compose config

services:
  app:
    build:
      context: /Users/xx/xx/project
      dockerfile: app.dockerfile
    environment:
      DB_HOST: database
      DB_PORT: '3306'
    volumes:
    - /Users/xx/xx/project:/var/www:rw
    working_dir: /var/www
  database:
    environment:
      MYSQL_DATABASE: xxx
      MYSQL_PASSWORD: xxx
      MYSQL_ROOT_PASSWORD: xxx
      MYSQL_USER: xxx
    image: mariadb:10.1.26
    ports:
    - 33061:3306/tcp
    volumes:
    - dbdata:/var/lib/mysql:rw
  web:
    build:
      context: /Users/xx/xx/project
      dockerfile: web.dockerfile
    ports:
    - 8080:80/tcp
    volumes:
    - /Users/xx/xx/project:/var/www:rw
    working_dir: /var/www
version: '3.0'
volumes:
  dbdata: {}

Steps to reproduce the issue

  1. docker-compose -H 'ssh://xxx@192.168.1.1' up

Expected result

Creating huelle_app_1      ... done
Creating huelle_database_1 ... done
Creating huelle_web_1      ... done

Stacktrace / full error message

 docker-compose -H 'ssh://xxx@192.168.1.1' up
Connected (version 2.0, client OpenSSH_7.6p1)
Authentication (publickey) successful!
Creating huelle_app_1 ...
Creating huelle_database_1 ...
Creating huelle_web_1      ...
ERROR: Secsh channel 10 open FAILED: open failed: Connect failed

ERROR: for huelle_app_1  (2, 'Connect failed')
ERROR: Secsh channel 11 open FAILED: open failed: Connect failed

ERROR: for huelle_web_1  (2, 'Connect failed')
ERROR: Secsh channel 12 open FAILED: open failed: Connect failed

ERROR: for huelle_database_1  (2, 'Connect failed')

ERROR: for app  (2, 'Connect failed')

ERROR: for web  (2, 'Connect failed')

ERROR: for database  (2, 'Connect failed')
Traceback (most recent call last):
  File "docker-compose", line 6, in <module>
  File "compose/cli/main.py", line 71, in main
  File "compose/cli/main.py", line 127, in perform_command
  File "compose/cli/main.py", line 1085, in up
  File "compose/cli/main.py", line 1081, in up
  File "compose/project.py", line 555, in up
  File "compose/parallel.py", line 112, in parallel_execute
  File "compose/parallel.py", line 210, in producer
  File "compose/project.py", line 541, in do
  File "compose/service.py", line 533, in execute_convergence_plan
  File "compose/service.py", line 455, in _execute_convergence_create
  File "compose/parallel.py", line 112, in parallel_execute
  File "compose/parallel.py", line 210, in producer
  File "compose/service.py", line 453, in <lambda>
  File "compose/service.py", line 441, in create_and_start
  File "compose/service.py", line 337, in create_container
  File "compose/container.py", line 52, in create
  File "site-packages/docker/api/container.py", line 427, in create_container
  File "site-packages/docker/api/container.py", line 437, in create_container_from_config
  File "site-packages/docker/api/client.py", line 284, in _post_json
  File "site-packages/docker/utils/decorators.py", line 46, in inner
  File "site-packages/docker/api/client.py", line 221, in _post
  File "site-packages/requests/sessions.py", line 572, in post
  File "site-packages/requests/sessions.py", line 524, in request
  File "site-packages/requests/sessions.py", line 637, in send
  File "site-packages/requests/adapters.py", line 449, in send
  File "site-packages/urllib3/connectionpool.py", line 600, in urlopen
  File "site-packages/urllib3/connectionpool.py", line 354, in _make_request
  File "http/client.py", line 1239, in request
  File "http/client.py", line 1285, in _send_request
  File "http/client.py", line 1234, in endheaders
  File "http/client.py", line 1026, in _send_output
  File "http/client.py", line 964, in send
  File "site-packages/docker/transport/sshconn.py", line 29, in connect
  File "site-packages/paramiko/transport.py", line 806, in open_session
  File "site-packages/paramiko/transport.py", line 944, in open_channel
paramiko.ssh_exception.ChannelException: (2, 'Connect failed')
[4866] Failed to execute script docker-compose

Additional information

OS version local: macOS Mojave, 10.14.2 (18C54) OS version remote: Ubuntu 18.04.1 LTS

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 22
  • Comments: 28 (5 by maintainers)

Most upvoted comments

Just ran into this issue with a fairly standard compose configuration. I think the issue should be reopened. Setting MaxSessions in /etc/ssh/sshd_config on my remote host to

MaxSessions 500

followed by a

sudo service ssh restart

did fix the issue for me. Running on ubuntu bionic

+59
nick-brady on May 4, 2020

Hey @Vmadmax,

I was able to reproduce this error.

The issue is that compose opens too much sessions per connection, ssh limits them to 10 by default. A temporary workaround would be to change this value in your remote sshd configuration (the MaxSessions parameter) and set it to ~30. We are working on a fix for this.

Please note that you need Docker 18.09 if you want to connect to your daemon over SSH, your remote server has 18.06.1-ce.

+34
rumpl on Jan 29, 2019

Why is this closed? Original issue is not solved. Still encountering this with Docker 19.03.8.

+29
trevorlloydelliott on May 12, 2020

It seems like this issue should have been fixed by https://github.com/docker/docker-py/pull/2275 which was merged early 2019, but weirdly the commits in the MR does not show up as part of any branches… Maybe my github-fu is not up to snuff, but if it is the case that the changes never hit the production branch, this issue and the issues that the MR was initially submitted to fix should probably be re-opened.

+14
andoks on Sep 18, 2020

@ulyssessouza Are you sure the original issue is solved? Running latest versions of docker-compose (1.24.0) and docker (18.09.3) I’m still facing a paramiko.ssh_exception.ChannelException: (1, 'Administratively prohibited') issue. Increasing maxSession option of sshd fixes this issue. This was suggested as a work-a-round by @chris-crone in second post of this thread.

I could confirm that the AttributeError: 'NoneType' object has no attribute 'open_session' reported by @Vmadmax in forth post of this thread seems to be fixed.

+7
jelhan on Mar 29, 2019

Just to call out: This issue will probably start getting more traffic now --compose has been added in 1.26.0 (in #7202), because that’s how I ended up here.

@aiordache it might be worth updating your blog post to mention this issue (and the MaxSession workaround). Thanks for that post by the way, it was very helpful! 🙏

+6
davetapley on Jun 9, 2020

Today I faced the same bug and I can confirm that Adding MaxSessions 500 fixed the issue.

Still, because default is only 10 there is a huge chance that this will be encountered by most users trying to use the removing interface.

We need to verify this and give a meaning error to the user as we are talking about a default system configuration that will trigger this in 9/10 cases and not some corner-case scalability issue.

+5
ssbarnea on Dec 6, 2019

A simple client-side “fix” can be accomplished by using an ssh ControlMaster. There are, or course some trade-offs, but for most use-cases, this should resolve the initially reported issue.

Just add a few lines to your client-side ~/.ssh/config file:

Host *
  ControlMaster auto
  ControlPath ~/.ssh/master-%r@%h:%p
+4
rmrfslashbin on Dec 24, 2020

I can also reproduce the AttributeError: 'NoneType' object has no attribute 'open_session' error. Some observations below plus docker-compose --verbose logs attached.

  • It happens only when/after an image is built using the “up” command. The image is still built successfully.
  • The first docker-compose -H 'ssh://...' up always fails because an image needs to be built. Repeating the command works with no error as the image now already exists on the remote host.
  • docker-compose -H 'ssh://...' up --build always produces the error (but the image is still built).

Attached log file of this call, docker-compose --verbose -H "ssh://ubuntu@remotehost" up --build &> docker-compose-up-build.log using docker-compose version 1.24.0-rc1 (build 0f3d4dda) locally and Docker version 18.09.1 on the remote host.

Thanks! docker-compose-up-build.log

Edit: Remote host sshd MaxSessions was set to 500. As a workaround we are first calling docker-compose -H 'ssh://...' build and then docker-compose -H 'ssh://...' up. This does not cause the error.

+4
tobyfoo on Jan 31, 2019

yea 500 seemed to do the trick.

+2
amovfx on Apr 20, 2020

Had this bug, too. Since not everybody is able (or wants) to configure MaxSessions, the issue can be bypassed:

docker-compose -f docker-compose.yml -p $PROJECT config | ssh $USER@$HOST docker-compose -f - -p $PROJECT up --remove-orphans -d

Still assumes you have docker-compose available on both ends. Using this in CI.

+2
derfabianpeter on Dec 11, 2019

Dear all,

I have been banging my head against the wall for a while with this issue. I wanted to share my workaround in case it can help someone else. It does not need to change the SSH config of the client or the server.

I replaced: docker-compose -H 'ssh://remoteUser@remoteHost' up -d by: ssh -fTNL localhost:2377:/var/run/docker.sock remoteUser@remoteHost #Forward local port to docker socket in server export DOCKER_HOST=localhost:2377 docker-compose -H 'ssh://remoteUser@remoteHost' up -d

I run this in a Gitlab CI container, which dies after it does its job. For other use cases, remember to clean after yourself (i.e: close the port forwarding and reset DOCKER_HOST to whatever it was before you changed).

I found the solution here: Note that their ssh command is different to mine. They do not send it to the background, so cleaning it is easier for them.

Hope it helps!

+1
damiarita on Feb 11, 2022

Thanks @tobyfoo!

I opened an issue on docker-py, the SSH transport is not behaving like the others.

Hopefully we will have a fix soon.

+1
rumpl on Jan 31, 2019
Read more comments on GitHub
← compose: Secrets invalid mount config for type 'bind' mount path must be absolute
compose: Service can't be reached by defined hostname →