compose: error on launching docker-compose by piping to sh ( echo 'docker-compose ... ' | sh )

I’ve a application which has multi container. to easy installation, I decide to use package image which contains docker-comopse.yaml file. what I expected is like below

$ docker run my_application install | sh this will pull all related image

$ docker run my_application up | sh this will do “docker-compose up” and some intial job for application using “docker-compose exec”

But I got below error

$ echo "docker-compose exec cassandra cqlsh -e 'desc keyspaces'" | sh
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "compose/cli/main.py", line 57, in main
  File "compose/cli/main.py", line 108, in perform_command
  File "compose/cli/main.py", line 353, in exec_command
  File ".tox/py27/lib/python2.7/site-packages/dockerpty/pty.py", line 338, in start
  File ".tox/py27/lib/python2.7/site-packages/dockerpty/io.py", line 32, in set_blocking
ValueError: file descriptor cannot be a negative integer (-1)
docker-compose returned -1

and I confiremd that below two commands works well

$ echo "docker exec my_application_cassandra_1 cqlsh -e 'desc keyspaces'" | sh

system_traces  system

$ sh -c "docker-compose exec cassandra cqlsh -e 'desc keyspaces'"

system_traces  system

Does someone have any idea why the error comes?

below is my version info and logs from docker-compose

$ docker version
Client:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Sat Mar 12 19:18:57 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Sat Mar 12 19:18:57 2016
 OS/Arch:      linux/amd64
$ docker-compose version
docker-compose version 1.7.0rc1, build 1ad8866
docker-py version: 1.8.0-rc2
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013

$ sh -c "docker-compose --verbose exec cassandra cqlsh -e 'desc keyspaces'"
compose.config.config.find: Using configuration files: ./docker-compose.yml
docker.auth.auth.load_config: Found 'auths' section
docker.auth.auth.parse_auth: Found entry (registry=u'dtr.test.org', username=u'keyolk')
docker.auth.auth.parse_auth: Found entry (registry=u'test.org', username=u'$oauthtoken')
compose.cli.command.get_client: docker-compose version 1.7.0rc1, build 1ad8866
docker-py version: 1.8.0-rc2
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
compose.cli.command.get_client: Docker base_url: http+docker://localunixsocket
compose.cli.command.get_client: Docker version: KernelVersion=4.3.2-0-vanilla, Os=linux, BuildTime=2016-03-12T19:18:57.542338402+01:00, ApiVersion=1.22, Version=1.10.3, GitCommit=20f81dd, Arch=amd64, GoVersion=go1.5.3
compose.cli.verbose_proxy.proxy_callable: docker containers <- (filters={u'label': [u'com.docker.compose.project=insator', u'com.docker.compose.service=cassandra', u'com.docker.compose.oneoff=False', u'com.docker.compose.container-number=1']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 1 items)
compose.cli.verbose_proxy.proxy_callable: docker exec_create <- (u'63c121aa356c5435a9bae43c08cd805268f7253734d44e01aa2d044a41304253', ['cqlsh', '-e', 'desc keyspaces'], tty=True, stdin=True, privileged=False, user=None)
compose.cli.verbose_proxy.proxy_callable: docker exec_create -> {u'Id': u'638f966cd2cf003da3b6fb32238a8750c61bfe147eabd103540501e55f8af725'}
compose.cli.verbose_proxy.proxy_callable: docker exec_start <- ({u'Id': u'638f966cd2cf003da3b6fb32238a8750c61bfe147eabd103540501e55f8af725'}, tty=True, socket=True)
compose.cli.verbose_proxy.proxy_callable: docker exec_start -> <socket object, fd=7, family=1, type=1, protocol=0>
compose.cli.verbose_proxy.proxy_callable: docker exec_inspect <- ({u'Id': u'638f966cd2cf003da3b6fb32238a8750c61bfe147eabd103540501e55f8af725'})
compose.cli.verbose_proxy.proxy_callable: docker exec_inspect -> {u'CanRemove': False,
 u'ContainerID': u'63c121aa356c5435a9bae43c08cd805268f7253734d44e01aa2d044a41304253',
 u'DetachKeys': u'',
 u'ExitCode': None,
 u'ID': u'638f966cd2cf003da3b6fb32238a8750c61bfe147eabd103540501e55f8af725',
 u'OpenStderr': True,
 u'OpenStdin': True,
 u'OpenStdout': True,
 u'ProcessConfig': {u'arguments': [u'-e', u'desc keyspaces'],
                    u'entrypoint': u'cqlsh',
...
compose.cli.verbose_proxy.proxy_callable: docker exec_resize <- ({u'Id': u'638f966cd2cf003da3b6fb32238a8750c61bfe147eabd103540501e55f8af725'}, width=190, height=42)
                                                                                                                                                                    compose.cli.verbose_proxy.proxy_callable: docker exec_resize -> None

system_traces  system

compose.cli.verbose_proxy.proxy_callable: docker exec_inspect <- ({u'Id': u'638f966cd2cf003da3b6fb32238a8750c61bfe147eabd103540501e55f8af725'})
compose.cli.verbose_proxy.proxy_callable: docker exec_inspect -> {u'CanRemove': False,
 u'ContainerID': u'63c121aa356c5435a9bae43c08cd805268f7253734d44e01aa2d044a41304253',
 u'DetachKeys': u'',
 u'ExitCode': 0,
 u'ID': u'638f966cd2cf003da3b6fb32238a8750c61bfe147eabd103540501e55f8af725',
 u'OpenStderr': True,
 u'OpenStdin': True,
 u'OpenStdout': True,
 u'ProcessConfig': {u'arguments': [u'-e', u'desc keyspaces'],
                    u'entrypoint': u'cqlsh',
...
echo "docker-compose --verbose exec cassandra cqlsh -e 'desc keyspaces'" | sh                                   [2/22580]
compose.config.config.find: Using configuration files: ./docker-compose.yml
docker.auth.auth.load_config: Found 'auths' section
docker.auth.auth.parse_auth: Found entry (registry=u'dtr.test.org', username=u'keyolk')
docker.auth.auth.parse_auth: Found entry (registry=u'test.org', username=u'$oauthtoken')
compose.cli.command.get_client: docker-compose version 1.7.0rc1, build 1ad8866
docker-py version: 1.8.0-rc2
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
compose.cli.command.get_client: Docker base_url: http+docker://localunixsocket
compose.cli.command.get_client: Docker version: KernelVersion=4.3.2-0-vanilla, Os=linux, BuildTime=2016-03-12T19:18:57.542338402+01:00, ApiVersion=1.22, Version=1.10.3, GitCommit=20f81dd, A$ch=amd64, GoVersion=go1.5.3
compose.cli.verbose_proxy.proxy_callable: docker containers <- (filters={u'label': [u'com.docker.compose.project=insator', u'com.docker.compose.service=cassandra', u'com.docker.compose.oneo$f=False', u'com.docker.compose.container-number=1']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 1 items)
compose.cli.verbose_proxy.proxy_callable: docker exec_create <- (u'63c121aa356c5435a9bae43c08cd805268f7253734d44e01aa2d044a41304253', ['cqlsh', '-e', 'desc keyspaces'], tty=True, stdin=True$ privileged=False, user=None)
compose.cli.verbose_proxy.proxy_callable: docker exec_create -> {u'Id': u'28344216442c0d58abce8a9face0430d1551d66eed3ec95477a1b333c67faf01'}
compose.cli.verbose_proxy.proxy_callable: docker exec_start <- ({u'Id': u'28344216442c0d58abce8a9face0430d1551d66eed3ec95477a1b333c67faf01'}, tty=True, socket=True)
compose.cli.verbose_proxy.proxy_callable: docker exec_start -> <socket object, fd=7, family=1, type=1, protocol=0>
compose.cli.verbose_proxy.proxy_callable: docker exec_inspect <- ({u'Id': u'28344216442c0d58abce8a9face0430d1551d66eed3ec95477a1b333c67faf01'})
compose.cli.verbose_proxy.proxy_callable: docker exec_inspect -> {u'CanRemove': False,
 u'ContainerID': u'63c121aa356c5435a9bae43c08cd805268f7253734d44e01aa2d044a41304253',
 u'DetachKeys': u'',
 u'ExitCode': None,
 u'ID': u'28344216442c0d58abce8a9face0430d1551d66eed3ec95477a1b333c67faf01',
 u'OpenStderr': True,
 u'OpenStdin': True,
 u'OpenStdout': True,
 u'ProcessConfig': {u'arguments': [u'-e', u'desc keyspaces'],
                    u'entrypoint': u'cqlsh',
...
compose.cli.verbose_proxy.proxy_callable: docker exec_resize <- ({u'Id': u'28344216442c0d58abce8a9face0430d1551d66eed3ec95477a1b333c67faf01'}, width=190, height=42)
compose.cli.verbose_proxy.proxy_callable: docker exec_resize -> None
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "compose/cli/main.py", line 57, in main
  File "compose/cli/main.py", line 108, in perform_command
  File "compose/cli/main.py", line 353, in exec_command
  File ".tox/py27/lib/python2.7/site-packages/dockerpty/pty.py", line 338, in start
  File ".tox/py27/lib/python2.7/site-packages/dockerpty/io.py", line 32, in set_blocking
ValueError: file descriptor cannot be a negative integer (-1)
docker-compose returned -1

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Reactions: 48
  • Comments: 38

Commits related to this issue

Most upvoted comments

I could get it to work by adding the -T parameter to not create a Pseudo-TTY.

docker-compose exec -T container_name ./build.sh for example.

for all those curious about getting around this while still getting information from docker-compose…

docker exec -i $(docker-compose ps -q cassandra) < someexample.cql

The -i keeps stdin open while not allotting a tty and the ps -q on the docker-compose provides us with the container ID we want.

I assume this will provide multiple IDs if you have many running, you’ll have to do some munging to use this method.

It would appear docker-compose isn’t implicitly using a -i when passing the exec along.

So basically, you need -i for docker exec when you are piping in data to the command.

And you should not use -t for docker exec when you are piping data out from the command.

And docker-compose exec only have -T, which means neither -i or -t. But you cannot represent -i without -t in docker-compose.

That’s the issue, right?

Usecase TTY Stdin Option
< Does not matter Yes N/A (This issue!)
> No Does not matter -T
Neither Does not matter Does not matter Does not matter
Both No Yes N/A (This issue!)

I’m having the same issue when running: docker-compose exec mycontainer cmd_that_takes_stdin < my/file/with/content

Everything seems to work after all (the command executes as expected), it’s just that nasty crash that docker-compose gives me afterwards:

$ docker-compose exec mycontainer cmd_that_takes_stdin < my/file/with/content
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "compose/cli/main.py", line 61, in main
  File "compose/cli/main.py", line 113, in perform_command
  File "compose/cli/main.py", line 441, in exec_command
  File "site-packages/dockerpty/pty.py", line 338, in start
  File "site-packages/dockerpty/io.py", line 32, in set_blocking
ValueError: file descriptor cannot be a negative integer (-1)
docker-compose returned -1

$ echo $?
255

When I use the -T flag as recommended though, my command doesn’t work and I even get stuck on the docker-compose command (doesn’t return to my shell but just hangs instead).

I’m having the same problem, when piping docker-compose output via gzip to a file. When I run the command myself in Bash, it works just fine. However, as a cron job, I’m getting the same error as you — and the same stack trace file & line numbers & function names, as you, and return code -1.

Here’s the command:

/usr/local/bin/docker-compose exec postgres pg_dumpall --username=postgres \
   | gzip > $postgres_backup_path

(I have that line in a backup script that I run sometimes manually (works fine) and sometimes as a cron job (won’t work)).


Now I’ve worked around my problem by calling Docker directly like so:

/usr/bin/docker exec edm_postgres_1 pg_dumpall --username=postgres \
    | gzip > $postgres_backup_path

— works fine from both Cron and manually in Bash.

reproduced by gitlab-ci

Same here. docker-compose exec fails when running as part of a Jenkins job (works locally), but docker exec works. Thanks @paolomainardi !

I can reproduce this bug just when docker-compose gets called from jenkins or gitlab-ci

I’d say that the problem is that -T in docker-compose exec is mostly equivalent to not using any of -t or -i flags in docker run, so with -T there is not tty but also not attached stdin, and it seems that there is no way of attaching stdin without tty in docker-compose.

docker-compose exec should probably have different flags for tty and for attaching stdin as docker run has.

same problem on jenkins, solved by using -T: docker-compose exec -T container_name cmd

I have the same problem, but just by launching this from jenkins (it fails just there): docker-compose exec drupal bash -c "bin/behat --profile=ci", now changed to docker exec ${COMPOSE_PROJECT_NAME}_drupal_1 bash -c "bin/behat --profile=ci".

now I try in cron (again) crontab -e

*/10 * * * * time -p /usr/local/bin/docker-compose -f /path/to/docker-compose.yml exec php ./yii command

now I have same problem in cron (again)

[17362] Failed to execute script docker-compose
Traceback (most recent call last):
  File "site-packages/dockerpty/pty.py", line 334, in start
  File "site-packages/dockerpty/pty.py", line 367, in _hijack_tty
  File "site-packages/dockerpty/io.py", line 59, in select
  File "site-packages/dockerpty/io.py", line 351, in fileno
  File "site-packages/dockerpty/io.py", line 103, in fileno
  File "socket.py", line 635, in fileno
ValueError: I/O operation on closed file.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bin/docker-compose", line 6, in <module>
  File "compose/cli/main.py", line 71, in main
  File "compose/cli/main.py", line 127, in perform_command
  File "compose/cli/main.py", line 519, in exec_command
  File "site-packages/dockerpty/pty.py", line 338, in start
  File "site-packages/dockerpty/io.py", line 32, in set_blocking
  File "site-packages/dockerpty/io.py", line 351, in fileno
  File "site-packages/dockerpty/io.py", line 103, in fileno
  File "socket.py", line 635, in fileno
ValueError: I/O operation on closed file.
Command exited with non-zero status 255
real 2.09
user 0.98

after add into crontab -e

COMPOSE_INTERACTIVE_NO_CLI=1
# Edit this file to introduce tasks to be run by cron.

https://stackoverflow.com/a/10657111/7670492

I try again with -T

/usr/local/bin/docker-compose -f /path/to/docker-compose.yml exec -T php ./yii command

success

So, as I understand, docker exec is the solution for now?

Just want to leave this minimal test-case for docker-compose:

echo 123 | docker-compose exec container_name tee

It triggers error:

Traceback (most recent call last):
  File "bin/docker-compose", line 3, in <module>
  File "compose/cli/main.py", line 88, in main
  File "compose/cli/main.py", line 140, in perform_command
  File "compose/cli/main.py", line 486, in exec_command
  File "site-packages/dockerpty/pty.py", line 338, in start
  File "site-packages/dockerpty/io.py", line 32, in set_blocking
ValueError: file descriptor cannot be a negative integer (-1)
Failed to execute script docker-compose

echo 123 | docker exec -i $(docker-compose ps -q service) cat - for now…

For people who are stumbling into this while trying to run docker-compose exec on Jenkins pipeline like myself:

The -T flag mentioned bypasses the error, but it does not actually execute the command for me. Jenkins just thinks the command executed successfully and moves on.

I had to fall back to docker exec, like so:

environment {
  PROJECT_NAME = "${env.BRANCH_NAME}.${env.BUILD_ID}"
  CONTAINER_PREFIX = "${PROJECT_NAME.replace('-', '').replace('.', '').toLowerCase()}"
}
stages {
  stage('run') {
    steps {
      sh 'docker-compose -p $PROJECT_NAME up -d --build'
      sh "docker exec ${CONTAINER_PREFIX}_app_1 bash -c \"hello world\""
    }
  }
}

The $CONTAINER_PREFIX is a workaround to get the docker compose prefix. It’s not pretty, and a little too verbose, but it’s the only way I could get it to work for me.

env: ansible playbook + ubuntu(vagrant)

error:

Traceback (most recent call last):
  File "/usr/local/bin/docker-compose", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/compose/cli/main.py", line 65, in main
    command()
  File "/usr/local/lib/python2.7/dist-packages/compose/cli/main.py", line 117, in perform_command
    handler(command, command_options)
  File "/usr/local/lib/python2.7/dist-packages/compose/cli/main.py", line 462, in exec_command
    pty.start()
  File "/usr/local/lib/python2.7/dist-packages/dockerpty/pty.py", line 338, in start
    io.set_blocking(pump, flag)
  File "/usr/local/lib/python2.7/dist-packages/dockerpty/io.py", line 32, in set_blocking
    old_flag = fcntl.fcntl(fd, fcntl.F_GETFL)
ValueError: file descriptor cannot be a negative integer (-1)

add -T fix the issue.

-T isn’t working for me. My only workaround is to use docker exec directly but means I have to parse the container ID from docker ps which isn’t ideal.

To test I have a basic python script that just prints stdin

$ cat myScript.py
#!/usr/bin/python
import sys
print "started"
for line in sys.stdin:
    print line

$ echo test | docker-compose exec -T myContainer /usr/src/myScript.py
.. nothing happens, just waits ..

$ echo test | docker-compose exec myContainer /usr/src/myScript.py
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "compose/cli/main.py", line 65, in main
  File "compose/cli/main.py", line 117, in perform_command
  File "compose/cli/main.py", line 462, in exec_command
  File "site-packages/dockerpty/pty.py", line 338, in start
  File "site-packages/dockerpty/io.py", line 32, in set_blocking
ValueError: file descriptor cannot be a negative integer (-1)
docker-compose returned -1

$ echo test | docker exec -i d6380314bb89  /usr/src/myScript.py
started
test
docker-compose exec container_name cat logfile > logfile

also fails for me, but with -T

docker-compose exec -t container_name cat logfile > logfile

it just produce empty file, so no any output! Even there no error, it is unusable.

I think he means you can use docker-compose ps -q {app} to get the container ID. For example,

docker exec $(docker-compose ps -q {app}) bash -c …

I’m running into this exact issue as well. Would love to be able to pipe data into containers via docker-compose.

Have you tried adding the -T flag to docker-compose exec?