compose: Timeout to docker host causes containers to stop

Before 1.5.0, from time to time, python stack trace was popping up.

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/Cellar/docker-compose/1.4.2/libexec/lib/python2.7/site-packages/compose/cli/multiplexer.py", line 41, in _enqueue_output
    for item in generator:
  File "/usr/local/Cellar/docker-compose/1.4.2/libexec/lib/python2.7/site-packages/compose/cli/log_printer.py", line 59, in _make_log_generator
    for line in line_generator:
  File "/usr/local/Cellar/docker-compose/1.4.2/libexec/lib/python2.7/site-packages/compose/cli/utils.py", line 100, in split_buffer
    for data in reader:
  File "/usr/local/Cellar/docker-compose/1.4.2/libexec/vendor/lib/python2.7/site-packages/docker/clientbase.py", line 238, in _stream_raw_result
    for out in response.iter_content(chunk_size=1, decode_unicode=True):
  File "/usr/local/Cellar/docker-compose/1.4.2/libexec/vendor/lib/python2.7/site-packages/requests/utils.py", line 332, in stream_decode_response_unicode
    for item in iterator:
  File "/usr/local/Cellar/docker-compose/1.4.2/libexec/vendor/lib/python2.7/site-packages/requests/models.py", line 680, in generate
    raise ConnectionError(e)
ConnectionError: HTTPSConnectionPool(host='192.168.99.104', port=2376): Read timed out.

In 1.5.0 this stack trace doesn’t show up anymore, but all containers are stopped instead. Even though the target docker machine is running.

Reverted to 1.4.2 and it works fine.

About this issue

Original URL
State: closed
Created 9 years ago
Comments: 21

Commits related to this issue

Comment out tty This option let's us view stdout print statements but causes timeouts with docker-compose. It can be enabled momentarily when debugging until a fix is found. https://github.com/docke... — committed to November-Project/tracker-api by tonyd256 9 years ago
Add docker timeouts for tty https://github.com/docker/compose/issues/3927 https://github.com/docker/compose/issues/3106 https://github.com/docker/compose/issues/2338 https://github.com/docker/docker-... — committed to astronomer/airflow by tedmiston 7 years ago

Most upvoted comments

@ChrisCinelli After running with -d you can simply run:

docker-compose logs; while [ $? -ne 0 ]; do docker-compose logs; done;

and you will have the logs just like you’re used to. This line will give you the logs for the orchestrated container(s) and will keep them (logs) alive if it loses connection with docker engine.

nbap on Dec 1, 2015

Well, I’m still thinking that it’s not bad to disable timeout. Regarding to this timeouts documentation http://docs.python-requests.org/en/latest/user/advanced/#timeouts there are connect and read timeouts. docker-py sets the same timeout value for both - connect and read by invoking _set_request_timeout method - https://github.com/docker/docker-py/blob/master/docker/client.py#L107. So, in either case we have already set connect timeout. With stream enabled and interactive mode (pseudo-TTY enabled) read timeout doesn’t really make sense, since interactive is not really about receiving data all the time. At this step - https://github.com/docker/docker-py/blob/master/docker/client.py#L249 - we have already established connection, so that line only disables read timeout. Also, please take a look at the following documentation from https://docs.python.org/2/library/socket.html#socket.socket.settimeout:

Note that the connect() operation is subject to the timeout setting, and in general it is recommended to call settimeout() before calling connect() or pass a timeout parameter to create_connection(). The system network stack may return a connection timeout error of its own regardless of any Python socket timeout setting.

Further, it’s good idea to enable TCP keep-alive - once in a certain period of time probe packet with no data will be sent to make sure that connection is still alive and it doesn’t disappeared accidentally. That’s the point how to find out if something is wrong with a connection. No output provided is not the reason for decision that something is wrong with connection. More about TCP keep-alive - https://delog.wordpress.com/2013/08/16/handling-tcp-keepalive/, http://tldp.org/HOWTO/html_single/TCP-Keepalive-HOWTO/#whatis.

So, just to summarize - connect timeout is left as is, disable read timeout and enable TCP keep-alive.

ux on Dec 4, 2015

In my case Read timed out error happens just if pseudo-TTY option is enabled and no output is provided during COMPOSE_HTTP_TIMEOUT period:

tty: true

If it’s not enabled then everythings works as expected.

Docker compose version information:

docker-compose version 1.6.0dev, build 707281a
docker-py version: 1.5.0
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013

Digging deeper I noticed that problem seems to be in the docker-py client. I cannot be sure about that, since I’m not very familiar with docker-compose internals. So, for a response stream docker-py disables timeout on the underlying socket - see https://github.com/docker/docker-py/blob/1.5.0/docker/client.py#L247-L250, but this doesn’t happen if pseudo-TTY is enabled - see https://github.com/docker/docker-py/blob/1.5.0/docker/client.py#L291-L293 and https://github.com/docker/docker-py/blob/1.5.0/docker/client.py#L273-L277.

So, disabling the timeout if pseudo-TTY enabled solves the above mentioned problem - https://gist.github.com/ux/ac4fd45392aedb380903. I don’t know if it’s a correct way to solve this problem, but it works for me.

ux on Dec 3, 2015

I think a workaround for now is to run with docker-compose up -d. That way connection failures won’t stop any containers. You can still tail logs with docker-compose logs.

dnephin on Nov 19, 2015