docker-py: error while using docker for mac client 2.5.0.0

Hello, same/similar issue.

Python version: 3.8.6 Docker version: 2.5.0 docker-py version: 4.3.1 urllib3 version: 1.25.11

reproducer:

import docker

client = docker.from_env()
client.containers.get("<non-existent-container-name>")

expected result: Exception at docker api level (docker.errors.NotFound or docker.errors.APIError)

result:

Traceback (most recent call last):
File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/urllib3/response.py", line 696, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/urllib3/response.py", line 436, in _error_catcher
    yield
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/urllib3/response.py", line 763, in read_chunked
    self._update_chunk_length()
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/urllib3/response.py", line 700, in _update_chunk_length
    raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/requests/models.py", line 751, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/urllib3/response.py", line 571, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/urllib3/response.py", line 792, in read_chunked
    self._original_response.close()
  File "/usr/local/Cellar/python@3.8/3.8.6/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/urllib3/response.py", line 454, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "reproducer.py", line 5, in <module>
    client.containers.get("asdrubale")
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/docker/models/containers.py", line 887, in get
    resp = self.client.api.inspect_container(container_id)
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/docker/api/container.py", line 771, in inspect_container
    self._get(self._url("/containers/{0}/json", container)), True
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/docker/api/client.py", line 228, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/requests/sessions.py", line 685, in send
    r.content
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/requests/models.py", line 829, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/Users/mconcas/.virtualenvs/alidock/lib/python3.8/site-packages/requests/models.py", line 754, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))```

We use this pattern to check if a container exists already before starting a container or when fetching the status of a container, as our software uses deterministic container names. In the case where the container isn’t running we catch docker.errors.NotFound, which is now broken as @mconcas points out. After initial testing, it does appear this is the only place in the API that is broken like this.

A very temporary fix is catching the unexpected error, again this is unexpected, fragile, and possibly not useful for everyone’s use case:

import docker

client = docker.from_env()
try:
    client.containers.get("<non-existent-container-name>")
except docker.errors.NotFound:
    pass
except requests.exceptions.ChunkedEncodingError:
    pass

Seeing the same issue using wsl1 on windows:

Python 3.8.2 Linux 5.4.39-linuxkit x86_64 Docker version 18.09.1, build 4c52b90 docker-py 4.3.1

This should be fixed in docker desktop 2.5.2 and up;

curl -v --unix-socket /var/run/docker.sock http://localhost/containers/nosuchcontainer/json

*   Trying /var/run/docker.sock:0...
* Connected to localhost (docker.sock) port 80 (#0)
> GET /containers/nosuchcontainer/json HTTP/1.1
> Host: localhost
> User-Agent: curl/7.65.3
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Api-Version: 1.41
< Content-Type: application/json
< Date: Tue, 01 Dec 2020 16:22:01 GMT
< Docker-Experimental: true
< Ostype: linux
< Server: Docker/20.10.0-rc1 (linux)
< Transfer-Encoding: chunked
<
{"message":"No such container: nosuchcontainer"}
* Connection #0 to host localhost left intact

Oh, actually, there might be a workaround; the “raw” socket is also available on the mac side in;

$ ls -la ~/Library/Containers/com.docker.docker/Data/docker.raw.sock

srwxr-xr-x   1 sebastiaan  staff         0 Nov 16 11:24 docker.raw.sock=

e.g.;

$ docker -H unix:///Users/sebastiaan/Library/Containers/com.docker.docker/Data/docker.raw.sock version
Client: Docker Engine - Community
 Version:           20.10.0-beta1
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        ac365d7
 Built:             Tue Oct 13 18:13:53 2020
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true
...

⚠️ ⚠️ BIG FAT WARNING!!! ⚠️ ⚠️

I think the name of that socket will be renamed soon, so only use this as a temporary workaround. Also, when “skipping” the API proxy, some things may not work as expected (for example, paths used in bind-mounts are not rewritten, network settings may not be adapted for how Docker Desktop is setup)

☝️ ☝️ ☝️ ☝️ IOW: the recommended socket to use will be /var/run/docker.sock, which is the official API to connect (for reasons above).

Mac facing the same problem 😃 hope it gets resolved! thanks guys

An interesting observation on various version. I’m seeing this same issue on:

• latest Docker Desktop for Mac stable channel (2.5.0.0)
• latest Docker Desktop for Windows edge channel (2.4.2.0) on HyperV

I do not see this issue on:

• latest Docker Desktop for Windows stable channel (2.5.0.0) on WSL2
• latest Docker Desktop for Windows stable channel (2.5.0.0) on HyperV