docker-py: urllib3 v2 incompatibility

Docker is installed over pip with the dependencies:

       "Requirement already up-to-date: docker in /usr/local/lib/python3.8/dist-packages (6.0.1)",
        "Requirement already satisfied, skipping upgrade: requests>=2.26.0 in /usr/local/lib/python3.8/dist-packages (from docker) (2.29.0)",
        "Requirement already satisfied, skipping upgrade: packaging>=14.0 in /usr/local/lib/python3.8/dist-packages (from docker) (23.1)",
        "Requirement already satisfied, skipping upgrade: websocket-client>=0.32.0 in /usr/local/lib/python3.8/dist-packages (from docker) (1.5.1)",
        "Requirement already satisfied, skipping upgrade: urllib3>=1.26.0 in /usr/local/lib/python3.8/dist-packages (from docker) (2.0.1)",
        "Requirement already satisfied, skipping upgrade: charset-normalizer<4,>=2 in /usr/local/lib/python3.8/dist-packages (from requests>=2.26.0->docker) (3.1.0)",
        "Requirement already satisfied, skipping upgrade: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests>=2.26.0->docker) (2.8)",
        "Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests>=2.26.0->docker) (2019.11.28)"

And we have the same problem: “msg”: “Error connecting: Error while fetching server API version: request() got an unexpected keyword argument ‘chunked’”

The only solution for us was:

pip3 remove requests pip3 install requests=2.28.1

Wow this post is really helpful as I’ve just encounter an issue with Ansible managing Docker.

I think urllib3 2.0.0 (or vendored urllib3 verions in requests) is a red herring; the problem is requests 2.29.0 itself because of a new feature: https://github.com/psf/requests/pull/6226/files

Docker SDK for Python uses requests with its own HTTP adapters for Unix connections by default (if you don’t talk to the Docker daemon over TCP connections). For that, the code in Docker SDK for Python that provides a Unix connection HTTP adapter needs to be adjusted to support the same chunking support that urllib3 (already < 2.0.0) provides, and that requests 2.29.0 uses. (The other HTTP adapters might have the same problems.)

Hello, urllib3 2.0 maintainer here 👋

First, let me mention that right now, if you use pip install docker with a recent enough pip that can do dependency resolution you will end up with requests 2.29.0 and urllib3 1.26.15. With those versions, the docker.from_env reproducer above works. I’m surprised that @felixfontein sees issues with those versions, can you please explain how to reproduce them?

I can confirm however that docker-py is not compatible with urllib3 2.0. Since requests will soon allow it, docker-py should modify setup.py to use urllib3 >= 1.26.0, < 2.0.0 or fix the bug.

---

So what is the issue? Here is how docker-py supports Unix domain sockets:

https://github.com/docker/docker-py/blob/a02ba743338c27fd9348af2cf7767b140501734d/docker/transport/unixconn.py#L17-L37

It assumes that urllib3.connectionpool.HTTPConnectionPool will call conn.request in a way that is compatible with http.client, but this has changed in urllib3 2.0, and HTTPConnectionPool has its own request() function with its own signature.

I’ve opened https://github.com/docker/docker-py/pull/3117 for a fix.

I can confirm just pinning requests<2.2.29 is solving this for me

Help please! 🙏🏼

How do I pin to urllib3<2 ?

@MaritzaB pip install 'urllib3<2' or add urllib3<2 in your requirements list.

Docker is installed over pip with the dependencies:

       "Requirement already up-to-date: docker in /usr/local/lib/python3.8/dist-packages (6.0.1)",
        "Requirement already satisfied, skipping upgrade: requests>=2.26.0 in /usr/local/lib/python3.8/dist-packages (from docker) (2.29.0)",
        "Requirement already satisfied, skipping upgrade: packaging>=14.0 in /usr/local/lib/python3.8/dist-packages (from docker) (23.1)",
        "Requirement already satisfied, skipping upgrade: websocket-client>=0.32.0 in /usr/local/lib/python3.8/dist-packages (from docker) (1.5.1)",
        "Requirement already satisfied, skipping upgrade: urllib3>=1.26.0 in /usr/local/lib/python3.8/dist-packages (from docker) (2.0.1)",
        "Requirement already satisfied, skipping upgrade: charset-normalizer<4,>=2 in /usr/local/lib/python3.8/dist-packages (from requests>=2.26.0->docker) (3.1.0)",
        "Requirement already satisfied, skipping upgrade: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests>=2.26.0->docker) (2.8)",
        "Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests>=2.26.0->docker) (2019.11.28)"

And we have the same problem: “msg”: “Error connecting: Error while fetching server API version: request() got an unexpected keyword argument ‘chunked’”

The only solution for us was:

pip3 remove requests pip3 install requests=2.28.1

this fixed the issue for me

pip uninstall requests pip install requests==2.28.1

A huge thank you to everyone here for the debugging, discussion, and PR(s) to fix the issue! ❤️

A new release is available with this fix as well as several other queued up improvements: https://github.com/docker/docker-py/releases/tag/6.1.0

The target 22.04 box has

❯ pip list | egrep 'request|urllib'
requests               2.30.0
urllib3                2.0.2

I downgraded requests to 2.29.0, which also brought urllib down

❯ pip install requests==2.29.0 --force-reinstall

❯ pip list | egrep 'request|urllib'
requests               2.29.0
urllib3                1.26.15

Unfortunately, still get the same error

Also note that Ansible 7.5.0 includes community.docker 3.4.3; if you use modules that use the vendored Docker SDK for Python code instead of Docker SDK for Python directly, you need to make sure that you have community.docker 3.4.5 installed instead if you have requests >= 2.29.0 or urllib3 >= 2.0 on the target machine. It will be part of Ansible 7.6.0 (which should get released in ~two weeks).

I created a PR for making Docker SDK for Python compatible with requests 2.29.0: #3116. It fixes the problems for me (as long as I stick to urllib3 < 2.0, what request has been requiring for a long time now). It seems to fix the problems I had with requests 2.29.0 and urllib3 < 2.0; would be nice to have more real life testing (especially for all the transports I don’t use).

This was fixed in community.docker 3.4.5, are you using an older version? Please report any further issues you have to https://github.com/ansible-collections/community.docker instead.

@komkomissarov you need to upgrade community.docker. You are using an outdated version (7 months old!) which still has this bug (which only got fixed a ~month ago). You need version 3.4.5 (released 24 days ago) or later.

You can run the following with requests==0.29.0 and urllib3==1.26.13 to trigger an error:

from docker import APIClient
from docker.utils import kwargs_from_env

kwargs = kwargs_from_env()
client = APIClient(**kwargs)

def generate_data():
    yield b'123'

container = client.create_container(image='debian:bullseye', command=['/bin/sh', '-c', 'ls -la /'], stdin_open=True, detach=True, name='test')
client.put_archive(container['Id'], '/', generate_data())

Stacktrace:

Traceback (most recent call last):
  File "/path/to/my-script.py", line 11, in <module>
    client.put_archive(container['Id'], '/', generate_data())
  File "/path/to/docker-py/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/path/to/docker-py/docker/api/container.py", line 980, in put_archive
    res = self._put(url, params=params, data=data)
  File "/path/to/docker-py/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/path/to/docker-py/docker/api/client.py", line 241, in _put
    return self.put(url, **self._set_request_timeout(kwargs))
  File "/home/felix/.local/lib/python3.10/site-packages/requests/sessions.py", line 647, in put
    return self.request("PUT", url, data=data, **kwargs)
  File "/home/felix/.local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/felix/.local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/home/felix/.local/lib/python3.10/site-packages/requests/adapters.py", line 487, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 396, in _make_request
    conn.request_chunked(method, url, **httplib_request_kw)
AttributeError: 'UnixHTTPConnection' object has no attribute 'request_chunked'

So yes, you need requests < 2.29.0 as well with the currently released Docker SDK for Python.

I think also the request version needs to be pinned, as there was a change? Works with requests 2.28.1, but fails with 2.29.x

/tmp/ansible_docker_swarm_payload_315b1r1d/ansible_docker_swarm_payload.zip/ansible_collections/community/docker/plugins/module_utils/common.py", line 222, in _init_
    super(AnsibleDockerClientBase, self)._init_(**self._connect_params)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 197, in _init_
    self._version = self._retrieve_server_version()
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(...
....
},
    "msg": "Error connecting: Error while fetching server API version: request() got an unexpected keyword argument 'chunked'"
}

I was, yes. I’d also tried switching between the O/S (Ubuntu 22.04) package and PIP installs of docker-py on both host and target, and getting nowhere, but hadn’t through to update the Ansible collection, which has resolved the issue!

I faced the same issue when was tying to set up server with ansible , two work around which helped me:

1)Do not change versions of requests(latest) and urllib3(latest) and install docker-py instead of docker 2)remove urlib3 , install urllib3==1.26.15 and then install docker with pip3

Hope it will help anyone )

How the heck it is possible, that this issue has status fixed by a PR done in totally unrelated project microsoft/Olive?! How?

Any hints ?

In our case the problem was the requests/urllib3 version on the target machine. That’s where docker-py was executed with docker access via socket.

I can confirm just pinning requests<2.2.29 is solving this for me

The digits are transposed here. Pinning requests<2.29.2 does the trick for me as requests 2.29.0 still pins urllib3 < 2.