urllib3: Behavior change: v2.x raises SSLEOFError with Python3.10 where v2.x does not with Python3.8
Subject
Setup: Using robotframework-requests library which in turn utilizes requests library to issue GET requests to a non-public web server over https. The web server is third-party. The GET requests work fine using Python3.8, openssl [1.1.1f|3.0.2] and urllib3 2.x The GET requests, however, raise following error when using the same context with only Python changed to 3.10:
SSLError: HTTPSConnectionPool(host='10.1.1.32', port=443): Max retries exceeded with url: /netio.json (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1007)')))
Environment
Reproduced on Ubuntu 20.04 LTS and 22.04 LTS (openssl 1.1.1f and openssl 3.0.2)
Combination of libs that raise the error (Ubuntu 22.04 LTS):
OS Linux-5.4.0-131-generic-x86_64-with-glibc2.35
Python 3.10.12
OpenSSL 3.0.2 15 Mar 2022
urllib3 2.0.4
Combination of libs that do not raise the error (Ubuntu 22.04 LTS):
OS Linux-5.4.0-131-generic-x86_64-with-glibc2.35
Python 3.10.12
OpenSSL 3.0.2 15 Mar 2022
urllib3 1.26.15
Difference between the two combinations:
pip install --upgrade --force-reinstall urllib3==1.26.15
Combination of libs that do not raise the error as well (Ubuntu 20.04 LTS):
OS Linux-5.4.0-131-generic-x86_64-with-glibc2.29
Python 3.8.10
OpenSSL 1.1.1f 31 Mar 2020
urllib3 2.0.4
Steps to Reproduce
It’s hard to reproduce publicly because the web server is not public. But my findings lead to the SSL_OP_IGNORE_UNEXPECTED_EOF flag.
Expected Behavior
v2.x shall behave the same way with Python 3.10 as it does with Python 3.8 regarding the error described above (i. e. not raising any SSLEOFError)
Actual Behavior
v2.x in combination with Python 3.10 raises SSLEOFError v2.x in combination with Python 3.8 does not raise SSLEOFError
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 16 (8 by maintainers)
My bad. I didn’t want to spread the traces throughout the thread, so I edit’ed several times…
Aha, if we don’t set a default, we get the default from CPython, defined in https://github.com/python/cpython/blob/65ce3652fa47a34acf715ee07bf0a2fae2e0da51/Modules/_ssl.c#L151-L186
And indeed, with
ctx.set_ciphers("@SECLEVEL=2:ECDH+AESGCM:ECDH+CHACHA20:ECDH+AES:DHE+AES:!aNULL:!eNULL:!aDSS:!SHA1:!AESCCM")I get the first block above, withoutAES256-SHA256. So callingset_ciphers("DEFAULT")explicitly makes sense.Oh, wow, so using the following script on the Ubuntu Focal Docker image:
I get the following output, with corresponds to your Wireshark ciphers:
But now, if I add
ctx.set_ciphers("DEFAULT"), I get something totally different:Which includes the line of interest:
On my Fedora laptop, I also see a difference where DEFAULT adds some CBC ciphers and removes some CCM ciphers.
We can easily call
set_ciphers("DEFAULT")here which would fix your immediate issue:https://github.com/urllib3/urllib3/blob/53368dfe6609aec8f3d933b62021cf568f24b794/src/urllib3/util/ssl_.py#L297-L300
But I’d have to understand what is the difference between not calling
set_ciphers()and callingset_ciphers("DEFAULT").Thanks for the responsiveness, it’s very appreciated.
I used tcpdump to capture pcap files directly from within the Docker container which is connecting to the web server. Hope this helps identifying the difference.
pip install --upgrade --force-reinstall urllib3==2.0.0For reference/comparison: working py38 urllib2 context
pcap.zip
I was trying to find a public server where I could reproduce the issue but had no luck. Going to see if I can set up wireshark to capture traffic (it’s a bit tricky since everything is executed within Docker containers from a sand-boxed network)
maybe related to SSLEOFError in Python 3.10 not in Python 3.9