requests: The latest version of requests (2.29.0) does not support urllib3 2.0.0

Many, if not most, projects pull urllib3 into their closure through another dependency. Pinning transitive dependencies, while good practice through something like a lockfile, is not widely observed. We have a decade of issues on this project as empirical data on that. Often times, users aren’t even aware they’re using Requests, let alone urllib3. Those are the cases where this can go wrong very easily.

We’ll likely move the pin within the next week, but pushing a major version update out isn’t something we’ll do lightly. Giving urllib3 the opportunity release before we pull everyone else in is the safest approach and least amount of pain for operation teams.

This is something that shouldn’t be upgraded in a minor version of requests, because it causes breaking changes.

For example, we saw our elasticsearch client pull in the 2.0.0 version of urllib3 and it caused this failure on our machine. To resolve, we’re adding a constraints.txt file, but not something that should be included in a minor release of requests.

ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2k-fips  26 Jan 2017. See: https://github.com/urllib3/urllib3/issues/2168

Hi @mdmintz, this is intentional as discussed here. We’ll move the pin once we get more data points on any issues in the major version bump. We have a responsibility to keep the majority of the Python ecosystem stable during the transition.

As pointed out above pip install urllib3==1.26.15 worked for me. Thanks!

To resolve it, just force the installation of version 1.26.15 of urllib3.

@sethmlarson Previous releases of requests had pinned urllib3 to <1.27, and now in your current non-major-version-bumping release 2.30.0, you’ve bumped urllib3 to <3, which means everyone who currently has pinned requests to <3 in one of their past releases (in order to not have any breaking changes) will inevitably make users install urllib3 2.x unless they add the version constraint themselves and publish a new release with the updated version constraints. This is a breaking change of requests, unfortunately.

@nateprewitt I would like to note that if requests blocks urllib3 2.0.0 and everyone has requests installed for something, most people will not be able to get any new data about urllib3 2.0.0 in practice because it just cannot be installed. Is that picture missing something? Regarding the blast radius mentioned at https://github.com/psf/requests/pull/6430#issuecomment-1522542220 I would like to note that the responsibility is with every project using urllib3 directly, not with requests. Why would it be requests’ responsibility?

@bastimeyer This is a question of software development hygiene. If you have a system that relies on stability, you should (ideally) be generating a lockfile, or at a minimum reviewing changes prior to deployment. The addition of support is not a breaking change, choosing to pull in a new major version may be breaking which is why we highlighted to carefully review before installing Requests 2.30.0.

Moving the pin was specifically because Requests was incompatible with urllib3 2.0 previously. That constraint is no longer in play, so users are free to choose their versions as needed.

For anyone upgrading, we have yanked Requests 2.30.0 as urllib3 appears to be silently dropping data on compressed response bodies. We’ll be tracking investigation in urllib3/urllib3#3009.

This is something that shouldn’t be upgraded in a minor version of requests, because it causes breaking changes.

For example, we saw our elasticsearch client pull in the 2.0.0 version of urllib3 and it caused this failure on our machine. To resolve, we’re adding a constraints.txt file, but not something that should be included in a minor release of requests.

ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2k-fips  26 Jan 2017. See: https://github.com/urllib3/urllib3/issues/2168

To make what @nateprewitt already eloquently stated a bit more explicit:

We will not be requiring urllib3 2.0, we will be adding support for it by raising the version constraint and ensuring requests works correctly with that version.

If you cannot use 2.0 you can continue to prevent it from being installed and we won’t break of you use 1.26.x

@bastimeyer It was a major version release for the project which had incompatibilities introduced. Requests allows you to install either urllib3 1.26.x or 2.x, that is not backwards incompatible.

As was originally stated, Requests 2.30.0 has been released within a week of the urllib3 launch. Users who wish to stay on urllib3 1.x should ensure they’re pinned to urllib<2 prior to upgrading. This may include users on legacy systems with OpenSSL <1.1.1 or those relying on the Google App Engine integration.

I’ll resolve this now as it’s unblocked, thanks for your patience.

Mixing apt and pip installations of packages is nearly guaranteed to put your system into a broken state, and is basically unsupported (and unsupportable). I’d strongly encourage you to pick a singe package manager and learn about its pinning faculties.

Landon, that’s ultimately a decision for users of Requests to decide. We’re doing a very temporary overlap of this pin during launch to prevent legitimate bugs being pushed to millions of software environments.

We have no intention to release a new major version of Requests. Urllib3 2.0 was painstakingly made largely backwards compatible. We’ll provide a prominent notice that we now support this version in the next minor release, and users are free to pin to older versions as needed.

I prefer to use pip install "urllib3<2" so if there would be an eg 1.26.16 patch release that I would get it.

Take 2 🎬 We’ve unyanked 2.30.0 now that the regression in urllib3 has been resolved. I’ll leave this open for tracking today in the event of other late arising issues.

Hi @jplock, the default Boto3 provided with the Python Lambda images does not support urllib3 2.0. You’ll either need to upgrade to a recent version of Boto3 (and Botocore) or ensure you’re not bringing urllib3 2.0 into the package closure.

Shouldn’t this have been a major version bump here due to the introduced incompabilities?

@nateprewitt Please yank it again. This is causing issues once more. Ansible builds that were failing all week, stating working again 24 hours ago (I’m guessing this is when you yanked the library). These are now failing again.

Again, please open a new issue. If you’re running on AWS, you’ll either need to choose an image with OpenSSL 1.1.1 support or pin your environment to urllib3<2. This is not an issue with Requests or urllib3 though.

@andrew-pickin-epi Please open a separate issue with a full description, we don’t have much to go off of your comment.

I don’t know what I’m missing, but for testing I can upgrade urllib3

You’re not missing anything. Some people like to over complicate things. Every pre-release I’ve ever done has been tested by too few people to ever ward off the hateful emails. It’s always been more work for no benefit

@nyetwurk there are solutions to this, but I consider it out of scope of this ticket. It’s not specific to requests or urllib3.

Mixing apt and pip installations of packages is nearly guaranteed to put your system into a broken state, and is basically unsupported (and unsupportable). I’d strongly encourage you to pick a singe package manager and learn about its pinning faculties.

Understood, and we avoid pip whenever possible for exactly this reason. Unfortunately, the wealth of python-env tooling and the tendency of python library developers to lean towards pip releases vs sane distro packaging make this a losing proposition. Going 100% to pip is untenable as well, as we depend on LTS releases of distros to provide some protection against regressions. If one could only convince all python developers that they should learn how to create packages for popular distros…

@nateprewitt & @sethmlarson Ansible build using docker module, which in turn uses requests, which it turn … I’m build in AWS amazon linux instances. …

Ansible reports.

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2k-fips  26 Jan 2017. See: https://github.com/urllib3/urllib3/issues/2168
fatal: [sl-monitor]: FAILED! => {"changed": false, "msg": "Failed to import the required Python library (Docker SDK for Python: docker above 5.0.0 (Python >= 3.6) or docker before 5.0.0 (Python 2.7) or docker-py (Python 2.6)) on sl-monitor.epimorphics.net's Python /usr/bin/python3.7. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter, for example via `pip install docker` (Python >= 3.6) or `pip install docker==4.4.4` (Python 2.7) or `pip install docker-py` (Python 2.6). The error was: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2k-fips  26 Jan 2017. See: https://github.com/urllib3/urllib3/issues/2168"}

These are fresh OS installs that of deployment code that’s been working for months but started failing last week, meaning all our deployment had to be doen by hand, but started working approx 2020-05-03 1400 GMT.

I’ve just done a fresh set of clean builds and they are all faling once more.

RequestsDependencyWarning: urllib3 (2.0.1) or chardet (None)/charset_normalizer (3.0.1) doesn’t match a supported version!

We’ll move the pin once we get more data points on any issues in the major version bump.

most people will not be able to get any new data about urllib3 2.0.0 in practice because it just cannot be installed

Just a thought, one way to gain more information on compatibility in the field would be to intially make the urllib3 version bump in a requests release that is marked alpha/beta/pre-release. Pip won’t install pre-releases by default. This way people who want can opt in for testing.

Hi @nateprewitt, thanks for your reply, I appreciate it! 👍

Many, if not most, projects pull urllib3 into their closure through another dependency. Pinning transitive dependencies, while good practice through something like a lockfile, is not widely observed. We have a decade of issues on this project as empirical data on that. Often times, users aren’t even aware they’re using Requests, let alone urllib3. Those are the cases where this can go wrong very easily.

What I’m reading here is that you are concerned about users that (1) do not use urllib3 directly and (2) use urllib3 indirectly and (3) have urllib3 pulled in without a semver guard like “<2” and (4) the code using urllib3 would break with urllib3 2.0.0. Can you confirm that’s the correct reading of your scenario?

Could you name example tickets where I could have a closer look at reports like that? Thanks!