requests: Inconsistent urllib3 version requirements between setup.py and __init__.py

I totally understand the motivation of this design decision to generate warnings on untested urllib3 versions… but this behavior is definitely breaking in some cases (and not just a warning).

Simple test case; create a distribution with an entry_point. For example:

#!/usr/bin/env python
from setuptools import find_packages, setup

setup(
    name="test",
    version="0.1.0",
    packages=find_packages(),
    install_requires=[
        "requests==2.18.4",
        "urllib3==1.23",
    ],
    entry_points={
        "example": [
            # it doesn't matter if test.foo exists
            "foo = test.foo",
        ],
    },
)

Create a test program that resolves this entry point (we don’t even have to load it):

#!/usr/bin/env python
from pkg_resources import iter_entry_points


for entry_point in iter_entry_points(group="example"):
    entry_point.require()

Then install the distribution:

python3 -m venv test
source test/bin/activate
pip install -U -e

This works fine. Even though we have version conflicts, pip lets us continue.

But then try to ask setuptools to resolve the entry point:

> python3 ./test.py
Traceback (most recent call last):
  File "./test.py", line 6, in <module>
    entry_point.require()
  File "/private/tmp/test/test/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2346, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/private/tmp/test/test/lib/python3.6/site-packages/pkg_resources/__init__.py", line 783, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (urllib3 1.23 (/private/tmp/test/test/lib/python3.6/site-packages), Requirement.parse('urllib3<1.23,>=1.21.1'), {'requests'})

TL/DR; putting an upper bound on urllib3 versions is breaking for any application that uses entry points in this way and “happens” to upgrade dependencies. That might be your intention, but this behavior definitely goes beyond “warnings” and definitely breaks some workflows (e.g. ours).

Here’s how I solved it yesterday:

pip install --upgrade "urllib3==1.22" awscli awsebcli

Hey everyone, to be clear this isn’t a bug but rather a specific design decision. We cannot guarantee support for new releases of urllib3 before they’ve been tested with the Requests code base. In the event of a new release we haven’t tested you will see a warning message informing you about the issue.

We are currently slated to release Requests 2.19 in the next week or two which will move the warning pin. Our standard procedure is to wait a week or so and see if any major bugs are raised in urllib3’s new release. Things will continue to work fine with 1.22 in the meantime.

@jessemyers, yeah, this is the first time in almost a year, and the third time ever we’ve had a release with this process. We previously owned both the release of urllib3 and Requests so it was a bit easier to sync these things. We’ll take your suggestions into account if we review the policy with recent changes. Thanks again for clarifying your issues, it’s appreciated feedback!

Seems like this issue broke installation of awscli

Also has been experiencing this with awsebcli on my CircleCI instances since 2-3 weeks ago. Upgrading pip and requests before installing awsebcli solved the issue.

Here’s what my CI script was doing before:

sudo apt-get update && sudo apt-get install -y python-pip python-dev build-essential
pip install awsebcli --user

Here’s what it’s doing now:

sudo apt-get update && sudo apt-get install -y python-pip python-dev build-essential
export PATH=~/.local/bin:$PATH
sudo pip install --upgrade pip
sudo pip install --upgrade requests
pip install awsebcli --upgrade --user

Seems like this issue broke installation of awscli

Alright, 2.19.0 is out the door! Closing this out since it should be resolved now.