requests: requests 2.14.0 cannot be installed using pip < 8.1.2

Ok, so right now with 56% of our downloaders already safe and most of the rest one pip install -U pip away from being ok, I’m pretty tempted to say that we should consider toughing this out. It’s going to annoy a whole bunch of people, but if we can drag the ecosystem forward a bit to the point that people can actually meaningfully use environment markers for conditional dependencies that would be one hell of a public service.

Ok folks I just shipped v2.14.1, which should vastly widen the number of supported pip and setuptools versions. Please all try using the new release and see how it works for you.

the rest are one pip install -U pip away from being ok

Just to point out, they are also one pip install requests==2.13.0 away from being ok. Seriously, this is going to impact a lot of people who use Travis. Before today I had never even heard of requests, but all of a sudden every unit test for our project is failing because codecov depends on requests. I agree that Travis should install a newer version of pip, but that doesn’t mean this is the right way to do it. I would prefer rolling back whatever change requires environment markers until Travis updates pip.

I don’t want to be dismissive about concerns, as this did break some things, but if you’re hitting this, it’s because you’re running the very latest version of requests, released like an hour ago. It doesn’t seem totally out of line to say that if you’re running the very newest version of a package, you might also need a new version of something else.

The desire for a “stable” OS (e.g. RHEL, CentOS, Ubuntu LTS), with all the latest packages and all the oldest tools is a contradiction in terms; and pretending like this is a reasonable practice is a massive maintenance burden for OSS developers.

when you have a userbase as large as ours it’s basically impossible to avoid breaking someone

Is “less than half of our users are affected” the bar? When you have a lot of users, half your users is still a lot of users.

I think everyone should always upgrade to the latest pip! Whether enough people have done that for requests to consider it reasonable to depend on it or not, I dunno.

This is a common feature of shipping Requests releases: when you have a userbase as large as ours it’s basically impossible to avoid breaking someone. This was a little bit bigger than I expected, I admit, but still. 😁

It’s certainly saddening to see “base” packages like pip and requests are taking more of an approach of “well, we’re using the latest cut of master, so everyone else needs to now too”. I guess lack of stability and endless work for library maintainers who try and do proper CI are the way of the future in the Python ecosystem?

@martin-langhoff Is it a stable environment if you’re pip installing the latest Requests? RHEL ships with a copy of Requests in their repositories that they use for system tools, and you really should not be shadowing it with one from pip. On the other hand, if you’re using a virtualenv (as you should be in this case), you can just pip install -U pip in the virtualenv to resolve the issue.

@Lukasa Completely understand. I haven’t dug deep into the change but if the pip version could be detected and a better error message emitted it would go a long way to ameliorating your users.

@Lukasa, 8.1.1 seems to likely be coming from Ubuntu Xenial too. It seems reasonable for people to be upgrading a version of pip that in some cases is more than 3 years old. I’m on board with waiting this one out as well with the current information. We can’t support legacy versions of pip forever.

Ok, so that means it works with a version of pip that’s a year old. I think I’m ok with this, though naturally the rest of the dev team should express opinions here. Generally, I’m a bit loathe to be tied down by older pips: generally speaking for most people it’s easy to upgrade pip, and if it’s hard to upgrade pip they’re usually using a distro and so should use a virtualenv to install Requests, at which point they should just install a new pip inside the virtualenv.

Any thoughts on that team? @nateprewitt @sigmavirus24 @kennethreitz?

Well done on finding a mitigation that improves compatibility with earlier pip versions.

Longer term, it likely makes sense to use this case as an example for why wrapping a virtual environment in an immutable deployment artifact (e.g. container, deb, RPM) or locking all your transitive Python level dependencies (e.g. with pip-tools or Pipfile.lock) is a good idea - even the most diligent of developers can accidentally introduce incompatibilities with older distros, and tools like pipenv and pyup.io help ensure your worst case scenario in such instances is a failure in pre-merge CI for a dependency update rather than a broken deployment pipeline or production environment.

Thanks to @Lukasa for handling this so well. It’s not always easy to deal with the users, especially when they’re all on your GitHub issue at once, but it’s clear the requests folks care 😄 .

Stable environments – say, RHEL6/CentOS6 is where a ton of software gets deployed. If you are going to break them, it better be for a good reason, no?

Do you need a newer pip hard enough to push work on thousands of other people? Or can this be solved with a quick one liner?

(This is not an abstract question, I’ve done releng for many FOSS projects, over many years, and I releng for a project where we just got hit by this. You can google me up if you want to.)

Pain is a part of software engineering, if it was easy no one would pay us. I think this accidental “breakage” is actually a benefit to the ecosystem as a whole.

For people interested in a possible mitigation, see #4007.

@Lukasa This might help: https://gsnedders.github.io/python-marker-test/results.html

As to whether we need it, it depends on whose needs we consider. The reality is that right now, if you install the socks dependencies on Windows for a version of Requests before 2.14 you get a broken install, because we need the win_inet_pton package. Naturally, this package is not needed on any other system, and may indeed misbehave quite badly on those systems.

Working around this requires forcing the execution of setup.py, which means no longer distributing wheels, which means forcing client code to build them and cache them locally. Not so good. There’s a balance here: either we leave Windows users broken, or we rely on Windows users having no wheel support (not likely, Windows users often have up-to-date pips), or we break people running old tools. There is no trivial easy answer here, or we’d have taken it.

If you want a CI that doesn’t fail on arbitrary upstream changes, pin your dependencies. pip, setuptools etc. are build-time dependencies.

So instead of pip install -U pip, consider pip install pip==X.Y.Z. It’s your choice which of the two you prefer. Maybe you want to have CI jobs for both.

IMHO we should spread awareness of this rather than supporting super-old stuff forever.

I’m pretty sure the justifcation is they’re Rubyists and don’t think much about pip 😃

I just found this issue because the default pip on Travis CI is 6.x (easily fixed with the addition of pip install --upgrade pip but that’ll be in every project until they update the base image)

@cjw296 I’m sorry you think that’s what we’re doing. I think this issue represents a reasoned and reasonable discussion of the pros and cons of this decision, with no definitive position being taken yet, but that’s just me.

So okay I need to upgrade setuptools as well on RHEL 7 to both run our build and run sphinx-build. pip install --upgrade pip setuptools produces this output:

  Running setup.py egg_info for package setuptools
    Traceback (most recent call last):
      File "<string>", line 3, in <module>
      File "setuptools/__init__.py", line 10, in <module>
        from six.moves import filter, map
    ImportError: No module named six.moves
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 3, in <module>

  File "setuptools/__init__.py", line 10, in <module>

    from six.moves import filter, map

ImportError: No module named six.moves

So now I have to upgrade six, setuptools, and pip, like so:

pip install --upgrade pip six
pip install --upgrade setuptools

@mhils If we can get an idea of what pips it’ll work for we can make a data-driven decision about how helpful it is. If it brings support up 10% or more I’m probably sold on that being a good approach.

I might be mistaken, but we are doing the following in @mitmproxy for a while now without any complaints:

    extras_require={
        ':sys_platform == "win32"': [
            "pydivert>=2.0.3, <2.1",
        ],

I think this also works for older pip versions, but I unfortunately cannot check that on mobile right now. This might be acceptable for requests as well?

@blamarvt Yeah, that’d be really nice. Sadly, we’ve been publishing wheels for a long time, and wheels are preferred by pip and don’t run any code, so anyone who just types pip install requests and can use wheels won’t get that handy error message. 😞

@tgamblin Right now we don’t have a defined bar. If we did, this would be a much shorter conversation: either it would meet it, or it would not. Part of the reason we’re having this discussion is to try to kick around the idea of how much pain we’re willing to inflict for (IMO) positive ecosystem gain. As I’ve said before, we are not committed to keeping this.

The flip side to this is that if we can bring even 20% of our users forward to a much newer pip, that opens up an enormous chunk of the ecosystem to being able to use these newer Python packaging tools. Python packaging is such a commonly complained about topic, and one reason is that an enormous amount of the newer nicer features cannot actually be used because we have to support older pips. Putting on some ecosystem pressure to upgrade might be a good thing. But it might not. Hence: this thread.

@adamjstewart Yeah, that’s absolutely true, and some number of projects will probably do that. All we can do is make a decision about at what point we decide to stop supporting the older releases of pip: at what level of usage do we give up on them?

I should note that Travis-CI is included in the above numbers, which means even with Travis-CI less than half of our downloads are currently affected. I don’t consider that number to be at a level where I want to immediately revert this.

I’m still keeping this open for discussion: I want the community and the other maintainers to weigh in. @nateprewitt and I have had our say, two maintainers have yet to get involved, and there are many more community folk who should feel free to drop in and tell us what they think. All I am saying is that right now I am open to leaving this as-is.

As a fellow maintainer of OSS projects who wants to use environment markers, ❤️. It seems like we should also engage with the travis folks to fix this, as that’ll reduce a lot of the pain.

This also broke our CI a minute ago: so is Travis gonna fix their env or are projects supposed to do an update within their Travis scripts?

So I wasn’t around for the change in question, but reading the thread there are a few things worth noting:

1. Some people are trying to be helpful but antagonizing others who are experiencing pain and those people need to stop.

2. This change was made as a best effort approach to unbreak users without breaking far more users. It had the unintended consequence of educating people about how ancient their versions of pip, setuptools, etc. are.

3. The error here fundamentally stems from old versions of critical infrastructure. Yes, @martin-langhoff, some of our consumers are unknowingly consuming requests through other packages that expect people to be using reasonably recent (not the latest) versions of software. If anything, this further highlights the fact that most people pay little attention to their dependencies.

4. Making a data driven decision for this issue is hard with people repeating themselves trying to be correct. There is no one right answer. There is just a least bad option in this case. I’d encourage people to take a walk and get away from their keyboards. Tensions are unnecessarily high. There are several workarounds and #4007 is a potential permanent fix.

For end-users whose projects have been surprise-broken by this change - I would strongly recommend pinning your dependencies via --require-hashes mode (which warns about unpinned sub-dependencies amongst other things): https://pip.pypa.io/en/stable/user_guide/#ensuring-repeatability https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode

Doing so used to add hassle in the form of having to continually update requirements files by hand, but there are now tools out that support automatically updating the pinned versions even when they have hashes (eg https://pyup.io).

For a project I work on there was no surprise breakage, just a bot-opened PR with a failing Travis run.

(Alternatively check out https://github.com/kennethreitz/pipenv if you don’t need bot support.)

There’s a balance here: either we leave Windows users broken, or we rely on Windows users having no wheel support (not likely, Windows users often have up-to-date pips), or we break people running old tools. There is no trivial easy answer here, or we’d have taken it.

This is a far better justification than “only 46% of our users are affected” or “I think this accidental “breakage” is actually a benefit to the ecosystem as a whole.”

I will say I suspect this would be a shorter thread if the error message was better. The fix for this is easy (update pip!). But the message you get when you hit the problem doesn’t even remotely imply that’s what you should do. Per pypa/pip#4469:

Collecting requests>=2.7.9 (from codecov)
  Downloading requests-2.14.0-py2.py3-none-any.whl (559kB)
    100% |################################| 561kB 801kB/s 
  Exception:
  Traceback (most recent call last):
    File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pip/basecommand.py", line 232, in main
      status = self.run(options, args)
    File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pip/commands/install.py", line 339, in run
      requirement_set.prepare_files(finder)
    File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pip/req/req_set.py", line 436, in prepare_files
      req_to_install.extras):
    File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2496, in requires
      dm = self._dep_map
    File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2697, in _dep_map
      self.__dep_map = self._compute_dependencies()
    File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2730, in _compute_dependencies
      common = frozenset(reqs_for_extra(None))
    File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2727, in reqs_for_extra
      if req.marker_fn(override={'extra':extra}):
    File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/pip/_vendor/_markerlib/markers.py", line 113, in marker_fn
      return eval(compiled_marker, environment)
    File "<environment marker>", line 1, in <module>
  NameError: name 'platform_system' is not defined

It would be awesome if that just said “your pip is too old”. I’m not intimately familiar with the code here, so I’m not sure whether requests could print such a message or whether pip would have to do it (in which case you can’t really backport it to an old pip).

MORE DATA!

If we restrict ourselves to the two most recent Requests releases, to try to exclude people who aren’t actually tracking requests releases, the data looks like this:

In this variation the versions of pip that support the marker account for 3,040,360 downloads of a total of 5,076,901, or a total of 59.8%. So some portion of our older downloads are downloading older Requests versions, and they will continue to be unaffected.

@oberstet Very much agreed, I’d prefer a world where Travis has some kind of rolling approach to pip versions, and I’m happy to work with them to make that possible if they need outside help. In the meantime, though, we shall do what we can in their absence! 😁