pip: Error with `import pip` in pip 9.0.2

In the long term… well, depends on how many people decide it would be more cost effective to replace you than to fix everything you’ve broken.

I always find this “threat” amusing. It comes with it a fundamental misunderstanding about the amount of effort that is put into something like pip and how much that really costs (and how little people are willing to actually pay for it). If you feel you’re able to do better, then by all means do so, I (and I assume the others) welcome it. A non trivial amount of effort in the past years has been on documenting and designing standards with one of the explicit goals being to make it possible for such a thing to happen.

It’s important to keep a sense of perspective about these things. There are… less than 15? 20? people commenting about how this broke them (though there is always a “tip of the iceberg” problem with these metrics), meanwhile pip 9.0.2 is already the second most used installer in the past week (counting days it hasn’t even existed for) and has downloaded 17 million files from PyPI since it has been released. Looking at just the last day, it’s 80% of the way to overtaking pip 9.0.1 as the most used installer on PyPI.

So while I recognize that this broke things for some people, it is a relatively small number of people doing either unsupported things, or in fairly edge case situations.

IF I can find the time, I may try to cut a 9.0.3, primarily to solve the case that @thet has run into where it is just an auto-importer trying to import things, and they’re not actually attempting to use pip’s internal APIs in any way. If that also ends up resolving the behavior for people using pip’s internal APIs I’m not going to go out of my way to break them specifically, but if it doesn’t I’m also not going to go out of my way to fix them specifically either.

Please do note that it takes many hours to release a 9.0.x at this point (things have diverged in master enough that it requires some finagling to do a release) and that doesn’t count the time required to debug and fix the actual issue. If you want to increase the chance that this happens, doing that debugging and fixing and providing a patch (or a branch off of the 9.0.2 tag in your own fork) is the best way to do so.

+16

@pfmoore

While I understand the frustration here, I’m getting increasingly annoyed by the willingness of people to blame the pip maintainers for things we never promised.

Think of it this way: pip is a command line utility, not a library. The fact that it’s written in Python is irrelevant. That’s the reality of the matter.

The bottom line is it really doesn’t matter what you promised. What matters is what you did and what the resulting situation is. Maybe you thought of it as a command line utility. But you released it as a library. That is the reality of the matter:

You’ve released a library with Feature X. People started using Feature X. Sure, you said “guys, please don’t use Feature X”. But you kept releasing it with Feature X. For years. And so people kept using it. Tens, maybe hundreds, of thousands of projects, big and small, use it now. Suddenly you remove it in a minor update without adequate warning.

What do you think happens next?

In the short term, people will just pin the old version and not unpin it unless there’s a good reason.

In the long term… well, depends on how many people decide it would be more cost effective to replace you than to fix everything you’ve broken.

+16

MikeHart85 on Mar 20, 2018

@RonnyPfannschmidt

that just putting blame because you dont like to be at fault

So you are saying that I’m at fault for using 3rd-party packages that rely on an undocumented feature of pip. Maybe, I am… I should have screened those 3rd-party packages for those errors and submitted an issue or even better a PR.

But let’s face the reality: There are plenty of projects out there that are used in production that are now broken. This is not a question of moral, not a question of whos fault this is. It’s a matter of “can you please fix this and provide a proper warning/deprecation”.

+14

If a third-party package you use breaks because it relies on undocumented, un-guaranteed internal APIs, it seems to me that you should file a bug against that package.

There’s an easy-to-discover line between what parts of pip are and aren’t intended for third-party consumption, and I don’t support trying to force its developers to maintain internal APIs that never should have been consumed by third-party packages. I don’t support treating this as the fault of the pip developers and sending angry users their way. If you want to go be angry at someone, go be angry at any packages you use which relied on internal APIs, and push for their maintainers to fix their own code.

+12

ubernostrum on Mar 20, 2018

Think of it this way: pip is a command line utility, not a library. The fact that it’s written in Python is irrelevant. That’s the reality of the matter.

+11

Hey @dstufft , thanks for chiming in. This thread is getting pretty hot!

First of all, I thank you for your work! I know that maintenance is a endless, thankless task, and I imagine it only gets worse when you manage a project as large and popular as pip!

Now, to the issue at hand: Although twenty maintainers have reported this problem so far, I know that is already affecting thousands of people - some of the projects affected are massive in their own right: Anaconda, OpenStack, SpaCy, Zappa, and have many tens of thousands of users . I know our Slack channel is flooded with discussion about this. (Literally, as I was typing this, a new issue appeared.)

Clearly, there are many important Python projects which need the ability to programmatically inspect their environments. This is a totally reasonable thing to need to be able to do. Furthermore, the documentation never warned against doing this - and still doesn’t! (Click the Docs link from this repo’s README!)

I think the situation so far is:

Given the format of the version string, we all believed that pip maintainers are following semantic versioning
The pip maintainers released a “patch” which introduced a massively breaking change
This change was unannounced and undocumented - and I assume, unexpected and unintended
Now, just calling import pip immediately breaks a program, which is extremely developer-hostile
This is causing major headaches across the Python ecosystem
The documentation does not (and still does not - click on the Docs link in this repo’s README) warn against using pip programmatically.
There was no announcement that import pip would cause this damage in a PATCH update - that announcement came for a version that hasn’t been released yet.
This change wasn’t even mentioned in the Changelog
We don’t want to replace pip or the pip developers! We love you!
…but we don’t want this type of thing to happen again!
…so we do want semver to be followed!
We would really, really like another PATCH which undoes this major breaking change!

The need for a programmatic way to inspect an environment in a pip>=10.0.0 world is a topic for another discussion, but the fact of the matter at hand is that we were never told we shouldn’t use pip programmatically in the pip<=9 world, and there was a massive, unannounced change, and we’d really really like to see it reversed because it is negatively affecting thousands of people.

Thanks again, Rich

+10

Miserlou on Mar 20, 2018

Is there any chance of us getting a 9.0.3 that fixes this - ideally quite soon?

This is already affecting lots of major projects (Anaconda, SpaCy, Zappa), and there are more than 850k+ uses of this on GitHub alone that are now broken by this supposed “patch” version update.

Can you at least revert this massive change in 9.0.3 and then announce to downstreams your intention to change this behavior for a future 10.x.x or at least 9.x.x release?

+10

Miserlou on Mar 19, 2018

As far as I understand just importing pip - without using it - breaks already. This is not a good citizens behavior in Python land. Not supporting using it/ not providing a public API is one thing, just breaking on import another. This affects a lot of auto-inspecting code.

jensens on Mar 20, 2018

you just demonstrated perfectly that turning around of words that so bad - the telling was “dont rely on it it will break” - now it broke and suddenly the project that told you before that it will break is at fault

that just putting blame because you dont like to be at fault

RonnyPfannschmidt on Mar 20, 2018

I can confirm this as well. Just about to file same issue.

nx-jackal on Mar 17, 2018

With no warnings being raised by the code for those using the old way, nothing denoting this as “internal” with starting with _, and this just being a bugfix bump, I actually think it’s quite easy to see this as a sudden, unexpected breakage.

stmcginnis on Mar 19, 2018

@RonnyPfannschmidt This is like saying “We’ve told you a 100 times you are not allowed to drive over the speed limit” and then enforcing the speed limit by replacing all engine-cars with flintstone-cars, so people can not go over the speed limit anymore.

https://pip.pypa.io/en/latest/user_guide/#using-pip-from-your-program

Clearly, there are many important Python projects which need the ability to programmatically inspect their environments. This is a totally reasonable thing to need to be able to do. Furthermore, the documentation never warned against doing this - and still doesn’t! (Click the Docs link from this repo’s README!)

It’s not clear to me why you’d have to warn against this. The use of pip as anything other than a command line tool is completely undocumented. Searching the documentation I don’t see any reference to importing pip at all. It’s like complaining because you linked against some .so used by Chrome and they broke ABI compatibility.

The usual interpretation of SemVer is that it applies to the supported, public interface (in this case, the CLI), not the undocumented internals. Anyone using the internals should be pinning their pip versions anyway, since they are subject to arbitrary change.

pganssle on Mar 20, 2018

@ all, accusing the maintainers for broken code doesn’t help with solving the problem.

thet on Mar 20, 2018

@dstufft I can confirm, the fix in #5101 solves the problem for me.

thet on Mar 21, 2018

There wasn’t - and there still isn’t - anything in the documentation saying not to do this.

@pfmoore

If @dstufft wants to make an emergency 9.0.3 release, I’m fine with that. But it will only be a very short term benefit, and I’m a little disappointed that people haven’t already moved away from import pip. The people hit by this issue will still need to prepare for pip 10, and should understand that simply switching to import pip._internal is not something we will support or recommend.

I can not believe that this is a statement by the owner of this repository… You literally just said “f*ck semantic versioning” and “who needs a deprecation policy anyway”.

pip is Python’s package manager. Python is a programming language. People need to inspect their environments programmatically. 1+1=2.

It was perfectly reasonable for people to assume that this was the correct way to do it. There wasn’t - and there still isn’t - anything in the documentation saying not to do this. As a reminder, there are over 700,000 applications which came to this same conclusion! Also - there isn’t any other way to do it!

Just because something isn’t documented in ReadTheDocs doesn’t mean that it is forbidden - we all encounter and use projects every day that simply provide code this way.

If anything, the fact that pip even has a command line interface at all seems to be a good indication that it can be used as a library, since it already has a Python client!

Furthermore, we’re not talking about private APIs namespaced with a _, as is convention, or even any specific public method, we’re only talking about just calling import which causes this catastrophic failure.

Miserlou on Mar 20, 2018

I always find this “threat” amusing.

It’s not a threat at all, apologies if it came across that way.

It’s an observation about something that happens all too often in the software world. People don’t work with your promises or intentions. People work with the product you deliver. If your product has a feature that many people come to depend on, you can no longer just yank it out from under their feet. If you do, they start seeing your product as broken. If the only explanation you provide is “well, we never promised this anyway” then they start seeing your unwillingness to acknowledge their concerns as a liability.

The more this happens, the bigger the desire and demand for an alternative. Sure, people underestimate how much work that takes. But that only makes them more, not less, likely to start working on one. Once that ball is rolling, it has a life of its own.

Alternatives aren’t always a good thing. They tend to make things messy. Personally, I’d prefer a single package manager. For everything. But I’ll settle for a single package manager for Python.

When people report you’ve broken a feature, intended or otherwise, consider briefly explaining why you had to break it, or why keeping it wasn’t viable, rather than dismissing concerns with “this wasn’t supported all along”.

MikeHart85 on Mar 20, 2018

First of all, sorry if I accused the maintainers of the repo for faulty code. Any accusation were made out of frustration, and, if at all, point to the fact that IMHO the 9.0.2 release is by the definition of semver a patch release (though @pfmoore made it clear that pip is not necessarily using semver - which is another issue for another day I guess).

@MikeHart85

In the short term, people will just pin the old version and not unpin it unless there’s a good reason. In the long term… well, depends on how many people decide it would be more cost effective to replace you than to fix everything you’ve broken.

Pretty much. I mean look at JavaScripts package managers: npm, bower, yarn, whatever is released next ~year~ week

No, but accusing the pip maintainers rather than those projects is not acceptable. We’re trying to help you but can’t be held responsible for what those projects do. Air your grievances with them.

I understand that my (our?) grievances within this issue might be misdirected. Though you have to admit that it is really hard to convince anyone that this is the fault of 3rd-party-maintainers.

We tried to provide a warning. We sent announcements out months ago, we added docs explaining the situation

Was this for pip 9.0.2 or for pip 10? If it was for pip 9.0.2, I would find it nice if there was a note in the Changelog. Anyway, thank you for being very proactive in the development of pip 10 and sending out announcements about not using import pip, that’s good! Keep it up!

Adding deprecations is far from being as easy as you suggest, as pip itself would spew those warnings (or there’d be a way for pip to switch them off, which others would just copy - we’re already hearing of people planning to just import pip._internal, so even that change won’t stop them disappointed).

I don’t think this would be too hard… You already have this in the __init__.py:

if __name__ == '__main__':
    sys.exit(main())

How about just adding an else?

if __name__ == '__main__':
    sys.exit(main())
else:
    logger.warning("You are importing PIP as a Python library. This behaviour is deprecated and should no longer be used. We will break the API with version 10.0!!!111eleven Please see https://pip.readthedocs.org/some_link_where_you_explain_this.html")

edit: You can even raise an exception here if you want to be “strict” about it, and add similar statements to the submodules.

The fact that people will work around this, is something else. If you want to hack a library, you can always reverse engineer it. You can always find a way around it. But you should not consider all the ways around it within your design decisions.

https://github.com/ContinuumIO/anaconda-issues/issues/8911

Pip 10 is due out in about a month. As I understand it, the issue is with code that uses import pip to access internal functionality of pip. We’ve never supported such usage officially, and we have explicitly documented that lack of support for some time now (albeit only in the “latest” version of the docs at https://pip.pypa.io/en/latest/user_guide/#using-pip-from-your-program, because we haven’t had a new stable release since that doc section was added). We also announced a reorganisation of the internals to make it clear that use of internal APIs is unsupported, last October (see https://mail.python.org/pipermail/distutils-sig/2017-October/031642.html). That change, which is in pip 10, will break any such usage regardless of what pip a possible pip 9.0.3 release would do. So it’s hard to see this as a sudden, unexpected breakage.

If @dstufft wants to make an emergency 9.0.3 release, I’m fine with that. But it will only be a very short term benefit, and I’m a little disappointed that people haven’t already moved away from import pip. The people hit by this issue will still need to prepare for pip 10, and should understand that simply switching to import pip._internal is not something we will support or recommend.

Proposals for reintroducing some level of API support are certainly an option (see #5080 for example) but at this stage, it’s too late for any such changes to make it into pip 10.

pfmoore on Mar 19, 2018

This also broke Anaconda and I came up with the same solution as @Miserlou:

Errors are being reported across a lot of projects.

FlipperPA on Mar 18, 2018

@drunkwcodes Just to be clear, pip.main() is the easiest usage to replace - you simply need to use subprocess.run([sys.executable, '-m', 'pip', <rest of args>]).

Also, the reason WheelFile.install() isn’t promoted is that the wheel project have also stated that they do not provide a user-visible API - we originally mentioned wheel in the pip docs, but removed it at their request. The wheel PEP is pretty clear about how you install a wheel, though - it’s not hard to implement in a 3rd party module.

pfmoore on Mar 23, 2018

unexpected breaks in backward compatibility

Please indicate to me where import pip was documented to be a stable API that falls under the backwards compatibility promises we make? I would like to make sure that I remove such documentation since we emphatically do not support that use.

Unless you’re of the opinion that absolutely nothing can break, even unsupported, untested uses. In which case you probably want to pin == because it is entirely unknowable what all you may be using at that point, and every change potentially becomes a breaking change for someone (obligatory xkcd).

Think of it this way: pip is a command line utility, not a library. The fact that it’s written in Python is irrelevant. That’s the reality of the matter.

Fact: It can not be imported, if imported even by auto-inspecting code for some reason: all breaks. I suppose this happens in @thet’s case. Conclusio: pip must isolate itself from the global or current virtualenv/venv Python namespace it installs packages into. That way it does not pollute it and not even import by accident ever happens.

jensens on Mar 20, 2018

It might be worth asking the distlib project to clarify that distinction a little better.

Already done that 😃 See: https://bitbucket.org/pypa/distlib/issues/100/clarify-project-positioning-and-status

tuukkamustonen on Mar 26, 2018

Ah, that’s a different “packaging” - it was the proposed stdlib module that was intended to replace distutils. It’s completely different from the PyPI packaging project. It might be worth asking the distlib project to clarify that distinction a little better.

pfmoore on Mar 24, 2018

@drunkwcodes I suspect that what you propose is already implemented in the existing packages that the pip documentation mentions, packaging, setuptools, and distlib. As far as I can tell from this thread, there’s no problem with a gap in functionality, but some people have problems with tools that try to import and inspect every module, and treat import failures as fatal errors.

It seems to me that such tools could work around this problem by wrapping import statements in a try/except block, but that also seems like a questionable precedent to set. Given the release of pip 9.0.3, though, I think it’s probably not worth the time to debate the import issue unless the problem happens again in pip 10. Anyway, as long as the maintainers don’t go out of their way to make import pip fail, or summarily reject patches that fix such failures, there would be common ground to stand on.

sruggier on Mar 22, 2018

Hooray! Thanks for this!

I’d also really love a deprecation warning, and, more importantly - proper instruction on how to programmatically inspect python environments in a pip10 world! Clearly there is a need for that, given that over 700,000 applications were affected by this bug.

Miserlou on Mar 22, 2018

Thanks for the fast resolution, from someone who never wrote import pip in their life but was a consumer of a package that did. After reading through this thread sounds like lots of apps have imported pip, unknowingly against best practices, and lots of dev and users are potentially effected by v9.0.2 and v10.

I second/third/Nth a proper deprecation warning being added to make things smoother. maybe even a pypi.python.org post?

diginc on Mar 22, 2018

Ok, last thing I swear-- pip 10 already moved all relevant code to pip._internal (we didn’t use _pip because that would break python -m pip, which is supported).

I believe that this discussion is unlikely to be productive nor is there really anything additional to be said. There is:

An adequate description of the problem.
Possible work arounds that someone could employ if they run into it.
Rationale for why we don’t consider this a breaking change as per our backwards compatibility guidelines.
A statement that I will attempt to find time to fix it (though no promises about it).
A Call to Action that one can be done if someone affected by this wants to make it more likely that a 9.0.3 exists with the fix.

At this point the discussion serves no real purpose except to further frustrate everyone involved, so I am bowing out of this issue for now. This issue will be closed either upon the release of 10.0, or 9.0.3. If someone does put effort into the Call To Action, they are welcome to either open a pull request, or submit a diff on this issue.

If you click on the “Docs” link of this very repo, you don’t get to that page. Nobody is referencing latest. Your own links to documentation don’t link to latest. There is no way to get to that page. Everything goes to stable, which does not include that section.

Miserlou on Mar 20, 2018

In response to this comment: Whether or not pip adheres to semver, or promises to adhere to it, the use of a major.minor.micro versioning scheme still carries an implicit promise of some kind of restraint around micro releases. If a compatible release pin like ~= 9.0.1 isn’t enough to shield users from unexpected breaks in backward compatibility, the only safe alternative left is plain version matching. If there’s no intention to support anything more than version matching, then pip might as well switch to a Chrome-style version scheme that has only one monotonically increasing component.

Edit: I see that @dstufft already proposed moving in this direction by adopting date-based versions. I think that would be an unfortunate piece of collateral damage to result from this incident.

So yes, as users of a volunteer driven software project, we need to balance our expectations with an appreciation for the nature of the user/maintainer relationship. It’s also clear that this release wasn’t intended to make import pip suddenly start to fail. On the other hand, though, I think it’s reasonable for people to be frustrated about this sort of regression happening in a micro release, and releasing a 9.0.3 version that fixes the issue seems like a reasonable remedy.

As an aside, I can reproduce this in a virtualenv (2 or 3, it doesn’t matter) with the following steps:

virtualenv -p python3 /tmp/pip
/tmp/pip/bin/pip install -e path/to/clone/of/pip/9.0.2
/tmp/pip/bin/pip install requests
/tmp/pip/bin/python

Then, within the python shell:

>>> import requests
>>> import pip

If requests hasn’t been imported already, then import pip succeeds.

sruggier on Mar 20, 2018

@Miserlou

Given the format of the version string, we all believed that pip maintainers are following semantic versioning

As someone who is strongly opposed to semver, can you let me know what format my version strings should be in to eliminate this confusion? “Three dotted numbers” does not imply “strictly follows semver” to me, so this assumption comes as a surprise.

habnabit on Mar 20, 2018

First of all, sorry if I accused the maintainers of the repo for faulty code. Any accusation were made out of frustration

Thank you. Frustration is high here, and I understand that.

How about just adding an else?

Good idea - I wish we’d thought of this some time ago. Of course, there’s no real point now, as the internal namespace will be reorganised in pip 10, so it’s too late. I’ll certainly remember this trick for the future.

Was this for pip 9.0.2 or for pip 10?

For 10.0. There was no original intention to release a 9.0.2, we only did so as an emergency release so that users of Mac OS wouldn’t lose all access to PyPI when the TLS changes come into effect.

@dstufft has responded here much more fully, so I think this is as resolved as it’s likely to be for now.

Same issue here with pip 9.0.2 in a gitlab-ci with docker python 3.4: KeyError

  File "/usr/local/lib/python3.4/site-packages/pip/__init__.py", line 45, in <module>
    from pip.vcs import git, mercurial, subversion, bazaar  # noqa
  File "/usr/local/lib/python3.4/site-packages/pip/vcs/mercurial.py", line 9, in <module>
    from pip.download import path_to_url
  File "/usr/local/lib/python3.4/site-packages/pip/download.py", line 40, in <module>
    from pip._vendor import requests, six
  File "/usr/local/lib/python3.4/site-packages/pip/_vendor/requests/__init__.py", line 98, in <module>
    from . import packages
  File "/usr/local/lib/python3.4/site-packages/pip/_vendor/requests/packages.py", line 12, in <module>
    sys.modules['pip._vendor.requests.packages.' + mod] = sys.modules["pip._vendor." + mod]
KeyError: 'pip._vendor.urllib3.contrib'