pip: editable cannot be installed when requiring hashes

This is completely baffling and bizarre. Why should one requirement having a hash affect any other line in the requirements file? What if you only want hashes on some requirements?

I’m aware of the fact that hash compare doesn’t work on editables 😉 (Of course, pip could create a hash over all checked out files. But that is not the topic here.)

I have a requirements.txt with some editables and with normal PyPi packages. The hashes are only attached to the normal package names.

I would assume that pip check the hashes for the normal PyPi downloads. But pip simply aborts with the message from above and doesn’t do anything more.

So: I can’t use the hash feature for normal PyPi packages if i have one or more editables in requirements.txt ?!?

This issue appear to be invalid, since there’s no reasonable improvement to the error message. Out of curiosity, why did you not understand the error message and what do you think it should be?

Just my 2 cents as to the behavior that I would personally naturally expect:

• Default: All deps are valid, hash or not, but any that specify hashes have them enforced. (Unsurprising behavior: use all information present, don’t make additional assumptions.)
• --require-hashes: The input is invalid and pip aborts immediately unless all deps specify hashes. (Unsurprising behavior: all deps “require hashes” just like the option says.)
• --ignore-hashes: All deps are valid, hash or not, and any hashes that are present are not enforced. (Unsurprising behavior: pip “ignores hashes” just like the option says.)

I’m not trying to weigh in on anything else that’s being discussed, but I agree with @carlmjohnson that having --hash on a single line in a requirements.txt fundamentally ‘coupled’ to every other line in that same file both violates the principle of least surprise and in general feels like incorrect software design. In fact, it’s basically impossible to imagine an implementation of this that isn’t more complex than the straightforward implementation where each line of a requirements.txt handles its flags individually.

FWIW, I have written a workaround wrapper for this that works well for our internal use case but likely could be improved if other people wanted to start using it and trying it out. But it takes ‘advantage’ of the odd situation by basically attempting a --no-deps install of every requirement individually, assuming that dependency resolution has already been accomplished separately, so it probably isn’t a truly general purpose solution.

Could we add something like --dont-require-hashes? or disable automatic enabling hash checking for every direct dependency and force use of --dont-require-hashes if someones desires it?

Ah, I think what you’ll want to do is break up the requirements.txt into 2 and invoke pip 2 times - once without the editable requirements in hash-checking mode and then once with editable requirements.

I’m actually not sure which ones you’ll want to install first. I imagine the editables with --no-deps followed by the hash checked dependencies via requirements.txt would be a good process.

On Fri, 26 Jan 2018, 21:11 Jens Diemer, notifications@github.com wrote:

I’m aware of the fact that hash compare doesn’t work on editables 😉 (Of course, pip could create a hash over all checked out files. But that is not the topic here.)

I have a requirements.txt with some editables and with normal PyPi packages. The hashes are only attached to the normal package names.

I would assume that pip check the hashes for the normal PyPi downloads. But pip simply aborts with the message from above and doesn’t do anything more.

So: I can’t use the hash feature for normal PyPi packages if i have one or more editables in requirements.txt ?!?

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/pypa/pip/issues/4995#issuecomment-360819310, or mute the thread https://github.com/notifications/unsubscribe-auth/ADH7SRAQOKUuGjajlz4dVACil63LnHwJks5tOfI9gaJpZM4RshLV .

I’m going to go ahead and hide all the comments w.r.t. the “this is crazy” discussion, since IMO they’re not very useful for someone who’s trying to actually solve the problem, and have negative effects on any contributor that’s considering contributing here. If you disagree with this move, please file a new issue for further discussion.

I see the purpose of this issue as a place to discuss whether changes are warranted, and if so, which ones to make. So, let me respond in slightly more depth to https://github.com/pypa/pip/issues/4995#issuecomment-593773239:

Personally I would prefer to keep the current default; editable installs are by definition cannot be pinned, so it does not make sense to ask for it when hashes are required.

Hashes being required for all dependencies is not an explicit choice that users are making. Users are making the choice to include some hashes. Having some hashes has value even if not everything has a hash. For example, I may have a private repo that I install as editable but still want to have pinned hashes for external dependencies. There are many other such scenarios.

Making it a special case feels to me like creating a loophole.

It’s a loophole to a problem created by an unexpected pip behavior. The root problem is one hash making all hashes required. Either change pip to always require hashes and opt out with a flag (seems unlikely because it would be breaking change for the ecosystem), or do the simpler change of adding a flag to require hashes. Editable repos don’t come into it one way or the other. If I had passed a flag like “–require-hashes” I would not have been surprised, done a search, and found this issue. I found the issue because the behavior is apparently an unintended side-effect of requiring one hash.

The two-step solution not working for tools sounds like a toolchain problem to me; tools should provide a way to run multiple commands in the installation phase, instead of pip (or any other installer command) needing to cram steps into one command.

This strikes me as side-stepping of pip’s core functionality. Pip is a tool to install Python dependencies. It need not do everything related to dependencies, but needing to split into multiple files or use a separate tool so as not to trigger an unexpected, undocumented, and undesirable behavior is just passing off responsibility for pip’s core value to other tools.

I have been thinking about this again, as I came across the pip install -e . -c constraints.txt use case, where constraints.txt has requirements with hashes. This cannot be resolved by splitting into two pip install calls while keeping the desired semantics.

So I’m wondering we could accept requirements provided as local directories in --require-hashes mode. I am under the impression (from skimming this thread again) that, combined with #11968, this could resolve most use cases.

Of course that means that --require-hashes will not detect when a user adds a local directory to requirements anymore, but I’m not sure that was ever a goal of --require-hashes.

The raison d’etre of editable installs is to make the installed code editable. If all you want is to install something from private locations, you can just install from path without the editable flag. This issue is specifically about editable installs, and your latter reason thus does not apply.

I’ve just reviewed all of the comments from pip developers (and experienced contributors to pip) in this thread. None of them are “rationalisations” or any sort of objection. They explain the current behaviour and discuss options for addressing the requirement, nothing more.

I’ve added the label “deferred till PR”, which says that further discussion should wait till there’s a PR. I personally think that the discussion was perfectly fine, and would have been happy to continue here, but if you insist on characterising the contributions of the pip developers and experienced contributors as obstructive, then I suggest it would be better to wait for a PR. Hopefully tempers will have cooled by then (and it’s not like anything will happen until there’s a PR anyway).

Why respond to my comment at all?

Misguidedly, I thought you might appreciate an explanation. My bad.

having --hash on a single line in a requirements.txt fundamentally ‘coupled’ to every other line in that same file both violates the principle of least surprise and in general feels like incorrect software design.

I believe that it was a deliberate decision, based on the wishes of the people who asked for hash checking mode in the first place - the idea being to avoid a security loophole where you are hash-checking everything, but accidentally add a dependency without a hash, leaving your application insecure when you think it’s secure. I do agree that explicitly specifying --require-hashes seems like a reasonable thing in that case, though.

My memory is very vague on the details here, and I could easily be completely wrong. I encourage anyone who wants to argue for dropping the implicit enabling of --require-hashes whenever a hash is specified, to do some research to confirm (or otherwise) the original intent here. (And please do add a link here, to help others - including me - confirm the details).

I’m sorry if my tone was off, but I really feel like we’re being gaslit. Look at the comments and emoji reactions from people not on the PyPA team. They all agree: this would be a nice feature and it’s annoying not have by default. If it can’t happen for priority reasons and you need someone else to find the time do a patch, again that’s fine, that’s life. But why make up these rationalizations as though we’re wrong for wanting an obvious feature? Why respond to my comment at all? Delete it as off topic if you thought I was too harsh or just leave it with no reply. As I read it, it just comes out to a long way of saying “submit the patch, otherwise forget it.”

I personally think editable installs must continue to be rejected when --require-hashes is used, because the first thing pip does with these is update the src directory which may have local changes.

I’m in favor of https://github.com/pypa/pip/issues/6469, however.

@Qu4tro assuming #6469 gets traction, a third approach will be to try to understand why pip wrappers such as poetry or pip-tools use editable installs. Since I’m working towards implementing #609, I’m curious to understand if pip-tools and poetry would benefit from it and/or PEP 610. My general goal with this work is to reduce the need of using --editable for working with VCS requirements.

@pradyunsg if possible I would like your opinion on this subject. (Disclaimer: I’m moving things to Poetry and tbh this is my vision as an end-user for Poetry)

From my reading there are two main schools of thoughts here:

• Allow pip to install editable packages alongside hashed packages.

  • Pro: Most friction-less approach for users
  • Pro: Doesn’t require changes in pip wrappers. (i.e. Poetry).
  • Cons: Require changes in pip

• Use a separate requirements.txt to install editable packages.

  • Pro: Doesn’t require changes in pip
  • Pro: Splits “safe” packages, from “unsafe” packages.
  • Cons: Wrappers will be required to output multiple requirements.txt

I might be missing others, possibly important, concerns.

I’m partial to having a separate requirements.txt, due to the splitting between “safe” and “unsafe”. I think an application should strive to have “safe” packages and only install “unsafe” explicitly.

I have not seen this idea mentioned anywhere so I thought I’d put it on the radar; Allow hashes to be explicitly ignored using a wildcard e.g. like --hash=*.

For context this would facilitate developing multiple packages out of one repo with internal and external dependencies. If for instance the dependency graph looks like below then it would be convenient to be able to do something like pip install -c constraints.txt intarnal1. At present this works only without hashes.

external<--+
internal0<-+
           +-internal1

There are many reasons to use an editable install. One is to edit them. Another is to use a package which isn’t on PyPI, such as a fork or private repo. For the former case, a hash is impractical, but for the latter it can be useful.

I often use editable packages just to make it easier to read the source for an important dependency in my text editor by putting it in a top level folder instead of buried in a virtual env somewhere (although I admit this is a niche usecase, so I wouldn’t put much weight on it per se).

Personally I would prefer to keep the current default; editable installs are by definition cannot be pinned, so it does not make sense to ask for it when hashes are required. Making it a special case feels to me like creating a loophole. The two-step solution not working for tools sounds like a toolchain problem to me; tools should provide a way to run multiple commands in the installation phase, instead of pip (or any other installer command) needing to cram steps into one command.

I do feel though it would be acceptable to have some way for the user to explicitly say “hey I know I said I want hashes, but this one is special”. The flag can be added to the command (i.e. the --no-require-hashes proposal) or applied on a per-package bases (i.e. add something like --no-hash or --hash=none to a line in requirements.txt). It would be a conscious decision if the user decides to supply that, and if that’s the case, fine (we’re grown-ups here).