pipenv: Idea: Capture expected Python version & artifact names in Pipfile.lock to improve error messages
Working on a pipenv based development setup in https://github.com/leapp-to/prototype/pull/8 we’re finding it fairly easy for people to get into a situation where they accidentally create the venv with Python 2, but the lock file was generated with Python 3, so the hashes for pre-built wheel files may not match.
Example from https://github.com/leapp-to/prototype/pull/8#issuecomment-289438239:
$ sha256sum ~/Downloads/cffi-1.10.0*
267dd2c66a5760c5f4d47e2ebcf8eeac7ef01e1ae6ae7a6d0d241a290068bc38 /home/podvody/Downloads/cffi-1.10.0-cp34-cp34m-manylinux1_x86_64.whl
4fc9c2ff7924b3a1fa326e1799e5dd58cac585d7fb25fe53ccaa1333b0453d65 /home/podvody/Downloads/cffi-1.10.0-cp35-cp35m-manylinux1_x86_64.whl
e7175287f7fe7b1cc203bb958b17db40abd732690c1e18e700f10e0843a58598 /home/podvody/Downloads/cffi-1.10.0-cp36-cp36m-manylinux1_x86_64.whl
b3b02911eb1f6ada203b0763ba924234629b51586f72a21faacc638269f4ced5 /home/podvody/Downloads/cffi-1.10.0.tar.gz
@shaded-enmity and I thought of two possible ways to make the resulting hash mismatch errors less confusing:
- Record the implementation and version of Python used to generate the lock file in the lock file, since that may affect the pre-built wheel files that pip finds and downloads
- For each locked artifact, record the filename in addition to the hash
For the first option, any hash mismatches would be preceded by a warning that the Python versions didn’t match so the lock file may need to be regenerated.
For the second, if a hash mismatch was found, it would be reported as a filename mismatch first, and only reported as a hash mismatch if the filenames matched.
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 26 (24 by maintainers)
✨🍰✨
This is now released!
Well, only partially. I think we have a ways to go still.
@kennethreitz AFAICT pypi.python.org currently provides only
md5digests but pypi.org also providessha256digests - https://pypi.org/pypi/pipenv/jsonNice! How would you feel about something like
environment-markersas the key name, rather than embedding the PEP number?Of the two suggestions, with the current “single universal lock file” structure, the one I’d recommend is to track the filename that the hash belongs to. When pip picks up a pre-built non-universal wheel, any environment mismatch is going to cause a hash conflict anyway, and having the filenames available will make for significantly more meaningful error messages.
However, I think a more important outcome of that approach would be the fact that tracking the filenames creates the opportunity to switch to a multi-level lockfile scheme, where you take the hashes out of the platform independent version lockfile entirely and move them to a separate environment specific artifact manifest. That is:
Pipfile: the hand-edited file that rules out dependency versions that are known not to workPipfile.lock: only locks the versions of the dependencies, not the specific downloaded artifactsPipfile.<PEP 425 compatibilty tag>.artifacts: expected artifact filenames and hashesWhen
pipenvlocks an environment, it would then generate the artifact manifest based on the most specific binary compatibility tag present in the downloaded artifacts. So if a project only had dependencies on source distributions and pure Python wheel files (or if the--no-binary :all:option was passed to pip), you’d get manifests like:Depending on whether it was Python 2 or Python 3 that was used to generate Pipfile.lock.
While if it had binary dependencies (as with the cffi example above) you might get:
pipenv installwould then ignore any artifact manifest where the compatibility tags didn’t match, as well as any that had been generated from a different version of Pipfile.lock (which could be tracked by storing a hash ofPipfile.lockas one of the fields in the manifest files)Actually taking that second step of splitting out the artifact manifest from the version lock files would involve talking to @dstufft et al about moving pip’s compatibility tag handling and wheel filename parsing out of pip itself and into the
packagingutility library, but that’s a desirable refactoring anyway.Such a model then only fails in the presence of environment markers that result in
Pipfile.lockgetting different dependency lists based on which version of Python and which platform are used to do the install, and even that could potentially be handled at some point in the future by allowing for optional compatibility tags in lock files as well.