pipenv: Pipenv fails when a wheel of a dependency has been uploaded after the Pipfile was generated

Pipenv installations fail if an egg was uploaded for a dependency after Pipenv had already written the sdist hash to the Lockfile.

$ python3 -m pipenv.help

$ python -m pipenv.help output

Pipenv version: '11.8.3'

Pipenv location: '/usr/lib/python3/dist-packages/pipenv'

Python location: '/usr/bin/python3'

Other Python installations in PATH:

2.7: /usr/bin/python2.7
2.7: /usr/bin/python2.7
3.6: /usr/bin/python3.6m
3.6: /usr/bin/python3.6
2.7.14: /usr/bin/python
2.7.14: /usr/bin/python2
3.6.3: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.3',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.13.0-39-generic',
 'platform_system': 'Linux',
 'platform_version': '#44-Ubuntu SMP Thu Apr 5 14:25:01 UTC 2018',
 'python_full_version': '3.6.3',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

CLUTTER_IM_MODULE
LC_MEASUREMENT
LESSCLOSE
LC_PAPER
LC_MONETARY
ANDROID_HOME
XDG_MENU_PREFIX
LANG
LESS
MANAGERPID
DISPLAY
INVOCATION_ID
GNOME_SHELL_SESSION_MODE
COLORTERM
USERNAME
JAVA_HOME
XDG_VTNR
SSH_AUTH_SOCK
MANDATORY_PATH
LC_NAME
XDG_SESSION_ID
USER
DESKTOP_SESSION
QT4_IM_MODULE
TEXTDOMAINDIR
DEFAULTS_PATH
QT_QPA_PLATFORMTHEME
PWD
HOME
LESSHISTFILE
JOURNAL_STREAM
TEXTDOMAIN
SSH_AGENT_PID
QT_ACCESSIBILITY
XDG_SESSION_TYPE
XDG_DATA_DIRS
PYTEST_ADDOPTS
XDG_SESSION_DESKTOP
TILIX_ID
LC_ADDRESS
DBUS_STARTER_ADDRESS
LC_NUMERIC
GTK_MODULES
WINDOWPATH
SELECTED_EDITOR
VTE_VERSION
SHELL
TERM
QT_IM_MODULE
XMODIFIERS
IM_CONFIG_PHASE
DBUS_STARTER_BUS_TYPE
XDG_CURRENT_DESKTOP
MOZ_USE_OMTC
PYTHONSTARTUP
SHLVL
XDG_SEAT
LC_TELEPHONE
GDMSESSION
GNOME_DESKTOP_SESSION_ID
LOGNAME
DBUS_SESSION_BUS_ADDRESS
XDG_RUNTIME_DIR
XAUTHORITY
XDG_CONFIG_DIRS
PATH
LC_IDENTIFICATION
SYSTEMD_NSS_BYPASS_BUS
SESSION_MANAGER
GCC_COLORS
LESSOPEN
GTK_IM_MODULE
LC_TIME
OLDPWD
_
PIP_PYTHON_PATH
PYTHONUNBUFFERED

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /home/remco/.local/share/android-sdk/tools/bin:/home/remco/.local/share/android-sdk/tools:/home/remco/.local/share/android-sdk/platform-tools:/home/remco/.local/share/gradle/bin:/home/remco/.local/bin:/home/remco/.local/share/android-sdk/build-tools/27.0.3:/home/remco/.gem/ruby/2.3.0/bin:/home/remco/.local/share/android-sdk/tools/bin:/home/remco/.local/share/android-sdk/tools:/home/remco/.local/share/android-sdk/platform-tools:/home/remco/.local/share/gradle/bin:/home/remco/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
SHELL: /bin/bash
LANG: en_US.UTF-8
PWD: /home/remco

Steps to replicate

Given a project which has the following Pipfile:

[requires]
python_version = "3.6"

[packages]
pluggy = "*"

If this was installed before April 15th, 2018, this would have produced the following lockfile:

{
    "_meta": {
        "hash": {
            "sha256": "c9c0edfb60fe650018ada97d2fd71a66171e3d74def36de484f6a156802bcc5a"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "pluggy": {
            "hashes": [
                "sha256:7f8ae7f5bdf75671a718d2daf0a64b7885f74510bcd98b1a0bb420eb9a9d0cff"
            ],
            "index": "pypi",
            "version": "==0.6.0"
        }
    },
    "develop": {}
}

The pluggy hash matches the sha value from the sdist found on https://pypi.org/project/pluggy/#files.

Actual result

If pipenv sync is run from this project now (after pluggy wheels have been published), this outputs the following warning:

THESE PACKAGES DO NOT MATCH THE HASHES FROM Pipfile.lock!. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    pluggy==0.6.0 from https://pypi.python.org/packages/ba/65/ded3bc40bbf8d887f262f150fbe1ae6637765b5c9534bd55690ed2c0b0f7/pluggy-0.6.0-py3-none-any.whl#md5=295745cab038ef139c75aa2cdb79a5b0 (from -r /tmp/pipenv-odt4x3d3-requirements/pipenv-5t_4hr2b-requirement.txt (line 1)):
        Expected sha256 7f8ae7f5bdf75671a718d2daf0a64b7885f74510bcd98b1a0bb420eb9a9d0cff
             Got        e160a7fcf25762bb60efc7e171d4497ff1d8d2d75a3d0df7a21b76821ecbf5c5

It appears that pipenv downloads the newly uploaded wheel, but matches this against the known hash of the sdist.

Expected result

I guess pipenv should fall back to the sdist if the wheel doesn’t match any hashes.

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 5
Comments: 15 (7 by maintainers)

Commits related to this issue

Updated Pipfile.lock to prevent dev builds from breaking. On April 15, 2018, the pluggy 0.6.0 wheel was updated (by one of its maintainers) and the hash changed. This does not affect production build... — committed to nditech/apollo by dodumosu 6 years ago

Most upvoted comments

@Overdrivr I’m not quite sure what your question is. pip does not know what hash belongs to which file; it simply receives a set of hashes, and make sure the file it downloads matches one of it. If you lock file includes only the tarball’s hash, but pip downloads a wheel file, the check would fail. Which is the problem.

What OP did was actually conceptually correct in terms of how this check is supposed to work:

Maintain a list of expected hashes.
Download a file.
Oops, the file does not match the hashes.
OP: “Hey so I downloaded this file and it does not match the previous one; did you change it?”
Maintainer: “Oh yeah that’s me, no worry.”
Okay cool so I update the list of expected hashes to include the new one.

Note that pipenv lock does not hash the downloaded file. The hashes it uses are from PyPI. It simply automates the process.

uranusjr on Apr 20, 2018

I think this issue is rare in practice. Also the cause is fairly simple to find by simply looking at the pypi page of the mismatched hash. In this case: https://pypi.org/project/pluggy/#files

The workaround for now is to verify the mismatched hash against the newly uploaded dist and adding it to your lockfile manually.

remcohaszing on Apr 17, 2018

I’m going to close this for now as it’s uncommon enough that we can just add a documentation note for it

techalchemy on Apr 20, 2018

for others finding this thread, https://github.com/pytest-dev/pluggy/issues/134 clarified a lot for me 😃

msheiny on Apr 17, 2018