pipenv: Pipenv fails to resolve subdependency when viable solution exists

This is being worked on here: https://github.com/sarugaku/passa

Issue description

You have two projects:

.
├── base-proj
│   ├── Pipfile
│   └── Pipfile.lock
└── sub-package
    └── setup.py

2 directories, 3 files

The setup.py looks like this:

from setuptools import setup

setup(
    name='sub-package',
    version='0.1',
    install_requires=['tensorflow'],
)

And the Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
tensorflow = "==1.8.0"

[dev-packages]

[requires]
python_version = "3.6"

From the base-proj directory, when you run pipenv install -e ../sub-package/, you get the following output:

Adding -e ../sub-package/ to Pipfile's [packages]...
Pipfile.lock (f43471) out of date, updating to (9beaad)...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.6.0rc0, 1.6.0rc0
There are incompatible versions in the resolved dependencies.
Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
Could not find a version that matches tensorboard<1.10.0,<1.9.0,>=1.8.0,>=1.9.0
Tried: 1.6.0, 1.6.0, 1.7.0, 1.7.0, 1.8.0, 1.8.0, 1.9.0, 1.9.0
Skipped pre-versions: 1.0.0a0, 1.0.0a0, 1.0.0a1, 1.0.0a1, 1.0.0a2, 1.0.0a2, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.6.0rc0, 1.6.0rc0
There are incompatible versions in the resolved dependencies.

How can Pipenv resolve tensorboard to '<1.10.0,<1.9.0,>=1.8.0,>=1.9.0'? It seems it is somehow pulling in the dependency tensorboard of tensorflow==1.9.0 and 1.8.0, and combining them.

Expected result

The tensorflow dependency lock in the main project seems to conflict with the one defined in the setup.py. Specifically the tensorboard sub-dependency seems to become immediately unsolvable (see lock --verbose output below).

Actual result

After running pipenv lock --verbose:

Locking [dev-packages] dependencies...
Locking [packages] dependencies...
using sources: [{'url': 'https://pypi.org/simple', 'verify_ssl': True, 'name': 'pypi'}]
Using pip: -i https://pypi.org/simple

                          ROUND 1                           
Current constraints:
  file:///Users/tom/projects/pipenv-test/sub-package (from -r /var/folders/9w/twrv54nd0v19gsbjhjs3mlhc0000gn/T/pipenv-btuknzld-requirements/pipenv-lxr0x3iq-constraints.txt (line 3))
  tensorflow==1.8.0 (from -r /var/folders/9w/twrv54nd0v19gsbjhjs3mlhc0000gn/T/pipenv-btuknzld-requirements/pipenv-lxr0x3iq-constraints.txt (line 2))

Finding the best candidates:
  found candidate -e file:///Users/tom/projects/pipenv-test/sub-package (constraint was <any>)
  found candidate tensorflow==1.8.0 (constraint was ==1.8.0)

Finding secondary dependencies:
  tensorflow==1.8.0         requires absl-py>=0.1.6, astor>=0.6.0, bleach==1.5.0, gast>=0.2.0, grpcio>=1.8.6, html5lib==0.9999999, markdown>=2.6.8, numpy>=1.13.3, protobuf>=3.4.0, setuptools, six>=1.10.0, tensorboard<1.9.0,>=1.8.0, tensorflow==1.8.0, termcolor>=1.1.0, werkzeug>=0.11.10, wheel>=0.26

New dependencies found in this round:
  adding ['absl-py', '>=0.1.6', '[]']
  adding ['astor', '>=0.6.0', '[]']
  adding ['bleach', '==1.5.0', '[]']
  adding ['gast', '>=0.2.0', '[]']
  adding ['grpcio', '>=1.8.6', '[]']
  adding ['html5lib', '==0.9999999', '[]']
  adding ['markdown', '>=2.6.8', '[]']
  adding ['numpy', '>=1.13.3', '[]']
  adding ['protobuf', '>=3.4.0', '[]']
  adding ['six', '>=1.10.0', '[]']
  adding ['sub-package', '==0.1', '[]']
  adding ['tensorboard', '<1.10.0,<1.9.0,>=1.8.0,>=1.9.0', '[]']
  adding ['tensorflow', '==1.8.0', '[]']
  adding ['termcolor', '>=1.1.0', '[]']
  adding ['werkzeug', '>=0.11.10', '[]']
  adding ['wheel', '>=0.26', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2                           
Current constraints:
  absl-py>=0.1.6
  astor>=0.6.0
  bleach==1.5.0
  gast>=0.2.0
  grpcio>=1.8.6
  html5lib==0.9999999
  markdown>=2.6.8
  numpy>=1.13.3
  protobuf>=3.4.0
  six>=1.10.0
  sub-package==0.1 from file:///Users/tom/projects/pipenv-test/sub-package (from -r /var/folders/9w/twrv54nd0v19gsbjhjs3mlhc0000gn/T/pipenv-btuknzld-requirements/pipenv-lxr0x3iq-constraints.txt (line 3))
  tensorboard<1.10.0,<1.9.0,>=1.8.0,>=1.9.0
  tensorflow==1.8.0 (from -r /var/folders/9w/twrv54nd0v19gsbjhjs3mlhc0000gn/T/pipenv-btuknzld-requirements/pipenv-lxr0x3iq-constraints.txt (line 2))
  termcolor>=1.1.0
  werkzeug>=0.11.10
  wheel>=0.26

Finding the best candidates:
  found candidate absl-py==0.2.2 (constraint was >=0.1.6)
  found candidate astor==0.7.1 (constraint was >=0.6.0)
  found candidate bleach==1.5.0 (constraint was ==1.5.0)
  found candidate gast==0.2.0 (constraint was >=0.2.0)
  found candidate grpcio==1.13.0 (constraint was >=1.8.6)
  found candidate html5lib==0.9999999 (constraint was ==0.9999999)
  found candidate markdown==2.6.11 (constraint was >=2.6.8)
  found candidate numpy==1.14.5 (constraint was >=1.13.3)
  found candidate protobuf==3.6.0 (constraint was >=3.4.0)
  found candidate six==1.11.0 (constraint was >=1.10.0)
  found candidate -e file:///Users/tom/projects/pipenv-test/sub-package (constraint was ==0.1)
Using pip: -i https://pypi.org/simple

                          ROUND 1                           
Current constraints:
  file:///Users/tom/projects/pipenv-test/sub-package (from -r /var/folders/9w/twrv54nd0v19gsbjhjs3mlhc0000gn/T/pipenv-btuknzld-requirements/pipenv-f2vkysg2-constraints.txt (line 3))
  tensorflow==1.8.0 (from -r /var/folders/9w/twrv54nd0v19gsbjhjs3mlhc0000gn/T/pipenv-btuknzld-requirements/pipenv-f2vkysg2-constraints.txt (line 2))

Finding the best candidates:
  found candidate -e file:///Users/tom/projects/pipenv-test/sub-package (constraint was <any>)
  found candidate tensorflow==1.8.0 (constraint was ==1.8.0)

Finding secondary dependencies:
  tensorflow==1.8.0         requires absl-py>=0.1.6, astor>=0.6.0, bleach==1.5.0, gast>=0.2.0, grpcio>=1.8.6, html5lib==0.9999999, markdown>=2.6.8, numpy>=1.13.3, protobuf>=3.4.0, setuptools, six>=1.10.0, tensorboard<1.9.0,>=1.8.0, tensorflow==1.8.0, termcolor>=1.1.0, werkzeug>=0.11.10, wheel>=0.26

New dependencies found in this round:
  adding ['absl-py', '>=0.1.6', '[]']
  adding ['astor', '>=0.6.0', '[]']
  adding ['bleach', '==1.5.0', '[]']
  adding ['gast', '>=0.2.0', '[]']
  adding ['grpcio', '>=1.8.6', '[]']
  adding ['html5lib', '==0.9999999', '[]']
  adding ['markdown', '>=2.6.8', '[]']
  adding ['numpy', '>=1.13.3', '[]']
  adding ['protobuf', '>=3.4.0', '[]']
  adding ['six', '>=1.10.0', '[]']
  adding ['sub-package', '==0.1', '[]']
  adding ['tensorboard', '<1.10.0,<1.9.0,>=1.8.0,>=1.9.0', '[]']
  adding ['tensorflow', '==1.8.0', '[]']
  adding ['termcolor', '>=1.1.0', '[]']
  adding ['werkzeug', '>=0.11.10', '[]']
  adding ['wheel', '>=0.26', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2                           
Current constraints:
  absl-py>=0.1.6
  astor>=0.6.0
  bleach==1.5.0
  gast>=0.2.0
  grpcio>=1.8.6
  html5lib==0.9999999
  markdown>=2.6.8
  numpy>=1.13.3
  protobuf>=3.4.0
  six>=1.10.0
  sub-package==0.1 from file:///Users/tom/projects/pipenv-test/sub-package (from -r /var/folders/9w/twrv54nd0v19gsbjhjs3mlhc0000gn/T/pipenv-btuknzld-requirements/pipenv-f2vkysg2-constraints.txt (line 3))
  tensorboard<1.10.0,<1.9.0,>=1.8.0,>=1.9.0
  tensorflow==1.8.0 (from -r /var/folders/9w/twrv54nd0v19gsbjhjs3mlhc0000gn/T/pipenv-btuknzld-requirements/pipenv-f2vkysg2-constraints.txt (line 2))
  termcolor>=1.1.0
  werkzeug>=0.11.10
  wheel>=0.26

Finding the best candidates:
  found candidate absl-py==0.2.2 (constraint was >=0.1.6)
  found candidate astor==0.7.1 (constraint was >=0.6.0)
  found candidate bleach==1.5.0 (constraint was ==1.5.0)
  found candidate gast==0.2.0 (constraint was >=0.2.0)
  found candidate grpcio==1.13.0 (constraint was >=1.8.6)
  found candidate html5lib==0.9999999 (constraint was ==0.9999999)
  found candidate markdown==2.6.11 (constraint was >=2.6.8)
  found candidate numpy==1.14.5 (constraint was >=1.13.3)
  found candidate protobuf==3.6.0 (constraint was >=3.4.0)
  found candidate six==1.11.0 (constraint was >=1.10.0)
  found candidate -e file:///Users/tom/projects/pipenv-test/sub-package (constraint was ==0.1)

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
Could not find a version that matches tensorboard<1.10.0,<1.9.0,>=1.8.0,>=1.9.0
Tried: 1.6.0, 1.6.0, 1.7.0, 1.7.0, 1.8.0, 1.8.0, 1.9.0, 1.9.0
Skipped pre-versions: 1.0.0a0, 1.0.0a0, 1.0.0a1, 1.0.0a1, 1.0.0a2, 1.0.0a2, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.6.0rc0, 1.6.0rc0
There are incompatible versions in the resolved dependencies.
Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
Could not find a version that matches tensorboard<1.10.0,<1.9.0,>=1.8.0,>=1.9.0
Tried: 1.6.0, 1.6.0, 1.7.0, 1.7.0, 1.8.0, 1.8.0, 1.9.0, 1.9.0
Skipped pre-versions: 1.0.0a0, 1.0.0a0, 1.0.0a1, 1.0.0a1, 1.0.0a2, 1.0.0a2, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a3, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a4, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a5, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.0.0a6, 1.6.0rc0, 1.6.0rc0
There are incompatible versions in the resolved dependencies.
Steps to replicate

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

$ pipenv --support

Pipenv version: '2018.7.1'

Pipenv location: '/usr/local/Cellar/pipenv/2018.7.1/libexec/lib/python3.7/site-packages/pipenv'

Python location: '/usr/local/Cellar/pipenv/2018.7.1/libexec/bin/python3.7'

Other Python installations in PATH:

  • 2.7: /usr/local/bin/python2.7

  • 2.7: /usr/local/bin/python2.7

  • 2.7: /usr/bin/python2.7

  • 3.6: /Users/tom/.pyenv/shims/python3.6m

  • 3.6: /Users/tom/.local/share/virtualenvs/base-proj-lBpX7WJ2/bin/python3.6

  • 3.6: /Users/tom/.pyenv/shims/python3.6

  • 3.6: /Users/tom/.pyenv/shims/python3.6

  • 3.7: /usr/local/bin/python3.7m

  • 3.7: /usr/local/bin/python3.7

  • 3.6.5: /Users/tom/.local/share/virtualenvs/base-proj-lBpX7WJ2/bin/python

  • 3.6.5: /Users/tom/.pyenv/shims/python

  • 3.6.5: /Users/tom/.pyenv/shims/python

  • 2.7.15: /usr/local/bin/python

  • 2.7.10: /usr/bin/python

  • 2.7.15: /usr/local/bin/python2

  • 3.6.5: /Users/tom/.local/share/virtualenvs/base-proj-lBpX7WJ2/bin/python3

  • 3.6.5: /Users/tom/.pyenv/shims/python3

  • 3.6.5: /Users/tom/.pyenv/shims/python3

  • 3.7.0: /usr/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.7.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT '
                     '2018; root:xnu-4570.71.2~1/RELEASE_X86_64',
 'python_full_version': '3.7.0',
 'python_version': '3.7',
 'sys_platform': 'darwin'}

System environment variables:

  • PATH
  • TERM_PROGRAM
  • TERM
  • SHELL
  • PIP_PYTHON_PATH
  • OP_entry_name
  • TMPDIR
  • OL_client_secret
  • Apple_PubSub_Socket_Render
  • TERM_PROGRAM_VERSION
  • OP_SESSION
  • HOMEBREW_AUTO_UPDATE_SECS
  • TERM_SESSION_ID
  • LC_ALL
  • USER
  • ONFIDO_ROLE
  • SSH_AUTH_SOCK
  • __CF_USER_TEXT_ENCODING
  • nvm_prefix
  • VIRTUAL_ENV_DISABLE_PROMPT
  • VIRTUAL_ENV
  • PIPENV_ACTIVE
  • PWD
  • OL_client_id
  • EDITOR
  • LANG
  • ITERM_PROFILE
  • XPC_FLAGS
  • OL_region
  • PYTHONDONTWRITEBYTECODE
  • XPC_SERVICE_NAME
  • SHLVL
  • PYENV_SHELL
  • HOME
  • COLORFGBG
  • ITERM_SESSION_ID
  • LOGNAME
  • _OLD_VIRTUAL_PATH
  • LC_CTYPE
  • OL_username
  • __pipenv_fish_initial_pwd
  • COLORTERM

Pipenv–specific environment variables:

  • PIPENV_ACTIVE: 1

Debug–specific environment variables:

  • PATH: /usr/local/Cellar/pipenv/2018.7.1/libexec/tools:/Users/tom/.local/share/virtualenvs/base-proj-lBpX7WJ2/bin:/Users/tom/.pyenv/shims:/usr/local/Cellar/pipenv/2018.7.1/libexec/tools:/Users/tom/.pyenv/shims:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Applications/Wireshark.app/Contents/MacOS
  • SHELL: /usr/local/bin/fish
  • EDITOR: nano
  • LANG: en_US.UTF-8
  • PWD: /Users/tom/projects/pipenv-test/base-proj
  • VIRTUAL_ENV: /Users/tom/.local/share/virtualenvs/base-proj-lBpX7WJ2

Contents of Pipfile (‘/Users/tom/projects/pipenv-test/base-proj/Pipfile’):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
tensorflow = "==1.8.0"
sub-package = {editable = true, path = "./../sub-package"}

[dev-packages]

[requires]
python_version = "3.6"

Contents of Pipfile.lock (‘/Users/tom/projects/pipenv-test/base-proj/Pipfile.lock’):

{
    "_meta": {
        "hash": {
            "sha256": "d9be781a0c99deda926abe1f0ad149d718c738ffe906d002d6f79bf053edb66f"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "absl-py": {
            "hashes": [
                "sha256:e0eb8358b549552b1cc5972350bc3e41dd0a926c15b3ff95ce60f3c78c80824c"
            ],
            "version": "==0.2.2"
        },
        "astor": {
            "hashes": [
                "sha256:95c30d87a6c2cf89aa628b87398466840f0ad8652f88eb173125a6df8533fb8d",
                "sha256:fb503b9e2fdd05609fbf557b916b4a7824171203701660f0c55bbf5a7a68713e"
            ],
            "markers": "python_version != '3.0.*' and python_version != '3.3.*' and python_version != '3.2.*' and python_version != '3.1.*' and python_version >= '2.7'",
            "version": "==0.7.1"
        },
        "bleach": {
            "hashes": [
                "sha256:978e758599b54cd3caa2e160d74102879b230ea8dc93871d0783721eef58bc65",
                "sha256:e67f46adcec78dbc3c04462f3aba3213a673d5652eba2609ed1ef15492a44b8d"
            ],
            "version": "==1.5.0"
        },
        "gast": {
            "hashes": [
                "sha256:7068908321ecd2774f145193c4b34a11305bd104b4551b09273dfd1d6a374930"
            ],
            "version": "==0.2.0"
        },
        "grpcio": {
            "hashes": [
                "sha256:002f9170d8c0c10c33f643240c2332ce6eb8c8cc8c2b08d6a8f3172ef574751b",
                "sha256:0f80b28033105e99e08d84361b899c45bd6eb31d2765ff2ed7cb66b8b1d12820",
                "sha256:1012969abbec9a6c5d7b09ac829df296cb0a2ddebe70dd314abb881fa62cbcc9",
                "sha256:19af04506fec213de9a889343d4b090e9d0b675e1d9b8397ea6fc6521f939a48",
                "sha256:37cdffc26987ae2a077cc80fc0d87273e0e01ba15df40ec0a20d107e635700cf",
                "sha256:4e72b30a276e519d687dc53d86ecf9d65edf31dad473f3bbd041542bbf9df12f",
                "sha256:564499d84c2e90eb87819f7a299eaa2aee32db8208a8e8d00e8941a0c66413c0",
                "sha256:6324581e215157f0fbe335dff2e21a65b4406db98ac7cca05f1e23b4f510b426",
                "sha256:64758d2718f14792286b32d31560edb10c3726ce5d5875c3472c95908b658aeb",
                "sha256:67b36c6b0070ef858e5438e82d3a3fb19db33a174572019744d7b965c95fa9b0",
                "sha256:83766cdfc3492a693902eff193648b3cc9710e4a131815cd0cc60e30e9b7cf8f",
                "sha256:935a0b328b79e03a47d87960836513bab1150d0faab44aff7968f8623ed48d62",
                "sha256:982349cc24df7569ab955bafa3ba7575140db171c3bd757fa135d0c0c0d6990d",
                "sha256:9c79eaca4b0b8fb973c6e0c4c6cd4be44e76dfd09d56cfc7b2a959289fda6682",
                "sha256:9efcdfe1cc1670dde940f86e51fe080184f637106cf9b6d8dfc14cab9660d710",
                "sha256:a24a37e441c36d6605029cfb035806ef4f888a37d757e64fb0488d03c3fff806",
                "sha256:ab33b5965df89fd6e4ea7846cb9a28ece4e6d9ded23434d3321b992051a62de0",
                "sha256:ad6c12e9ab809fd4f8891a085a7655ca2690a5753bb6258d9d602084f610a223",
                "sha256:b1e4c83209bce5548029ed7df2af6a94415bb7dd37a2203183919d1d5a5249c9",
                "sha256:b87cdf8c4291875bda4b2a0f6747cab008fd9ea6c682a43d0869d308a16d0956",
                "sha256:b9e3793e0d1498e5c72993ba91f14e617e06e945d6562716bfbbbc6a9e7ab7de",
                "sha256:ba388412f64d6ac02098fcb77c409896297b058a8942d946bd1999699d35c123",
                "sha256:baaf1a0d5a5d9af67ec3ff0d9dfc1d642e7aba38e59ae60de1c6d1bd46406177",
                "sha256:be0d79c3253f7d23facc4dab96ada086e9b17048a36843041a5fac3bad9415e7",
                "sha256:ce3c23b1110238c1f440cdbceefd0c5fc7fcf3022c82c8a349514038aa69ac3d",
                "sha256:d9b9d309e7db3a988df0d12ba3c1ca4a7059c502c10ce34d4d65779bebbb6949",
                "sha256:ddd489b4730d7eccf9b836216d7137f85b3e68258a292a618a12e9ce5a7bcfb0",
                "sha256:dee025675506fc84f475b9fb0c8ce2cc9b663f9d7c7a22281ba878538be78fe8",
                "sha256:e6ac3198f4174c1b58e3a6b765d0b9cdcead231ba8bf4ddd30709320960b6e39",
                "sha256:e738782d0216087cb7ee9acc54305db9a30bf9a784bff7a407b748a91dd8c942",
                "sha256:f136b98861f27e2628f824c7c7e8d4bce47c9e18953fd00a0aca1d1c9cfd1b6c"
            ],
            "version": "==1.13.0"
        },
        "html5lib": {
            "hashes": [
                "sha256:2612a191a8d5842bfa057e41ba50bbb9dcb722419d2408c78cff4758d0754868"
            ],
            "version": "==0.9999999"
        },
        "markdown": {
            "hashes": [
                "sha256:9ba587db9daee7ec761cfc656272be6aabe2ed300fece21208e4aab2e457bc8f",
                "sha256:a856869c7ff079ad84a3e19cd87a64998350c2b94e9e08e44270faef33400f81"
            ],
            "version": "==2.6.11"
        },
        "numpy": {
            "hashes": [
                "sha256:07379fe0b450f6fd6e5934a9bc015025bb4ce1c8fbed3ca8bef29328b1bc9570",
                "sha256:085afac75bbc97a096744fcfc97a4b321c5a87220286811e85089ae04885acdd",
                "sha256:2d6481c6bdab1c75affc0fc71eb1bd4b3ecef620d06f2f60c3f00521d54be04f",
                "sha256:2df854df882d322d5c23087a4959e145b953dfff2abe1774fec4f639ac2f3160",
                "sha256:381ad13c30cd1d0b2f3da8a0c1a4aa697487e8bb0e9e0cbeb7439776bcb645f8",
                "sha256:385f1ce46e08676505b692bfde918c1e0b350963a15ef52d77691c2cf0f5dbf6",
                "sha256:4130e5ae16c656b7de654dc5e595cfeb85d3a4b0bb0734d19c0dce6dc7ee0e07",
                "sha256:4d278c2261be6423c5e63d8f0ceb1b0c6db3ff83f2906f4b860db6ae99ca1bb5",
                "sha256:51c5dcb51cf88b34b7d04c15f600b07c6ccbb73a089a38af2ab83c02862318da",
                "sha256:589336ba5199c8061239cf446ee2f2f1fcc0c68e8531ee1382b6fc0c66b2d388",
                "sha256:5ae3564cb630e155a650f4f9c054589848e97836bebae5637240a0d8099f817b",
                "sha256:5edf1acc827ed139086af95ce4449b7b664f57a8c29eb755411a634be280d9f2",
                "sha256:6b82b81c6b3b70ed40bc6d0b71222ebfcd6b6c04a6e7945a936e514b9113d5a3",
                "sha256:6c57f973218b776195d0356e556ec932698f3a563e2f640cfca7020086383f50",
                "sha256:758d1091a501fd2d75034e55e7e98bfd1370dc089160845c242db1c760d944d9",
                "sha256:8622db292b766719810e0cb0f62ef6141e15fe32b04e4eb2959888319e59336b",
                "sha256:8b8dcfcd630f1981f0f1e3846fae883376762a0c1b472baa35b145b911683b7b",
                "sha256:91fdd510743ae4df862dbd51a4354519dd9fb8941347526cd9c2194b792b3da9",
                "sha256:97fa8f1dceffab782069b291e38c4c2227f255cdac5f1e3346666931df87373e",
                "sha256:9b705f18b26fb551366ab6347ba9941b62272bf71c6bbcadcd8af94d10535241",
                "sha256:9d69967673ab7b028c2df09cae05ba56bf4e39e3cb04ebe452b6035c3b49848e",
                "sha256:9e1f53afae865cc32459ad211493cf9e2a3651a7295b7a38654ef3d123808996",
                "sha256:a4a433b3a264dbc9aa9c7c241e87c0358a503ea6394f8737df1683c7c9a102ac",
                "sha256:baadc5f770917ada556afb7651a68176559f4dca5f4b2d0947cd15b9fb84fb51",
                "sha256:c725d11990a9243e6ceffe0ab25a07c46c1cc2c5dc55e305717b5afe856c9608",
                "sha256:d696a8c87315a83983fc59dd27efe034292b9e8ad667aeae51a68b4be14690d9",
                "sha256:e1864a4e9f93ddb2dc6b62ccc2ec1f8250ff4ac0d3d7a15c8985dd4e1fbd6418",
                "sha256:e1d18421a7e2ad4a655b76e65d549d4159f8874c18a417464c1d439ee7ccc7cd"
            ],
            "markers": "python_version != '3.0.*' and python_version != '3.3.*' and python_version != '3.2.*' and python_version != '3.1.*' and python_version >= '2.7'",
            "version": "==1.14.5"
        },
        "protobuf": {
            "hashes": [
                "sha256:12985d9f40c104da2f44ec089449214876809b40fdc5d9e43b93b512b9e74056",
                "sha256:12c97fe27af12fc5d66b23f905ab09dd4fb0c68d5a74a419d914580e6d2e71e3",
                "sha256:327fb9d8a8247bc780b9ea7ed03c0643bc0d22c139b761c9ec1efc7cc3f0923e",
                "sha256:3895319db04c0b3baed74fb66be7ba9f4cd8e88a432b8e71032cdf08b2dfee23",
                "sha256:695072063e256d32335d48b9484451f7c7948edc3dbd419469d6a778602682fc",
                "sha256:7d786f3ef5b33a04e6538089674f244a3b0f588155016559d950989010af97d0",
                "sha256:8bf82bb7a466a54be7272dcb492f71d55a2453a58d862fb74c3f2083f2768543",
                "sha256:9bbc1ae1c33c1bd3a2fc05a3aec328544d2b039ff0ce6f000063628a32fad777",
                "sha256:9f1087abb67b34e55108bc610936b34363a7aac692023bcbb17e065c253a1f80",
                "sha256:9fefcb92a3784b446abf3641d9a14dad815bee88e0edd10b9a9e0e144d01a991",
                "sha256:a37836aa47d1b81c2db1a6b7a5e79926062b5d76bd962115a0e615551be2b48d",
                "sha256:cca22955443c55cf86f963a4ad7057bca95e4dcde84d6a493066d380cfab3bb0",
                "sha256:d7ac50bc06d31deb07ace6de85556c1d7330e5c0958f3b2af85037d6d1182abf",
                "sha256:dfe6899304b898538f4dc94fa0b281b56b70e40f58afa4c6f807805261cbe2e8"
            ],
            "version": "==3.6.0"
        },
        "six": {
            "hashes": [
                "sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9",
                "sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb"
            ],
            "version": "==1.11.0"
        },
        "tensorboard": {
            "hashes": [
                "sha256:2651a4d9261a6593cb2c3514576e4bf25e273837c79e98f68a3cf51759f68725",
                "sha256:7776cc8bcfd0d07cd106e9e86c011ab8eca38c97b57f5433b9f0fd6bbf31a36e"
            ],
            "markers": "python_version != '3.0.*' and python_version != '3.3.*' and python_version != '3.2.*' and python_version != '3.1.*' and python_version >= '2.7'",
            "version": "==1.8.0"
        },
        "tensorflow": {
            "hashes": [
                "sha256:1fc4eb267bc973f2a8466778ad0c6f110f8356373e534c8463a7bb9b6d86e5f0",
                "sha256:24c3d3f87dc6108f4de3bbb7f647bcd6f303f874721b176185f260adea88fe40",
                "sha256:3087797a85610b756066df294f98e666e49f4fa60c6e1dbcf4b190b21397af71",
                "sha256:36a6671d39a3aec89cb4cce125e90305c3f8a69da16d2fb72eff40fe478e1de9",
                "sha256:373c637f9c4f06346b1b02e280b30517d262a1651b252ff889f7c22716ef4548",
                "sha256:397c0766e166fb768613b1498a2a6f67eeea077425d234c0138d55e85408473d",
                "sha256:68bda870c355aa64dc3dc377d5b9d13c829281467a7c3b95cb8529031d8f68b2",
                "sha256:899e53e01efcc8e906b3d1f53fe6a6264edf5dfc275b32cadbc6f4e33dca78f7",
                "sha256:92801b9aebcc195d3b5a492d1d46330c47c11c6f0c0f7ab6b605da489482c64e",
                "sha256:d345d296aeb05eeb50d9de43a1dcb66ceaba6a2bd603f58aeefaa07b2c1bfac1",
                "sha256:dccc205e84cd33a240a601046e88eacefe12d677a1bbdf17f0ebafd1a7c84c70",
                "sha256:de51f60021ea8160ea6d0340e827a26331cd549f6e7c470fba7ee83aeab4f818"
            ],
            "index": "pypi",
            "version": "==1.8.0"
        },
        "termcolor": {
            "hashes": [
                "sha256:1d6d69ce66211143803fbc56652b41d73b4a400a2891d7bf7a1cdf4c02de613b"
            ],
            "version": "==1.1.0"
        },
        "werkzeug": {
            "hashes": [
                "sha256:c3fd7a7d41976d9f44db327260e263132466836cef6f91512889ed60ad26557c",
                "sha256:d5da73735293558eb1651ee2fddc4d0dedcfa06538b8813a2e20011583c9e49b"
            ],
            "version": "==0.14.1"
        },
        "wheel": {
            "hashes": [
                "sha256:0a2e54558a0628f2145d2fc822137e322412115173e8a2ddbe1c9024338ae83c",
                "sha256:80044e51ec5bbf6c894ba0bc48d26a8c20a9ba629f4ca19ea26ecfcf87685f5f"
            ],
            "markers": "python_version >= '3'",
            "version": "==0.31.1"
        }
    },
    "develop": {}
}

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 10
  • Comments: 42 (17 by maintainers)

Most upvoted comments

Consider this simple scenario. There are packages A, B, C, and D.

  • A depends on B and C.
  • B depends on C and D.

And you specify A in your Pipfile.

The resolver can be thought of working in rounds. For each round, we look inside each package to find their dependencies, add them into the list, and repeat, until we cannot find anymore.

R0: A. R1: A depends on B and C. Now we have A, B, C. R2: B depends on C and D; C does not depend on anything. Now A, B, C, D. R3: D does not depend on anything. Still A, B, C, D. Resolution finished successfully.

Now let’s consider versions. Say A, B, and D each has one version (to keeps things simple), while C has two, C1 and C2.

  • A depends on B and C.
  • B depends on C and D, but requires C1 (i.e. specifies C<2 as a dependency).

R0: A. R1: A wants on B and C. Which C should we choose? We don’t know.

What Pipenv current does is to always prefer the latest, so…

R1: A, B, C2. R2: B wants C1 and D. Now we have a problem, we can’t install C1 and C2 at the same time. Resolution fails.

What backtracking allows, is to save before we made that decision in round 1.

R1: A, B, C… save here. R1a: A, B, C2. R2-1a: This does not work. Load. R1b: A, B, C1. R2-1b: A, B, C1, D. Success!

Now generalise this. Package X may require P<5 in v1, and P>5 in v2. We might choose v2 first, but if that prooves to be a problem later on (perhaps Y requires P<5 in all versions), we can backtrack and choose the v1 series of X instead. Those conflicting version requirements you see generally come from this kind of more complicated graphs, and backtracking is the solution to it (if there is a solution at all).

Backtracking resolution algorithm is largely a solved problem, and there are implementations in various languages. Python, however, is more lacking in this regard. Mixology, which is used by Poetry (another tool that also uses a lock file), and largely similar to Ruby’s Molinillo (used by Bundler and Cocoapods), is the only one I know in real-world production use. Zazo is another, but it is in an even earlier stage.

I hope the above clarifies things somewhat. This is not a simple topic to work on (though again, not difficult, because everything is already solved, you only need to implement it with a straight head), but would help Pipenv—and I’dsay, the whole Python comminuty—tremendously, if done correctly.

+19
uranusjr on Jul 17, 2018

pipenv lock --clear will clear your dependency cache and resolve it properly

+14
techalchemy on Aug 1, 2018

I have the issue again with apache-airflow = "==1.10.3". It worked fine, but started to fail about week ago. Dependency resolver complains about jinja2 dependency. Is there any ongoing work on extracting the resolving mechanism from Poetry?

+13
Kreshu on Jul 11, 2019

nor to add jinja2 = "==2.9.0" explicitly in you Pipfile not pipenv lock --clear helped… How can I use pipenv together with airflow?

+11
nickdani on Aug 24, 2018

@uranusjr The resolver has a backtracking algorithm but it’s rarely used since the cases where a true conflict occurs are rare.

I’ve been looking into Mixology (and Milinillo), and the problem to me is that the dependency information for each version is fetched too eagerly when populating a specification.

I agree and that’s why I rewrote the resolver completely since it was too slow sometimes to be usable.

I was under the impression that by implementing possibilities as a lazily-evaluated list would solve most of the problem.

I tried that but it didn’t lead anywhere, at least with the previous algorithm. It would make this worse since you couldn’t check that a package’s dependency set had already been seen before and thus ditch the package if it had been, so you would inspect packages that did not warrant being checked, slowing down the process even more.

@uranusjr @orf @techalchemy I plan on extract it from Poetry at some point, yes. However, I don’t have an ETA yet since right now it’s tightly coupled with other parts of the codebase (like a version constraint module). So I have to abstract some of the concepts used internally to make it a trully independent library.

+7
sdispater on Jul 17, 2018

@gzagatti @Mokubyow

I’ve got airflow 1.10 working with something like this and pipenv==2018.5.18.

[requires]
python_version = "2.7"

[packages]
apache-airflow = "==1.10.0"
flask-appbuilder = "==1.11.1"

I also have notes that these were needed:

backports_abc = ">=0.4"
singledispatch = "~=3.4"
+6
jvstein on Oct 9, 2018

FWIW, this appears to be a recent regression and downgrading to pipenv==2018.5.18 is a workaround until this issue is fixed. I suspect it’s related to the upgrade from pip 9 to pip 10 (i.e. #2255).

+5
ltm on Jul 24, 2018

Mixology, which is used by Poetry (another tool that also uses a lock file), and largely similar to Ruby’s Molinillo

Just for information this is no longer the case since it was way too slow for Python packaging ecosystem. It works well if all the metadata is available via an API but not otherwise. So I implemented in Poetry another resolver which is lazy (it gets the information only when needed), has conflict resolution and a backtracking algorithm. This led to cutting down extreme cases which would take 30 minutes to less than a minute now. And most of the time it will resolve dependencies in less than 10 seconds.

+5
sdispater on Jul 17, 2018

Is there anyway to override the version that has a conflict? I can’t find how to set the version of the conflicting package. I tried setting it in the Pipfile but I still get the conflict and I can’t generate the lock file.

+2
aleksarias on Jul 28, 2020

For the airflow folks, poetry worked for me:

https://gist.github.com/jasonnerothin/aaacd3c8aec30bfca991fd04e2847ec9

+2
jasonnerothin on Aug 21, 2019

Can confirm that running pipenv lock --clear worked

+2
benhowes on Aug 1, 2018

@lukesmurray poetry is an alternative dependency version management tool that serves the same purpose of pipenv, but they claim to have a better dependency resolution algorithm than pipenv. In case you can’t afford to way this bug to be fixed, you could use it.

Pipenv is already integrated in my deployment process, so I particularly rather wait for this to be fixed.

+2
thalesmello on Jul 23, 2018

@sdispater Would you mind if I try to extract Mixology from Poetry into a standalone repository on my own? I’m trying to work on the resolver logic, and it would be great if I could have something to interface against. I will include appropriate attribution, of course.

+2
uranusjr on Jul 18, 2018

I would like to mention that, as always, contributions toward usability improvements and better error messaging are highly welcomed, and tend to be accepted very quickly, if anyone really cares enough to submit a pull request 😃

+1
uranusjr on Aug 1, 2018

I can report that I am having a similar issue when trying to install pytest and pytest-cov, which is a plugin of pytest.

Since there are no restrictions on both of the items, there should be no conflicts between both since pytest-cov is developed to be used with pytest as a plugin. There should be a version of pytest (if not the latest) which works with pytest-cov.

A minimal Pipfile like the one below will fail complaining that it could not find a version that matches pluggy<0.7,>=0.5,>=0.7:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pytest-cov = "*"
pytest = "*"

[requires]
python_version = "3.6"
+1
gzagatti on Jul 31, 2018
Read more comments on GitHub
← pipenv: Pipenv fails to install/upgrade setuptools after 2022.8.19 release. UBI9
pipenv: Pipenv fails when a wheel of a dependency has been uploaded after the Pipfile was generated →