pipenv: tensorflow/html5lib "Your dependencies could not be resolved" but can't see problem

omg is this real

Well I found the problem and a solution, if not an explanation for how it happened.

I spent a while stepping through the underlying code, i.e.

from pipenv.patched.piptools.repositories import PyPIRepository
from pipenv.patched.piptools.scripts.compile import get_pip_command
from pipenv.patched.notpip.req.req_install import InstallRequirement
from pipenv.patched.piptools.resolver import Resolver

pip_command = get_pip_command()
pip_options, _ = pip_command.parse_args([])
session = pip_command._build_session(pip_options)
pypi = PyPIRepository(pip_options=pip_options, use_json=False, session=session)

# first observe that this returns correct constraints:
ireq = InstallRequirement.from_line('bleach==1.5.0')
pypi.get_dependencies(ireq)
{<InstallRequirement object: six (from bleach==1.5.0) editable=False>,
 <InstallRequirement object: html5lib!=0.9999,!=0.99999,<0.99999999,>=0.999 (from bleach==1.5.0) editable=False>}

# I added the following breakpoint at pipenv/patched/piptools/resolver.py:200
            if str(best_match.req) == 'bleach==1.5.0':
                import ipdb; ipdb.set_trace()

# then ran the resolver:
ireq = InstallRequirement.from_line('tensorflow==1.3.0')
resolver = Resolver(constraints=[ireq], repository=pypi)
resolver.resolve()

# stepping through we fail this check, showing that 'html5lib' is already in the cache
    if ireq not in self.dependency_cache:

# so next it fetches dep from cache:
ipdb> pp self.dependency_cache[ireq]
['html5lib!=1.0b1,!=1.0b2,!=1.0b3,!=1.0b4,!=1.0b5,!=1.0b6,!=1.0b7,!=1.0b8,>=0.99999999pre',
 'six']
# aha!
ipdb> self.dependency_cache._cache_file
'/Users/anentropic/Library/Caches/pipenv/depcache-py3.6.json'

Somehow the bogus html5lib>=0.99999999pre requirement had got stuck in the dependency cache file.

Now I knew what to do…

pipenvtemp-py3 $ rm /Users/anentropic/Library/Caches/pipenv/depcache-py3.6.json
pipenvtemp-py3 $ pipenv lock
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project.
Locking [dev-packages] dependencies…
Locking [packages] dependencies…

Success - lock file was created without error.

This also explains why it worked under Python 2.7 - I assume because pipenv has a separate dependency cache file per python version.

@techalchemy thanks again for the work around. It solved my actual problem (installing jupyter + tensorflow): I put jupyter into the development packages and kept tensorflow in the production packages. This way, all packages can be locked.

@henadzit Because spaCy is dropping its required version in 2.0.11, it felt more correct to use the TensorFlow version. I’m not sure what the canonical way to do this is, but I just (temporarily, I hope) pulled tensorflow out of my Pipfile and installed my packages like this:

pipenv install
pip install tensorflow

@henadzit try pipenv lock --clear or wipe any of the following directories if they exist:

/Users/.../Library/Caches/pipenv
/Users/.../Library/Caches/pip-tools
/Users/.../Library/Caches/pip

@anentropic just uninstall all of the pipenvs you have, everywhere, and reinstall using system 3.6.4 with --user

this versioning scheme is just absurd

I can also reproduce with a minimal example… in a new shell, create an empty directory, the following Pipfile and run pipenv install

[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[packages]

tensorflow = "*"


[requires]

python_version = "3.6"