pipenv: Any package requiring importlib-metadata breaks locking

With pip install -U pipenv was solved for me.

The workaround of deleting the Pipfile.lock file and then re-running pipenv lock always seems to work but here is what I did for a more permanent fix.

1. Destroy local pipenv virtualenv pipenv --rm
2. Uninstall pipenv brew uninstall pipenv
3. Upgrade virtualenv pip install -U virtualenv
4. Reinstall pipenv pip install --user pipenv
5. Recreate pipenv virtualenv pipenv install

Repeat runs of pipenv lock work as expected and do not re-trigger the conflict error message.

With importlib-metadata 3.0.0 this error seems to be occurring again, even after having the latest version of virtualenv

⠹ Locking..✘ Locking Failed! 
Traceback (most recent call last):
  File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.6/site-packages/pipenv/resolver.py", line 764, in <module>
    main()
  File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.6/site-packages/pipenv/resolver.py", line 760, in main
    dev=parsed.dev)
  File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.6/site-packages/pipenv/resolver.py", line 741, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
  File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.6/site-packages/pipenv/resolver.py", line 714, in resolve_packages
    results = clean_results(results, resolver, project, dev)
  File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.6/site-packages/pipenv/resolver.py", line 590, in clean_results
    reverse_deps = project.environment.reverse_dependencies()
  File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.6/site-packages/pipenv/project.py", line 376, in environment
    self._environment = self.get_environment(allow_global=allow_global)
  File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.6/site-packages/pipenv/project.py", line 366, in get_environment
    environment.extend_dists(pipenv_dist)
  File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.6/site-packages/pipenv/environment.py", line 127, in extend_dists
    extras = self.resolve_dist(dist, self.base_working_set)
  File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.6/site-packages/pipenv/environment.py", line 122, in resolve_dist
    deps |= cls.resolve_dist(dist, working_set)
  File "/home/user/.local/share/virtualenvs/user-5PivF_Od/lib/python3.6/site-packages/pipenv/environment.py", line 121, in resolve_dist
    dist = working_set.find(req)
  File "/tmp/tmppbnfl0ad/.venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 627, in find
    raise VersionConflict(dist, req)
pkg_resources.VersionConflict: (importlib-metadata 3.0.0 (/tmp/tmppbnfl0ad/.venv/lib/python3.6/site-packages), Requirement.parse('importlib-metadata<3,>=0.12; python_version < "3.8"'))

Any suggestions on what we could do to avoid this? @joshsleeper

I found that my issue was actually just matter of updating to the newest virtualenv patch release 20.0.32

• https://virtualenv.pypa.io/en/stable/changelog.html#v20-0-32-2020-10-01
• https://github.com/pypa/virtualenv/pull/1953

once that version was released and incorporated into my lockfiles everything worked like a charm on both python 3.7 and 3.8.

This issue is fixed by the newest release of virtualenv

Any newcomers to this issue should upgrade the version of virtualenv

Here are my debugging notes. Still not sure which package is causing the problem, but this does seem to fix it.

The error you get when you first run pipenv install with pipenv version 2020.8.13.

Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/cli/command.py", line 252, in install
    site_packages=state.site_packages
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 1928, in do_install
    site_packages=site_packages,
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 580, in ensure_project
    pypi_mirror=pypi_mirror,
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 512, in ensure_virtualenv
    python=python, site_packages=site_packages, pypi_mirror=pypi_mirror
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 999, in do_create_virtualenv
    project._environment.add_dist("pipenv")
  File "/usr/local/lib/python3.6/site-packages/pipenv/environment.py", line 135, in add_dist
    self.extend_dists(dist)
  File "/usr/local/lib/python3.6/site-packages/pipenv/environment.py", line 127, in extend_dists
    extras = self.resolve_dist(dist, self.base_working_set)
  File "/usr/local/lib/python3.6/site-packages/pipenv/environment.py", line 122, in resolve_dist
    deps |= cls.resolve_dist(dist, working_set)
  File "/usr/local/lib/python3.6/site-packages/pipenv/environment.py", line 121, in resolve_dist
    dist = working_set.find(req)
  File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 642, in find
    raise VersionConflict(dist, req)
pkg_resources.VersionConflict: (importlib-metadata 2.0.0 (/usr/local/lib/python3.6/site-packages), Requirement.parse('importlib-metadata<2,>=0.12; python_version < "3.8"'))

If you run pip install -U pipenv it seems to change the importlib-metadata version:

Installing collected packages: importlib-metadata
  Attempting uninstall: importlib-metadata
    Found existing installation: importlib-metadata 2.0.0
    Uninstalling importlib-metadata-2.0.0:
      Successfully uninstalled importlib-metadata-2.0.0
Successfully installed importlib-metadata-1.7.0

Now if you run pipenv install -d --skip-lock it will finish. It seems like a library is requiring a version >= importlib-metadata 2.0.

When I pinned the following dependencies it didn’t work at first when running pipenv lock, however, if I removed the lock file (rm Pipenv.lock) then it worked when I ran pipenv lock again.

virtualenv = "==20.0.31"
importlib-metadata = "==1.7.0"

@nicofuchsg if you’re implying that simply running the latest version of pipenv solves the problem, fyi that’s not the case universally.

I encountered this issue using pipenv 2020.8.13 from start to finish.

How about if we are using pip instead of virtualenv.

The sequence of calls causing this error are fairly elaborate.

1. Call pipenv lock
2. This results in a call to main in resolver.py within a subprocess
3. This causes a call to clean_results only if there is an already existing lock file
4. This results in an attempt to instantiate the project environment to find reverse dependencies.
5. When instantiating the project environment the pipenv distribution is added to the environment using extend_dists
6. In extend_dists a search is conducted for all dependencies of pipenv in the working set
7. This results in a search for package versions matching the requirements of virtualenv including importlib-metadata

If this search results in finding no version of the package installed, then there is no error.

However, if there is only a version of the package that conflicts with virtualenv’s requirements installed then the error discussed above is thrown.

So to see this specific error you would likely need to fall under the following conditions:

• Use importlib-metadata in your project (directly or as a dependency)
• Use a version of python in your project for which virtualenv requires importlib-metadata <2 (e.g. 3.7)
• Not have importlib-metadata 1.7 or earlier installed elsewhere so that there is a non-conflicting version of the package in the working set.

Without fully understanding the logic of adding pipenv to the project environment, it is hard to know the right point at which to address this issue.

However, it would seem that the dependency check for pipenv should use the version of python which will be used to run pipenv. Then an error would reflect a real problem, rather than the current situation.

Alternatively, error handling around adding pipenv to the environment could handle VersionConflict errors. Either silently, which would assume that pipenv was correctly installed with its dependencies, but that might obviate the point of adding it to the environment. Or perhaps by installing required dependencies for pipenv for the project version of python.

I wonder if there are plans for the pipenv to use the new built-in pip dependency resolver.

pip uninstall -y importlib-metadata and run pipenv lock again work for me.

Following @scuerda comment it works for me to fix importlib-metadata 3.0.0 with last pipenv version, hope it helps

The workaround of deleting the Pipfile.lock file and then re-running pipenv lock always seems to work but here is what I did for a more permanent fix.

1. Destroy local pipenv virtualenv pipenv --rm
2. Uninstall pipenv brew uninstall pipenv
3. Upgrade virtualenv pip install -U virtualenv
4. Reinstall pipenv pip install --user pipenv
5. Recreate pipenv virtualenv pipenv install

Repeat runs of pipenv lock work as expected and do not re-trigger the conflict error message.