pip: pip install -e does not install setup.py install_requires dependencies

  • Pip version: 9.0.1
  • Python version: 3.6.3
  • Operating system: Gentoo Linux

Description:

I have a local package that specifies a number of dependencies via the install_requires argument in setup.py. Installing it with pip install --user -e . does not install these dependencies, whereas pip install --user . does.

What I’ve run:

Consider the case where one of the install_requires dependencies (here gitpython) is not present on the system. Below is the output of pip install in both cases. Note that gitpython is only collected and installed in the second case.

➜  package git:(master) ✗ pip install --user -e .
Obtaining file:///home/leonard/projects/package
Requirement already satisfied: mxnet in /home/leonard/software/mxnet/python (from leezu-package==0.1.dev0)
Requirement already satisfied: numpy in /usr/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: addict in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: numba in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: pyyaml in /usr/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: spacy in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: en_core_web_sm in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: en_core_web_md in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: matplotlib in /usr/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: pandas in /usr/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: xarray in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: netCDF4 in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: progressbar2 in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: requests in /home/leonard/.local/lib64/python3.6/site-packages (from mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: graphviz in /home/leonard/.local/lib64/python3.6/site-packages (from mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: llvmlite in /usr/lib64/python3.6/site-packages (from numba->leezu-package==0.1.dev0)
Requirement already satisfied: murmurhash<0.27,>=0.26 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: cymem<1.32,>=1.30 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: preshed<2.0.0,>=1.0.0 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: thinc<6.6.0,>=6.5.0 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: plac<1.0.0,>=0.9.6 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: pip<10.0.0,>=9.0.0 in /usr/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: six in /usr/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: pathlib in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: ujson>=1.35 in /usr/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: dill<0.3,>=0.2 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: regex<2017.12.1,>=2017.4.1 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: ftfy<5.0.0,>=4.4.2 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: python-dateutil in /usr/lib64/python3.6/site-packages (from matplotlib->leezu-package==0.1.dev0)
Requirement already satisfied: pytz in /usr/lib64/python3.6/site-packages (from matplotlib->leezu-package==0.1.dev0)
Requirement already satisfied: cycler>=0.10 in /usr/lib64/python3.6/site-packages (from matplotlib->leezu-package==0.1.dev0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=1.5.6 in /usr/lib64/python3.6/site-packages (from matplotlib->leezu-package==0.1.dev0)
Requirement already satisfied: python-utils>=2.1.0 in /home/leonard/.local/lib64/python3.6/site-packages (from progressbar2->leezu-package==0.1.dev0)
Requirement already satisfied: idna<2.7,>=2.5 in /home/leonard/.local/lib64/python3.6/site-packages (from requests->mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/lib64/python3.6/site-packages (from requests->mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib64/python3.6/site-packages (from requests->mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/lib64/python3.6/site-packages (from requests->mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: wrapt in /home/leonard/.local/lib64/python3.6/site-packages (from thinc<6.6.0,>=6.5.0->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: tqdm<5.0.0,>=4.10.0 in /home/leonard/.local/lib64/python3.6/site-packages (from thinc<6.6.0,>=6.5.0->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: cytoolz<0.9,>=0.8 in /home/leonard/.local/lib64/python3.6/site-packages (from thinc<6.6.0,>=6.5.0->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: termcolor in /home/leonard/.local/lib64/python3.6/site-packages (from thinc<6.6.0,>=6.5.0->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: wcwidth in /usr/lib64/python3.6/site-packages (from ftfy<5.0.0,>=4.4.2->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: html5lib in /home/leonard/.local/lib64/python3.6/site-packages (from ftfy<5.0.0,>=4.4.2->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: toolz>=0.8.0 in /home/leonard/.local/lib64/python3.6/site-packages (from cytoolz<0.9,>=0.8->thinc<6.6.0,>=6.5.0->spacy->leezu-package==0.1.dev0)
Installing collected packages: leezu-package
  Running setup.py develop for leezu-package
Successfully installed leezu-package



➜  package git:(master) ✗ pip install --user .                                                                                                                                <<<
Processing /home/leonard/projects/package
  Requirement already satisfied (use --upgrade to upgrade): leezu-package==0.1.dev0 from file:///home/leonard/projects/package in ./src
Requirement already satisfied: mxnet in /home/leonard/software/mxnet/python (from leezu-package==0.1.dev0)
Requirement already satisfied: numpy in /usr/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: addict in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: numba in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: pyyaml in /usr/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: spacy in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: en_core_web_sm in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: en_core_web_md in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: matplotlib in /usr/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: pandas in /usr/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: xarray in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: netCDF4 in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: progressbar2 in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: traitlets>=5.0.0.dev in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: singleton-decorator in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Requirement already satisfied: visdom in /home/leonard/.local/lib64/python3.6/site-packages (from leezu-package==0.1.dev0)
Collecting gitpython (from leezu-package==0.1.dev0)
  Using cached GitPython-2.1.7-py2.py3-none-any.whl
Requirement already satisfied: requests in /home/leonard/.local/lib64/python3.6/site-packages (from mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: graphviz in /home/leonard/.local/lib64/python3.6/site-packages (from mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: llvmlite in /usr/lib64/python3.6/site-packages (from numba->leezu-package==0.1.dev0)
Requirement already satisfied: murmurhash<0.27,>=0.26 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: cymem<1.32,>=1.30 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: preshed<2.0.0,>=1.0.0 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: thinc<6.6.0,>=6.5.0 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: plac<1.0.0,>=0.9.6 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: pip<10.0.0,>=9.0.0 in /usr/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: six in /usr/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: pathlib in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: ujson>=1.35 in /usr/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: dill<0.3,>=0.2 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: regex<2017.12.1,>=2017.4.1 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: ftfy<5.0.0,>=4.4.2 in /home/leonard/.local/lib64/python3.6/site-packages (from spacy->leezu-package==0.1.dev0)
Requirement already satisfied: python-dateutil in /usr/lib64/python3.6/site-packages (from matplotlib->leezu-package==0.1.dev0)
Requirement already satisfied: pytz in /usr/lib64/python3.6/site-packages (from matplotlib->leezu-package==0.1.dev0)
Requirement already satisfied: cycler>=0.10 in /usr/lib64/python3.6/site-packages (from matplotlib->leezu-package==0.1.dev0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=1.5.6 in /usr/lib64/python3.6/site-packages (from matplotlib->leezu-package==0.1.dev0)
Requirement already satisfied: python-utils>=2.1.0 in /home/leonard/.local/lib64/python3.6/site-packages (from progressbar2->leezu-package==0.1.dev0)
Requirement already satisfied: ipython_genutils in /usr/lib64/python3.6/site-packages (from traitlets>=5.0.0.dev->leezu-package==0.1.dev0)
Requirement already satisfied: decorator in /home/leonard/.local/lib64/python3.6/site-packages (from traitlets>=5.0.0.dev->leezu-package==0.1.dev0)
Requirement already satisfied: pillow in /usr/lib64/python3.6/site-packages (from visdom->leezu-package==0.1.dev0)
Requirement already satisfied: tornado in /home/leonard/.local/lib64/python3.6/site-packages (from visdom->leezu-package==0.1.dev0)
Requirement already satisfied: pyzmq in /usr/lib64/python3.6/site-packages (from visdom->leezu-package==0.1.dev0)
Requirement already satisfied: torchfile in /home/leonard/.local/lib64/python3.6/site-packages (from visdom->leezu-package==0.1.dev0)
Requirement already satisfied: gitdb2>=2.0.0 in /home/leonard/.local/lib64/python3.6/site-packages (from gitpython->leezu-package==0.1.dev0)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib64/python3.6/site-packages (from requests->mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/lib64/python3.6/site-packages (from requests->mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: idna<2.7,>=2.5 in /home/leonard/.local/lib64/python3.6/site-packages (from requests->mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /usr/lib64/python3.6/site-packages (from requests->mxnet->leezu-package==0.1.dev0)
Requirement already satisfied: wrapt in /home/leonard/.local/lib64/python3.6/site-packages (from thinc<6.6.0,>=6.5.0->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: tqdm<5.0.0,>=4.10.0 in /home/leonard/.local/lib64/python3.6/site-packages (from thinc<6.6.0,>=6.5.0->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: cytoolz<0.9,>=0.8 in /home/leonard/.local/lib64/python3.6/site-packages (from thinc<6.6.0,>=6.5.0->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: termcolor in /home/leonard/.local/lib64/python3.6/site-packages (from thinc<6.6.0,>=6.5.0->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: html5lib in /home/leonard/.local/lib64/python3.6/site-packages (from ftfy<5.0.0,>=4.4.2->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: wcwidth in /usr/lib64/python3.6/site-packages (from ftfy<5.0.0,>=4.4.2->spacy->leezu-package==0.1.dev0)
Requirement already satisfied: olefile in /usr/lib64/python3.6/site-packages (from pillow->visdom->leezu-package==0.1.dev0)
Requirement already satisfied: smmap2>=2.0.0 in /home/leonard/.local/lib64/python3.6/site-packages (from gitdb2>=2.0.0->gitpython->leezu-package==0.1.dev0)
Requirement already satisfied: toolz>=0.8.0 in /home/leonard/.local/lib64/python3.6/site-packages (from cytoolz<0.9,>=0.8->thinc<6.6.0,>=6.5.0->spacy->leezu-package==0.1.dev0)
Installing collected packages: gitpython
Successfully installed gitpython-2.1.7

About this issue

  • Original URL
  • State: open
  • Created 7 years ago
  • Reactions: 14
  • Comments: 24 (8 by maintainers)

Most upvoted comments

Hey @pradyunsg @xoviat . I have investigated this issue a bit further and found it was related to an project_name.egg-info folder next to the setup.py whose ./project_name.egg-info/requires.txt didn’t contain the up to date requirments.

Which means this bug boils down to:

  • pip install -e uses the project_name.egg-info folder in the directory of ./setup.py and just reads the (possibly outdated) ./project_name.egg-info/requires.txt.
  • pip install correctly parses setup.py and updates ./src/project_name.egg-info/requires.txt (assuming a layout where the code is in ./src/project_name )

Furthermore I found renaming the outdated project_name.egg-info folder to e.g. test.egg-info will not stop pip install -e from reading the (now not only outdated but also not wrongly named) test.egg-info/requires.txt file.

The following setup.py is sufficient to reproduce the bug:

from setuptools import setup
setup(name='project', install_requires=['gitpython'])

with the following files present in a test.egg-info folder in the same directory: PKG-INFO, requires.txt.

Not sure if the usage of the wrongly named egg-info file constitutes a bug. However I would expect pip install -e to at least double check the requires.txt file it uses with the contents of setup.py?

+7
leezu on Oct 17, 2017

I stumbled into this too today. Using the latest pip 19.0.3 on Python 3.7.

I edited the list of install_requires in setup.py to add some-new-package and re-ran:

pip install -e .

But the new package wasn’t installed. 😦

The package was mentioned in inside remote_settings_uptake_health.egg-info/requires.txt but the comments above got me wondering if the reason was because I had two *.egg-info/ directories. Indeed I had.

The name key in setup(name=THIS, ...) had indeed changed and I suspect pip incorrectly looked at old_name.egg-info/requires.txt.

So I think the steps to reproduce are as follows:

  1. Create a new package with just a setup.py file:

    ▶ cat setup.py from setuptools import setup

    setup(name=“name-one”, install_requires=[“click”])

  2. Install things:

▶ pip install -e .
Obtaining file:///Users/peterbe/dev/PYTHON/reproduce-pypa-issue-4780
Collecting click (from name-one==0.0.0)
  Using cached https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl
Installing collected packages: click, name-one
  Running setup.py develop for name-one
Successfully installed click-7.0 name-one
  1. Change your mind about the name and the list of requires:
▶ cat setup.py
from setuptools import setup

setup(name="name-one-v2", install_requires=["click", "requests"])
  1. Try to install again:
▶ pip install -e .
Obtaining file:///Users/peterbe/dev/PYTHON/reproduce-pypa-issue-4780
Requirement already satisfied: click in /Users/peterbe/virtualenvs/reproduce-pypa-issue-4780/lib/python3.7/site-packages (from name-one==0.0.0) (7.0)
Installing collected packages: name-one
  Found existing installation: name-one 0.0.0
    Uninstalling name-one-0.0.0:
      Successfully uninstalled name-one-0.0.0
  Running setup.py develop for name-one
Successfully installed name-one

No requests installed this time!! Also, not the console output mentions “Installing collected packages: name-one” but that’s not the name in my setup.py.

(Note; I edited the comment mentioning that the reason why it picks different *.egg-info/requires.txt might be related to the alphabetical order. I don’t think that was right.)

+3
peterbe on Feb 25, 2019

I wonder if this is the problem I described in https://github.com/pypa/pip/pull/9147#discussion_r527750168? If editable requirements are always re-installed, setuptools would have a chance to re-build the egg-info directory when pip install -e is re-run.

Note that this may or may not be what we want, however, without PEP 517 standardising editable and incremental builds. Running setup.py can have a significant performance impact (even only for egg_info), and I can easily imagine projects requiring binary compilation may be relying on the current behaviour in their development workflow.

+1
uranusjr on Nov 23, 2020

Sorry, I checked again: So just having an outdated ./src/project_name.egg-info/requires.txt is actually no problem. pip install -e . seems to update that correctly.

So the bug here really boils down to pip install -e . getting confused by the leftover ./project_name.egg-info/requires.txt which was created before I switched my project layout from having all code in ./src to having all code in ./src/project_name as advocated by https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure

While this is arguably an edge case, I think pip install -e . shouldn’t be confused by it and at least double check the requires.txt file matches the setup.py requirements… Or it should warn the user about the presence of this “wrong” ./project_name.egg-info folder.

What do you think?

+1
leezu on Oct 17, 2017
Read more comments on GitHub
← pip: pip install --user misbehaves inside venv
pip: pip install -e inconsistent checkout with git url and branch →