pip: Cannot install into user site directory with editable source.

Following the tip from @agoose77, I’m putting this in my setup.py as a workaround:

import site
import sys
site.ENABLE_USER_SITE = "--user" in sys.argv[1:]
...

@ssbarnea PEP 517 doesn’t support editable installs, so I don’t see how PEP 517 is relevant here?

I don’t know, but doing a

pip install --user --no-use-pep517 -e .

succeeds, and after that a

pip install --user -e .

doesn’t fail anymore. However, this still fails:

pip install --user --force-reinstall -e .

Nobody was able to fix this in pip? The whole idea of pep517 was to reduce use of setup.py ending in its removal, not adding more junk into to it.

Another workaround is --prefix=~/.local rather than --user

I’ve been using the workaround mentioned in https://github.com/pypa/pip/issues/7953#issuecomment-645133255 for a while now:

site.ENABLE_USER_SITE = "--user" in sys.argv[1:]

but it stopped working recently.

The good news is that I don’t need that old workaround anymore with this new workaroud:

python3 -m pip install --prefix=$(python3 -m site --user-base) -e .

that I found here: https://github.com/pypa/setuptools/issues/3063#issuecomment-1026030252 (-- same workaround as in @mixmastamyk post https://github.com/pypa/pip/issues/7953#issuecomment-1027612802)

EDIT: I forgot about my own workaround here (in https://github.com/pypa/pip/pull/9990):

python3 -m pip install --no-build-isolation -e <path>

– it also works.

I have seen this failing for a project that uses pyproject.toml (PEP517 install flow), where it works fine for projects without pyproject.toml. @siddalmia has described the same behavior in https://github.com/pytorch/fairseq/issues/1977. I wonder if there is something in the PEP517 install flow that breaks this…?

Hmm, I’ve been using this functionality for years. “We broke it and you are welcome to fix it” is not particularly neighborly.

How 'bout a link to the broken function or general ballpark?

P.S. This got me thru the install:

pip install --prefix=~/.local -e .

Ran into this again today. Very surprised, that this issue is still not fixed.

Hit this issue today trying out the currently-recommended way to package Python code for a new package. Everything works fine exclusively using the previously-recommended setup.py, but trying to use a setup.cfg or even including a pyproject.toml results in failing to install editable source. Unfortunate that this issue has been around for more than a year now…

Ran into this today on debian 9, not sure what has changed but we have been using pip3 install --user -e . for a long time for development purposes, it is strange for this to suddenly stop working.

@pradyunsg

# 0) Prepare a reproducible environment, you can probably skip this step, but just in case
docker run -it python:3.6 /bin/bash
useradd -m -s /bin/bash someone
su someone
cd $(mktemp -d)

# 1) Prepare setup.py
cat >setup.py <<EOF
from setuptools import setup

if __name__ == "__main__":
    setup()
EOF

# 2) Prepare setup.cfg
cat >setup.cfg <<EOF
[metadata]
name = thepackage
[options]
package_dir =
    =src
packages = find_namespace:
[options.packages.find]
where = src
EOF

# 3) Prepare pyproject.toml
cat >pyproject.toml <<EOF
[build-system]
build-backend = "setuptools.build_meta"
requires = ["setuptools", "wheel"]
EOF

# 4) Prepare dummy package data
mkdir src
mkdir src/thepackage
touch src/thepackage/__init__.py

# 5) Boom!
pip install --user --editable .

Not sure if this is a minimal example, probably not, but I don’t have the time to simplify this any further.

If I try --no-use-pep517 I get

ERROR: Disabling PEP 517 processing is invalid: project specifies a build backend of setuptools.build_meta in pyproject.toml

The only workaround I’ve found so far is to “hide” pyproject.toml temporarily, like

mv pyproject.toml pyproject.nope
pip install -e .  # etc
mv pyproject.nope pyproject.toml

While this feels a little gross, I guess this is “fine” in the sense that there isn’t much use for me having a pyproject.toml if I’m going to do an editable install anyway. I also suppose this means PEP517 might still be used for other packages that are dependencies, which would be a good thing.

the setup.py is deprecated and the new proposed way is not working at the same time. What should we do now? You should not mark it deprecated at least.

Are there any plans to fix this?

Not in the sense that any of the pip maintainers is currently expecting to work on it, no. It’s simply a matter of too much work and not enough people, though - there’s no objection in principle to a fix.

We’re really relying on the possibility of someone who is actually affected by the issue having the time and willingness to contribute a fix, and we can’t really plan for when or if that might happen. That’s just the nature of open source, I’m afraid.

I also ran into this today. Is this intentionally not supported? I’m not sure why, I’d find it useful.

Raised bugs against distros shipping outdated setuptools:

• https://bugs.launchpad.net/ubuntu/+source/setuptools/+bug/1994016 Ubuntu 22.04 and 22.10
• Fedora does not need it as it includes newer setuptools with 37 (up to release tomorrow)

@StanczakDominik the minimal reproducible example, that I provided earlier in this thread is still broken. Here it is, updated for the latest pip/python versions:

# 0) Prepare a reproducible environment, you can probably skip this step, but just in case
# Run these commands line-by-line (they spawn new shells, you can't copy paste the whole block)
docker run -it python:3.10 /bin/bash
pip install --upgrade pip
useradd -m -s /bin/bash someone
su someone
cd $(mktemp -d)

# 1) Prepare setup.py
cat >setup.py <<EOF
from setuptools import setup

if __name__ == "__main__":
    setup()
EOF

# 2) Prepare setup.cfg
cat >setup.cfg <<EOF
[metadata]
name = thepackage
[options]
package_dir =
    =src
packages = find_namespace:
[options.packages.find]
where = src
EOF

# 3) Prepare pyproject.toml
cat >pyproject.toml <<EOF
[build-system]
build-backend = "setuptools.build_meta"
requires = ["setuptools", "wheel"]
EOF

# 4) Prepare dummy package data
mkdir src
mkdir src/thepackage
touch src/thepackage/__init__.py

# 5) Boom!
pip --version
pip install --user --editable .

The pip install --user --editable . fails with the following output:

Obtaining file:///tmp/tmp.sPIIeXNutX
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Installing collected packages: thepackage
  Running setup.py develop for thepackage
    error: subprocess-exited-with-error

    × python setup.py develop did not run successfully.
    │ exit code: 1
    ╰─> [32 lines of output]
        running develop
        /tmp/pip-build-env-d3bklpt1/overlay/lib/python3.10/site-packages/setuptools/command/easy_install.py:160: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
          warnings.warn(
        WARNING: The user site-packages directory is disabled.
        /tmp/pip-build-env-d3bklpt1/overlay/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
          warnings.warn(
        error: can't create or remove files in install directory

        The following error occurred while trying to add or remove files in the
        installation directory:

            [Errno 13] Permission denied: '/usr/local/lib/python3.10/site-packages/test-easy-install-39.write-test'

        The installation directory you specified (via --install-dir, --prefix, or
        the distutils default setting) was:

            /usr/local/lib/python3.10/site-packages/

        Perhaps your account does not have write access to this directory?  If the
        installation directory is a system-owned directory, you may need to sign in
        as the administrator or "root" account.  If you do not have administrative
        access to this machine, you may wish to choose a different installation
        directory, preferably one that is listed in your PYTHONPATH environment
        variable.

        For information on other options, you may wish to consult the
        documentation at:

          https://setuptools.pypa.io/en/latest/deprecated/easy_install.html

        Please make the appropriate changes for your system and try again.

        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [32 lines of output]
    running develop
    /tmp/pip-build-env-d3bklpt1/overlay/lib/python3.10/site-packages/setuptools/command/easy_install.py:160: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    WARNING: The user site-packages directory is disabled.
    /tmp/pip-build-env-d3bklpt1/overlay/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    error: can't create or remove files in install directory

    The following error occurred while trying to add or remove files in the
    installation directory:

        [Errno 13] Permission denied: '/usr/local/lib/python3.10/site-packages/test-easy-install-39.write-test'

    The installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:

        /usr/local/lib/python3.10/site-packages/

    Perhaps your account does not have write access to this directory?  If the
    installation directory is a system-owned directory, you may need to sign in
    as the administrator or "root" account.  If you do not have administrative
    access to this machine, you may wish to choose a different installation
    directory, preferably one that is listed in your PYTHONPATH environment
    variable.

    For information on other options, you may wish to consult the
    documentation at:

      https://setuptools.pypa.io/en/latest/deprecated/easy_install.html

    Please make the appropriate changes for your system and try again.

    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

Seeing this on macOS as well.

Funnily enough, this seems to work for me after pip install --upgrade pip grabbing pip 22.0.3. Can anyone affected reproduce that? If so, could this issue be closed?

Can the pip maintainers at least clarify, why is the user site being disabled in the first place?

From our (the users) perspective, it seems that the problem is that the user directory is disabled for some reason. As outlined in the OPs post, the fix can be as simple as removing the 'PYTHONNOUSERSITE': '1', line from src/pip/_internal/build_env.py. However, this fix ignores the original reason for disabling the user install site. What was that reason?

This isolation mechanism, was originally introduced in PR #4144 in commit 92bfdf702f2cc867f27e98d3eae1e930bdd25edc “Isolate wheel builds when using PEP 518 build dependencies”. My best guess is that PYTHONNOUSERSITE is used so that the build dependencies are installed into the temporary prefix instead of the user site, but I am not 100% sure. One more thing that I don’t understand is why does setuptools/easy_install try to install the editable package during the build step.

Also also, @pfmoore earlier in this thread you mentioned that PEP 517 doesn’t support editable installs. If my understanding is correct, this is no longer true after PEP 660 got accepted. Maybe, the actual problem here is that setuptools currently doesn’t implement PEP660 pypa/setuptools#2816?

Issue still present with pip version pip 21.1.2.

I would encourage people affected by this to dig into the source. The code path described does not make much sense. The code path for an editable install should not go through build_env.py at all, so there is either a misidentification to the cause, or a more obscure bug than the sympton shows.

Facing this with root installs into non-user directory on ubuntu 20 + python 3.8. Eaten a ton of my time.

I will take a look, but I don’t know, how pip internally works (e.g. why is that class only entered when pyproject.toml exists, why disabling the user install with an envvariable, why does the reset not work or is the actual installation done in that context, …).

Not sure, if I am able to understand enough of pip in a reasonable time. If I get an idea how to fix it, I will open a PR.

You’re welcome to step up and fix this, because you clearly care enough to step up and complain that this hasn’t been fixed.

@pfmoore you can support an editable install in a project using PEP 517 by adding a rather minimal setup.py.

But then, it is impossible to have -e and --user