pip: pip install --user misbehaves inside venv

The issue of this bug is not to do with explicit specification of --user on the command line. The issue is that a user having user mode set in his ~/.config/pip/pip.conf currently can NEVER use pip inside a virtualenv, because it just ignores the virtualenv and installs to ~/.local. That is why I described the bug above as “serious”. It seems to me that pip should ignore the user mode setting in ~/.config/pip/pip.conf when it is run inside a virtualenv.

Whether pip errors out on an explicit --user when run within a virtualenv, or just installs to ~/.local as the user has explicitly asked, is another matter and another debate. Personally I think pip in that case should respect the added --user.

It is arguably a design error that by default pip installs globally (see the long running and still open debate in issue #1668) so many users just set the user default in their config and forget about it, like I did a month or so ago then scratched my head for ages why my next virtualenv was not getting set up correctly.

Dropping this from 18.1, since this is not something introduced in 18.0 and there’s no clear plan on how to fix this within pip anyway.

If someone could try it out and let me know if that works well for them, that’ll be great!

With #7155, pip install --user inside a venv now throws the expected error “Can not perform a --user install. …”

Does the following serve as a demonstration?

1. Install Ubuntu 18.04, install systems pythons (2.7 and 3.6.8), install system pip. system pip is v.9.0.1

2. install/upgrade pip in the user space. If i am not mistaken, i did it with the following command:

$ pip install --user --upgrade pip
$ pip --version
=> 19.2.3

1. scenario that works (you can skip it)

$ mkdir bigbang; cd bigbang
$ python3 -m venv pip-no-user
	
$ . pip-no-user/bin/activate
$ which pip
=> /path/to/bigbang/pip-no-user/bin/pip

$ pip --version
=> 9.0.1 # this is system pip in ubuntu 18.04

$ pip install --upgrade pip
# upgrades the version inside virtual environment

$ pip --version
=> 19.2.3

$ pip list | grep tabulate
=> nothing, as expected

$ pip install tabulate
$ pip list | grep tabulate
=> tabulate      0.8.3 

# lets do a closer test by loading the library
$ python
>>> import tabulate
=> OK

# type ctrl+d to exit repl

$ deactivate

conclusion: as expected, the package tabulate was installed in the virtual environment

1. configure pip to always perform user install. this is accomplished by adding the following to the configuration file $HOME/.pip/pip.conf

[install]
user = true

Lets do some testing

$ pip list | grep tabulate
=> nothing

$ pip install tabulate
$ pip install tabulate
=> Requirement already satisfied: tabulate in HOME/.local/lib/python3.6/site-packages (0.8.3)

$ pip list | grep tabulate
=> tabulate      0.8.3 

$ python3
>>> import tabulate
=>OK

Conclusion: the package was successfully installed into user space without explicitly giving --user option to pip

1. Based on the setup outlined in 4). This is the failing scenario: when in virtual environment, the libraries are not installed in the environment but rather in user space (HOME/.local).

$ python3 -m venv pip-user-true
$ . pip-user-true/bin/activate
$ pip --version
=> pip 9.0.1

At this point one could run

$ pip install --upgrade pip

but i am not doing it, because it fucks up HOME/.local/bin/pip by changing shebang to be #!/path/to/bigbang/pip-user-true/bin/python3

The question is why HOME/.local/bin/pip was updated if environment-specific pip should have been updated, namely this one: /path/to/bigbang/pip-user-true/bin/pip.

Next we check if tabulate (installed in step 4 in user space) is available:

$ pip list |grep tabulate
=> nothing
	
$ python
>>> import tabulate
=> ModuleNotFoundError: No module named 'tabulate'

Conclusion: as expected, virtual environment isolates current project from user space.

And finally the painful part.

$ pip install tabulate
=> Collecting tabulate
=> Cache entry deserialization failed, entry ignored
=> Installing collected packages: tabulate
=> Successfully installed tabulate-0.8.3

Weird message ‘Cache entry deserialization failed, entry ignored’. Heh.

$ pip list | grep tabulate
=> nothing... weird

$ python
>>> import tabulate
=> ModuleNotFoundError: No module named 'tabulate'

It smells bad. Lets continue the journey. Installing tabulate library also drops a command line tool. It should be in the environment bin /path/to/bigbang/pip-user-true/bin/tabulate but in reality it is found in user bin:

$ ls HOME/.local/bin/tabulate
=> ok

Lets look at the script more closely:

$ head -n1 HOME/.local/bin/tabulate
=> #!/path/to/bigbang/pip-user-true/bin/python3

This is weird! Shebang is environment-specific but the script is in user space. Something is wrong here.

And the last chance. Try loading the library we (think we) have just installed.

$ python
>>> import tabulate
ModuleNotFoundError: No module named 'tabulate'

hmmm. It is not available in the environment.

1. Scratching the head… Undoing modifications in .pip/pip.conf for the benefit of everything.

Sorry for the long read.

@piotr-dobrogost Agreed. I’d be against having behaviour differ based on where the user specifies --user. What about an environment variable? Should a persistent environment variable be treated differently than PIP_USER=yes pip install foo? I don’t think we want to start down this route…

I understand that people see setting --user in a config file as being somehow different in intention, as more of a choice to default to user than a request to always force user, but that’s just not what it is.

many users just set the user default in their config

And that’s the problem - setting user=true in the pip config file is not “setting the user default”, rather it’s “asking pip to use --user for all installs”, which is subtly (but significantly) different. I guess there may be some value in adding a warning note to the docs explaining that point.

Having user installs be the default is what #1668 is about, and discussions about that should take place over there.

Thank you for that! It helps a lot!

–user is supposed to abort when installing from within an activated virtualenv. That seems to not be happening.

I’ll look into why that’s happening and print an error message instead of doing the wrong thing.

I have run into this issue with pip 19.0.3 and 19.2.2. Spent the whole day trying to figure out why packages do not get installed while in the virtual environment. It is a pity that such important issues are not fixed. I think we need another PEP for this situation… 😦