setuptools: [BUG] setuptools>=60.0.0 causes virtualenv to create abnormal tree structure on ubuntu 22.04

setuptools version

setuptools 60, 61, 62

59.x was the last version I found to not cause trouble. This also happens to be the default version installed with ubuntu 22.04 via apt.

Python version

Python 3.10

OS

Ubuntu 22.04

Also, I was unable to reproduce this on python:3.10-buster

Additional environment information

No response

Description

Installing setuptools==62.1.0 causes virtualenv venv to create an abnormal directory structure. Uninstalling setuptools solves the problem. This is 100% reproducible in ubuntu:22.04 docker image (ubuntu@sha256:06b5d30fabc1fc574f2ecab87375692299d45f8f190d9b71f512deb494114e1f)

The failure of virtualenv also affects poetry, which has no error checking for venv activate and will attempt to modify system packages when it thinks the venv is activated but it is not.

Without setuptools, venv/bin exists. With setuptools, venv/local/bin exists and no venv/bin dir exists.

Expected behavior

Setuptools does not interfere with virtualenv directory structure.

How to Reproduce

  1. Create Dockerfile:
    FROM ubuntu:22.04
    
    RUN apt-get update
    RUN apt-get install -y python3 python3-pip python3-virtualenv tree
    RUN pip install setuptools==62.1.0 ; \
        virtualenv venv ; \
        tree -L 3 venv ; \
        . venv/bin/activate
    
  2. Build it.
  3. Notice error.

Output

$ cat Dockerfile
FROM ubuntu:22.04

RUN apt-get update
RUN apt-get install -y python3 python3-pip python3-virtualenv tree
RUN pip install setuptools==62.1.0 ; \
    virtualenv venv ; \
    tree -L 3 venv ; \
    . venv/bin/activate
$ docker build -t test1 .
Sending build context to Docker daemon  24.01MB
Step 1/4 : FROM ubuntu:22.04
 ---> f0b07b45d05b
Step 2/4 : RUN apt-get update
 ---> Using cache
 ---> f3a881e3158b
Step 3/4 : RUN apt-get install -y python3 python3-pip python3-virtualenv tree
 ---> Using cache
 ---> 1b02afb3b2a1
Step 4/4 : RUN pip install setuptools==62.1.0 ;     virtualenv venv ;     tree -L 3 venv ;     . venv/bin/activate
 ---> Running in 5a7b54dc3a32
Collecting setuptools==62.1.0
  Downloading setuptools-62.1.0-py3-none-any.whl (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 8.8 MB/s eta 0:00:00
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 59.6.0
    Not uninstalling setuptools at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'setuptools'. No files were found to uninstall.
Successfully installed setuptools-62.1.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
created virtual environment CPython3.10.4.final.0-64 in 308ms
  creator CPython3Posix(dest=/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==22.0.2, setuptools==59.6.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
venv
|-- lib
|   `-- python3.10
|-- local
|   |-- bin
|   |   |-- activate
|   |   |-- activate.csh
|   |   |-- activate.fish
|   |   |-- activate.nu
|   |   |-- activate.ps1
|   |   |-- activate_this.py
|   |   |-- deactivate.nu
|   |   |-- pip
|   |   |-- pip-3.10
|   |   |-- pip3
|   |   |-- pip3.10
|   |   |-- python -> /usr/bin/python3
|   |   |-- python3 -> python
|   |   |-- python3.10 -> python
|   |   |-- wheel
|   |   |-- wheel-3.10
|   |   |-- wheel3
|   |   `-- wheel3.10
|   `-- lib
|       `-- python3.10
`-- pyvenv.cfg

6 directories, 19 files
/bin/sh: 1: .: cannot open venv/bin/activate: No such file
The command '/bin/sh -c pip install setuptools==62.1.0 ;     virtualenv venv ;     tree -L 3 venv ;     . venv/bin/activate' returned a non-zero code: 2

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 14
  • Comments: 19 (8 by maintainers)

Commits related to this issue

Most upvoted comments

Having this issue on WSL Ubuntu 22.04. This export SETUPTOOLS_USE_DISTUTILS=stdlib works around the issue.

I just upgraded multiple computers to Ubuntu 22.04 and all of them have this problem. Replacing os.environ.get(var, 'local') with os.environ.get(var, 'stdlib') fixes the problem.

I’ve filed a bug with Debian about this problem, since as far as I can tell nobody explicitly informed them that there is a problem: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1022749

I used this dockerfile:

from ubuntu:22.04
RUN apt update
RUN apt install -y python3 python3-pip python3-virtualenv tree
RUN pip install -U setuptools
RUN virtualenv venv
CMD tree -L 3 venv

Because it allows me to see the failed expectation but also replace it and inspect the environment.

The problem can be observed by adding a breakpoint to virtualenv/discovery/py_info.py:88:

root@9b995eb67229:~# cat /usr/lib/python3/dist-packages/virtualenv/discovery/py_info.py | head -n 95 | tail -n 15
            # we cannot use distutils at all if "venv" exists, distutils don't know it
            self.distutils_install = {}
        else:
            self.sysconfig_scheme = None
            self.sysconfig_paths = {u(i): u(sysconfig.get_path(i, expand=False)) for i in sysconfig.get_path_names()}
            self.distutils_install = {u(k): u(v) for k, v in self._distutils_install().items()}

        breakpoint()

        # https://bugs.python.org/issue22199
        makefile = getattr(sysconfig, "get_makefile_filename", getattr(sysconfig, "_get_makefile_filename", None))
        self.sysconfig = {
            u(k): u(v)
            for k, v in [
                # a list of content to store from sysconfig

And inspecting the value:

root@9b995eb67229:~# python3 -m virtualenv venv
> /usr/lib/python3/dist-packages/virtualenv/discovery/py_info.py(91)__init__()
-> makefile = getattr(sysconfig, "get_makefile_filename", getattr(sysconfig, "_get_makefile_filename", None))
(Pdb) self.distutils_install
{'purelib': 'local/lib/python3.10/dist-packages', 'platlib': 'local/lib/python3.10/dist-packages', 'headers': 'usr/include/python3.10', 'scripts': 'local/bin', 'data': 'local'}

The fact that scripts is local/bin is probably what is leading virtualenv to install executables to that directory.

My guess is that something about the assumptions made between _distutils_system_mod and virtualenv are in conflict.

Probably what we need is someone from virtualenv to analyze what expectation is being missed here and determine what’s the best way to meet that expectation. @gaborbernat Do you have any advice here?

I also observe that virtualenv has a codepath that bypasses distutils if a venv scheme is supplied. Perhaps the best thing would be for debian to supply that install scheme. @FFY00 Do you have a recommendation on what Debian should be doing to support a virtualenv?

@doko42, It does appear to me you authored the patch. Will you be able to help work through this issue?

setuptools 60 didn’t ship in Ubuntu 22.04, it came later in 23.04.

The setuptools solution to this problem is the addition of the venv scheme to the python interpreter. That isn’t something that is reasonable to do in a stable Ubuntu, to resolve a bug like this, that occurs only when a package is updated to a version newer than Ubuntu ships. That’s outside of what’s reasonable for Ubuntu support.

I suggest leaving your system setuptools as shipped by Ubuntu, and using the system virtualenv to create virtualenvs. Use the newer setuptools inside virtualenvs.

Or use venv instead of virtualenv.

Workaround / fix

I have verified with 62.1.0 that changing $HOME/.local/lib/python3.10/site-packages/distutils-precedence.pth from:

import os; var = 'SETUPTOOLS_USE_DISTUTILS'; enabled = os.environ.get(var, 'local') == 'local'; enabled and __import__('_distutils_hack').add_shim();

to

import os; var = 'SETUPTOOLS_USE_DISTUTILS'; enabled = os.environ.get(var, 'stdlib') == 'local'; enabled and __import__('_distutils_hack').add_shim();

Solves the problem.

See also