virtualenv: Scripts installed with wrong shebang under subprocess

On Python 3.5.1 but not on 2.7.10, when I invoke pip through a subprocess in a virtualenv, the scripts are created with the wrong shebang and thus fail to run:

$ cat > invoke.py
import sys
import subprocess

subprocess.Popen(sys.argv[1:]).wait()
$ rm -Rf env
$ python -m virtualenv --version
14.0.0
$ python --version
Python 3.5.1
$ python -m virtualenv env
Using base prefix '/Library/Frameworks/Python.framework/Versions/3.5'
New python executable in /Users/jaraco/Dropbox/code/public/aspen/env/bin/python3
Also creating executable in /Users/jaraco/Dropbox/code/public/aspen/env/bin/python
Installing setuptools, pip, wheel...done.
$ python invoke.py env/bin/pip install pytest
Collecting pytest
  Using cached pytest-2.8.5-py2.py3-none-any.whl
Collecting py>=1.4.29 (from pytest)
  Using cached py-1.4.31-py2.py3-none-any.whl
Installing collected packages: py, pytest
Successfully installed py-1.4.31 pytest-2.8.5
$ head -n 1 env/bin/py.test
#!/Library/Frameworks/Python.framework/Versions/3.5/bin/python3

One would expect the py.test script to have a shebang with ‘env’ in it.

Using pyvenv instead is not subject to the issue. Removing the __PYVENV_LAUNCHER__ environment variable before launching the subprocess works around the issue, as reported here. The issue was observed with 13.1.2 and 14.0.0.

Is this behavior expected? If so, is removing the environment variable the appropriate thing for a parent process to do to work around the issue?

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Comments: 27 (15 by maintainers)

Commits related to this issue

Most upvoted comments

I was just bitten by this exact bug. It’s quite a severe bug in my opinion, mostly because it’s so extremely confusing and hard to figure out when it hits.

Some thoughts:

  • __PYVENV_LAUNCHER__ was added because for earlier versions of Python (< 3.3, according to Ronald) but not for later versions, Python’s site.py on OS X needed to distinguish the location of stub launchers from framework executables. According to this comment by Ned Deily, sys.executable now points to the stub launcher, and this is why distlib stopped using __PYVENV_LAUNCHER__. The only usage of it now is in site.py - if available, on OS X, it’s used to locate the pyvenv.cfg file.
  • It may be worth checking to see what happens in e.g. Python 3.4, to see if’s some sort of regression, or whether all Python >= 3.3 has the same behaviour.
  • My guess is that something, somewhere, is doing an os.path.realpath() call when it shouldn’t be. It doesn’t look like it’s distlib doing that. If (as per Ned’s comment) sys.executable and the env var are always the same, then it shouldn’t matter; but if something does a realpath on a sys.executable and executes Python through the result, then that would result in an unexpected shebang (because sys.executable would then be a dereferenced path).