vscode-python: `pyvsc-run-isolated.py` removes pip "editable" installations from `sys.path`, breaks pytest

@brettcannon , if I understand you correctly the motivation of the isolate script is to make tests/linters function even if they would not work when run from the terminal emulator, e.g. due to name clashes between files in workspace root and stdlib modules. If this is so, I would like to offer my view on this matter for your consideration:

Before coming to VSCode I have been a PyCharm user (many of my colleagues still are). For me, the most important advantage of using VSCode over PyCharm is that VSCode puts the user in charge of far more aspects of workspace configuration than PyCharm does. Moreover, VSCode is configured through text files and allows the user to configure tests/linters/whatnot using the standard configuration files of the respective programs. These files can than be committed to repository, reused by other users (even using other IDEs) and (very importantly) in CI context. This has huge value when it comes to educating users how to use all these tools without VSCode. VSCode encouraged me to learn things and made me a better programmer, PyCharm prevents users from learning.

To illustrate how important this can be I can say the following: It is very common for my colleagues who exclusively use PyCharm to have problems with correct configuration of their projects, to the point that some of the software they write works only when run from within PyCharm! I recall that one piece of software had installation instructions that started with “1. Install PyCharm”.

For this reason I feel that IDEs should not try to fix what users have broken. This makes them ignorant of how projects should be organized. In my opinion testing and linting should be run directly in user’s environment.

For me; pytest and pylint were finding finding modules using the PYTHONPATH=. setting in my ${workspaceFolder}/.env file. (as suggested here)

The behaviour introduced in #14014 that removes $(cwd) from sys.path breaks this setting; causing both to fail.

Same her: running tests from GUI is broken on a package installed using ‘setup.py develop’. Likely related, pylint is also broken, marking same package imports as not found.

@schperplata If you can try the latest insiders build, it has a flag to turn off isolation. Please try that and let us know if it works for you. The setting is "python.useIsolation": false. This should be set in the User settings.

FYI we’re planning to remove isolation script with https://github.com/microsoft/vscode-python/pull/16274 so this can be closed.

@schperplata we’re waiting on a feature from VS Code which we hope will become available in the next couple of releases/months, at which point we can drop the setting.

@schperplata If you can try the latest insiders build, it has a flag to turn off isolation. Please try that and let us know if it works for you. The setting is "python.useIsolation": false. This should be set in the User settings.

Thank you! I chose weekly insiders, added the setting: "python.useIsolation": false and that fixed it for me!

I’ve had to disable pylint linting in my projects, which is annoying, since we have it turned on in our CI/CD, so disabling it in my editor is definitely not ideal.

Another option maybe – ability to choose whether to run in isolated mode?

Edit: also interesting that “Debug Test” adornment works but not “Run Test”

@karthiknadig , I am glad to see that this has been approved for fixing in next release.

I also wanted to start a discussion on how this should be fixed. I see two ways:

• change behavior of the isolate script to remove at most one instance of '' or cwd from sys.path
• run tests and linters without the use of the isolate script (The question being: what is the purpose of running those via the isolate script? Is running pytest or pylint directly in the workspace root considered a risk?)

Sorry, I am still fairly new to python. What is the expected way for the target package to be installed locally when running pytest? From the pytest documentation it seems like installing pip install -e is suggested.

@ashrauma it might be, but without more info it’s hard to tell. Please open a new issue and fill out the bug report template.

The insiders build v2021.2.531887795-dev works for me too.

@schperplata If you can try the latest insiders build, it has a flag to turn off isolation. Please try that and let us know if it works for you. The setting is "python.useIsolation": false. This should be set in the User settings.

Thank you! I chose weekly insiders, added the setting: "python.useIsolation": false and that fixed it for me!

fixed it for me too!

@brettcannon Q: As this is a breaking change (we have to debug instead of run tests via VSC), is it possible that the breaking change is reverted? As it is currently broken, we (all users of VSC and pytest) have to use older version: 2020.9.114305. This also adds a lot of instructions on how to use VSC with pytest, since one has to install specific extension version (vs latest).

I agree with @akukuq that the IDE/Editor/whatever should not try to fix something that is broken in users code. However, I have to assume the reason that the pyvsc-run-isolated.py exists is because there were enough people making “false” issues in this repo that is was in the best interest of the developers to create a workaround. So it seems that we need to find a happy medium that doesn’t overburden the extension developers with false issues and still allows people who are using the tool correctly to run their code.

One potential solution is to search through the module list, get the location of editable modules, and only remove the cwd if it is not listed as an editable module:

# We "isolate" the script/module (sys.argv[1]) by removing current working
# directory or '' in sys.path and then sending the target on to runpy.
# However, we should only do this if the current working directory
# is not listed as an editable directory.

# TODO: how to know which pip to use
modules = os.popen("pip list").readlines()[2:]  # remove first two lines which are header and separator

# editable modules have the location listed, which is the third field
editable_modules = [normalize(m.split()[2]) for m in modules if len(m.split()) == 3]

cwd = normalize(os.getcwd())
if cwd in editable_modules:
    cwd = ""

sys.path[:] = (p for p in sys.path if p != "" and normalize(p) != cwd)
...

(note that I wrote this quickly without much thought to formatting or correctness, but I’ll add the dreaded phrase that all developers love to hear: “it works on my machine.”)

In addition to pytest, this regression seems to effect notebooks as well. I get module not found errors when importing a project in the same folder (i.e. using cwd) in a notebook. Import works fine when downgrading to 2020.9.114305.