vscode-python: Pytest: Unable to run tests for non-installed modules

Environment data

VS Code version: 1.15.2
Extension version (available under the Extensions sidebar): 2020.12.42445261
OS and version: Manjaro (up to date @ 2020-12-27); Linux x64 5.9.11
Python version (& distribution if applicable, e.g. Anaconda): 3.8.6
Type of virtual environment used (N/A | venv | virtualenv | conda | …): n/a
Relevant/affected Python packages and their versions: Pytest 6.2.1
Relevant/affected Python-related VS Code extensions and their versions: (probably not relevant) Pylance 2020.12.2
Value of the python.languageServer setting: Pylance

Setup

Consider the following project layout:

├ .vscode/
│  └ settings.json
├ somepkg/
│  └ __init__.py
└ tests/
   └ test_foo.py

The contents of those files don’t really matter, expect that tests/test_foo.py imports somepkg.

Example .vscode/settings.json:

{
    "python.testing.pytestArgs": ["."],
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true
}

Example somepkg/__init__.py:

def foo():
    return "foo"

Example tests/test_foo.py:

import somepkg
def test_foo():
    assert somepkg.foo() == "foo"

With such project layout, it is impossible to setup VS Code to run those tests without also installing somepkg (for example in a venv).

Here are solutions that I have tried:

Case 0: Do nothing

Obviously, this doesn’t work, because invoking bare pytest doesn’t add current directory to sys.path

VS Code fails to discover tests:

Test Discovery failed: 
Error: ============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 0 items / 1 error

==================================== ERRORS ====================================
______________________ ERROR collecting tests/test_foo.py ______________________
ImportError while importing test module '/home/mikolaj/Documents/VSCode-Pytest-Bug/tests/test_foo.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_foo.py:1: in <module>
    import somepkg
E   ModuleNotFoundError: No module named 'somepkg'
=========================== short test summary info ============================
ERROR tests/test_foo.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
===================== no tests collected, 1 error in 0.06s =====================

Traceback (most recent call last):
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 2)

And similarly command pytest fails:

$ pytest
================================================ test session starts ================================================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 0 items / 1 error                                                                                         

====================================================== ERRORS =======================================================
________________________________________ ERROR collecting tests/test_foo.py _________________________________________
ImportError while importing test module '/home/mikolaj/Documents/VSCode-Pytest-Bug/tests/test_foo.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_foo.py:1: in <module>
    import somepkg
E   ModuleNotFoundError: No module named 'somepkg'
============================================== short test summary info ==============================================
ERROR tests/test_foo.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================= 1 error in 0.06s ==================================================

Case 1: python -m pytest

This is the solution from pytest docs. I added "python.testing.pytestPath": "python -m pytest" to .vscode/settings.json and tried to launch tests.

VS Code, however, still fails on discovery.

> /usr/bin/python ~/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir ~/Documents/VSCode-Pytest-Bug -s --cache-clear .
cwd: ~/Documents/VSCode-Pytest-Bug
Error 2020-12-27 19:55:08: Failed to parse discovered Test [r [Error]: ============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 0 items / 1 error

==================================== ERRORS ====================================
______________________ ERROR collecting tests/test_foo.py ______________________
ImportError while importing test module '/home/mikolaj/Documents/VSCode-Pytest-Bug/tests/test_foo.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_foo.py:1: in <module>
    import somepkg
E   ModuleNotFoundError: No module named 'somepkg'
=========================== short test summary info ============================
ERROR tests/test_foo.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
===================== no tests collected, 1 error in 0.06s =====================

Traceback (most recent call last):
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 2)

	at ChildProcess.<anonymous> (/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/out/client/extension.js:9:567859)
	at Object.onceWrapper (events.js:313:26)
	at ChildProcess.emit (events.js:223:5)
	at maybeClose (internal/child_process.js:1021:16)
	at Socket.<anonymous> (internal/child_process.js:430:11)
	at Socket.emit (events.js:223:5)
	at Pipe.<anonymous> (net.js:664:12)]

And running pytest manually works fine.

$ python -m pytest
================================================ test session starts ================================================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 1 item                                                                                                    

tests/test_foo.py .                                                                                           [100%]

================================================= 1 passed in 0.01s =================================================

Case 2: Absolute PYTHONPATH

Here, two new keys were added to the original settings.json: "python.envFile": "${workspaceFolder}/.vscode/.env" and "terminal.integrated.env.linux": { "PYTHONPATH": "${workspaceFolder}" }; and the .vscode/.env file was created with the following content. The path is hard-coded to circumvent issue #13749.

# Somehow using ${workspaceFolder} or ${env:PWD} doesn't work here
PYTHONPATH = /home/mikolaj/Documents/VSCode-Pytest-Bug/

And now test discovery works! But then, running the tests errors out.

> /usr/bin/python ~/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/pyvsc-run-isolated.py pytest --override-ini junit_family=xunit1 --rootdir ~/Documents/VSCode-Pytest-Bug --junit-xml=/tmp/tmp-12233yWmGcDH2XD1w.xml .
cwd: ~/Documents/VSCode-Pytest-Bug

============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 0 items / 1 error

==================================== ERRORS ====================================
______________________ ERROR collecting tests/test_foo.py ______________________
ImportError while importing test module '/home/mikolaj/Documents/VSCode-Pytest-Bug/tests/test_foo.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_foo.py:1: in <module>
    import somepkg
E   ModuleNotFoundError: No module named 'somepkg'
-------------- generated xml file: /tmp/tmp-10236qdrGjj8MkHVf.xml --------------
=========================== short test summary info ============================
ERROR tests/test_foo.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.07s ===============================

And of course, pytest works like a charm:

$ PYTHONPATH=/home/mikolaj/Documents/VSCode-Pytest-Bug/ pytest
================================================ test session starts ================================================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 1 item                                                                                                    

tests/test_foo.py .                                                                                           [100%]

================================================= 1 passed in 0.01s =================================================

Case 3: relative PYTHONPATH

This time settings.json looks like this:

{
    "python.testing.pytestArgs": [
        "."
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.testing.pytestPath": "python -m pytest",
    "python.envFile": "${workspaceFolder}/.vscode/.env",
    "terminal.integrated.env.linux": {
        "PYTHONPATH": "."
    },
    "python.testing.cwd": "${workspaceFolder}"
}

And .vscode/.env contains only PYTHONPATH=.

VS Code now correctly discovers the tests, but crashes when trying to run them:

> /usr/bin/python ~/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/pyvsc-run-isolated.py pytest --override-ini junit_family=xunit1 --rootdir ~/Documents/VSCode-Pytest-Bug --junit-xml=/tmp/tmp-10236MyRbp3P5xcEN.xml .
cwd: ~/Documents/VSCode-Pytest-Bug

============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 0 items / 1 error

==================================== ERRORS ====================================
______________________ ERROR collecting tests/test_foo.py ______________________
ImportError while importing test module '/home/mikolaj/Documents/VSCode-Pytest-Bug/tests/test_foo.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_foo.py:1: in <module>
    import somepkg
E   ModuleNotFoundError: No module named 'somepkg'
-------------- generated xml file: /tmp/tmp-10236whE7Tokmzock.xml --------------
=========================== short test summary info ============================
ERROR tests/test_foo.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.07s ===============================

And, yet again, pytest works fine:

$ PYTHONPATH=. pytest                                         
================================================ test session starts ================================================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 1 item                                                                                                    

tests/test_foo.py .                                                                                           [100%]

================================================= 1 passed in 0.01s =================================================

Expected behavior

I would expect that modifications from cases 1, 2 and 3 allow testing non-installed modules.

Related issues

#6891 - that’s not the case. starting from case 3, I have created a separate test in tests/test_env.py:

import os

def test_pythonpath():
    assert os.getenv("PYTHONPATH") == "."

And it passes correctly.

#12420 - there “Run tests” work, here it doesn’t

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 32
Comments: 19 (6 by maintainers)

Most upvoted comments

After some time I also found that workaround, so yes, adding tests/__init__.py fixes the issue. Another workaround is to add an empty conftest.py to project root.

MKuranowski on Jan 28, 2021

I am having the same issue with an editable install (pip install -e). To make the extension run the tests i have to add an __init__.py as @luabud has suggested and others have also successfully tried. Tests discovery works fine without.

I am just wondering why i can easily run pytest from the command line WITHOUT adding an __init__.py in the tests folder. Personally i don’t want to clutter the test folder with unnecessary files only to make the extension being able to run the tests.

How does the extension invoke pytest?

phloose on Feb 11, 2021

Hey @MKuranowski, can you try adding a __init__.py file to your tests folder and see if it works?

luabud on Jan 27, 2021

I’m using a devcontainer and I’m having the same issue with my modules. Our tests on the command line also use python -m pytest to run the tests. Adding conftest.py to root of the workspace or __init__.py to tests\ has not worked either for me.

I got the test discovery to work as well by adding "containerEnv": {"PYTHONPATH": "${containerWorkspaceFolder}"}, to the .devcontainer/.devcontainer.json. The Run Test highlight still fails, however, on a very surprising note, debug test does work for running the tests.

EliiseS on Jun 1, 2021

Thanks @luabud I was having the same issue (exactly as described, but in Windows), adding the __init__.py to my tests folder fixes it for me.

SuaveFool on Jan 29, 2021

As suggested in https://github.com/microsoft/vscode-python/issues/14570#issuecomment-754791239 using the user setting "python.useIsolation": false fixes the issue for me.

phloose on Feb 16, 2021

I believe I have a related, but not identical issue. I have been downgrading to extension version v2020.9.111407 (or so) every time it auto-updates to get around it. In my case, the package is installed with pip install -e , so there is an egg-link in the package directory. VS code discovers the tests, the tests complete in debug mode, but give an import error for the package when I run them.

jocelyne8 on Jan 19, 2021