vscode-python: pythonTestAdapter fails if sockets are disabled for tests
Type: Bug
Behaviour
Expected vs. Actual
Actual
When using pytest
with pytest_socket
to disable network connections in test, the test suite fails. Test Results displays the following error:
vscode_pytest.VSCodePytestError: Error attempting to connect to extension communication socket[vscode-pytest]: A test tried to use socket.socket.connect() with host “localhost” (allowed: “localhost”).
Meanwhile, the test suite passes with on the terminal with:
pytest --disable-socket
The test suite can be made to pass by opting out of the pythonTestAdapter
experiment in Settings:
"python.experiments.optOutFrom": ["pythonTestAdapter"],
Expected
The test runner succeeds like the CLI.
Steps to reproduce:
I’ve created a minimal reproduction repo: https://github.com/johnchristopherjones/vscode-python-test-adapter-socket
The really short version is:
- Install
pytest_socket
- Decorate a test with
@pytest.mark.disable_socket
:
@pytest.mark.disable_socket
def test_example():
assert True
- Try to run the test with the test runner. (⌘; C)
Or, alternatively:
-
Install
pytest_socket
-
Add this to your
pyproject.toml
:[tool.pytest.ini_options] addopts = [ "--disable-socket", # Disable network in test with pytest-socket
-
Try to run any tests. (⌘; A)
Diagnostic data
- Python version (& distribution if applicable, e.g. Anaconda): 3.11.6
- Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Venv
- Value of the
python.languageServer
setting: Pylance
Output for Python
in the Python Test Log
panel
Starting now, all test run output will be sent to the Test Result panel, while test discovery output will be sent to the "Python" output channel instead of the "Python Test Log" channel. The "Python Test Log" channel will be deprecated within the next month. See https://github.com/microsoft/vscode-python/wiki/New-Method-for-Output-Handling-in-Python-Testing for details.CLIENT: Server listening on port 50821...
Received JSON data in run script
============================= test session starts ==============================
platform darwin -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0
rootdir: /Users/jcj/src/vscode-python-test-adapter-socket
configfile: pyproject.toml
plugins: socket-0.6.0
collected 1 item
tests/test_always_fail.py Error attempting to connect to extension communication socket[vscode-pytest]: A test tried to use socket.socket.
If you are on a Windows machine, this error may be occurring if any of your tests clear environment variables as they are required to communicate with the extension. Please reference https://docs.pytest.org/en/stable/how-to/monkeypatch.html#monkeypatching-environment-variablesfor the correct way to clear environment variables during testing.
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 721, in send_post_request
INTERNALERROR> __socket.connect()
INTERNALERROR> File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/testing_tools/socket_manager.py", line 32, in connect
INTERNALERROR> self.socket = socket.socket(
INTERNALERROR> ^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pytest_socket.py", line 80, in __new__
INTERNALERROR> raise SocketBlockedError()
INTERNALERROR> pytest_socket.SocketBlockedError: A test tried to use socket.socket.
INTERNALERROR>
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR>
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/main.py", line 271, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/main.py", line 325, in _main
INTERNALERROR> config.hook.pytest_runtestloop(session=session)
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR> return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 152, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_result.py", line 114, in get_result
INTERNALERROR> raise exc.with_traceback(exc.__traceback__)
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/main.py", line 350, in pytest_runtestloop
INTERNALERROR> item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR> return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 152, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_result.py", line 114, in get_result
INTERNALERROR> raise exc.with_traceback(exc.__traceback__)
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 114, in pytest_runtest_protocol
INTERNALERROR> runtestprotocol(item, nextitem=nextitem)
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 133, in runtestprotocol
INTERNALERROR> reports.append(call_and_report(item, "call", log))
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 226, in call_and_report
INTERNALERROR> hook.pytest_runtest_logreport(report=report)
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR> return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 113, in _multicall
INTERNALERROR> raise exception.with_traceback(exception.__traceback__)
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/terminal.py", line 574, in pytest_runtest_logreport
INTERNALERROR> *self.config.hook.pytest_report_teststatus(report=rep, config=self.config)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR> return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 152, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_result.py", line 114, in get_result
INTERNALERROR> raise exc.with_traceback(exc.__traceback__)
INTERNALERROR> File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 62, in _multicall
INTERNALERROR> next(wrapper_gen) # first yield
INTERNALERROR> ^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 233, in pytest_report_teststatus
INTERNALERROR> execution_post(
INTERNALERROR> File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 660, in execution_post
INTERNALERROR> send_post_request(payload)
INTERNALERROR> File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 732, in send_post_request
INTERNALERROR> raise VSCodePytestError(error_msg)
INTERNALERROR> vscode_pytest.VSCodePytestError: Error attempting to connect to extension communication socket[vscode-pytest]: A test tried to use socket.socket.
Error attempting to connect to extension communication socket[vscode-pytest]: A test tried to use socket.socket.
If you are on a Windows machine, this error may be occurring if any of your tests clear environment variables as they are required to communicate with the extension. Please reference https://docs.pytest.org/en/stable/how-to/monkeypatch.html#monkeypatching-environment-variablesfor the correct way to clear environment variables during testing.
Traceback (most recent call last):
File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 721, in send_post_request
__socket.connect()
File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/testing_tools/socket_manager.py", line 32, in connect
self.socket = socket.socket(
^^^^^^^^^^^^^^
File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pytest_socket.py", line 80, in __new__
raise SocketBlockedError()
pytest_socket.SocketBlockedError: A test tried to use socket.socket.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/run_pytest_script.py", line 67, in <module>
pytest.main(arg_array)
File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 169, in main
ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 113, in _multicall
raise exception.with_traceback(exception.__traceback__)
File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
res = hook_impl.function(*args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/main.py", line 318, in pytest_cmdline_main
return wrap_session(config, _main)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/main.py", line 306, in wrap_session
config.hook.pytest_sessionfinish(
File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 130, in _multicall
teardown[0].send(outcome)
File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/_pytest/terminal.py", line 857, in pytest_sessionfinish
outcome.get_result()
File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_result.py", line 114, in get_result
raise exc.with_traceback(exc.__traceback__)
File "/Users/jcj/src/vscode-python-test-adapter-socket/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
res = hook_impl.function(*args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 367, in pytest_sessionfinish
execution_post(
File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 660, in execution_post
send_post_request(payload)
File "/Users/jcj/.vscode/extensions/ms-python.python-2023.19.13031019/pythonFiles/vscode_pytest/__init__.py", line 732, in send_post_request
raise VSCodePytestError(error_msg)
vscode_pytest.VSCodePytestError: Error attempting to connect to extension communication socket[vscode-pytest]: A test tried to use socket.socket.
Starting now, all test run output will be sent to the Test Result panel, while test discovery output will be sent to the "Python" output channel instead of the "Python Test Log" channel. The "Python Test Log" channel will be deprecated within the next month. See https://github.com/microsoft/vscode-python/wiki/New-Method-for-Output-Handling-in-Python-Testing for details.
User Settings
languageServer: "Pylance"
testing
• pytestEnabled: true
terminal
• launchArgs: "<placeholder>"
Extension version: 2023.19.13031019 VS Code version: Code 1.83.1 (f1b07bd25dfad64b0167beb15359ae573aecd2cc, 2023-10-10T23:57:32.750Z) OS version: Darwin arm64 22.6.0 Modes:
About this issue
- Original URL
- State: closed
- Created 8 months ago
- Reactions: 12
- Comments: 37 (16 by maintainers)
@eleanorjboyd As @karthiknadig mentioned, sockets are disabled by the pytest-socket plugin for pytest, which is a good practice to ensure the code under test is not opening unexpected network connections. For instance, when testing code that might exercise a third-party API, you want to make sure your test suite isn’t REALLY sending requests to that API—especially if your test suite runs in CI.
As I mention in my reproduction repo, I suspect this might have a straightforward cause and possibly a straightforward fix. The
vscode_pytest/run_pytest_script.py
script is directly invokingpytest.run()
in-process. That means the script itself is being subjected to the test suite, which can include all kinds of modifications to the runtime that are very reasonable for test suites that are not at all reasonable for production code, such as patching os.environ and very routine test activities such as opening and closing any number of sockets and files.I think the test adapter script shouldn’t run in the same process as the test suite. Consider spinning off a subprocess, perhaps with asyncio.subprocess.
@magnasilvar 👋 Thanks for the tip! However, I think that’s a separate bug. It’s related insofar as that bug prevents a partial workaround using
--allow-hosts=localhost
withpytest-socket
.That’s not a workaround I want to use because I also specifically don’t want connections to services running on localhost, like databases.
Allowing localhost would allow the test runner to work, much like turning off your firewall or setting a DMZ allows friends to connect to your Minecraft server—you’re just circumventing the firewall. The underlying problem is that the test runner runtime environment is being controlled by the test suite environment.
Consider the following use-cases:
localhost:5432
)Relatedly, we closed #21645, regarding
pytest-recording
breaking under the new test runner, with this understanding. Is it the same bug inpytest-recording
? Maybe, but they’re not dependencies of each other, so fixing one won’t fix the other.Hi @johnchristopherjones 👋 It’s not a bug in
vscode-python
, but a bug inpytest-socket
. I have opened this PR to resolve the issue: https://github.com/miketheman/pytest-socket/pull/275Current workaround
Downgrade your version of
pytest-socket
to the previous version:5.1.0
.I have been hit by this in some projects where sockets are disabled by design. Took a very long time to figure it out 😦
The issue goes away when I opt out from the experiment.
I’m just observing this as a bystander, but I do see that pytest-socket does allow for that:
I also see you can configure it to only allow connections to (for example) localhost, which might be another possible workaround:
I suspect that the pytest_socket pligin is getting loaded before vscode_pytest.
@strawgate One thing you could try is to disable Unix Domain socket from being blocked by the pytest_socket plugin. It may have a optional setting to do this.
@eleanorjboyd We can try and match to copied (monkey patched) sockets, but I think the real solution would be this: https://github.com/microsoft/vscode-python/issues/23279
It has more details on what is going on and why we need to use this communication channel, and the limitations of other modes of communication.
@eleanorjboyd Thanks for working on this. I just installed Code - Insiders and can confirm that my tests fail with Python extension v2024.4.1 but pass with v2024.5.11101014 (pre-release), so all looks good.
Hello! We have now switched our extension to using named pipes! This should resolve the socket disabled issue. If someone could try it out and let me know if it works that would be extremely helpful. To do so you just need to be on the latest version of the python insiders extension and a vscode version >= 1.89.0-20240415- the easiest way to do this is download vscode insiders and get the pre-release of the python extension.
Thanks so much and happy coding!
You can also use the command palette to go the the user settings json and add it there
@johnchristopherjones, we will look to work on this change soon but cannot promise a timeline. @karthiknadig or I will keep you in the loop as we merge changes related to switching to named pipes. Thanks!