spyder: Missing Python packages due to cleared `$PYTHONPATH`

Problem Description

We install Spyder alongside other globally installed Python packages that are in non-standard locations. To make Python find them we either add the locations to $PYTHONPATH or use a custom sitecustomize.py whose location must be in $PYTHONPATH

When starting Spyder it fails with errors such as No module named 'zmq'.

I traced that to https://github.com/spyder-ide/spyder/blob/a857397bb4fc80c597ae3ba1ac478765137a1117/spyder/app/start.py#L13-L18

That removes the entries that are required such that Python finds the modules and our sitecustomize

Removing that results in a similar problem as the IDE console shows

Error while finding module specification for ‘spyder_kernels.console’ (ModuleNotFoundError: No module named ‘spyder_kernels’)

That arises due to removing $PYTHONPATH from the environment the kernel starts in: https://github.com/spyder-ide/spyder/blob/a857397bb4fc80c597ae3ba1ac478765137a1117/spyder/plugins/ipythonconsole/utils/kernelspec.py#L172

I’m not fully sure what the intention was, but maybe that can be solved differently. Is that possible?

Versions

  • Spyder version: 5.5.1
  • Python 3.11.3 64-bit | Qt 5.15.10 | PyQt5 5.15.10 | Linux 4.18.0-513.24.1.el8_9.x86_64 (x86_64)

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 24 (24 by maintainers)

Most upvoted comments

@mrclary Correct. And just to be clarify on the 2nd point: When started from a console on Linux PYTHONPATH from the environment is honored except paths matching a certain pattern

And for both we want an option to disable any filtering

`` it could be something like SPY_NOCLEAN_PYTHONPATH

I like this. 👍🏼

What exactly is that filtering?

I’m referring to the same thing you commented on here.

And what are “unsafe PYTHONPATH paths”?

Anything that is caught by the filtering is considered (potentially) unsafe 😄.

I propose the following solution.

  • Spyder will move PYTHONPATH paths to the end of sys.path in its runtime environment at startup. This will allow Spyder to launch if non-standard library locations are listed in PYTHONPATH. Currently Spyder removes these paths, however moving them to the end should provide the same protection.
  • Spyder will not remove PYTHONPATH from the subprocess environment used to start Spyder-kenels.
  • Spyder-kernels will move PYTHONPATH paths to the end of sys.path at the beginning of startup. Currently, Spyder-kernels is protected when Spyder removes PYTHONPATH from the subprocess environment used to launch Spyder-kernels; Spyder-kernels will now be responsible for its own protection.
  • Spyder-kernels will remove PYTHONPATH paths from sys.path immediately after startup. Spyder-kernels will restore PYTHONPATH paths to sys.path in their original position immediately after startup, allowing remote environments to execute user code as expected. Upon receiving the sig_pythonpath_changed signal from the PYTHONPATH Manager, Spyder-kernels will remove the original PYTHONPATH paths from sys.path and add those from the signal.
  • The IPython console shell will only pass the PYTHONPATH Manager’s sig_pythonpath_changed signal to Spyder-kernels if Spyder-kernels is not running in a remote environment. This will allow proper integration with the PYTHONPATH Manager plugin when not using a remote kernel, but honors the use of PYTHONPATH in the remote environment. Unlike Spyder, Spyder-kernels’ runtime environment is the same as the user’s Python environment (where the user runs his own code) so paths from the PYTHONPATH Manager will be injected into the sys.path in this environment, when not running in a remote environment. This makes sense since paths available to a local user will not be available to the remote Python kernel.
  • get_user_environment_variables will continue to use os.environ from Spyder’s runtime environment if Spyder was launched from the commandline, otherwise it will attempt to get environment variables from an interactive login shell subprocess. Note this is important for Spyder launched from a shortcut: macOS will not see any shell startup variables, while Linux will only see login startup variables (i.e. ~/.bash_profile) but not interactive variables (i.e. ~/.bashrc).
  • Add a preference “Allow unsafe PYTHONPATH paths” to Preferences -> Application -> Advanced settings (default unchecked). This will allow bypassing the path filtering performed by PYTHONPATH Manager. The PYTHONPATH path filtering resides entirely within the PYTHONPATH Manager, is not used anywhere else, and is unrelated to Spyder startup; therefore I think it is more appropriate to use a preference setting rather than an environment variable.

Since #21769 includes a lot of refactoring of the PYTHONPATH Manager, I propose making these changes either after that PR or as part of that PR.

@Flamefire @smax1, I’ll summarize what I think are the primary issues:

  • Spyder removes PYTHONPATH paths from its runtime environment sys.path, causing Spyder to fail to launch due to its inability to find necessary packages in non-standard locations (specified in PYTHONPATH). Spyder-kernels may be affected in the same way.
  • To pass PYTHONPATH on to user processes (IPython Console and others), Spyder only obtains PYTHONPATH from shell startup files (interactive + login) and not from the environment variables in its current launch environment, precluding possible changes from e.g. environment modules.

Note: Could this be related/should supersede #21633 ?

No; I think it is related to Windows subsystem for Linux. I left a comment. However, the proposed solution for #22681 (current issue), may provide a workaround for #21633. But I think #21633 requires a dedicated fix.

So, in conclusion, I agree that @Flamefire’s suggestion to use an environment variable to switch PYTHONPATH treatment in Spyder may be the best solution for now. I propose SPY_ADD_PYTHONPATH. Spyder will not add paths from PYTHONPATH to sys.path upon startup if SPY_ADD_PYTHONPATH is unset or evaluates to False; Spyder will add paths from PYTHONPATH to sys.path upon startup if SPY_ADD_PYTHONPATH evaluates to True. What do @spyder-ide/core-developers think?

I agree that that’s the best we could do to allow this use case.

PYTHONPATH is for user-provided and non-standard-library paths and should not be relied upon for Python or installed packages to work properly. Unfortunately, any user-provided path via PYTHONPATH presents a potential hazard to Spyder, not just the current working directory, so we cannot selectively “clean” this environment variable.

Rather than PYTHONPATH, I would recommend using a .pth file located in ../lib/python3.11/site-packages, relative to the Python executable you are using. See https://docs.python.org/3.11/library/site.html#module-site