spyder: Missing Python packages due to cleared `$PYTHONPATH`
Problem Description
We install Spyder alongside other globally installed Python packages that are in non-standard locations. To make Python find them we either add the locations to $PYTHONPATH or use a custom sitecustomize.py whose location must be in $PYTHONPATH
When starting Spyder it fails with errors such as No module named 'zmq'.
I traced that to https://github.com/spyder-ide/spyder/blob/a857397bb4fc80c597ae3ba1ac478765137a1117/spyder/app/start.py#L13-L18
That removes the entries that are required such that Python finds the modules and our sitecustomize
Removing that results in a similar problem as the IDE console shows
Error while finding module specification for ‘spyder_kernels.console’ (ModuleNotFoundError: No module named ‘spyder_kernels’)
That arises due to removing $PYTHONPATH from the environment the kernel starts in:
https://github.com/spyder-ide/spyder/blob/a857397bb4fc80c597ae3ba1ac478765137a1117/spyder/plugins/ipythonconsole/utils/kernelspec.py#L172
I’m not fully sure what the intention was, but maybe that can be solved differently. Is that possible?
Versions
- Spyder version: 5.5.1
- Python 3.11.3 64-bit | Qt 5.15.10 | PyQt5 5.15.10 | Linux 4.18.0-513.24.1.el8_9.x86_64 (x86_64)
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 24 (24 by maintainers)
@mrclary Correct. And just to be clarify on the 2nd point: When started from a console on Linux PYTHONPATH from the environment is honored except paths matching a certain pattern
And for both we want an option to disable any filtering
I like this. 👍🏼
I’m referring to the same thing you commented on here.
Anything that is caught by the filtering is considered (potentially) unsafe 😄.
I propose the following solution.
PYTHONPATHpaths to the end ofsys.pathin its runtime environment at startup. This will allow Spyder to launch if non-standard library locations are listed inPYTHONPATH. Currently Spyder removes these paths, however moving them to the end should provide the same protection.PYTHONPATHfrom the subprocess environment used to start Spyder-kenels.PYTHONPATHpaths to the end ofsys.pathat the beginning of startup. Currently, Spyder-kernels is protected when Spyder removesPYTHONPATHfrom the subprocess environment used to launch Spyder-kernels; Spyder-kernels will now be responsible for its own protection.Spyder-kernels will removeSpyder-kernels will restorePYTHONPATHpaths fromsys.pathimmediately after startup.PYTHONPATHpaths tosys.pathin their original position immediately after startup, allowing remote environments to execute user code as expected. Upon receiving thesig_pythonpath_changedsignal from the PYTHONPATH Manager, Spyder-kernels will remove the originalPYTHONPATHpaths fromsys.pathand add those from the signal.sig_pythonpath_changedsignal to Spyder-kernels if Spyder-kernels is not running in a remote environment. This will allow proper integration with the PYTHONPATH Manager plugin when not using a remote kernel, but honors the use ofPYTHONPATHin the remote environment. Unlike Spyder, Spyder-kernels’ runtime environment is the same as the user’s Python environment (where the user runs his own code) so paths from the PYTHONPATH Manager will be injected into thesys.pathin this environment, when not running in a remote environment. This makes sense since paths available to a local user will not be available to the remote Python kernel.get_user_environment_variableswill continue to useos.environfrom Spyder’s runtime environment if Spyder was launched from the commandline, otherwise it will attempt to get environment variables from an interactive login shell subprocess. Note this is important for Spyder launched from a shortcut: macOS will not see any shell startup variables, while Linux will only see login startup variables (i.e. ~/.bash_profile) but not interactive variables (i.e. ~/.bashrc).PYTHONPATHpath filtering resides entirely within the PYTHONPATH Manager, is not used anywhere else, and is unrelated to Spyder startup; therefore I think it is more appropriate to use a preference setting rather than an environment variable.Since #21769 includes a lot of refactoring of the PYTHONPATH Manager, I propose making these changes either after that PR or as part of that PR.
@Flamefire @smax1, I’ll summarize what I think are the primary issues:
PYTHONPATHpaths from its runtime environmentsys.path, causing Spyder to fail to launch due to its inability to find necessary packages in non-standard locations (specified inPYTHONPATH). Spyder-kernels may be affected in the same way.PYTHONPATHon to user processes (IPython Console and others), Spyder only obtainsPYTHONPATHfrom shell startup files (interactive + login) and not from the environment variables in its current launch environment, precluding possible changes from e.g. environment modules.No; I think it is related to Windows subsystem for Linux. I left a comment. However, the proposed solution for #22681 (current issue), may provide a workaround for #21633. But I think #21633 requires a dedicated fix.
I agree that that’s the best we could do to allow this use case.
PYTHONPATHis for user-provided and non-standard-library paths and should not be relied upon for Python or installed packages to work properly. Unfortunately, any user-provided path viaPYTHONPATHpresents a potential hazard to Spyder, not just the current working directory, so we cannot selectively “clean” this environment variable.Rather than
PYTHONPATH, I would recommend using a.pthfile located in../lib/python3.11/site-packages, relative to the Python executable you are using. See https://docs.python.org/3.11/library/site.html#module-site