spyder: IPython Console not working when connecting standalone app to external interpreter on Mac

Description

What steps will reproduce the problem?

Change Python Interpretor to System’s Python path
Install spyder-kernal

Traceback

Traceback (most recent call last):
  File "/Applications/Spyder.app/Contents/Resources/lib/python3.9/jupyter_client/manager.py", line 91, in wrapper
    self._ready.set_result(None)
asyncio.exceptions.InvalidStateError: invalid state

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "qtconsole/manager.pyc", line 27, in poll
  File "/Applications/Spyder.app/Contents/Resources/lib/python3.9/jupyter_client/restarter.py", line 143, in poll
    self.kernel_manager.restart_kernel(now=True, newports=newports)
  File "jupyter_core/utils/__init__.pyc", line 173, in wrapped
  File "asyncio/base_events.pyc", line 647, in run_until_complete
  File "/Applications/Spyder.app/Contents/Resources/lib/python3.9/jupyter_client/manager.py", line 596, in _async_restart_kernel
    await self._async_start_kernel(**self._launch_args)
  File "/Applications/Spyder.app/Contents/Resources/lib/python3.9/jupyter_client/manager.py", line 94, in wrapper
    self._ready.set_exception(e)
asyncio.exceptions.InvalidStateError: invalid state

Versions

Spyder version: 5.5.0 b4c4e1a16 (standalone)
Python version: 3.9.14 64-bit
Qt version: 5.15.11
PyQt5 version: 5.15.10
Operating System: macOS-14.1.1-x86_64-i386-64bit

Dependencies

# Mandatory:
atomicwrites >=1.2.0             :  1.4.1 (OK)
chardet >=2.0.0                  :  5.2.0 (OK)
cloudpickle >=0.5.0              :  3.0.0 (OK)
cookiecutter >=1.6.0             :  2.5.0 (OK)
diff_match_patch >=20181111      :  20230430 (OK)
intervaltree >=3.0.2             :  3.1.0 (OK)
IPython >=8.13.0,<9.0.0,!=8.17.1 :  8.17.2 (OK)
jedi >=0.17.2,<0.20.0            :  0.19.1 (OK)
jellyfish >=0.7                  :  1.0.3 (OK)
jsonschema >=3.2.0               :  4.20.0 (OK)
keyring >=17.0.0                 :  24.3.0 (OK)
nbconvert >=4.0                  :  7.11.0 (OK)
numpydoc >=0.6.0                 :  1.6.0 (OK)
parso >=0.7.0,<0.9.0             :  0.8.3 (OK)
pexpect >=4.4.0                  :  4.8.0 (OK)
pickleshare >=0.4                :  0.7.5 (OK)
psutil >=5.3                     :  5.9.6 (OK)
pygments >=2.0                   :  2.17.1 (OK)
pylint >=2.5.0,<3.1              :  3.0.2 (OK)
pylint_venv >=3.0.2              :  None (OK)
pyls_spyder >=0.4.0              :  0.4.0 (OK)
pylsp >=1.9.0,<1.10.0            :  1.9.0 (OK)
pylsp_black >=1.2.0,<3.0.0       :  1.3.0 (OK)
qdarkstyle >=3.2.0,<3.3.0        :  3.2.1 (OK)
qstylizer >=0.2.2                :  0.2.2 (OK)
qtawesome >=1.2.1                :  1.2.3 (OK)
qtconsole >=5.5.0,<5.6.0         :  5.5.1 (OK)
qtpy >=2.1.0                     :  2.4.1 (OK)
rtree >=0.9.7                    :  1.1.0 (OK)
setuptools >=49.6.0              :  69.0.2 (OK)
sphinx >=0.6.6                   :  5.1.1 (OK)
spyder_kernels >=2.5.0,<2.6.0    :  2.5.0 (OK)
textdistance >=4.2.0             :  4.6.0 (OK)
three_merge >=0.1.1              :  0.1.1 (OK)
watchdog >=0.10.3                :  3.0.0 (OK)
zmq >=22.1.0                     :  25.1.1 (OK)

# Optional:
cython >=0.21                    :  3.0.5 (OK)
matplotlib >=3.0.0               :  3.8.2 (OK)
numpy >=1.7                      :  1.26.2 (OK)
pandas >=1.1.1                   :  2.1.3 (OK)
scipy >=0.17.0                   :  1.11.4 (OK)
sympy >=0.7.3                    :  1.12 (OK)

About this issue

Original URL
State: open
Created 7 months ago
Comments: 24 (16 by maintainers)

Most upvoted comments

Ryan: I realized that I never confirmed the solution on this issue. The download works great! I appreciate the support from you and Carlos, and I still kind of marvel at how things progressed from workaround to solution to enhanced product. You don’t see that every day. Regards, Tom Barson

On Sun, Jan 21, 2024 at 5:22 PM Ryan Clary @.***> wrote:

@tomb1949 https://github.com/tomb1949, I’ve uploaded an arm64 build of Spyder 5.5.0 to our release page. This is a full standalone version, but built locally on my M1 iMac, using a universal/arm64 Python environment only. You may want to give that a try; this should not require any of the workarounds that I posted earlier.

https://github.com/spyder-ide/spyder/releases/download/v5.5.0/Spyder-arm64.dmg

— Reply to this email directly, view it on GitHub https://github.com/spyder-ide/spyder/issues/21575#issuecomment-1902785528, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATBPTTAGZHY656ESHQQK64LYPWIL3AVCNFSM6AAAAABAEDUE3OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBSG44DKNJSHA . You are receiving this because you were mentioned.Message ID: @.***>

– Tom Barson East Lansing, MI

tomb1949 on Mar 13, 2024

@tomb1949, I’ve uploaded an arm64 build of Spyder 5.5.0 to our release page. This is a full standalone version, but built locally on my M1 iMac, using a universal/arm64 Python environment only. You may want to give that a try; this should not require any of the workarounds that I posted earlier. https://github.com/spyder-ide/spyder/releases/download/v5.5.0/Spyder-arm64.dmg

mrclary on Jan 21, 2024

@mrclary, is it possible to detect if an external environment is universal or not on Mac? That way I think we could post your recommendations as a message in the console:

What I’ve learned is that for Spyder (x86_64) to work with external environments on arm64:

If the environment Python executable is universal, then all binaries in the Python environment must be x86_64. See below on how to accomplish this.

If the environment Python executable is a single architecture, then all the binaries in the Python environment must be the same architecture.

It is possible to detect whether the environment Python executable is x86_64, arm64, or Universal. It is possible to detect whether Spyder is running under Rosetta since platform.version will contain ARM64 (the true architecture), while platform.platform will indicate x86_64.

>>> platform.version()                                            
'Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:34 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T8103'

>>> platform.platform()
'macOS-14.2.1-x86_64-i386-64bit'

I’ll reinvestigate building the standalone installer on arm64. As I recall, there was an issue because PyQt5 was not available in arm64. If I can build a native arm64 standalone, I think other issues may be resolved as well (#20201, #20629).

We can discuss possible strategies for moving forward at our next developer meeting.

mrclary on Jan 10, 2024

@tomb1949, according to what you posted above, please try these commands to check if they fix your problem:
pip uninstall psutil
pip install --no-binary :all: psutil

I would note that from the above traceback, all we know is that psutil is the first problematic binary encountered. It is possible that other packages will also have arm64 only architecture installed.

mrclary on Jan 8, 2024

Ryan: Thank you - this is above and beyond. I will attempt your suggestions tomorrow am when I have a full day to try them and deal with the consequences. Appreciate your help very much. Some other tools (VS Code, Jupyter) are doing fine with my setup, but I miss Spyder. Your suggestions will be useful even if (or especially if) I revert to Anaconda. TB

On Fri, Jan 5, 2024 at 2:41 PM Ryan Clary @.***> wrote:

TL;DR

What I’ve learned is that for Spyder (x86_64) to work with external environments on arm64:

If the environment Python executable is universal, then all binaries in the Python environment must be x86_64. See below on how to accomplish this.

If the environment Python executable is a single architecture, then all the binaries in the Python environment must be the same architecture.

Details

Ryan: Thank you for looking at this. Some possible things of interest.

I don’t have a usr/bin/python instance.

Actually, you should have /usr/bin/python3, which comes with macOS, but it is best to pretend that you don’t have anything there 😉.

Anaconda used to be installed but I removed it because I kept having failed update issues. So I’ve been building a standalone environment (successfully, so far) except for this problem.

Sorry to hear that. I prefer to use miniforge https://github.com/conda-forge/miniforge since it is much smaller.

The python interpreter that is failing to load in Spyder iConsole is: /Library/Frameworks/Python.framework/Versions/3.12/bin/python3 There is also a python3.12-intel64 executable in that ‘bin’ but it doesn’t help if I point to it explicity in Spyder Preferences. It would want to be started by Rosettta 2, I think.

Okay, this is the Python.org installation and it is a universal2 binary (has both x86_64 and arm64 built in). As you observe, there are also python3.12-intel64 and python3.12-arm64 executables; these are also universal binaries but only have 1 architecture. You are correct that these executables may not be useful in solving any of the problems that you observe. There are two problems I think may be occurring for you here.

You should avoid using the base Python executable environment. This is true for any Python install, even for conda (Anaconda or miniforge). One should always use a virtual environment instead. For a conda install you would do the following.

$ conda create -n my-venv python=3.12 spyder-kernels=2.5

For any other Python install, use venv.

$ /usr/local/bin/python3 -m venv /path/to/desired/my-venv $ source /path/to/desired/my-venv/bin/activate (my-venv) $ python -m pip install spyder-kernels==2.5

You should point Spyder to the python executable in the virtual environment. If you were using the Anaconda base environment to “work” in, then I suspect that is the reason you had update issues. I’ve also run into all sorts of problems whenever I’ve used base environments (even accidentally). 2. When I create a virtual environment as above with /usr/local/bin/python3 (which just points to the universal binary /Library/Frameworks/Python.framework/Versions/3.12/bin/python3) and install spyder-kernels I get the following error in Spyder’s IPython console:

ImportError: dlopen(/Users/ryan/my-venv/lib/python3.12/site‑packages/psutil/_psutil_osx.abi3.so, 0x0002): tried: ‘/Users/ryan/my-venv/lib/python3.12/site‑packages/psutil/_psutil_osx.abi3.so’ (mach‑o file, but is an incompatible architecture (have ‘arm64’, need ‘x86_64’))

I think the problem here is that _psutil_osx.abi3.so is not the same architecture as the Python executable (x86_64 version selected by Rosetta). As I’ve found with conda environments, when the Python executable is arm64 and _psutil_osx.abi3.so is arm64, then there doesn’t seem to be a problem. The workaround is to ensure x86_64 binaries are installed in the virtual environment. After creating the virtual environment as shown above, when installing spyder-kernels (or any package in the environment) use the arch -x86_64 pre-command:

$ /usr/local/bin/python3 -m venv /path/to/desired/my-venv $ source /path/to/desired/my-venv/bin/activate (my-venv) $ arch -x86_64 python -m pip install spyder-kernels==2.5

This ensures that x86_64 packages are installed instead of arm64.

I was NOT prompted to install Rosetta 2 when i started Spyder standalone. I know it’s on my machine (and that it works) because I needed to install it for MATLAB.

It will only prompt if Rosetta is not already installed.

Happy to try something if you have ideas. Thanks, TB

See above.

— Reply to this email directly, view it on GitHub https://github.com/spyder-ide/spyder/issues/21575#issuecomment-1879174466, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATBPTTFE7UVMCY2OAPVWFVTYNBJPFAVCNFSM6AAAAABAEDUE3OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZZGE3TINBWGY . You are receiving this because you commented.Message ID: @.***>

– Tom Barson East Lansing, MI

tomb1949 on Jan 5, 2024