spyder: IPython kernel hangs when using other graphics backend than "inline"

Issue Report Checklist

Searched the issues page for similar reports
Read the relevant sections of the Spyder Troubleshooting Guide and followed its advice
Reproduced the issue after updating with conda update spyder (or pip, if not using Anaconda)
Could not reproduce inside jupyter qtconsole (if console-related)
Tried basic troubleshooting (if a bug/error)
- Restarted Spyder
- Reset preferences with spyder --reset
- Reinstalled the latest version of Anaconda
- Tried the other applicable steps from the Troubleshooting Guide
Completed the Problem Description, Steps to Reproduce and Version sections below

Problem Description

Potentially the same issue as: https://github.com/spyder-ide/spyder/issues/16400

When using a different graphics backend than “Inline”, (i.e. Automatic or Qt5), the kernel hangs when running a script. Infrequently, it works as expected, but there doesn’t seem to be a pattern to this. The kernel must be restarted to regain functionality. Interrupt does not work.

What steps reproduce the problem?

Set graphics backend to Automatic or Qt5
Run a script that produces a plot

What is the expected output? What do you see instead?

Expected: A new plot window and any console output from the script Actual: No plot window, no console output

Versions

Spyder version: 5.4.3
Python version: 3.9.17
Qt version: 5.15.2
PyQt version: 5.15.7
Operating System name/version: Windows 10 22H2

Dependencies

# Mandatory:
atomicwrites >=1.2.0                             :  1.4.0 (OK)
chardet >=2.0.0                                  :  4.0.0 (OK)
cloudpickle >=0.5.0                              :  2.2.1 (OK)
cookiecutter >=1.6.0                             :  1.7.3 (OK)
diff_match_patch >=20181111                      :  20200713 (OK)
intervaltree >=3.0.2                             :  3.1.0 (OK)
IPython >=7.31.1,<9.0.0,!=8.8.0,!=8.9.0,!=8.10.0 :  8.12.2 (OK)
jedi >=0.17.2,<0.19.0                            :  0.18.1 (OK)
jellyfish >=0.7                                  :  0.9.0 (OK)
jsonschema >=3.2.0                               :  4.17.3 (OK)
keyring >=17.0.0                                 :  23.13.1 (OK)
nbconvert >=4.0                                  :  6.5.4 (OK)
numpydoc >=0.6.0                                 :  1.5.0 (OK)
paramiko >=2.4.0                                 :  2.8.1 (OK)
parso >=0.7.0,<0.9.0                             :  0.8.3 (OK)
pexpect >=4.4.0                                  :  4.8.0 (OK)
pickleshare >=0.4                                :  0.7.5 (OK)
psutil >=5.3                                     :  5.9.0 (OK)
pygments >=2.0                                   :  2.15.1 (OK)
pylint >=2.5.0,<3.0                              :  2.16.2 (OK)
pylint_venv >=2.1.1                              :  2.3.0 (OK)
pyls_spyder >=0.4.0                              :  0.4.0 (OK)
pylsp >=1.7.2,<1.8.0                             :  1.7.2 (OK)
pylsp_black >=1.2.0                              :  1.2.1 (OK)
qdarkstyle >=3.0.2,<3.2.0                        :  3.0.2 (OK)
qstylizer >=0.2.2                                :  0.2.2 (OK)
qtawesome >=1.2.1                                :  1.2.2 (OK)
qtconsole >=5.4.2,<5.5.0                         :  5.4.2 (OK)
qtpy >=2.1.0                                     :  2.2.0 (OK)
rtree >=0.9.7                                    :  1.0.1 (OK)
setuptools >=49.6.0                              :  68.0.0 (OK)
sphinx >=0.6.6                                   :  5.0.2 (OK)
spyder_kernels >=2.4.3,<2.5.0                    :  2.4.4 (OK)
textdistance >=4.2.0                             :  4.2.1 (OK)
three_merge >=0.1.1                              :  0.1.1 (OK)
watchdog >=0.10.3                                :  2.1.6 (OK)
zmq >=22.1.0                                     :  23.2.0 (OK)

# Optional:
cython >=0.21                                    :  None (NOK)
matplotlib >=3.0.0                               :  3.7.1 (OK)
numpy >=1.7                                      :  1.24.3 (OK)
pandas >=1.1.1                                   :  2.0.3 (OK)
scipy >=0.17.0                                   :  1.11.1 (OK)
sympy >=0.7.3                                    :  1.11.1 (OK)

About this issue

Original URL
State: closed
Created 10 months ago
Comments: 18 (8 by maintainers)

Commits related to this issue

Change eventloop iteration scheduling Testing this as a possible fix for https://github.com/spyder-ide/spyder/issues/21299 — committed to jdranczewski/ipykernel by jdranczewski 4 months ago
Change eventloop iteration scheduling Testing this as a possible fix for https://github.com/spyder-ide/spyder/issues/21299 — committed to jdranczewski/ipykernel by jdranczewski 4 months ago
Use msg_queue for scheduling eventloop advances This ensures that the msg_queue is truly empty before entering the eventloop, fixing a possible race condition where process_one has consumed a dispatch... — committed to jdranczewski/ipykernel by jdranczewski 4 months ago
Merge from 5.x: PR #21834 Fixes #21299 — committed to mrclary/spyder by ccordoba12 4 months ago

Most upvoted comments

I also tried @jdranczewski’s fix, and it seems to work for me as well!

gica3618 on Feb 23, 2024

This particular fix is not part of that PR yet, as it’s a different issue (handling of call aborting in eventloops vs scheduling of eventloop calls), so I wasn’t sure if I should bring it in there. I’ll check with the maintainers over at IPykernel tomorrow (I still want to run some sanity checks) if they think this could be merged into one PR under the eventloop umbrella or if warrants a separate PR.

jdranczewski on Feb 23, 2024

Hi there, although I initially was unable to reproduce this, checked @jdranczewski spamming the run button way to reproduce this and I was indeed able to reproduce 👍

Also, applying the patch to ipykernel to my installation indeed seems like fixes things 🎉 I’m even able to see some output from matplotlib related with memory consumption in the console due to the amount of plots being open:

dalthviz on Feb 22, 2024

@James-F1dev could you specifically confirm that this happens on Windows under Tkinter for you? The Qt and Tkinter loops work in very different ways on Windows, so this would be very useful to know. I know you said that Tkinter is also affected in your original report, but I am very confused as to how this could happen under Tkinter on Windows (as it should return focus to the main kernel thread every 10ms or so), so thought I’d double-check.

Hi @jdranczewski yes you are completely correct, it appears that Tkinter is not subject to this issue. Only automatic and Qt5 exhibit the behaviour described in my original post.

Tkinter sometimes fails to produce the plot window, but does not hang the kernel. This is resolved simply by running the script again, so it seems like a separate and less severe issue.

Thanks for looking at this.

James-F1dev on Feb 19, 2024

I’ve had some luck reproducing this on Windows under Qt by spamming the run button, which occasionally makes a bizarre interaction of the Tornado io_loop, the kernel’s message queue, and the GUI loop occur, with the run call being put on the queue but the GUI loop kicking in before the run call can be processed. This manifests in the same way (the run call is put into the console and nothing happens until the kernel is prodded to exit the GUI loop, for example by setting the working directory), but feels much more localised to me mashing the run button than the issue as described originally.

@James-F1dev could you specifically confirm that this happens on Windows under Tkinter for you? The Qt and Tkinter loops work in very different ways on Windows, so this would be very useful to know. I know you said that Tkinter is also affected in your original report, but I am very confused as to how this could happen under Tkinter on Windows (as it should return focus to the main kernel thread every 10ms or so), so thought I’d double-check.

I think there is a fairly simple fix to the way IPykernel schedules GUI loop runs that we could implement, but it would be useful to pinpoint the issue first so we know this is for sure the problem. @James-F1dev @AstroALee @gica3618 sorry for the mass ping, but you seem to be able to reproduce this - if you feel like doing small surgery on your ipykernel installation (most likely in lib/site-packages/ipykernel in your Python installation/virtual environment folder) it would be super useful to know if applying the changes from https://github.com/jdranczewski/ipykernel/commit/30da44ab4d355ce82750743ef3cd89bed7912f21 resolves the issue for you. Thanks!

jdranczewski on Feb 18, 2024

I tried the current, unpatched version of IPykernel. Will do more testing over the weekend!

jdranczewski on Feb 17, 2024

Hi @ccordoba12, my understanding is that https://github.com/ipython/ipykernel/pull/1212 only solves the kernel hanging after an exception is raised and the kernel starts aborting future requests (which doesn’t actually happen in my tests in SpyderShell ~~for reasons I don’t yet understand~~ edit: because SpyderShell or some related widget/process sends an additional is_complete_request message to the kernel before the execute_request message - the eventloop exits due to the is_complete_request message, the abort stop method is called, and only then the execute_request message arrives).

This issue seems to be the kernel getting stuck (potentially in the eventloop) without an exception, which shouldn’t really be impacted by my PR, but may be an adjacent problem? The Tkinter (on Mac and Linux) and Qt (on any platform) ipykernel eventloops are set up to exit on a ZMQ socket event, which a call to execute code should produce I think? From what I can see in the YouTube video, perhaps the execution of the file fails to exit the eventloop and then setting the working directory sends a ZMQ message and exits correctly, resuming file execution? If this was the problem I’d expect it to be reproducible on any platform where the eventloops exit on socket event rather than every kernel interval (so on any platform for Qt, and Linux and Mac for Tkinter). I wasn’t able to reproduce this behaviour on Windows, so this may be something quite different? I don’t have access to a Mac, but could maybe try on Ubuntu…

jdranczewski on Feb 16, 2024

Thank you @ccordoba12. Now looking closer at https://github.com/spyder-ide/spyder/issues/16400, I think it’s a different issue from the one I have. I can restart the kernel even when choosing Backend “Automatic”. The problem I have is exactly like described here and shown in the video by @AstroALee. I tried the workaround anyway, but it didn’t solve the issue.

gica3618 on Feb 16, 2024