jupyter-scheduler: RuntimeError: "This event loop is already running" with nbclient==0.8.0

Related #330 but happening with nbclient==0.8.0

Description

When I execute a job, I get “This event loop is already running” error in the UI.

In the service logs, I get the following error.

Sep 29 11:57:51 tubuntu jupyter-notebook[857]: [I 2023-09-29 11:57:51.712 ServerApp] Connecting to kernel a687ffa6-7465-4686-b209-7c1af177a8c2.
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]: Traceback (most recent call last):
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:   File "/home/taras/mambaforge/envs/notebook_env/lib/python3.11/site-packages/jupyter_scheduler/executors.py", line 60, in process
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:     self.execute()
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:   File "/home/taras/mambaforge/envs/notebook_env/lib/python3.11/site-packages/jupyter_scheduler/executors.py", line 140, in execute
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:     ep.preprocess(nb)
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:   File "/home/taras/mambaforge/envs/notebook_env/lib/python3.11/site-packages/nbconvert/preprocessors/execute.py", line 94, in preprocess
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:     with self.setup_kernel():
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:   File "/home/taras/mambaforge/envs/notebook_env/lib/python3.11/contextlib.py", line 137, in __enter__
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:     return next(self.gen)
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:            ^^^^^^^^^^^^^^
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:   File "/home/taras/mambaforge/envs/notebook_env/lib/python3.11/site-packages/nbclient/client.py", line 596, in setup_kernel
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:     self.start_new_kernel(**kwargs)
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:   File "/home/taras/mambaforge/envs/notebook_env/lib/python3.11/site-packages/jupyter_core/utils/__init__.py", line 168, in wrapped
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:     return loop.run_until_complete(inner)
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:   File "/home/taras/mambaforge/envs/notebook_env/lib/python3.11/asyncio/base_events.py", line 629, in run_until_complete
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:     self._check_running()
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:   File "/home/taras/mambaforge/envs/notebook_env/lib/python3.11/asyncio/base_events.py", line 588, in _check_running
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:     raise RuntimeError('This event loop is already running')
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]: RuntimeError: This event loop is already running
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]: /home/taras/mambaforge/envs/notebook_env/lib/python3.11/site-packages/jupyter_scheduler/executors.py:64: RuntimeWarning: coroutine 'NotebookClient.async_start_new_kernel' was never awaited
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]:   self.on_failure(e)
Sep 29 11:58:02 tubuntu jupyter-notebook[36064]: RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Sep 29 11:58:22 tubuntu jupyter-notebook[857]: [W 2023-09-29 11:58:22.103 ServerApp] Notebook Matrix optimization/Post Sep 29/scripts/Trust_ColAdd_1.ipynb is not trusted

Reproduce

I’m using mamba to install Jupyter Notebook & JupyterLab and PIP to install Jupyter scheduler.

Expected behavior

Would love to be able to run jobs 😦

Context

• Troubleshoot Output
• Command Line Output