notebook: Connection failed, kernel does not exist
Describe the bug
I’m experiencing connection/kernel issues with my local copy of Jupyter. I’m not sure if this is the right place to report this, but I encounter the issue with both jupyter lab and jupyter notebook so it may be an issue with a common core/server dependency.
To Reproduce
- Launch Jupyter with
jupyter laborjupyter notebook - Create a new notebook with a simple
print('hello world')line - Try to run this line
- Nothing happens, errors in log
Expected behavior
I would Jupyter to find and use the kernel and run my code properly.
Screenshots
With jupyter lab I see the following log:
$ jupyter lab
[I 2022-05-25 17:58:18.321 ServerApp] jupyterlab | extension was successfully linked.
[I 2022-05-25 17:58:18.398 LabApp] JupyterLab extension loaded from /Users/ajstewart/spack/opt/spack/darwin-monterey-m1/apple-clang-13.1.6/py-jupyterlab-3.4.2-edi2q5wc43g627lu2gkmo2xtawiuv7jy/lib/python3.9/site-packages/jupyterlab
[I 2022-05-25 17:58:18.398 LabApp] JupyterLab application directory is /Users/ajstewart/spack/opt/spack/darwin-monterey-m1/apple-clang-13.1.6/py-jupyterlab-3.4.2-edi2q5wc43g627lu2gkmo2xtawiuv7jy/share/jupyter/lab
[I 2022-05-25 17:58:18.400 ServerApp] jupyterlab | extension was successfully loaded.
[I 2022-05-25 17:58:18.400 ServerApp] Serving notebooks from local directory: /Users/ajstewart/Downloads
[I 2022-05-25 17:58:18.400 ServerApp] Jupyter Server 1.17.0 is running at:
[I 2022-05-25 17:58:18.400 ServerApp] http://localhost:8888/lab?token=9375a706106d9afad83c4a544c502ad6237d5d3e21ddb9fe
[I 2022-05-25 17:58:18.400 ServerApp] or http://127.0.0.1:8888/lab?token=9375a706106d9afad83c4a544c502ad6237d5d3e21ddb9fe
[I 2022-05-25 17:58:18.400 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2022-05-25 17:58:18.404 ServerApp]
To access the server, open this file in a browser:
file:///Users/ajstewart/Library/Jupyter/runtime/jpserver-67212-open.html
Or copy and paste one of these URLs:
http://localhost:8888/lab?token=9375a706106d9afad83c4a544c502ad6237d5d3e21ddb9fe
or http://127.0.0.1:8888/lab?token=9375a706106d9afad83c4a544c502ad6237d5d3e21ddb9fe
[W 2022-05-25 17:58:22.067 LabApp] Could not determine jupyterlab build status without nodejs
[I 2022-05-25 17:58:22.717 ServerApp] Kernel started: 280c43a0-d3b9-4763-af38-8e49846055a3
[I 2022-05-25 17:58:25.714 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (1/5), new random ports
[I 2022-05-25 17:58:28.743 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (2/5), new random ports
[I 2022-05-25 17:58:31.787 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (3/5), new random ports
[I 2022-05-25 17:58:34.825 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (4/5), new random ports
[I 2022-05-25 17:58:37.868 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (5/5), new random ports
[W 2022-05-25 17:58:40.896 ServerApp] AsyncIOLoopKernelRestarter: restart failed
[W 2022-05-25 17:58:40.898 ServerApp] Kernel 280c43a0-d3b9-4763-af38-8e49846055a3 died, removing from map.
[W 2022-05-25 17:59:22.731 ServerApp] Timeout waiting for kernel_info reply from 280c43a0-d3b9-4763-af38-8e49846055a3
[W 2022-05-25 17:59:22.734 ServerApp] 404 GET /api/kernels/280c43a0-d3b9-4763-af38-8e49846055a3/channels?session_id=1fa21670-fa85-4f65-8c05-12b7f13f9272 (::1): Kernel does not exist: 280c43a0-d3b9-4763-af38-8e49846055a3
[W 2022-05-25 17:59:22.735 ServerApp] 404 GET /api/kernels/280c43a0-d3b9-4763-af38-8e49846055a3/channels?session_id=41ef9206-5db6-4404-88c4-37c5feed8356 (127.0.0.1): Kernel does not exist: 280c43a0-d3b9-4763-af38-8e49846055a3
[I 2022-05-25 17:59:56.360 ServerApp] Kernel started: 63cced8e-a8b3-417e-a20a-6a2d8be40be1
[I 2022-05-25 17:59:59.348 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (1/5), new random ports
[I 2022-05-25 18:00:02.374 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (2/5), new random ports
[I 2022-05-25 18:00:05.418 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (3/5), new random ports
[I 2022-05-25 18:00:08.451 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (4/5), new random ports
[I 2022-05-25 18:00:11.477 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (5/5), new random ports
[W 2022-05-25 18:00:14.523 ServerApp] AsyncIOLoopKernelRestarter: restart failed
[W 2022-05-25 18:00:14.525 ServerApp] Kernel 63cced8e-a8b3-417e-a20a-6a2d8be40be1 died, removing from map.
With jupyter notebook I see the following log:
$ jupyter notebook test.ipynb
[I 18:00:22.873 NotebookApp] Serving notebooks from local directory: /Users/ajstewart/Downloads
[I 18:00:22.873 NotebookApp] Jupyter Notebook 6.4.11 is running at:
[I 18:00:22.873 NotebookApp] http://localhost:8888/?token=002d2f8c5e2fe78ac0d44b9829a661f0f0fcb4f3fa269d8d
[I 18:00:22.873 NotebookApp] or http://127.0.0.1:8888/?token=002d2f8c5e2fe78ac0d44b9829a661f0f0fcb4f3fa269d8d
[I 18:00:22.873 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 18:00:22.879 NotebookApp]
To access the notebook, open this file in a browser:
file:///Users/ajstewart/Library/Jupyter/runtime/nbserver-67477-open.html
Or copy and paste one of these URLs:
http://localhost:8888/?token=002d2f8c5e2fe78ac0d44b9829a661f0f0fcb4f3fa269d8d
or http://127.0.0.1:8888/?token=002d2f8c5e2fe78ac0d44b9829a661f0f0fcb4f3fa269d8d
[W 18:00:25.247 NotebookApp] 404 GET /nbextensions/widgets/notebook/js/extension.js (::1) 4.050000ms referer=http://localhost:8888/notebooks/test.ipynb
[I 18:00:25.328 NotebookApp] Kernel started: dc6a45c0-2768-48c9-bbec-ff8e340f3320, name: python3
[I 18:00:28.320 NotebookApp] KernelRestarter: restarting kernel (1/5), new random ports
[I 18:00:31.367 NotebookApp] KernelRestarter: restarting kernel (2/5), new random ports
[I 18:00:34.403 NotebookApp] KernelRestarter: restarting kernel (3/5), new random ports
[I 18:00:37.437 NotebookApp] KernelRestarter: restarting kernel (4/5), new random ports
[I 18:00:40.470 NotebookApp] KernelRestarter: restarting kernel (5/5), new random ports
[W 18:00:43.502 NotebookApp] KernelRestarter: restart failed
[W 18:00:43.504 NotebookApp] Kernel dc6a45c0-2768-48c9-bbec-ff8e340f3320 died, removing from map.
[W 18:01:25.337 NotebookApp] Timeout waiting for kernel_info reply from dc6a45c0-2768-48c9-bbec-ff8e340f3320
[E 18:01:25.344 NotebookApp] Error opening stream: HTTP 404: Not Found (Kernel does not exist: dc6a45c0-2768-48c9-bbec-ff8e340f3320)
[W 18:01:26.800 NotebookApp] 404 GET /api/kernels/dc6a45c0-2768-48c9-bbec-ff8e340f3320/channels?session_id=b7dbf7410a154f7d944fea051d6d4e17 (::1): Kernel does not exist: dc6a45c0-2768-48c9-bbec-ff8e340f3320
[W 18:01:26.803 NotebookApp] 404 GET /api/kernels/dc6a45c0-2768-48c9-bbec-ff8e340f3320/channels?session_id=b7dbf7410a154f7d944fea051d6d4e17 (::1) 7.810000ms referer=None
[W 18:01:29.604 NotebookApp] 404 GET /api/kernels/dc6a45c0-2768-48c9-bbec-ff8e340f3320/channels?session_id=b7dbf7410a154f7d944fea051d6d4e17 (::1): Kernel does not exist: dc6a45c0-2768-48c9-bbec-ff8e340f3320
[W 18:01:29.607 NotebookApp] 404 GET /api/kernels/dc6a45c0-2768-48c9-bbec-ff8e340f3320/channels?session_id=b7dbf7410a154f7d944fea051d6d4e17 (::1) 7.040000ms referer=None
[W 18:01:34.349 NotebookApp] 404 GET /api/kernels/dc6a45c0-2768-48c9-bbec-ff8e340f3320/channels?session_id=b7dbf7410a154f7d944fea051d6d4e17 (::1): Kernel does not exist: dc6a45c0-2768-48c9-bbec-ff8e340f3320
[W 18:01:34.351 NotebookApp] 404 GET /api/kernels/dc6a45c0-2768-48c9-bbec-ff8e340f3320/channels?session_id=b7dbf7410a154f7d944fea051d6d4e17 (::1) 5.120000ms referer=None
[W 18:01:42.372 NotebookApp] 404 GET /api/kernels/dc6a45c0-2768-48c9-bbec-ff8e340f3320/channels?session_id=b7dbf7410a154f7d944fea051d6d4e17 (::1): Kernel does not exist: dc6a45c0-2768-48c9-bbec-ff8e340f3320
[W 18:01:42.376 NotebookApp] 404 GET /api/kernels/dc6a45c0-2768-48c9-bbec-ff8e340f3320/channels?session_id=b7dbf7410a154f7d944fea051d6d4e17 (::1) 7.250000ms referer=None
I also see a popup in the browser:
Connection failed
A connection to the notebook server could not be established. The notebook will continue trying to reconnect. Check your network connection or notebook server configuration.
Desktop (please complete the following information):
- OS: macOS 12.4 (Apple M1 Pro, arm64)
- Browser: Chrome 101.0.4951.64 (Official Build) (arm64)
- Version: 6.4.11
Additional context
All Jupyter libraries are built from source using the Spack package manager. Here are all versions:
py-jupyterlab@3.4.2
py-ipython@8.0.1
py-appnope@0.1.0
py-pip@21.3.1
python@3.9.12
apple-libuuid@1353.100.2
bzip2@1.0.8
diffutils@3.8
gnuconfig@2021-08-14
libiconv@1.16
expat@2.4.8
gdbm@1.19
readline@8.1
ncurses@6.2
pkgconf@1.8.0
gettext@0.21
libxml2@2.9.13
xz@5.2.5
zlib@1.2.12
tar@1.34
libffi@3.4.2
openssl@1.1.1o
perl@5.34.1
berkeley-db@18.1.40
sqlite@3.38.5
py-setuptools@59.4.0
py-wheel@0.37.0
py-backcall@0.2.0
py-flit-core@3.6.0
py-black@22.1.0
py-click@8.0.3
py-mypy-extensions@0.4.3
py-pathspec@0.9.0
py-platformdirs@2.4.0
py-setuptools-scm@6.3.2
py-packaging@21.3
py-pyparsing@3.0.6
py-tomli@1.2.2
py-setuptools@62.3.2
py-setuptools-scm@6.3.2
py-typing-extensions@4.1.1
py-decorator@5.1.1
py-jedi@0.18.1
py-parso@0.8.2
py-matplotlib-inline@0.1.3
py-traitlets@5.1.1
py-pexpect@4.8.0
py-ptyprocess@0.7.0
py-pickleshare@0.7.5
py-prompt-toolkit@3.0.29
py-wcwidth@0.2.5
py-pygments@2.10.0
py-stack-data@0.2.0
py-asttokens@2.0.5
py-six@1.16.0
py-executing@0.8.2
py-pure-eval@0.2.2
py-jinja2@3.0.3
py-markupsafe@2.0.1
py-jupyter-core@4.9.2
py-jupyter-packaging11@0.12.0
py-deprecation@2.1.0
py-tomlkit@0.7.2
py-poetry-core@1.0.8
py-jupyter-server@1.17.0
py-anyio@3.5.0
py-idna@3.3
py-sniffio@1.2.0
py-argon2-cffi@21.3.0
py-argon2-cffi-bindings@21.2.0
py-cffi@1.15.0
py-pycparser@2.20
py-jupyter-client@7.1.2
py-entrypoints@0.4
py-nest-asyncio@1.5.4
py-python-dateutil@2.8.2
py-pyzmq@22.3.0
libzmq@4.3.4
libsodium@1.0.18
py-cython@0.29.24
py-gevent@1.5.0
py-greenlet@1.1.2
py-py@1.11.0
py-tornado@6.1
py-nbconvert@6.5.0
py-beautifulsoup4@4.10.0
py-soupsieve@2.2.1
py-bleach@4.1.0
py-webencodings@0.5.1
py-defusedxml@0.7.1
py-jupyterlab-pygments@0.1.2
py-mistune@0.8.4
py-nbclient@0.5.5
py-nbformat@5.4.0
py-fastjsonschema@2.15.1
py-jsonschema@4.4.0
py-attrs@21.4.0
py-pyrsistent@0.18.0
py-pandocfilters@1.5.0
py-tinycss2@1.1.1
py-pre-commit@2.17.0
py-cfgv@2.0.1
py-identify@1.4.7
py-nodeenv@1.3.3
py-pyyaml@6.0
libyaml@0.2.5
py-toml@0.10.2
py-virtualenv@20.10.0
py-backports-entry-points-selectable@1.1.1
py-distlib@0.3.4
py-filelock@3.5.0
py-prometheus-client@0.12.0
py-send2trash@1.8.0
py-terminado@0.12.1
py-websocket-client@1.2.1
py-jupyterlab-server@2.10.3
py-babel@2.9.1
py-pytz@2021.3
py-json5@0.9.6
py-requests@2.26.0
py-certifi@2021.10.8
py-charset-normalizer@2.0.12
py-urllib3@1.26.6
py-nbclassic@0.3.5
py-notebook@6.4.11
py-ipykernel@6.9.1
py-debugpy@1.5.1
py-ipython-genutils@0.2.0
Let me know if there are any additional steps I can take to debug this or at least figure out which of these many dependencies has an issue. Also let me know if any of these dependencies should be updated.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 23 (10 by maintainers)
@adamjstewart I see the same thing with my installation from July 12 😦
ipykernel is not a direct dependency in the Python sense. nbclient doesn’t call ipykernel methods directly. It issues messages to configured ZMQ sockets (that the two processes know about via the shared connection information). This decoupling enables the ability to have different kernel implementations for various languages, etc., because this communication goes over “generic” sockets.
However, in this case, nbclient is not able to start issuing cell contents as messages because the initial handshaking that occurs during a kernel’s startup has not been completed successfully.
Thanks for the response (and your patience)! This is helpful, although I still cannot determine the cause of the kernel’s abrupt termination.
I believe this issue should be investigated from the
ipykernelside of things, but given this appears to only occur in a Spack env, it probably means you will need to troubleshoot further.Based on the ipykernel debug output, we can determine that, in all likelihood, the kernel has completed the
init_io()method since the “Seeing logger to stderr” message is logged there in what appears to be general behavior (not an error condition). Theinit_io()method is called frominitialize()and followinginitialize()is thestart()method.Barring other ideas, I think it would be interesting to instrument additional logging in
initialize()andstart()and see where that takes you. (You should probably confirm the issue still occurs using a custom-built ipykernel prior to making any changes.)@blink1073 - does it make sense to transfer this issue to ipykernel for now? I suppose it could be an issue in
jupyter_clientbut it seems like it should be looked at fromipykernelat the moment. I think the related issue injupyter_clientcan be closed since we would likely transfer this issue there if it’s determined to be a jupyter_client issue.Also, do you happen to have any ideas here?
Given the cross-post to
jupyter_clientI’m assuming this is still an issue. These symptoms are indicative of the kernel process starting just enough to satisfy the start kernel request, but then that process terminates, at which time the auto-restart behavior is triggered, attempting the additional restart attempts.I would recommend adding
--debugwhen starting the notebook server and seeing if that produces any additional (and useful) information.You might also try running the notebook via an application like papermill (e.g.,
papermill my-nb.ipynb my-nb-output.ipynb) to better isolate the issue since that would remove the web portion of things and may provide additional datapoints via its error reporting. You will not see the 5 restart attempts with this approach.