infrastructure: Debug intermittent failure of servers to start in LEAP hub

In https://2i2c.freshdesk.com/a/tickets/365, it was reported that there was failure to start on some LEAP hub users intermittently. Looking at the hub logs, I discovered a number of:

[W 2023-01-11 19:22:05.428 JupyterHub user:881] <user> server never showed up at http://10.0.81.5:8888/user/<user>/ after 30 seconds. Giving up.
    
    Common causes of this timeout, and debugging tips:
    
    1. The server didn't finish starting,
       or it crashed due to a configuration issue.
       Check the single-user server's logs for hints at what needs fixing.
    2. The server started, but is not accessible at the specified URL.
       This may be a configuration issue specific to your chosen Spawner.
       Check the single-user server logs and resource to make sure the URL
       is correct and accessible from the Hub.
    3. (unlikely) Everything is working, but the server took too long to respond.
       To fix: increase `Spawner.http_timeout` configuration
       to a number of seconds that is enough for servers to become responsive.
    
[I 2023-01-11 19:22:05.429 JupyterHub spawner:2808] Deleting pod prod/jupyter-<user>

at relevant times.

Upon looking at the user’s pod logs (using techniques described in #2046), I see at the matching timestamp:

[C 2023-01-11 19:22:05.491 SingleUserNotebookApp notebookapp:1971] received signal 15, stopping

This means the pod started, but notebook process inside the pod hadn’t started in time, and the ‘Deleting pod’ sent the appropriate signal to the user pod to shut down. So this isn’t a spawn timeout but a http timeout.

In the user logs, the line preceeding the ‘received signal 15’ is:

[I 2023-01-11 21:36:56.540 LabApp] JupyterLab application directory is /srv/conda/envs/notebook/share/jupyter/lab

If I look at the logs from a successful startup, I see:

[I 2023-01-11 22:15:01.495 SingleUserNotebookApp mixins:609] Starting jupyterhub single-user server version 3.1.0
[I 2023-01-11 22:15:01.495 SingleUserNotebookApp mixins:623] Extending notebook.notebookapp.NotebookApp from notebook 6.5.2
[I 2023-01-11 22:15:03.047 SingleUserNotebookApp manager:91] [nb_conda_kernels] enabled, 0 kernels found
[I 2023-01-11 22:15:03.155 SingleUserNotebookApp notebookapp:1592] Authentication of /metrics is OFF, since other authentication is disabled.
[W 2023-01-11 22:15:10.165 LabApp] Config option `kernel_spec_manager_class` not recognized by `LabApp`.
[W 2023-01-11 22:15:10.169 LabApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[W 2023-01-11 22:15:10.169 LabApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[W 2023-01-11 22:15:10.171 LabApp] Config option `kernel_spec_manager_class` not recognized by `LabApp`.
[W 2023-01-11 22:15:10.181 LabApp] Config option `kernel_spec_manager_class` not recognized by `LabApp`.
[I 2023-01-11 22:15:10.183 LabApp] JupyterLab extension loaded from /srv/conda/envs/notebook/lib/python3.10/site-packages/jupyterlab
[I 2023-01-11 22:15:10.184 LabApp] JupyterLab application directory is /srv/conda/envs/notebook/share/jupyter/lab
[I 2023-01-11 22:15:22.292 SingleUserNotebookApp mixins:670] Starting jupyterhub-singleuser server version 3.1.0
[W 2023-01-11 22:15:22.304 SingleUserNotebookApp _version:68] jupyterhub version 3.0.0 != jupyterhub-singleuser version 3.1.0. This could cause failure to authenticate and result in redirect loops!
[I 2023-01-11 22:15:22.304 SingleUserNotebookApp notebookapp:2328] Serving notebooks from local directory: /home/jovyan
[I 2023-01-11 22:15:22.304 SingleUserNotebookApp notebookapp:2328] Jupyter Notebook 6.5.2 is running at:

So right after:

[I 2023-01-11 22:15:10.184 LabApp] JupyterLab application directory is /srv/conda/envs/notebook/share/jupyter/lab

you see

[I 2023-01-11 22:15:22.292 SingleUserNotebookApp mixins:670] Starting jupyterhub-singleuser server version 3.1.0

However, it took 12 seconds to arrive! That is a very long time for something to be happening here, and the notebook server is not ‘ready’ until well after that. It is quite likely that sometimes, it takes longer than 30s and hence the timeout happens.

Ideally, all this should really be taking under a second so it is unclear why things are taking as long as they are. My suspicion is there’s a notebook extension that’s doing some blocking code execution on startup, causing this - but not sure how exactly to look for that.

Items to resolve support ticket

Items to resolve to close this

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 4
  • Comments: 29 (27 by maintainers)

Commits related to this issue

Most upvoted comments

@jcrist suggested in https://hachyderm.io/@jcristharif/109707037385576616 that the performance issue might be the write time of .pyc files, not the parsing itself - and suggested a method to test this.

So I did!

No .pyc files, no env var:

(notebook) jovyan@jupyter-yuvipanda:~$ time python -c 'import panel.io.jupyter_server_extension'

real    0m18.088s
user    0m10.299s
sys     0m1.258s
(notebook) jovyan@jupyter-yuvipanda:~$ time python -c 'import panel.io.jupyter_server_extension'

real    0m4.797s
user    0m3.652s
sys     0m0.521s

The .pyc files are created during the first invocation, so the second invocation is about 360% faster.

No .pyc files, but PYTHONDONTWRITEBYTECODE=1

(notebook) jovyan@jupyter-yuvipanda:~$ time python -c 'import panel.io.jupyter_server_extension'

real    0m12.763s
user    0m9.966s
sys     0m0.783s
(notebook) jovyan@jupyter-yuvipanda:~$ time python -c 'import panel.io.jupyter_server_extension'

real    0m12.287s
user    0m10.496s
sys     0m0.784s

I validated that no .pyc files are created, and the second invocation is about the same speed.

So from this, while not creating .pyc files at all does help with startup performance (12s vs 18s), it’s still a lot more than with .pyc files present (~4s).

I checked the size of the .pyc files created, and that’s 17MB in this case. I think shipping 17MB is a reasonable trade-off here…

Will make Panel 0.14.3 available some time next week which should reduce import time by at least half based on my testing.

To mark this issue as completed, I’d want to bump the LEAP hub image to include all the fixes we have in place.

I opened https://github.com/pangeo-data/pangeo-docker-images/pull/433 to include xarray-leaflet. I don’t think bumping jupyter_server is required, but would be good to get panel in there.

Thanks for the writeup Yuvi! Turns out my intuition was wrong (as it often is). Your conclusion to not strip the .pyc files makes sense to me 👍.

Amazing, @philippjfr! Thank you for your prompt work 😃

I also opened https://github.com/jupyter-server/jupyter_server/issues/1177 to get jupyter-server to stop importing python packages if they are disabled.

I’ve also made this PR to jupyter_server to surface this performance information more easily, so we can see the culprits: https://github.com/jupyter-server/jupyter_server/pull/1171

Since server startup is definitely much faster when .pyc files are present, I opened a PR in pangeo-docker-images to stop deleting them: https://github.com/pangeo-data/pangeo-docker-images/pull/426.

I also opened https://github.com/xarray-contrib/xarray_leaflet/issues/79 in xarray-leaflet

https://github.com/2i2c-org/infrastructure/pull/2048 will temporarily increase the start timeout to make launches more robust, as we try to figure out why in the world it takes so long to start up