infrastructure: Debug intermittent failure of servers to start in LEAP hub
In https://2i2c.freshdesk.com/a/tickets/365, it was reported that there was failure to start on some LEAP hub users intermittently. Looking at the hub logs, I discovered a number of:
[W 2023-01-11 19:22:05.428 JupyterHub user:881] <user> server never showed up at http://10.0.81.5:8888/user/<user>/ after 30 seconds. Giving up.
Common causes of this timeout, and debugging tips:
1. The server didn't finish starting,
or it crashed due to a configuration issue.
Check the single-user server's logs for hints at what needs fixing.
2. The server started, but is not accessible at the specified URL.
This may be a configuration issue specific to your chosen Spawner.
Check the single-user server logs and resource to make sure the URL
is correct and accessible from the Hub.
3. (unlikely) Everything is working, but the server took too long to respond.
To fix: increase `Spawner.http_timeout` configuration
to a number of seconds that is enough for servers to become responsive.
[I 2023-01-11 19:22:05.429 JupyterHub spawner:2808] Deleting pod prod/jupyter-<user>
at relevant times.
Upon looking at the user’s pod logs (using techniques described in #2046), I see at the matching timestamp:
[C 2023-01-11 19:22:05.491 SingleUserNotebookApp notebookapp:1971] received signal 15, stopping
This means the pod started, but notebook process inside the pod hadn’t started in time, and the ‘Deleting pod’ sent the appropriate signal to the user pod to shut down. So this isn’t a spawn timeout but a http timeout.
In the user logs, the line preceeding the ‘received signal 15’ is:
[I 2023-01-11 21:36:56.540 LabApp] JupyterLab application directory is /srv/conda/envs/notebook/share/jupyter/lab
If I look at the logs from a successful startup, I see:
[I 2023-01-11 22:15:01.495 SingleUserNotebookApp mixins:609] Starting jupyterhub single-user server version 3.1.0
[I 2023-01-11 22:15:01.495 SingleUserNotebookApp mixins:623] Extending notebook.notebookapp.NotebookApp from notebook 6.5.2
[I 2023-01-11 22:15:03.047 SingleUserNotebookApp manager:91] [nb_conda_kernels] enabled, 0 kernels found
[I 2023-01-11 22:15:03.155 SingleUserNotebookApp notebookapp:1592] Authentication of /metrics is OFF, since other authentication is disabled.
[W 2023-01-11 22:15:10.165 LabApp] Config option `kernel_spec_manager_class` not recognized by `LabApp`.
[W 2023-01-11 22:15:10.169 LabApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[W 2023-01-11 22:15:10.169 LabApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[W 2023-01-11 22:15:10.171 LabApp] Config option `kernel_spec_manager_class` not recognized by `LabApp`.
[W 2023-01-11 22:15:10.181 LabApp] Config option `kernel_spec_manager_class` not recognized by `LabApp`.
[I 2023-01-11 22:15:10.183 LabApp] JupyterLab extension loaded from /srv/conda/envs/notebook/lib/python3.10/site-packages/jupyterlab
[I 2023-01-11 22:15:10.184 LabApp] JupyterLab application directory is /srv/conda/envs/notebook/share/jupyter/lab
[I 2023-01-11 22:15:22.292 SingleUserNotebookApp mixins:670] Starting jupyterhub-singleuser server version 3.1.0
[W 2023-01-11 22:15:22.304 SingleUserNotebookApp _version:68] jupyterhub version 3.0.0 != jupyterhub-singleuser version 3.1.0. This could cause failure to authenticate and result in redirect loops!
[I 2023-01-11 22:15:22.304 SingleUserNotebookApp notebookapp:2328] Serving notebooks from local directory: /home/jovyan
[I 2023-01-11 22:15:22.304 SingleUserNotebookApp notebookapp:2328] Jupyter Notebook 6.5.2 is running at:
So right after:
[I 2023-01-11 22:15:10.184 LabApp] JupyterLab application directory is /srv/conda/envs/notebook/share/jupyter/lab
you see
[I 2023-01-11 22:15:22.292 SingleUserNotebookApp mixins:670] Starting jupyterhub-singleuser server version 3.1.0
However, it took 12 seconds to arrive! That is a very long time for something to be happening here, and the notebook server is not ‘ready’ until well after that. It is quite likely that sometimes, it takes longer than 30s and hence the timeout happens.
Ideally, all this should really be taking under a second so it is unclear why things are taking as long as they are. My suspicion is there’s a notebook extension that’s doing some blocking code execution on startup, causing this - but not sure how exactly to look for that.
Items to resolve support ticket
- Wait for panel to make a release https://github.com/holoviz/panel/issues/4271#issuecomment-1396441302, then bump pangeo-docker image to use it panel 0.14.3 released, and updated in pangeo-docker-images
- Use newer version of xarary-leaflet in the docker image https://github.com/pangeo-data/pangeo-docker-images/pull/433 Observed to be resolved
- Use newly bumped pangeo-docker image in the LEAP hub Done in #2183
- Communicate the outcome in the freshdesk ticket
Items to resolve to close this
- Figure out if we can get https://jcristharif.com/conda-docker-tips.html amended
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 4
- Comments: 29 (27 by maintainers)
Commits related to this issue
- Temporarily increase HTTP timeout This gives the single user server process more time to start up, pending investigation in https://github.com/2i2c-org/infrastructure/issues/2047 — committed to yuvipanda/pilot-hubs by yuvipanda 4 years ago
- Don't delete *.pyc files from the image Panel and Xarray-Leaflet are heavy enough imports that without .pyc files, they sometimes together take as much as 15s to import?! This causes jupyterhub to fa... — committed to yuvipanda/pangeo-docker-images by yuvipanda 4 years ago
- Log how long each extension module takes to import Based on some detailed debugging in https://github.com/2i2c-org/infrastructure/issues/2047 to figure out why a Jupyter Server *process* somtimes tak... — committed to yuvipanda/jupyter_server by yuvipanda 4 years ago
- Log how long each extension module takes to import (#1171) * Log how long each extension module takes to import Based on some detailed debugging in https://github.com/2i2c-org/infrastructure/issu... — committed to jupyter-server/jupyter_server by yuvipanda 4 years ago
@jcrist suggested in https://hachyderm.io/@jcristharif/109707037385576616 that the performance issue might be the write time of .pyc files, not the parsing itself - and suggested a method to test this.
So I did!
No .pyc files, no env var:
The .pyc files are created during the first invocation, so the second invocation is about 360% faster.
No .pyc files, but
PYTHONDONTWRITEBYTECODE=1I validated that no .pyc files are created, and the second invocation is about the same speed.
So from this, while not creating .pyc files at all does help with startup performance (12s vs 18s), it’s still a lot more than with .pyc files present (~4s).
I checked the size of the .pyc files created, and that’s 17MB in this case. I think shipping 17MB is a reasonable trade-off here…
Will make Panel 0.14.3 available some time next week which should reduce import time by at least half based on my testing.
To mark this issue as completed, I’d want to bump the LEAP hub image to include all the fixes we have in place.
I opened https://github.com/pangeo-data/pangeo-docker-images/pull/433 to include xarray-leaflet. I don’t think bumping jupyter_server is required, but would be good to get panel in there.
Thanks for the writeup Yuvi! Turns out my intuition was wrong (as it often is). Your conclusion to not strip the
.pycfiles makes sense to me 👍.Amazing, @philippjfr! Thank you for your prompt work 😃
I also opened https://github.com/jupyter-server/jupyter_server/issues/1177 to get jupyter-server to stop importing python packages if they are disabled.
I’ve also made this PR to jupyter_server to surface this performance information more easily, so we can see the culprits: https://github.com/jupyter-server/jupyter_server/pull/1171
Since server startup is definitely much faster when
.pycfiles are present, I opened a PR in pangeo-docker-images to stop deleting them: https://github.com/pangeo-data/pangeo-docker-images/pull/426.I also opened https://github.com/xarray-contrib/xarray_leaflet/issues/79 in xarray-leaflet
I opened https://github.com/holoviz/panel/issues/4271 in panel
https://github.com/2i2c-org/infrastructure/pull/2048 will temporarily increase the start timeout to make launches more robust, as we try to figure out why in the world it takes so long to start up