infrastructure: [Incident] PaleoHack User sessions won't start
Summary
The PaleoHackWeek hub is not putting users on the new nodes that were pre-cooked for the event. Here are the hub logs:
2021-10-28T17:38:46Z [Warning] 0/23 nodes are available: 20 Insufficient cpu, 3 node(s) didn't match node selector.
2021-10-28T17:38:48Z [Normal] pod didn't trigger scale-up: 3 node(s) didn't match node selector, 1 max node group size reached
In addition, I’m not sure if this is relevant but it seems like all of the CPU commit %s are pinned at 62.6% and I’m not sure why. I wonder if that’s related to the 20 Insufficient cpu error above. Here’s an image of the grafana plot:

Timeline (if relevant)
See the comments below for the timeline details, all happened within a few hours. Here is a summary:
- Initial report
- Identified that the issue was one user per node
- Realized that we did not add the proper helm config value to fix this
- We realized that the paleohack image didn’t have nbgitpuller and updated the environment there
- We realized that the configurator had its own Docker image configured that was clashing with our updates
- Confirmed the incident was resolved
After-action report
What went wrong
Two major things:
- We had a sub-optimal helm config that was causing Kubernetes to request all of the resources available to a pod.
- The PaleoHack image didn’t have nbgitpuller on it, so their nbgitpuller links didn’t work
Action items
These are only sample subheadings. Every action item should have a GitHub issue
(even a small skeleton of one) attached to it, so these do not get forgotten. These issues don’t have to be in infrastructure/, they can be in other repositories.
Process improvements
- Add steps to send to the community representative to make sure the hub is working properly before the event (# )
Documentation improvements
Technical improvements
- Enforce that guarantees are always set for userpods, not just limits: https://github.com/2i2c-org/infrastructure/pull/801
- Validate our config against helm schema in the PR via CI: https://github.com/2i2c-org/infrastructure/issues/279
Actions
- Incident has been dealt with or is over
- Sections above are filled out
- Incident title and after-action report is cleaned up
- All actionable items above have linked GitHub Issues
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 18 (18 by maintainers)
Commits related to this issue
- Use JupyterHub specific terminology for setting resource guarantees I think z2jh should support the kubernetes native terminology (requests) as well Ref https://github.com/2i2c-org/infrastructure/is... — committed to yuvipanda/pilot-hubs by yuvipanda 5 years ago
- Set a tiny CPU guarantee for all user pods When limit is set but not guarantee, guarantee is set to match limit! For most use cases, when we set limit but not guarantee, we want to offer no guarantee... — committed to yuvipanda/pilot-hubs by yuvipanda 5 years ago
It should be singleuser.cpu.guarantee. I’m just doing a quick manual deploy
OK I’ve updated the top comment and included links to follow-up issues, closing this one now! thanks everybody for your hard work 😃
I think everything seems to be in order now, and all users that are pending have been whiped. So, a restart by users should now make them fit on nodes.
Note that the deploy is still running! It is cycling through all of the hubs in the 2i2c pilot cluster, and is taking some time to get to the paleohack hub. I’m just writing this down here for future reference in the debrief, because it seems sub-optimal that we are waiting for the deploys on all hubs even though only the paleohack config has changed 😃
This is because the pods requests the same they do as they are limited to, making only a single user fit on each node.
By setting only limit, the requests were defaulting to the limit. The solution is to set requests and limit where requests are far lower than the limit.
Assigning @damianavila and @yuvipanda for visibility since I think they are the ones that did the pre-cooking