infrastructure: [Incident] PaleoHack User sessions won't start

Summary

The PaleoHackWeek hub is not putting users on the new nodes that were pre-cooked for the event. Here are the hub logs:

2021-10-28T17:38:46Z [Warning] 0/23 nodes are available: 20 Insufficient cpu, 3 node(s) didn't match node selector.
2021-10-28T17:38:48Z [Normal] pod didn't trigger scale-up: 3 node(s) didn't match node selector, 1 max node group size reached

In addition, I’m not sure if this is relevant but it seems like all of the CPU commit %s are pinned at 62.6% and I’m not sure why. I wonder if that’s related to the 20 Insufficient cpu error above. Here’s an image of the grafana plot:

image

Timeline (if relevant)

See the comments below for the timeline details, all happened within a few hours. Here is a summary:


After-action report

What went wrong

Two major things:

  • We had a sub-optimal helm config that was causing Kubernetes to request all of the resources available to a pod.
  • The PaleoHack image didn’t have nbgitpuller on it, so their nbgitpuller links didn’t work

Action items

These are only sample subheadings. Every action item should have a GitHub issue (even a small skeleton of one) attached to it, so these do not get forgotten. These issues don’t have to be in infrastructure/, they can be in other repositories.

Process improvements

  1. Add steps to send to the community representative to make sure the hub is working properly before the event (# )

Documentation improvements

  1. https://github.com/2i2c-org/pilot/issues/111

Technical improvements

  1. Enforce that guarantees are always set for userpods, not just limits: https://github.com/2i2c-org/infrastructure/pull/801
  2. Validate our config against helm schema in the PR via CI: https://github.com/2i2c-org/infrastructure/issues/279

Actions

  • Incident has been dealt with or is over
  • Sections above are filled out
  • Incident title and after-action report is cleaned up
  • All actionable items above have linked GitHub Issues

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 18 (18 by maintainers)

Commits related to this issue

Most upvoted comments

It should be singleuser.cpu.guarantee. I’m just doing a quick manual deploy

OK I’ve updated the top comment and included links to follow-up issues, closing this one now! thanks everybody for your hard work 😃

I think everything seems to be in order now, and all users that are pending have been whiped. So, a restart by users should now make them fit on nodes.

Note that the deploy is still running! It is cycling through all of the hubs in the 2i2c pilot cluster, and is taking some time to get to the paleohack hub. I’m just writing this down here for future reference in the debrief, because it seems sub-optimal that we are waiting for the deploys on all hubs even though only the paleohack config has changed 😃

This is because the pods requests the same they do as they are limited to, making only a single user fit on each node.

    resources:
      limits:
        cpu: "2"
        memory: "4294967296"
      requests:
        cpu: "2"
        memory: "268435456"

By setting only limit, the requests were defaulting to the limit. The solution is to set requests and limit where requests are far lower than the limit.

Assigning @damianavila and @yuvipanda for visibility since I think they are the ones that did the pre-cooking