infrastructure: [UToronto] Pod scheduled on full node and spawn failure
Description of problem and opportunity to address it
New ticket from UToronto folks reporting a spawn failure (that has also happened in the past). This is the ticket https://2i2c.freshdesk.com/a/tickets/79
Relevant logs reported:

Updates and ongoing work
Initially
I first thought the spawn timeout was because the new node took to much time to come up, but as @consideRatio pointed out, there isn’t actually any scale-up event in the logs.
To investigate
I’m copy-pasting what @consideRatio noted in the ticket because I believe it’s very relevant and presents a great debugging strategy of this issue:
If it tried to scale up, I would expect to see a note about a scale up event - but instead there are two “successfully assigned pod” events - but then there is a note about “too many pods” and the pod being kicked out of the node.
Questions to debug:
- why are the user pod being assigned to a full node?
- one lead to consider is the custom pod scheduler we use to schedule the user pods onto nodes. The JupyterHub Helm chart starts a dedicated kube-scheduler that may fail to acknowledge that a node may have too many pods and schedule it there even though it shouldn’t. The user-scheduler is running a kube-scheduler binary with custom configuration. This kube-scheduler binary comes from a fixed version of k8s, maybe it should be updated because the scheduling logic is different now compared to before - or had a bug related to this etc?
- the logs of the user-scheduler pod assigning this user pod to a node is a source of relevant information I think, but probably quite technical in nature.
Possible theory
Race condition because we have multiple schedulers running on the cluster - the default cluster scheduler and the custom user scheduler. They could be racing each other, and the following sequence of events might be happening.
- User scheduler sees nodeA with 109 pods
- Cluster scheduler sees nodeA with 109 pods
- Cluster scheduler assigns podA to nodeA
- User scheduler doesn’t update current status of nodeA, but assigns podB to nodeA
- Kubelet on nodeA pulls podA, and when it tries to pull podB it fails with this error message - 110 is a kubernetes default per-pod node limit.
Steps that could fix it
- Update
kube-schedulerversion ->https://github.com/2i2c-org/infrastructure/issues/1055
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 17 (17 by maintainers)
Commits related to this issue
- Use OPTIMIZE_UTILIZATION cluster autoscaler profile GKE supports setting a cluster autoscaler profile that optimizes utilization[1]. This changes the behavior of the autoscaler *and* the scheduler, p... — committed to yuvipanda/pilot-hubs by yuvipanda 5 years ago
- Use OPTIMIZE_UTILIZATION cluster autoscaler profile GKE supports setting a cluster autoscaler profile that optimizes utilization[1]. This changes the behavior of the autoscaler *and* the scheduler, p... — committed to yuvipanda/pilot-hubs by yuvipanda 5 years ago
OK so it sounds like this one will be resolved (hopefully!) when this issue is resolved:
I’ll add that to the top comment, and since this is an improvement over time and not a continuous active fire, I think that we can close out this incident. I’ll let the Toronto folks know to watch that issue.
If nobody objects in an hour or two, I’ll close this and email them w/ this info
Since the email has been sent, let’s close the incident!
@damianavila good point - I’ve sent them an email today!
Hey all - I see that in this comment:
We suggested closing this one once those two PRs are merged. They are now both merged. Should we close this issue? If not, what is our next step here?
This can be tricky sometimes due to how kube-scheduler has changed, and given that we want to support a few versions of k8s at the same time. Here is a relevant note from z2jh about the need to bump RBAC resources.
https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/6f4609e2f73258c7e0146f8e849d31aa7fe7f09f/jupyterhub/values.yaml#L432-L436
I’ll look into this right now a bit. See https://github.com/jupyterhub/zero-to-jupyterhub-k8s/pull/2590
One possibility is that it’s a race condition because we have multiple schedulers running on the cluster - the default cluster scheduler and the custom user scheduler. They could be racing each other, and the following sequence of events might be happening.
I encountered something like this (but not exactly this) at the Berkeley cluster. There, I worked around it by configuring the GKE cluster scheduler itself to behave closer to our user scheduler (see https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler#autoscaling_profiles). I don’t think such a thing is possible in AKS yet. If so, I think we should:
@damianavila, I believe it’s 110 with 110. But yes, I’ve checked and 110 is the max number of pods allowed per node.
Findings
I analyzed the dashboards a bit and this is what it shows:
The peaks are aprox 20min after the timestamp of the error reported (btw, the log timestamps are UTC, right?)
Also, I never see that node in the logs having 110 pods on it 🤔 It was at 108 at the time the logs reported the node full.
There are also a few pods consuming a significant amount of RAM in that time interval.
A new node did come up 15min after this error, even if the logs didn’t show a scale-up event