infrastructure: [UToronto] Pod scheduled on full node and spawn failure

Description of problem and opportunity to address it

New ticket from UToronto folks reporting a spawn failure (that has also happened in the past). This is the ticket https://2i2c.freshdesk.com/a/tickets/79

Relevant logs reported: toront-logs

Updates and ongoing work

Initially

I first thought the spawn timeout was because the new node took to much time to come up, but as @consideRatio pointed out, there isn’t actually any scale-up event in the logs.

To investigate

I’m copy-pasting what @consideRatio noted in the ticket because I believe it’s very relevant and presents a great debugging strategy of this issue:

If it tried to scale up, I would expect to see a note about a scale up event - but instead there are two “successfully assigned pod” events - but then there is a note about “too many pods” and the pod being kicked out of the node.

Questions to debug:

  • why are the user pod being assigned to a full node?
  • one lead to consider is the custom pod scheduler we use to schedule the user pods onto nodes. The JupyterHub Helm chart starts a dedicated kube-scheduler that may fail to acknowledge that a node may have too many pods and schedule it there even though it shouldn’t. The user-scheduler is running a kube-scheduler binary with custom configuration. This kube-scheduler binary comes from a fixed version of k8s, maybe it should be updated because the scheduling logic is different now compared to before - or had a bug related to this etc?
  • the logs of the user-scheduler pod assigning this user pod to a node is a source of relevant information I think, but probably quite technical in nature.

Possible theory

Race condition because we have multiple schedulers running on the cluster - the default cluster scheduler and the custom user scheduler. They could be racing each other, and the following sequence of events might be happening.

  1. User scheduler sees nodeA with 109 pods
  2. Cluster scheduler sees nodeA with 109 pods
  3. Cluster scheduler assigns podA to nodeA
  4. User scheduler doesn’t update current status of nodeA, but assigns podB to nodeA
  5. Kubelet on nodeA pulls podA, and when it tries to pull podB it fails with this error message - 110 is a kubernetes default per-pod node limit.

Steps that could fix it

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 17 (17 by maintainers)

Commits related to this issue

Most upvoted comments

OK so it sounds like this one will be resolved (hopefully!) when this issue is resolved:

I’ll add that to the top comment, and since this is an improvement over time and not a continuous active fire, I think that we can close out this incident. I’ll let the Toronto folks know to watch that issue.

If nobody objects in an hour or two, I’ll close this and email them w/ this info

Since the email has been sent, let’s close the incident!

@damianavila good point - I’ve sent them an email today!

Hey all - I see that in this comment:

With https://github.com/2i2c-org/infrastructure/pull/1026, I believe we should close this issue once https://github.com/2i2c-org/infrastructure/pull/1087 that updates the kube-scheduler is merged.

We suggested closing this one once those two PRs are merged. They are now both merged. Should we close this issue? If not, what is our next step here?

See if version of user scheduler needs to be upgraded, and upgrade it

This can be tricky sometimes due to how kube-scheduler has changed, and given that we want to support a few versions of k8s at the same time. Here is a relevant note from z2jh about the need to bump RBAC resources.

https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/6f4609e2f73258c7e0146f8e849d31aa7fe7f09f/jupyterhub/values.yaml#L432-L436

I’ll look into this right now a bit. See https://github.com/jupyterhub/zero-to-jupyterhub-k8s/pull/2590

One possibility is that it’s a race condition because we have multiple schedulers running on the cluster - the default cluster scheduler and the custom user scheduler. They could be racing each other, and the following sequence of events might be happening.

  1. User scheduler sees nodeA with 109 pods
  2. Cluster scheduler sees nodeA with 109 pods
  3. Cluster scheduler assigns podA to nodeA
  4. User scheduler doesn’t update current status of nodeA, but assigns podB to nodeA
  5. Kubelet on nodeA pulls podA, and when it tries to pull podB it fails with this error message - 110 is a kubernetes default per-pod node limit.

I encountered something like this (but not exactly this) at the Berkeley cluster. There, I worked around it by configuring the GKE cluster scheduler itself to behave closer to our user scheduler (see https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler#autoscaling_profiles). I don’t think such a thing is possible in AKS yet. If so, I think we should:

  1. See if version of user scheduler needs to be upgraded, and upgrade it
  2. See if more CPU and memory for user scheduler will help.

That used: 110 capacity:100 smells bad…

@damianavila, I believe it’s 110 with 110. But yes, I’ve checked and 110 is the max number of pods allowed per node.

Findings

I analyzed the dashboards a bit and this is what it shows:

toronto-dashboard

  • The peaks are aprox 20min after the timestamp of the error reported (btw, the log timestamps are UTC, right?)

  • Also, I never see that node in the logs having 110 pods on it 🤔 It was at 108 at the time the logs reported the node full.

  • There are also a few pods consuming a significant amount of RAM in that time interval. high-mem

  • A new node did come up 15min after this error, even if the logs didn’t show a scale-up event

new-node