velero: Fail early DM pod checks have false positives

What steps did you take and what happened: In Velero 1.13, we added code to fail early if the datamover pods are in an unrecoverable state: https://github.com/vmware-tanzu/velero/pull/7052 In 1.14, some additional changes were made here: https://github.com/vmware-tanzu/velero/pull/7437

In our OADP testing with Velero 1.14, we found that in some cases DDs or DUs were being canceled almost immediately. Digging into the code added in the above 2 PRs, there are a few situations where Velero considers a pod unrecoverable:

  1. pod phase of Failed or Unknown
  2. pod phase of Pending with an “unschedulable” condition
  3. container status with ImagePullBackOff or ErrImgNeverPull

It was the second situation that was causing the failure for us. The pod will have an “unschedulable” condition until the PVC is is bound to a PV. In many cases, this all happens quickly enough that by the time Velero checks, the PVC is bound and the Pod is no longer unschedulable, but if the provisioner takes a few seconds to bind the PV and PVC together, the Pod may still have an “unschedulable” condition when the node agent first checks, which will cause the DU or DD to be immediately canceled.

We are seeing this happen frequently with Ceph volumes.

We have a couple of options here.

  1. The simplest option would be to remove “unschedulable” from the list of unrecoverable conditions. As we see here, “unschedulable” does not always indicate an unrecoverable state. The only downside here is that we lose fast fail for permanently-unschedulable pods.

  2. Slightly more complicated – we could wrap the unschedulable check in a PollUntilContextTimeout call with some hard-coded timeout (2 minutes maybe?) – this will result in almost-fast-failing for unschedulable pods. This assumes that no provisioner should take more than 2 minutes to provision the PV. That’s probably safe.

I will be posting a draft PR for option 1) – I’m open to moving to 2) if the consensus is that it’s a better solution.

Environment:

  • Velero version (use velero version): 1.14
  • Velero features (use velero client config get features):
  • Kubernetes version (use kubectl version):
  • Kubernetes installer & version:
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project’s top voted issues listed here.
Use the “reaction smiley face” up to the right of this comment to vote.

  • 👍 for “I would like to see this bug fixed as soon as possible”
  • 👎 for “There are more important bugs to focus on right now”

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 6
  • Comments: 16 (8 by maintainers)

Commits related to this issue

Most upvoted comments

I personally think ignoring unschedulable makes the most sense… an admin (or autoscaler) can scale up more nodes etc for scheduling to resolve. timing out at 1 minute and marking restore fail can be misleading. velero did everything right, infra was just not yet avail to schedule but could be totally fine 30 minutes later.