velero: EBS VolumeSnapshotter fails to create volume from snapshot if the snapshot is still pending

What steps did you take and what happened: Restoring a volume with an EBS snapshot sometimes fails if the snapshot is still pending. Most of the time I don’t hit it, but it happened at least once. The case where it’s likely to come up is where the backup is happening from one cluster and the restore is happening into another cluster immediately following, since there is no enforced downtime around namespace/volume deletion.

What did you expect to happen: I expected the volume creation to succeed.

The output of the following commands will help us better understand what’s going on: Error message from restore: error executing PVAction for persistentvolumes/pvc-89e0cd19-c53c-11e9-9f8d-0a0e2c8f9874: rpc error: code = Unknown desc = IncorrectState: Snapshot is in invalid state - pending

One possible solution is for the aws volume_snapshotter to check for the snapshot state before moving on. This actually solves two potential errors:

  1. if the volume state is ‘error’, then we can just return an error without attempting to create
  2. if the volume state is ‘pending’, then wait and try again

Here’s an idea of what I’m suggesting. As a first approximation, just make a Sleep call and try again. The real answer would need to include a timeout and possibly a better way of waiting, but this gets the concept down.

Basically replace the lines here: https://github.com/heptio/velero/blob/master/pkg/cloudprovider/aws/volume_snapshotter.go#L104-L106 With something like this (timeout, error checking, etc. omitted to just show the concept):

	if count := len(snapRes.Snapshots); count != 1 {
		return "", errors.Errorf("expected 1 snapshot from DescribeSnapshots for %s, got %v", snapshotID, count)
	}
	for (snapRes.Snapshots[0].State == "pending" {
		time.Sleep(time.Second * 5)
		snapRes, _ = b.ec2.DescribeSnapshots(snapReq)
	}
	if snapRes.Snapshots[0].State == "error" {
		return "", errors.Errorf("Snapshot is in 'error' state")
	}

About this issue

  • Original URL
  • State: open
  • Created 7 years ago
  • Comments: 15 (7 by maintainers)

Commits related to this issue

Most upvoted comments

We just discovered that GCP has the same problem (for the same reasons). This isn’t unexpected, and I think everything discussed above still applies. When the plugin API is updated to allow post-backup snapshot status checks, then both GCP and AWS (and probably Azure too) will need to check snapshot status, taking into account the snapshotCreationTimeout in the VSL.

I think a new phase that indicates we’re waiting for the snapshot data to replicate makes sense. And we’ll still want that “wait for snapshot completion” step to be non-blocking, so subsequent backups can proceed.

Couple related things we’ll want to consider:

  • we’d still want any post hooks to run immediately after the snapshot was “created”, rather than waiting until they’re completed/replicated
  • for some storage platforms (typically on-prem), it may be possible to restore from a local snapshot after it’s created, before the data is replicated off to some durable backend. so, we may want some provider-specific way to indicate whether the snapshot/backup is restorable yet, independent of whether it’s completed replicating its data somewhere