velero: Not able to restore EFS volumes using restic

Backup is completed successfully as can be seen here -

  Completed:
    xyz/app-xyz-64b9cdb6cb-tbbn8: app-common-volume, app-log-volume```

But while restore, the PVCs are stuck in pending state - 

`Name:          app-common-volume-pvc
Namespace:     xyz
StorageClass:  efs-sc
Status:        Pending
Volume:
Labels:        prophecy.io/cluster=prophecydev
               prophecy.io/component=app
               prophecy.io/controlplane=xyz
               prophecy.io/logconfmapversion=1659308
               velero.io/backup-name=xyz-backup2
               velero.io/restore-name=xyz-backup2-20200921165054
Annotations:   volume.beta.kubernetes.io/storage-provisioner: efs.csi.aws.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Mounted By:    app-xyz-64b9cdb6cb-tbbn8
Events:
  Type    Reason                Age                  From                         Message
  ----    ------                ----                 ----                         -------
  Normal  ExternalProvisioning  64s (x202 over 51m)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "efs.csi.aws.com" or manually created by system administrator`

Not sure why it doesnt get created on its own when PV  resource itself is part of backup

About this issue

Original URL
State: open
Created 4 years ago
Reactions: 1
Comments: 15 (3 by maintainers)

Most upvoted comments

I’m also facing the same problem. @vaishali-prophecy as a workaround, you can create the PV by hand, just be sure to correctly setup the PV name, PVC namespace and name, as well as volumeHandle parts (for example using aws efs):

apiVersion: v1
kind: PersistentVolume
metadata:
  name: <pv-name>
spec:
  capacity:
    storage: 1Gi
  csi:
    driver: efs.csi.aws.com
    volumeHandle: 'fs-XXXXXXXX::fsap-XXXXXXXXXXXXXXXXX'
  accessModes:
    - ReadWriteMany
  claimRef:
    kind: PersistentVolumeClaim
    namespace: <claim-namespace>
    name: <claim-name>
  persistentVolumeReclaimPolicy: Retain
  storageClassName: efs-storage-class
  volumeMode: Filesystem

Then you can run the restore process and everything should work fine.

headyj on Mar 26, 2021

Based on the message waiting for a volume to be created, either by external provisioner "efs.csi.aws.com", the EFS CSI driver hasn’t yet created the volume.

Doing some Googling, as I’m not familiar with the driver, I came across their documentation which states

Only static volume provisioning is supported. This means that an Amazon EFS file system needs to be created outside of Amazon EKS before being used by pods in your cluster.

Docs in https://github.com/kubernetes-sigs/aws-efs-csi-driver state this, too. So it appears AWS has changed how EFS is provisioned and we need to revisit how it’s approached in Velero; this is going to require some research on our part.

nrb on Sep 30, 2020

Hi @nrb ,

Yaml for the PV resource -

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/bound-by-controller: "yes"
  finalizers:
  - kubernetes.io/pv-protection
  labels:
    prophecy.io/cluster: prophecydev
    prophecy.io/component: efs
    pvselector: app-common-volume-aaa
    velero.io/backup-name: aaa-backup
  name: app-app-common-volume-aaa
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 5Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: app-common-volume-pvc
    namespace: aaa
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-e2900360:/aaa/app/app/common
  persistentVolumeReclaimPolicy: Delete
  storageClassName: efs-sc
  volumeMode: Filesystem

Yaml for storage class -

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    meta.helm.sh/release-name: vish
    meta.helm.sh/release-namespace: prophecy
  creationTimestamp: "2020-09-17T15:35:23Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  name: efs-sc
  resourceVersion: "388862"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/efs-sc
  uid: df3f7a5b-55e0-4e73-ba35-236c6b9d06d2
provisioner: efs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

Version of velero - 1.4 K8s version - 1.16

Note that PV resource itself isn’t restored on restore action for the PVCs which are marked in pod annotation. Also just shared PV yaml corresponding to some other deployment as the previous one doesnt exist as of now.

vaishali-prophecy on Sep 21, 2020