aws-efs-csi-driver: Files not getting deleted on the EFS when PVCs & Access Points are Deleted

/kind bug

What happened? I’m using Dynamic Provisioning and I was wondering if it is expected behavior or a bug for my files to still be in the EFS file system even after we have cleaned up our pods and PVC/PVs. This is resulting in our meter usage keep going up even though we are redeploying new pods and the old files from already deleted pods and PVs are still in the file system.

I expected the Reclaim Policy: Delete option of the storage class to be able to remove those files as well, not just the Access Points.

AWS Support suggests that we attach a cleanup sidecar pod to remove the files before it gets deleted, but it would be really nice if that could be handled by the storage class itself to avoid us having to create a separate pod and handle the cleaning part separately.

What you expected to happen? After we delete the PVC/PV that was being used by the pods, it should have deleted the files in the attached Filesystem

How to reproduce it (as minimally and precisely as possible)?

Use dynamic provisioning option of the storage class with the reclaim policy set to Delete:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
reclaimPolicy: Delete
parameters:
  provisioningMode: efs-ap
  fileSystemId: $EFS_FILE_SYSTEM_ID
  directoryPerms: "700"

Create a Pod with a PVC that uses this storage class. Make sure the pod creates a file in the volume
Delete the Pod and the PVC that was created
Mount the EFS Filesystem to an instance that can mount it
You’ll find the file still exists in the file system under the folder of its previously existing Access Point

Anything else we need to know?:

Environment

Kubernetes version (use kubectl version): 1.18*
Driver version: 1.2.0

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 4
Comments: 22 (8 by maintainers)

Most upvoted comments

Looking through the code, I found the solution using an argument that’s passed into the EFS driver container. I wish this was documented a bit more since it can be misleading for users when it comes to the Delete option of reclaimPolicy.

To enable this functionality here’s what I did:

go to the deployment spec of the EFS CSI Driver and add this into the arguments section: --delete-access-point-root-dir=true

    spec:
      containers:
      - args:
        - --delete-access-point-root-dir=true
        - --endpoint=\$(CSI_ENDPOINT)
        - --logtostderr
        - --v=5

MarkSpencerTan on Apr 13, 2021

@kbasv yeah we might need to tag the access point or something in createvolume and then have deletevolume descibe it… If we add parameter, we still have to reconcile what happens when both flag and parameter are set, I’ll leave that design to others.

For now let’s add a big note to README clarifying basically what has been said in the previous comments "NOTE: if your EFS PV’s ReclaimPolicy is Delete and you delete its PVC then the EFS access point will deleted but its data remains in the EFS volume. this behavior can be toggled with the flag x " , we can all review it. Users should never have to look at code to figure this out. Any volunteers can add this, otherwise i’ll do it later this week.

wongma7 on Apr 14, 2021

Throwing out an alternative, does it make sense to allow users to configure this as part of the StorageClass? That way a user can decide if they want their contents to be wiped for some volumes but not for others?

chrishenzie on Apr 14, 2021

Just making sure I understand correctly, reclaimPolicy: Delete only works if the above flag is supplied to the EFS CSI driver? If that’s the case, can we just flip it to be default true (or remove this as a toggle)?

reclaimPolicy: Delete works as expected. Provisioned volume and the underlying access points are deleted. However, EFS does not delete Access Point contents when deleting an Access Point. We provided --delete-access-point-root-dir to the driver and if enabled, driver will delete access point contents while deleting PV .

Hi @kbasv , this was my confusion since I’ve worked with a couple of other persistent volume storage classes in the past, and this is not the default behavior in their logic when reclaim Policy is ‘Delete’. I would be interested to find out what different use case you guys have in the cloud why a user would still need this data if they explicitly defined ‘Delete’ as their reclaim policy. I figured the user should set the reclaim policy to ‘Reclaim’ if they still need that data afterward as documented here in the official k8s docs: https://kubernetes.io/docs/tasks/administer-cluster/change-pv-reclaim-policy/#why-change-reclaim-policy-of-a-persistentvolume.

I think the extra documentation you mentioned and the helm config to enable that as brought up by @KarstenSiemer would be great additions for the user to gain visibility over this flag! Thanks guys 😃

MarkSpencerTan on Apr 14, 2021

Just making sure I understand correctly, reclaimPolicy: Delete only works if the above flag is supplied to the EFS CSI driver? If that’s the case, can we just flip it to be default true (or remove this as a toggle)?

reclaimPolicy: Delete works as expected. Provisioned volume and the underlying access points are deleted. However, EFS does not delete Access Point contents when deleting an Access Point. We provided --delete-access-point-root-dir to the driver and if enabled, driver will delete access point contents while deleting PV .

kbasv on Apr 14, 2021

I see; I don’t think this is supported by the driver, but we could open an issue for this as a feature addition.

chrishenzie on Aug 31, 2021