kubernetes: Can't remove terminating persistent volumes (PV)

What happened:

The pod and namespaces got removed, but sometimes the pv still remains in “Terminating” status even with ReclaimPolicy Delete.

grafik

Name:            pvc-3252ac79-674e-11e9-9651-5404a6f17789
Labels:          <none>
Annotations:     pv.kubernetes.io/provisioned-by: rancher.io/longhorn
Finalizers:      [external-attacher/io-rancher-longhorn]
StorageClass:    longhorn1rep
Status:          Terminating (lasts <invalid>)
Claim:           sentry/sentry-data
Reclaim Policy:  Delete
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        10Gi
Node Affinity:   <none>
Message:
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            io.rancher.longhorn
    VolumeHandle:      pvc-3252ac79-674e-11e9-9651-5404a6f17789
    ReadOnly:          false
    VolumeAttributes:      numberOfReplicas=1
                           staleReplicaTimeout=60
                           storage.kubernetes.io/csiProvisionerIdentity=1556113772288-8081-rancher.io/longhorn
Events:                <none>

I tried to delete it with kubectl delete pv pvc-3252ac79-674e-11e9-9651-5404a6f17789 --force --grace-period=0 but it gets stuck and remains terminating.

What you expected to happen:

PV’s should get removed.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.7", GitCommit:"6f482974b76db3f1e0f5d24605a9d1d38fad9a2b", GitTreeState:"clean", BuildDate:"2019-03-25T02:52:13Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration: single node setup with rancher
  • OS (e.g: cat /etc/os-release):
NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
  • Kernel (e.g. uname -a): Linux node2 4.4.0-146-generic #172-Ubuntu SMP Wed Apr 3 09:00:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools: rancher
  • Network plugin and version (if this is a network-related bug):
  • Others: We use Longhorn as our Storageclass

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 18
  • Comments: 41 (8 by maintainers)

Commits related to this issue

Most upvoted comments

You can get rid of this issue by manually editing the pv and then removing the finalizers which looked something like this:

- kubernetes.io/pv-protection

e.g

kubectl patch pvc pvc_name -p '{"metadata":{"finalizers":null}}'
kubectl patch pv pv_name -p '{"metadata":{"finalizers":null}}'
kubectl patch pod pod_name -p '{"metadata":{"finalizers":null}}'
+324
ravibagri5 on Jun 14, 2019

Here’s a one-liner to remove finalizers from all pv in the system

kubectl get pv | tail -n+2 | awk '{print $1}' | xargs -I{} kubectl patch pv {} -p '{"metadata":{"finalizers": null}}'

+122
palashgoel7 on Jul 24, 2019

@Cherishty Because, in your case, the finalizer value is set to ‘protect’ which will block the deletion. You can read more about finalizers here : https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#finalizers And don’t forget to give ‘Thumbs up’ if it worked for you. 😉

+37
ravibagri5 on Jun 26, 2019

The problem is still in 1.18.0

+19
41tair on Mar 29, 2020

You could check if VolumeAttachment still exist using Kubectl get volumeattachment. Besides, you can check if the volume detached successfully in csi driver logs.

+5
cwdsuzhou on May 6, 2019

When I tried running the above one-liner provided by @palashgoel7 (although on a different type of object), I got the following error Error from server (UnsupportedMediaType): the body of the request was in an unknown format - accepted media types include: application/json-patch+json, application/merge-patch+json

The solution was to add --type='merge'. The final command was

kubectl get pg | tail -n+2 | awk '{print $1}' | xargs -I{} kubectl patch pg {} --type='merge' -p '{"metadata":{"finalizers": null}}'

notice the object type is pg (postgres.kubedb.com) not pv, probably not important

+2
sryabkov on Oct 10, 2019

@ravibagri5 your suggestion does work! But I still wonder why the finalizer is always blocking the deletion?

+2
Cherishty on Jun 26, 2019

/reopen

+1
XzenTorXz on Feb 24, 2022
Read more comments on GitHub
← kubernetes: Can't make a ref to pod "{pod_name}", container {container_name}: selfLink was empty, can't make reference
kubernetes: Can't upgrade to 1.16.0: [ERROR CoreDNSUnsupportedPlugins]: there are unsupported plugins in the CoreDNS  →