kubernetes: Custom resources with finalizers can "deadlock" customresourcecleanup.apiextensions.k8s.io finalizer

edit on 2022-11-08

The following is not recommended and should not be required in any recent Kubernetes version (>1.10.0)

Removing the finalizer from your CRD orphans the child custom resources, and breaks the expected cleanup path for them. The following comment was only a workaround for the deadlock with that cleanup process, not a process you should follow to remove CRDs normally.

original comment:

to get around this, I had to remove the customresourcecleanup.apiextensions.k8s.io finalizer from the CRD, then recreate it and go about removing the finalizers from the custom resources

# remove the CRD finalizer blocking on custom resource cleanup
kubectl patch crd/crontabs.stable.example.com -p '{"metadata":{"finalizers":[]}}' --type=merge

# now the CRD can be deleted (orphaning custom resources in etcd)
kubectl delete -f crd.yaml 
customresourcedefinition.apiextensions.k8s.io "crontabs.stable.example.com" deleted
Error from server (NotFound): error when deleting "crd.yaml": the server could not find the requested resource (delete crontabs.stable.example.com my-new-cron-object)

# when the CRD is recreated, it resurfaces the existing custom resources
kubectl create -f crd.yaml
customresourcedefinition.apiextensions.k8s.io "crontabs.stable.example.com" created
Error from server (AlreadyExists): error when creating "crd.yaml": object is being deleted: crontabs.stable.example.com "my-new-cron-object" already exists

# the custom resources can now be edited to remove finalizers
kubectl patch crontab/my-new-cron-object -p '{"metadata":{"finalizers":[]}}' --type=merge
crontab.stable.example.com "my-new-cron-object" patched

# and now both custom resource and CRD can be deleted
kubectl delete -f crd.yaml 
customresourcedefinition.apiextensions.k8s.io "crontabs.stable.example.com" deleted
Error from server (NotFound): error when deleting "crd.yaml": crontabs.stable.example.com "my-new-cron-object" not found

Still hitting deadlock in cert-manager v1.7.1 in Kubernetes v1.23.3

1. Locate remaining instances of the custom resource (not the CRD itself). They should all have have metadata.deletionTimestamp set, indicating the API server is trying to remove them as part of cleaning up the CRD.

kubectl get $mycustomtype -A -o yaml

1. Find any that have metadata.finalizers set. That means something added a finalizer to that instance, indicating it wanted to block deletion of that instance, but has not yet removed its own finalizer.

apiVersion: mygroup.example.com/v1
kind: MyCustomType
metadata:
  name: foo
  deletionTimestamp: 2022-01-12T15:46:38Z
  finalizers:
  - myfinalizer.example.com
...

1. If it is safe to override the finalizer on those instances, remove the finalizers from the custom resource instances (not the CRD):

kubectl patch $mycustomtype/foo -p '{"metadata":{"finalizers":[]}}' --type=merge

That will unblock deletion of those instances, which will unblock deletion of the CRD

to get around this, I had to remove the customresourcecleanup.apiextensions.k8s.io finalizer from the CRD, then recreate it and go about removing the finalizers from the custom resources

# remove the CRD finalizer blocking on custom resource cleanup
kubectl patch crd/crontabs.stable.example.com -p '{"metadata":{"finalizers":[]}}' --type=merge

# now the CRD can be deleted (orphaning custom resources in etcd)
kubectl delete -f crd.yaml 
customresourcedefinition.apiextensions.k8s.io "crontabs.stable.example.com" deleted
Error from server (NotFound): error when deleting "crd.yaml": the server could not find the requested resource (delete crontabs.stable.example.com my-new-cron-object)

# when the CRD is recreated, it resurfaces the existing custom resources
kubectl create -f crd.yaml
customresourcedefinition.apiextensions.k8s.io "crontabs.stable.example.com" created
Error from server (AlreadyExists): error when creating "crd.yaml": object is being deleted: crontabs.stable.example.com "my-new-cron-object" already exists

# the custom resources can now be edited to remove finalizers
kubectl patch crontab/my-new-cron-object -p '{"metadata":{"finalizers":[]}}' --type=merge
crontab.stable.example.com "my-new-cron-object" patched

# and now both custom resource and CRD can be deleted
kubectl delete -f crd.yaml 
customresourcedefinition.apiextensions.k8s.io "crontabs.stable.example.com" deleted
Error from server (NotFound): error when deleting "crd.yaml": crontabs.stable.example.com "my-new-cron-object" not found

I was having issue with AgroCD’s “applications.argoproj.io” crd. I accidently deleted the namespace and now this CRD is not getting deleted. This work around worked like a charm. Thank you !

kubectl patch crd/applications.argoproj.io -p '{"metadata":{"finalizers":[]}}' --type=merge

This is marked as closed, but I got the same problem and could fix the problem by the solution /workaround provided here in v1.17. Is ti still not closed?

` Client Version: version.Info{Major:“1”, Minor:“17”, GitVersion:“v1.17.0”, GitCommit:“70132b0f130acc0bed193d9ba59dd186f0e634cf”, GitTreeState:“clean”, BuildDate:“2019-12-07T21:20:10Z”, GoVersion:“go1.13.4”, Compiler:“gc”, Platform:“linux/amd64”}

Server Version: version.Info{Major:“1”, Minor:“17”, GitVersion:“v1.17.0”, GitCommit:“70132b0f130acc0bed193d9ba59dd186f0e634cf”, GitTreeState:“clean”, BuildDate:“2019-12-07T21:12:17Z”, GoVersion:“go1.13.4”, Compiler:“gc”, Platform:“linux/amd64”} `

An integration test for this would be a nice addition and a good task for new contributors.

@xmudrii can you work on this as your first issue? You can base your test on top of @liggitt’s PR above.

Here are some things that can help you get started:

• If you are new to the Kubernetes API in general, I’d recommend reading the API conventions first.
• CustomResourceDefinition (CRD) is a way to extend the Kubernetes API. You can read more about it here: https://kubernetes.io/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/.
• If you’d like to do some additional reading about this, I had written something up during the time I did GSoC (shamless plug): https://nikhita.github.io/gsoc-kubernetes-making-customresources-more-awesome.
• There are lots of integration tests for CRDs here and you can follow a similar pattern for writing the integration test for this.
• This doc contains useful information about how you can run tests.

Feel free to ping me on Slack if you need any help with this! 😃

!!! IMPORTANT !!!

This issue is still happening in 1.20.15_1568

former FIX from @liggitt still works:

remove the CRD finalizer blocking on custom resource cleanup

kubectl patch crd/name-of-the-CRD -p ‘{“metadata”:{“finalizers”:[]}}’ --type=merge

having same issue while deleting crd of splunk of operator. Patching crd with empty finalizers throws error linke invalid json. Any pointers to troubleshoot.

it would probably be useful to improve the status reported on the CRD about which specific custom resource instances remain that are blocking the deletion (https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiextensions-apiserver/pkg/controller/finalizer/crd_finalizer.go#L232-L253)

HI. I’m not sure why did you close that. Could you point me where is the solution, please?

In my dev I had a problem with { “apiVersion”: “http.keda.sh/v1alpha1”, “kind”: “HTTPScaledObject”, “metadata”: { “annotations”: { “kubectl.kubernetes.io/last-applied-configuration”: “{"apiVersion":"http.keda.sh/v1alpha1","kind":"HTTPScaledObject","metadata":{"annotations":{},"name":"k8sdemoingress","namespace":"k8sdemoingress"},"spec":{"host":"myhost.com","replicas":{"max":10,"min":5},"scaleTargetRef":{"deployment":"k8sdemoingress","port":80,"service":"k8sdemoingress"}}}\n” }, “creationTimestamp”: “2022-01-12T15:24:47Z”, “deletionGracePeriodSeconds”: 0, “deletionTimestamp”: “2022-01-12T15:46:38Z”, “finalizers”: [ “httpscaledobject.http.keda.sh” ],

As I don’t have too much time to dig, I just removed the cluster 😃 LOL - and it worked 😄 hahahahaha

Seriously - it there better solution to remove deadlock customresource (or in general any resource) ?

Thank You.

Same problem here using cert-manager. 👎

Versions

EKS 1.18 cert-manager 1.1.0

Having the same issue with fluxcd/fluxv2 on eks-1.18

@wojtek-viirtue @fernando-villalba Same problem here after accidentally deleting the argocd namespace (deleting namespace gets stuck). I’m trying this workaround. Phew, it worked.

I am also getting this issue when deleting the applications.argoproj.io CRD from argocd I thought that Kubernetes was injecting the finalizer customresourcecleanup.apiextensions.k8s.io on creation so I declared an empty finalizer in the yaml with finalizers: [] If you apply this when it is stuck, it automatically deletes. However what baffles me is that this finalizer is only injected by kubernetes into the definition on delete, so even if I declare empty finalizers beforehand this does not solve the issue.

Using EKS 1.18

Does anyone have any links to documentation on how this works? What’s this customeresourcleanup and why is it injected on deletion?

I’m running into this issue with CRD for cert-manager v1.18.6

Edit: Referenced thread below to patch the cert manager to force deletion.

@nikhita This sounds good to me. Thank you so much!

I’ll take this one and I’ll try to put up a PR soon.