kubernetes: There is no way to force delete Namespaces with invalid finalizers

sorry, PUT.

You essentially have to mimic what a controller that was sending PUT requests to the finalize subresource would do to drop those spec.finalizers.

cat ns.yaml 

apiVersion: v1
kind: Namespace
metadata:
  name: delete-me
spec:
  finalizers:
    - foregroundDeletion

Create with a spec.finalizer:

kubectl create -f ns.yaml 

namespace/delete-me created

Request deletion:

kubectl delete -f ns.yaml --wait=false

namespace "delete-me" deleted

kubectl get ns delete-me -o json | jq '.spec.finalizers=[]' > ns-without-finalizers.json
cat ns-without-finalizers.json

{
  "apiVersion": "v1",
  "kind": "Namespace",
  "metadata": {
    "creationTimestamp": "2019-04-25T21:11:48Z",
    "deletionTimestamp": "2019-04-25T21:11:53Z",
    "name": "delete-me",
    "resourceVersion": "355",
    "selfLink": "/api/v1/namespaces/delete-me",
    "uid": "a7e756b3-53f3-4a3d-a26b-fe794ee7954d"
  },
  "spec": {
    "finalizers": []
  },
  "status": {
    "phase": "Terminating"
  }
}

PUT the namespace without finalizers to the /finalize subresource (kubectl proxy just lets you easily use curl against the subresource):

kubectl proxy &
PID=$!
curl -X PUT http://localhost:8001/api/v1/namespaces/delete-me/finalize -H "Content-Type: application/json" --data-binary @ns-without-finalizers.json
kill $PID

And now the delete can succeed.

kubectl delete -f ns.yaml

namespace "delete-me" deleted

thanks @liggitt , that worked well

One liner (two lines if you are picky) for @liggitt 's solution which avoids creating intermediate yaml file:

Starting kubectl proxy in one terminal:

kubectl proxy

In another terminal:

kubectl get ns delete-me -o json | \
  jq '.spec.finalizers=[]' | \
  curl -X PUT http://localhost:8001/api/v1/namespaces/delete-me/finalize -H "Content-Type: application/json" --data @-

so please use the other tips to delete the finalizer 😃

What other tips? I have tried:

• Editing and reapplying the YAML
• kubectl edit ns delete-me
• kubectl proxy & and curl to run finalize
• kubectl delete ns delete-me --force --grace-period=0
• Waiting a week
• etcdctl --endpoint=http://127.0.0.1:8001 rm /namespaces/delete-me
• ./kill-kube-ns delete-me

I recently found this method works every time allowing you to identify the reason for the supposedly stuck state.

kubectl api-resources --verbs=list --namespaced -o name \
  | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace>

I believe this results in properly deleting whatever resource is stuck on terminating

If you kubectl edit and manually remove the finalizer, either it will be deleted or you’ll be shown a probably useful error message.

If that makes it get deleted then either the GC has a bug or it’s stuck trying to delete some other resource.

Thanks @lavalamp , but that doesn’t work.

Things I’ve tried:

None of these work or modify the Namespace. After any of these the problematic finalizer still exists.

Edit the YAML

Apply:

apiVersion: v1
kind: Namespace
metadata:
  name: delete-me
spec:
  finalizers:

The command finishes with no error, but the Namespace is not udpated.

The below YAML has the same result:

apiVersion: v1
kind: Namespace
metadata:
  name: delete-me
spec:

kubectl edit

kubectl edit ns delete-me, and remove the finalizer. Ditto removing the list entirely. Ditto removing spec.

This shows no error message but does not update the Namespace.

kubectl proxy &

• kubectl proxy &
• curl -k -H "Content-Type: application/yaml" -X PUT --data-binary @tmp.yaml http://127.0.0.1:8001/api/v1/namespaces/delete-me/finalize

As above, this exits successfully but does nothing.

Force Delete

kubectl delete ns delete-me --force --grace-period=0

This actually results in an error:

warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
Error from server (Conflict): Operation cannot be fulfilled on namespaces "delete-me": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.

However, it doesn’t actually do anything.

Wait a long time

In the test cluster I set up to debug this issue, I’ve been waiting over a week. Even if the Namespace might eventually decide to be deleted, I need it to be deleted faster than a week.

Make sure the Namespace is empty

The Namespace is empty.

$ kubectl get -n delete-me all

No resources found.

etcdctl

$ etcdctl --endpoint=http://127.0.0.1:8001 rm /namespaces/delete-me

Error:  0:  () [0]

I’m pretty sure that’s an error, but I have no idea how to interpret that. It also doesn’t work. Also tried with --dir and -r.

ctron/kill-kube-ns

There is a script for force deleting Namespaces. This also does not work.

$ ./kill-kube-ns delete-me

Killed namespace: delete-me

$ kubectl get ns delete-me 

NAME        STATUS        AGE
delete-me   Terminating   1h

@souravb11 Thanks this is a simple and easy solution to fix the issue edited namespace and removed below “finalizers”: [ “controller.cattle.io/namespace-auth” ]

I faced the same issue and i have edit my ns yaml file and remove “finalizers” parameter from file and saved. Thats work for me its deleted immediate.

I recently found this method works every time allowing you to identify the reason for the supposedly stuck state.

kubectl api-resources --verbs=list --namespaced -o name \
  | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace>

I believe this results in properly deleting whatever resource is stuck on terminating

THIS SOLUTION is the best! GREAT, thanks. I just use the command to find all resources installed, then edit all resources related to Rancher to have none ‘finalizers’. After that you could edit rancher´s namespaces to have none ‘finalizers’, then delete all.

kubectl api-resources --verbs=list --namespaced -o name | while read line; do echo $line; kubectl get -A $line --ignore-not-found -o name ;done;