kubernetes: Kubernetes namespaces stuck in terminating state

@paralin - there is no code issue, but maybe I can look to improve in the openshift example clean-up scripts or document the steps. When you created a project in openshift, it created a namespace for that project, and annotated the namespace with a finalizer token that said before a namespace is deleted, an external agent needs to remove its lock on the object that says it was done clean-up. Since you are no longer running openshift, its agent did not remove the lock and take part in the termination flow.

A quick fix:

# find each namespace impacted
$ kubectl get namespaces -o json | grep "openshift.io/origin"
$ kubect get namespace <ns> -o json > temp.json
# vi temp.json and remove the finalizer entry for "openshift.io/origin"
# for example
{
    "kind": "Namespace",
    "apiVersion": "v1",
    "metadata": {
        "name": "testing",
        "selfLink": "/api/v1/namespaces/testing",
        "uid": "33074e57-cb72-11e5-9d3d-28d2444e470d",
        "resourceVersion": "234",
        "creationTimestamp": "2016-02-04T19:05:04Z",
        "deletionTimestamp": "2016-02-04T19:05:54Z"
    },
    "spec": {
        "finalizers": [
            "openshift.io/org"  <--- remove me
        ]
    },
    "status": {
        "phase": "Terminating"
    }
}

$ curl -H "Content-Type: application/json" -X PUT --data-binary @temp.json http://127.0.0.1:8080/api/v1/namespaces/<name_of_namespace>/finalize
# wait a moment, and you should see your namespace removed
$ kubectl get namespaces 

That will remove the lock that blocks the namespace from being completely terminated, and you should quickly see that the namespace is removed from your system.

Closing the issue, but feel free to comment if you continue to have problems or hit me up on slack.

same issue on 1.15 , nothing worked in my case.

kubectl delete ns test

Error from server (Conflict): Operation cannot be fulfilled on namespaces “test”: The system is ensuring all content is removed from this namespace. Upon completion, this namespace will automatically be purged by the system.

have you tried to delete it using the following script ?

#!/bin/bash

###############################################################################
# Copyright (c) 2018 Red Hat Inc
#
# See the NOTICE file(s) distributed with this work for additional
# information regarding copyright ownership.
#
# This program and the accompanying materials are made available under the
# terms of the Eclipse Public License 2.0 which is available at
# http://www.eclipse.org/legal/epl-2.0
#
# SPDX-License-Identifier: EPL-2.0
###############################################################################

set -eo pipefail

die() { echo "$*" 1>&2 ; exit 1; }

need() {
	which "$1" &>/dev/null || die "Binary '$1' is missing but required"
}

# checking pre-reqs

need "jq"
need "curl"
need "kubectl"

PROJECT="$1"
shift

test -n "$PROJECT" || die "Missing arguments: kill-ns <namespace>"

kubectl proxy &>/dev/null &
PROXY_PID=$!
killproxy () {
	kill $PROXY_PID
}
trap killproxy EXIT

sleep 1 # give the proxy a second

kubectl get namespace "$PROJECT" -o json | jq 'del(.spec.finalizers[] | select("kubernetes"))' | curl -s -k -H "Content-Type: application/json" -X PUT -o /dev/null --data-binary @- http://localhost:8001/api/v1/namespaces/$PROJECT/finalize && echo "Killed namespace: $PROJECT"

# proxy will get killed by the trap ```
`#!/bin/bash

###############################################################################
# Copyright (c) 2018 Red Hat Inc
#
# See the NOTICE file(s) distributed with this work for additional
# information regarding copyright ownership.
#
# This program and the accompanying materials are made available under the
# terms of the Eclipse Public License 2.0 which is available at
# http://www.eclipse.org/legal/epl-2.0
#
# SPDX-License-Identifier: EPL-2.0
###############################################################################

set -eo pipefail

die() { echo "$*" 1>&2 ; exit 1; }

need() {
	which "$1" &>/dev/null || die "Binary '$1' is missing but required"
}

# checking pre-reqs

need "jq"
need "curl"
need "kubectl"

PROJECT="$1"
shift

test -n "$PROJECT" || die "Missing arguments: kill-ns <namespace>"

kubectl proxy &>/dev/null &
PROXY_PID=$!
killproxy () {
	kill $PROXY_PID
}
trap killproxy EXIT

sleep 1 # give the proxy a second

kubectl get namespace "$PROJECT" -o json | jq 'del(.spec.finalizers[] | select("kubernetes"))' | curl -s -k -H "Content-Type: application/json" -X PUT -o /dev/null --data-binary @- http://localhost:8001/api/v1/namespaces/$PROJECT/finalize && echo "Killed namespace: $PROJECT"

# proxy will get killed by the trap`

usage :  ./kill-kube-ns <name of the namespace>

I have already the same problem. I restarted the virtual machine, and also run “kubectl delete ns ***** --force --grace-period=0”. The namespace concerned is still stucking in Terminating.

@attila123 I was having the same issue. In the case of rook it creates a finalizer which was causing it to get stuck.

Directly from the cleanup docs:

The operator is responsible for removing the finalizer after the mounts have been cleaned up. If for some reason the operator is not able to remove the finalizer (ie. the operator is not running anymore), you can delete the finalizer manually.

kubectl -n rook edit cluster rook

Look for the finalizers element and delete the following line:

  - cluster.rook.io

Now save the changes and exit the editor. Within a few seconds you should see that the cluster CRD has been deleted and will no longer block other cleanup such as deleting the rook namespace.

Resolved using the script from this comment.

Getting this issue in 1.14.3.

I deleted a namespace, and it shows as permanently “Terminating”. Removing the finalizer made no difference, and there are no pods running in the ns.

Hi, I got the same problem with Kubernetes 1.9.0 running by Minikube v0.24.1. I started too many pods my laptop kept swapping and not responding. So I powered that off. Upon restart lots of pods stucked, but I could delete them with kubectl -n some_namespace delete pod --all --grace-period=0 --force (first I deleted all the deployments, services, etc.) I also installed rook (0.7.0) and its namespace get stuck, even after minikube restart (stop + start).

[vagrant@localhost ~]$ kubectl get ns rook -o json
{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "annotations": {
            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"name\":\"rook\",\"namespace\":\"\"}}\n"
        },
        "creationTimestamp": "2018-03-09T16:13:08Z",
        "deletionTimestamp": "2018-03-10T07:54:03Z",
        "name": "rook",
        "resourceVersion": "29736",
        "selfLink": "/api/v1/namespaces/rook",
        "uid": "c1a4f8e6-23b4-11e8-8129-525400ad3b43"
    },
    "spec": {
        "finalizers": [
            "kubernetes"
        ]
    },
    "status": {
        "phase": "Terminating"
    }
}

[vagrant@localhost ~]$ kubectl delete ns rook --grace-period=0 --force
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
Error from server (Conflict): Operation cannot be fulfilled on namespaces "rook": The system is ensuring all content is removed from this namespace.  Upon completion, this namespace will automatically be purged by the system.

I needed to move on with my work, so I stopped minikube, stopped the host VM, created a full clone of it in Virtualbox, so if there is anything I can check, any log, etc., I can provide it. Then I minikube delete-ed my cluster. If I remember correctly, no resource was present in the rook namespace.

I’m facing the same issue

# oc version
oc v1.3.0-alpha.2
kubernetes v1.3.0-alpha.1-331-g0522e63

I have deleted the project named “gitlab” via Openshift Origin web console. But it is not removed.

As said by @derekwaynecarr I did the following

# kubectl get namespace gitlab -o json > temp.json

# cat temp.json

{
    "kind": "Namespace",
    "apiVersion": "v1",
    "metadata": {
        "name": "gitlab",
        "selfLink": "/api/v1/namespaces/gitlab",
        "uid": "cd86c372-481e-11e6-aebc-408d5c676116",
        "resourceVersion": "3115",
        "creationTimestamp": "2016-07-12T10:53:01Z",
        "deletionTimestamp": "2016-07-12T11:11:36Z",
        "annotations": {
            "openshift.io/description": "",
            "openshift.io/display-name": "GitLab",
            "openshift.io/requester": "developer",
            "openshift.io/sa.scc.mcs": "s0:c8,c7",
            "openshift.io/sa.scc.supplemental-groups": "1000070000/10000",
            "openshift.io/sa.scc.uid-range": "1000070000/10000"
        }
    },
    "spec": {
        "finalizers": [
            "kubernetes"   <---removed
        ]
    },
    "status": {
        "phase": "Terminating"
    }
}

and

# curl -k -H "Content-Type: application/json" -X PUT --data-binary @temp.json https://10.28.27.65:8443/api/v1/namespaces/gitlab/finalize
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "User \"system:anonymous\" cannot update namespaces/finalize in project \"gitlab\"",
  "reason": "Forbidden",  <--- seems like nothing happened
  "details": {
    "name": "gitlab",
    "kind": "namespaces/finalize"
  },
  "code": 403
}

but it is removed.

kubectl get namespace annoying-namespace-to-delete -o json > tmp.json then edit tmp.json and remove"kubernetes"

curl -k -H “Content-Type: application/json” -X PUT --data-binary @tmp.json https://kubernetes-cluster-ip/api/v1/namespaces/annoying-namespace-to-delete/finalize

and it should delete your namespace,

Hi, I got this:

curl -H “Content-Type: application/json” -X PUT --data-binary @temp.json http://127.0.0.1:8001/api/v1/namespaces/cattle-system/finalize/ { “kind”: “Status”, “apiVersion”: “v1”, “metadata”: {

}, “status”: “Failure”, “message”: “Operation cannot be fulfilled on namespaces "cattle-system": the object has been modified; please apply your changes to the latest version and try again”, “reason”: “Conflict”, “details”: { “name”: “cattle-system”, “kind”: “namespaces” }, “code”: 409

UPDATE: for those who got it, remove in json all line:

“resourceVersion”: “somenumber”,

And rerun curl

in my case, it was cert-managaer ns, which stuck in termination state. It could not be delete because of CRDs which was created before but was not seen by cmd kubectl get all but link to the script which has been provided by @difabion do the job. thx @difabion

Getting this issue in 1.14.3. I deleted a namespace, and it shows as permanently “Terminating”. Removing the finalizer made no difference, and there are no pods running in the ns.

Have you solved it?

I have not, no. Incredible that such an issue has been unfixed for over three years.

curl -k -H “Content-Type: application/json” -X PUT --data-binary @tmp.json https://kubernetes-cluster-ip/api/v1/namespaces/annoying-namespace-to-delete/finalize

Also worked for me in EKS. Just needed to use proxy k proxy and then curl localhost

Removing finalizer worked for me. I PUT empty spec property and removed status property completely. Used kubectl proxy and http://localhost:8001/api/v1/namespaces/{namespace_name}/finalizer as the endpoint

{
    "kind": "Namespace",
    "apiVersion": "v1",
    "metadata": {
        "name": "testing",
        "selfLink": "/api/v1/namespaces/testing",
        "uid": "33074e57-cb72-11e5-9d3d-28d2444e470d",
        "resourceVersion": "234",
        "creationTimestamp": "2016-02-04T19:05:04Z",
        "deletionTimestamp": "2016-02-04T19:05:54Z"
    },
    "spec": {
    }
}

I came across this issue when cleaning up our staging cluster, which our developers use a lot. We use a cluster with nodes provisioned through rancher at Digitalocean. For the other people ending here after googling this issue and looking for an easy way to remove these namespaces, i will leave the shell script I’ve written for these cases here, please use it with care:

# This script deletes namespaces created through rancher with dangling finalizers
namespace=undefined

# path to your kube config file, e.g.: ~/.kube/config
kubeconfig=
# URL of your rancher, e.g.: https://rancher.example.com
rancher_url=
# ID of the cluster, will be found in the URL of the cluster start page: https://rancher.example.com/c/<CLUSTER_ID>/monitoring
cluster_id=

# Your Rancher Bearer Token generated at 'APIs & Keys' in Rancher
RANCHER_BEARER=

# Ask which namespace will be delete
echo "Enter Namespace you want to delete:"
read namespace

echo "Get Namespace $namespace"
kubectl --kubeconfig $kubeconfig get ns $namespace -o json > $namespace.json

# Removes the whole "Spec" block of the namespace
echo "Removing spec block"
sed -i -e '/\"spec\"/,/}/ d; /^$/d' $namespace.json

# Push namespace back, will be deleted immediately if already dangling
echo "Send edited json file back to rancher"
curl -k -H "Content-Type: application/json" -H "Authorization: Bearer $RANCHER_BEARER" -X PUT --data-binary @$namespace.json $rancher_url/k8s/clusters/$cluster_id/api/v1/namespaces/$namespace/finalize```

The (old but effective) comment from @derekwaynecarr did the trick for me

https://github.com/kubernetes/kubernetes/issues/19317#issuecomment-180003984

the only missing step for me was kubectl proxy and changing the port number accordingly (8001 instead of 8080)

I’m hitting this issue on v1.6.10-gke.1 – ns stuck in ‘terminating’ after 1d. Looks like a regression. Manually deleting the finalizer fixed the problem. Here’s a dump of my NS data;

$ kubectl get ns review-bugfix-cb-response-logging-qwil-2210 -o yaml                                             
apiVersion: v1                                                                 
kind: Namespace                                                                
metadata:                                                                      
  creationTimestamp: 2017-10-17T19:45:14Z                                      
  deletionTimestamp: 2017-10-18T22:56:53Z                                      
  name: review-bugfix-cb-response-logging-qwil-2210                            
  resourceVersion: "53696648"                                                  
  selfLink: /api/v1/namespacesreview-bugfix-cb-response-logging-qwil-2210      
  uid: b1fd43a1-b373-11e7-96b9-42010a80000b                                    
spec:                                                                          
  finalizers:                                                                  
  - kubernetes                                                                 
status:                                                                        
  phase: Terminating                                                           

$ kubectl describe ns review-bugfix-cb-response-logging-qwil-2210                                                                                                                                                                                                               
Name:           review-bugfix-cb-response-logging-qwil-2210                    
Labels:         <none>                                                         
Annotations:    <none>                                                         
Status:         Terminating                                                    

No resource quota.                                                             

No resource limits.                                   

I use kubectl edit namespace **** and remove the finalizer part, and save.

https://github.com/rancher/rancher/issues/14715#issuecomment-407900343

same issue on 1.15 , nothing worked in my case.

kubectl delete ns test

Error from server (Conflict): Operation cannot be fulfilled on namespaces “test”: The system is ensuring all content is removed from this namespace. Upon completion, this namespace will automatically be purged by the system.

have you tried to delete it using the following script ?

#!/bin/bash

###############################################################################
# Copyright (c) 2018 Red Hat Inc
#
# See the NOTICE file(s) distributed with this work for additional
# information regarding copyright ownership.
#
# This program and the accompanying materials are made available under the
# terms of the Eclipse Public License 2.0 which is available at
# http://www.eclipse.org/legal/epl-2.0
#
# SPDX-License-Identifier: EPL-2.0
###############################################################################

set -eo pipefail

die() { echo "$*" 1>&2 ; exit 1; }

need() {
	which "$1" &>/dev/null || die "Binary '$1' is missing but required"
}

# checking pre-reqs

need "jq"
need "curl"
need "kubectl"

PROJECT="$1"
shift

test -n "$PROJECT" || die "Missing arguments: kill-ns <namespace>"

kubectl proxy &>/dev/null &
PROXY_PID=$!
killproxy () {
	kill $PROXY_PID
}
trap killproxy EXIT

sleep 1 # give the proxy a second

kubectl get namespace "$PROJECT" -o json | jq 'del(.spec.finalizers[] | select("kubernetes"))' | curl -s -k -H "Content-Type: application/json" -X PUT -o /dev/null --data-binary @- http://localhost:8001/api/v1/namespaces/$PROJECT/finalize && echo "Killed namespace: $PROJECT"

# proxy will get killed by the trap ```
`#!/bin/bash

###############################################################################
# Copyright (c) 2018 Red Hat Inc
#
# See the NOTICE file(s) distributed with this work for additional
# information regarding copyright ownership.
#
# This program and the accompanying materials are made available under the
# terms of the Eclipse Public License 2.0 which is available at
# http://www.eclipse.org/legal/epl-2.0
#
# SPDX-License-Identifier: EPL-2.0
###############################################################################

set -eo pipefail

die() { echo "$*" 1>&2 ; exit 1; }

need() {
	which "$1" &>/dev/null || die "Binary '$1' is missing but required"
}

# checking pre-reqs

need "jq"
need "curl"
need "kubectl"

PROJECT="$1"
shift

test -n "$PROJECT" || die "Missing arguments: kill-ns <namespace>"

kubectl proxy &>/dev/null &
PROXY_PID=$!
killproxy () {
	kill $PROXY_PID
}
trap killproxy EXIT

sleep 1 # give the proxy a second

kubectl get namespace "$PROJECT" -o json | jq 'del(.spec.finalizers[] | select("kubernetes"))' | curl -s -k -H "Content-Type: application/json" -X PUT -o /dev/null --data-binary @- http://localhost:8001/api/v1/namespaces/$PROJECT/finalize && echo "Killed namespace: $PROJECT"

# proxy will get killed by the trap`

usage :  ./kill-kube-ns <name of the namespace>

got the issue on EKS (Kubernetes version: 1.15) kubectl version:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.4", GitCommit:"c96aede7b5205121079932896c4ad89bb93260af", GitTreeState:"clean", BuildDate:"2020-06-17T11:41:22Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.11-eks-065dce", GitCommit:"065dcecfcd2a91bd68a17ee0b5e895088430bd05", GitTreeState:"clean", BuildDate:"2020-07-16T01:44:47Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}

The kill-kube-ns script works for me. Thank you.

None of these suggestions worked and after realising I could be looking at wasting hours trying to fix it, I came to the conclusion that if many equally capable devs had tried before me then I would be wasting my time for something so insignificant. Just deleting the kubernetes data and settings (through the Docker Desktop client) worked fine. You should have scripts for setting up your clusters and stuff anyway so no harm there if you’re in a dev environment.

Can we reopen this issue?

following @derekwaynecarr workaround - which helped me allot - wrote this script which will delete all terminating projects. sort of cleanup script. please someone tell me, i am one RedHat Openshift Enterprise! 3.7 and facing this issue. its so stupid that even i will just create an project and immediatley will delete it i will stumble on this issue. isnt this crazy?!?!!? the enterprise version cost fortune. please just tell me i am not crazy.

this is the script; (uses jq, tested with redhat openshift ocp 3.7)

kubectl proxy & serverPID=$! for row in $(oc get ns -o json | jq -r ‘.items[] | select(.status.phase==“Terminating”) | .metadata.name’); do echo “force deleting name space ${row}” oc project $row oc delete --all all,secret,pvc > /dev/null oc get ns $row -o json > tempns sed -i ‘’ ‘/“kubernetes”/d’ ./tempns curl --silent --output /dev/null -H “Content-Type: application/json” -X PUT --data-binary @tempns http://127.0.0.1:8001/api/v1/namespaces/$row/finalize done kill -9 $serverPID

we are hitting this issue atm on 1.5.2

https://github.com/kubernetes/kubernetes/issues/37554 https://github.com/kubernetes/kubernetes/issues/37278

we are hitting this issue atm on 1.4.6; edit: actually our issue is https://github.com/kubernetes/kubernetes/issues/37278

In my case, ThirdPartyResource had been kept on Etcd. Stucked namespaces was removed after deleting it like this.

etcdctl rm /registry/thirdpartyresources/default/network-policy.net.alpha.kubernetes.io