rancher: Pods/namespace stuck at removing/terminating state

What kind of request is this (question/bug/enhancement/feature request): bug

Steps to reproduce (least amount of steps as possible): I am not sure if the issue will reappear. But this is exactly what I did to have these pods stuck at terminating state. Launch a wordpress app from catalog with NFS persistent storage claim. Delete it after successful deployment from catalog page. It gets stuck at terminating state forever. It has been more than 6 days now. It is not always the catalog apps, sometimes it happens with custom images as well.

Result: Pods and namespace stuck at removing/terminating state.

Other details that may be helpful: I tried to delete the pods using kubectl delete pod NAME --grace-period=0 --force as well as kubectl delete pod NAME --now The above command returns: warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "nginx-59487766c9-6g6w7" not found

Environment information

• Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): v2.1.1
• Installation option (single install/HA): HA

Cluster information

• Cluster type (Hosted/Infrastructure Provider/Custom/Imported):DigitalOcean
• Machine type (cloud/VM/metal) and specifications (CPU/memory):cloud
• Kubernetes version (use kubectl version):

(paste the output here)

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.5", GitCommit:"32ac1c9073b132b8ba18aa830f46b77dcceb0723", GitTreeState:"clean", BuildDate:"2018-06-21T11:46:00Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

• Docker (use docker info):

(paste the output here)

Containers: 56 Running: 26 Paused: 0 Stopped: 30 Images: 16 Server Version: 17.03.2-ce Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 201 Dirperm1 Supported: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe init version: 949e6fa Security Options: apparmor seccomp Profile: default Kernel Version: 4.4.0-138-generic Operating System: Ubuntu 16.04.5 LTS OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 3.859 GiB Name: rancher1 ID: C2T2:7OS5:RWYA:4VGZ:C6OD:C2JY:GBFL:A6R4:OIZF:M6JB:B5KL:WO3Y Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

WARNING: No swap limit suppor`