kubernetes: Orphaned pod found, but error not a directory occurred when trying to remove the volumes dir

Experienced this issue on k8s v1.25.5 running on AKS using the disk.csi.azure.com provisioner.

Manual deletion of the vol_data.json file ultimately did the trick but was not fun to troubleshoot, 0/10 would not recommend.

It happends in k8s 1.22, removing those orphaned pods folders solve this issue (in /var/lib/kubelet/pods/), kubelet is not able to do it automatically.

Discussed in the SIG Storage bug triage meeting. This will be addressed when we work on re-design of reconstruction logic.

Here’s a related bug: https://github.com/kubernetes/kubernetes/issues/111933

As part of SELinux feature, some of the reconstruction issues are currently addressed under an alpha feature gate: https://github.com/kubernetes/enhancements/pull/3548

We will consider separating out the reconstruction logic with a different feature gate.

Found this bug, still in:

k3s version v1.24.3+k3s1 (990ba0e8) go version go1.18.1

We are facing the same issue on a 1.22 k8s cluster

@sgielen from the code: https://github.com/kubernetes/kubernetes/blob/e054181e517b48a3c862207537092c28604aaad9/pkg/kubelet/kubelet_volumes.go#L130 It only can remove empty dir.

I ran into this with many MANY orphaned directories, so I made a one-line shell script to at least save me from copy/paste. This will look at the log, figure out the orphaned pod, and remove its directory. This is being used on k3os, v1.21.5. Check it yourself before running, as all things from the internet. I take no responsibility for breaking your server!

tail /var/log/k3s-service.log | grep "orphaned pod" | awk '{print $18}' | cut -d\\ -f2 | cut -d\" -f2 | uniq | xargs -I % sh -c 'echo "deleting /var/lib/kubelet/pods/%"; rm -rf /var/lib/kubelet/pods/%;'
deleting /var/lib/kubelet/pods/4966fdcc-16db-4c16-ad75-624edfddf546

If you want to confirm the deletion first, before automating it, here’s another version:

k3os [/var/lib/kubelet/pods]# tail  /var/log/k3s-service.log | grep "orphaned pod" | awk '{print $18}' | cut -d\\ -f2 | cut -d\" -f2 | uniq | xargs -p -I % sh -c 'rm -rf %;'
sh -c 'rm -rf 2198391e-ed80-468d-b6f8-a65d3899853d;' ?...y
sh -c 'rm -rf 221b57a5-2c94-40ef-81b1-e8212612a2c0;' ?...y

And if you want to be really lazy/reckless, here’s a short script which will keep running and deleting directories. Stop the script with ctrl+c when no new directories are being deleted, and check the log that it stopped spamming.

k3os [~]# cat delete-orphans.sh 
#!/bin/bash
while true
do
        tail /var/log/k3s-service.log | grep "orphaned pod" | awk '{print $18}' | cut -d\\ -f2 | cut -d\" -f2 | uniq | xargs -I % sh -c 'echo "deleting /var/lib/kubelet/pods/%"; rm -rf /var/lib/kubelet/pods/%;'
        sleep 1
done

Same issue happened on : v1.24.11

Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.11", GitCommit:"0f75679e3346160939924550fd3591462a4afec6", GitTreeState:"clean", BuildDate:"2023-02-22T13:32:00Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}

I’m seeing the same thing on v1.23.6 while using rook-ceph. I also occasionally use nfs-server-provisioner.