kubernetes: Pods with subPath are stuck on Terminating

What happened:

Pods with subPath are stuck on Terminating when deleting

What you expected to happen:

Pod succefully removed.

How to reproduce it (as minimally and precisely as possible):

crate:

---
apiVersion: v1
kind: Pod
metadata:
  name: test
spec:
  containers:
  - image: docker.io/library/alpine
    name: test
    command: ['sleep', 'infinity']
    volumeMounts:
    - mountPath: /blabla/sss
      name: keys
      subPath: somekey
  volumes:
  - name: keys
    secret:
      secretName: test
---
apiVersion: v1
kind: Secret
metadata:
  name: test
data:
  somekey: R29vZCBhdHRlbXBsdCwgYnV0IG5vdCE=
  someotherkey: QXJlIHlvdSBraWRkaW5nIG1lPw==

remove the pod:

kubectl delete pod test

check the kubelet logs:

Mar 17 20:40:31 m1c4 kubelet[13866]: E0317 20:40:31.230957   13866 nestedpendingoperations.go:270] Operation for "\"kubernetes.io/secret/89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238-keys\" (\"89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238\")" failed. No retries permitted until 2020-03-17 20:42:33.230929328 +0100 CET m=+1192722.291304745 (durationBeforeRetry 2m2s). Error: "error cleaning subPath mounts for volume \"keys\" (UniqueName: \"kubernetes.io/secret/89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238-keys\") pod \"89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238\" (UID: \"89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238\") : error processing /var/lib/kubelet/pods/89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238/volume-subpaths/keys/test: error cleaning subpath mount /var/lib/kubelet/pods/89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238/volume-subpaths/keys/test/0: unmount failed: exit status 32\nUnmounting arguments: /var/lib/kubelet/pods/89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238/volume-subpaths/keys/test/0\nOutput: umount: /var/lib/kubelet/pods/89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238/volume-subpaths/keys/test/0: not mounted.\n"
Mar 17 20:42:33 m1c4 kubelet[13866]: I0317 20:42:33.234534   13866 reconciler.go:183] operationExecutor.UnmountVolume started for volume "keys" (UniqueName: "kubernetes.io/secret/89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238-keys") pod "89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238" (UID: "89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238")
Mar 17 20:42:33 m1c4 kubelet[13866]: E0317 20:42:33.257819   13866 nestedpendingoperations.go:270] Operation for "\"kubernetes.io/secret/89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238-keys\" (\"89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238\")" failed. No retries permitted until 2020-03-17 20:44:35.257684981 +0100 CET m=+1192844.318060559 (durationBeforeRetry 2m2s). Error: "error cleaning subPath mounts for volume \"keys\" (UniqueName: \"kubernetes.io/secret/89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238-keys\") pod \"89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238\" (UID: \"89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238\") : error processing /var/lib/kubelet/pods/89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238/volume-subpaths/keys/test: error cleaning subpath mount /var/lib/kubelet/pods/89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238/volume-subpaths/keys/test/0: unmount failed: exit status 32\nUnmounting arguments: /var/lib/kubelet/pods/89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238/volume-subpaths/keys/test/0\nOutput: umount: /var/lib/kubelet/pods/89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238/volume-subpaths/keys/test/0: not mounted.\n"

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:20:10Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:12:17Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: Bare Metal
• OS (e.g: cat /etc/os-release):

NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

• Kernel (e.g. uname -a):

Linux m1c4 4.15.18-24-pve #1 SMP PVE 4.15.18-52 (Thu, 05 Dec 2019 10:14:17 +0100) x86_64 x86_64 x86_64 GNU/Linux

• Install tools: kubeadm
• Network plugin and version (if this is a network-related bug):
• Others: docker version: 19.3.5

docker is running with the next config:

{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "iptables": false,
  "ip-forward": false,
  "bridge": "none",
  "log-driver": "journald",
  "storage-driver": "overlay2"
}

kubelet versions:

Kubernetes v1.17.1
Kubernetes v1.17.4

created file is empty and not mountpoint, path is not symlink:

# mountpoint /var/lib/kubelet/pods/89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238/volume-subpaths/keys/test/0 
/var/lib/kubelet/pods/89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238/volume-subpaths/keys/test/0 is not a mountpoint

# ls -lah
-rw-r----- 1 root root 0 Mar 17 20:37 /var/lib/kubelet/pods/89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238/volume-subpaths/keys/test/0

# readlink -f /var/lib/kubelet/pods/89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238/volume-subpaths/keys/test/0 
/var/lib/kubelet/pods/89b3eaa9-0bd8-4cc6-89a0-2fd9a346e238/volume-subpaths/keys/test/0

Not sure, but maybe problem caused by overlayfs root:

# df -h /var/lib/kubelet/pods/
Filesystem               Size  Used Avail Use% Mounted on
/run/initramfs/ltsp/cow   16G  412M   16G   3% /