kubernetes: Can not umount nfs subpath and the pod stack in Terminating status

What happened?

My deployment mount the nfs volume and use subpath to mount the different path in the container. The mount Setting like this

# default the nfs volume
     volumes:
      - name: data-volume
        nfs:
          path: /export/xxxxxx/xxxxxx/xxxxxx
          server: 10.10.10.10
          
# and mount 2 subpaths
        volumeMounts:
        - mountPath: <mp1>
          name: data-volume
          subPath: <sp1>
        - mountPath: <mp2>
          name: data-volume
          subPath: <sp2>

But when I delete the pod, the pod will stack in Terminating status. In the kubelet log, I found the error

Jun 28 11:51:00 k8s-worker kubelet: E0628 11:51:00.387689    1192 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/nfs/93db04c8-1a67-4aa2-bd22-487c135e7a27-data-volume podName:93db04c8-1a67-4aa2-bd22-487c135e7a27 nodeName:}" failed. No retries permitted until 2022-06-28 11:51:08.387656785 +0800 CST m=+1333.758393955 (durationBeforeRetry 8s). Error: "error cleaning subPath mounts for volume \"data-volume\" (UniqueName: \"kubernetes.io/nfs/93db04c8-1a67-4aa2-bd22-487c135e7a27-data-volume\") pod \"93db04c8-1a67-4aa2-bd22-487c135e7a27\" (UID: \"93db04c8-1a67-4aa2-bd22-487c135e7a27\") : error processing /var/lib/kubelet/pods/93db04c8-1a67-4aa2-bd22-487c135e7a27/volume-subpaths/data-volume/expressdaemon: error cleaning subpath mount /var/lib/kubelet/pods/93db04c8-1a67-4aa2-bd22-487c135e7a27/volume-subpaths/data-volume/expressdaemon/2: Failed to unmount path /var/lib/kubelet/pods/93db04c8-1a67-4aa2-bd22-487c135e7a27/volume-subpaths/data-volume/expressdaemon/2"

according to the umount code，the unmount function mounter.Unmount(mountPath) returns nil

func doCleanupMountPoint(mountPath string, mounter Interface, extensiveMountPointCheck bool, corruptedMnt bool) error {
	if !corruptedMnt {
		var notMnt bool
		var err error
		if extensiveMountPointCheck {
			notMnt, err = mounter.IsNotMountPoint(mountPath)
		} else {
			notMnt, err = mounter.IsLikelyNotMountPoint(mountPath)
		}

		if err != nil {
			return err
		}

		if notMnt {
			klog.Warningf("Warning: %q is not a mountpoint, deleting", mountPath)
			return os.Remove(mountPath)
		}
	}

	// Unmount the mount path
	klog.V(4).Infof("%q is a mountpoint, unmounting", mountPath)
	if err := mounter.Unmount(mountPath); err != nil {
		return err
	}

	notMnt, mntErr := mounter.IsLikelyNotMountPoint(mountPath)
	if mntErr != nil {
		return mntErr
	}
	if notMnt {
		klog.V(4).Infof("%q is unmounted, deleting the directory", mountPath)
		return os.Remove(mountPath)
	}
	return fmt.Errorf("Failed to unmount path %v", mountPath)
}

What did you expect to happen?

the subpath umount successfully and the pod can be deleted

How can we reproduce it (as minimally and precisely as possible)?

just like the description in what happened

and the subpath in nfs server mounted with --bind flag

Anything else we need to know?

No response

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.11", GitCommit:"d966b365d87fb2783deab0d244047d340f1be146", GitTreeState:"clean", BuildDate:"2022-01-19T01:10:01Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"


$ uname -a
Linux k8s-dqnew-d04 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, …) and versions (if applicable)

nfs server system version: NAME="Ubuntu" VERSION="18.04.4 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.4 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic

nfs version dpkg -l |grep nfs ii libnfsidmap2:amd64 0.25-5.1 amd64 NFS idmapping library ii nfs-common 1:1.3.4-2.1ubuntu5.2 amd64 NFS support files common to client and server ii nfs-kernel-server 1:1.3.4-2.1ubuntu5.2 amd64 support for NFS kernel server

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 15 (7 by maintainers)

Commits related to this issue

Make sure auto-mounted subpath mount source is already mounted For example, we have two filesystems, one is embedded into another: /a/test # first filesystem with a directory "/a/test/b2" /a/test/b... — committed to Dieken/kubernetes by Dieken 2 years ago
Make sure auto-mounted subpath mount source is already mounted For example, we have two filesystems, one is embedded into another: /a/test # first filesystem with a directory "/a/test/b2" /a/test/b... — committed to andyzhangx/kubernetes by Dieken 2 years ago
Make sure auto-mounted subpath mount source is already mounted For example, we have two filesystems, one is embedded into another: /a/test # first filesystem with a directory "/a/test/b2" /a/test/b... — committed to andyzhangx/kubernetes by Dieken 2 years ago
Make sure auto-mounted subpath mount source is already mounted For example, we have two filesystems, one is embedded into another: /a/test # first filesystem with a directory "/a/test/b2" /a/test/b... — committed to andyzhangx/kubernetes by Dieken 2 years ago

Most upvoted comments

That means: 1. if using mount bind --no-canonicalize subpath, the code does umount subpath, the umount command exits 0, with no error, but the mountpoint still exists 2. you execute umount subpath command manually again, it can delete the mountpoint successfully 3. if removing --no-canonicalize flag, umonut always successes can you test again that if using --no-canonicalize flag with umount command, whether the bug still exists?

249043822 on Jul 4, 2022

I increased our kubelet verbosity up to 4. Unfortunately, I can’t easily isolate this job from other jobs so it’s hard to get the debug - but here is my best shot using grep with context.

Jul  1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34.246087    7422 kubelet.go:2198] "SyncLoop (housekeeping) end"
Jul  1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34.438899    7422 httplog.go:129] "HTTP" verb="GET" URI="/metrics/cadvisor" latency="85.194498ms" userAgent="Prometheus/2.28.1" audit-ID="" srcIP="172.31.32.247:53774" resp=200
Jul  1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34.861641    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/124aee32274342c597bb75ef7656af76eec64c1d42dfe09489f382841c5242e2/merged
Jul  1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34.866086    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/aa4677a3-2ac4-44f9-b9e3-6f7b3c482942/volume-subpaths/depot-pv-smoke/bubblestatus/2
Jul  1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34.866122    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/affc65e1-0964-48fc-a4fb-1019424b51de/volumes/kubernetes.io~nfs/depot-pv-schirato
Jul  1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34.866136    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/affc65e1-0964-48fc-a4fb-1019424b51de/volumes/kubernetes.io~projected/kube-api-access-d9j7q
Jul  1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34.866895    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/aa4677a3-2ac4-44f9-b9e3-6f7b3c482942/volumes/kubernetes.io~nfs/depot-pv-smoke/p4
Jul  1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34.867806    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/aa4677a3-2ac4-44f9-b9e3-6f7b3c482942/volumes/kubernetes.io~nfs/u-smoke-pv-smoke
Jul  1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34.867877    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/aa4677a3-2ac4-44f9-b9e3-6f7b3c482942/volumes/kubernetes.io~nfs/usr-software-rhel7-pv-smoke
Jul  1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34.867892    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/a111cfc33a717069e798806bf7f6c1f1e6b90ba8d3c86a93e753b401607577f0/merged
Jul  1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34.868274    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/ffb20b27fed75b16ba288838dc18b24300ad527068d76350bf031014ba8ed8df/merged
Jul  1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34.869172    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/aa4677a3-2ac4-44f9-b9e3-6f7b3c482942/volumes/kubernetes.io~projected/kube-api-access-szfg9
Jul  1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34.869196    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/db051adec95bca7d895c26bd282bcef0061d7f1f2e1be7d9442b6c9d1e799354/merged
Jul  1 13:52:34 inf-k8s-node003 kubelet[7422]: I0701 13:52:34.869213    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/affc65e1-0964-48fc-a4fb-1019424b51de/volume-subpaths/depot-pv-schirato/bubblestatus/2
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.246826    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/ffb20b27fed75b16ba288838dc18b24300ad527068d76350bf031014ba8ed8df/merged
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.247827    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/aa4677a3-2ac4-44f9-b9e3-6f7b3c482942/volumes/kubernetes.io~projected/kube-api-access-szfg9
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.247852    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/db051adec95bca7d895c26bd282bcef0061d7f1f2e1be7d9442b6c9d1e799354/merged
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.247869    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/affc65e1-0964-48fc-a4fb-1019424b51de/volume-subpaths/depot-pv-schirato/bubblestatus/2
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.247918    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/124aee32274342c597bb75ef7656af76eec64c1d42dfe09489f382841c5242e2/merged
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.252233    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/affc65e1-0964-48fc-a4fb-1019424b51de/volumes/kubernetes.io~nfs/depot-pv-schirato
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.252699    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/aa4677a3-2ac4-44f9-b9e3-6f7b3c482942/volume-subpaths/depot-pv-smoke/bubblestatus/2
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.252722    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/aa4677a3-2ac4-44f9-b9e3-6f7b3c482942/volumes/kubernetes.io~nfs/depot-pv-smoke/p4
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.252734    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/affc65e1-0964-48fc-a4fb-1019424b51de/volumes/kubernetes.io~projected/kube-api-access-d9j7q
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.254384    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/aa4677a3-2ac4-44f9-b9e3-6f7b3c482942/volumes/kubernetes.io~nfs/u-smoke-pv-smoke
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.254480    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/aa4677a3-2ac4-44f9-b9e3-6f7b3c482942/volumes/kubernetes.io~nfs/usr-software-rhel7-pv-smoke
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.254495    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/a111cfc33a717069e798806bf7f6c1f1e6b90ba8d3c86a93e753b401607577f0/merged
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.314637    7422 reconciler.go:192] "operationExecutor.UnmountVolume started for volume \"infra\" (UniqueName: \"kubernetes.io/nfs/886fdd6b-a775-406b-b461-aa3e17c7a5c8-infra-pv-schirato\") pod \"886fdd6b-a775-406b-b461-aa3e17c7a5c8\" (UID: \"886fdd6b-a775-406b-b461-aa3e17c7a5c8\") "
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.314754    7422 subpath_linux.go:244] Cleaning up subpath mounts for /var/lib/kubelet/pods/886fdd6b-a775-406b-b461-aa3e17c7a5c8/volume-subpaths/infra-pv-schirato
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.314846    7422 subpath_linux.go:259] Cleaning up subpath mounts for container bubblestatus
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.326844    7422 subpath_linux.go:310] Cleaning up subpath mounts for subpath 6
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.326914    7422 mount_helper_common.go:99] "/var/lib/kubelet/pods/886fdd6b-a775-406b-b461-aa3e17c7a5c8/volume-subpaths/infra-pv-schirato/bubblestatus/6" is a mountpoint, unmounting
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.326928    7422 mount_linux.go:294] Unmounting /var/lib/kubelet/pods/886fdd6b-a775-406b-b461-aa3e17c7a5c8/volume-subpaths/infra-pv-schirato/bubblestatus/6
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: E0701 13:52:35.408497    7422 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/nfs/886fdd6b-a775-406b-b461-aa3e17c7a5c8-infra-pv-schirato podName:886fdd6b-a775-406b-b461-aa3e17c7a5c8 nodeName:}" failed. No retries permitted until 2022-07-01 13:53:39.40845013 -0400 EDT m=+640.205267069 (durationBeforeRetry 1m4s). Error: error cleaning subPath mounts for volume "infra" (UniqueName: "kubernetes.io/nfs/886fdd6b-a775-406b-b461-aa3e17c7a5c8-infra-pv-schirato") pod "886fdd6b-a775-406b-b461-aa3e17c7a5c8" (UID: "886fdd6b-a775-406b-b461-aa3e17c7a5c8") : error processing /var/lib/kubelet/pods/886fdd6b-a775-406b-b461-aa3e17c7a5c8/volume-subpaths/infra-pv-schirato/bubblestatus: error cleaning subpath mount /var/lib/kubelet/pods/886fdd6b-a775-406b-b461-aa3e17c7a5c8/volume-subpaths/infra-pv-schirato/bubblestatus/6: Failed to unmount path /var/lib/kubelet/pods/886fdd6b-a775-406b-b461-aa3e17c7a5c8/volume-subpaths/infra-pv-schirato/bubblestatus/6
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.430060    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/aa4677a3-2ac4-44f9-b9e3-6f7b3c482942/volume-subpaths/depot-pv-smoke/bubblestatus/2
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.430117    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/affc65e1-0964-48fc-a4fb-1019424b51de/volumes/kubernetes.io~nfs/depot-pv-schirato
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.431155    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/aa4677a3-2ac4-44f9-b9e3-6f7b3c482942/volumes/kubernetes.io~nfs/depot-pv-smoke/p4
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.431190    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/affc65e1-0964-48fc-a4fb-1019424b51de/volumes/kubernetes.io~projected/kube-api-access-d9j7q
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.432261    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/aa4677a3-2ac4-44f9-b9e3-6f7b3c482942/volumes/kubernetes.io~nfs/u-smoke-pv-smoke
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.432292    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/a111cfc33a717069e798806bf7f6c1f1e6b90ba8d3c86a93e753b401607577f0/merged
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.432343    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/aa4677a3-2ac4-44f9-b9e3-6f7b3c482942/volumes/kubernetes.io~nfs/usr-software-rhel7-pv-smoke
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.432358    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/ffb20b27fed75b16ba288838dc18b24300ad527068d76350bf031014ba8ed8df/merged
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.433222    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/affc65e1-0964-48fc-a4fb-1019424b51de/volume-subpaths/depot-pv-schirato/bubblestatus/2
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.433649    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/aa4677a3-2ac4-44f9-b9e3-6f7b3c482942/volumes/kubernetes.io~projected/kube-api-access-szfg9
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.433673    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/db051adec95bca7d895c26bd282bcef0061d7f1f2e1be7d9442b6c9d1e799354/merged
Jul  1 13:52:35 inf-k8s-node003 kubelet[7422]: I0701 13:52:35.433730    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/124aee32274342c597bb75ef7656af76eec64c1d42dfe09489f382841c5242e2/merged
Jul  1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36.123564    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/aa4677a3-2ac4-44f9-b9e3-6f7b3c482942/volume-subpaths/depot-pv-smoke/bubblestatus/2
Jul  1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36.123613    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/affc65e1-0964-48fc-a4fb-1019424b51de/volumes/kubernetes.io~nfs/depot-pv-schirato
Jul  1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36.124555    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/aa4677a3-2ac4-44f9-b9e3-6f7b3c482942/volumes/kubernetes.io~nfs/depot-pv-smoke/p4
Jul  1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36.124592    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/affc65e1-0964-48fc-a4fb-1019424b51de/volumes/kubernetes.io~projected/kube-api-access-d9j7q
Jul  1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36.125987    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/aa4677a3-2ac4-44f9-b9e3-6f7b3c482942/volumes/kubernetes.io~nfs/u-smoke-pv-smoke
Jul  1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36.126053    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/aa4677a3-2ac4-44f9-b9e3-6f7b3c482942/volumes/kubernetes.io~nfs/usr-software-rhel7-pv-smoke
Jul  1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36.126069    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/a111cfc33a717069e798806bf7f6c1f1e6b90ba8d3c86a93e753b401607577f0/merged
Jul  1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36.126476    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/ffb20b27fed75b16ba288838dc18b24300ad527068d76350bf031014ba8ed8df/merged
Jul  1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36.127353    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/aa4677a3-2ac4-44f9-b9e3-6f7b3c482942/volumes/kubernetes.io~projected/kube-api-access-szfg9
Jul  1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36.127377    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/db051adec95bca7d895c26bd282bcef0061d7f1f2e1be7d9442b6c9d1e799354/merged
Jul  1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36.127395    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/kubelet/pods/affc65e1-0964-48fc-a4fb-1019424b51de/volume-subpaths/depot-pv-schirato/bubblestatus/2
Jul  1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36.127455    7422 fs.go:423] unable to determine file system type, partition mountpoint does not exist: /var/lib/docker/overlay2/124aee32274342c597bb75ef7656af76eec64c1d42dfe09489f382841c5242e2/merged
Jul  1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36.192294    7422 kubelet.go:2190] "SyncLoop (housekeeping)"
Jul  1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36.199363    7422 kubelet_pods.go:1082] "Clean up pod workers for terminated pods"
Jul  1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36.199454    7422 kubelet_pods.go:1111] "Clean up probes for terminating and terminated pods"
Jul  1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36.206822    7422 kubelet_pods.go:1148] "Clean up orphaned pod statuses"
Jul  1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36.213308    7422 kubelet_pods.go:1167] "Clean up orphaned pod directories"
Jul  1 13:52:36 inf-k8s-node003 kubelet[7422]: I0701 13:52:36.213637    7422 kubelet_volumes.go:194] "Orphaned pod found, but volumes are not cleaned up" podUID=4116346c-b0be-42b8-8796-37976709b859

Let me know if that doesn’t include what you are looking for.

We end up with a orphaned mount everytime this cron runs. If we remove the mount manually - kubelet detects it and cleans up fine. I rolled us back from 1.23.8 to 1.22.11 and were we seeing this in that release too. We also see it in 1.21.1. We only see the pods get stuck Terminating if we delete the pod while it is runnning. It gets to a completed state if we leave it alone, but we end up with all these orphaned mounts. We didn’t notice we were running out of inotifies until we upgraded to 1.23.6 (also 1.23.8) but it looks like in 1.23 we are leaking faster and that is why we saw it.

dschirato on Jul 1, 2022