kubernetes: CSI volume reconstruction does not work for ephemeral volumes

1. The IsLikelyNotMountPoint() call that verult already called out, which is known not to work for bind mounts. The mountpoint never gets found, so volume reconstruction fails with “… is not mounted”, the PVC never never gets added to the list of attached volumes, and that’s why we end up hitting this error:

There is an ongoing work that changes IsNotMountPoint to utilize openat2(2) syscall to detect mount point by using MountedFast. By using openat2(2), bind mount will be properly detected fast, but it requires kernel version is v5.6 or later. So, we would be able to also utilize openat2(2) in IsLikelyNotMountPoint. Then, the issue of bind mount can be resolved at least for kernel v5.6+, and we will be able to focus on how we solve this issue for old kernels.

I added a couple of extra debug lines to that lines in the actual_state_of_the_world.go file, and can see that when DeletePodFromVolume tries to see if the volume exists, it doesn’t find it in the asw.attachedVolumes struct, like so:

func (asw *actualStateOfWorld) DeletePodFromVolume(
	podName volumetypes.UniquePodName, volumeName v1.UniqueVolumeName) error {
	asw.Lock()
	defer asw.Unlock()

	volumeObj, volumeExists := asw.attachedVolumes[volumeName]
	klog.InfoS("DEBUG:", "volumeName", volumeName)
	klog.InfoS("DEBUG:", "volumeObj", volumeObj, "volumeExists", volumeExists, "asw.attachedVolumes", asw.attachedVolumes)
	klog.Info("DEBUG:", "volumeExists",volumeExists)
	if !volumeExists {
		return fmt.Errorf(
			"no volume with the name %q exists in the list of attached volumes",
			volumeName)
	}

	_, podExists := volumeObj.mountedPods[podName]
	if podExists {
		delete(asw.attachedVolumes[volumeName].mountedPods, podName)
	}

	return nil
}

Oct 07 00:04:30 e2e-test-build-minion-group-klvd kubelet[4740]: I1007 00:04:30.677329    4740 actual_state_of_world.go:662] "DEBUG:" volumeName=pvc-295aa817-8cdd-4dac-818a-6c81b79778f5
Oct 07 00:04:30 e2e-test-build-minion-group-klvd kubelet[4740]: I1007 00:04:30.679563    4740 actual_state_of_world.go:663] "DEBUG:" volumeObj={volumeName: mountedPods:map[] spec:<nil> pluginName: pluginIsAttachable:false deviceMountState: devicePath: deviceMountPath: volumeInUseErrorForExpansion:false} volumeExists=false

Probably we should search for the name of the attached volume differently for ephemeral volumes?