kubernetes: Failed to run the pod with duplicate persistent volumes

What happened:

Create a pod:

apiVersion: v1
kind: Pod
metadata:
  name: test
spec:
  containers:
    - name: test
      image: nginx
      volumeMounts:
        - name: config-a
          mountPath: /etc/a
          subPath: a
        - name: config-b
          mountPath: /etc/b
          subPath: b
  volumes:
    - name: config-a
      persistentVolumeClaim:
        claimName: test-nfs-claim
    - name: config-b
      persistentVolumeClaim:
        claimName: test-nfs-claim

the pod is always ContainerCreating, and describes as:

Warning FailedMount 4s kubelet Unable to attach or mount volumes: unmounted volumes=[config-a], unattached volumes=[config-a config-b default-token-t2wrn]: timed out waiting for the condition

kubelet log:

I0225 13:41:14.817380 2005690 desired_state_of_world_populator.go:361] Added volume "config-a" (volSpec="pvc-a0b70cdf-ac23-43f2-a7bc-14467138ffa9") for pod "64659405-169c-4450-b006-99d00d4018bd" to desired state.
I0225 13:41:14.823159 2005690 desired_state_of_world_populator.go:361] Added volume "config-b" (volSpec="pvc-a0b70cdf-ac23-43f2-a7bc-14467138ffa9") for pod "64659405-169c-4450-b006-99d00d4018bd" to desired state.
I0225 13:41:14.905775 2005690 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-a0b70cdf-ac23-43f2-a7bc-14467138ffa9" (UniqueName: "kubernetes.io/nfs/64659405-169c-4450-b006-99d00d4018bd-pvc-a0b70cdf-ac23-43f2-a7bc-14467138ffa9") pod "test" (UID: "64659405-169c-4450-b006-99d00d4018bd")
I0225 13:41:15.006182 2005690 reconciler.go:254] Starting operationExecutor.MountVolume for volume "pvc-a0b70cdf-ac23-43f2-a7bc-14467138ffa9" (UniqueName: "kubernetes.io/nfs/64659405-169c-4450-b006-99d00d4018bd-pvc-a0b70cdf-ac23-43f2-a7bc-14467138ffa9") pod "test" (UID: "64659405-169c-4450-b006-99d00d4018bd")
I0225 13:41:15.006213 2005690 reconciler.go:269] operationExecutor.MountVolume started for volume "pvc-a0b70cdf-ac23-43f2-a7bc-14467138ffa9" (UniqueName: "kubernetes.io/nfs/64659405-169c-4450-b006-99d00d4018bd-pvc-a0b70cdf-ac23-43f2-a7bc-14467138ffa9") pod "test" (UID: "64659405-169c-4450-b006-99d00d4018bd")
I0225 13:41:15.006319 2005690 nfs.go:246] NFS mount set up: /var/lib/kubelet/pods/64659405-169c-4450-b006-99d00d4018bd/volumes/kubernetes.io~nfs/pvc-a0b70cdf-ac23-43f2-a7bc-14467138ffa9 false stat /var/lib/kubelet/pods/64659405-169c-4450-b006-99d00d4018bd/volumes/kubernetes.io~nfs/pvc-a0b70cdf-ac23-43f2-a7bc-14467138ffa9: no such file or directory
I0225 13:41:15.006482 2005690 mount_linux.go:171] Mounting cmd (mount) with arguments (-t nfs -o vers=4.1 10.246.250.159:/mnt/nfs/default-test-nfs-claim-pvc-a0b70cdf-ac23-43f2-a7bc-14467138ffa9 /var/lib/kubelet/pods/64659405-169c-4450-b006-99d00d4018bd/volumes/kubernetes.io~nfs/pvc-a0b70cdf-ac23-43f2-a7bc-14467138ffa9)
I0225 13:41:15.084139 2005690 operation_generator.go:672] MountVolume.SetUp succeeded for volume "pvc-a0b70cdf-ac23-43f2-a7bc-14467138ffa9" (UniqueName: "kubernetes.io/nfs/64659405-169c-4450-b006-99d00d4018bd-pvc-a0b70cdf-ac23-43f2-a7bc-14467138ffa9") pod "test" (UID: "64659405-169c-4450-b006-99d00d4018bd")

What you expected to happen:

Deny the pod orchestration with duplicate volumes, OR the pod can be running successfully.

How to reproduce it (as minimally and precisely as possible):

Create a pod with the duplicate pvc as above.

Anything else we need to know?:

Root cause：

The key of persistent volume info in DSW is volumeName named afterpluginName/podUid-persistenVolumeName, and the OuterVolumeSpecName is a string:

https://github.com/kubernetes/kubernetes/blob/27c89b9aec73f66529a497910c460af2b25ab6dd/pkg/kubelet/volumemanager/cache/desired_state_of_world.go#L288-L293

When getting the unmounted volumes, it uses OuterVolumeSpecName:

https://github.com/kubernetes/kubernetes/blob/27c89b9aec73f66529a497910c460af2b25ab6dd/pkg/kubelet/volumemanager/volume_manager.go#L435-L441

In our case, config-b volume will corve the volume kubernetes.io/nfs/64659405-169c-4450-b006-99d00d4018bd-pvc-a0b70cdf-ac23-43f2-a7bc-14467138ffa9 in DSW, and the config-a volume info is lost in ASW.

Solution:

2 choices:

• Consider it as an invalid request, and deny creating it.
• Construct uniqueVolumeName with OuterVolumeSpecName when the volume is a persistent volume.

Environment:

• Kubernetes version (use kubectl version): v1.20.4
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

/sig apps /sig node /sig storage