kubernetes: PVC is not annotated with "volume.kubernetes.io/selected-node" when .spec.nodeName is used

What happened: PVC stays forever in Pending state when the Pod that uses the PVC is created with .spec.nodeName.

What you expected to happen: I would expect to be able to schedule a Pod on a predefined Node (using .spec.nodeName) that use a PVC (kubelet running with --cloud-provider=external and CSI migration feature gates are enabled on the corresponding components).

How to reproduce it (as minimally and precisely as possible):

1. The default storage class is the following one:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: default
provisioner: ebs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

1. Apply a PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 4Gi

1. Apply a Pod with .spec.nodeName that uses this PVC

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: centos
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"]
    volumeMounts:
    - name: persistent-storage
      mountPath: /data
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: ebs-claim
  nodeName: ip-10-222-0-45.eu-west-1.compute.internal

1. Ensure that the PVC does not have the volume.kubernetes.io/selected-node annotation. The Pod stays in ContainerCreating with events:

$ k describe po app

  Warning  FailedMount  3m40s (x300 over 93m)  kubelet, ip-10-222-0-45.eu-west-1.compute.internal  Unable to attach or mount volumes: unmounted volumes=[persistent-storage], unattached volumes=[persistent-storage default-token-65nsk]: error processing PVC default/ebs-claim-coreos: PVC is not bound

The PVC events

$ k describe pvc ebs-claim

Events:
  Type    Reason                Age                   From                         Message
  ----    ------                ----                  ----                         -------
  Normal  WaitForFirstConsumer  100s (x382 over 96m)  persistentvolume-controller  waiting for first consumer to be created before binding

kube-controller-manager is waiting forever for the volume.kubernetes.io/selected-node annotation to be added on the PVC:

I0408 11:09:43.546351       1 event.go:278] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ebs-claim", UID:"f93a5a92-4ef8-4625-a8eb-5124ff3cd24f", APIVersion:"v1", ResourceVersion:"11588", FieldPath:""}): type: 'Normal' reason: 'WaitForFirstConsumer' waiting for first consumer to be created before binding
I0408 11:09:58.546487       1 event.go:278] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ebs-claim", UID:"f93a5a92-4ef8-4625-a8eb-5124ff3cd24f", APIVersion:"v1", ResourceVersion:"11588", FieldPath:""}): type: 'Normal' reason: 'WaitForFirstConsumer' waiting for first consumer to be created before binding
I0408 11:09:58.546514       1 event.go:278] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ebs-claim-coreos", UID:"a9d72c3c-1440-4780-ba6e-660b1a0e2679", APIVersion:"v1", ResourceVersion:"14685", FieldPath:""}): type: 'Normal' reason: 'WaitForFirstConsumer' waiting for first consumer to be created before binding

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): v1.18.0
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others: