vsphere-csi-driver: Volume provisioning fails with "invalid metadata" when using a native key provider

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug

What happened: PV provisioning fails when using a native key provider. Specifically, I’m seeing the following messages on the ESXi host (my provider is named provider-0, my cluster is named cl-0):

warning kmxa[2097952] [Originator@6876 sub=Libs opID=resolveKey-52a1d087-b13e-71e3-dd80-4aca154a5817-48] Failed to resolve key /provider-0: invalid metadata.

and

2022-04-30T23:48:58.034Z info vpxa[2100223] [Originator@6876 sub=Default opID=0fea314b-ea6c-4664-b319-34d81a6f7b69-362537-12-e] [VpxLRO] -- ERROR task-72673 -- vstorageObjectManager -- vim.vslm.host.VStorageObjectManager.createDisk: vmodl.fault.SystemError:
--> Result:
--> (vmodl.fault.SystemError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    reason = "Key locator error: api = DiskLib_Create, path = /vmfs/volumes/{redacted}/fcd/{redacted}.vmdk"
-->    msg = "A general system error occurred: Key locator error: api = DiskLib_Create, path = /vmfs/volumes/{redacted}/fcd/{redacted}.vmdk"
--> }
--> Args:
-->
--> Arg spec:
--> (vim.vslm.CreateSpec) {
-->    name = "pvc-{redacted}",
-->    keepAfterDeleteVm = true,
-->    backingSpec = (vim.vslm.CreateSpec.DiskFileBackingSpec) {
-->       datastore = 'vim.Datastore:{redacted}',
-->       path = <unset>,
-->       virtualDiskFormat = <unset>,
-->       provisioningType = <unset>
-->    },
-->    capacityInMB = 1024,
-->    profile = (vim.vm.ProfileSpec) [
-->       (vim.vm.DefinedProfileSpec) {
-->          profileId = "{redacted}",
-->          replicationSpec = (vim.vm.replication.ReplicationSpec) null,
-->          profileData = (vim.vm.ProfileRawData) {
-->             extensionKey = "com.vmware.vim.sps",
-->             objectData = "<ns1:storageProfile xmlns:ns1="http://profile.policy.data.vasa.vim.vmware.com/xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ns1:StorageProfile"><ns1:constraints><ns1:subProfiles><ns1:capability><capabilityId xmlns="http://capability.policy.data.vasa.vim.vmware.com/xsd"><id>vmwarevmcrypt@ENCRYPTION</id><namespace>IOFILTERS</namespace></capabilityId><constraint xmlns="http://capability.policy.data.vasa.vim.vmware.com/xsd"><propertyInstance><id>AllowCleartextFilters</id><value xmlns:s90="http://www.w3.org/2001/XMLSchema" xsi:type="s90:string">False</value></propertyInstance></constraint></ns1:capability><ns1:name>Host based services</ns1:name></ns1:subProfiles></ns1:constraints><ns1:createdBy>Temporary user handle</ns1:createdBy><ns1:creationTime>2022-03-12T06:39:17.817+00:00</ns1:creationTime><ns1:description>Sample storage policy for VMware's VM and virtual disk encryption</ns1:description><ns1:generationId>2</ns1:gen
erationId><ns1:lastUpdatedBy>Temporary user handle</ns1:lastUpdatedBy><ns1:lastUpdatedTime>2022-04-29T16:11:20.912-07:00</ns1:lastUpdatedTime><ns1:name>VM Encryption Policy</ns1:name><ns1:profileId>{redacted}</ns1:profileId></ns1:storageProfile>"
-->          },
-->          profileParams = <unset>
-->       }
-->    ],
-->    crypto = (vim.encryption.CryptoSpecEncrypt) {
-->       cryptoKeyId = (vim.encryption.CryptoKeyId) {
-->          keyId = "",
-->          providerId = (vim.encryption.KeyProviderId) {
-->             id = "provider-0"
-->          }
-->       },
-->       inPlace = <unset>
-->    },
-->    metadata = (vim.KeyValue) [
-->       (vim.KeyValue) {
-->          key = "cns.tag",
-->          value = "true"
-->       },
-->       (vim.KeyValue) {
-->          key = "cns.version",
-->          value = "3"
-->       },
-->       (vim.KeyValue) {
-->          key = "cns.containerCluster.clusterId",
-->          value = "cl-0"
-->       },
-->       (vim.KeyValue) {
-->          key = "cns.containerCluster.clusterType",
-->          value = "KUBERNETES"
-->       },
-->       (vim.KeyValue) {
-->          key = "cns.containerCluster.vSphereUser",
-->          value = "{redacted}"
-->       },
-->       (vim.KeyValue) {
-->          key = "cns.containerCluster.clusterFlavor",
-->          value = "VANILLA"
-->       }
-->    ]
--> }

I have the storagepolicyname param set to VM Encryption Policy. All node VM’s and all disks are already encrypted using provider-0. The provisioner successfully creates unencrypted PV’s when using the default storage policy. provider-0 has a Key ID in vSphere, but it’s clearly missing in the log message. Am I missing part of the setup or is the csi-provisioner not setting a required param?

What you expected to happen: The csi-provisioner to successfully create an encrypted FCD.

How to reproduce it (as minimally and precisely as possible): Create a Native Key Provider. Create a storage policy that uses the VMware VM Encryption provider. Create a storage class that uses the policy. Create a PVC using the storage class.

Anything else we need to know?: DRS and Storage DRS are enabled.

Environment:

• csi-vsphere version: v2.5.1
• vsphere-cloud-controller-manager version: v1.22.6
• Kubernetes version: v1.22.9
• vSphere version: 7.0.3 (build 19480866)
• OS (e.g. from /etc/os-release): Ubuntu 22.04 LTS
• Kernel (e.g. uname -a): 5.15.0-27-generic
• Install tools: kubeadm + kubectl
• ESXi version: 7.0.3 (build 19482537)