rancher: vSphere/ESXi 7.0 VM provisioning that requires modifying the VM properties fails (Ie adding devices)

What kind of request is this (question/bug/enhancement/feature request):

Bug

Steps to reproduce (least amount of steps as possible):

Create a vSphere node template containing multiple networks. Use this template to provision a cluster to a vSphere 7 & ESXi7 environment

Result:

The following is presented in vSphere and the node fails to provision. Note, single NIC nodes provision without issue.

Other details that may be helpful:

Trace logs from Rancher (However, these don’t add anything extra)

2020/06/25 12:40:52 [INFO] [node-controller-rancher-machine] (m1-1) Adding network: /Homelab/network/VDS-MGMT-DEFAULT
2020/06/25 12:40:52 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-9nsw7/nodes/m-lwljm
2020/06/25 12:40:52 [DEBUG] stdout: (m1-1) Adding network: /Homelab/network/VDS-NEST-DEFAULT
2020/06/25 12:40:52 [INFO] [node-controller-rancher-machine] (m1-1) Adding network: /Homelab/network/VDS-NEST-DEFAULT
2020/06/25 12:40:52 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-9nsw7/nodes/m-lwljm

2020/06/25 12:40:55 [TRACE] REST UPDATE apis/management.cattle.io/v3/c-9nsw7/nodes/m-lwljm
2020/06/25 12:40:55 [TRACE] NodeController calling handler rke-worker-upgrader c-9nsw7/m-lwljm
2020/06/25 12:40:55 [ERROR] NodeController c-9nsw7/m-lwljm [node-controller] failed with : Error creating machine: Error in driver during machine creation: A specified parameter was not correct: deviceChange[1].device.key
2020/06/25 12:40:55 [TRACE] NodeController calling handler cluster-provisioner-controller c-9nsw7/m-lwljm
2020/06/25 12:40:55 [TRACE] NodeController calling handler cluster-stats c-9nsw7/m-lwljm
2020/06/25 12:40:55 [TRACE] NodeController calling handler nodepool-provisioner c-9nsw7/m-lwljm
2020/06/25 12:40:55 [TRACE] NodeController calling handler node-controller c-9nsw7/m-lwljm
2020/06/25 12:40:55 [TRACE] NodeController calling handler rke-worker-upgrader c-9nsw7/m-lwljm
2020/06/25 12:40:55 [TRACE] NodeController c-9nsw7/m-lwljm completed with dropped err: [node-controller] failed with : Error
2020/06/25 12:40:55 [DEBUG] Triggering auth refresh on u-4r7xs

vCenter VPXD logs (Note the key as -1 for both vNICs)

2020-06-25T13:20:04.116Z info vpxd[10851] [Originator@6876 sub=Default opID=97e94a2-01] [VpxLRO] -- ERROR lro-1115251 -- vm-8840 -- vim.VirtualMachine.reconfigure: vmodl.fault.InvalidArgument:
--> Result:
--> (vmodl.fault.InvalidArgument) {
-->    faultCause = (vmodl.MethodFault) null, 
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.vim.vpxd.vmprov.duplicateDeviceKey", 
-->          arg = <unset>, 
-->          message = <unset>
-->       }
-->    ], 
-->    invalidProperty = "deviceChange[1].device.key"
-->    msg = ""
--> }
--> Args:
--> 
--> Arg spec:
--> (vim.vm.ConfigSpec) {
-->    changeVersion = <unset>, 
-->    name = <unset>, 
-->    version = <unset>, 
-->    createDate = "2020-06-25T13:20:00.543008Z", 
-->    uuid = <unset>, 
-->    instanceUuid = <unset>, 
-->    npivNodeWorldWideName = <unset>, 
-->    npivPortWorldWideName = <unset>, 
-->    npivWorldWideNameType = <unset>, 
-->    npivDesiredNodeWwns = <unset>, 
-->    npivDesiredPortWwns = <unset>, 
-->    npivTemporaryDisabled = <unset>, 
-->    npivOnNonRdmDisks = <unset>, 
-->    npivWorldWideNameOp = <unset>, 
-->    locationId = <unset>, 
-->    guestId = <unset>, 
-->    alternateGuestName = <unset>, 
-->    annotation = <unset>, 
-->    files = (vim.vm.FileInfo) null, 
-->    tools = (vim.vm.ToolsConfigInfo) null, 
-->    flags = (vim.vm.FlagInfo) null, 
-->    consolePreferences = (vim.vm.ConsolePreferences) null, 
-->    powerOpInfo = (vim.vm.DefaultPowerOpInfo) null, 
-->    numCPUs = <unset>, 
-->    vcpuConfig = <unset>, 
-->    numCoresPerSocket = <unset>, 
-->    autoCoresPerSocket = <unset>, 
-->    memoryMB = <unset>, 
-->    memoryHotAddEnabled = <unset>, 
-->    cpuHotAddEnabled = <unset>, 
-->    cpuHotRemoveEnabled = <unset>, 
-->    virtualICH7MPresent = <unset>, 
-->    virtualSMCPresent = <unset>, 
-->    deviceChange = (vim.vm.device.VirtualDeviceSpec) [
-->       (vim.vm.device.VirtualDeviceSpec) {
-->          operation = "add", 
-->          fileOperation = <unset>, 
-->          device = (vim.vm.device.VirtualVmxnet3) {
-->             dynamicProperty = <unset>, 
-->             key = -1, 
-->             deviceInfo = (vim.Description) null, 
-->             backing = (vim.vm.device.VirtualEthernetCard.DistributedVirtualPortBackingInfo) {
-->                port = (vim.dvs.PortConnection) {
-->                   switchUuid = "50 3b f7 94 0f 8f 73 20-24 0f a4 38 2b 85 b7 c9", 
-->                   portgroupKey = "dvportgroup-27", 
-->                   portKey = <unset>, 
-->                   connectionCookie = <unset>
-->                }
-->             }, 
-->             connectable = (vim.vm.device.VirtualDevice.ConnectInfo) null, 
-->             slotInfo = (vim.vm.device.VirtualDevice.BusSlotInfo) null, 
-->             controllerKey = <unset>, 
-->             unitNumber = <unset>, 
-->             addressType = <unset>, 
-->             macAddress = <unset>, 
-->             wakeOnLanEnabled = <unset>, 
-->             resourceAllocation = (vim.vm.device.VirtualEthernetCard.ResourceAllocation) null, 
-->             externalId = <unset>, 
-->             uptCompatibilityEnabled = <unset>
-->          }, 
-->          profile = <unset>, 
-->          backing = (vim.vm.device.VirtualDeviceSpec.BackingSpec) null
-->       }, 
-->       (vim.vm.device.VirtualDeviceSpec) {
-->          operation = "add", 
-->          fileOperation = <unset>, 
-->          device = (vim.vm.device.VirtualVmxnet3) {
-->             dynamicProperty = <unset>, 
-->             key = -1, 
-->             deviceInfo = (vim.Description) null, 
-->             backing = (vim.vm.device.VirtualEthernetCard.DistributedVirtualPortBackingInfo) {
-->                port = (vim.dvs.PortConnection) {
-->                   switchUuid = "50 3b f7 94 0f 8f 73 20-24 0f a4 38 2b 85 b7 c9", 
-->                   portgroupKey = "dvportgroup-48", 
-->                   portKey = <unset>, 
-->                   connectionCookie = <unset>
-->                }
-->             }, 
-->             connectable = (vim.vm.device.VirtualDevice.ConnectInfo) null, 
-->             slotInfo = (vim.vm.device.VirtualDevice.BusSlotInfo) null, 
-->             controllerKey = <unset>, 
-->             unitNumber = <unset>, 
-->             addressType = <unset>, 
-->             macAddress = <unset>, 
-->             wakeOnLanEnabled = <unset>, 
-->             resourceAllocation = (vim.vm.device.VirtualEthernetCard.ResourceAllocation) null, 
-->             externalId = <unset>, 
-->             uptCompatibilityEnabled = <unset>
-->          }, 
-->          profile = <unset>, 
-->          backing = (vim.vm.device.VirtualDeviceSpec.BackingSpec) null
-->       }
-->    ], 
-->    cpuAllocation = (vim.ResourceAllocationInfo) null, 
-->    memoryAllocation = (vim.ResourceAllocationInfo) null, 
-->    latencySensitivity = (vim.LatencySensitivity) null, 
-->    cpuAffinity = (vim.vm.AffinityInfo) null, 
-->    memoryAffinity = (vim.vm.AffinityInfo) null, 
-->    networkShaper = (vim.vm.NetworkShaperInfo) null, 
-->    cpuFeatureMask = <unset>, 
-->    extraConfig = <unset>, 
-->    swapPlacement = <unset>, 
-->    bootOptions = (vim.vm.BootOptions) null, 
-->    vAppConfig = (vim.vApp.VmConfigSpec) null, 
-->    ftInfo = (vim.vm.FaultToleranceConfigInfo) null, 
-->    repConfig = (vim.vm.ReplicationConfigSpec) null, 
-->    vAppConfigRemoved = <unset>, 
-->    vAssertsEnabled = <unset>, 
-->    changeTrackingEnabled = <unset>, 
-->    firmware = <unset>, 
-->    maxMksConnections = <unset>, 
-->    guestAutoLockEnabled = <unset>, 
-->    managedBy = (vim.ext.ManagedByInfo) null, 
-->    memoryReservationLockedToMax = true, 
-->    nestedHVEnabled = <unset>, 
-->    vPMCEnabled = <unset>, 
-->    scheduledHardwareUpgradeInfo = (vim.vm.ScheduledHardwareUpgradeInfo) null, 
-->    vmProfile = <unset>, 
-->    messageBusTunnelEnabled = <unset>, 
-->    crypto = (vim.encryption.CryptoSpec) null, 
-->    migrateEncryption = <unset>, 
-->    sgxInfo = (vim.vm.SgxInfo) null, 
-->    ftEncryptionMode = <unset>, 
-->    guestMonitoringModeInfo = (vim.vm.GuestMonitoringModeInfo) null
--> }
2020-06-25T13:20:04.116Z info vpxd[10851] [Originator@6876 sub=vpxLro opID=97e94a2] [VpxLRO] -- FINISH task-3072
2020-06-25T13:20:04.116Z info vpxd[10851] [Originator@6876 sub=Default opID=97e94a2] [VpxLRO] -- ERROR task-3072 -- vm-8840 -- vim.VirtualMachine.reconfigure: vmodl.fault.InvalidArgument:
--> Result:
--> (vmodl.fault.InvalidArgument) {
-->    faultCause = (vmodl.MethodFault) null, 
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.vim.vpxd.vmprov.duplicateDeviceKey", 
-->          arg = <unset>, 
-->          message = <unset>
-->       }
-->    ], 
-->    invalidProperty = "deviceChange[1].device.key"
-->    msg = ""
--> }

Environment information

• Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI):
• Installation option (single install/HA):

Cluster information

• Cluster type (Hosted/Infrastructure Provider/Custom/Imported):
• Machine type (cloud/VM/metal) and specifications (CPU/memory):
• Kubernetes version (use kubectl version):

vSphere IaaS
2vCPU 4GB Ram
N/A - Cluster does not provision

• Docker version (use docker version):

N/A - Cluster does not provision

---

Useful	Info
Versions	Rancher v2.4.3 UI: v2.4.22
Route	undefined