kubernetes: Persistence Volume Azure Virtual Machine Scale Sets mount failed

What happened: Controller Manager can’t request to attach the Disk (PV) to Virtual Machine from VMSS

I0319 12:50:40.197498 1 event.go:221] Event(v1.ObjectReference{Kind:“Pod”, Namespace:“u001”, Name:“u001-cassandra-csb-0”, UID:“900a6abd-4a45-11e9-a32f-000d3a6cc167”, APIVersion:“v1”, ResourceVersion:“2242297”, FieldPath:“”}): type: ‘Warning’ reason: ‘FailedAttachVolume’ AttachVolume.Attach failed for volume “pvc-f852fa33-43b5-11e9-b42a-000d3a6cc167” : Attach volume “kubernetes-dynamic-pvc-f852fa33-43b5-11e9-b42a-000d3a6cc167” to instance “/subscriptions/xxxx/resourceGroups/qus7uw2-xxxx/providers/Microsoft.Compute/virtualMachineScaleSets/xxxx-qus7uw2l-k8sworker-7/virtualMachines/1” failed with compute.VirtualMachineScaleSetVMsClient#Update: Failure sending request: StatusCode=0 – Original Error: autorest/azure: Service returned an error. Status=<nil> Code=“OperationNotAllowed” Message=“Operation Put on Virtual Machine Scale Set VM Instance 1 of virtual machine scale set xxxx-qus7uw2l-k8sworker-7 is not allowed because the VM instance is not provisioned yet.”

How to reproduce it (as minimally and precisely as possible): It just happened after some days work correctly.

Anything else we need to know?: The Virtual Machine Scale Set VM Instance 1 started correctly with cloud-config and cloud-provider=azure are set. (Verified from Azure console and kubectl get node and ProviderID was set when node join to cluster by kube-controller-manager)

Environment:

• Kubernetes version (use kubectl version): 1.13.4
• Cloud provider or hardware configuration: Azure
• OS (e.g: cat /etc/os-release): Centos7
• Kernel (e.g. uname -a): 3.10.0-862.14.4.el7.x86_64
• Install tools: self-deployment
• Others:

/sig azure