kubernetes: Persistent Volume failing to mount - timeout - AKS

@andyzhangx

What happened: We observe that mounting a persistent volume claim times out for volumes of 1TB or larger

What you expected to happen: That the volume mounts immediately or atleast within the timeout window

How to reproduce it (as minimally and precisely as possible): Create new stateful set and PV with 1tb

Anything else we need to know?: Tried it with different instance types to no avail

Environment: kubernetes v1.12.4 Azure region: West Europe

`Name: graphite-0 Namespace: metrics Priority: 0 PriorityClassName: <none> Node: vmss-vm92409000000/10.240.6.80 Start Time: Wed, 20 Mar 2019 18:06:23 +0000 Labels: controller-revision-hash=graphite-5fb5c75765 statefulset.kubernetes.io/pod-name=graphite-0 workload.user.cattle.io/workloadselector=statefulSet-prometheus-graphite Annotations: cattle.io/timestamp=2019-03-20T17:30:49Z field.cattle.io/ports=[[{“containerPort”:8080,“dnsName”:“graphite-nodeport”,“kind”:“NodePort”,“name”:“80tcp801”,“protocol”:“TCP”,“sourcePort”:80},{“containerPort”:2003,“dnsName”:“graphite-nodeport”,"k… Status: Pending IP:
Controlled By: StatefulSet/graphite Containers: graphite: Container ID:
Image: graphiteapp/graphite-statsd:1.1.5-10 Image ID:
Ports: 8080/TCP, 2003/TCP, 2023/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Limits: memory: 500Mi Requests: memory: 500Mi Environment: <none> Mounts: /opt/graphite/conf/ from graphite-config (rw) /opt/graphite/storage/whisper from graphite-data (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-rrp2l (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: graphite-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: graphite-server-conf Optional: false graphite-data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: graphite-data ReadOnly: false default-token-rrp2l: Type: Secret (a volume populated by a Secret) SecretName: default-token-rrp2l Optional: false QoS Class: Burstable Node-Selectors: <none> Tolerations: large node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message

---

Warning FailedAttachVolume 4m (x515 over 18h) attachdetach-controller AttachVolume.Attach failed for volume “pvc-25ba6630-3b6e-11e9-8700-000d3a450542” : Attach volume “k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542” to instance “/subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/virtualMachineScaleSets/sig-0901eba7_standard_f8s_v2_low/virtualMachines/0” failed with compute.VirtualMachineScaleSetVMsClient#Update: Failure sending request: StatusCode=0 – Original Error: autorest/azure: Service returned an error. Status=<nil> Code=“OperationNotAllowed” Message=“” Warning FailedMount 3m (x474 over 17h) kubelet, vmss-vm92409000000 Unable to mount volumes for pod “graphite-0_metrics(f02668ea-4b38-11e9-8700-000d3a450542)”: timeout expired waiting for volumes to attach or mount for pod “metrics”/“graphite-0”. list of unmounted volumes=[graphite-data]. list of unattached volumes=[graphite-config graphite-data default-token-rrp2l]`

• Cloud provider AKS

Kubelet logs

I0320 10:33:00.772129 3260 kubelet.go:1883] SyncLoop (ADD, "api"): "graphite-0_metrics(2dcb2656-4afb-11e9-8700-000d3a450542)" I0320 10:33:00.924599 3260 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-25ba6630-3b6e-11e9-8700-000d3a450542" (UniqueName: "kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542") pod "graphite-0" (UID: "2dcb2656-4afb-11e9-8700-000d3a450542") I0320 10:33:00.924655 3260 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-rrp2l" (UniqueName: "kubernetes.io/secret/2dcb2656-4afb-11e9-8700-000d3a450542-default-token-rrp2l") pod "graphite-0" (UID: "2dcb2656-4afb-11e9-8700-000d3a450542") I0320 10:33:00.924680 3260 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "graphite-config" (UniqueName: "kubernetes.io/configmap/2dcb2656-4afb-11e9-8700-000d3a450542-graphite-config") pod "graphite-0" (UID: "2dcb2656-4afb-11e9-8700-000d3a450542") E0320 10:33:00.924785 3260 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\"" failed. No retries permitted until 2019-03-20 10:33:01.4247423 +0000 UTC m=+32738.178114201 (durationBeforeRetry 500ms). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-25ba6630-3b6e-11e9-8700-000d3a450542\" (UniqueName: \"kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\") pod \"graphite-0\" (UID: \"2dcb2656-4afb-11e9-8700-000d3a450542\") " I0320 10:33:01.025091 3260 reconciler.go:252] operationExecutor.MountVolume started for volume "default-token-rrp2l" (UniqueName: "kubernetes.io/secret/2dcb2656-4afb-11e9-8700-000d3a450542-default-token-rrp2l") pod "graphite-0" (UID: "2dcb2656-4afb-11e9-8700-000d3a450542") I0320 10:33:01.025150 3260 reconciler.go:252] operationExecutor.MountVolume started for volume "graphite-config" (UniqueName: "kubernetes.io/configmap/2dcb2656-4afb-11e9-8700-000d3a450542-graphite-config") pod "graphite-0" (UID: "2dcb2656-4afb-11e9-8700-000d3a450542") I0320 10:33:01.025891 3260 operation_generator.go:568] MountVolume.SetUp succeeded for volume "graphite-config" (UniqueName: "kubernetes.io/configmap/2dcb2656-4afb-11e9-8700-000d3a450542-graphite-config") pod "graphite-0" (UID: "2dcb2656-4afb-11e9-8700-000d3a450542") I0320 10:33:01.035414 3260 operation_generator.go:568] MountVolume.SetUp succeeded for volume "default-token-rrp2l" (UniqueName: "kubernetes.io/secret/2dcb2656-4afb-11e9-8700-000d3a450542-default-token-rrp2l") pod "graphite-0" (UID: "2dcb2656-4afb-11e9-8700-000d3a450542") I0320 10:33:01.426606 3260 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-25ba6630-3b6e-11e9-8700-000d3a450542" (UniqueName: "kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542") pod "graphite-0" (UID: "2dcb2656-4afb-11e9-8700-000d3a450542") E0320 10:33:01.426731 3260 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\"" failed. No retries permitted until 2019-03-20 10:33:02.4266917 +0000 UTC m=+32739.180063501 (durationBeforeRetry 1s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-25ba6630-3b6e-11e9-8700-000d3a450542\" (UniqueName: \"kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\") pod \"graphite-0\" (UID: \"2dcb2656-4afb-11e9-8700-000d3a450542\") " E0320 10:33:02.420594 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope I0320 10:33:02.429843 3260 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-25ba6630-3b6e-11e9-8700-000d3a450542" (UniqueName: "kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542") pod "graphite-0" (UID: "2dcb2656-4afb-11e9-8700-000d3a450542") E0320 10:33:02.429939 3260 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\"" failed. No retries permitted until 2019-03-20 10:33:04.4299013 +0000 UTC m=+32741.183273101 (durationBeforeRetry 2s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-25ba6630-3b6e-11e9-8700-000d3a450542\" (UniqueName: \"kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\") pod \"graphite-0\" (UID: \"2dcb2656-4afb-11e9-8700-000d3a450542\") " I0320 10:33:04.436035 3260 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-25ba6630-3b6e-11e9-8700-000d3a450542" (UniqueName: "kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542") pod "graphite-0" (UID: "2dcb2656-4afb-11e9-8700-000d3a450542") E0320 10:33:04.439748 3260 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\"" failed. No retries permitted until 2019-03-20 10:33:08.439709 +0000 UTC m=+32745.193080801 (durationBeforeRetry 4s). Error: "Volume not attached according to node status for volume \"pvc-25ba6630-3b6e-11e9-8700-000d3a450542\" (UniqueName: \"kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\") pod \"graphite-0\" (UID: \"2dcb2656-4afb-11e9-8700-000d3a450542\") " I0320 10:33:08.448440 3260 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-25ba6630-3b6e-11e9-8700-000d3a450542" (UniqueName: "kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542") pod "graphite-0" (UID: "2dcb2656-4afb-11e9-8700-000d3a450542") E0320 10:33:08.451579 3260 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\"" failed. No retries permitted until 2019-03-20 10:33:16.4515383 +0000 UTC m=+32753.204910101 (durationBeforeRetry 8s). Error: "Volume not attached according to node status for volume \"pvc-25ba6630-3b6e-11e9-8700-000d3a450542\" (UniqueName: \"kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\") pod \"graphite-0\" (UID: \"2dcb2656-4afb-11e9-8700-000d3a450542\") " E0320 10:33:12.452878 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope I0320 10:33:16.475975 3260 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-25ba6630-3b6e-11e9-8700-000d3a450542" (UniqueName: "kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542") pod "graphite-0" (UID: "2dcb2656-4afb-11e9-8700-000d3a450542") E0320 10:33:16.479669 3260 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\"" failed. No retries permitted until 2019-03-20 10:33:32.4796277 +0000 UTC m=+32769.232999601 (durationBeforeRetry 16s). Error: "Volume not attached according to node status for volume \"pvc-25ba6630-3b6e-11e9-8700-000d3a450542\" (UniqueName: \"kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\") pod \"graphite-0\" (UID: \"2dcb2656-4afb-11e9-8700-000d3a450542\") " E0320 10:33:22.480255 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope E0320 10:33:32.504488 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope I0320 10:33:32.523123 3260 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-25ba6630-3b6e-11e9-8700-000d3a450542" (UniqueName: "kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542") pod "graphite-0" (UID: "2dcb2656-4afb-11e9-8700-000d3a450542") E0320 10:33:32.526347 3260 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\"" failed. No retries permitted until 2019-03-20 10:34:04.5263106 +0000 UTC m=+32801.279682401 (durationBeforeRetry 32s). Error: "Volume not attached according to node status for volume \"pvc-25ba6630-3b6e-11e9-8700-000d3a450542\" (UniqueName: \"kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\") pod \"graphite-0\" (UID: \"2dcb2656-4afb-11e9-8700-000d3a450542\") " E0320 10:33:42.528080 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope E0320 10:33:52.551149 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope E0320 10:34:02.574950 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope I0320 10:34:04.622625 3260 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-25ba6630-3b6e-11e9-8700-000d3a450542" (UniqueName: "kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542") pod "graphite-0" (UID: "2dcb2656-4afb-11e9-8700-000d3a450542") E0320 10:34:04.628919 3260 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\"" failed. No retries permitted until 2019-03-20 10:35:08.628879 +0000 UTC m=+32865.382250901 (durationBeforeRetry 1m4s). Error: "Volume not attached according to node status for volume \"pvc-25ba6630-3b6e-11e9-8700-000d3a450542\" (UniqueName: \"kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\") pod \"graphite-0\" (UID: \"2dcb2656-4afb-11e9-8700-000d3a450542\") " W0320 10:34:09.363969 3260 kubelet_pods.go:842] Unable to retrieve pull secret v2/carbon for v2/profile-v2-prod-workers-6c74db66d4-tzb98 due to secret "carbon" not found. The image pull may not succeed. E0320 10:34:12.603522 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope E0320 10:34:22.632095 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope E0320 10:34:32.661833 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope E0320 10:34:42.689052 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope E0320 10:34:52.709259 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope E0320 10:35:02.732061 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope E0320 10:35:03.806748 3260 kubelet.go:1655] Unable to mount volumes for pod "graphite-0_metrics(2dcb2656-4afb-11e9-8700-000d3a450542)": timeout expired waiting for volumes to attach or mount for pod "metrics"/"graphite-0". list of unmounted volumes=[graphite-data]. list of unattached volumes=[graphite-config graphite-data default-token-rrp2l]; skipping pod E0320 10:35:03.806816 3260 pod_workers.go:186] Error syncing pod 2dcb2656-4afb-11e9-8700-000d3a450542 ("graphite-0_metrics(2dcb2656-4afb-11e9-8700-000d3a450542)"), skipping: timeout expired waiting for volumes to attach or mount for pod "metrics"/"graphite-0". list of unmounted volumes=[graphite-data]. list of unattached volumes=[graphite-config graphite-data default-token-rrp2l] I0320 10:35:08.721970 3260 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-25ba6630-3b6e-11e9-8700-000d3a450542" (UniqueName: "kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542") pod "graphite-0" (UID: "2dcb2656-4afb-11e9-8700-000d3a450542") E0320 10:35:08.735797 3260 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\"" failed. No retries permitted until 2019-03-20 10:37:10.7357557 +0000 UTC m=+32987.489127601 (durationBeforeRetry 2m2s). Error: "Volume not attached according to node status for volume \"pvc-25ba6630-3b6e-11e9-8700-000d3a450542\" (UniqueName: \"kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\") pod \"graphite-0\" (UID: \"2dcb2656-4afb-11e9-8700-000d3a450542\") " E0320 10:35:12.762291 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope E0320 10:35:22.786661 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope E0320 10:35:32.812393 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope W0320 10:35:37.363860 3260 kubelet_pods.go:842] Unable to retrieve pull secret v2/carbon for v2/profile-v2-prod-workers-6c74db66d4-tzb98 due to secret "carbon" not found. The image pull may not succeed. E0320 10:35:42.844320 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope E0320 10:35:52.875053 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope E0320 10:36:02.900388 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope E0320 10:36:12.927191 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope E0320 10:36:22.950467 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope E0320 10:36:32.977422 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope W0320 10:36:42.364109 3260 kubelet_pods.go:842] Unable to retrieve pull secret v2/carbon for v2/profile-v2-prod-workers-6c74db66d4-tzb98 due to secret "carbon" not found. The image pull may not succeed. E0320 10:36:43.002311 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope E0320 10:36:53.023743 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope E0320 10:37:03.050171 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope I0320 10:37:10.802390 3260 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-25ba6630-3b6e-11e9-8700-000d3a450542" (UniqueName: "kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542") pod "graphite-0" (UID: "2dcb2656-4afb-11e9-8700-000d3a450542") E0320 10:37:10.811328 3260 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\"" failed. No retries permitted until 2019-03-20 10:39:12.8112054 +0000 UTC m=+33109.564577301 (durationBeforeRetry 2m2s). Error: "Volume not attached according to node status for volume \"pvc-25ba6630-3b6e-11e9-8700-000d3a450542\" (UniqueName: \"kubernetes.io/azure-disk//subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/disks/k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542\") pod \"graphite-0\" (UID: \"2dcb2656-4afb-11e9-8700-000d3a450542\") " E0320 10:37:13.071378 3260 azure_dd.go:165] failed to list vm sizes in GetVolumeLimits, plugin.host: vmss-vm13309000174, location: westeurope

describe pod

`Name: graphite-0 Namespace: metrics Priority: 0 PriorityClassName: <none> Node: vmss-vm13309000174/10.240.6.49 Start Time: Wed, 20 Mar 2019 10:33:00 +0000 Labels: controller-revision-hash=graphite-664477488 statefulset.kubernetes.io/pod-name=graphite-0 workload.user.cattle.io/workloadselector=statefulSet-prometheus-graphite Annotations: cattle.io/timestamp=2019-03-20T09:28:15Z field.cattle.io/ports=[[{“containerPort”:8080,“dnsName”:“graphite-nodeport”,“kind”:“NodePort”,“name”:“80tcp801”,“protocol”:“TCP”,“sourcePort”:80},{“containerPort”:2003,“dnsName”:“graphite-nodeport”,"k… Status: Pending IP:
Controlled By: StatefulSet/graphite Containers: graphite: Container ID:
Image: graphiteapp/graphite-statsd:1.1.5-10 Image ID:
Ports: 8080/TCP, 2003/TCP, 2023/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Limits: memory: 500Mi Requests: memory: 500Mi Environment: <none> Mounts: /opt/graphite/conf/ from graphite-config (rw) /opt/graphite/storage/whisper from graphite-data (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-rrp2l (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: graphite-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: graphite-server-conf Optional: false graphite-data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: graphite-data ReadOnly: false default-token-rrp2l: Type: Secret (a volume populated by a Secret) SecretName: default-token-rrp2l Optional: false QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message

---

Normal Scheduled 49m default-scheduler Successfully assigned metrics/graphite-0 to vmss-vm13309000174 Warning FailedAttachVolume 3m (x23 over 49m) attachdetach-controller AttachVolume.Attach failed for volume “pvc-25ba6630-3b6e-11e9-8700-000d3a450542” : Attach volume “k8s-cluster-production-dynamic-pvc-25ba6630-3b6e-11e9-8700-000d3a450542” to instance “/subscriptions/a71140a1-1719-4ca9-8800-b03e4bb64914/resourceGroups/k8s-cluster-production/providers/Microsoft.Compute/virtualMachineScaleSets/sig-5acb9b56_standard_f4_low/virtualMachines/1552” failed with compute.VirtualMachineScaleSetVMsClient#Update: Failure sending request: StatusCode=0 – Original Error: autorest/azure: Service returned an error. Status=<nil> Code=“OperationNotAllowed” Message=“” Warning FailedMount 2m (x20 over 45m) kubelet, vmss-vm13309000174 Unable to mount volumes for pod “graphite-0_metrics(2dcb2656-4afb-11e9-8700-000d3a450542)”: timeout expired waiting for volumes to attach or mount for pod “metrics”/“graphite-0”. list of unmounted volumes=[graphite-data]. list of unattached volumes=[graphite-config graphite-data default-token-rrp2l]`