secrets-store-csi-driver-provider-azure: aks failed to get mounted files, err: failed to list all files in target path

What steps did you take and what happened: [A clear and concise description of what the bug is.] I am only seeing this error on k8s init windows jobs. I am unsure of linux jobs are impacted as I don’t run any linux init jobs.

13m         Warning   FailedToCreateSecret   /mssql-init-sr859   failed to get mounted files, err: failed to list all files in target path c:\var\lib\kubelet\pods\a7529eb1-16eb-4b23-9674-d8fc2c7e8424\volumes\kubernetes.io~csi\secrets-store-inline\mount, err: open c:\var\lib\kubelet\pods\a7529eb1-16eb-4b23-9674-d8fc2c7e8424\volumes\kubernetes.io~csi\secrets-store-inline\mount: The system cannot find 
the path specified.
10m         Warning   FailedToCreateSecret   /solr-init-pgzvc    failed to get mounted files, err: failed to list all files in target path c:\var\lib\kubelet\pods\bd78698c-b15e-4d71-bf6d-e58f4e125f0e\volumes\kubernetes.io~csi\secrets-store-inline\mount, err: open c:\var\lib\kubelet\pods\bd78698c-b15e-4d71-bf6d-e58f4e125f0e\volumes\kubernetes.io~csi\secrets-store-inline\mount: The system cannot find 
the path specified.

This does not stop the k8s job from running and completing successfully.

I did not see this issue in 0.0.8, but I am on 0.0.9 aks 1.18.8 vmss with latest windows node image

What did you expect to happen: no warning error.

Which access mode did you use to access the Azure Key Vault instance: [e.g. Service Principal, Pod Identity, User Assigned Managed Identity, System Assigned Managed Identity] User Assigned Managed Identity

Environment:

  • Secrets Store CSI Driver version: (use the image tag): 0.0.16
  • Azure Key Vault provider version: (use the image tag):0.0.9
  • Kubernetes version: (use kubectl version and kubectl get nodes -o wide):
kubectl version

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"73ec19bdfc6008cd3ce6de96c663f70a69e2b8fc", GitTreeState:"clean", BuildDate:"2020-09-17T04:17:08Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

kubectl get nodes -o wide

NAME                                STATUS   ROLES   AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION     CONTAINER-RUNTIME
aks-nodepool1-27591730-vmss000000   Ready    agent   8h    v1.18.8   10.200.12.4     <none>        Ubuntu 18.04.5 LTS               5.4.0-1026-azure   docker://19.3.12
aks-nodepool1-27591730-vmss000001   Ready    agent   8h    v1.18.8   10.200.12.20    <none>        Ubuntu 18.04.5 LTS               5.4.0-1026-azure   docker://19.3.12
aks-npsolr-27591730-vmss000000      Ready    agent   8h    v1.18.8   10.200.12.129   <none>        Ubuntu 18.04.5 LTS               5.4.0-1026-azure   docker://19.3.12
aks-npsolr-27591730-vmss000001      Ready    agent   8h    v1.18.8   10.200.12.140   <none>        Ubuntu 18.04.5 LTS               5.4.0-1026-azure   docker://19.3.12
aks-npsolr-27591730-vmss000002      Ready    agent   8h    v1.18.8   10.200.12.151   <none>        Ubuntu 18.04.5 LTS               5.4.0-1026-azure   docker://19.3.12
aks-npzk-27591730-vmss000000        Ready    agent   8h    v1.18.8   10.200.12.162   <none>        Ubuntu 18.04.5 LTS               5.4.0-1026-azure   docker://19.3.12
aks-npzk-27591730-vmss000001        Ready    agent   8h    v1.18.8   10.200.12.173   <none>        Ubuntu 18.04.5 LTS               5.4.0-1026-azure   docker://19.3.12
aks-npzk-27591730-vmss000002        Ready    agent   8h    v1.18.8   10.200.12.184   <none>        Ubuntu 18.04.5 LTS               5.4.0-1026-azure   docker://19.3.12
aksnpwin000000                      Ready    agent   8h    v1.18.8   10.200.12.36    <none>        Windows Server 2019 Datacenter   10.0.17763.1397    docker://19.3.11
aksnpwin000001                      Ready    agent   8h    v1.18.8   10.200.12.67    <none>        Windows Server 2019 Datacenter   10.0.17763.1397    docker://19.3.11
aksnpwin000002                      Ready    agent   8h    v1.18.8   10.200.12.98    <none>        Windows Server 2019 Datacenter   10.0.17763.1397    docker://19.3.11

  • Cluster type: (e.g. AKS, aks-engine, etc): AKS

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 18 (7 by maintainers)

Most upvoted comments

We’ve released driver v0.0.17 and AKV provider 0.0.10 - https://github.com/Azure/secrets-store-csi-driver-provider-azure/releases/tag/0.0.10. With v0.0.17 we have added checks to skip rotation and reconcile for deleting pods.

Closing this issue with the release and also the primary issue for pod stuck in Terminating was caused due to windows insights. Please feel free to open a new issue for any further queries.

@aramase they are pushing the fix across all the regions and hope to have it done by end of week. css case # 120101424001816