azure-key-vault-to-kubernetes: [BUG] starting container process caused "exec: \"/azure-keyvault/azure-keyvault-env\": stat /azure-keyvault/azure-keyvault-env: no such file or directory"

Note: Make sure to check out known issues (https://github.com/sparebankenvest/azure-key-vault-to-kubernetes#known-issues) before submitting

Describe the bug Using environment injection with either the supplied test image or my own results in the container not starting, apparently because the expected volume mounts are not actually created.

AKS with kubernetes 1.17.0

To Reproduce Steps to reproduce the behavior:

Helm installation as per manual.

apiVersion: v1
kind: Namespace
metadata:
  name: akv-test
  labels:
    azure-key-vault-env-injection: enabled
---
apiVersion: spv.no/v1alpha1
kind: AzureKeyVaultSecret
metadata:
  name: db-secret-inject
  namespace: akv-test
spec:
  vault:
    name: akvtest # name of key vault
    object:
      name: login-secret # name of the akv object
      type: secret # akv object type
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: akv2k8s-test-injection
  namespace: akv-test
  labels:
    app: akv2k8s-test-injection
spec:
  selector:
    matchLabels:
      app: akv2k8s-test-injection
  template:
    metadata:
      labels:
        app: akv2k8s-test-injection
    spec:
      containers:
      - name: akv2k8s-env-test
        image: spvest/akv2k8s-env-test
        env:
        - name: TEST_SECRET
          value: "secret-inject@azurekeyvault"
        - name: SP_SECRET
          value: "db-secret-inject@azurekeyvault"
        - name: ENV_INJECTOR_LOG_LEVEL
          value: debug

Expected behavior

SP_SECRET environment variable defined with value from AKV, inside a running container.

Logs If applicable, add logs to help explain your problem.

The env-injector pod seems to think it did its work:

time="2020-03-11T09:20:02Z" level=info msg="found pod to mutate in namespace 'akv-test'"
time="2020-03-11T09:20:02Z" level=info msg="found container 'akv2k8s-env-test' to mutate"
time="2020-03-11T09:20:02Z" level=info msg="checking for env vars containing '@azurekeyvault' in container akv2k8s-env-test"
time="2020-03-11T09:20:02Z" level=info msg="found env var: secret-inject@azurekeyvault"
time="2020-03-11T09:20:02Z" level=info msg="found env var: db-secret-inject@azurekeyvault"
time="2020-03-11T09:20:02Z" level=info msg="did not find credentials to use with registry 'spvest' - getting default credentials"
time="2020-03-11T09:20:02Z" level=info msg="registry host 'spvest' is not a acr registry"
time="2020-03-11T09:20:02Z" level=info msg="pulling docker image docker.io/spvest/akv2k8s-env-test:latest to get entrypoint and cmd, timeout is 120 seconds"
time="2020-03-11T09:20:04Z" level=info msg="docker image docker.io/spvest/akv2k8s-env-test:latest pulled successfully"
time="2020-03-11T09:20:04Z" level=info msg="inspecting container image docker.io/spvest/akv2k8s-env-test:latest, looking for entrypoint and cmd"
time="2020-03-11T09:20:04Z" level=info msg="using 'entrypoint.sh' as arguments for env-injector"
time="2020-03-11T09:20:04Z" level=info msg="containers mutated and pod updated with init-container and volumes"

But in the pod description:

Events:
  Type     Reason     Age                   From                                        Message
  ----     ------     ----                  ----                                        -------
  Normal   Scheduled  <unknown>             default-scheduler                           Successfully assigned akv-test/akv2k8s-test-injection-6c796746df-vqspz to aks-nodepool1-40643695-vmss000001
  Normal   Pulled     13m                   kubelet, aks-nodepool1-40643695-vmss000001  Container image "spvest/azure-keyvault-env:1.0.1" already present on machine
  Normal   Created    13m                   kubelet, aks-nodepool1-40643695-vmss000001  Created container copy-azurekeyvault-env
  Normal   Started    13m                   kubelet, aks-nodepool1-40643695-vmss000001  Started container copy-azurekeyvault-env
  Normal   Started    13m                   kubelet, aks-nodepool1-40643695-vmss000001  Started container akv2k8s-env-test
  Normal   Pulled     12m (x4 over 13m)     kubelet, aks-nodepool1-40643695-vmss000001  Successfully pulled image "spvest/akv2k8s-env-test"
  Normal   Created    12m (x4 over 13m)     kubelet, aks-nodepool1-40643695-vmss000001  Created container akv2k8s-env-test
  Warning  Failed     12m (x3 over 12m)     kubelet, aks-nodepool1-40643695-vmss000001  Error: failed to start container "akv2k8s-env-test": Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused "exec: \"/azure-keyvault/azure-keyvault-env\": stat /azure-keyvault/azure-keyvault-env: no such file or directory": unknown
  Normal   Pulling    11m (x5 over 13m)     kubelet, aks-nodepool1-40643695-vmss000001  Pulling image "spvest/akv2k8s-env-test"
  Warning  BackOff    3m11s (x44 over 12m)  kubelet, aks-nodepool1-40643695-vmss000001  Back-off restarting failed container

The init container does not show any error state (exit code 0). Both containers have the /azure-keyvault/ mount listed.

The logs for the init container simply say that the file was copied:

kubectl logs -n akv-test akv2k8s-test-injection-6c796746df-vqspz -c copy-azurekeyvault-env
Copying /azure-keyvault/azure-keyvault-env to /azure-keyvault/

Additional context

This did work yesterday! Nothing has changed in the AKS cluster. I have redeployed the akv2k8s and application deployments a few times to be sure.