azure-key-vault-to-kubernetes: [BUG] when using aad-pod-identity, env-injector fail to pull image from ACR

Note: Make sure to check out known issues (https://github.com/sparebankenvest/azure-key-vault-to-kubernetes#known-issues) before submitting

Your question Does the env-injector use some custom mechanism to authenticate with a private docker registry to inspect the image in order to overwrite the entrypoint or cmd? I’m having some issues where the env-injector container is showing logs where it attempts to pull the image from ACR but it fails.

To Reproduce If question relates to a certain behavior, describe steps to reproduce: I’m using AKS + Azure Container Registry with a service principle authentication mechanism that allows me to spin up containers in AKS available in my private registry (ACR). The original pod itself was deployed successfully and i can verify that the AKS cluster has the necessary permission to pull images from ACR. However, I’ve noticed that not env-injector debug logs were showing up in my container so i poked around the env-injector container and noticed some error messages related to it’s inability to pull docker images from ACR

Logs Here is the command I am running to find that information: $ kubectl logs -n akv2k8s azure-key-vault-env-injector-5458b4bc9d-njxrz -c azure-key-vault-env-injector

Here are the logs indicating the problem mentioned:

time="2020-03-27T19:43:58Z" level=info msg="found pod to mutate in namespace 'argo'"
time="2020-03-27T19:43:58Z" level=info msg="found container 'api-test' to mutate"
time="2020-03-27T19:43:58Z" level=info msg="checking for env vars containing '@azurekeyvault' in container api-test"
time="2020-03-27T19:43:58Z" level=info msg="found env var: db-connection-string@azurekeyvault"
time="2020-03-27T19:43:58Z" level=info msg="did not find credentials to use with registry '****.azurecr.io' - getting default credentials"
time="2020-03-27T19:43:58Z" level=info msg="failed to read azure.json to get default credentials, error: open /etc/kubernetes/azure.json: no such file or directory"
time="2020-03-27T19:43:58Z" level=info msg="pulling docker image ***.azurecr.io/***:latest to get entrypoint and cmd, timeout is 120 seconds"
2020/03/27 19:43:59 [ERROR] admission webhook error: failed to get auto cmd, error: failed to pull docker image '****.azurecr.io/***:latest', error: Error response from daemon: Get https://****.azurecr.io/v2/***/manifests/latest: unauthorized: authentication required

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 89 (7 by maintainers)

Commits related to this issue

fixed issue #55 where auth with acs was not working — committed to SparebankenVest/azure-key-vault-to-kubernetes by torresdal 4 years ago

Most upvoted comments

looks like it works 😃 … I have not done anything really other than reinstallng injectior and my sample app. And I have done this at least 5 times yesterday 😃. Ok, now, I was using version 1.6

leonids2005 on Sep 24, 2020

@torresdal I’ve tried the second option you’ve mentioned but doesn’t seem to work.

Here are the steps I took:

I’ve created a docker registry k8s secret in the akv2k8s namespace.
I’ve patched the azure-key-vault-env-injector service account to inject the imagePullSecrets in every pod.
I’ve forced the creation of a new set of env-injector pods and verified that the imagePullSecrets information was correct.
I’ve redeployed my app and checked the env-injector pod for logs.

I got the same error I was getting before.

I think the reason why this doesn’t work is because the env-injector pod is trying to pull the image from within application code ? Therefore the imagePullSecrets that i’ve configured does not take effect on that request to the private registry.

I might just use the controller sync mechanism for now until we fix this.

I’m not sure on the effort to fix this but if you have any ideas how to easily fix this I might be able to convince some of the Go devs on our team to help you get that done.

rgbusato on Mar 29, 2020

I can confirm that the latest beta works for me as well 👍

zacharias33 on Feb 26, 2021

We’ve removed explicit support for aad-pod-identity (the podIdentitySelection option no longer exist), meaning you should just use labels, as you do with any other pod. autoInject does not exist either, as there is no longer any need for it (longer story).

But even if you set all this correctly, I think it will fail, because it will still try to use the authservice, but you don’t want in your case… I’ll look at this shortly and provide an option for not using the authservice.

Long story short: we need to provide you with an option to not use the authservice.

torresdal on Sep 25, 2020

I will try it later today. I have a question - do I need to do any extra configuration for ACR other than creating AcrPull role assignment. And maybe some extra specific config for AKS other than creating it using managed identity

leonids2005 on Sep 24, 2020

The official version with this fix is image 1.1.6 the latest helm charts are also updated.

torresdal on Sep 24, 2020

I’m sorry, it did actually work!

Here is last part of the log:

time="2020-09-24T12:19:07Z" level=debug msg="Increasing token expiration to: 60 seconds"
time="2020-09-24T12:19:07Z" level=debug msg="GET https://xxxxxxxxx.azurecr.io/v2/akv2k8s-env-test/manifests/2.0.1"
time="2020-09-24T12:19:07Z" level=debug msg="Downloading /v2/akv2k8s-env-test/blobs/sha256:9384878a479ea85902def35189c2d430a43c9a4af501783510e0916608ec8d67"
time="2020-09-24T12:19:07Z" level=debug msg="GET https://xxxxxxxx.azurecr.io/v2/akv2k8s-env-test/blobs/sha256:9384878a479ea85902def35189c2d430a43c9a4af501783510e0916608ec8d67"
time="2020-09-24T12:19:08Z" level=info msg="using 'entrypoint.sh TEST_SECRET' as arguments for env-injector"
time="2020-09-24T12:19:08Z" level=debug msg="signed arguments to prevent override"
time="2020-09-24T12:19:08Z" level=debug msg="public signing key for argument verification: \n-----BEGIN RSA PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxcgGyMq2XIJnC8vjpBfB\nwK+8vxgjJwsWzOAOOc01GMiq2vNfTtDS99cHTROsTY2XawJi44m6m6AKTl8Eent4\nwnDMRY8mvD8es7RrsQuGe/EAW5GOzgLF3CjD4xNAPxrdJ13Yjw6mP9GkGq3Lc3Rf\n+/9WH5Za/sWkORmV5GlT0YvTrtA9zFDqHsm5n+kkGATaWihWOy1UuTu3oVYMMr7j\nBL9EPqJ42EOVOyf6l18ZVRYuk8qXET5mGfpdYDj9ZzEZnEcHkFOLdmggaCVDI/fu\nFr+52WP5i/mFtdLqeejmnOTRPevxJeWUsIZPrRkTkaIpJ9YyWTWasSiExBHxp939\n1QIDAQAB\n-----END RSA PUBLIC KEY-----\n"
time="2020-09-24T12:19:08Z" level=debug msg="full exec path: /azure-keyvault/azure-keyvault-env"
time="2020-09-24T12:19:08Z" level=debug msg="container args: [entrypoint.sh TEST_SECRET]"
time="2020-09-24T12:19:08Z" level=debug msg="mounting volume 'azure-keyvault-env' to '/azure-keyvault/'"
time="2020-09-24T12:19:08Z" level=debug msg="configure init-container to use auth service"
time="2020-09-24T12:19:08Z" level=info msg="containers mutated and pod updated with init-container and volumes"

Why I thought it didn’t work was because there was ErrImagePull error some time, and I checked the events and I read it carelessly that it fetched the image from the spvest registry after failing, but that is the initContainer!

NAMESPACE             NAME                                                              READY   STATUS         RESTARTS   AGE
akv-test              akvs-secret-app-6676d76f76-v2spp                                  0/1     ErrImagePull   0          36s

and events said:

LAST SEEN   TYPE      REASON              OBJECT                                  MESSAGE
43s         Normal    Scheduled           pod/akvs-secret-app-6676d76f76-v2spp    Successfully assigned akv-test/akvs-secret-app-6676d76f76-v2spp to aks-nodepool-34239724-vmss000000
42s         Normal    Pulling             pod/akvs-secret-app-6676d76f76-v2spp    Pulling image "spvest/azure-keyvault-env:1.1.0"
41s         Normal    Pulled              pod/akvs-secret-app-6676d76f76-v2spp    Successfully pulled image "spvest/azure-keyvault-env:1.1.0"
40s         Normal    Created             pod/akvs-secret-app-6676d76f76-v2spp    Created container copy-azurekeyvault-env
40s         Normal    Started             pod/akvs-secret-app-6676d76f76-v2spp    Started container copy-azurekeyvault-env
25s         Normal    Pulling             pod/akvs-secret-app-6676d76f76-v2spp    Pulling image "xxxxxx.azurecr.io/akv2k8s-env-test:2.0.1"
25s         Warning   Failed              pod/akvs-secret-app-6676d76f76-v2spp    Failed to pull image "xxxxxx.azurecr.io/akv2k8s-env-test:2.0.1": rpc error: code = Unknown desc = Error response from daemon: Get https://xxxxxxx.azurecr.io/v2/akv2k8s-env-test/manifests/2.0.1: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.
25s         Warning   Failed              pod/akvs-secret-app-6676d76f76-v2spp    Error: ErrImagePull
10s         Normal    BackOff             pod/akvs-secret-app-6676d76f76-v2spp    Back-off pulling image "xxxxxxx.azurecr.io/akv2k8s-env-test:2.0.1"
10s         Warning   Failed              pod/akvs-secret-app-6676d76f76-v2spp    Error: ImagePullBackOff
10s         Normal    Pulled              pod/akvs-secret-app-6676d76f76-v2spp    Successfully pulled image "xxxxxxx.azurecr.io/akv2k8s-env-test:2.0.1"
43s         Normal    SuccessfulCreate    replicaset/akvs-secret-app-6676d76f76   Created pod: akvs-secret-app-6676d76f76-v2spp
45s         Normal    ScalingReplicaSet   deployment/akvs-secret-app              Scaled up replica set akvs-secret-app-6676d76f76 to 1

Great work, thank you so much!

jpohjolainen on Sep 24, 2020

and this is what I get in \etc\kubernetes\azure.json on the node

    "aadClientId": "msi",                                                                                                                                         
    "aadClientSecret": "msi",

leonids2005 on Sep 23, 2020

@VpavlovLumedic I believe this should now work in v1.1.0

torresdal on Sep 21, 2020

Thanks @pvmraghunandan. The bug is still there though, when not using imagePullSecrets, so I’m working on fixing that before releasing…

torresdal on Sep 11, 2020

Thanks for taking the time to test @pvmraghunandan. I looked through the code once more, and there is an issue indeed. When looking for credentials to use for image registries, it does not take into account using custom auth. I’ll patch this asap and get the 1.1.0 version out the door. Thanks.

torresdal on Sep 10, 2020