azure-key-vault-to-kubernetes: [BUG] env-injector fails to pull image from a private registry when using imagePullSecrets

Describe the bug When a deployment specifies imagePullSecrets the env-injector is unable to mutate containers because it cannot pull the image from the private registry resulting in the pod(s) not starting

To Reproduce I am using the latest env injector (v1.1.7) but I also confirmed the issue exists in v1.0.2 I use helm to generate the yaml files , I am using the default configuration. helm template azure-key-vault-env-injector --namespace akv2k8s --output-dir ‘./yamls’ ./azure-key-vault-env-injector/

I managed to reproduce the issue with a very simple rabbitmq setup. see attached rabbitmq-test yaml rabbitmq-test.yaml.zip

1. When the rabbitmq image is pulled from the docker hub (anonymous access) everything works fine and the cookie environment variable is injected. See successful env-injector log below.

2. When I push the same rabbitmq image to our private gitlab registry and adapt the yaml accordingly by changing the image and adding the imagePullSecrets then the rabbitmq pod no longer starts. When I remove erlang-cookie@azurekeyvault in the yaml then the rabbitmq pods are starting without issue so the gitlab credentials are working fine as the image can be pulled. See unsuccessful env-injector log below.

3. When I push the rabbitmq image to an Azure Container Registry and adapt the yaml to pull from ACR then the pods are also started without issue.

4. When I pull the image from ACR and change the configuration of the env-injector to DOCKER_IMAGE_INSPECTION_USE_ACS_CREDENTIALS=false then the pods also don’t start. See another unsuccessful env-injector log below, the error is different compared to (2)

Expected behavior env-injector can mutate the containers when using imagepullsecrets , resulting in environment variables being injected

Logs

1. injector logs when pulling from docker hub and successful mutation

time="2020-10-02T09:50:26Z" level=info msg="found pod to mutate in namespace 'seam-dev-03'"
time="2020-10-02T09:50:26Z" level=info msg="found container 'rabbitmq' to mutate"
time="2020-10-02T09:50:26Z" level=info msg="checking for env vars containing '@azurekeyvault' in container rabbitmq"
time="2020-10-02T09:50:26Z" level=info msg="found env var: erlang-cookie@azurekeyvault"
time="2020-10-02T09:50:26Z" level=info msg="we are running inside azure aks, trying to get acr credentials"
time="2020-10-02T09:50:26Z" level=warning msg="no acr credentials found for "
time="2020-10-02T09:50:26Z" level=debug msg="no cmd override in kubernetes for container rabbitmq, checking docker image configuration for entrypoint and cmd for rabbitmq:3.8.8-management-alpine"
time="2020-10-02T09:50:26Z" level=debug msg="docker image inspection timeout: 20 seconds"
time="2020-10-02T09:50:26Z" level=debug msg="Trying to access \"docker.io/library/rabbitmq:3.8.8-management-alpine\""
time="2020-10-02T09:50:26Z" level=debug msg="Returning credentials from DockerAuthConfig"
time="2020-10-02T09:50:26Z" level=debug msg="Using registries.d directory /etc/containers/registries.d for sigstore configuration"
time="2020-10-02T09:50:26Z" level=debug msg=" No signature storage configuration found for docker.io/library/rabbitmq:3.8.8-management-alpine"
time="2020-10-02T09:50:26Z" level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io"
time="2020-10-02T09:50:26Z" level=debug msg="GET https://registry-1.docker.io/v2/"
time="2020-10-02T09:50:26Z" level=debug msg="Ping https://registry-1.docker.io/v2/ status 401"
time="2020-10-02T09:50:26Z" level=debug msg="GET https://auth.docker.io/token?scope=repository%3Alibrary%2Frabbitmq%3Apull&service=registry.docker.io"
time="2020-10-02T09:50:27Z" level=debug msg="GET https://registry-1.docker.io/v2/library/rabbitmq/manifests/3.8.8-management-alpine"
time="2020-10-02T09:50:27Z" level=debug msg="GET https://registry-1.docker.io/v2/library/rabbitmq/manifests/sha256:b8addf0167de3ca3f7a552760173cb3ba931172c3dc1f7723a1982644690e74a"
time="2020-10-02T09:50:28Z" level=debug msg="Downloading /v2/library/rabbitmq/blobs/sha256:5b60d37d2763a18d9d9ed14ac070be3ad3b4f129141ff25afea17ede66d4b4fb"
time="2020-10-02T09:50:28Z" level=debug msg="GET https://registry-1.docker.io/v2/library/rabbitmq/blobs/sha256:5b60d37d2763a18d9d9ed14ac070be3ad3b4f129141ff25afea17ede66d4b4fb"
time="2020-10-02T09:50:28Z" level=info msg="using 'docker-entrypoint.sh rabbitmq-server' as arguments for env-injector"
time="2020-10-02T09:50:29Z" level=debug msg="signed arguments to prevent override"
time="2020-10-02T09:50:29Z" level=debug msg="public signing key for argument verification: \n-----BEGIN RSA PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA05sWt3qKJKPvXsu06Gdz\nxBF0XKe3e/NnRsgDlP5K1ZZrjXvMecyxvEG2i462ecHnM+k6Bn3alf3+QgGq/F5v\nR5nw4L3E4bMXhOWngUYhLycYEIScbdwKFwzas9z5FpIYGaGHgpHyaNVXBUHJf3pE\nEXh3BoL3zKN9x/nRVrVDCzoemgDHBNl54tTo5FqSnuyRczB2SUAdo0OWeD30C7+P\nSQTO9BNMZZQnlCzSs3lwZRWE0JuO/Ma/igmjssrmF3ASEzI+W2MTkJ+ANKOEAHkv\nDBf1tlYuss3RPfDiu0HHYzwiY28mYGjZJypXxYtHqxfXmWFKm2ZuWdKdvJZzEMln\nZQIDAQAB\n-----END RSA PUBLIC KEY-----\n"
time="2020-10-02T09:50:29Z" level=debug msg="full exec path: /azure-keyvault/azure-keyvault-env"
time="2020-10-02T09:50:29Z" level=debug msg="container args: [docker-entrypoint.sh rabbitmq-server]"
time="2020-10-02T09:50:29Z" level=debug msg="mounting volume 'azure-keyvault-env' to '/azure-keyvault/'"
time="2020-10-02T09:50:29Z" level=debug msg="configure init-container to use auth service"
time="2020-10-02T09:50:29Z" level=info msg="containers mutated and pod updated with init-container and volumes"

2. injector logs when pulling from gitlab, mutation unsuccessful

time="2020-10-02T09:56:10Z" level=info msg="found pod to mutate in namespace 'seam-dev-03'"
time="2020-10-02T09:56:10Z" level=info msg="found container 'rabbitmq' to mutate"
time="2020-10-02T09:56:10Z" level=info msg="checking for env vars containing '@azurekeyvault' in container rabbitmq"
time="2020-10-02T09:56:10Z" level=info msg="found env var: erlang-cookie@azurekeyvault"
time="2020-10-02T09:56:10Z" level=info msg="we are running inside azure aks, trying to get acr credentials"
time="2020-10-02T09:56:10Z" level=warning msg="no acr credentials found for registry.gitlab.com"
time="2020-10-02T09:56:10Z" level=debug msg="no cmd override in kubernetes for container rabbitmq, checking docker image configuration for entrypoint and cmd for registry.gitlab.com/bynubian/docker/rabbitmq:8b0d9ec66637c444f627a54c31333284e7f8fcfa"
time="2020-10-02T09:56:10Z" level=debug msg="docker image inspection timeout: 20 seconds"
time="2020-10-02T09:56:10Z" level=debug msg="Trying to access \"registry.gitlab.com/bynubian/docker/rabbitmq:8b0d9ec66637c444f627a54c31333284e7f8fcfa\""
time="2020-10-02T09:56:10Z" level=debug msg="Returning credentials from DockerAuthConfig"
time="2020-10-02T09:56:10Z" level=debug msg="Using registries.d directory /etc/containers/registries.d for sigstore configuration"
time="2020-10-02T09:56:10Z" level=debug msg=" No signature storage configuration found for registry.gitlab.com/bynubian/docker/rabbitmq:8b0d9ec66637c444f627a54c31333284e7f8fcfa"
time="2020-10-02T09:56:10Z" level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.gitlab.com"
time="2020-10-02T09:56:10Z" level=debug msg="GET https://registry.gitlab.com/v2/"
time="2020-10-02T09:56:11Z" level=debug msg="Ping https://registry.gitlab.com/v2/ status 401"
time="2020-10-02T09:56:11Z" level=debug msg="GET https://gitlab.com/jwt/auth?scope=repository%3Abynubian%2Fdocker%2Frabbitmq%3Apull&service=container_registry"
time="2020-10-02T09:56:11Z" level=debug msg="Accessing \"registry.gitlab.com/bynubian/docker/rabbitmq:8b0d9ec66637c444f627a54c31333284e7f8fcfa\" failed: Requesting bear token: invalid status code from registry 403 (Forbidden)"
time="2020-10-02T09:56:11Z" level=error msg="failed to mutate pod, error: failed to get auto cmd, error: failed to get docker image \"docker://registry.gitlab.com/bynubian/docker/rabbitmq:8b0d9ec66637c444f627a54c31333284e7f8fcfa\": Requesting bear token: invalid status code from registry 403 (Forbidden)\ngithub.com/containers/image/v5/docker.httpResponseToError\n\t/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/errors.go:44\ngithub.com/containers/image/v5/docker.(*dockerClient).getBearerToken\n\t/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/docker_client.go:674\ngithub.com/containers/image/v5/docker.(*dockerClient).setupRequestAuth\n\t/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/docker_client.go:565\ngithub.com/containers/image/v5/docker.(*dockerClient).makeRequestToResolvedURLOnce\n\t/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/docker_client.go:512\ngithub.com/containers/image/v5/docker.(*dockerClient).makeRequestToResolvedURL\n\t/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/docker_client.go:466\ngithub.com/containers/image/v5/docker.(*dockerClient).makeRequest\n\t/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/docker_client.go:426\ngithub.com/containers/image/v5/docker.(*dockerImageSource).fetchManifest\n\t/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/docker_image_src.go:188\ngithub.com/containers/image/v5/docker.(*dockerImageSource).ensureManifestIsLoaded\n\t/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/docker_image_src.go:221\ngithub.com/containers/image/v5/docker.newImageSourceAttempt\n\t/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/docker_image_src.go:126\ngithub.com/containers/image/v5/docker.newImageSource\n\t/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/docker_image_src.go:69\ngithub.com/containers/image/v5/docker.newImage\n\t/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/docker_image.go:27\ngithub.com/containers/image/v5/docker.dockerReference.NewImage\n\t/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/docker_transport.go:138\nmain.(*imageOptions).getConfigFromManifest\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/registry.go:109\nmain.getContainerCmd\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/registry.go:54\nmain.mutateContainers\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:129\nmain.mutatePodSpec\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:254\nmain.vaultSecretsMutator\n\t/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/main.go:137\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.MutatorFunc.Mutate\n\t/go/pkg/mod/github.com/slok/kubewebhook@v0.4.0/pkg/webhook/mutating/mutator.go:25\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.(*staticWebhook).mutatingAdmissionReview\n\t/go/pkg/mod/github.com/slok/kubewebhook@v0.4.0/pkg/webhook/mutating/webhook.go:131\ngithub.com/slok/kubewebhook/pkg/webhook/mutating.(*staticWebhook).Review\n\t/go/pkg/mod/github.com/slok/kubewebhook@v0.4.0/pkg/webhook/mutating/webhook.go:123\ngithub.com/slok/kubewebhook/pkg/webhook/internal/instrumenting.(*Webhook).Review\n\t/go/pkg/mod/github.com/slok/kubewebhook@v0.4.0/pkg/webhook/internal/instrumenting/instrumenting.go:42\ngithub.com/slok/kubewebhook/pkg/http.HandlerFor.func1\n\t/go/pkg/mod/github.com/slok/kubewebhook@v0.4.0/pkg/http/handler.go:64\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\t/go/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2802\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1890\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"
2020/10/02 09:56:11 [ERROR] admission webhook error: failed to get auto cmd, error: failed to get docker image "docker://registry.gitlab.com/bynubian/docker/rabbitmq:8b0d9ec66637c444f627a54c31333284e7f8fcfa": Requesting bear token: invalid status code from registry 403 (Forbidden)
github.com/containers/image/v5/docker.httpResponseToError
	/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/errors.go:44
github.com/containers/image/v5/docker.(*dockerClient).getBearerToken
	/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/docker_client.go:674
github.com/containers/image/v5/docker.(*dockerClient).setupRequestAuth
	/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/docker_client.go:565
github.com/containers/image/v5/docker.(*dockerClient).makeRequestToResolvedURLOnce
	/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/docker_client.go:512
github.com/containers/image/v5/docker.(*dockerClient).makeRequestToResolvedURL
	/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/docker_client.go:466
github.com/containers/image/v5/docker.(*dockerClient).makeRequest
	/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/docker_client.go:426
github.com/containers/image/v5/docker.(*dockerImageSource).fetchManifest
	/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/docker_image_src.go:188
github.com/containers/image/v5/docker.(*dockerImageSource).ensureManifestIsLoaded
	/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/docker_image_src.go:221
github.com/containers/image/v5/docker.newImageSourceAttempt
	/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/docker_image_src.go:126
github.com/containers/image/v5/docker.newImageSource
	/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/docker_image_src.go:69
github.com/containers/image/v5/docker.newImage
	/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/docker_image.go:27
github.com/containers/image/v5/docker.dockerReference.NewImage
	/go/pkg/mod/github.com/containers/image/v5@v5.3.0/docker/docker_transport.go:138
main.(*imageOptions).getConfigFromManifest
	/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/registry.go:109
main.getContainerCmd
	/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/registry.go:54
main.mutateContainers
	/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:129
main.mutatePodSpec
	/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:254
main.vaultSecretsMutator
	/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/main.go:137
github.com/slok/kubewebhook/pkg/webhook/mutating.MutatorFunc.Mutate
	/go/pkg/mod/github.com/slok/kubewebhook@v0.4.0/pkg/webhook/mutating/mutator.go:25
github.com/slok/kubewebhook/pkg/webhook/mutating.(*staticWebhook).mutatingAdmissionReview
	/go/pkg/mod/github.com/slok/kubewebhook@v0.4.0/pkg/webhook/mutating/webhook.go:131
github.com/slok/kubewebhook/pkg/webhook/mutating.(*staticWebhook).Review
	/go/pkg/mod/github.com/slok/kubewebhook@v0.4.0/pkg/webhook/mutating/webhook.go:123
github.com/slok/kubewebhook/pkg/webhook/internal/instrumenting.(*Webhook).Review
	/go/pkg/mod/github.com/slok/kubewebhook@v0.4.0/pkg/webhook/internal/instrumenting/instrumenting.go:42
github.com/slok/kubewebhook/pkg/http.HandlerFor.func1
	/go/pkg/mod/github.com/slok/kubewebhook@v0.4.0/pkg/http/handler.go:64
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2007
github.com/gorilla/mux.(*Router).ServeHTTP
	/go/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210
net/http.serverHandler.ServeHTTP
	/usr/local/go/src/net/http/server.go:2802
net/http.(*conn).serve
	/usr/local/go/src/net/http/server.go:1890
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1357
Logs from Oct 2, 2020 to Oct 2, 2020 UTC

3. injector logs when pulling from ACR with DOCKER_IMAGE_INSPECTION_USE_ACS_CREDENTIALS=true

time="2020-10-02T10:03:47Z" level=info msg="found pod to mutate in namespace 'seam-dev-03'"
time="2020-10-02T10:03:47Z" level=info msg="found container 'rabbitmq' to mutate"
time="2020-10-02T10:03:47Z" level=info msg="checking for env vars containing '@azurekeyvault' in container rabbitmq"
time="2020-10-02T10:03:47Z" level=info msg="found env var: erlang-cookie@azurekeyvault"
time="2020-10-02T10:03:47Z" level=info msg="we are running inside azure aks, trying to get acr credentials"
time="2020-10-02T10:03:47Z" level=debug msg="no cmd override in kubernetes for container rabbitmq, checking docker image configuration for entrypoint and cmd for bynubiandockerregistry.azurecr.io/rabbitmq:8b0d9ec66637c444f627a54c31333284e7f8fcfa"
time="2020-10-02T10:03:47Z" level=debug msg="docker image inspection timeout: 20 seconds"
time="2020-10-02T10:03:47Z" level=debug msg="Trying to access \"bynubiandockerregistry.azurecr.io/rabbitmq:8b0d9ec66637c444f627a54c31333284e7f8fcfa\""
time="2020-10-02T10:03:47Z" level=debug msg="Returning credentials from DockerAuthConfig"
time="2020-10-02T10:03:47Z" level=debug msg="Using registries.d directory /etc/containers/registries.d for sigstore configuration"
time="2020-10-02T10:03:47Z" level=debug msg=" No signature storage configuration found for bynubiandockerregistry.azurecr.io/rabbitmq:8b0d9ec66637c444f627a54c31333284e7f8fcfa"
time="2020-10-02T10:03:47Z" level=debug msg="Looking for TLS certificates and private keys in /etc/docker/certs.d/bynubiandockerregistry.azurecr.io"
time="2020-10-02T10:03:47Z" level=debug msg="GET https://bynubiandockerregistry.azurecr.io/v2/"
time="2020-10-02T10:03:47Z" level=debug msg="Ping https://bynubiandockerregistry.azurecr.io/v2/ status 401"
time="2020-10-02T10:03:47Z" level=debug msg="GET https://bynubiandockerregistry.azurecr.io/oauth2/token?account=786f6893-8b13-4bce-b586-8b9a612b9461&scope=repository%3Arabbitmq%3Apull&service=bynubiandockerregistry.azurecr.io"
time="2020-10-02T10:03:48Z" level=debug msg="Increasing token expiration to: 60 seconds"
time="2020-10-02T10:03:48Z" level=debug msg="GET https://bynubiandockerregistry.azurecr.io/v2/rabbitmq/manifests/8b0d9ec66637c444f627a54c31333284e7f8fcfa"
time="2020-10-02T10:03:48Z" level=debug msg="Downloading /v2/rabbitmq/blobs/sha256:5b60d37d2763a18d9d9ed14ac070be3ad3b4f129141ff25afea17ede66d4b4fb"
time="2020-10-02T10:03:48Z" level=debug msg="GET https://bynubiandockerregistry.azurecr.io/v2/rabbitmq/blobs/sha256:5b60d37d2763a18d9d9ed14ac070be3ad3b4f129141ff25afea17ede66d4b4fb"
time="2020-10-02T10:03:48Z" level=info msg="using 'docker-entrypoint.sh rabbitmq-server' as arguments for env-injector"
time="2020-10-02T10:03:49Z" level=debug msg="signed arguments to prevent override"
time="2020-10-02T10:03:49Z" level=debug msg="public signing key for argument verification: \n-----BEGIN RSA PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAubytL/s5833iZ2Mk6DBT\n24W6U1u0I9vZqEI1gnLdzl2AQjY62VBqNs6ZhhGnrRrK2jp+F6P+sIGfat6/JpYG\n1TehcQyWlbUjAR5U42VEf+K14hqkoaqOve2mGIQz42Zr2C46rG4nmq87cSbQDZ5D\n3pMRvYeI5fhWRNgjDjpNHq6FR5AGfXGEgixERr+s0ye9v7T1tLwitVxzWlS6d3cg\nliVauhl8BP+kaUqS0Oyl68OoM5TClAUAPpY38YfFO+NE7amH9NbkrQG8ptFdz5sX\nNEaBywNez47RxeOZG47/OQeELQJ/8S3DECxMV8SW72poNY3x+qwiofU2miSAVIV8\nawIDAQAB\n-----END RSA PUBLIC KEY-----\n"
time="2020-10-02T10:03:49Z" level=debug msg="full exec path: /azure-keyvault/azure-keyvault-env"
time="2020-10-02T10:03:49Z" level=debug msg="container args: [docker-entrypoint.sh rabbitmq-server]"
time="2020-10-02T10:03:49Z" level=debug msg="mounting volume 'azure-keyvault-env' to '/azure-keyvault/'"
time="2020-10-02T10:03:49Z" level=debug msg="configure init-container to use auth service"
time="2020-10-02T10:03:49Z" level=info msg="containers mutated and pod updated with init-container and volumes"

4. injector logs when pulling from ACR with DOCKER_IMAGE_INSPECTION_USE_ACS_CREDENTIALS=false

time="2020-10-02T11:25:38Z" level=info msg="found pod to mutate in namespace 'seam-dev-03'"
time="2020-10-02T11:25:38Z" level=info msg="found container 'rabbitmq' to mutate"
time="2020-10-02T11:25:38Z" level=info msg="checking for env vars containing '@azurekeyvault' in container rabbitmq"
time="2020-10-02T11:25:38Z" level=info msg="found env var: erlang-cookie@azurekeyvault"
time="2020-10-02T11:25:38Z" level=debug msg="not trying to get acr credentials, as we are not on aks or configured to not use aks credentials with acr"
time="2020-10-02T11:25:38Z" level=debug msg="no cmd override in kubernetes for container rabbitmq, checking docker image configuration for entrypoint and cmd for bynubiandockerregistry.azurecr.io/rabbitmq:8b0d9ec66637c444f627a54c31333284e7f8fcfa"
time="2020-10-02T11:25:38Z" level=warning msg="no credentials provided/found to access remote docker image configuration for bynubiandockerregistry.azurecr.io/rabbitmq:8b0d9ec66637c444f627a54c31333284e7f8fcfa - going anonymous"
2020/10/02 11:25:38 http: panic serving 10.244.0.4:54596: runtime error: invalid memory address or nil pointer dereference
goroutine 171 [running]:
net/http.(*conn).serve.func1(0xc00012edc0)
	/usr/local/go/src/net/http/server.go:1767 +0x139
panic(0x151e540, 0x23b5fe0)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
main.getContainerCmd(0xc0002835c8, 0x8, 0xc00008eae0, 0x53, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/registry.go:51 +0x1ae
main.mutateContainers(0xc000557540, 0x1, 0x1, 0xc0005e9ef0, 0xc000590100, 0x0, 0x0)
	/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:129 +0x5f9
main.mutatePodSpec(0xc0000a1c00, 0x4, 0x170a37e)
	/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/pod.go:254 +0x1a2
main.vaultSecretsMutator(0x1933ee0, 0xc0005fd8f0, 0x196c020, 0xc0000a1c00, 0xc0002d5740, 0xc00f31, 0xc000104320)
	/go/src/github.com/SparebankenVest/azure-key-vault-to-kubernetes/cmd/azure-keyvault-secrets-webhook/main.go:137 +0x15a
github.com/slok/kubewebhook/pkg/webhook/mutating.MutatorFunc.Mutate(0x1790ed8, 0x1933ee0, 0xc0005fd8f0, 0x196c020, 0xc0000a1c00, 0x0, 0x408deb, 0xc000030000)
	/go/pkg/mod/github.com/slok/kubewebhook@v0.4.0/pkg/webhook/mutating/mutator.go:25 +0x4e
github.com/slok/kubewebhook/pkg/webhook/mutating.(*staticWebhook).mutatingAdmissionReview(0xc000335f20, 0x1933ee0, 0xc0005fd8f0, 0xc0005fd7a0, 0x196c020, 0xc0000a1880, 0x196c020, 0xc0000a1c00, 0xc0000a1880)
	/go/pkg/mod/github.com/slok/kubewebhook@v0.4.0/pkg/webhook/mutating/webhook.go:131 +0x9d
github.com/slok/kubewebhook/pkg/webhook/mutating.(*staticWebhook).Review(0xc000335f20, 0x1933ee0, 0xc0005fd8f0, 0xc0005fd7a0, 0x16616c0)
	/go/pkg/mod/github.com/slok/kubewebhook@v0.4.0/pkg/webhook/mutating/webhook.go:123 +0x3b0
github.com/slok/kubewebhook/pkg/webhook/internal/instrumenting.(*Webhook).Review(0xc0000bc4b0, 0x1933ee0, 0xc0005fd8c0, 0xc0005fd7a0, 0x0)
	/go/pkg/mod/github.com/slok/kubewebhook@v0.4.0/pkg/webhook/internal/instrumenting/instrumenting.go:42 +0x275
github.com/slok/kubewebhook/pkg/http.HandlerFor.func1(0x192dd60, 0xc0005e0460, 0xc000337500)
	/go/pkg/mod/github.com/slok/kubewebhook@v0.4.0/pkg/http/handler.go:64 +0x1e7
net/http.HandlerFunc.ServeHTTP(0xc0003437a0, 0x192dd60, 0xc0005e0460, 0xc000337500)
	/usr/local/go/src/net/http/server.go:2007 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0003ec300, 0x192dd60, 0xc0005e0460, 0xc000337300)
	/go/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0xe2
net/http.serverHandler.ServeHTTP(0xc0001542a0, 0x192dd60, 0xc0005e0460, 0xc000337300)
	/usr/local/go/src/net/http/server.go:2802 +0xa4
net/http.(*conn).serve(0xc00012edc0, 0x1933e20, 0xc00036d780)
	/usr/local/go/src/net/http/server.go:1890 +0x875
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2927 +0x38e

Additional context Add any other context about the problem here.