kubernetes: CredentialProvider fails to read environment variable passed into the CredentialProviderConfig file
What happened:
I am attempting to pull an image from ECR using the new kubelet image credential provider feature which is an alpha features in Kubernetes v1.20. My CredentialProviderConfig file includes the environment variable AWS_PROFILE which resides in the default (~/.aws) directory. The profile references a credential_process which allows me to execute a binary that returns a set of temporary AWS credentials. When I try running a pod with a container from ECR, I see the following errors in the kubelet logs:
Jun 09 21:47:05 ecr-kubelet-worker kubelet[225990]: E0609 21:47:05.229849 225990 plugin.go:198] Failed getting credential from external registry credential provider: error execing credential provider plugin ecr-credential-provider for image 820537372947.dkr.ecr.us-east-1.amazonaws.com/cats: exit status 1
Jun 09 21:47:05 ecr-kubelet-worker kubelet[225990]: E0609 21:47:05.597552 225990 remote_image.go:113] PullImage "820537372947.dkr.ecr.us-east-1.amazonaws.com/cats:v2" from image service failed: rpc error: code = Unknown desc = failed to pull and unpack image "820537372947.dkr.ecr.us-east-1.amazonaws.com/cats:v2": failed to resolve reference "820537372947.dkr.ecr.us-east-1.amazonaws.com/cats:v2": unexpected status code [manifests v2]: 401 Unauthorized
However, when I type the following into the command line on the kubelet:
echo '{"kind": "CredentialProviderRequest", "apiVersion": "credentialprovider.kubelet.k8s.io/v1alpha1", "image": "820537372947.dkr.ecr.us-east-1.amazonaws.com/cats:v2"}' | AWS_PROFILE=default /usr/local/kubelet/bin/ecr-credential-provider get-credentials
I get the following response:
{"kind":"CredentialProviderResponse","apiVersion":"credentialprovider.kubelet.k8s.io/v1alpha1","cacheKeyType":"Registry","cacheDuration":"21.6µs","auth":{"820537372947.dkr.ecr.us-east-1.amazonaws.com":{"username":"AWS","password":"eyJwYXlsb2FkIjoiak1RV0pidXpzVnRDd28wNHExdmlScmlwaVRLd3BOc25hejFlUktxVDlBOWZES0FVTjlsbkozSGl0TGh4OHhnejViU0xrRncxNGtaU3htd2R6Rks3MHNCd2FRL0M3eWdtSlVueGNrZGI0eWFjZUhIQ1NSbVZyMjNHam9VOE1yTjJadDdNUnprK0c0RzVWdmYrMVhjODdKdHhFNEZOTWU1dytoSjFRZXYydTlaNXBCVExhTHdaRDNxT0t4dk1aTi9iK0E5Vys2ajI0L1kxUGxER2wybnpoSWxDdFV3aXBxTDhaTS9MdzFwT1RmYmY1eXp6OFIxVDd0MUZvbER3SWVKNUNzQkZBQWZJanNITmpES2tTL3dYdXVlMDBOZGZNeVo2UHlPMjlVanBLdGdqMU8vZTBwZ0J4SzRGelpRVldROHE5Wk1paWtFL3BxRXMvZzN0WXNJcDhTalR6RkZqTys4UVB4WVdFZ2ZrVDFRcjZnbmtrOXZLRkdOSGd6SEV3Z0JMNUJQeDFReXpTSGJyNmlPMDd2Sjh1QzE3eHpITEZwcmpuR3o3T3AyQ1U3SFYydjVwSjdlNFM1d0Qrenljbk5NZFVabjBsQ2dzdlBpUGtuYkV5Rk1lZ050Y3dmM3JKdnNkSm5FZ3IrVGowTTlsdktTWHJBL3hGVW9vZXFQRFhQY2t4SlN6eXM2dkYvZGdXbWJZcXJPTW1jdnBaZ3VpUVo3aTF0NnZncEVIcUQ1RzIzbyt2UnlzQ1FjOElTNVRHVk1peDhGdXp1UjkyT0djbnhIVm85U0gzdWhVL096clZKdWJQQTdFcGJHTjJCM2lFUTFsaHBGeXdvdGtyNit2dFhvU01GSFBFeXFKQ2syL0VWTmphWi9Vb05ZS3JrL3hZYmJPS29UQmlpNUFrTkZDcE9IU1BqM1JVTHpQcXhJVCtPM2tkZ3ZwZ0RMSTI1c3AvYVVUV0Fpd21KYm5BZTFTR1RrdTcyOUk1cVY0bkxVdWtzK2Y0OEJqUlNXUlI0YWlqZnIxbFpLczBLRnlFTk16WkJvbGdBYWZMV01NNnEwaXlMd1Nzc1pJK2xPYmgxU3JUa3IwRFpYRDR4N05QQlN4RTByU1FmSDdzR1M3Y3VLbG85cUUrVDBMSGhxbE0velU1eDU2aTN5MkhZYnV5RDRlSHVvd1F5dVh6UTFRSXRhZDF3Q0U1blJCWjlOU1o1VDE1R1ZheklDcVFCbmFIaFl4WnZqemZuVm1vU1JrL0d1LzFESFU1c2c5U3Z0SXBkL0hXeFkxNXBTQk9nUHNaK2c5WTdOSXAyYXRHNEg0Qmh0eU40T215bm5VaGFOQjZPZU0xV3l3VFY1dXh5cHhINXk3STJwYkE4MDVMR1U4cGNzMGNvTU5Kbmdpam5aRGhzVDRhMisrS3F5d0VPaVpkZ29JZ0ZPdnFwWnIycEVuZlo2MUlEN2pjbjgwR0tMUExvalEzVGV4bjhtVFNINjhzS1l5ZHlyZGJ0MDg0bXZUbjEvNE1PeW9yOUVYeU9qMzY4UzZ2Sno1YmQ4dSsxc1Fva2lBb3FkNkIyWndLYm1QSWx4T0h6bVl1a3N6UGxnWDA1NFNQVUNNSGJJTjJacncxTmJIb3V1UXU1WFVjeGZPR2diK0g1bnBIL1VvTzJFdmR2T29TOWNlczBjMkNDU2hKbHY4MVE4ZUhGRExGRS9tajJaVkF6aFN1RXhsbTcvazlHaVJ0bW9JS3VISFBzRUhDVTRsQTBWVk1HNG44SFpEY1A3bWNrTTRCTS9ibmZTVXNabUc0ZjFEK29WdmpGQS9oWTQ3R0JrQmpVNGJ3OFRoeXprOXFxL2MxbE9tWDc4QllwUDRuMmdKbGd5M0wzcnFTMFBMc2xkNHZzZUFFSWRML29XdjlqZ2FWYjM4bHJuaEdHeTFXc09hOVJDWFMvWTRrc25vRGtnaHVmd1hGcHQveWE0d1VMcUtRL1d3bTZyZ2R0TEdGb3dPbERPOHY1cnZZWDY4WVNOSzFrcEJTcXZoak5haG5ZQU1pNEFrOXVGVFVqaDRDQmFLVTg4OFVYdUFBNERNbktMWEk0a29HK3pGK2ZVYVljOW52Vmo2dDZVdzFaQjFKTm5vNG00b1ZIUlo4U1E5c2g0Z2U3bFd4b2FiZVZxWDE0N0k4T0orZDV1L2g1MHdmWkRQT3AzZXViKzJzR28wZEl3SE5jM1lxREpBd0szSmltdFozakVJYldodmJzWUZUZzlFMkk5OUJqQ2ZnSWx6c092eXlxMXV2RW9DZ0RMN3ZySWI4amFaQWtsSDJWelQ5RUNzU3REc2J2K3ZuaGg4K2VaK2hnRzVOQXV2ZTRWL2NMVjZOL01KbzVadU1vMEg0THZyTnVRVDNXVlIzZjBqYnZlVmMrT2tyZWQxdEFOczdCUGJFdUlCZE5VaURMSmNpWkFJb3J6aGkvaDZxU3EzUE9Jc1pSencvczVBOCt0NERNQlptYXRsbDFZekJYV2EvN3pxbGVpZWlod01XU2ZqWUpDOTRVa1l1c3hkbjZCclJWVzhKNDdscHNqUVlNbG96V01raWRPalpIWUJqc0hVRXpVSytFdTdyQjQ2eDF6U2RkYSs5a3hZY0tXS1UxOUtKWTVQUTVUd1dVRXZKbXJna3dwZjdjZWRhTzFtWUwvZEpMK2ZMOFA4M09jMDJxSXJ0MzFRQUZRNGRmOFNkcmxCQVp4MExCQVdtTU0vd0RPTTJMNXlFckI0cVNTRXEvbHo4Rnc5RWFFTll3UW93N2hDQzFIRXZJSzZUWTQ1YXBZZU55clZ6UzFUUTJjSkVRWi9ZUGFlUVNFZVE1UEVnRkxpQ3RmdEhnPT0iLCJkYXRha2V5IjoiQVFFQkFIaHdtMFlhSVNKZVJ0Sm01bjFHNnVxZWVrWHVvWFhQZTVVRmNlOVJxOC8xNHdBQUFINHdmQVlKS29aSWh2Y05BUWNHb0c4d2JRSUJBREJvQmdrcWhraUc5dzBCQndFd0hnWUpZSVpJQVdVREJBRXVNQkVFRExzbWJPRldpeXcxbkhEVEVnSUJFSUE3RFVyUmcwd1ZWRlB4OFpGbkdQdkNGTVUwM21zQjZNTVZESUg5TnJLejN3Uy84L0NFZFQydy9RbEo4S1piaG1YNmlwdTl6RmNZdEQvOGFhRT0iLCJ2ZXJzaW9uIjoiMiIsInR5cGUiOiJEQVRBX0tFWSIsImV4cGlyYXRpb24iOjE2MjMzMTg2MTl9"}}}
It doesn’t appear the environment variables in my Config are being passed to the Kubelet when it executes the ecr-credential-provider binary.
Below is Config file:
kind: CredentialProviderConfig
apiVersion: kubelet.config.k8s.io/v1alpha1
providers:
- name: ecr-credential-provider
matchImages:
- "*.dkr.ecr.*.amazonaws.com"
defaultCacheDuration: "12h"
apiVersion: credentialprovider.kubelet.k8s.io/v1alpha1
args:
- get-credentials
env:
- name: "AWS_PROFILE"
value: "default"
What you expected to happen:
I expect for the Kubelet to pass the environment variables in the Config file when it executes the ecr-credential-provider binary.
How to reproduce it (as minimally and precisely as possible):
I am using KinD to test this feature. Here is my config:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
runtimeConfig:
"api/alpha": "true"
name: ecr-kubelet
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: ClusterConfiguration
apiServer:
extraArgs:
feature-gates: 'KubeletCredentialProviders=true'
image: kindest/node:v1.20.7@sha256:e645428988191fc824529fd0bb5c94244c12401cf5f5ea3bd875eb0a787f0fe9
- role: worker
image: kindest/node:v1.20.7@sha256:e645428988191fc824529fd0bb5c94244c12401cf5f5ea3bd875eb0a787f0fe9
extraMounts:
- hostPath: /Users/jicowan/go/bin
containerPath: /usr/local/kubelet/bin
- hostPath: /Users/jicowan/conf
containerPath: /usr/local/kubelet/config
kubeadmConfigPatches:
- |
kind: JoinConfiguration
nodeRegistration:
kubeletExtraArgs:
feature-gates: 'KubeletCredentialProviders=true'
image-credential-provider-config: '/usr/local/kubelet/config/ecr.conf'
image-credential-provider-bin-dir: '/usr/local/kubelet/bin/'
Anything else we need to know?:
No
Environment:
- Kubernetes version (use
kubectl version): 1.20 - Cloud provider or hardware configuration: None
- OS (e.g:
cat /etc/os-release): Ubuntu 20.10 - Kernel (e.g.
uname -a): 5.10 - Install tools: KinD
- Network plugin and version (if this is a network-related bug):
- Others:
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 48 (28 by maintainers)
@n4j : I am facing the similar issue on my bare metal cluster
This is my config
Can you help. It works when I try to execute the credentials binary from local but doesn’t work in the pod. Can you pls guide me
@n4j Thanks for looking into the issue, you are right the spec does says that. Feel free to fix it https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/config/types.go#L550-#L554 /triage accepted
@n4j sure thing.
@jicowan After several rounds of troubleshooting I was able to get
ecr-credential-providerworking in my KinD environment.My credential loading setup is
ecr-credential-provider<->cognito<->AWS Cognitoand below is my configuration of credential processInitially, I got the same error as yours, then I added couple of custom log lines to print env vars in
ecr-credential-providerin/tmpdirectory and I could verify that the env vars specified in the credentials spec were indeed being passed.What I noticed in my KinD setup is that AWS SDK was logging an error because it couldn’t find any “providers”, though I could verify in the
kind-workercontainer that/root/.aws/configfile and/root/.aws/credentialswere indeed present.I altered my env vars section of the
CredentialProviderConfigas below and it worked for meApparently, what I noticed is
ecr-credential-provideris being passed ONLY the env vars which are present in the spec and everything else was missing! Or at least this is the behaviour in my local cluster setup + KinD cluster.Can you please add
HOMEand other env vars necessary for the correct execution of AWS SDK and see if it works for you?If this doesn’t work, then I’ll send you the modified version of the
ecr-credential-providerwhich puts some additional logs to troubleshoot this issue further.@n4j Adding the PATH did not work for me.
I’m happy to send you the cognito binary with a username/password that you can use for testing. I have only tested this with KinD, but I don’t think that is the issue here since it fails with exit code when when I try using
exec.Env().@jicowan Yes, I am planning to setup Cognito Identity pool in my AWS Account.
I included my KinD configuration in the beginning of this thread. Can you use that @n4j? The
cognitobinary has to be built from source, https://github.com/jicowan/cognito (or I can send it to you along with a temp username/password). There is an accompanying blog post for the binary at https://jicowan.medium.com/pulling-docker-images-from-ecr-private-registries-with-a-cognito-user-pool-identity-95f4c233fc4a.The flow @adisky is as @n4j describes above. The
cognitobinary returns a json blob to the calling application that looks like this:During my testing (without the kubelet) I am able to get the proper response for the kubelet when referencing the AWS profile using
os.Getenvor by settingexec.Envtoos.Environ(). However, when I try to pass the AWS profile in using exec.Env(“AWS_PROFILE=default”) I getexit status 1and no further information. I have a feeling thatexecdoesn’t properly chain or pass information back to the calling application.@jicowan isn’t it the issue of ecr-credential-provider? This is the job of ecr-credential-provider to fetch credentials and return result in expected format to kubelet.
When I switch to a profile with credential process I get the following error:
@n4j kubelet registers intree credential providers while startup, you need not to explicitly add any!! https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/plugins_providers.go#L23-#L25
And I recently tried GCP credential provider, It does fall back to intree GCP credential provider after getting failed response from the one i set up.
@n4j can you check kubelet logs for
Failed getting credential from external registry credential provider, It might be possible that is falling back to in-tree aws credential provider.This is the binary I’m calling with credential_process, https://github.com/jicowan/cognito.
@n4j i am using credential_process, https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sourcing-external.html in my profile. I am also using KinD for testing, but I don’t think that should matter. I don’t have a credentials file. I only have a config file.
Yes @n4j, it is not passing the Envs to the spawned process. The code for ecr-credential-provider is in the AWS cloud controller repository, https://github.com/kubernetes/cloud-provider-aws/blob/master/cmd/ecr-credential-provider/main.go.