amazon-ecr-credential-helper: Doesn't work with AWS SSO

Seconding (Nth-ing?) the request for a homebrew release with this change included 🙏

thanks @awilkins sad

anyone here able to kick off a release and get it on brew?

Same problem here, setting AWS_PROFILE= doesn’t work aswell. I tried many different combinations and the credential helper did not work. no basic auth credentials

Let us take a look this week.

This issue is becoming more important with the release of the latest version of docker v2.4.0. There seems to be a limit of 2500 characters on the auth token that makes the aws ecr get-login-password method not working (check here aws/aws-cli#5636).

It’s not a solution in all situations but for those looking for a workaround for trying to push from a local computer the following might work:

• Navigate to https://your-company.awsapps.com/start
• Under the desired account select “Command line or programmatic access”
• Copy the AWS environment vars for the desired operating system
• Paste them in a console
• Amazon-ecr-credential-helper will now work (at least it does for me 😁)

@gondalez thanks for the interest in the project and the feedback. As others have noted building from main will help the immediate pain. That said let me bring this up with the team to see about an updated release with these fixes.

This command works

❯ aws ecr get-login-password --region ap-east-1 --profile myprofile | docker login --username AWS --password-stdin XXX.dkr.ecr.ap-east-1.amazonaws.com
❯ docker push XXX.dkr.ecr.ap-east-1.amazonaws.com/my-ecr-repo:latest
The push refers to repository [XXX.dkr.ecr.ap-east-1.amazonaws.com/my-ecr-repo]
370e6XXXXXXX: Pushed 

latest: digest: sha256:XXXXXXXXXX size: 1987

But this don’t, is there anyone has similar problem?

❯ AWS_PROFILE=myprofile docker push XXX.dkr.ecr.ap-east-1.amazonaws.com/my-ecr-repo:latest
The push refers to repository [XXX.dkr.ecr.ap-east-1.amazonaws.com/my-ecr-repo:latest
370e6XXXXXXX: Preparing 
no basic auth credentials
❯ cat ~/.ecr/log/ecr-login.log
time="2023-02-02T10:48:11+08:00" level=debug msg="Could not fetch credentials for cache prefix, disabling cache" error="the SSO session has expired or is invalid: open /Users/me/.aws/sso/cache/XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.json: no such file or directory"
time="2023-02-02T10:48:11+08:00" level=debug msg="Retrieving credentials" region=ap-east-1 registry=XXX serverURL=XXX.dkr.ecr.ap-east-1.amazonaws.com service=ecr
time="2023-02-02T10:48:11+08:00" level=debug msg="Calling ECR.GetAuthorizationToken" registry=XXX
time="2023-02-02T10:48:11+08:00" level=error msg="Error retrieving credentials" error="ecr: Failed to get authorization token: operation error ECR: GetAuthorizationToken, failed to sign request: failed to retrieve credentials: the SSO session has expired or is invalid: open /Users/me/.aws/sso/cache/XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.json: no such file or directory"

For those that need to support many teams with this, I’ve created a temporary workaround credential helper. https://github.com/dougrday/docker-credential-plaintext

It’s working with our teams with AWS SSO, but it’s definitely not the “secure” solution. Our tokens only live for 4 hours though, so the security window is fairly small.

I’d definitely prefer that other credential helpers step up and resolve the issue in a more secure fashion.

This issue is becoming more important with the release of the latest version of docker v2.4.0. There seems to be a limit of 2500 characters on the auth token that makes the aws ecr get-login-password method not working (check here https://github.com/aws/aws-cli/issues/5636).

FYI, SSO-related issues seem to be gone in a new binary version (077b4a9), which can be built manually from the sources. Not sure why new releases are not posted for such a long time already

Thanks for the heads-up @vtatarin. I’ve had a few issues with SSO bugs in my team.

@samuelkarp @austinvazquez I noticed you have changed CHANGELOG.md in the past… any insights as to why there are master commits but no releases past 0.6.0 for ?

I’d love to see the newer version with bugfixes released. I assume that would allow it to make its way into a future Docker Desktop release as well for ultimate 😌

This is what I see locally; my credentials helper is packaged with with docker and pinned at 0.6.0:

» which docker-credential-ecr-login
/usr/local/bin/docker-credential-ecr-login

» ls -lah /usr/local/bin/docker-credential-ecr-login

lrwxr-xr-x  1 root  wheel    75B  1 Dec  2021 /usr/local/bin/docker-credential-ecr-login -> /Applications/Docker.app/Contents/Resources/bin/docker-credential-ecr-login

» docker-credential-ecr-login -v
amazon-ecr-credential-helper
Version:    0.6.0
Git commit: 69c85dc22db6511932bbf119e1a0cc5c90c69a7f

FYI, SSO-related issues seem to be gone in a new binary version (077b4a9), which can be built manually from the sources. Not sure why new releases are not posted for such a long time already

@matuszeman yes, I have the permsision GetAuhtorizationToken permission. I’m also able to perform pulls authenticating via aws ecr get-login-password | docker login, but not via credential helper =/

I noticed a log file inside .ecr folder with this line that might help:

time="2022-10-04T00:25:03-03:00" level=error msg="Error retrieving credentials" error="ecr: Failed to get authorization token: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"

It looks like the Homebrew formula is updated now. In my case, I had to force-link it after installing, to overwrite the 0.5.0 version installed with Docker Desktop. I also had to clean up a bunch of pre-SSO config, but the most relevant things were:

• Deleting old ECR cache: rm ~/.ecr/cache.json
• Deleting old AWS CLI credentials: rm ~/.aws/credentials (not 100% sure if this is required, but it’s not needed/used anymore with SSO)
• Re-logging to establish new tokens in the expected places: aws sso login

After all of the above, I was able to painlessly pull images from ECR again.

For anyone wandering in from Google, I can confirm that #267 fixes this for me after building from source.

Special thanks to @fangn2 who worked to push this through. v0.7.0 is available with fix for the issue.

@samuelkarp I noticed you have changed CHANGELOG.md in the past…

@gondalez I no longer work at Amazon.

I am using amazon-ecr-credential-helper for first time, I installed 0.6.0 version via brew. When I pull from private ECR repo, I see Error response from daemon: Head "https://***.dkr.ecr.us-east-1.amazonaws.com/v2/prisidio/service-base-image/manifests/latest": no basic auth credentials

What am I missing? 🤔 I use AWS SSO. Exporting correct profile, and sos sso login as well.

My dockerconfig is:

{
        "credsStore": "ecr-login",
        "credHelpers": {
          "public.ecr.aws": "ecr-login",
          "XXX.dkr.ecr.us-east-1.amazonaws.com": "ecr-login"
        }
}

brew info docker-credential-helper-ecr                                                                                                                                   main
==> docker-credential-helper-ecr: stable 0.6.0 (bottled)
Docker Credential Helper for Amazon ECR
https://github.com/awslabs/amazon-ecr-credential-helper
/usr/local/Cellar/docker-credential-helper-ecr/0.6.0 (7 files, 8.6MB)
  Poured from bottle on 2022-09-16 at 09:59:58
...

I was wondering if a release is planned in the next coming weeks 😃 This is also one of the last tools I need for AWS SSO adoption. For now I guess I will build it from source, but a release would be better!

The solution I use with other applications that do not yet natively support AWS SSO is an external credential_process, using a tool like https://github.com/benkehoe/aws-sso-credential-process

However, this doesn’t currently work with amazon-ecr-credential-helper. In the logs I get an error like the following:

... error="ecr: Failed to get authorization token: ProcessProviderParseError: parse failed of credential_process output: <JSON REDACTED>\ncaused by: unexpected end of JSON input"

This is because the output from the credential process is longer than 1024 characters, but the max buffer size was only increased in aws-sdk-go v1.25.42.

Until proper SSO support is added here, it would be great to bump the SDK version to allow use of credential_process as a workaround.

I can confirm this particular issue with credential_process not working with was fixed with this PR: https://github.com/awslabs/amazon-ecr-credential-helper/pull/240 - however a version of amazon-ecr-credential-helper hasn’t been released with this update.

As for native support of SSO I imagine this is stalled until something like this: https://github.com/aws/aws-sdk-go/pull/3610 is merged to the SDK, then it will Just Work™ with another SDK upgrade because it seems it’ll be part of the default profile credential parser.

Also, sidebar - it seems that Docker Desktop for Mac is overwriting /usr/local/bin/docker-credential-ecr-login when the application is started, which is seemingly terrible and and they should feel bad for doing that. So that’s also a mess 🤸

@gautam-nutalapati I had the same problem as you reported. I fixed it by setting AWS profile via environment variable.

AWS_PROFILE=<your-profile> docker pull <ecr-id>.dkr.ecr.eu-central-1.amazonaws.com/<repo>

This is still a problem for Homebrew users as the current formula is pinned at 0.5.0 (b19192b6522b2da02d14ec394c331f3b1a70efe2).

I’ve also taken a stab at this, and ended up with something that doesn’t rely on node and doesn’t store temp credentials anywhere*. It’s working well for me, and hopefully it’s of some value to the rest of you as well.

https://github.com/TylerLubeck/docker-credentials-aws-ecr-sso

*Not to say that these things are bad, they just don’t fit for my use case

I @otaviomedeirossb, as already stated, AWS CLI v2 will write AWS SSO temporary credentials under ~/.aws/cli/cache folder, which in many cases ends up to a “credentials not found” issue. I’m currently working on an open source project that addresses this problem too. Indeed, it provides support to AWS SSO and lists you all the IAM Roles that your AWS SSO User can access. By clicking a session card associated with a IAM Role, Leapp will generate temporary credentials to access your AWS Organization’s Account, inside ~/.aws/credentials file. Let me know if it makes sense to you and if you have questions about it.

@dougrday . Your solution is the only one that worked for me. I understand the risk of exposing the password for a short period of time. That said, it is the only workaround until other credential helpers solve this issue. Thanks!