aws-sdk-go: Get credentials from role has become slow
Please fill out the sections below to help us address your issue.
Version of AWS SDK for Go?
v1.25.41
Version of Go (go version)?
go version go1.11.1 linux/amd64
What issue did you see?
Getting credentials from an ec2 role takes 20 seconds.
We have this issue on consistently on multiple different ec2 instances since a couple of days ago.
To me it looks like an issue with the service, but I am not sure.
Steps to reproduce
package slow
import (
"fmt"
"time"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds"
"github.com/aws/aws-sdk-go/aws/ec2metadata"
"github.com/aws/aws-sdk-go/aws/session"
)
func getCredentialsFromRole() (*credentials.Credentials, error) {
roleProvider := &ec2rolecreds.EC2RoleProvider{
Client: ec2metadata.New(session.New()),
}
creds := credentials.NewCredentials(roleProvider)
start := time.Now().UTC()
if _, err := creds.Get(); err != nil { // this takes 20 seconds
return nil, err
}
fmt.Printf("getting credentails from role took %s\n", time.Now().UTC().Sub(start))
return creds, nil
}
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 30
- Comments: 22 (4 by maintainers)
Commits related to this issue
- `aws/ec2metadata`: Reduces timeout and number of max retries for EC2Metadata client (#3066) The PR address the issues related to EC2Metadata client having long timeouts in case of failure in obtainin... — committed to aws/aws-sdk-go by skotambkar 4 years ago
- Release v1.27.2 (2020-01-07) === ### Service Client Updates * `service/AWSMigrationHub`: Updates service API, documentation, and paginators * `service/codebuild`: Updates service API and documentatio... — committed to aws/aws-sdk-go by deleted user 4 years ago
- Release v1.27.2 (2020-01-07) (#3068) Release v1.27.2 (2020-01-07) === ### Service Client Updates * `service/AWSMigrationHub`: Updates service API, documentation, and paginators * `service/codebuild`... — committed to aws/aws-sdk-go by aws-sdk-go-automation 4 years ago
- Reduce timeout for accessing EC2 instance metadata service Previously we were re-using our shared HTTP client, which has a rather high timeout (120 seconds) that causes the HTTP client to wait around... — committed to pganalyze/collector by lfittl 3 years ago
- Reduce timeout for accessing EC2 instance metadata service Previously we were re-using our shared HTTP client, which has a rather high timeout (120 seconds) that causes the HTTP client to wait around... — committed to pganalyze/collector by lfittl 3 years ago
Our problem is the same as described above.
aws ec2 modify-instance-metadata-options --instance-id i-34215432543254235 --http-endpoint enabled --http-put-response-hop-limit 2is a temporary solution.I really hope their will be a better solution in the future. Running Go with this sdk in a docker container on an EC2 instance is one of the most common use cases of this sdk that I can think of. This really should just work out of the box.
@alexd765 's solution worked for me - thanks Alex!
You can implement his solution in terraform like so:
Also note that it’s not just go that runs into this problem - even just using curl as described in https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html results in a long delay.
I’m also seeing this issue (I believe it’s the same) and I’ve been able to pin point
v1.25.38as the culprit release.Before
v1.25.38I can see the following requests being made by the sdk (when running on a docker container on EC2):while on
v1.25.38and newer:I run this app without specifying AWS env keys nor with a shared profile.
@pmalekn - They introduced the new EC2 metadata feature in v1.25.38 - and a backwards compatibility issue by setting the default hop limit to 1 - which means the replies get dropped while transiting the docker bridge network.
You have to increase the hop limit yourself or it just hit lengthy timeouts & retries before eventual fallback to the old method
Can we remove the stale lifecycle on this one please @skmcgrail ? I’m presuming it’s still an issue and the reason people aren’t commenting on it is because of the workaround. A lot of people have ran into this problem and it would be good to have explicit confirmation that it is fixed when it gets fixed.
Running on Fargate in ECS, this is not only slow but completely broken in latest SDK (
v1.33.4at time of testing). Errors with could not find credentials configuration. Only downgrading to1.25.37resolves the issue, since the hop limit can not be configured for Fargate.Tested with this https://github.com/particleflux/s3fetch/blob/master/s3fetch.go run directly in docker entrypoint.
@vincentjorgensen This article summarizes very well how this TTL thing works: https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/ see the
Protecting against open layer 3 firewalls and NATssection.We also saw similar behavior, we noticed issues on the operations
The above curl is also super-fast for us (only tested on t2.small)
Ran into the same problem using the AWS go SDK in an Elastic Beanstalk (EB) Docker environment, but didn’t see a way to set
http_put_response_hop_limitin EB without manipulating the EC2 instance directly.Fixed (or rather sidestepped) it by setting AWS user credentials and default region as environment variables - hope it helps others in the same situation.
I have a delay of 3 seconds as of v1.29.24 when trying to create a new session using this code:
I am using:
Will revert to v1.25.19 until this is fixed, because manually increasing the hop limit is not an acceptable solution.
No global option I know of, though hoping AWS might see sense & implement something like https://github.com/aws/aws-sdk-go/issues/2980
In the mean time you can only stay on old version of aws-sdk or update all EC2 instances to increase the hop limit, e.g. with: