aws-sdk-php: InstanceProfileProvider: bug with Guzzle 7.0.1, works with 6.4.5

Confirm by changing [ ] to [x] below to ensure that it’s a bug:

Describe the bug S3Client used in the context of InstanceProfilePovider. The description is the same as #2022 Aws\Exception\CredentialsException Timeout, but it fails only with guzzlehttp/guzzle v7.0.1. The same SDK version on same instance (and same application) with guzzlehttp/guzzle v 6.5.4 works fine. Therefore I don’t think that the issue is the EC2 instance being unable to respond quick enough.

Version of AWS SDK for PHP? v3.138.9 (but I believe it’s the same with 3.145)

Version of PHP (php -v)? php 7.4

To Reproduce (observed behavior)

When running this code on an EC2 instance with an IAM role granting the appropriate S3 permissions:

$client = new S3Client([
    'version' => 'latest',
    'region' => 'us-east-1,
]);

$client->getObject([
  'Bucket' => $bucket,
  'Key' => $key,
  'SaveAs' => $localFilename,
]);

with the Guzzlehttp version reported above (7.0.1): I get an exception:

Aws\\Exception\\CredentialsException
Code: 0
Message: Error retrieving credentials from the instance profile metadata service. (cURL error 28: Operation timed out after 1000 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html))
File: /srv/app/vendor/aws/aws-sdk-php/src/Credentials/InstanceProfileProvider.php
Line: 226
Trace:
#0 /srv/app/vendor/guzzlehttp/promises/src/Promise.php(203): Aws\\Credentials\\InstanceProfileProvider->Aws\\Credentials\\{closure}()
#1 /srv/app/vendor/guzzlehttp/promises/src/Promise.php(156): GuzzleHttp\\Promise\\Promise::callHandler()
#2 /srv/app/vendor/guzzlehttp/promises/src/TaskQueue.php(47): GuzzleHttp\\Promise\\Promise::GuzzleHttp\\Promise\\{closure}()
#3 /srv/app/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(155): GuzzleHttp\\Promise\\TaskQueue->run()
#4 /srv/app/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(182): GuzzleHttp\\Handler\\CurlMultiHandler->tick()
#5 /srv/app/vendor/guzzlehttp/promises/src/Promise.php(246): GuzzleHttp\\Handler\\CurlMultiHandler->execute()
#6 /srv/app/vendor/guzzlehttp/promises/src/Promise.php(223): GuzzleHttp\\Promise\\Promise->invokeWaitFn()
#7 /srv/app/vendor/guzzlehttp/promises/src/Promise.php(267): GuzzleHttp\\Promise\\Promise->waitIfPending()
#8 /srv/app/vendor/guzzlehttp/promises/src/Promise.php(225): GuzzleHttp\\Promise\\Promise->invokeWaitList()
#9 /srv/app/vendor/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\\Promise\\Promise->waitIfPending()
#10 /srv/app/vendor/guzzlehttp/promises/src/Coroutine.php(65): GuzzleHttp\\Promise\\Promise->wait()
#11 /srv/app/vendor/guzzlehttp/promises/src/Promise.php(246): GuzzleHttp\\Promise\\Coroutine->GuzzleHttp\\Promise\\{closure}()
#12 /srv/app/vendor/guzzlehttp/promises/src/Promise.php(223): GuzzleHttp\\Promise\\Promise->invokeWaitFn()
#13 /srv/app/vendor/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\\Promise\\Promise->waitIfPending()
#14 /srv/app/vendor/guzzlehttp/promises/src/Coroutine.php(85): GuzzleHttp\\Promise\\Promise->wait()
#15 /srv/app/vendor/guzzlehttp/promises/src/Promise.php(273): GuzzleHttp\\Promise\\Coroutine->wait()
#16 /srv/app/vendor/guzzlehttp/promises/src/Promise.php(225): GuzzleHttp\\Promise\\Promise->invokeWaitList()
#17 /srv/app/vendor/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\\Promise\\Promise->waitIfPending()
#18 /srv/app/vendor/aws/aws-sdk-php/src/AwsClientTrait.php(59): GuzzleHttp\\Promise\\Promise->wait()
#19 /srv/app/vendor/aws/aws-sdk-php/src/AwsClientTrait.php(87): Aws\\AwsClient->execute()

** Expected behaviour**

Exact same code, same machine, works fine if I downgrade guzzlehttp/guzzle to version 6.5.4.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 2
  • Comments: 17 (5 by maintainers)

Most upvoted comments

Just confirming this bug as well.

In our case, we were running into Error retrieving credentials from the instance profile metadata service. trying to send messages via SES from pods running in EKS using IAM via service account with the eks.amazonaws.com/role-arn annotation.

We were fairly certain from the beginning of troubleshooting that this was something specific to the AWS PHP SDK since we have Node.js applications running on the same cluster under the same service account using SES/SNS/SQS without issue.

But can confirm that rolling Guzzle back to 6.5.4 fixed the issue.

Thanks for your insight, everyone!

@fischaz This does look a case of the thrown exceptions changing in Guzzle 7, and we’ve added a fix in #2117 that will go out in the next release.

Let us know if anyone still has any issues.

Hi all,

I believe we have the same issue (reading the ticket and the condition). debugging internally, in our case, we’ve narrowed it down to the way the PHP/Guzzle first try to access the AWS Metadata v2 Token API http://169.254.169.254/latest/api/token which:

  • does connect with TCP
  • but times out waiting for the PUT call to respond.

I could reproduce the issue within the same docker container using curl:

curl -X PUT -v http://169.254.169.254/latest/api/token -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"

Strangely, the same curl command was successful on the host itself, so there is something fishy on the way AWS might deal with the docker network and the token API call.

Also, access to any other EC2 metadata endpoint (like the direct Ec2 credentials endpoint) is still working from within the container normally with or without the token header (because we don’t enforce metadata V2 just yet).

I believe this might explain why the older version of guzzle is working as it might not yet have implemented the code to use metadata v2 access.

I’ll have to dig further on my side as to why AWS metadata v2 token API access is timeout within a docker container (this does not quite make sense yet from a TCP/HTTP point of view).

Hopefully this is helpful.

a better way might be to use

aws ec2 modify-instance-metadata-options --instance-id .... --http-put-response-hop-limit 2

to set the instance metadata options, because this can be done for launch configurations and templates too.

This is the recommended way of dealing with this use case. For more context, EC2 has a guide on configuring IMDSv2.

iptables -t mangle -A PREROUTING -i eth0 -m tcp -p tcp -s 169.254.169.254 --sport 80 -m ttl --ttl 1 -j TTL --ttl-set 2

a better way might be to use

aws ec2 modify-instance-metadata-options --instance-id .... --http-put-response-hop-limit 2

to set the instance metadata options, because this can be done for launch configurations and templates too.

@fischaz even if that’s the issue… how does this explain that it works with Guzzle6? Do you have any idea?