botocore: Error when retrieving credentials from iam-role: Credential refresh failed, response did not contain: access_key, secret_key, token, expiry_time

We are seeing a strange issue relating to boto3 and botocore. The following error is being thrown sporadically when we try to read from S3 or utilize an SQS client.

Error when retrieving credentials from iam-role: Credential refresh failed, response did not contain: access_key, secret_key, token, expiry_time

It appears that the credentials are not correctly getting refreshed via the assumed IAM role. This a Python application running inside of a Docker container within EKS. An example piece of code is below.

def fetch_message(s3, bucket, key):
    response = s3.get_object(Bucket=bucket, Key=key)

Does anybody have any ideas why this is happening and whether or not this is a known issue with boto?

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 19 (2 by maintainers)

Most upvoted comments

Hi, I was facing this issue running python in a pod in an EKS cluster, and it seems at first glance the retries/timeout solution worked. Did anyone figure out a reason why these requests fail? I’ve seen pods restart hundreds of times because of this and I’m curious if there is something in the EKS setup that can be used to mitigate.

martimors on Oct 27, 2020

Hey guys, did you figure out this issue by any chance?

spinus on Mar 5, 2020

I’m also using kube2iam to have a pod assume an IAM role and seeing this error sporadically. It sometimes happens at the start of the container, but we’ve also seen it happen after the containers been running for a while. Any suggestions on workarounds? We’ve set AWS_METADATA_SERVICE_NUM_ATTEMPTS but it seems to have no effect.

shshe on May 6, 2019

It looks like you’re sourcing credentials from the EC2 Instance Metadata and the request to fetch them failed. By default we don’t retry those requests, but you can add retries with metadata_service_num_attempts and metadata_service_timeout in the config file (docs).

JordonPhillips on Dec 11, 2018