terraform-provider-aws: AWS profile does not work with role_arn and credential_source = Ec2InstanceMetadata

The root cause of this issue explained here https://github.com/hashicorp/aws-sdk-go-base/issues/7

I have a slightly different use case - running TF in EKS pod that uses IAM attached to Service Account as described here https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

The long story short, I get ~/.aws/config file like this:

[profile profile1]
role_arn = arn:aws:iam::xxx:role/pod
web_identity_token_file = /var/run/secrets/eks.amazonaws.com/serviceaccount/token
[profile profile2]
source_profile = profile1
role_arn = arn:aws:iam::xxx:role/some-other-role-allowed-to-be-assumed-from-profile1

Then I have export AWS_PROFILE=profile2 just before calling terraform. I have simple TF code to test this:

provider "aws" {
  version = "2.34.0"
  region  = "us-west-2"
}

data "aws_caller_identity" "current" {}

output "aws_caller_identity" {
  value = data.aws_caller_identity.current
}

Terraform picking up EKS node instance profile instead of everything defined in ~/.aws/config. I think I have slightly better workaround than skip_metadata_api_check - trick AWS SDK into thinking it’s not running in AWS by defining AWS_METADATA_URL environment variable to some absurd endpoint:

export AWS_METADATA_URL="http://localhost/not/existent/url"

For my particular use case, AWS metadata IP should be anyway iptabled out so not accessible by EKS pods, I just haven’t got there yet. Still this is a bug in https://github.com/hashicorp/aws-sdk-go-base/ worth fixing - I can imagine there might be use cases these workarounds does not apply. Such as using role_arn and instance profile for different provider instances or something if EKS node instance profile not intended to be hidden from it’s pods. Anyway diverging from official AWS SDK credentials chain logic (or official AWS SDK in general) sounds like a bad practice. It may come in all sorts of unintended behaviour/bugs, AWS systems are pretty complex and heavily rely on conventions and standards like this AWS credentials chaining order.

Hi folks 👋 Version 3.0 of the Terraform AWS Provider will include a few authentication changes that should help in this case including:

• Enabling the AWS shared configuration file (e.g. ~/.aws/config) by default
• Ensuring the AWS shared configuration and credential files are read before falling back to the EC2 Instance Metadata Service (should no longer need workarounds to disable it via AWS_METADATA_URL)

This major version update will release in the next two weeks or so. Please follow the v3.0.0 milestone for tracking the progress of that release. If you are still having trouble after updating when its released, please file a new issue. Thanks!

@llibicpep I think your analysis nailed the problem, and was a huge help to me in putting together this proposed fix. It could still use some additional test cases if anyone else has time to pitch in. https://github.com/hashicorp/aws-sdk-go-base/pull/20

@bflad could you review?

Using AWS_SDK_LOAD_CONFIG=1 fixed a similar issue for me where I was using roles to switch from a master account. Obviously not related to this issue, but thought I’d leave a comment here for future seekers

@shanee-spring (and future readers of this) the thing that doesn’t work until TF v0.12 is using the ~/.aws/config to get the role arn (allowing you to not specify that in the Terraform backend directly).

@shanee-spring - The s3 backend does not use the provider block. It uses the terraform { backend block. I was only able to get the provider block to work with skip_metadata_api_check = true, not the backend block. I tested with Terraform v0.12a and found that the backend block also works. I think the tracking issue for that is: https://github.com/hashicorp/terraform/issues/18213

Easy test case for this with 2 accounts (this is Terraform v0.12.5) …

In account 1: Create an EC2 Instance, assign an IAM role to that instance In account 2: Create a role with a policy that allows account 1 to assume it (here it’s called dev-account-role) In account 1: On the instance, plop the following into ~/.aws/config where 12345678901 is account 2 id:

[profile dev]
role_arn = arn:aws:iam::12345678901:role/dev-account-role
credential_source = Ec2InstanceMetadata
region = us-west-2

Run this .tf on this instance:

# instance profile:
provider "aws" {}

# assumed role from credential_source:
provider "aws" {
    profile = "dev"
    alias = "assumed_role"
}

data "aws_caller_identity" "instance" {}

data "aws_caller_identity" "assumed_role" {
    provider = "aws.assumed_role"
}

output "instance_profile_role_arn" {
    value = "${data.aws_caller_identity.instance.arn}"
}

output "assumed_role_arn" {
    value = "${data.aws_caller_identity.assumed_role.arn}"
}

Expected: assumed_role_arn and instance_profile_role_arn are not the same.

Actual: they are the same.

As stated, skip_metadata_api_check = true fixes it. Also worth mentioning that this fix doesn’t seem to work when this same tf file is applied by Atlantis. 😢

Mine is when EC2 attached to a Role because needed for AWS Session Manager and i use shared credentials file for Terraform Backend when running init (try access S3) got 403 denied turns out the caller identity is the Role from EC2 Instance Profile not from shared credentials, using skip_metadata_api_check=true not working but exporting AWS_METADATA_URL to non existent url works. Thanks @dee-kryvenko

This should be supported in version 1.41.0 since the aws-sdk-go dependency was updated to v1.15.55 in #6164. See also: https://github.com/terraform-providers/terraform-provider-aws/blob/v1.41.0/vendor/vendor.json#L177-L184

https://github.com/aws/aws-sdk-go/pull/2201 just got merged which adds support for credential_source