terraform-provider-aws: Terraform AWS provider unable to assume role using profile that assumes a role itself

Hi folks 👋

This should be resolved in the S3 Backend as of Terraform version 0.12.3 and in the Terraform AWS Provider as of version 2.16.0.

I verified this locally via this configuration:

provider "aws" {
  region  = "us-east-2"
  version = "2.17.0"
}

data "aws_caller_identity" "current" {}

output "caller_arn" {
  value = "${data.aws_caller_identity.current.arn}"
}

This setup of AWS credentials and configuration files locally:

$ cat ~/.aws/config

[profile tf-acc-assume-role]
role_arn = arn:aws:iam::--OMITTED--:role/tf-acc-assume-role
source_profile = tf-acc

[profile tf-acc-assume-role-2]
role_arn = arn:aws:iam::--OMITTED--:role/tf-acc-assume-role-2
source_profile = tf-acc-assume-role

$ cat ~/.aws/credentials

[tf-acc]
aws_access_key_id = --OMITTED--
aws_secret_access_key = --OMITTED--

And running:

$ export AWS_SDK_LOAD_CONFIG=1
$ export AWS_PROFILE=tf-acc-assume-role-2
$ terraform apply
data.aws_caller_identity.current: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

caller_arn = arn:aws:sts::--OMITTED--:assumed-role/tf-acc-assume-role-2/1562206728701794000

For future bug reports or feature requests relating to provider authentication, even if they look similar to the error messages reported here, please submit new GitHub issues following the bug report and feature request issue templates for further triage. These types of issues tend to be very environment specific. 👍

I believe this is fixed with hashicorp/aws-sdk-go-base#5 PR. It sounds very similar.

So I have determined why this is occurring. terraform-provider-aws uses the library aws-sdk-go-base which takes care of retrieving credentials for the provider. Within aws-sdk-go-base, the aws-go-sdk credentials package is used to obtain credentials for the provider via a ChainProvider. Now you would think that the EnvProvider used in the ChainProvider would behave the same as the aws-go-sdk session package, in that it would respect the environment variable AWS_SDK_LOAD_CONFIG, however it does not, and because of this, any profile that doesn’t have credentials in the shared credentials file (by default ~/.aws/credentials) will not work with the terraform aws provider assume_role or profile options.

I’m happy to submit a PR to fix this, however feel that the PR would be better suited for the aws-go-sdk instead of the terraform-provider-aws or aws-sdk-go-base, as this issue will occur for any user of the aws-go-sdk credential package.

See:

• https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html
• https://godoc.org/github.com/aws/aws-sdk-go/aws/credentials

@rekahsoft If you have a minute, can you contribute this to my collection of credential tests? I’m trying to get an easily reproducible set of problems together: https://github.com/YakDriver/terraform-cred-tests

@bflad Unfortunately I’m still encountering this issue. Note that my validation method was slightly different. I am using a profile with only a single layer of assumed roles (tf-acc-assume-role, in your example above), and am receiving an error on the below provider block, which itself assumes a role:

provider "aws" { # This works fine
  version             = "~> 2.17.0"
  region              = var.region
}

provider "aws" { # Error happens here
  version = "~> 2.17.0"
  alias   = "my_alias"
  region  = var.region

  assume_role {
    role_arn = var.my_arn_var
  }
}

› terraform --version 
Terraform v0.12.3

Error

Error: No valid credential sources found for AWS Provider.
  Please see https://terraform.io/docs/providers/aws/index.html for more information on
  providing credentials for the AWS Provider

  on main.tf line 8, in provider "aws":
   8: provider "aws" {

I believe this is more similar to the use case for the original comment than that you provided. Could we reopen the issue?

I’m encountering what I believe to be the same issue, using an AWS profile with a source_profile, eg

[profile me]
role_arn = ROLE_ARN
source_profile = foobar

I first noticed this when trying to add a provider which used an assume_role to access a resource in another AWS account, but have noticed this happens even when I do not provide the assume_role part - all I need to do is provide a second AWS provider to encounter the error

provider "aws" {
  region = "${var.region}"
}

provider "aws" {
  alias   = "my_alias"
  region = "${var.region}"
}

The error I see:

No valid credential sources found for AWS Provider.
  Please see https://terraform.io/docs/providers/aws/index.html for more information on
  providing credentials for the AWS Provider

Some project owners have a policy of closing tickets when they are too hard to fix so that it doesn’t run up their median time for opened tickets

@bflad Still encountering this issue, can we reopen it?

I have tried @YakDriver 's solution, but it does not seem to work for me

$ export AWS_SDK_LOAD_CONFIG=1
$ export AWS_PROFILE=base
$ terraform apply
data.aws_caller_identity.current: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

caller_arn = arn:aws:sts::--OMITTED--:assumed-role/--OMITTED--/--OMITTED--

$ export AWS_SDK_LOAD_CONFIG=1
$ export AWS_PROFILE=tf-acc-assume-role
$ terraform apply

Error: No valid credential sources found for AWS Provider.
	Please see https://terraform.io/docs/providers/aws/index.html for more information on
	providing credentials for the AWS Provider

  on main.tf line 1, in provider "aws":
   1: provider "aws" {

$ terraform -v
Terraform v0.12.6
+ provider.aws v2.23.0

$ tail -3 ~/go/src/github.com/terraform-providers/terraform-provider-aws/go.mod 
)

replace github.com/hashicorp/aws-sdk-go-base => github.com/YakDriver/aws-sdk-go-base v0.0.0-20190503174753-82bd97734e8f

$ cat main.tf 
provider "aws" {
  region  = "us-west-2"
}

data "aws_caller_identity" "current" {}

output "caller_arn" {
  value = "${data.aws_caller_identity.current.arn}"
}

$ cat ~/.aws/config 
[default]
region=us-west-2
output=json

[profile base]
region=us-west-2
output=json

[profile tf-acc-assume-role]
role_arn = arn:aws:iam::--OMITTED--:role/aws-reserved/sso.amazonaws.com/us-west-2/--OMITTED--
source_profile = base

$ cat ~/.aws/credentials 
[default]
aws_access_key_id=--OMITTED--
aws_secret_access_key=--OMITTED--

[base]
aws_access_key_id = --OMITTED--
aws_secret_access_key = --OMITTED--
aws_session_token = --OMITTED--

I followed YakDriver’s instructions posted above to do the build with the addition of:

$ brew install go
$ brew install bzr
$ export GOPATH=~/go
$ export PATH=$PATH:$GOPATH/bin

This is failing for me as well with Terraform v0.12.5 and provider 2.20.0. These are roles that work fine with TF 0.11.

Both of these scenarios fail for me:

• Running Terraform locally using AWS credentials set via environment variables with aws-vault
• Running Terraform via CI/CD from an ECS service with a task role

Interestingly in my case, the Terraform plan works fine. It’s only the apply that fails. This is especially odd because the remote state backend is configured to assume the same role, and that part seems to be working since Terraform can read the remote state during the plan.

Hi folks, the fix @YakDriver described above is scheduled to be released with v2.32.0 next week. If you upgrade and the problem you had is still happening, please open a new issue so we can address the errors separately. Thanks!

Actually this worked for me. I used a better strategy although this is not documented anywhere.

My learning is remove the Access and Secret key credentials from the environment variables.if not remove the TF does not behave as expected.

Set the config and credentials environment variables.

AWS_SHARED_CREDENTIALS_FILE – Specifies the location of the file that the AWS CLI uses to store access keys. The default path is ~/.aws/credentials).

AWS_CONFIG_FILE – Specifies the location of the file that the AWS CLI uses to store configuration profiles. The default path is ~/.aws/config).

AWS_SDK_LOAD_CONFIG=“true”

Instead of assuming roles as stated above set them under config. You can go any level in assuming role and all you have to do is set the profile in providers definition and use it as alias(if required).

e.g config [default]

[profile AnAccount] role_arn=arn:aws:iam::1111111111111:role/SuperAdmin source_profile=default Above code shall change to this

provider “aws” { version = “~> 2.8” region = “ap-southeast-2” }

provider “aws” { version = “~> 2.8” profile = “AnAccount” alias = “AnAccount_ap2” region = “ap-southeast-2” }

#How to use it module “create_account” { source = “./account” params = local.params providers = { aws = aws.AnAccount_ap2 } } Read about provider when using with modules & alias.

This logic works perfectly.

I have credentails in env variables, set credentials and config environment vars. Here is my scenarios

1. user tfdev (account A) assume role to org_admin under (Payers’s account B) alias it B_org_admin
2. Call module “setup” with provider alias B_org_admin
3. Under Setup Module create a new provider alias “C_org_admin” which tries to switch to “org_admin” under account C
4. Provider cannot assume Role org_admin under Account C

I could verify that while executing module setup the role is org_admin under account C (using caller identity)

but I see cloudtrail under Account A that it failed to assume role org_admin under Account C.

Should it not try to assume role from Account B to Account C. Why is provider still trying to Assume from it from account A -> Account C when provider was created under setup module which was invoked with provider B_org_admin.

Questions

1. Is provider always trying to switch from default provider.
2. I have also created profiles and setup roles under this but TF isnt picking it,