aws-vault: InvalidClientTokenId error when creating IAM role with vault credentials and terraform

OS: 10.13.4 (High Sierra) Terraform version: v0.11.7 provider.aws: v1.26.0

The problem: When I attempt to create an IAM role via terraform while using vault credentials for a user called “terraform”, a 403 error occurs.

Shanes-MacBook-Pro:terraform_test shanekeller$ aws-vault --debug exec terraform -- terraform apply --auto-approve 
2018/07/09 22:43:27 [keyring] Considering backends: [keychain file]
2018/07/09 22:43:27 Loading config file /Users/shanekeller/.aws/config
2018/07/09 22:43:27 Parsing config file /Users/shanekeller/.aws/config
2018/07/09 22:43:27 Looking for sessions for terraform
2018/07/09 22:43:27 Looking up all keys in keyring
2018/07/09 22:43:27 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2018/07/09 22:43:27 [keyring] Found 3 results
2018/07/09 22:43:27 Session "terraform session (1531204691)" expires in 54m43.768571954s
2018/07/09 22:43:27 [keyring] Querying keychain for service="aws-vault", account="terraform session (1531204691)", keychain="aws-vault.keychain"
2018/07/09 22:43:30 [keyring] Found item "aws-vault session for terraform"
2018/07/09 22:43:30 Using session ********************, expires in 54m40.363844684s
2018/07/09 22:43:30 Setting subprocess env: AWS_DEFAULT_REGION=us-west-2, AWS_REGION=us-west-2
2018/07/09 22:43:30 Setting subprocess env: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
2018/07/09 22:43:30 Setting subprocess env: AWS_SESSION_TOKEN, AWS_SECURITY_TOKEN
aws_iam_role.iam_for_lambda: Creating...
  arn:                   "" => "<computed>"
  assume_role_policy:    "" => "{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Action\": \"sts:AssumeRole\",\n      \"Principal\": {\n        \"Service\": \"lambda.amazonaws.com\"\n      },\n      \"Effect\": \"Allow\",\n      \"Sid\": \"\"\n    }\n  ]\n}\n"
  create_date:           "" => "<computed>"
  force_detach_policies: "" => "false"
  max_session_duration:  "" => "3600"
  name:                  "" => "iam_for_lambda"
  path:                  "" => "/"
  unique_id:             "" => "<computed>"

Error: Error applying plan:

1 error(s) occurred:

* aws_iam_role.iam_for_lambda: 1 error(s) occurred:

* aws_iam_role.iam_for_lambda: Error creating IAM Role iam_for_lambda: InvalidClientTokenId: The security token included in the request is invalid
	status code: 403, request id: 3124a1fa-8404-11e8-b8c9-0f4f651e0f50

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Here are the terraform debug logs just before the error:

2018-07-09T22:46:53.942-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Accept-Encoding: gzip
2018-07-09T22:46:53.942-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: 
2018-07-09T22:46:53.942-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Action=GetRole&RoleName=iam_for_lambda&Version=2010-05-08
2018-07-09T22:46:53.942-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: -----------------------------------------------------
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: 2018/07/09 22:46:54 [DEBUG] [aws-sdk-go] DEBUG: Response iam/GetRole Details:
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: ---[ RESPONSE ]--------------------------------------
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: HTTP/1.1 403 Forbidden
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Connection: close
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Content-Length: 305
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Content-Type: text/xml
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Date: Tue, 10 Jul 2018 05:46:53 GMT
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: X-Amzn-Requestid: a68ca1e8-8404-11e8-b6a6-e1ba4d916180
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: 
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: 
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: -----------------------------------------------------
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: 2018/07/09 22:46:54 [DEBUG] [aws-sdk-go] <ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">

I am able to successfully create the same IAM role via terraform by hardcoding the “terraform” user credentials into the provider definition in the .tf file.

Shanes-MacBook-Pro:terraform_test shanekeller$ aws-vault exec terraform -- terraform apply  --auto-approve

aws_iam_role.iam_for_lambda: Creating...
  arn:                   "" => "<computed>"
  assume_role_policy:    "" => "{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Action\": \"sts:AssumeRole\",\n      \"Principal\": {\n        \"Service\": \"lambda.amazonaws.com\"\n      },\n      \"Effect\": \"Allow\",\n      \"Sid\": \"\"\n    }\n  ]\n}\n"
  create_date:           "" => "<computed>"
  force_detach_policies: "" => "false"
  max_session_duration:  "" => "3600"
  name:                  "" => "iam_for_lambda"
  path:                  "" => "/"
  unique_id:             "" => "<computed>"
aws_iam_role.iam_for_lambda: Creation complete after 1s (ID: iam_for_lambda)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

tf file

provider "aws" {
  region     = "us-west-2"
}

resource "aws_iam_role" "iam_for_lambda" {
  name = "iam_for_lambda"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

~/.aws/config file

[profile terraform]
mfa_serial = arn:aws:iam::207283649878:user/terraform
region = us-west-2
source_profile = terraform

What I’ve tried

Creating other resources with the aws-vault credentials and terraform. I’m able to successfully create s3 buckets and a vpc.

resource "aws_s3_bucket" "terraform_test"{
  bucket = "terraform-test-1983451"
}

resource "aws_vpc" "test" {
  # https://serverfault.com/questions/630022/what-is-the-recommended-cidr-when-creating-vpc-on-aws
  # "...there is no harm in starting with a small prefix such as /16 because you can always create subnets."
  cidr_block = "11.0.0.0/16"

  # enabled by default, but just to be sure
  enable_dns_support   = true
  enable_dns_hostnames = true
}

It seems like the security token generated by aws-vault has different permissions than I’m expecting. But I’m not sure what additional debugging steps to take.

There’s another issue that addresses security tokens, but I don’t think the solution applies to me because the poster had two roles: https://github.com/99designs/aws-vault/issues/262

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 25 (7 by maintainers)

Most upvoted comments

yes that is the expected behaviour. AWS does not allow IAM operations with an assumed role unless it’s authenticated with an MFA

+12

frezbo on Jul 13, 2018

I would guess the issue is layering of an STS session and then trying to assume a role with that.

aws-vault is designed with a primary objective of “protect your root credentials”. E.g. only disclose short-lived temporary credentials to subprocesses. There are two ways to do this, either creating an STS session and providing those credentials to the subprocess, or assuming a role and providing the session credentials form that to the subprocess.

AWS has complicated rules about chaining temporary credentials together with assuming roles, or assuming roles from other roles.

If you really want, you can disable the session that aws-vault creates with aws-vault exec <profile> --no-session, but I think it waters down a lot of the security that it provides.

In terms of your specific problem, I’d suggest troubleshooting it by getting the simplest config you can working and then adding complexity to it. Start without Terraform, get aws-vault simply assuming a role in your CLI. Prove that you can run something like aws s3 ls or an equivalent simple command.

Note that your config should look something like:

[profile ljd]
region=us-east-1

[profile ljd-admin]
region=us-east-1
source_profile=ljd
mfa_serial=arn:aws:iam::xxx:mfa/lachlan.donald
role_arn=arn:aws:iam::xxx:role/assume-admin-access

The source_profile refers to the profile to look for the credentials in. In this case, credentials are against my IAM user, in myljd profile. I then use an ljd-admin profile to assume a role, via the credentials in the ljd profile.

lox on Jul 12, 2018

What does this give:

aws-vault exec deployment_user -- aws sts assume-role --role-arn <arn> --external-id <if there is one> --role-session-name dummy

Make sure to remove the creds when posting 😉

frezbo on Jul 10, 2018

Sounds like you are missing MFA

FernandoMiguel on Jul 10, 2018