aws-vault: InvalidClientTokenId error when creating IAM role with vault credentials and terraform
OS: 10.13.4 (High Sierra) Terraform version: v0.11.7 provider.aws: v1.26.0
The problem: When I attempt to create an IAM role via terraform while using vault credentials for a user called “terraform”, a 403 error occurs.
Shanes-MacBook-Pro:terraform_test shanekeller$ aws-vault --debug exec terraform -- terraform apply --auto-approve
2018/07/09 22:43:27 [keyring] Considering backends: [keychain file]
2018/07/09 22:43:27 Loading config file /Users/shanekeller/.aws/config
2018/07/09 22:43:27 Parsing config file /Users/shanekeller/.aws/config
2018/07/09 22:43:27 Looking for sessions for terraform
2018/07/09 22:43:27 Looking up all keys in keyring
2018/07/09 22:43:27 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2018/07/09 22:43:27 [keyring] Found 3 results
2018/07/09 22:43:27 Session "terraform session (1531204691)" expires in 54m43.768571954s
2018/07/09 22:43:27 [keyring] Querying keychain for service="aws-vault", account="terraform session (1531204691)", keychain="aws-vault.keychain"
2018/07/09 22:43:30 [keyring] Found item "aws-vault session for terraform"
2018/07/09 22:43:30 Using session ********************, expires in 54m40.363844684s
2018/07/09 22:43:30 Setting subprocess env: AWS_DEFAULT_REGION=us-west-2, AWS_REGION=us-west-2
2018/07/09 22:43:30 Setting subprocess env: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
2018/07/09 22:43:30 Setting subprocess env: AWS_SESSION_TOKEN, AWS_SECURITY_TOKEN
aws_iam_role.iam_for_lambda: Creating...
arn: "" => "<computed>"
assume_role_policy: "" => "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Action\": \"sts:AssumeRole\",\n \"Principal\": {\n \"Service\": \"lambda.amazonaws.com\"\n },\n \"Effect\": \"Allow\",\n \"Sid\": \"\"\n }\n ]\n}\n"
create_date: "" => "<computed>"
force_detach_policies: "" => "false"
max_session_duration: "" => "3600"
name: "" => "iam_for_lambda"
path: "" => "/"
unique_id: "" => "<computed>"
Error: Error applying plan:
1 error(s) occurred:
* aws_iam_role.iam_for_lambda: 1 error(s) occurred:
* aws_iam_role.iam_for_lambda: Error creating IAM Role iam_for_lambda: InvalidClientTokenId: The security token included in the request is invalid
status code: 403, request id: 3124a1fa-8404-11e8-b8c9-0f4f651e0f50
Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.
Here are the terraform debug logs just before the error:
2018-07-09T22:46:53.942-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Accept-Encoding: gzip
2018-07-09T22:46:53.942-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4:
2018-07-09T22:46:53.942-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Action=GetRole&RoleName=iam_for_lambda&Version=2010-05-08
2018-07-09T22:46:53.942-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: -----------------------------------------------------
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: 2018/07/09 22:46:54 [DEBUG] [aws-sdk-go] DEBUG: Response iam/GetRole Details:
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: ---[ RESPONSE ]--------------------------------------
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: HTTP/1.1 403 Forbidden
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Connection: close
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Content-Length: 305
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Content-Type: text/xml
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Date: Tue, 10 Jul 2018 05:46:53 GMT
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: X-Amzn-Requestid: a68ca1e8-8404-11e8-b6a6-e1ba4d916180
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4:
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4:
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: -----------------------------------------------------
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: 2018/07/09 22:46:54 [DEBUG] [aws-sdk-go] <ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">
I am able to successfully create the same IAM role via terraform by hardcoding the “terraform” user credentials into the provider definition in the .tf file.
Shanes-MacBook-Pro:terraform_test shanekeller$ aws-vault exec terraform -- terraform apply --auto-approve
aws_iam_role.iam_for_lambda: Creating...
arn: "" => "<computed>"
assume_role_policy: "" => "{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Action\": \"sts:AssumeRole\",\n \"Principal\": {\n \"Service\": \"lambda.amazonaws.com\"\n },\n \"Effect\": \"Allow\",\n \"Sid\": \"\"\n }\n ]\n}\n"
create_date: "" => "<computed>"
force_detach_policies: "" => "false"
max_session_duration: "" => "3600"
name: "" => "iam_for_lambda"
path: "" => "/"
unique_id: "" => "<computed>"
aws_iam_role.iam_for_lambda: Creation complete after 1s (ID: iam_for_lambda)
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
tf file
provider "aws" {
region = "us-west-2"
}
resource "aws_iam_role" "iam_for_lambda" {
name = "iam_for_lambda"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}
~/.aws/config file
[profile terraform]
mfa_serial = arn:aws:iam::207283649878:user/terraform
region = us-west-2
source_profile = terraform
What I’ve tried
Creating other resources with the aws-vault credentials and terraform. I’m able to successfully create s3 buckets and a vpc.
resource "aws_s3_bucket" "terraform_test"{
bucket = "terraform-test-1983451"
}
resource "aws_vpc" "test" {
# https://serverfault.com/questions/630022/what-is-the-recommended-cidr-when-creating-vpc-on-aws
# "...there is no harm in starting with a small prefix such as /16 because you can always create subnets."
cidr_block = "11.0.0.0/16"
# enabled by default, but just to be sure
enable_dns_support = true
enable_dns_hostnames = true
}
It seems like the security token generated by aws-vault has different permissions than I’m expecting. But I’m not sure what additional debugging steps to take.
There’s another issue that addresses security tokens, but I don’t think the solution applies to me because the poster had two roles: https://github.com/99designs/aws-vault/issues/262
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 25 (7 by maintainers)
yes that is the expected behaviour. AWS does not allow IAM operations with an assumed role unless it’s authenticated with an MFA
I would guess the issue is layering of an STS session and then trying to assume a role with that.
aws-vault
is designed with a primary objective of “protect your root credentials”. E.g. only disclose short-lived temporary credentials to subprocesses. There are two ways to do this, either creating an STS session and providing those credentials to the subprocess, or assuming a role and providing the session credentials form that to the subprocess.AWS has complicated rules about chaining temporary credentials together with assuming roles, or assuming roles from other roles.
If you really want, you can disable the session that
aws-vault
creates withaws-vault exec <profile> --no-session
, but I think it waters down a lot of the security that it provides.In terms of your specific problem, I’d suggest troubleshooting it by getting the simplest config you can working and then adding complexity to it. Start without Terraform, get
aws-vault
simply assuming a role in your CLI. Prove that you can run something likeaws s3 ls
or an equivalent simple command.Note that your config should look something like:
The
source_profile
refers to the profile to look for the credentials in. In this case, credentials are against my IAM user, in myljd
profile. I then use anljd-admin
profile to assume a role, via the credentials in theljd
profile.What does this give:
Make sure to remove the creds when posting 😉
Sounds like you are missing MFA