terraform-provider-aws: terraform commands fail to detect expired AWS security tokens

This issue was originally opened by @scubahub as hashicorp/terraform#18024. It was migrated here as a result of the provider split. The original body of the issue is below.

Im using the AWS provider and relying on terraform to access using the values defined in the default ~/.aws/credentials and ~/.aws/config files. The credentials entries include aws_session_token which terraform does seem to pick up. However when that token is expired and I try to terraform plan, terraform tries to connect to AWS 15 times then just hangs.

At that point ctl-c says it’s gracefully shutting down, but it never does. Have to ctl-c it a second time.

Terraform Version

Terraform v0.11.7

Terraform Configuration Files

provider "aws" {
  profile = "dz"
}

resource "aws_instance" "example" {
  ami           = "ami-f4533694"
  instance_type = "t2.micro"
}

Debug Output

The following is the result for every attempt to terraform makes to login to aws:

2018-05-10T10:46:37.580-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2018-05-10T10:46:37.580-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: POST / HTTP/1.1
2018-05-10T10:46:37.580-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: Host: sts.amazonaws.com
2018-05-10T10:46:37.580-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: User-Agent: aws-sdk-go/1.13.42 (go1.9.2; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.7
2018-05-10T10:46:37.580-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: Content-Length: 43
2018-05-10T10:46:37.580-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: Authorization: AWS4-HMAC-SHA256 Credential=***redacted***
2018-05-10T10:46:37.580-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: Content-Type: application/x-www-form-urlencoded; charset=utf-8
2018-05-10T10:46:37.580-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: X-Amz-Date: 20180510T174637Z
2018-05-10T10:46:37.581-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: X-Amz-Security-Token: *** redacted ***
2018-05-10T10:46:37.581-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: Accept-Encoding: gzip
2018-05-10T10:46:37.581-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: Action=GetCallerIdentity&Version=2011-06-15
2018-05-10T10:46:37.581-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: -----------------------------------------------------
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: 2018/05/10 10:46:37 [DEBUG] [aws-sdk-go] DEBUG: Response sts/GetCallerIdentity Details:
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: ---[ RESPONSE ]--------------------------------------
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: HTTP/1.1 403 Forbidden
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: Connection: close
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: Content-Length: 297
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: Content-Type: text/xml
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: Date: Thu, 10 May 2018 17:46:37 GMT
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: X-Amzn-Requestid: 16c459b6-547a-11e8-90ea-59844d81b50c
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4:
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4:
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: -----------------------------------------------------
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: 2018/05/10 10:46:37 [DEBUG] [aws-sdk-go] <ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4:   <Error>
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4:     <Type>Sender</Type>
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4:     <Code>ExpiredToken</Code>
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4:     <Message>The security token included in the request is expired</Message>
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4:   </Error>
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4:   <RequestId>16c459b6-547a-11e8-90ea-59844d81b50c</RequestId>
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: </ErrorResponse>

Here is the output after ctl-c:

2018/05/10 10:48:08 [WARN] terraform: Stop called, initiating interrupt sequence
2018/05/10 10:48:08 [WARN] terraform: run context exists, stopping
Interrupt received.
Please wait for Terraform to exit or data loss may occur.
Gracefully shutting down...
stopping operation...
2018-05-10T10:48:08.846-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: plugin received interrupt signal, ignoring: timestamp=2018-05-10T10:48:08.845-0700 count=1

Expected Behavior

Terraform should exit and notify the user their credentials are expired.

Actual Behavior

Terraform never returns

Steps to Reproduce

terraform init terraform plan

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 17
Comments: 22 (6 by maintainers)

Most upvoted comments

Hitting this too. If you are using a system that rotates credentials frequently (like hourly) this gets really annoying.

gkrizek on Sep 20, 2019

Ideally, this error should be exposed to the user rather than only the debug logs:

The security token included in the request is expired

bernadinm on Jan 16, 2019

Ideally, this error should be exposed to the user rather than only the debug logs:
The security token included in the request is expired

I have faced with an issue in terraform 0.12.29. I have received the correct token and can use aws cli but cant forming terraform plan.

dshuvar on Jul 30, 2020

Pretty sure this behavior is coming from the SDK itself and probably related to https://github.com/aws/aws-sdk-go/issues/925

devonbleak on Apr 17, 2019

Related to issues #1351, #1307, #2068, and PR #6992.

tl;dr If credentials expire and the problem cannot be resolved, terraform should report that there is a problem. Currently credential expiration is opaque to users (it isn’t reported at the default logging level).

Currently, when credentials expire, terraform loops until the modification timeout for its active resource expires, then complains about being unable to write its state. This does not clearly convey that there is an issue that needs to be resolved while trying to modify resources, nor does it convey that there was a credential problem earlier on when the persistence of the state file fails. To the user, it appears that the active resource was taking a long time to modify, and as a result credentials expired, resulting in the end of the build at that time. In truth, cause and effect were flipped: the credentials expired, causing the active resource to “take a long time” to modify.

Here are the scenarios when credentials expire midway through apply as I understand them:

Terraform is running in EC2 and was provided a role to assume for credentials. Terraform assumes the role again to get new credentials.
Credentials were obtained through the awscli credentials file or environment variables and cannot be refreshed. Terraform default logging level exposes this to the user, and the user restarts the apply with fresh credentials.

The second use case is not addressed. The first use case may be addressed, but I haven’t tried it so I can’t say.

longbowrocks on Mar 5, 2019

Bug still in latest -

$ terraform version
Terraform v1.0.10
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v3.64.2

jacobm3 on Nov 8, 2021

I think this could be not a bug because a user can refresh credential while Terraform is retrying. But this silent retry is still very annoying. Please print some warning messages while retrying for expired credentials.

kaznak on Jan 22, 2019

In my work environment we authenticate to aws via a Federated connection. I found (the very hard way) that having failed to “login” in this morning terraform plan would ‘hang’ and I had to use Ctrl + C to stop the action. The only information provided with a TF_LOG="ERROR" set in my MacBookPro environment.

Ideal result would be for terraform plan to halt and report an error message to the user reporting connection failure(s). A simple message of “Connection Failure” would be far more informative than the vague log data. As well, I disagree with forcing the user to depend on having to view a log file to obtain this level of data. This does defeat various levels of automation and allowing ‘junior’ or ‘lesser skilled persons’ executing scripts. < My 2 cents …

Log output:


2019/01/18 10:06:22 [ERROR] root: eval: *terraform.EvalConfigProvider, err: unexpected EOF
2019/01/18 10:06:22 [ERROR] root: eval: *terraform.EvalSequence, err: unexpected EOF
2019/01/18 10:06:22 [ERROR] root: eval: *terraform.EvalOpFilter, err: unexpected EOF
2019/01/18 10:06:22 [ERROR] root: eval: *terraform.EvalSequence, err: unexpected EOF
2019/01/18 10:06:22 [TRACE] [walkRefresh] Exiting eval tree: provider.aws

k7faq on Jan 18, 2019