terraform-provider-aws: Intermittent error using s3 state

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave “+1” or “me too” comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform 0.11.7 aws 1.17.0

Affected Resource(s)

s3 backend

Terraform Configuration Files

(names have been changed to protect the innocent…)


provider "aws" { 
  region = "${var.Region}"
  version = "~> 1.17.0"
}

terraform {
  backend "s3" {
    profile              = "TheCloud"
    bucket               = "the-cloud-terraform-state"
    key                  = "instance-terraform.tfstate"
    region               = "us-west-2"
    workspace_key_prefix = "instances"
  }
}

# Retrieve state data from S3
data "terraform_remote_state" "state" {
  backend = "s3"

  config {
    profile              = "TheCloud"
    bucket               = "the-cloud-terraform-state"
    key                  = "instance-terraform.tfstate"
    region               = "us-west-2"
    workspace_key_prefix = "instances"
  }
}
...

Output

...
aws_launch_configuration.Instance: Refreshing state... (ID: instance-launch-config-20180531060133610900000001) 
aws_autoscaling_group.Instance: Refreshing state... (ID: Instance-204aaaba-fb33-48e8-88b2-aa190e763b71) 
Error: Error refreshing state: 1 error(s) occurred: 
* data.terraform_remote_state.state: 1 error(s) occurred: 
* data.terraform_remote_state.state: data.terraform_remote_state.state: error loading the remote state: RequestError: send request failed 
caused by: Get https://the-cloud-terraform-state.s3.us-west-2.amazonaws.com/?prefix=instances%2F: dial tcp 54.231.177.36:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. 
...

Expected Behavior

Terraform to work as expected, consistently

Actual Behavior

Intermittent error talking to s3

Steps to Reproduce

Its intermittent 😦

terraform apply

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 81
Comments: 24 (8 by maintainers)

Most upvoted comments

+1 here, hitting lots of this, even with max_retries set to 10 in the aws provider block.

+16

lanmalkieri on Sep 21, 2018

Also seeing this issue intermittently on an EC2 instance in the same region as the s3 bucket. 0.11.8, 1.36.0. Sometimes it is failing to load the main state file from s3 and other times it is failing to load a terraform_remote_state also from s3.

$ terraform init
Error refreshing state: RequestError: send request failed
caused by: Get https://my-s3-bucket.s3.us-west-2.amazonaws.com/path/to/terraform.state: EOF

$ terraform plan
* data.terraform_remote_state.my-remote-state: data.terraform_remote_state.my-remote-state: error loading the remote state: RequestError: send request failed
caused by: Get https://my-s3-bucket.s3.us-west-2.amazonaws.com/?prefix=env%3A%2F: EOF

$ terraform plan
* data.terraform_remote_state.my-remote-state: data.terraform_remote_state.my-remote-state: RequestError: send request failed
caused by: Get https://my-s3-bucket.s3.us-west-2.amazonaws.com/path/to/terraform.state: EOF

And occasionally states are failing to be uploaded which results in the state file and dynamo lock table getting out of sync and needing to be manually repaired:

$ terraform apply terraform.plan
Failed to load backend: Error writing state: failed to upload state: RequestError: send request failed
caused by: Put https://my-s3-bucket.s3.us-west-2.amazonaws.com/path/to/terraform.state: EOF

The s3 backend code (that seems to be used by the terraform_remote_state datasource as well) appears to do some basic retrying but this EOF issue is either not getting caught by s3ErrCodeInternalError or is occurring twice in a row. It would be great to make the retry logic more intelligent with an exponential backoff of some sort. If needed I can try to repro this with more verbose logging so we can get the AWS error code.

rifelpet on Sep 25, 2018

I have submitted a pull request upstream, which should resolve this: https://github.com/hashicorp/terraform/pull/19951

bflad on Jan 9, 2019

My case is on Terraform v0.11.10 and it is not intermitten. All terraform init command failed with same error

Initializing the backend...
Error loading state: RequestError: send request failed
caused by: Get https://[REDACTED].s3.ap-southeast-1.amazonaws.com/?prefix=env%3A%2F: dial tcp 52.219.36.19:443: connect: connection refused

puzzloholic on Dec 5, 2018

We’ve been experiencing the same issue in our CI pipeline, and have narrowed it down to two problems.

1. ListObjects doesn’t retry on any error

2018/09/18 14:01:03 [DEBUG] [aws-sdk-go] DEBUG: Send Request s3/ListObjects failed, not retrying, error RequestError: send request failed
caused by: Get https://s3.eu-central-1.amazonaws.com/OUR-BUCKET?prefix=env%3A%2F: EOF
2018/09/18 14:01:03 [DEBUG] plugin: waiting for all plugin processes to complete...
[31mError loading state: RequestError: send request failed
caused by: Get https://s3.eu-central-1.amazonaws.com/OUR-BUCKET?prefix=env%3A%2F: EOF[0m[0m

See https://github.com/hashicorp/terraform/blob/49d62d3a1b99abf65711f5c8fdf2396931044db3/backend/remote-state/s3/backend_state.go#L29

2. GetObject only retries on S3-specific errors

2018/09/18 15:39:38 [DEBUG] [aws-sdk-go] DEBUG: Send Request s3/GetObject failed, not retrying, error RequestError: send request failed
caused by: Get https://s3.eu-central-1.amazonaws.com/OUR-BUCKET/dev/ae-auth/certificate/terraform.tfstate: EOF

See https://github.com/hashicorp/terraform/blob/49d62d3a1b99abf65711f5c8fdf2396931044db3/backend/remote-state/s3/client.go#L104

/cc @bracki

mlafeldt on Sep 19, 2018

For issues relating to version 1.34.0+ of the AWS provider plan/apply being much slower, you might be interested in this thread potentially related to DNS handling and Go 1.11 (v1.34.0 was the first release on Go 1.11): https://github.com/terraform-providers/terraform-provider-aws/issues/5822#issuecomment-424712521

bflad on Sep 26, 2018

Same here. Yesterday was HORRIBLE with a ton of failures executing terragrunt apply-all across many projects, AWS accounts, and buckets. Things seem improved today, but not completely. We are using terraform 0.11.7, and aws provider 1.37.0. We have backed it down aws provider 1.33.0 as a precaution. Applies seem faster, but no guarantee it solved anything because of the intermittent nature of the issue.

akrapfl on Sep 26, 2018

Seeing this very often when using terragrunt with the apply-all command that runs several Terraform instances in parallel across different modules. It’s so bad that I can’t do deployments from my own laptop anymore and have to instead deploy an EC2 instance and git clone my code there. Is it possible AWS/S3 is throttling? Or is there some concurrency issue with Go?

brikis98 on Sep 25, 2018