terraform-provider-aws: err="rpc error: code = Unavailable desc = transport is closing"

Setting up this GODEBUG environment variable before launching terraform fixes it for me.

export GODEBUG=asyncpreemptoff=1;

Running Terraform 0.14.11

Terraform v0.14.11
+ provider registry.terraform.io/hashicorp/archive v2.2.0
+ provider registry.terraform.io/hashicorp/aws v3.72.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/hashicorp/template v2.2.0

I see the same issue on

Terraform v0.13.5
+ provider registry.terraform.io/hashicorp/aws v3.22.0

This is also on a M1 Mac, could be important

I have thumbs upped this, but wanted to add that I would commit a bounty of a couple hundreds dollars for someone who can fix this ASAP.

I’ve been struggling for weeks with this problem. Been trying terraform 0.13 and 0.14 of various versions, and various versions of the aws provider. The problem for me is intermittent, where an “apply” just hangs for a very long time (~5m) while gathering facts and with trace logging on I just see a bunch of this rpc error transport is closing. I’m having to do terraform apply multiple times before it finally goes through, but meanwhile this is frustrating me to no end.

This is impacting my daily life as I live and breathe Terraform. I would be available to try dev/beta builds as well and give feedback, as I’ve tried everything besides jumping into code at this point and I’m still frustrated and wasting tons of time. I suspect that not only a logging resolution would need to be made, but if the logging fix would give me more clarity on why my runs fail so regularly I could fix the actual problem whatever that may be.

Cheers, hoping for a speedy resolution!

I encountered a similar error but it was related to this issue which is caused by a bug in the cloudflare provider version 2.19.1.

Pinning the cloudflare provider to 2.19.0 fixed the issue for me

Hello all, for those of you encountering the error message err="rpc error: code = Unavailable desc = transport is closing", are you running Terraform on either Kubernetes or an M1 Mac?

In the Kubernetes case, the error is often associated with running out of memory.

On M1 Macs, it appears that there is an issue in Rosetta 2 (the system that allows running amd64 executables on arm64). See also https://yaleman.org/post/2021/2021-01-01-apple-m1-terraform-and-golang/. This can be resolved by setting the environment variable GODEBUG=asyncpreemptoff=1. This might also be resolved by ensuring that you are using an arm64-native version of Terraform and the AWS provider.

If you are encountering this error on platforms other than Kubernetes or M1 Macs, please include the output of terraform version, including the machine architecture.

Hello, It’s seems that @rastakajakwanna is right, this issue comes from a not found data. However I’ve rolled back to version 2.70 of aws provider ant it adds where :

Version 3.26 Error: Your query returned no results. Please change your search criteria and try again.

Version 2.70

Error: Your query did not return any results. Please try a different search criteria.

  on base.tf line 267, in data "aws_autoscaling_group" "my_asg":
 267: data "aws_autoscaling_group" "my_asg" {

So it looks like a logging regression since user is now unable to find quickly what is the missing “data” resource.

Getting the same issue with aws provider 3.24.1, (issue tested also with 3.23.0, 3.22.0, 3.21.0 and failing) and tf 0.12.29.

The issue is inconsistent, it fails only for one remote state (stage in our case) and not for the others. Provider versions configuration are the same. It also fails at different places - I can see kubernetes provider failing, next time it is random provider…

I’ve also deleted all non-default workspaces as suggested in linked issue but no change.

Trace log shows just the following and I could find randomly failing providers with the rpc error in earlier part of the log.

2021/01/19 19:49:19 [TRACE] dag/walk: upstream of "provider.aws (close)" errored, so skipping
2021/01/19 19:49:19 [TRACE] dag/walk: upstream of "root" errored, so skipping

Also no change whether I am using proxy or not.

Is it possible that changes applied with the aws.provider 3.24.0 have somehow messed up the state somewhere? (regarding https://github.com/hashicorp/terraform-provider-aws/issues/17125 )

I can also see a lot of rpc errors of three kinds:

[WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unimplemented desc = unknown service plugin.GRPCStdio"
[WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
[WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Canceled desc = context canceled"

Terraform refresh is crashing around different resources than plan, however, both of them are crashing always around IAM policy attachments “aws_iam_role_policy_attachment”. But it does not fail if I target these resources with plan.

Our issue has been resolved. It was a data resource which was looking for a deprecated AMI. I am only wondering why I could not find the exact source of the issue in the TRACE log of Terraform.

@agustinlare : I would bet that you’re running out of resources on whatever platform you’re building on. Are you using Codebuild? If so, increase memory and CPU allocation and see if that helps.

Facing same issue on doing cdktf get,

2021-10-20T17:19:53.794+0100 [TRACE] GRPCProvider: GetProviderSchema
⠧ downloading and generating modules and providers...
2021-10-20T17:19:53.890+0100 [TRACE] GRPCProvider: Close
2021-10-20T17:19:53.891+0100 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-10-20T17:19:53.892+0100 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/aws/3.63.0/linux_amd64/terraform-provider-aws_v3.63.0_x5 pid=7564
2021-10-20T17:19:53.893+0100 [DEBUG] provider: plugin exited

I was following guide from https://learn.hashicorp.com/tutorials/terraform/cdktf-build-python?in=terraform/cdktf and it did not work for me.

terraform: v1.0.9 cdktf: 0.7.0 python: 3.8.5 os: Ubuntu 20.04.1LTS

I just got this when it was trying to set an S3 bucket policy to one that was invalid because it had a zero length array of principals.

I was only able to work that out by setting TF_LOG to TRACE, otherwise I didn’t get enough detail in the logs and it just kept going for ever.

      "Principal": {
        "AWS": []
      }
      

@cdimitroulas You are my hero mate. Hopefully this also addresses OP’s concern.

I can confirm, I needed to update Terraform from 0.12 to 0.14, and I had to rollback the Cloudflare plugin from 2.19.1 to 2.19.0, and it worked well.

The error message was related to an AWS EC2 instance, but it really was the Cloudflare plugin rollback that fixed the issue. Thanks for the input @cdimitroulas!

Receiving the same error when when feeding a module that calls another module output from another module on provider 3.23.0.

// main.tf
module "network" {
  source = "../../../../modules/vpc/basic"
  input vars...
}

module "instance_type" {
  source = "../../../../modules/instance_type"

  vpc_id                       = module.network.vpc_id
  subnet_id                    = module.network.public_subnets[0]
  additional_policy = {
    json = data.aws_iam_policy_document.policy.json
  }
}

My debug shows the plan completed before the error.

2021/01/15 11:56:43 [INFO] backend/local: plan operation completed

Error: Your query returned no results. Please change your search criteria and try again.



Error: Your query returned no results. Please change your search criteria and try again.


2021-01-15T11:56:43.867-0600 [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-01-15T11:56:43.867-0600 [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-01-15T11:56:43.867-0600 [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-01-15T11:56:43.868-0600 [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-01-15T11:56:43.868-0600 [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-01-15T11:56:43.873-0600 [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/aws/3.23.0/darwin_amd64/terraform-provider-aws_v3.23.0_x5 pid=24068
2021-01-15T11:56:43.874-0600 [DEBUG] plugin: plugin exited
2021-01-15T11:56:43.874-0600 [DEBUG] plugin: plugin process exited: path=/usr/local/Cellar/tfenv/2.0.0/versions/0.14.2/terraform pid=24064
2021-01-15T11:56:43.874-0600 [DEBUG] plugin: plugin exited