terraform-provider-aws: Plugin crashes when plan and apply

@francisATgwn I had exactly the same problem the other day. I use tfenv for managing terraform versions, so for me the fix was to

• set environment variable TFENV_ARCH to arm64
• re-install terraform
• run terraform init to get arm64-versions of plugins

Hope that helps!

FYI, this has happened to me for months with the x64 executable on an M1 mac. I just painfully have to terraform apply/plan many times, sometimes 4-5 times before it works properly. Is painful to use. Switching over to the ARM-based executable seems to have fixed the problem though.

Hello,

I (and another colleague) ran into the same problem. Here is a small example of the behaviour:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.65.0"
    }
  }
}

provider "aws" {
  region = "eu-central-1"
}

resource "aws_iam_role" "test_role" {
  name = "test_role"

  # Terraform's "jsonencode" function converts a
  # Terraform expression result to valid JSON syntax.
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Sid    = ""
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      },
    ]
  })

  tags = {
    tag-key = "tag-value"
  }
}

➜  $ terraform init         

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 3.65.0"...
- Installing hashicorp/aws v3.65.0...
- Installed hashicorp/aws v3.65.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
➜  $ terraform -v
Terraform v1.0.11
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v3.65.0
➜  $ terraform plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following
symbols:
  + create

Terraform will perform the following actions:

  # aws_iam_role.test_role will be created
  + resource "aws_iam_role" "test_role" {
      + arn                   = (known after apply)
      + assume_role_policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = "sts:AssumeRole"
                      + Effect    = "Allow"
                      + Principal = {
                          + Service = "ec2.amazonaws.com"
                        }
                      + Sid       = ""
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + create_date           = (known after apply)
      + force_detach_policies = false
      + id                    = (known after apply)
      + managed_policy_arns   = (known after apply)
      + max_session_duration  = 3600
      + name                  = "test_role"
      + name_prefix           = (known after apply)
      + path                  = "/"
      + tags                  = {
          + "tag-key" = "tag-value"
        }
      + tags_all              = {
          + "tag-key" = "tag-value"
        }
      + unique_id             = (known after apply)

      + inline_policy {
          + name   = (known after apply)
          + policy = (known after apply)
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run
"terraform apply" now.
➜  $ terraform plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following
symbols:
  + create

Terraform will perform the following actions:

  # aws_iam_role.test_role will be created
  + resource "aws_iam_role" "test_role" {
      + arn                   = (known after apply)
      + assume_role_policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = "sts:AssumeRole"
                      + Effect    = "Allow"
                      + Principal = {
                          + Service = "ec2.amazonaws.com"
                        }
                      + Sid       = ""
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + create_date           = (known after apply)
      + force_detach_policies = false
      + id                    = (known after apply)
      + managed_policy_arns   = (known after apply)
      + max_session_duration  = 3600
      + name                  = "test_role"
      + name_prefix           = (known after apply)
      + path                  = "/"
      + tags                  = {
          + "tag-key" = "tag-value"
        }
      + tags_all              = {
          + "tag-key" = "tag-value"
        }
      + unique_id             = (known after apply)

      + inline_policy {
          + name   = (known after apply)
          + policy = (known after apply)
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run
"terraform apply" now.
➜  $ terraform plan
╷
│ Error: Plugin did not respond
│ 
│   with provider["registry.terraform.io/hashicorp/aws"],
│   on main.tf line 28, in provider "aws":
│   28: provider "aws" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).GetProviderSchema call. The plugin logs may
│ contain more details.
╵
➜  $ terraform plan
╷
│ Error: Could not load plugin
│ 
│ 
│ Plugin reinitialization required. Please run "terraform init".
│ 
│ Plugins are external binaries that Terraform uses to access and manipulate
│ resources. The configuration provided requires plugins which can't be located,
│ don't satisfy the version constraints, or are otherwise incompatible.
│ 
│ Terraform automatically discovers provider requirements from your
│ configuration, including providers used in child modules. To see the
│ requirements and constraints, run "terraform providers".
│ 
│ failed to retrieve schema from provider "registry.terraform.io/hashicorp/aws": Request cancelled: The
│ plugin.(*GRPCProvider).GetProviderSchema request was cancelled.
│ 
╵
➜  $ terraform plan
╷
│ Error: Could not load plugin
│ 
│ 
│ Plugin reinitialization required. Please run "terraform init".
│ 
│ Plugins are external binaries that Terraform uses to access and manipulate
│ resources. The configuration provided requires plugins which can't be located,
│ don't satisfy the version constraints, or are otherwise incompatible.
│ 
│ Terraform automatically discovers provider requirements from your
│ configuration, including providers used in child modules. To see the
│ requirements and constraints, run "terraform providers".
│ 
│ failed to instantiate provider "registry.terraform.io/hashicorp/aws" to obtain schema: Unrecognized remote plugin message: 
│ 
│ This usually means that the plugin is either invalid or simply
│ needs to be recompiled to support the latest protocol.
│ 
╵
➜  $ terraform plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following
symbols:
  + create

Terraform will perform the following actions:

  # aws_iam_role.test_role will be created
  + resource "aws_iam_role" "test_role" {
      + arn                   = (known after apply)
      + assume_role_policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = "sts:AssumeRole"
                      + Effect    = "Allow"
                      + Principal = {
                          + Service = "ec2.amazonaws.com"
                        }
                      + Sid       = ""
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + create_date           = (known after apply)
      + force_detach_policies = false
      + id                    = (known after apply)
      + managed_policy_arns   = (known after apply)
      + max_session_duration  = 3600
      + name                  = "test_role"
      + name_prefix           = (known after apply)
      + path                  = "/"
      + tags                  = {
          + "tag-key" = "tag-value"
        }
      + tags_all              = {
          + "tag-key" = "tag-value"
        }
      + unique_id             = (known after apply)

      + inline_policy {
          + name   = (known after apply)
          + policy = (known after apply)
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run
"terraform apply" now.

As you can see, it’s not consistent at all. Successive calls to plan doesn’t return the same error. I attached 2 stack traces here

I use also a M1 chip. Terraform was installed with tfenv and seems to run under Rosetta 2.

➜  $ uname -mprsv
Darwin 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:20 PDT 2021; root:xnu-7195.141.6~3/RELEASE_ARM64_T8101 arm64 arm

I don’t have time to investigate further right now unfortunately. Hope it helps!

Closing this Issue as newer versions of the provider and/or macOS seems to have stopped the problem occurring.

If you are using tfenv, you must enter this value in the environment variable.

export TFENV_ARCH=arm64

So darwin_arm64 should be output in terraform version.

$ tfenv uninstall 1.1.3
Uninstall Terraform v1.1.3
Terraform v1.1.3 is successfully uninstalled

$ export TFENV_ARCH=arm64

$ tfenv install
Installing Terraform v1.1.3
Downloading release tarball from https://releases.hashicorp.com/terraform/1.1.3/terraform_1.1.3_darwin_arm64.zip
########################################################################################################################################################################### 100.0%
Downloading SHA hash file from https://releases.hashicorp.com/terraform/1.1.3/terraform_1.1.3_SHA256SUMS
No keybase install found, skipping OpenPGP signature verification
Archive:  tfenv_download.Ezq9ad/terraform_1.1.3_darwin_arm64.zip
  inflating: /opt/homebrew/Cellar/tfenv/2.2.2/versions/1.1.3/terraform
Installation of terraform v1.1.3 successful. To make this your default version, run 'tfenv use 1.1.3'

$ terraform -v
Terraform v1.1.3
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v3.71.0

@kblcuk I’m using tfenv also. I tried that and it works. I’ll see if this helps decrease or eliminate the provider errors.

@smithatlanta Are you running with an M1 mac?

If you are using tfenv, you must enter this value in the environment variable.

export TFENV_ARCH=arm64

So darwin_arm64 should be output in terraform version.

$ tfenv uninstall 1.1.3
Uninstall Terraform v1.1.3
Terraform v1.1.3 is successfully uninstalled

$ export TFENV_ARCH=arm64

$ tfenv install
Installing Terraform v1.1.3
Downloading release tarball from https://releases.hashicorp.com/terraform/1.1.3/terraform_1.1.3_darwin_arm64.zip
########################################################################################################################################################################### 100.0%
Downloading SHA hash file from https://releases.hashicorp.com/terraform/1.1.3/terraform_1.1.3_SHA256SUMS
No keybase install found, skipping OpenPGP signature verification
Archive:  tfenv_download.Ezq9ad/terraform_1.1.3_darwin_arm64.zip
  inflating: /opt/homebrew/Cellar/tfenv/2.2.2/versions/1.1.3/terraform
Installation of terraform v1.1.3 successful. To make this your default version, run 'tfenv use 1.1.3'

$ terraform -v
Terraform v1.1.3
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v3.71.0

that’s a valid solution! but @nalbam, I think it would be good to fix this flakyness either way. I prefer to run the amd64 version of terraform because my colleagues, atlantis, jenkins, github actions, and every other place where terraform is being executed is using the amd64 binaries and I don’t want to deal with these differences and manage differences on the lock files. On top of that there’s some deprecated providers such as template that don’t have binaries for arm64 and having to refactor everything all at once would be a pain. 😞

About the providers that don’t have the binaries for arm64: Even some that are not deprecated don’t support it. F.i. datadog.

@kblcuk Thank you so much for the TFENV_ARCH fix. I’ve been fighting with terraform and terra grunt for a month since I got my M1 thinking it was some issue with the new AWS provider or something because SURELY tfenv and tgenv would’ve installed the proper version of everything right? right??

This should be in the docs.