terraform-provider-google: Crash When Doing Init For v4.53.0

Community Note

  • Please vote on this issue by adding a đź‘Ť reaction to the original issue to help the community and maintainers prioritize this request.
  • Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.
  • If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

1.3.1

Affected Resource(s)

  • google_*

Terraform Configuration Files

We set our credentials using the ENV VARs available.

provider google {}

Debug Output

https://gist.github.com/tgoodsell-tempus/bb4ce2a8a3d20752e00a170f1aae4751

Panic Output

Expected Behavior

Provider starts as expected

Actual Behavior

Crashing on init/setup

Steps to Reproduce

Important Factoids

  • Do not use the google provider block directly
  • All configuration is handled by the ENV VARs exposed by the provider

References

  • b/269246896

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 76
  • Comments: 24 (1 by maintainers)

Most upvoted comments

Same error on here as well since today, on apply and plan

+10
miguelpuiggarcia on Feb 14, 2023

4.53.1 is going out now, it should be available within the hour.

Edit: Releases for both providers are complete and 4.53.1 appears in the Terraform Registry (https://registry.terraform.io/providers/hashicorp/google/4.53.1), individual terraform init runs may be hitting a cache in the meantime

I’ve got a reproduction that’s more representative of a typical user configuration than my earlier one (supplying a bad access token). We have logic during provider initialization that logs the user’s identity into local debug output (using a different identity than expected is a common source of customer issue!) that’s expected to silently log an error if it encounters an issue and proceed, failing later in the configuration or apply. That’s largely because that logic requires an additional scope ("userinfo-email") that isn’t required by the rest of the provider. Using an access token that does not include that scope (such as one produced by the steps below) triggers the issue.

During a large code change to the provider initialization logic, the logging code was inadvertently changed. I’m speculating a bit to the exact cause here (other folks on the team drove the investigation), but I believe that we both accessed global state out of order (causing this crash) and threw an error when we encountered an error when checking the Terraform runner’s identity, rather than moving on. That meant that the initial solution in https://github.com/GoogleCloudPlatform/magic-modules/pull/7285, which resolves the crash, still failed at provider initialization time.

Configuration to produce an invalid access token:

data "google_service_account_access_token" "default" {
  target_service_account = "my-service-account@my-project.iam.gserviceaccount.com"
  scopes                 = ["cloud-platform"]
  lifetime               = "3600s"
}

output "token" {
  sensitive = true
  value = nonsensitive(data.google_service_account_access_token.default.access_token)
}

Use that with Terraform:

GOOGLE_OAUTH_ACCESS_TOKEN={{access token}} terraform apply

With the following (simple) configuration:

provider "google" {
  version = "= 4.53.0"
}

resource "google_compute_address" "default" {
  name = "my-address"
}
+9
rileykarson on Feb 14, 2023

Looks like the same issue we also facing with: We are running terraform inside Container image: hashicorp/terraform:1.0.8 Issues seems to be within this image, because also we tried to run on MAC and Ubuntu laptops everything works fine. Bellow is an stack trace we are getting:

$ terraform plan -out "planfile"
â•·
│ Error: Plugin did not respond
│ 
│   with provider["registry.terraform.io/hashicorp/google"],
│   on providers.tf line 1, in provider "google":
│    1: provider "google" {
│ 
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).ConfigureProvider call. The plugin logs may contain
│ more details.
╵
Stack trace from the terraform-provider-google_v4.53.0_x5 plugin:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x5ecc01]
goroutine 168 [running]:
github.com/hashicorp/terraform-plugin-log/internal/logging.GetProviderRootLogger(...)
	github.com/hashicorp/terraform-plugin-log@v0.7.0/internal/logging/provider.go:12
github.com/hashicorp/terraform-plugin-log/tflog.Info({0x0, 0x0}, {0x30d2361, 0x85}, {0x0, 0x0, 0x0})
	github.com/hashicorp/terraform-plugin-log@v0.7.0/tflog/provider.go:74 +0x61
github.com/hashicorp/terraform-provider-google/google.getCurrUserEmail(0xc0001e7800, {0xc000e5a100?, 0x356c300?}, 0xc001084dd0)
	github.com/hashicorp/terraform-provider-google/google/framework_utils.go:44 +0x25f
github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).logGoogleIdentities(_, {_, _}, {{0x0, {0x0, 0x0}}, {0x0, {0x0, 0x0}}, {0x0, ...}, ...}, ...)
	github.com/hashicorp/terraform-provider-google/google/framework_config.go:1219 +0x698
github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).SetupClient(_, {_, _}, {{0x0, {0x0, 0x0}}, {0x0, {0x0, 0x0}}, {0x0, ...}, ...}, ...)
	github.com/hashicorp/terraform-provider-google/google/framework_config.go:1149 +0x28f
github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).ConfigureWithData(_, {_, _}, {{0x0, {0x0, 0x0}}, {0x0, {0x0, 0x0}}, {0x0, ...}, ...}, ...)
	github.com/hashicorp/terraform-provider-google/google/framework_config.go:47 +0x23e
github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).Configure(0xc0001e7800, {0x3575238, 0xc000c0a300}, {{0xc0011061a8, 0x5}, {{{0x357b990, 0xc000c1fb60}, {0x2b3a1e0, 0xc000c1a720}}, {0x357ce58, ...}}}, ...)
	github.com/hashicorp/terraform-provider-google/google/framework_provider.go:821 +0x225
github.com/hashicorp/terraform-plugin-framework/internal/fwserver.(*Server).ConfigureProvider(0xc0004509a0, {0x3575238, 0xc000c0a300}, 0xc001084e40, 0xc001084dc0)
	github.com/hashicorp/terraform-plugin-framework@v1.1.1/internal/fwserver/server_configureprovider.go:15 +0x10f
github.com/hashicorp/terraform-plugin-framework/internal/proto5server.(*Server).ConfigureProvider(0xc0004509a0, {0x3575238?, 0xc000c0a1b0?}, 0x3?)
	github.com/hashicorp/terraform-plugin-framework@v1.1.1/internal/proto5server/server_configureprovider.go:36 +0x292
github.com/hashicorp/terraform-plugin-mux/tf5muxserver.muxServer.ConfigureProvider({0xc000974a80, 0xc000974ae0, {0xc000f67840, 0x2, 0x2}, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...}, ...)
	github.com/hashicorp/terraform-plugin-mux@v0.8.0/tf5muxserver/mux_server_ConfigureProvider.go:25 +0x1b3
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).Configure(0xc000aef680, {0x3575238?, 0xc000c03890?}, 0xc001084a80)
	github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/tf5server/server.go:556 +0x2ce
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_Configure_Handler({0x2f0[88](https://gitlab.com/aerialdev/platform/data-platform/infrastructure/datastream/-/jobs/3767225211#L88)60?, 0xc000aef680}, {0x3575238, 0xc000c03890}, 0xc0010b7f10, 0x0)
	github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:331 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc001012000, {0x357bc20, 0xc001020820}, 0xc000e9d200, 0xc000ffcc60, 0x47d66b8, 0x0)
	google.golang.org/grpc@v1.51.0/server.go:1340 +0xd13
google.golang.org/grpc.(*Server).handleStream(0xc001012000, {0x357bc20, 0xc001020820}, 0xc000e9d200, 0x0)
	google.golang.org/grpc@v1.51.0/server.go:1713 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/grpc@v1.51.0/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc@v1.51.0/server.go:963 +0x28a
Error: The terraform-provider-google_v4.53.0_x5 plugin crashed!
This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.
+6
glazgou on Feb 14, 2023

This seems to be the same bug we have encountered encountered as well.

+6
WTommyG on Feb 14, 2023

same error, work around with following for now:

required_providers {
    google = {
      source  = "hashicorp/google"
      version = ">=4.47.0, != 4.53.0"
    }
}
+3
DesmondH0 on Feb 14, 2023

I am having the same error.

+3
raffaelepaglietti on Feb 14, 2023

Having kinda the same issues/errors here with v4.53.0, everything worked before… I have some nil pointer reference errors:

Stack trace from the terraform-provider-google_v4.53.0_x5 plugin:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x5ecc01]

goroutine 141 [running]:
github.com/hashicorp/terraform-plugin-log/internal/logging.GetProviderRootLogger(...)
	github.com/hashicorp/terraform-plugin-log@v0.7.0/internal/logging/provider.go:12
github.com/hashicorp/terraform-plugin-log/tflog.Info({0x0, 0x0}, {0x30d2361, 0x85}, {0x0, 0x0, 0x0})
	github.com/hashicorp/terraform-plugin-log@v0.7.0/tflog/provider.go:74 +0x61
github.com/hashicorp/terraform-provider-google/google.getCurrUserEmail(0xc0001f3800, {0xc000a9a280?, 0x3570360?}, 0xc000e18690)
	github.com/hashicorp/terraform-provider-google/google/framework_utils.go:44 +0x25f
github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).logGoogleIdentities(_, {_, _}, {{0x0, {0x0, 0x0}}, {0x2, {0xc00005001a, 0x1d3}}, {0x0, ...}, ...}, ...)
	github.com/hashicorp/terraform-provider-google/google/framework_config.go:1219 +0x698
github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).SetupClient(_, {_, _}, {{0x0, {0x0, 0x0}}, {0x2, {0xc00005001a, 0x1d3}}, {0x0, ...}, ...}, ...)
	github.com/hashicorp/terraform-provider-google/google/framework_config.go:1149 +0x28f
github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).ConfigureWithData(_, {_, _}, {{0x0, {0x0, 0x0}}, {0x2, {0xc00005001a, 0x1d3}}, {0x0, ...}, ...}, ...)
	github.com/hashicorp/terraform-provider-google/google/framework_config.go:47 +0x23e
github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).Configure(0xc0001f3800, {0x3575238, 0xc000fd30b0}, {{0xc0001e35d0, 0x6}, {{{0x357b990, 0xc000fe4810}, {0x2b3a1e0, 0xc000fe0b40}}, {0x357ce58, ...}}}, ...)
	github.com/hashicorp/terraform-provider-google/google/framework_provider.go:821 +0x225
github.com/hashicorp/terraform-plugin-framework/internal/fwserver.(*Server).ConfigureProvider(0xc000452580, {0x3575238, 0xc000fd30b0}, 0xc000e18700, 0xc000e18680)
	github.com/hashicorp/terraform-plugin-framework@v1.1.1/internal/fwserver/server_configureprovider.go:15 +0x10f
github.com/hashicorp/terraform-plugin-framework/internal/proto5server.(*Server).ConfigureProvider(0xc000452580, {0x3575238?, 0xc000fd2f60?}, 0x3?)
	github.com/hashicorp/terraform-plugin-framework@v1.1.1/internal/proto5server/server_configureprovider.go:36 +0x292
github.com/hashicorp/terraform-plugin-mux/tf5muxserver.muxServer.ConfigureProvider({0xc0002bd2f0, 0xc0002bd350, {0xc0009803c0, 0x2, 0x2}, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...}, ...)
	github.com/hashicorp/terraform-plugin-mux@v0.8.0/tf5muxserver/mux_server_ConfigureProvider.go:25 +0x1b3
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).Configure(0xc0001fcc80, {0x3575238?, 0xc000fd2630?}, 0xc000e18340)
	github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/tf5server/server.go:556 +0x2ce
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_Configure_Handler({0x2f08860?, 0xc0001fcc80}, {0x3575238, 0xc000fd2630}, 0xc000def0a0, 0x0)
	github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:331 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000ab6000, {0x357bc20, 0xc0006b6820}, 0xc000e1efc0, 0xc000b895f0, 0x47d66b8, 0x0)
	google.golang.org/grpc@v1.51.0/server.go:1340 +0xd13
google.golang.org/grpc.(*Server).handleStream(0xc000ab6000, {0x357bc20, 0xc0006b6820}, 0xc000e1efc0, 0x0)
	google.golang.org/grpc@v1.51.0/server.go:1713 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/grpc@v1.51.0/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc@v1.51.0/server.go:963 +0x28a

Error: The terraform-provider-google_v4.53.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

â•·
│ Error: Plugin did not respond
│ 
│   with provider["registry.terraform.io/hashicorp/google"],
│   on main.tf line 10, in provider "google":
│   10: provider "google" {
│ 
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).ConfigureProvider call. The plugin logs may contain
│ more details.
╵
+3
mr-pascal on Feb 14, 2023

Avoiding 4.53.0 worked for us too as a workaround.

+2
ghost on Feb 14, 2023

Same error:

If you ever set or change modules or backend configuration for Terraform,

rerun this command to reinitialize your working directory. If you forget, other

commands will detect it and remind you to do so if necessary.

  • terraform plan

â•·

│ Error: Plugin did not respond

│

│ with provider[“registry.terraform.io/hashicorp/google”],

│ on main.tf line 1, in provider “google”:

│ 1: provider “google” {

│

│ The plugin encountered an error, and failed to respond to the

│ plugin.(*GRPCProvider).ConfigureProvider call. The plugin logs may contain

│ more details.

╵

Stack trace from the terraform-provider-google_v4.53.0_x5 plugin:

panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x5ecc01]

goroutine 163 [running]:

github.com/hashicorp/terraform-plugin-log/internal/logging.GetProviderRootLogger(…)

github.com/hashicorp/terraform-plugin-log@v0.7.0/internal/logging/provider.go:12

github.com/hashicorp/terraform-plugin-log/tflog.Info({0x0, 0x0}, {0x30d2361, 0x85}, {0x0, 0x0, 0x0})

github.com/hashicorp/terraform-plugin-log@v0.7.0/tflog/provider.go:74 +0x61

github.com/hashicorp/terraform-provider-google/google.getCurrUserEmail(0xc00011d500, {0xc000ac4100?, 0x356c300?}, 0xc000f10c50)

github.com/hashicorp/terraform-provider-google/google/framework_utils.go:44 +0x25f

github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).logGoogleIdentities(, {, _}, {{0x0, {0x0, 0x0}}, {0x0, {0x0, 0x0}}, {0x0, …}, …}, …)

github.com/hashicorp/terraform-provider-google/google/framework_config.go:1219 +0x698

github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).SetupClient(, {, _}, {{0x0, {0x0, 0x0}}, {0x0, {0x0, 0x0}}, {0x0, …}, …}, …)

github.com/hashicorp/terraform-provider-google/google/framework_config.go:1149 +0x28f

github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).ConfigureWithData(, {, _}, {{0x0, {0x0, 0x0}}, {0x0, {0x0, 0x0}}, {0x0, …}, …}, …)

github.com/hashicorp/terraform-provider-google/google/framework_config.go:47 +0x23e

github.com/hashicorp/terraform-provider-google/google.(*frameworkProvider).Configure(0xc00011d500, {0x3575238, 0xc0001dc000}, {{0xc00121e060, 0x5}, {{{0x357b990, 0xc00128b920}, {0x2b3a1e0, 0xc0001fb3b0}}, {0x357ce58, …}}}, …)

github.com/hashicorp/terraform-provider-google/google/framework_provider.go:821 +0x225

github.com/hashicorp/terraform-plugin-framework/internal/fwserver.(*Server).ConfigureProvider(0xc0004b0160, {0x3575238, 0xc0001dc000}, 0xc000f10cc0, 0xc000f10c40)

github.com/hashicorp/terraform-plugin-framework@v1.1.1/internal/fwserver/server_configureprovider.go:15 +0x10f

github.com/hashicorp/terraform-plugin-framework/internal/proto5server.(*Server).ConfigureProvider(0xc0004b0160, {0x3575238?, 0xc001091f20?}, 0x3?)

github.com/hashicorp/terraform-plugin-framework@v1.1.1/internal/proto5server/server_configureprovider.go:36 +0x292

github.com/hashicorp/terraform-plugin-mux/tf5muxserver.muxServer.ConfigureProvider({0xc000983650, 0xc0009836b0, {0xc000faa460, 0x2, 0x2}, {0x0, 0x0, 0x0}, {0x0, 0x0, …}, …}, …)

github.com/hashicorp/terraform-plugin-mux@v0.8.0/tf5muxserver/mux_server_ConfigureProvider.go:25 +0x1b3

github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).Configure(0xc000508d20, {0x3575238?, 0xc001090690?}, 0xc000f108c0)

github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/tf5server/server.go:556 +0x2ce

github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_Configure_Handler({0x2f08860?, 0xc000508d20}, {0x3575238, 0xc001090690}, 0xc000db6000, 0x0)

github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:331 +0x170

google.golang.org/grpc.(*Server).processUnaryRPC(0xc000001680, {0x357bc20, 0xc000bac340}, 0xc000ec4900, 0xc000ff9290, 0x47d66b8, 0x0)

google.golang.org/grpc@v1.51.0/server.go:1340 +0xd13

google.golang.org/grpc.(*Server).handleStream(0xc000001680, {0x357bc20, 0xc000bac340}, 0xc000ec4900, 0x0)

google.golang.org/grpc@v1.51.0/server.go:1713 +0xa1b

google.golang.org/grpc.(*Server).serveStreams.func1.2()

google.golang.org/grpc@v1.51.0/server.go:965 +0x98

created by google.golang.org/grpc.(*Server).serveStreams.func1

google.golang.org/grpc@v1.51.0/server.go:963 +0x28a

Error: The terraform-provider-google_v4.53.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely

helpful if you could report the crash with the plugin’s maintainers so that it

can be fixed. The output above should help diagnose the issue.

+1
rahulparashar01 on Feb 14, 2023
Read more comments on GitHub
← terraform-provider-google: Cloud Run Service Secret Environment Variables, throws spec.template.spec.containers[0].env.value_from should be empty
terraform-provider-google: Creation of Endpoints for Cloud Functions causes circular dependency between ESP and Endpoint services →