terraform-provider-google: google_apikeys_key missing X-Goog-User-Project (thus doesn't work with ADC)

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
  • Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.
  • If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

1.1.5

Affected Resource(s)

  • google_apikeys_key

Terraform Configuration Files

resource "google_apikeys_key" "key" {
  name         = "mykey"
  display_name = "mykey"
}

Debug Output

[...]
-----------[REQUEST]----------
GET /v2/projects/REDACTED/locations/global/keys/mykey?alt=json HTTP/1.1
Host: apikeys.googleapis.com
User-Agent: Terraform/1.1.5 (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google/dev DeclarativeClientLib/0.0.1
Content-Type: application/json
Accept-Encoding: gzip


-------[END REQUEST]--------: timestamp=2022-06-10T14:51:22.198+0200
[...]
2022-06-10T14:51:22.975+0200 [INFO]  provider.terraform-provider-google_v4.21.0_x5: 2022/06/10 14:51:22 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2022-06-10T14:51:22.975+0200
2022-06-10T14:51:22.975+0200 [INFO]  provider.terraform-provider-google_v4.21.0_x5: 2022/06/10 14:51:22 [DEBUG][DCL INFO] [RequestID:]  Google API Response: (id dkbr2) 
-----------[RESPONSE]----------
HTTP/2.0 403 Forbidden
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Fri, 10 Jun 2022 12:51:22 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
  "error": {
    "code": 403,
    "message": "Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the apikeys.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/. If you are getting this error with curl or similar tools, you may need to specify 'X-Goog-User-Project' HTTP header for quota and billing purposes. For more information regarding 'X-Goog-User-Project' header, please check https://cloud.google.com/apis/docs/system-parameters.",
    "status": "PERMISSION_DENIED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "SERVICE_DISABLED",
        "domain": "googleapis.com",
        "metadata": {
          "service": "apikeys.googleapis.com",
          "consumer": "projects/REDACTED"
        }
      }
    ]
  }
}

-------[END RESPONSE]--------: timestamp=2022-06-10T14:51:22.975+0200
2022-06-10T14:51:22.975+0200 [INFO]  provider.terraform-provider-google_v4.21.0_x5: 2022/06/10 14:51:22 [DEBUG][DCL INFO] Error appears not to be retryable: googleapi: Error 403: Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the apikeys.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/. If you are getting this error with curl or similar tools, you may need to specify 'X-Goog-User-Project' HTTP header for quota and billing purposes. For more information regarding 'X-Goog-User-Project' header, please check https://cloud.google.com/apis/docs/system-parameters.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "googleapis.com",
    "metadata": {
      "consumer": "projects/REDACTED",
      "service": "apikeys.googleapis.com"
    },
    "reason": "SERVICE_DISABLED"
  }
]: timestamp=2022-06-10T14:51:22.975+0200
2022-06-10T14:51:22.976+0200 [ERROR] vertex "module.mykeymodule.google_apikeys_key.key" error: Error when reading or editing ApikeysKey "projects/REDACTED/locations/global/keys/mykey": googleapi: Error 403: Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the apikeys.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/. If you are getting this error with curl or similar tools, you may need to specify 'X-Goog-User-Project' HTTP header for quota and billing purposes. For more information regarding 'X-Goog-User-Project' header, please check https://cloud.google.com/apis/docs/system-parameters.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "googleapis.com",
    "metadata": {
      "consumer": "projects/REDACTED",
      "service": "apikeys.googleapis.com"
    },
    "reason": "SERVICE_DISABLED"
  }
]
[...]

Panic Output

Expected Behavior

Able to terraform plan google_apikeys_key when using ADC and not logged in as a GCP SA.

Actual Behavior

$ terraform plan
[...]
╷
│ Error: Error when reading or editing ApikeysKey "projects/REDACTED/locations/global/keys/mykey": googleapi: Error 403: Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported by the apikeys.googleapis.com. We recommend configuring the billing/quota_project setting in gcloud or using a service account through the auth/impersonate_service_account setting. For more information about service accounts and how to use them in your application, see https://cloud.google.com/docs/authentication/. If you are getting this error with curl or similar tools, you may need to specify 'X-Goog-User-Project' HTTP header for quota and billing purposes. For more information regarding 'X-Goog-User-Project' header, please check https://cloud.google.com/apis/docs/system-parameters.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/REDACTED",
│       "service": "apikeys.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│
│   with module.mykeymodule.google_apikeys_key.key,
│   on ../../modules/gcp-api-key/main.tf line 1, in resource "google_apikeys_key" "key":
│    1: resource "google_apikeys_key" "key" {
│
╵

Steps to Reproduce

  1. terraform plan

Important Factoids

Authenticating as a user, with ADC.

The equivalent works fine when either using a service account, or when configuring the provider for example like this:

provider "google" {
  project               = var.gcp_project
  region                = var.gcp_region
  billing_project       = var.gcp_project
  user_project_override = true
}

References

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 16
  • Comments: 15

Most upvoted comments

Ahh the error message they give back encompasses the fix. Looks like they want you to set a billing project.

Try doing that on your provider cofig

provider "google" {
  project               = "some-project"
  region                = "us-central1"
  billing_project       = "some-project"
  user_project_override = true
}
+1
ScottSuarez on Oct 6, 2022

Thanks for reporting this bug! It looks like this is an issue with how the DCL makes its requests.

Here are my notes for future reference while looking into this bug:

  • x-goog-user-project is set on the core Config object.
  • google_apikeys_key is a DCL-based resource that reuses config.client, so it should have the same client as the core config, including any additional headers.
  • However, the DCL uses loggingTransport to round-trip its requests - as opposed to the Terraform provider, which uses headerTransportLayer as client.Transport. Since the DCL doesn’t call client.Get, it maybe sidesteps using client.Transport and doesn’t set the correct headers? But I’m having trouble tracking how it’s actually using the client.
  • b/236042741

Question for folks experiencing this: What provider version are you using? It looks like the user agent is being reported as terraform-provider-google/dev instead of a version number. The DCL version seems to be reported as 0.0.1, which is very out of date.

We are using provider version 4.24

+1
ghost on Jun 16, 2022
Read more comments on GitHub
← terraform-provider-google: Google provider latest version 4.42.0 is crashing
terraform-provider-google: google_bigquery_data_transfer_config.query_config example is not working →