terraform-provider-google: Applying IAM policy failed with "Request contains an invalid argument., badRequest" error

Thank you all for your great work!

When I try to apply IAM policy with below terraform config, it fails with googleapi: Error 400: Request contains an invalid argument., badRequest error.

In the debug output, I can see GCP is returning the user email with a case-sensitive manner whereas terraform is using only lowercase characters in the successive request.

I could get the same error message from GCP when I posted the same payload terraform made, but could successfully applied the policy when I updated the payload to include the exact user email.

It looks like not using the exact same email address is causing the issue.

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave “+1” or “me too” comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment
If an issue is assigned to the “modular-magician” user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to “hashibot”, a community member has claimed the issue already.

Terraform Version

Terraform version: 0.12.6
Terraform google plugin version: v2.13.0
Go runtime version: go1.12.7

Affected Resource(s)

google_project_iam_member

Terraform Configuration Files

variable "access_token" {}
variable "project_id" {}
variable "region" {}

provider "google" {
  access_token = var.access_token
  project = var.project_id
}

resource "google_service_account" "foobar" {
  account_id = "foobar"
  display_name = "foobar"
}

resource "google_project_iam_member" "storage_admin_foobar" {
  role = "roles/storage.admin"
  member = "serviceAccount:${google_service_account.foobar.email}"
}

Debug Output

https://gist.github.com/ryu1kn/87986c18a5908afc06d9d816d85bfba6

Expected Behavior

The service account foobar should get storage admin role.

Actual Behavior

The service account foobar doesn’t get storage admin role.

Steps to Reproduce

make access_token, project_id and region variables available and:

terraform init
terraform apply

Important Factoids

I was authenticating as a user instead of a service account

Additional info

curl request I used (replaced other IAM bindings with ... in the JSON payload). Replacing foo.bar.12@gmail.com with Foo.Bar.12@gmail.com will make the request succeed.

curl 'https://cloudresourcemanager.googleapis.com/v1/projects/my-project:setIamPolicy?alt=json&prettyPrint=false' \
    -H 'Content-Type: application/json' \
    -H 'User-Agent: google-api-go-client/0.5 Terraform/0.12.6 (+https://www.terraform.io) terraform-provider-google/2.13.0' \
    -H 'Accept-Encoding: gzip' \
    --compressed \
    -H "Authorization: Bearer $TF_VAR_access_token" \
    -d '{
 "policy": {
  "bindings": [
   {
    "members": [
     "serviceAccount:foobar@my-project.iam.gserviceaccount.com"
    ],
    "role": "roles/storage.admin"
   },
   {
    "members": [
     "user:foo.bar.12@gmail.com"
    ],
    "role": "roles/owner"
   },
   ...
  ],
  "etag": "BwWQPPqvXVw=",
  "version": 1
 },
 "updateMask": "bindings,etag,auditConfigs"
}'

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 28
Comments: 32 (5 by maintainers)

Most upvoted comments

I have a working repro for this, and am looking into a fix.

+12

danawillow on Apr 2, 2020

It looks like I’m also having the same issue where Terraform treats the IAM policy data as case-insensitive, so sends it as all lowecase even though the response it got from the API had case-sensitive information (such as the email address).

In this case I’m using the google_project_iam_member resource and Terraform pulls the current policy, makes changes locally and pushes the new one, which is fine. However when Terraform pushes the new policy it has has changed my email address from EmailAddress@gmail.com to emailaddress@gmail.com and it seems GCP treats this as you changing/removing the EmailAddress user. And since that user has the roles/owner permission it fails as you can’t change that via the API, only via the Console.

If I make the same request that Terraform is making but using the API Explorer then it fails. But if I change the email back to EmailAddress@gmail.com it works as expected.

Terraform Trace Log (click me!)

google_project_iam_member.pubsub: Creating...
2020/02/02 23:02:33 [DEBUG] google_project_iam_member.pubsub: applying the planned Create change
2020/02/02 23:02:33 [TRACE] GRPCProvider: ApplyResourceChange
2020-02-02T23:02:33.076Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: 2020/02/02 23:02:33 [DEBUG] Batching is disabled, sending single request for "Create IAM Members projects/project-id-here/roles/tf_pubsub serviceAccount:terraform@project-id-here.iam.gserviceaccount.com for \"project \\\"project-id-here\\\"\""
2020-02-02T23:02:33.076Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: 2020/02/02 23:02:33 [DEBUG] Locking "iam-project-project-id-here"
2020-02-02T23:02:33.076Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: 2020/02/02 23:02:33 [DEBUG] Locked "iam-project-project-id-here"
2020-02-02T23:02:33.076Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: 2020/02/02 23:02:33 [DEBUG]: Retrieving policy for project "project-id-here"
2020-02-02T23:02:33.076Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: 2020/02/02 23:02:33 [DEBUG] Google API Request Details:
2020-02-02T23:02:33.076Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: ---[ REQUEST ]---------------------------------------
2020-02-02T23:02:33.076Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: POST /v1/projects/project-id-here:getIamPolicy?alt=json&prettyPrint=false HTTP/1.1
2020-02-02T23:02:33.076Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Host: cloudresourcemanager.googleapis.com
2020-02-02T23:02:33.076Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: User-Agent: google-api-go-client/0.5 HashiCorp Terraform/0.12.18 (+https://www.terraform.io) Terraform Plugin SDK/1.4.0 terraform-provider-google/3.6.0
2020-02-02T23:02:33.076Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Content-Length: 3
2020-02-02T23:02:33.076Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Content-Type: application/json
2020-02-02T23:02:33.076Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: X-Goog-Api-Client: gl-go/1.13.0 gdcl/20191026
2020-02-02T23:02:33.076Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Accept-Encoding: gzip
2020-02-02T23:02:33.076Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: 
2020-02-02T23:02:33.076Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: {}
2020-02-02T23:02:33.076Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: 
2020-02-02T23:02:33.076Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: -----------------------------------------------------
2020-02-02T23:02:34.120Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: 2020/02/02 23:02:34 [DEBUG] Google API Response Details:
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: ---[ RESPONSE ]--------------------------------------
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: HTTP/2.0 200 OK
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Cache-Control: private
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Content-Type: application/json; charset=UTF-8
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Date: Sun, 02 Feb 2020 23:02:34 GMT
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Server: ESF
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Server-Timing: gfet4t7; dur=735
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Vary: Origin
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Vary: X-Origin
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Vary: Referer
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: X-Content-Type-Options: nosniff
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: X-Frame-Options: SAMEORIGIN
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: X-Xss-Protection: 0
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: 
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: {
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:  "version": 1,
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:  "etag": "BwWdn769RKc=",
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:  "bindings": [
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:   {
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:    "role": "roles/owner",
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:    "members": [
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:     "user:EmailAddress@gmail.com"
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:    ]
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:   }
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:  ]
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: }
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: -----------------------------------------------------
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: 2020/02/02 23:02:34 [DEBUG]: Retrieved policy for project "project-id-here": &{AuditConfigs:[] Bindings:[0xc0003c13e0] Etag:BwWdn769RKc= Version:1 ServerResponse:{HTTPStatusCode:200 Header:map[Alt-Svc:[quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000] Cache-Control:[private] Content-Type:[application/json; charset=UTF-8] Date:[Sun, 02 Feb 2020 23:02:34 GMT] Server:[ESF] Server-Timing:[gfet4t7; dur=735] Vary:[Origin X-Origin Referer] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[0]]} ForceSendFields:[] NullFields:[]}
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: 2020/02/02 23:02:34 [DEBUG]: Setting policy for project "project-id-here" to &{AuditConfigs:[] Bindings:[0xc0003c14a0 0xc0003c1500] Etag:BwWdn769RKc= Version:1 ServerResponse:{HTTPStatusCode:200 Header:map[Alt-Svc:[quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000] Cache-Control:[private] Content-Type:[application/json; charset=UTF-8] Date:[Sun, 02 Feb 2020 23:02:34 GMT] Server:[ESF] Server-Timing:[gfet4t7; dur=735] Vary:[Origin X-Origin Referer] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[0]]} ForceSendFields:[] NullFields:[]}
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: 2020/02/02 23:02:34 [DEBUG] Google API Request Details:
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: ---[ REQUEST ]---------------------------------------
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: POST /v1/projects/project-id-here:setIamPolicy?alt=json&prettyPrint=false HTTP/1.1
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Host: cloudresourcemanager.googleapis.com
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: User-Agent: google-api-go-client/0.5 HashiCorp Terraform/0.12.18 (+https://www.terraform.io) Terraform Plugin SDK/1.4.0 terraform-provider-google/3.6.0
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Content-Length: 292
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Content-Type: application/json
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: X-Goog-Api-Client: gl-go/1.13.0 gdcl/20191026
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Accept-Encoding: gzip
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: 
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: {
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:  "policy": {
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:   "bindings": [
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:    {
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:     "members": [
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:      "user:emailaddress@gmail.com"
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:     ],
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:     "role": "roles/owner"
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:    },
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:    {
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:     "members": [
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:      "serviceAccount:terraform@project-id-here.iam.gserviceaccount.com"
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:     ],
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:     "role": "projects/project-id-here/roles/tf_pubsub"
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:    }
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:   ],
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:   "etag": "BwWdn769RKc=",
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:   "version": 1
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:  },
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:  "updateMask": "bindings,etag,auditConfigs"
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: }
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: 
2020-02-02T23:02:34.121Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: -----------------------------------------------------
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: 2020/02/02 23:02:34 [DEBUG] Google API Response Details:
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: ---[ RESPONSE ]--------------------------------------
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: HTTP/2.0 400 Bad Request
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Cache-Control: private
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Content-Type: application/json; charset=UTF-8
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Date: Sun, 02 Feb 2020 23:02:34 GMT
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Server: ESF
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Server-Timing: gfet4t7; dur=371
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Vary: Origin
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Vary: X-Origin
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: Vary: Referer
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: X-Content-Type-Options: nosniff
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: X-Frame-Options: SAMEORIGIN
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: X-Xss-Protection: 0
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: 
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: {
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:   "error": {
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:     "code": 400,
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:     "message": "Request contains an invalid argument.",
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:     "errors": [
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:       {
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:         "message": "Request contains an invalid argument.",
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:         "domain": "global",
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:         "reason": "badRequest"
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:       }
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:     ],
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:     "status": "INVALID_ARGUMENT"
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5:   }
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: }
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: 
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: -----------------------------------------------------
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: 2020/02/02 23:02:34 [DEBUG] Unlocking "iam-project-project-id-here"
2020-02-02T23:02:34.531Z [DEBUG] plugin.terraform-provider-google_v3.6.0_x5: 2020/02/02 23:02:34 [DEBUG] Unlocked "iam-project-project-id-here"
2020/02/02 23:02:34 [DEBUG] google_project_iam_member.pubsub: apply errored, but we're indicating that via the Error pointer rather than returning it: Error applying IAM policy for project "project-id-here": Error setting IAM policy for project "project-id-here": googleapi: Error 400: Request contains an invalid argument., badRequest

MisguidedEmails on Feb 2, 2020

Worked with the IAM team internally to get this resolved on the API side, which means the fix should now work in all versions of the provider 🎉

Give it a try! Closing this out now that it seems to be fixed.

danawillow on Apr 13, 2020

For now, I’m avoiding this issue by pinning to the previous version (i.e. v2.12.0).

provider "google" {
  version = "2.12.0"
  access_token = var.access_token
  project = var.project_id
}

ryu1kn on Aug 25, 2019

I had a similar issue, it was happening in 2.11, 2.12, and 2.16.

I was using this:

data "google_service_account_access_token" "create_key_keyring" {
  target_service_account = "${google_service_account.service_account_creator.email}"
  delegates              = ["${google_service_account.terraform_cloudbuilder.name}"]
  scopes                 = ["cloud-platform"]
  lifetime               = "300s"
}

And replaced it with this to get it working:

data "google_service_account_access_token" "create_key_keyring" {
  target_service_account = "${google_service_account.service_account_creator.email}"
  delegates              = ["projects/-/serviceAccounts/${google_service_account.terraform_cloudbuilder.email}"]
  scopes                 = ["cloud-platform"]
  lifetime               = "300s"
}

It might have something to do with the project name being included in the request, but I can only confirm that for my case when using delegates to get the token.

captain-kark on Oct 4, 2019

@edwardmedia I encountered this problem earlier last week and took some time to look at this. It seems to me that the bug was indeed introduced when PR https://github.com/GoogleCloudPlatform/magic-modules/pull/1957 was merged to fix #3880. The issue then was that case sensitivity in IAM members causes a change loop, so that PR renders all IAM members case insensitive. As to how that PR relates to this bug, I think https://github.com/terraform-providers/terraform-provider-google/issues/4276#issuecomment-581190176 covered this well.

I was able to workaround this bug by removing roles/owner from users with non-lowercase emails. So, conversely, I think you should be able to reproduce this bug if you add any non-lowercase emails to IAM and assign roles/owner to it via UI, and then apply whatever IAM modifications you desire via Terraform.

Thank you for picking this up!

loozhengyuan on Mar 28, 2020

So I believe the same error can occur with google_project_iam_binding and google_project_iam_member.

After a little bit more investigation, I noticed that running the same policy update via Terraform vs the gcloud CLI generates slightly different error messages. (Does it show different messages based on API client?)

My Terraform definition is as follows:

resource "google_project_iam_binding" "ip_manager_user_role_binding" {
    provider = google-beta
    role    = "projects/${var.project_name}/roles/${google_project_iam_custom_role.test_role.role_id}"
    members = ["serviceAccount:${google_service_account.test_sa.email}"]
}

Using the gcloud CLI, I get:

$ gcloud projects set-iam-policy myproject-456 ./test2.json
ERROR: (gcloud.projects.set-iam-policy) INVALID_ARGUMENT: Request contains an invalid argument.
- '@type': type.googleapis.com/google.cloudresourcemanager.v1.ProjectIamPolicyError
  member: user:some_user2@gmail.com
  role: roles/owner
  type: SOLO_MUST_INVITE_OWNERS
- '@type': type.googleapis.com/google.cloudresourcemanager.v1.ProjectIamPolicyError
  member: user:some_user1@gmail.com
  role: roles/owner
  type: SOLO_MUST_INVITE_OWNERS

(The ./test2.json was a HTTP request payload taken from the debug output of a previous Terraform apply.)

Using Terraform:

2020-01-21T22:58:26.100-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5: ---[ RESPONSE ]--------------------------------------        
2020-01-21T22:58:26.100-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5: HTTP/2.0 400 Bad Request    
2020-01-21T22:58:26.100-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5: Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
2020-01-21T22:58:26.100-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5: Cache-Control: private    
2020-01-21T22:58:26.100-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5: Content-Type: application/json; charset=UTF-8    
2020-01-21T22:58:26.100-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5: Date: Wed, 22 Jan 2020 04:58:25 GMT    
2020-01-21T22:58:26.100-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5: Server: ESF    
2020-01-21T22:58:26.100-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5: Server-Timing: gfet4t7; dur=387    
2020-01-21T22:58:26.100-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5: Vary: Origin    
2020-01-21T22:58:26.100-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5: Vary: X-Origin    
2020-01-21T22:58:26.100-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5: Vary: Referer    
2020-01-21T22:58:26.101-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5: X-Content-Type-Options: nosniff    
2020-01-21T22:58:26.101-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5: X-Frame-Options: SAMEORIGIN    
2020-01-21T22:58:26.101-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5: X-Xss-Protection: 0    
2020-01-21T22:58:26.101-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5:     
2020-01-21T22:58:26.101-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5: {    
2020-01-21T22:58:26.101-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5:   "error": {    
2020-01-21T22:58:26.101-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5:     "code": 400,    
2020-01-21T22:58:26.101-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5:     "message": "Request contains an invalid argument.",        
2020-01-21T22:58:26.101-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5:     "errors": [    
2020-01-21T22:58:26.101-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5:       {    
2020-01-21T22:58:26.101-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5:         "message": "Request contains an invalid argument.",        
2020-01-21T22:58:26.101-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5:         "domain": "global",    
2020-01-21T22:58:26.101-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5:         "reason": "badRequest"    
2020-01-21T22:58:26.101-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5:       }    
2020-01-21T22:58:26.101-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5:     ],    
2020-01-21T22:58:26.101-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5:     "status": "INVALID_ARGUMENT"    
2020-01-21T22:58:26.101-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5:   }    
2020-01-21T22:58:26.101-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5: }    
2020-01-21T22:58:26.101-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5:     
2020-01-21T22:58:26.101-0600 [DEBUG] plugin.terraform-provider-google-beta_v3.4.0_x5: -----------------------------------------------------

Note that my Terraform resource does not even mention the users in the error message shown in the gcloud CLI.

It turns out that this SOLO_MUST_INVITE_OWNERS error message is returned if at least one of the following is true:

A new user is being granted an owner role via the API. (This is not allowed, according to the API Docs)
The user has not accepted the invitation to be an owner of the project/org/account.

Once I removed the offending users, the Terraform resource was created successfully. Hopefully this is helpful to some of you…

clee231 on Jan 22, 2020

I’ve got a similar issue with google_project_iam_member and google_project_iam_binding, and probably other google_project_iam_*. I’m on

Terraform v0.12.12
+ provider.google v2.18.1

So the issue is that I’m trying to grant a role (let’s say “roles/storage.objectCreator”) to a service account, but the projects IAM policy has the following entry (which I don’t touch with my change):

- members:
  - user:User.With.Capital@gmail.com
  role: roles/owner

for some reason once terraform tries to apply the new policy with the projects.setIamPolicy API call, it lowercases all members. So, in this case, GCP treats that as a new owner role member added and rejects the change due to A user cannot be granted the owner role using setIamPolicy(). restriction.

It is definitely a bug in the provider, it should not modify the IAM policy.

snowytoxa on Oct 31, 2019