terraform-provider-google: google_bigquery_dataset_iam_member doesn't handle deleted members

Hi @joerayme, here are the steps to reproduce it:

1. Create a dataset and two service accounts. Assign them roles on the dataset through google_bigquery_dataset_iam_member (in the example below I had to create the google_bigquery_dataset_iam_member resources after applying other resources to avoid the Terraform error: The “for_each” value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created.)

terraform {
  required_version = ">= 0.13"

  required_providers {
    google      = "~> 3.49.0"
    google-beta = "~> 3.49.0"
  }
}

resource "google_service_account" a_service_account {
  account_id = "a-service-account"
}

resource "google_service_account" another_service_account {
  account_id = "another-service-account"
}

resource "google_bigquery_dataset" a_dataset {
  dataset_id = "a_dataset"
  location   = "US"
}

locals {
  service_accounts = [
    google_service_account.a_service_account.email,
    google_service_account.another_service_account.email,
  ]
}

resource google_bigquery_dataset_iam_member "reader" {
  for_each   = toset(local.service_accounts)
  dataset_id = google_bigquery_dataset.a_dataset.dataset_id
  member     = "serviceAccount:${each.key}"
  role       = "roles/bigquery.dataViewer"
}

1. Manually deleted one of the two service accounts and perform an additional change on the IAM permissions of the dataset. In my example, I’m trying to delete the IAM role of the still existing service account:

terraform {
  required_version = ">= 0.13"

  required_providers {
    google      = "~> 3.49.0"
    google-beta = "~> 3.49.0"
  }
}

resource "google_service_account" a_service_account {
  account_id = "a-service-account"
}

// I DELETED THIS MANUALLY
//resource "google_service_account" another_service_account {
//  account_id = "another-service-account"
//}

resource "google_bigquery_dataset" a_dataset {
  dataset_id = "a_dataset"
  location   = "US"
}

locals {
  service_accounts = [
//    google_service_account.a_service_account.email,
//    google_service_account.another_service_account.email,
  ]
}

resource google_bigquery_dataset_iam_member "reader" {
  for_each   = toset(local.service_accounts)
  dataset_id = google_bigquery_dataset.a_dataset.dataset_id
  member     = "serviceAccount:${each.key}"
  role       = "roles/bigquery.dataViewer"
}

The diff of terraform apply is correct:

Terraform will perform the following actions:

  # google_bigquery_dataset_iam_member.reader["a-service-account@ra-testing-42.iam.gserviceaccount.com"] will be destroyed
  - resource "google_bigquery_dataset_iam_member" "reader" {
      - dataset_id = "a_dataset" -> null
      - id         = "projects/ra-testing-42/datasets/a_dataset/roles/bigquery.dataViewer/serviceaccount:a-service-account@ra-testing-42.iam.gserviceaccount.com" -> null
      - member     = "serviceAccount:a-service-account@ra-testing-42.iam.gserviceaccount.com" -> null
      - project    = "ra-testing-42" -> null
      - role       = "roles/bigquery.dataViewer" -> null
    }

Plan: 0 to add, 0 to change, 1 to destroy.

However, when applying changes I get this error:

google_bigquery_dataset_iam_member.reader["a-service-account@ra-testing-42.iam.gserviceaccount.com"]: Destroying... [id=projects/ra-testing-42/datasets/a_dataset/roles/bigquery.dataViewer/serviceaccount:a-service-account@ra-testing-42.iam.gserviceaccount.com]

Error: Error when reading or editing Resource projects/ra-testing-42/datasets/a_dataset for IAM Member (role "serviceAccount:a-service-account@ra-testing-42.iam.gserviceaccount.com", "roles/bigquery.dataViewer"): Error applying IAM policy for Bigquery Dataset ra-testing-42/a_dataset: Failed to parse BigQuery Dataset IAM member type: deleted:serviceAccount:another-service-account@ra-testing-42.iam.gserviceaccount.com?uid=115133331797674156835

Interesting enough, the bug doesn’t seem to happen if using two separate google_bigquery_dataset_iam_member resources (i.e. not using the for_each)

@edwardmedia GCP added the deleted: prefix after a change in September 14, 2020. This was quite an important change as people received information about it by email. These are some interesting parts from the email that I received at that time:

Roles that are granted to deleted accounts will be clearly labeled as deleted accounts, allowing you to differentiate between roles granted to active or deleted accounts with a particular given email.

[...]

If you manage IAM Policies with infrastructure as code tools such as Terraform:
* When a user, group, or service account is labeled as deleted on IAM policy, you should remove the deleted member from your master copy of the policy (this step is already required after accounts are irrevocably purged). Once an account is purged, setting IAM Policy with references to the account is not supported.
* You may also need to update configuration-as-code tools to recognize the new prefix. For example, if you use the Google provider for Terraform, be sure you have updated to at least version 3.3.0.

You may also need to update configuration-as-code tools to recognize the new prefix.

I think that the error Failed to parse BigQuery Dataset IAM member type: deleted:serviceAccount:<service-account>@<account>.iam.gserviceaccount.com is a clear indication that action needs to be taken in the provider to be compliant with Google’s guidelines

Exactly. This exact issue has cropped up for us; it’s why I opened the ticket. Things happen and my Terraform shouldn’t break because someone accidentally deleted a service account or even offboarded a member of staff that’s managed by some other bit of Terraform. Deleted members are handled elsewhere and a bug related to this was even opened pre-emptively (#7278) so I don’t see why this should be any different.

@joerayme on a side note, I also think that this will fail in the context of this toy example, but that could be a separate issue:

1. Create a service account A
2. Assign IAM permission to it
3. Delete the service account A
4. Re-create the service account A with the same name (aka email)
5. Re-apply the IAM permission to it <- FAIL

The reason why you want to perform step 5 (i.e. reassign the permission) is that GCP internally trackes IAM using service account IDs, which are unique even across re-creation of a seervice account with the same name. But, when applying, I think that GCP will complain about the unfeasible existence of both the deleted and the new member in the IAM policy, which is refused. To make it work you first have to remove the deleted: entry from the IAM policy than apply the new one.