terraform-provider-google: google_project_iam_member is slow when upgrading >= v3.14

When upgrading the google provider I noticed one of the workflow I am working one takes a lot longer to execute. Not even the actual deployment phase, just the terraform plan. The “refreshing state” step took ~10 minutes before and now it takes 3 hours. I probed the different versions available and found out that this issue started for v3.14

I have reduced this issue to a single component google_project_iam_member, to make things easier to talk about and reproduce. This is a list of ~100 members to deploy in iam. With v3.13, terraform plan takes <1 min to execute with v3.14 and higher, terraform plan takes 8 minutes to execute.

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v0.12.28

provider.google v3.38.0

Affected Resource(s)

google_project_iam_member

Terraform Configuration Files

resource "google_project_iam_member" "project_members" {
  count = "${length(var.members)}"

  project = "${var.project}"

  role   = "roles/browser"
  member = "${element(var.members, count.index)}"

}

Debug Output

https://gist.github.com/FrancoisPoinsot/07f597e7d3e5aef3705b31114e4b0b23

Expected Behavior

terraform plan would take <1 minute to execute

Actual Behavior

terraform plan takes 8 minutes to execute

Steps to Reproduce

create a var members of ~100 members
actually deploy them
execute terraform plan

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 3
Comments: 20 (4 by maintainers)

Most upvoted comments

I suspect https://github.com/hashicorp/terraform-provider-google/pull/5902 may have introduced this performance regression and will verify tomorrow.

nat-henderson on Oct 23, 2020

I ran a small test:

reduced the member list in the state to 12
reduced the member list in the variable to 0
run time terraform plan
remove one member from the state

repeated the last 2 step until I can plot a graph repeated the whole process with v3.13 and v3.14. Ok I did not plot all the point of 3.14 because I just did not want to wait the whole morning in front of my screen.

Here are the results:

member length in state	v3.13  	v3.14

0                     	1.21  	1.234
1                     	13.499	13.707
2                     	13.985	19.492
3                     	14.53
4                     	14.93
5                     	15.142	36.684
6                     	15.519
7                     	15.999
8                     	16.356	54.528
9                     	16.306
10                    	17.097	1:06.31
11                    	17.644	1:11.65
12                    	18.243	1:16.88

Reminder, my test project is still this only resource:

resource "google_project_iam_member" "project_members" {
  count = "${length(var.members)}"

  project = "${var.project}"

  role   = "roles/browser"
  member = "${element(var.members, count.index)}"

}

It looks to me there is an issue with how v3.14 (and higher) scales with the state size.

FrancoisPoinsot on Oct 22, 2020