terraform-provider-aws: [Bug]: Assigning multiple role policies using aws_iam_role_policy_attachment fails with ConcurrentModification error
Terraform Core Version
1.6.2
AWS Provider Version
5.25.0
Affected Resource(s)
aws_iam_role_policy_attachment
Expected Behavior
It should be possible to assign multiple policies to the same role.
Actual Behavior
When multiple policies are assigned, it very often fails with ConcurrentModification exception. It may be related to some changes in AWS, since the same code, with the same version of terraform and aws provider was working last week.
Relevant Error/Panic Output Snippet
Error: attaching policy arn:aws:iam::12545454:policy/test-policy-5 to IAM Role test-role: ConcurrentModification: Another request updating the entity is in progress. Please try again later.
│ status code: 409, request id: aaaaa-bbbbb-ccccc-ddddd-fea34e635b6a
│
│ with aws_iam_role_policy_attachment.test_attach[5],
│ on init.tf line 101, in resource "aws_iam_role_policy_attachment" "test_attach":
│ 101: resource "aws_iam_role_policy_attachment" "test_attach" {
Terraform Configuration Files
variable "profile" {}
variable "region" {}
provider "aws" {
profile = var.profile
region = var.region
}
terraform {
required_version = ">= 1.0"
backend "s3" {
key = "spikes/role-benchmark/terraform.tfstate"
}
required_providers {
aws = {
source ="hashicorp/aws"
version = "= 5.25.0"
}
}
}
data "aws_iam_policy_document" "assume_role" {
statement {
effect = "Allow"
principals {
type = "Service"
identifiers = ["ec2.amazonaws.com"]
}
actions = ["sts:AssumeRole"]
}
}
resource "aws_iam_role" "role" {
name = "test-role"
assume_role_policy = data.aws_iam_policy_document.assume_role.json
}
data "aws_iam_policy_document" "policy" {
statement {
effect = "Allow"
actions = ["ec2:Describe*"]
resources = ["*"]
}
}
resource "aws_iam_policy" "policy" {
count = 8
name = "test-policy-${count.index}"
description = "A test policy"
policy = data.aws_iam_policy_document.policy.json
}
resource "aws_iam_role_policy_attachment" "test_attach" {
count = 8
role = aws_iam_role.role.name
policy_arn = aws_iam_policy.policy[count.index].arn
}
Steps to Reproduce
apply the provided terraform. Or create new terraform, which assigns 8 polices to a role
Debug Output
No response
Panic Output
No response
Important Factoids
No response
References
No response
Would you like to implement a fix?
None
About this issue
- Original URL
- State: closed
- Created 8 months ago
- Reactions: 18
- Comments: 18 (9 by maintainers)
Commits related to this issue
- service/iam: retry attaching policy on ConcurrentModificationException Fixes #34371 Signed-off-by: Frank Lichtenheld <frank@lichtenheld.com> — committed to flichtenheld/terraform-provider-aws by flichtenheld 8 months ago
Got some updates from AWS side:
Hope that helps
FTR the error went away yesterday afternoon so I contacted AWS to get some explanation. I was told that recently AWS made changes in IAM API, in particular AttachRolePolicy, AttachUserPolicy, and AttachGroupPolicy started to throw ConcurrentModificationException in case of concurrent requests. After the change AWS noticed elevated ConcurrentModification errors when calling IAM APIs. Because of that the issue has been resolved and the service is operating normally. Will try to clarify if “resolved” means that the feature was rolled back or something else.
terraform apply -parallelism=1works around the problem.Yes, I tried bisecting this. But found no version with which it actually works. So it looks indeed like AWS-side change.
I updated last week from 5.20.0 to 5.24.0 and started to see these errors. So I think this was introduced earlier than 5.25.0
I have started seeing this with version 3.76.1. I suspect AWS IAM behaviour has changed.
This does not fail reliably for me. So it seems to depend on a race condition. Something to keep in mind when trying to bisect.
Community Note
Voting for Prioritization
Volunteering to Work on This Issue