terraform-provider-aws: Enable Aurora Enhanced Monitoring on existing instances - "IAM role invalid or wrong perms", followed by "MonitoringRoleARN value is required"

This issue was originally opened by @ssm1th as hashicorp/terraform#9435. It was migrated here as part of the provider split. The original body of the issue is below.

Similar issue as @deanmraz reported in hashicorp/terraform#5455 I think.

Attempting to enable Enhanced Monitoring on existing Aurora instances initially throws the following error:

InvalidParameterValue: IAM role ARN value is invalid or does not include the required permissions for: ENHANCED_MONITORING

When attempting an apply again following that error, I’m getting:

InvalidParameterCombination: A MonitoringRoleARN value is required if you specify a MonitoringInterval value other than 0

I suspect the latter is being thrown because the first apply actually added monitoring_role_arn to tfstate.

Have tried using the existing AWS provided managed policy and attaching it to a Terraform created role (all within Terraform), and I’ve also tried creating my own policy as per hashicorp/terraform#5455 and attaching that to a self made role too, but the behaviour is still the same.

The addition of monitoring_role_arn to tfstate following the first error sounds like a bug to me, but does anybody have any ideas or suggestions on how to get around this and get it working?

Thanks.

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 3
Comments: 21 (2 by maintainers)

Commits related to this issue

resource/aws_db_instance: Ensure monitoring attributes are always written to state and retry ModifyDBInstance on IAM eventual consistency error References: - https://github.com/terraform-providers/t... — committed to hashicorp/terraform-provider-aws by bflad 5 years ago
resource/aws_rds_cluster_instance: Ensure monitoring attributes are always written to state Reference: https://github.com/terraform-providers/terraform-provider-aws/issues/315 Output from acceptance... — committed to hashicorp/terraform-provider-aws by bflad 5 years ago
Merge upstream (#1) * provider: Ensure needs-triage issue labeling step in GitHub Actions v2 is conditional on opened action The payload is slightly different in v2, so actions/github@v1.0.0 fails... — committed to paulrigor/terraform-provider-aws by paulrigor 5 years ago

Most upvoted comments

In our experience, we have an RDS instance that we created via terraform a few weeks ago. When we tried adding enhanced monitoring by setting monitoring_interval and monitoring_role_arn at the same time, terraform failed with the error FIXME. Subsequent runs always failed with InvalidParameterCombination: A MonitoringRoleARN value is required if you specify a MonitoringInterval value other than 0., unless we modified the tfstate file to forget about the monitoring_* properties.

The initial failure we suspect is due to either the role not being created properly before its ARN was applied to the RDS instance (is that even possible?), or that the assumerole policy of our role was using the service rds.amazonaws.com instead of monitoring.rds.amazonaws.com.

Once we had role created with monitoring.rds.amazonaws.com as our AssumeRolePolicy Service, and removed the monitoring settings from our tfstate, terraform was able to successfully add enhanced monitoring to the RDS instance.

geofffranks on Dec 15, 2017

Yeah, got the same issue. There’s two problems here:

The first issue is a race condition problem: the RDS declaration has a direct dependency on the monitoring role, so as soon as the role is available, the modification of the RDS is fired. So the modification of the RDS (adding the monitoring role) and the aws_iam_role_policy_attachment are happening at the same time. RDS is checking if the role has indeed the right to use enhanced monitoring, but at that moment, that’s not the case, hence the first error.

I tried to fix this issue by forcing a dependency on aws_rds_instance to aws_iam_role_policy_attachment but in my tests, even if the calls are done sequentially, they happen so fast that RDS is still checking too soon if the role has the correct right, leading to the same error.

Here’s a breakdown of first run of TF, with the manual dependency on aws_iam_role_policy_attachment:

request creation of the role @ 2018-11-09T12:40:35.128+0100
answer HTTP 200 @ 2018-11-09T12:40:35.813+0100
request attachment policy AmazonRDSEnhancedMonitoringRole@ 2018-11-09T12:40:36.481+0100
answer HTTP 200 @ 2018-11-09T12:40:37.188+0100
request iam/ListAttachedRolePolicies (I suppose to confirm that the attachment is correctly done) @ 2018-11-09T12:40:37.848+0100
answer HTTP 200 with the policy AmazonRDSEnhancedMonitoringRole listed @ 2018-11-09T12:40:38.495+0100
request to modify the DB instance and add the enhanced monitoring @ 2018-11-09T12:40:38.509+0100
answer HTTP 400 Bad Request @ 2018-11-09T12:40:39.498+0100. Reason is IAM role ARN value is invalid or does not include the required permissions for: ENHANCED_MONITORING

As you can see, there’s less than 2s between the confirmation that the policy is correctly attached, and the call to modify the instance with the call. I suspect some cache on AWS side, that is not refreshed when we make the call to modify the RDS. Policy attachment is clearly done when we initiate the DB modify API call.

The second problem is a Terraform bug: whatever the outcome of the first call, the monitoring_role_arn attribute is set to the ARN, so next calls are only pushing the monitoring_interval. As a consequence, the monitoring ARN is not sent in the AWS API call, making the request incorrect.

TL;DR: this is the combination of two bugs, one on AWS side and one on TF side.

sterfield on Nov 9, 2018

I’ve opened an issue on AWS side for this. I’ll keep you guys posted, but looking at this forum thread, it’s very likely that IAM information is being propagated but haven’t reached yet the RDS API.

In that case, it means that TF may need to retry the RDS call a few times to see if the RDS api finally get the correct answer.

sterfield on Nov 12, 2018

Also facing this issue curently. When aws is handling this?

flmmartins on Jun 21, 2019

The fix for this has been merged and will release in version 2.25.0 of the Terraform AWS Provider, in about a week.

bflad on Aug 15, 2019

I’m encountering the same exact issue as @hegyre. I have an existing Postgres 10 instance and when I apply the monitoring_role_arn it never gets set which results in the exception.

InvalidParameterCombination: A MonitoringRoleARN value is required if you specify a MonitoringInterval value other than 0.

What’s the status on this issue?

brycehemme on Jul 17, 2019

Did this get addressed somehow? I am currently facing the exact same issues reported above.

albrechtsimon on Aug 23, 2017