terraform-provider-aws: Applying aws_s3_bucket_policy and aws_s3_bucket_public_access_block at the same time may cause an error

Using depends_on is a good way to force the public access block and policy to be applied one-by-one instead of concurrently:

resource "aws_s3_bucket_policy" "this" {
  depends_on = ["aws_s3_bucket_public_access_block.this"]

This worked for me.

I think that depends_on would work as a workaround, but it does not actually depend on it, we should implement an appropriate error handling. Since similar problems can occur with many s3 related resources, I’m not sure how do we handle it is the best.

We’ve been intermittently experiencing a variety of OperationAborted: A conflicting conditional operation... errors when creating new S3 buckets. For example:

• Error putting S3 versioning
• Error putting S3 notification configuration
• error creating public access block policy for S3 bucket
• Error putting S3 lifecycle
• error putting S3 server side encryption configuration

In our case, we typically aren’t creating aws_s3_bucket_policy resources and have not seen from triaging Terraform debug logs that parallel calls to the S3 service are being made for the same bucket.

In each of the cases we’ve triaged, the S3 bucket can eventually be created and set with the desired configuration if enough retries are attempted. Generally, a single retry within a minute of the first failed try has been sufficient. In some cases, though, we’ve had wait for up to 45 minutes before a retry succeeds.

From descriptions I’ve seen of this error, it sounds like a common condition that would trigger this is recreating an S3 bucket which has just just been deleted or hitting a soft limit on the number of S3 buckets in an account. I don’t believe either has been true in our case, though, since the number of buckets have been well under our account limit around the time of some failures and the buckets we’re creating are most often unique.

In the implementation the aws_s3_bucket in the Terraform AWS provider code, I’m not seeing code which would seem to be causing parallel S3 API calls for the same bucket to be made. We have seen at least occasionally that when Terraform makes an apparently successful Put call to an S3 API from the resourceAwsS3BucketUpdate function that the corresponding Get call made from the resourceAwsS3BucketRead function later might fail. For example, we saw a PutBucketLifecycleConfiguration call return an HTTP 200 OK response but the subsequent GetBucketLifecycleConfiguration call for the same bucket returned an HTTP 404 Not Found response. Perhaps this is due to eventual consistency issues with the AWS S3 service. The calls made from within resourceAwsS3BucketUpdate generally don’t appear to follow a Put with a Get to confirm that the S3 service has committed the desired configuration. Where multiple Put calls could be strung together (e.g., when encryption, lifecycle, and/or other features are all enabled together for the same S3 service), I wonder if adding Gets for each of the Puts would help serialize the configuration changes in AWS (and, therefore, help avoid “conflicting conditional operations”). It’s hard to tell if this would help, though, since it’s not really clear from the AWS error messaging what the “conflicting conditional operations” are in the event of a failure.

For the S3 server side encryption configuration one, we had put up a PR, which would add retries for 409 errors, although that PR has been up for over 10 months now without any attention. Perhaps it would make sense to add similar retries for each of the S3 API calls that the AWS provider could make. Unfortunately, though, the retry timeout might need to be very high (45 minutes or longer?) in order to reliably overcome the errors. It would be nice to figure out the sequences of events from interactions with the AWS S3 API that produce these errors so that we could avoid them (either in Terraform code or, at least where possible, within the Terraform AWS provider).

This is also experienced with the aws_s3_bucket_ownership_controls resource. I think the eventual consistency is the more likely issue and TF should just backoff and retry on this error message.

It seems this is also an issue when managing aws_s3_bucket_notification resources.

This should be resolved with #12949. However, since this issue is tricky to test, please open a new issue (and reference this issue) if you continue to see problems!

Some sort ordering seems needed, a related issue is that if you have AWS Guardduty enabled and you terraform destroy an s3 bucket you will get a security alert that the “Block public access policies” have been removed because they are removed before the bucket is removed rather than just destroying the bucket first.

seeing this on terraform 1.0.4 and AWS provider 0.57.0 as well

The same with aws_s3_bucket_public_access_block and aws_s3_bucket_ownership_controls configured for the bucket. I overcome this by making aws_s3_bucket_ownership_controls explicit depends_on aws_s3_bucket_public_access_block

Hi all 👋 Just letting you know that this is issue is featured on this quarters roadmap. If a PR exists to close the issue a maintainer will review and either make changes directly, or work with the original author to get the contribution merged. If you have written a PR to resolve the issue please ensure the “Allow edits from maintainers” box is checked. Thanks for your patience and we are looking forward to getting this merged soon!

Either that or sort out planned API calls to S3 so that they happen in sequence.

IMO documenting the use of depends_on to serialize creation would be sufficient.