terraform-provider-aws: S3 bucket slow to delete when destroyed during an apply
Community Note
- Please vote on this issue by adding a đź‘Ť reaction to the original issue to help the community and maintainers prioritize this request
- Please do not leave “+1” or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Terraform Version
- Terraform version: 0.12.9
- AWS provider version: 2.42
Affected Resource(s)
- aws_s3_bucket
Terraform Configuration Files
resource "aws_s3_bucket" "cdn_logs_bucket" {
bucket = "cdn.logs.${local.fqdn}"
acl = "private"
force_destroy = "true"
tags = "${var.tags}"
server_side_encryption_configuration {
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "AES256"
}
}
}
}
Expected Behavior
S3 bucket destroyed in a timely manner (e.g., within minutes)
Actual Behavior
S3 bucket takes multiple hours to destroy
Steps to Reproduce
- Create an S3 bucket via Terraform
terraform apply- Put some objects in the bucket
- Remove S3 bucket from Terraform
Important Factoids
- I’ve encountered this a few times and was able to delete the S3 bucket manually via the AWS console before Terraform finished deleting it.
- S3 bucket destroys during a
terraform destroyseem to work as expected.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 117
- Comments: 18 (6 by maintainers)
@mratoms Of what order of magnitude is the number of objects (including all versions) in the bucket? Internally the provider ends up listing all the object versions in a bucket and deleting them one at a time when the bucket is deleted. We could look at deleting objects in batches via the
DeleteObjectsAPI.Hi, Was there any update on this? I haven’t tried the new version of the provider but this is really blocking us to migrate to new version and try out the AWS features released in newer versions.
This issue first appears in v2.29.0 and looks related to the changes in #9942 - where
resource/aws_s3_bucket_objectlocks were introduced. For v2.28.1 and below, objects were deleted in batch API calls. From v2.29.0 onwards each object is deleted one at a time, leading to far more API calls and poorer efficiency.The changelog for v2.29.0 doesn’t suggest there’s any change in behaviour for
resource/aws_s3_bucket, only enhancements toresource/aws_s3_bucket_object. So I assume this was unintended.This issue can even be seen with as few as 200 objects in a bucket. terraform-provider-aws v2.28.1 takes ~11 seconds to
terraform destroy, whereas v2.29.0 takes considerably longer at ~38 seconds. And testing with 5,000 objects took 15 seconds on v2.28.1, and now takes 12 minutes on v2.29.0.Debug logging also shows v2.28.1 is deleting multiple objects in a single s3/DeleteObjects API call:
Whereas, terraform-provider-aws v2.29.0 is seen deleting single objects in multiple s3/DeleteObjects API call (there’s 200 s3/DeleteObjects API calls in the 200 objects example):
There doesn’t seem to be an elegant workaround for this issue and its limiting our ability to upgrade.
In my case the issue can be replicated with just 5000 objects in bucket without versioning enabled. Took around 30 minutes. Used 0.12.21 terraform and AWS terraform provider v2.49.0
Hi all 👋 Just letting you know that this is issue is featured on this quarters roadmap. If a PR exists to close the issue a maintainer will review and either make changes directly, or work with the original author to get the contribution merged. If you have written a PR to resolve the issue please ensure the “Allow edits from maintainers” box is checked. Thanks for your patience and we are looking forward to getting this merged soon!
@ojongerius Yes, it’s in my queue.
We found a really simple workaround that I have documented here: https://gist.github.com/bassmanitram/53a57988d9f4e4a2ffc94aed789bbf9e
When testing with a bucket containing 190,000 objects, this technique deleted the content and the bucket in about 30 minutes. Without this workaround, the terraform destroy command was still running 12 hours later with still over 75000 objects left in the bucket (I got bored waiting and deleted them via the AWS S3 console just to put TF out of its misery).
NOTE that this WON’T work for buckets with object versioning enabled since the CLI s3 rm command simply places a deleted marker in the object history rather than actually getting rid of the object.
I’m seeing this issue in 0.12.21. The order of magnitude of the objects in the bucket is hundreds of thousands. Deleting the bucket through the web console works just fine, but Terraform just says “Still destroying…” forever.