terraform-provider-aws: AWS security groups not being destroyed

This issue occurs when renaming a security group in Terraform and also updating the aws_instance resource to reference the security group by its new name.

Terraform tries to do the following:

1. Destroy existing security group
2. Create new security group
3. Modify EC2 instance (disassociate old SG and associate new SG)

This causes (1) to hang because the AWS API prevents deleting a SG that’s still associated with an instance. (2) succeeds, and (3) is never executed.

The correct order of actions should be:

1. Create new security group
2. Modify EC2 instance (disassociate old SG and associate new SG)
3. Destroy existing security group

I didn’t find any working solutions, except during the hanging going to the AWS Management Console Actions > Networking > Change Security Groups, disassociating the old security group, and associating the new security group.

With create_before_destroy = true in SG it works.

resource "aws_security_group" "intercluster" {
  name   = "some sg"
  vpc_id = var.vpc_id

  lifecycle {
    create_before_destroy = true
  }
<snip>

this is what I am trying to avoid - doing things manually in the console. if a security group contains a second security group and we do terraform destroy, the second SG rule should be removed, the second SG destroyed and then the first. Seems so logical to me.

To add to this thread. I often find upon a second run of the destroy command the previously endlessly hanging security_group deletion task will immediately delete. Perhaps suggesting Terraform needs to perform some sort of re-calculation at each retry attempt?

Has 0.12 introduced anything that’d allow for a fix to this? Having to manually alter >80 security groups via the console is not a pleasant experience.

Hi @szczad! Sorry for this limitation.

At the moment Terraform doesn’t have any mechanism to deal with these “enforced dependencies” in the underlying service, and indeed it affects a number of resources, such as what we see in #646, #151, #2201.

The problem is that this requires some extra coordination between operations – creating multiple coordinated steps as you mentioned – which Terraform’s provider model isn’t currently able to represent. In certain cases such dependencies could be represented in principle – for example, in this case where Terraform is managing both the security group and the other resources that belong to it – while in other cases Terraform can’t “see” the dependency at all, because e.g. the EC2 instances are being created implicitly by an aws_autoscaling_group.

We would like to find a way to address this limitation in the long run, for certain situations at least, but at this time we do not have a suitable design figured out to deal with it, and our current Terraform Core development focus is elsewhere. For now it is, as you noted, required to manually break the dependency somehow before making these changes, which is definitely not ideal.

In principle we could improve the behavior here by at least producing a helpful error when this situation arises. Unfortunately the long-running polling behavior you saw here was introduced to work around a different problem: network interfaces tend to live for a few minutes after their associated resources are destroyed, acting as dependencies on the security group that aren’t reflected in the API at all. Terraform therefore retries here so that it can wait until VPC has finished deallocating the network interface before proceeding, rather than failing with a hard error in that case.

Thanks @BcTpe4HbIu

  lifecycle {
    create_before_destroy = true
  }

worked in my case.

create_before_destroy = true doesn’t help. To destroy my SG, i need to MANUALLY remove the other SG from the rules. Simple test code available here

Terraform v0.12.26

• provider.aws v2.65.0

The issue is the the default SG is detroyed BEFORE the EC2 SG is and in fact it’s not true - the default SG is still there even if Terraform says “destroyed” (see 2nd line below)

module.VPCs.aws_default_security_group.default: Destroying... [id=sg-02057153bb443e13d]
module.VPCs.aws_default_security_group.default: Destruction complete after 0s
module.VPCs.aws_security_group.GC-SG-VPC-test: Destroying... [id=sg-02a06934c19a8efaa]

The GC-SG-VPC-test security group is part the the default SG rules !!!

Our scenario/workaround: TF couldn’t destroy/replace a security group because it was still attached to an ALB. We had to do 3 separate TF runs:

1. Create new security groups and attach them to ALB
2. Remove old SGs from ALB
3. Remove SG resources

This avoids horrid manual console/statefile intervention. Hope this helps someone!

This worked for me:

1. Add

  lifecycle {
    create_before_destroy = true
  }

to the old security group while renaming it.

1. Run terraform - it will create the new security group, do the attachments and it will hang at terminating the old security group.

2. Go to the AWS console and attach the new group, detach the old group from your targets. Ensure that old group attachments are 0.

3. Terraform will like that and delete the old group thereby not messing up your state file.