pulumi: delete-before-replace triggers delete-before-replace in all dependencies

Following up from this community Slack thread, there is a bug where:

  • The user changes data in a manually-named Kubernetes ConfigMap, which triggers a delete-before-replace.
  • The user references that ConfigMap in a Deployment.
  • Because step generator believes that any resource that is marked delete-before-replace must also have all its dependencies transitively marked delete-before-replace (specifically this line), we delete the entire Deployment.

This is very bad for Kubernetes. It means that changing data in a ConfigMap can trigger a complete delete-first tear-down and rebuild of your application, including a stateful application.

The comment explaining this asserts that this is true, but I’m not really sure why it would be – why, in particular, would we need to mark resources that actually don’t need to be deleted at all, as delete-before-create?

cc @pgavlin @swgillespie @lukehoban

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 1
  • Comments: 37 (33 by maintainers)

Most upvoted comments

@naveensrinivasan We apologize for the delay here. This is a subtle and tricky issue, so it’s taken a bit of time to discuss and design. That said, it’s in our current sprint, and we will be acting on this soon. Expect a more detailed update tomorrow on the plans and timeframe on a fix.

The PR for this is in progress at https://github.com/pulumi/pulumi/pull/2369. Currently expecting this to be available next week (hopefully early next week).

Great! We are being blocked by the EKS issue referred here.

Moreover, CloudFormation, Terraform, and all other such infrastructure as code solutions seem to have chosen our current behavior, as far as I know. I also prefer that we learn from others rather than invent. In fact, isn’t this Terraform’s precisely behavior for all resource replacements by default?

This may not in fact be TF’s behavior. Consider the following config:

resource "aws_s3_bucket" "b" {
  bucket = "my-tf-test-bucket-abcde"
  acl    = "private"

  tags {
    Name        = "My bucket"
    Environment = "Dev"
  }
}

resource "aws_s3_bucket" "c" {
  bucket = "my-tf-test-bucket-efghi"
  acl    = "private"

  tags {
    Name        = "My bucket"
    Environment = "${aws_s3_bucket.b.id}"
  }
}

If I deploy this, then change the name of the first bucket, then run an apply, the second bucket is not replaced:

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place
-/+ destroy and then create replacement

Terraform will perform the following actions:

-/+ aws_s3_bucket.b (new resource required)
      id:                          "my-tf-test-bucket-abcde" => <computed> (forces new resource)
      acceleration_status:         "" => <computed>
      acl:                         "private" => "private"
      arn:                         "arn:aws:s3:::my-tf-test-bucket-abcde" => <computed>
      bucket:                      "my-tf-test-bucket-abcde" => "my-tf-test-bucket-abcdef" (forces new resource)
      bucket_domain_name:          "my-tf-test-bucket-abcde.s3.amazonaws.com" => <computed>
      bucket_regional_domain_name: "my-tf-test-bucket-abcde.s3.amazonaws.com" => <computed>
      force_destroy:               "false" => "false"
      hosted_zone_id:              "Z3AQBSTGFYJSTF" => <computed>
      region:                      "us-east-1" => <computed>
      request_payer:               "BucketOwner" => <computed>
      tags.%:                      "2" => "2"
      tags.Environment:            "Dev" => "Dev"
      tags.Name:                   "My bucket" => "My bucket"
      versioning.#:                "1" => <computed>
      website_domain:              "" => <computed>
      website_endpoint:            "" => <computed>

  ~ aws_s3_bucket.c
      tags.%:                      "" => <computed>
      tags.Environment:            "my-tf-test-bucket-abcde" => ""
      tags.Name:                   "My bucket" => ""


Plan: 1 to add, 1 to change, 1 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_s3_bucket.b: Destroying... (ID: my-tf-test-bucket-abcde)
aws_s3_bucket.b: Destruction complete after 1s
aws_s3_bucket.b: Creating...
  acceleration_status:         "" => "<computed>"
  acl:                         "" => "private"
  arn:                         "" => "<computed>"
  bucket:                      "" => "my-tf-test-bucket-abcdef"
  bucket_domain_name:          "" => "<computed>"
  bucket_regional_domain_name: "" => "<computed>"
  force_destroy:               "" => "false"
  hosted_zone_id:              "" => "<computed>"
  region:                      "" => "<computed>"
  request_payer:               "" => "<computed>"
  tags.%:                      "" => "2"
  tags.Environment:            "" => "Dev"
  tags.Name:                   "" => "My bucket"
  versioning.#:                "" => "<computed>"
  website_domain:              "" => "<computed>"
  website_endpoint:            "" => "<computed>"
aws_s3_bucket.b: Creation complete after 7s (ID: my-tf-test-bucket-abcdef)
aws_s3_bucket.c: Modifying... (ID: my-tf-test-bucket-efghi)
  tags.Environment: "my-tf-test-bucket-abcde" => "my-tf-test-bucket-abcdef"
aws_s3_bucket.c: Modifications complete after 6s (ID: my-tf-test-bucket-efghi)

Apply complete! Resources: 1 added, 1 changed, 1 destroyed.

I also don’t know whether you’d want to let a resource make a decision about its dependents. In general, this would seem to be something that only those dependents would be qualified to decide.