aws-cdk: s3: possible race condition, adding new object to bucket between deletion of bucket objects and bucket deletion prevents deletion

Describe the bug

If I create a log bucket. Then a content bucket, using log bucket for server access logs, when I do a “cdk destroy” the access logs are not removed and the stack fails to destroy.

Expected Behavior

The destroy to complete successfully! (The log bucket and content to be removed.)

Current Behavior

Cloudwatch logs for the delete objects Lambda suggests it was successful :

START RequestId: 72426a95-6f49-4d26-9642-2e15450b205a Version: $LATEST
2022-08-26T10:14:21.449Z	72426a95-6f49-4d26-9642-2e15450b205a	INFO	
{
    "RequestType": "Delete",
    "ServiceToken": "arn:aws:lambda:eu-west-2:123492961234:function:CdkBucketStack-CustomS3AutoDeleteObjectsCustomReso-kArk0g4I9Il2",
    "ResponseURL": "...",
    "StackId": "arn:aws:cloudformation:eu-west-2:123492961234:stack/CdkBucketStack/0531d0e0-2522-11ed-abbb-0aa24eb467d4",
    "RequestId": "b81d7b6f-c28c-455f-83cf-8cf4be357191",
    "LogicalResourceId": "LogBucketAutoDeleteObjectsCustomResource7762F42C",
    "PhysicalResourceId": "f9dbbbc0-641f-4026-a15e-df345405a307",
    "ResourceType": "Custom::S3AutoDeleteObjects",
    "ResourceProperties": {
        "ServiceToken": "arn:aws:lambda:eu-west-2:123492961234:function:CdkBucketStack-CustomS3AutoDeleteObjectsCustomReso-kArk0g4I9Il2",
        "BucketName": "cdkbucketstack-logbucketcc3b17e8-79kbay52r42q"
    }
}

2022-08-26T10:14:27.491Z	72426a95-6f49-4d26-9642-2e15450b205a	INFO	submit response to cloudformation {
  Status: 'SUCCESS',
  Reason: 'SUCCESS',
  StackId: 'arn:aws:cloudformation:eu-west-2:123492961234:stack/CdkBucketStack/0531d0e0-2522-11ed-abbb-0aa24eb467d4',
  RequestId: 'b81d7b6f-c28c-455f-83cf-8cf4be357191',
  PhysicalResourceId: 'f9dbbbc0-641f-4026-a15e-df345405a307',
  LogicalResourceId: 'LogBucketAutoDeleteObjectsCustomResource7762F42C',
  NoEcho: undefined,
  Data: undefined
}
END RequestId: 72426a95-6f49-4d26-9642-2e15450b205a
REPORT RequestId: 72426a95-6f49-4d26-9642-2e15450b205a	Duration: 6312.29 ms	Billed Duration: 6313 ms	Memory Size: 128 MB	Max Memory Used: 81 MB	Init Duration: 174.79 ms	

But the result of the cdk destroy is a fail because there is content :

CdkBucketStack: destroying...
11:15:21 | DELETE_FAILED        | AWS::S3::Bucket             | LogBucketCC3B17E8
The bucket you tried to delete is not empty (Service: Amazon S3; Status Code: 409; Error Code: BucketNotEmpty; Request ID: HJVZC3J1DP4N2QSM; S3 Extended Request ID: /OcCq7vdx6Yxt6bShdNQhgr8fUI3/pG1gmCuC4cu7dBm3juB+Sa9U+FM+O41vk+FVCV8Xwk49YA=; Proxy: null)


 ❌  CdkBucketStack: destroy failed Error: The stack named CdkBucketStack is in a failed state. You may need to delete it from the AWS console : DELETE_FAILED (The following resource(s) failed to delete: [LogBucketCC3B17E8]. ): The bucket you tried to delete is not empty (Service: Amazon S3; Status Code: 409; Error Code: BucketNotEmpty; Request ID: HJVZC3J1DP4N2QSM; S3 Extended Request ID: /OcCq7vdx6Yxt6bShdNQhgr8fUI3/pG1gmCuC4cu7dBm3juB+Sa9U+FM+O41vk+FVCV8Xwk49YA=; Proxy: null)

Reproduction Steps

Deploy the template below. Upload your current directory to “bucket” and wait up to an hour until there are some logs in logBucket. Then run the destroy.

(Yes the code is a bit janky, I did a lazy copy/paste and removed some logic that isn’t relevant etc…)

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as s3 from 'aws-cdk-lib/aws-s3';
import * as iam from 'aws-cdk-lib/aws-iam';

export class CdkBucketStack extends cdk.Stack {
  public readonly Bucket: cdk.CfnOutput;
  public readonly logBucket: cdk.CfnOutput;
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);
    // Create a bucket for all the different logs.
    let logBucketProps: any = {
        removalPolicy: cdk.RemovalPolicy.DESTROY,
        autoDeleteObjects: true,
        accessControl: s3.BucketAccessControl.LOG_DELIVERY_WRITE,
    };
    logBucketProps["enforceSSL"] = true;
    logBucketProps["encryption"] = s3.BucketEncryption.S3_MANAGED;
    logBucketProps["blockPublicAccess"] = s3.BlockPublicAccess.BLOCK_ALL;
    const logBucket = new s3.Bucket(this, 'LogBucket', logBucketProps);
    logBucket.addToResourcePolicy(
      new iam.PolicyStatement({
        effect: iam.Effect.ALLOW,
        principals: [new iam.ServicePrincipal('logging.s3.amazonaws.com')],
        actions: ['s3:PutObject'],
        resources: [`${logBucket.bucketArn}/*`],
      }),
    );

    let BucketProps: any = {
        removalPolicy: cdk.RemovalPolicy.DESTROY,
        autoDeleteObjects: true,
    };
    BucketProps["enforceSSL"] = true;
    BucketProps["encryption"] = s3.BucketEncryption.S3_MANAGED;
    BucketProps["serverAccessLogsPrefix"] = "output-access-logs/";
    BucketProps["serverAccessLogsBucket"] = logBucket;
    BucketProps["blockPublicAccess"] = s3.BlockPublicAccess.BLOCK_ALL;
    const bucket = new s3.Bucket(this, 'OutputBucket', BucketProps);

    this.Bucket = new cdk.CfnOutput(this, 'S3Bucket', {
      value: bucket.bucketName,
      description: "The bucket",
     });
    this.logBucket = new cdk.CfnOutput(this, 'logS3Bucket', {
      value: logBucket.bucketName,
      description: "The log bucket",
    });
  }
}

Possible Solution

I imagine the “delete content” lambda is not checking some error condition it isn’t expecting and reporting success when it hasn’t succeeded. The Lambda does get deleted and the code in the Lambda editor is minified so I couldn’t easily retry or debug it.

Additional Information/Context

The logic I removed in the code example avoids deleting the content in our “production” deployments. But in dev, we have no need to keep the access logs. So if someone was thinking “but it’s log data, we can’t delete that”, please : do delete any data. Or at least provide a “actuallyAutoDeleteAllObjects: true” option.

CDK CLI Version

2.38.1 (build a5ced21)

Framework Version

No response

Node.js Version

Node.js v18.8.0

OS

OSX

Language

Typescript

Language Version

No response

Other information

No response

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 2
  • Comments: 18 (11 by maintainers)

Most upvoted comments

@peterwoodworth One solution might be to update bucket policy to block adding objects to the bucket and after that deleting the objects. That should prevent writes to the bucket between clearing the bucket and deleting the bucket. (Unless the used write method ignores the bucket policy)

Seeing this problem also.

Interestingly, we observe the issue when our system couldn’t cause any access events on the content bucket itself during the Stack destroy events. We don’t have any compute accessing the buckets during that time. I have no problem manually emptying and then deleting the access logs bucket because there’s nothing continuously accessing the content which be populating the access logs.

However, CDK fails to empty and then delete the access logs every time, reliably. Content bucket gets emptied and deleted, never the access logs.

Therefore, I wonder if it’s a timing in CDK itself. Perhaps marking its “empty the access logs bucket” process finished before its “empty the content bucket” has finished.

One theory: 1.In parallel:

  1. Begin clearing content bucket
  2. Begin clear access logs bucket
  3. Access logs completes emptying first
  4. Content bucket continues emptying, thus writing new entries to access logs
  5. Attempt to delete access logs fails because it’s not empty