aws-cdk: (CDK Pipelines) Deploy stage, vpc.prepare failed due to S3 Access Denied

I have a working code pipeline using CDK python. Recently someone else in the team did some change just move some of the code to different folder. then the code pipeline stopped working, failed at the first step in deploying stage, vpc.Prepare. the reason was S3 Access Denied.

My first thinking is to revert the change and try again, and I did it. it did not help.

Then I removed all the stacks in cloud formation, started over. it did not help.

I found it was due to S3 Access Denied. so I change the S3 bucket pipelinestack-mappipelineartifactsbucket822fb6ba-1cjz3xh6o8l9j to public access, and allow public read to the artifacts within this bucket. Then the Prepare succeeded.

To verify, I turned off the public access of the s3, then it stopped working, stuck at the Prepare step. Turning it on again, then it is in progress again.

in the cloud formation, I can see that the deployment role policy is administrator access, which should not have this kind access issue, correct? But why after granting public access, it worked?

Keep the bucket publicly accessible is not ideal.

Please help to solve the issue.

Thanks

Reproduction Steps

What did you expect to happen?

Prepare step should not be stuck

What actually happened?

Prepare step got stuck when code pipeline is deploy stage

Environment

**CDK CLI Version : 1.111.0 (build 556ca93)
Framework Version:
**Node.js Version:**v16.2.0
**OS : big sure
Language (Version): Python 3.8.2

Other

This is the error message:

Error message
Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: K4T79835ANWTDQY3; S3 Extended Request ID: Hw8uwS1+GrSvsbkem0Xh+XdEDzwdmr2u4yv7Szun9SSbgrtmbw8uwNLq+z1rinQFdqf0xpUiLic=; Proxy: null)

Could you track this request ID to figure out what is happening under the hood?

This is 🐛 Bug Report

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 25 (11 by maintainers)

Most upvoted comments

Hi @peterwoodworth,

Thanks for re-opening this issue, I would like to inform you that the issue was resolved yesterday with the help of an AWS support team(Hammad R) we had multiple calls, after spending one week of time.

He analyzed the error message deeply and found that issues with an Bootstrap bucket policy. We added the permission in the bootstrap bucket policy allowing cfn-exe-role, deployment-role, pipeline-role (Principle) and “Action”:s3* for the resources (Artifact bucket).

He saved me from the embarrassing situation, I really thankful to Hammad R (AWS Support) and aws-cdk-GitHub Team.

Regards, Deva

devendhirababu on Jul 15, 2021

No intention to nitpick on this, just trying to be precise, to check if I’m missing something.

I’m facing the same issue, so I’d like to know if the ACLs you refer to are indeed Bucket ACLs or is it Bucket Policy? Since my bucket has Object Ownership enforced, no ACLs are disabled, and I added the above specified permissions within the bucket policy, but to know avail.

When I turn off the object ownership enforcement, ACLs are enabled, but the interface doesn’t allow to specify them in this format, but just to enter grantee and check the boxes for read write on bucket/objects…

Thanks!

The permissions I wrote above are applied to the S3 bucket as a bucket policy.

Public Access (Block all public access: ON: Access: Bucket and objects not public)

{
    "PublicAccessBlockConfiguration": {
        "BlockPublicAcls": true,
        "IgnorePublicAcls": true,
        "BlockPublicPolicy": true,
        "RestrictPublicBuckets": true
    }
}

The object ownership config for that bucket is: Object Ownership Object writer

The bucket ACL is this (via aws s3api get-bucket-acl --bucket obscured-bucket-name):

{
    "Owner": {
        "DisplayName": "amazon.OBSCURED",
        "ID": "OBSCURED"
    },
    "Grants": [
        {
            "Grantee": {
                "DisplayName": "amazon.OBSCURED",
                "ID": "OBSCURED",
                "Type": "CanonicalUser"
            },
            "Permission": "FULL_CONTROL"
        }
    ]

Hope that helps

jy-metservice on Dec 10, 2023

ame the S3 permission denied error

@jy-metservice How did you do this ? I have added ACL, and also a policy like this but it is still failing


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::artifact-bucket-acc",
                "arn:aws:s3:::artifact-bucket-acc/*"
            ],
            "Condition": {
                "ArnLike": {
                    "aws:PrincipalArn": "arn:aws:iam::acc:role/cdk-hnb659fds-cfn-exec-role-acc-region"
                }
            }
        }
    ]
}

@leantorres73 I used the following ACL. (Note: I have altered our account ID and bucket name). I had to give access to both the cfn-exec-role + the cfn-deploy role

{
    "Version": "2012-10-17",
    "Id": "AccessControl",
    "Statement": [
        {
            "Sid": "CDKToolkitAccess",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::123456789:role/cdk-hnb659fds-cfn-exec-role-123456789-us-east-1",
                    "arn:aws:iam::123456789:role/cdk-hnb659fds-deploy-role-123456789-us-east-1"
                ]
            },
            "Action": [
                "s3:GetObject*",
                "s3:GetBucket*",
                "s3:List*",
                "s3:PutObject*",
                "s3:DeleteObject*"
            ],
            "Resource": [
                "arn:aws:s3:::our-bucket-name",
                "arn:aws:s3:::our-bucket-name/*"
            ]
        }
    ]
}

jy-metservice on Sep 27, 2022