aws-cdk: S3 error: Access Denied on cdk deploy
Around 10% of cdk deploy commands fail with S3 error: Access Denied.
Reproduction Steps
cdk "deploy" "-e" "true" "--require-approval" "never" "--no-ci" "--output" "1432235223.cdk.out" "--no-staging" "MyStack"
Error Log
978 MyStack: creating CloudFormation changeset...
979 ❌ MyStack failed: ValidationError: S3 error: Access Denied
980 For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
981 S3 error: Access Denied
982 For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
Environment
- CLI Version :1.25.0
- Framework Version:1.25.0
- OS :Linux Ubuntu 18.04
- Language :Typescript
Other
The stack itself is irrelevant. This can even happen with an almost empty stack.
This is 🐛 Bug Report
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 2
- Comments: 21 (7 by maintainers)
Commits related to this issue
- fix(assets): infrequent "ValidationError: S3 error: Access Denied" We're checking if an asset exists in S3 before uploading (to avoid re-uploading possibly large assets). This causes HEAD/GET-after-P... — committed to aws/aws-cdk by RomainMuller 4 years ago
- fix(assets): infrequent "ValidationError: S3 error: Access Denied" We're checking if an asset exists in S3 before uploading (to avoid re-uploading possibly large assets). This causes HEAD/GET-after-P... — committed to aws/aws-cdk by RomainMuller 4 years ago
- fix(assets): infrequent "ValidationError: S3 error: Access Denied" (#7556) We're checking if an asset exists in S3 before uploading (to avoid re-uploading possibly large assets). This causes HEAD/GET... — committed to aws/aws-cdk by RomainMuller 4 years ago
It seems to me that after naming something in the stack to get different stack generated (different hash), it started to work. Again when returning to the same stack (identical hash), I’m getting this error 100% of the rounds. It has to be that something with S3 file publishing has failed.
I intend on releasing a version of CDK with the fix in it today, so y’all can collecting evidence on whether that was it or not… 🤞🏻
Got the same error… I’ve never seen this before. I’m trying to deploy a resource in a VPC I haven’t visited in a while.
@NetaNir might actually have identified an S3 negative cache entry situation which could cause the symptoms you’re facing.
We’re checking if an asset exists in S3 before uploading (to avoid re-uploading possibly large assets). This causes HEAD/GET-after-PUT to become eventually consistent because it creates a negative cache entry…
We’re looking to switch to using a LIST-based check, which would not create the negative cache entry, and hopefully stop this issue altogether.
I keep seeing this on EKS stacks but only occasionally - maybe 1 in 10. Details: