copilot-cli: Environment upgrade from 1.8.0 -> 1.9.0 failed: AccessDenied when upload template to bucket
On a recent service deploy, copilot
CLI attempted to transitively upgrade the Copilot environment from 1.8.0
to 1.9.0
. For past environment upgrades, this has gone off without a hitch. Though this time, an error was received when attempting to upload the new service template to the s3 bucket in the App AWS account:
✘ execute "env upgrade --app foo-app --name preview": upgrade environment preview from version v1.8.0 to version v1.9.0: upload manual/templates/***/18918392bc70edd***.yml to bucket stackset-foo-app-inf-pipelinebuiltartifactbuc-1sgrupkj5d4eq****: AccessDenied: Access Denied
status code: 403, request id: MMDRNV2Y90MTNHH3, host id: rERY8wMubiJ+VpU8I7hd+z/8l1BSCwnwe2uePw5pPwtBflXDkL1TiLSDw2D3b3jeKhxAIE8+vpc=
I’ve obfuscated some of the error above with ****
I’m using the App AWS credentials, so not sure why i’d be getting an access denied on the s3. Thanks for you help!
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 2
- Comments: 15 (9 by maintainers)
Commits related to this issue
- fix(s3): fallback to no "bucket-owner-full-control" when access denied Fixes #3556 — committed to efekarakus/copilot-cli by efekarakus 2 years ago
- fix(s3): fallback to no "bucket-owner-full-control" when access denied (#3559) Fixes #3556 I'd like to also create a `Logger` object to log when this issue happens, but that'll be in a separate PR ... — committed to aws/copilot-cli by efekarakus 2 years ago
- chore: detect if EnvManagerRole has permissions to upload artifacts (#3562) This PR is step 1 in resolving #3556 We first need to detect if the env manager role is allowed to upload to S3. If so, t... — committed to aws/copilot-cli by efekarakus 2 years ago
- fix: ensure EnvManagerRole template has permissions to upload artifacts Related #3556 — committed to efekarakus/copilot-cli by efekarakus 2 years ago
- fix: ensure EnvManagerRole template has permissions to upload artifacts (#3567) Related #3556 Tested `env upgrade` from the following versions: - [x] `copilot v1.0` - [x] `copilot v1.13` - [x] `co... — committed to aws/copilot-cli by efekarakus 2 years ago
Hi folks! @codekitchen @TillaTheHun0 @coledcrawford We just released the fix: https://github.com/aws/copilot-cli/releases/tag/v1.18.1 🎉 Apologies for all the inconvenience 🙇
Please let us know if you run into any issues, thank you!
Thanks @codekitchen ! we’re in the process right now of making the changes and we will post here once we have the patch release done!
We have a fix for the issue here: https://github.com/aws/copilot-cli/pull/3559/, #3562
Hi @TillaTheHun0 !
To add onto Janice’s comment, we’re currently in the process of trying to reproduce the issue.
In the mean time to get you unblocked, our hypothesis is that it’s because the following permission is missing in the
EnvironmentManagerRole
: https://github.com/aws/copilot-cli/blob/0a2bd5ac31948ceb9e8f7e8bff67480dc1623c47/internal/pkg/template/templates/environment/partials/environment-manager-role.yml#L225 I think if you make a modification to the EnvironmentManagerRole’s policy to add the permission above, it should get you unblocked.Hypothesis
Between version
v1.8.0
andv1.9.0
, we started uploading the environment template to the S3 bucket (stackset-foo-app-inf-pipelinebuiltartifactbuc-1sgrupkj5d4eq****
) but with theObjectCannedACLBucketOwnerFullControl
ACL as it’s a recommended practice.However, since in
v1.8.0
the role doesn’t have thePutObjectAcl
permission then the template upload fails.I was just coming here to report the same issue. Thanks for the quick fix! We’ll keep an eye out for the next release.
@efekarakus good catch.
Yep, we have separate a AWS account for the App and then each of our environments.
That’s great to hear!
That’s correct 😃 until we have a patch for the fix, adding the permission will get you unblocked
@huanjani @efekarakus thanks for your help with this 🙏
I checked the
EnvironmentManagerRole
for the environment being upgraded. ThePutObjectsToArtifactBucket
policy is like:so no
s3:PutObjectAcl
there.There is another policy,
BuiltArtifactAccess
, that contains more s3 permissions, but still nos3:PutObjectAcl
there either.If that’s the case, and with the lack of the
s3:PutObjectAcl
permission on any of the existing policies, then your hypothesis seems like it could the issue I am seeing. I will add that permission to thePutObjectsToArtifactBucket
policy, and retry the upgrade and see what we get, and report back here.