copilot-cli: Environment upgrade from 1.8.0 -> 1.9.0 failed: AccessDenied when upload template to bucket

Hi folks! @codekitchen @TillaTheHun0 @coledcrawford We just released the fix: https://github.com/aws/copilot-cli/releases/tag/v1.18.1 🎉 Apologies for all the inconvenience 🙇

Please let us know if you run into any issues, thank you!

Thanks @codekitchen ! we’re in the process right now of making the changes and we will post here once we have the patch release done!

We have a fix for the issue here: https://github.com/aws/copilot-cli/pull/3559/, #3562

Hi @TillaTheHun0 !

To add onto Janice’s comment, we’re currently in the process of trying to reproduce the issue.

In the mean time to get you unblocked, our hypothesis is that it’s because the following permission is missing in the EnvironmentManagerRole: https://github.com/aws/copilot-cli/blob/0a2bd5ac31948ceb9e8f7e8bff67480dc1623c47/internal/pkg/template/templates/environment/partials/environment-manager-role.yml#L225 I think if you make a modification to the EnvironmentManagerRole’s policy to add the permission above, it should get you unblocked.

Hypothesis

Between version v1.8.0 and v1.9.0, we started uploading the environment template to the S3 bucket (stackset-foo-app-inf-pipelinebuiltartifactbuc-1sgrupkj5d4eq****) but with the ObjectCannedACLBucketOwnerFullControl ACL as it’s a recommended practice.

However, since in v1.8.0 the role doesn’t have the PutObjectAcl permission then the template upload fails.

I was just coming here to report the same issue. Thanks for the quick fix! We’ll keep an eye out for the next release.

@efekarakus good catch.

Yep, we have separate a AWS account for the App and then each of our environments.

That’s great to hear!

I am assuming I’ll need to do this for each environment, for now?

That’s correct 😃 until we have a patch for the fix, adding the permission will get you unblocked

@huanjani @efekarakus thanks for your help with this 🙏

I checked the EnvironmentManagerRole for the environment being upgraded. The PutObjectsToArtifactBucket policy is like:

      {
          "Action": [
              "s3:PutObject"
          ],
          "Resource": [
              "arn:aws:s3:::stackset-foo-app-inf-pipelinebuiltartifactbuc-1sgrupkj5d4eq",
              "arn:aws:s3:::stackset-foo-app-inf-pipelinebuiltartifactbuc-1sgrupkj5d4eq/*"
          ],
          "Effect": "Allow",
          "Sid": "PutObjectsToArtifactBucket"
      },

so no s3:PutObjectAcl there.

There is another policy, BuiltArtifactAccess, that contains more s3 permissions, but still no s3:PutObjectAcl there either.

Between version v1.8.0 and v1.9.0, we started uploading the environment template to the S3 bucket… with the ObjectCannedACLBucketOwnerFullControl ACL as it’s a recommended practice.

If that’s the case, and with the lack of the s3:PutObjectAcl permission on any of the existing policies, then your hypothesis seems like it could the issue I am seeing. I will add that permission to the PutObjectsToArtifactBucket policy, and retry the upgrade and see what we get, and report back here.