serverless: Serverless Deployment Access Denied Failures
Issue description
Hey We have just noticed that deployments via serverless deploy have started failing during the initializing phase with the following error
` Access denied when accessing /serverless-framework/state/s3-bucket SSM parameter and serverless-framework-state-xxxx-xxxx-xxx-xxxx-xxxxxxxxx S3 bucket. Please check your permissions and try again. You have the following options: • Ensure you have permission to create SSM and S3 resources. • Use the “state” field to specify an existing S3 bucket. • Manually create the S3 bucket and SSM parameter.
For more details, please refer to the documentation: https://slss.io/compose-state
Original error: An error occurred while fetching the SSM parameter “/serverless-framework/state/s3-bucket”: `
We arent using a custom s3 bucket for holding state, and as far as i am aware nothing has changed with our IAM permissions. Deployments were working fine until about 2 or 3 hours ago, I noticed that a new release got rolled out so tried bumping this new version - we were on v4.4.7 - but no luck. Can you suggest what we might try to troubleshoot / whether its related to the recent releases
Thanks
Context
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 7
- Comments: 20 (7 by maintainers)
This assumes that
us-east-1is available for all your users. Whilst that might be the case on a default account set up, many have to lock down accounts by region for compliance and privacy reasons.Please, please revert this change. Or make it flaggable.
The solution you mentioned above only changes where the bucket configuration is rather than SSM as you later point out.
I’m experiencing the same issue. The latest version (4.4.9) seems to default to the us-east-1 region for SSM getParameter, which is causing the problem. Pinning the version is a workaround for now.
Thank you all for your feedback and for sharing the challenges you’ve faced. We’ve reverted the use of State for individual Serverless Framework services in version v4.4.13. The State is now created and used only when using Compose, as it was prior to v4.4.8. This change should resolve the access and permissions issues introduced by the recent updates. Sorry for any disruption this caused, and thanks for your patience while we worked on resolving it. If you encounter any further issues or have additional feedback, feel free to open a new issue or continue the discussion.
I’m still trying to understand why a patch version introduced such a breaking change. Is this a bug or a feature?
The error you can see below is caused only by a change from Serverless v4.4.7 to Serverless v4.4.12. I’m not using any custom resolvers, and as far as I know, I was not using any means to store the “state” as until now, this doesn’t seem to be needed.
Due to my company security policies, I was the “lucky one” mentioned by @qmenchaca to find the “principle-of-least-privilege”-like IAM permissions set on my own. It was then shared across the company. Now, due to state-related changes the deployment and rollback processes of a buch of apps running SLS in my org is failing.
Due to the same company security policies, I doubt if we could get permissions to
ssm:PutParameteras that would pose an extra security risk, as we’re using Parameter Store to keep important information that could be read but shouldn’t be possible to be overwritten by the app.After experiencing some sudden pipeline failures, I was eventually led here. Without any warning either.
We deploy out our application via Bitbucket pipelines and an IAM user. It makes sense to me that the user deploying via serverless ought to have base IAM permissions to be authorized to access certain resources. It looks like set-up of this predates me during my time at this company.
Digging further through your documentation, it doesn’t look like there is a principle-of-least-privilege IAM permission defined anywhere. (If it is defined it certainly is not anywhere obvious.) The recommended ways to set up credentials with Serverless is to either create a new CloudFormation stack from scratch or to use AWS Organizations:
https://www.serverless.com/framework/docs/providers/aws/guide/credentials#production-configuration
Organizations seems like complete overkill for this situation, not to mention something some devs may not even be authorized to do in their professional role. And without any guidance on IAM permissions to grant to roles and policies, you are asking your users to either engage in frustrating trial-and-error or to give very insecure blanket “*” permissions to a lot of services. This seems like a pretty dangerous, insecure thing to ask of users of your services.
Can I please be linked a clear list of principle-of-least-privilege IAM permissions serverless requires for deployment so my organization can get our app deployed again?
@ChrisFord-Ovo Sorry for the inconvenience. It seems like the issue arises when the custom state resolver is used in combination with
ssm,s3, orcfvariables in the configuration. We’re aiming to release a fix for this on Monday. For now, you can eitherprovider.profilefield to use that AWS profile as credentials resolver ORssm,s3andcfvariables with the resolver name (in your case,awsAccount). For example, change from${ssm:example-param}to${awsAccount:ssm:example-param})ORframeworkVersionfield (see docs)When you have only one
awsresolver set instagesblock and you don’t haveprovider.profileset, the resolver fromstagesblock automatically becomes the credential resolver for deployment. If you want to use another configuration (e.g. another profile) you canprovider.profileORstagesblock and specify it using theprovider.resolverfield to select it as the credential resolver for deployment.To learn more see the docs here.
The state bucket is shared across all services deployed in the account. This design minimizes the number of additional resources created and simplifies sharing state between services in the account (even across regions) without requiring extra lookups to locate the state store for a specific service. The issue isn’t just the bucket itself but also the SSM parameter where the bucket details are stored. While you could technically put your own bucket details (from your region) into the parameter, the Serverless Framework only checks the SSM in the
us-east-1region, so this wouldn’t resolve your problemHey @dekpient, thanks for reporting this. We’ll add support for this in a similar way we do for deployment buckets. I’ve created a new issue to track it here.
@ChrisFord-Ovo Yeah, we’re going to release it in a few minutes.
Thanks for responding @czubocha. It might be worth considering this as 4.5.0 version or having this fact clearer in the release notes. I get most users wont have restrictions on regions - particularly us-east-1 - so wont affect them, but it will affect some and there is no obvious indicator in the release notes there could be a issue, I only noticed due to timings.