serverless: Serverless Deployment Access Denied Failures

Issue description

Hey We have just noticed that deployments via serverless deploy have started failing during the initializing phase with the following error

` Access denied when accessing /serverless-framework/state/s3-bucket SSM parameter and serverless-framework-state-xxxx-xxxx-xxx-xxxx-xxxxxxxxx S3 bucket. Please check your permissions and try again. You have the following options: • Ensure you have permission to create SSM and S3 resources. • Use the “state” field to specify an existing S3 bucket. • Manually create the S3 bucket and SSM parameter.

For more details, please refer to the documentation: https://slss.io/compose-state

Original error: An error occurred while fetching the SSM parameter “/serverless-framework/state/s3-bucket”: `

We arent using a custom s3 bucket for holding state, and as far as i am aware nothing has changed with our IAM permissions. Deployments were working fine until about 2 or 3 hours ago, I noticed that a new release got rolled out so tried bumping this new version - we were on v4.4.7 - but no luck. Can you suggest what we might try to troubleshoot / whether its related to the recent releases

Thanks

Context

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 7
  • Comments: 20 (7 by maintainers)

Most upvoted comments

The state bucket is shared across all services deployed in the account. This design minimizes the number of additional resources created and simplifies sharing state between services in the account (even across regions) without requiring extra lookups to locate the state store for a specific service.

This assumes that us-east-1 is available for all your users. Whilst that might be the case on a default account set up, many have to lock down accounts by region for compliance and privacy reasons.

Please, please revert this change. Or make it flaggable.

The solution you mentioned above only changes where the bucket configuration is rather than SSM as you later point out.

I’m experiencing the same issue. The latest version (4.4.9) seems to default to the us-east-1 region for SSM getParameter, which is causing the problem. Pinning the version is a workaround for now.

Thank you all for your feedback and for sharing the challenges you’ve faced. We’ve reverted the use of State for individual Serverless Framework services in version v4.4.13. The State is now created and used only when using Compose, as it was prior to v4.4.8. This change should resolve the access and permissions issues introduced by the recent updates. Sorry for any disruption this caused, and thanks for your patience while we worked on resolving it. If you encounter any further issues or have additional feedback, feel free to open a new issue or continue the discussion.

I’m still trying to understand why a patch version introduced such a breaking change. Is this a bug or a feature?

The error you can see below is caused only by a change from Serverless v4.4.7 to Serverless v4.4.12. I’m not using any custom resolvers, and as far as I know, I was not using any means to store the “state” as until now, this doesn’t seem to be needed.

Due to my company security policies, I was the “lucky one” mentioned by @qmenchaca to find the “principle-of-least-privilege”-like IAM permissions set on my own. It was then shared across the company. Now, due to state-related changes the deployment and rollback processes of a buch of apps running SLS in my org is failing.

Due to the same company security policies, I doubt if we could get permissions to ssm:PutParameter as that would pose an extra security risk, as we’re using Parameter Store to keep important information that could be read but shouldn’t be possible to be overwritten by the app.

Deploying "project" to stage "staging" (eu-west-1)
✖ Stack project-staging failed to deploy (0s)


✖ Access denied when accessing /serverless-framework/state/s3-bucket SSM parameter and serverless-framework-state-xxxxxxx-yyyy-zzzz-aaaa-bbbbbbbbb S3 bucket. Please check your permissions and try again. You have the following options:
• Ensure you have permission to create SSM and S3 resources.
• Use the "state" field to specify an existing S3 bucket.
• Manually create the S3 bucket and SSM parameter.

For more details, please refer to the documentation: https://slss.io/compose-state

Original error: User: arn:aws:sts::abcdabcdabcd:assumed-role/AWSReservedSSO_xzy_abcdefabcdefabcdef/user@somedomain.com is not authorized to perform: ssm:PutParameter on resource: arn:aws:ssm:us-east-1:abcdabcdabcd:parameter/serverless-framework/state/s3-bucket with an explicit deny in an identity-based policy

After experiencing some sudden pipeline failures, I was eventually led here. Without any warning either.

We deploy out our application via Bitbucket pipelines and an IAM user. It makes sense to me that the user deploying via serverless ought to have base IAM permissions to be authorized to access certain resources. It looks like set-up of this predates me during my time at this company.

Digging further through your documentation, it doesn’t look like there is a principle-of-least-privilege IAM permission defined anywhere. (If it is defined it certainly is not anywhere obvious.) The recommended ways to set up credentials with Serverless is to either create a new CloudFormation stack from scratch or to use AWS Organizations:

https://www.serverless.com/framework/docs/providers/aws/guide/credentials#production-configuration

Organizations seems like complete overkill for this situation, not to mention something some devs may not even be authorized to do in their professional role. And without any guidance on IAM permissions to grant to roles and policies, you are asking your users to either engage in frustrating trial-and-error or to give very insecure blanket “*” permissions to a lot of services. This seems like a pretty dangerous, insecure thing to ask of users of your services.

Can I please be linked a clear list of principle-of-least-privilege IAM permissions serverless requires for deployment so my organization can get our app deployed again?

@ChrisFord-Ovo Sorry for the inconvenience. It seems like the issue arises when the custom state resolver is used in combination with ssm, s3, or cf variables in the configuration. We’re aiming to release a fix for this on Monday. For now, you can either

  • add provider.profile field to use that AWS profile as credentials resolver OR
  • prepend all ssm, s3 and cf variables with the resolver name (in your case, awsAccount). For example, change from ${ssm:example-param} to ${awsAccount:ssm:example-param})OR
  • pin the Framwork version using frameworkVersion field (see docs)

Either way it seems like now we have to define a custom state resolver we now also have to change how our aws credentials get set?

When you have only one aws resolver set in stages block and you don’t have provider.profile set, the resolver from stages block automatically becomes the credential resolver for deployment. If you want to use another configuration (e.g. another profile) you can

  • add provider.profile OR
  • add another resolver to the stages block and specify it using the provider.resolver field to select it as the credential resolver for deployment.

To learn more see the docs here.

Out of interest why is the default region for the state buckets us-east-1? Cant it just use the region the stack is deployed in?

The state bucket is shared across all services deployed in the account. This design minimizes the number of additional resources created and simplifies sharing state between services in the account (even across regions) without requiring extra lookups to locate the state store for a specific service. The issue isn’t just the bucket itself but also the SSM parameter where the bucket details are stored. While you could technically put your own bucket details (from your region) into the parameter, the Serverless Framework only checks the SSM in the us-east-1 region, so this wouldn’t resolve your problem

Hey @dekpient, thanks for reporting this. We’ll add support for this in a similar way we do for deployment buckets. I’ve created a new issue to track it here.

@ChrisFord-Ovo Yeah, we’re going to release it in a few minutes.

Thanks for responding @czubocha. It might be worth considering this as 4.5.0 version or having this fact clearer in the release notes. I get most users wont have restrictions on regions - particularly us-east-1 - so wont affect them, but it will affect some and there is no obvious indicator in the release notes there could be a issue, I only noticed due to timings.