serverless: EC2 permissions for VPC not set on first deploy
Hi Guys,
I have multiple serverless projects up and running, but today I wanted to set up a new one, and I can’t figured it out.
My project is inside a VPC, and I have a permission error on the “AWS::Lambda::Function” role:
Your access has been denied by EC2, please make sure your function execution role have permission to CreateNetworkInterface. EC2 Error Code: UnauthorizedOperation. EC2 Error Message: You are not authorized to perform this operation.
But my serverless.yml is setup the same way than my working projects:
service: AwesomeService
frameworkVersion: "=1.2.1"
custom: ${file(./environments/serverless/${env:NODE_ENV}.yml)}
provider:
name: aws
cfLogs: true
runtime: nodejs4.3
region: eu-central-1
stage: ${self:custom.stage}
memorySize: ${self:custom.memorySize}
timeout: ${self:custom.timeout}
iamRoleStatements:
- Effect: "Allow"
Action:
- "ec2:CreateNetworkInterface"
- "ec2:DescribeNetworkInterfaces"
- "ec2:DetachNetworkInterface"
- "ec2:DeleteNetworkInterface"
Resource: "*"
vpc: ${self:custom.vpc}
functions:
hello:
handler: handler.hello
events:
- http:
method: post
path: hello
cors: true
integration: lambda
As you can see, I clearly set the iamRoleStatements for EC2. Moreover, I read this in the documentation:
Further, if you have specified VPC security groups and subnets for your lambdas to use then the EC2 rights necessary to attach to the VPC via an ENI will be added into the default IAM policy.
But when the role is created, the inline policy attached is the following:
{
"Version": "2012-10-17",
"Statement": [{
"Action": ["logs:CreateLogGroup", "logs:CreateLogStream"],
"Resource": ["arn:aws:logs:eu-central-1:111118746979:log-group:/aws/lambda/AwesomeService-development-hello:*"],
"Effect": "Allow"
}, {
"Action": ["logs:PutLogEvents"],
"Resource": ["arn:aws:logs:eu-central-1:111118746979:log-group:/aws/lambda/AwesomeService-development-hello:*:*"],
"Effect": "Allow"
}]
}
No rights about EC2 are added… Does anyone have an idea?
Many thanks
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Reactions: 3
- Comments: 48 (26 by maintainers)
I figured it out finally… You must do a first deploy of your service WITH the EC2 iamRoleStatements but WITHOUT the VPC key. Once the deployed has ended, then you can add the VPC info to your serverless.yml and make a new deploy…
Is it possible to get a fix of it?
I’m experiencing this using 1.24. I tried deploying first without the vpc configuration and then you modify serverless.yml to add the VPC configuration, then deploy again. That works but prevents automated deployments. Is there a way to specify an the “serverless.yml” file to use for deployment, then I could keep 2 around one without the VPC config and another with?
Any plans for fixing this issue?
Any resolution for this? I am getting this issue in v1.16.
Error log
serverless.yml
For anybody finding this issue, it’s not broken I guess you’re just missing the following properties within your IAM custom roles.
For more info: https://serverless.com/framework/docs/providers/aws/guide/iam#one-custom-iam-role-for-all-functions
When it was introduced: https://github.com/serverless/serverless/commit/e879b2a6bc6b531bd6e85c57a09b37e3f9f20fb8
@pmuens I strongly believe you can close this issue.
Just for clarity, FYI with v1.4 I still get this. 😉 In a few published projects that REQUIRE VPC, I have to specifically warn/explain to people to deploy with VPC commented out first, and then uncomment it. This bug is particularly annoying to have to explain, and I’m running into this more and more since I’m doing lots of in-VPC automation via Lambda these days. Hope someone can +1 work on this soon?
I got it working by adding DependsOn for both, role and policy, to function in Resources. It’s not very sleek to add manually that to every function. @eahefnawy, @flomotlik @pmuens, will there be any downsides if that
DependsOnis added to default cloudformation stack?+1 this is still an issue
This seems to definitely be a timing issue of adding the AWS managed policy taking a while to go into effect before the role attempts to use it.
The best workaround I have found is to manually go into the console, and add the
AWSLambdaVPCAccessExecutionRolemanaged policy to the role. Wait a few seconds and then the deploy should be successful. This is the same change the cloud formation template is doing so the change is idempotent and everything is fine from then on.Thanks for the update @vKongv and @lems3 👍
I’ll re-open this one since it looks like this bug is back again…
Sorry guess, encountered this issue again today on brand new stacks… 😦
Serverless 1.3.0:
Currently experimenting with AWS::CloudFormation::WaitCondition and the results are 👍
Example:
Already tested with adding this to existing stacks and performed stacks updates without issues. Updating the IAM roles/polices between stack updates triggered the wait condition properly. I’ll start working on a new PR with the wait condition fix.
If you are using a VPC and if you are using a custom execution role:
Then you need to make sure this role has
AWSLambdaVPCAccessExecutionRolepolicy attached to itHave you tried it lately @lems3 ? If you don’t override the role it automatically adds that managed role, if you do override the role then you just have to add the managed property to your custom role. I just confirmed with two of my projects that used to exhibit these problems, the first deploy issues seem to be completely gone.
@pmuens I’m on 1.16.1 and am using a
serverless.ymlvery similar to that of @TomVanBerlo.I also got the real project successfully deployed with PR version.
Same issue here using 1.25.0, the @inuwan suggestion not worked for me.
@nicka No. The function doesn’t have a custom role. It uses the iamRoleStatement provided by Serverless. And I wasn’t adding anything to the basic role at first when I tried my deployment. I had to add the policies manually first, then add the VPC configuration to the lambda.
So, if Serverless is supposed to add the policies automatically when using VPC in one function, I can confirm it was not working. If it’s working now, the issue can be closed.
From the research I did in the past it’s a 1/3 change it being able to deploy with the ec2 statements in your iamRoleStatements. Only way is to add the Managed Policy Arn from AWS into your custom IAM role (which the sls framework does).
I’m on Serverless 1.21.1 I tried adding an existing Lambda to the default VPC security group and subnet, and I got the “The provided execution role does not have permissions to call CreateNetworkInterface on EC2.” error.
I manually added the role statements like this
And configured my Lambda function like this (We have other functions in there, some have custom roles built in the resources section, and I replaced the group and subnet ID to random names for privacy)
EDIT : Adding the EC2 permissions, updating the serverless deployment then adding the VPC configuration works.
I have this issue too. I tried with v1.12.1 and v1.16.1. Specifying the IAM policy manually and removing the vpc code during the first run works.
I added the IamRoleStatements part to make it work.
@pmuens, I used the PR version to test that demo code and it deployed just fine. I can try later with a real project that uses VPC, but I think @nicka got it fixed.
+1
Confirmed what @gozup said, if you first deploy serverless without VPC configuration, it goes through, and then you modify serverless.yml to add the VPC then the deployment works fine, and I confirm the Lambdas are running from the VPC requested. Just something to do with the CF workflow for initial-deployment.