aws-cdk: custom_resources: Provider Lambda function is missing lambda:GetFunctionConfiguration
Describe the bug
The Landing Zone Accelerator solution leverages the custom_resources module to create service-linked roles via CDK custom resources. When this custom resource Lambda function is invoked several times in succession, users intermittently receive the following error:
Received response status [FAILED] from custom resource. Message returned: AccessDeniedException: Resource is not in the state functionActive
We believe this is the result of queuing incoming requests and the role attached to the cdk.custom_resources.Provider function is missing the permission: lambda:GetFunctionConfiguration
Expected Behavior
Custom resource provider implements appropriate permissions and retries to execute successfully when invoked several times in succession.
Current Behavior
Transient failures:
Received response status [FAILED] from custom resource. Message returned: AccessDeniedException: Resource is not in the state functionActive
Reproduction Steps
Deploy v1.4.3 of the Landing Zone Accelerator on AWS.
For a smaller sample that can be extracted without deploying the entire LZA solution, you may use this custom resource construct that is used by LZA to create the service-linked roles:
Possible Solution
Add lambda:GetFunctionConfiguration permission to the provider Lambda function’s IAM role.
Additional Information/Context
No response
CDK CLI Version
2.79
Framework Version
No response
Node.js Version
16.20.1
OS
Amazon Linux
Language
Typescript
Language Version
No response
Other information
No response
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 7
- Comments: 17 (9 by maintainers)
Commits related to this issue
- fix(custom-resource): provider framework lambda missing GetFunction permission (#32904) ### Issue # (if applicable) Closes #26838. ### Reason for this change In the Provider Framework lambda code,... — committed to samson-keung/aws-cdk by samson-keung 2 years ago
- fix(custom-resource): provider framework lambda missing GetFunction permission (#32904) ### Issue # (if applicable) Closes #26838. ### Reason for this change In the Provider Framework lambda code,... — committed to aws/aws-cdk by samson-keung 2 years ago
- fix(custom-resource): provider framework lambda missing GetFunction permission (#32904) ### Issue # (if applicable) Closes #26838. ### Reason for this change In the Provider Framework lambda code,... — committed to aws/aws-cdk by samson-keung 2 years ago
- fix(custom-resource): provider framework lambda missing GetFunction permission (#32904) ### Issue # (if applicable) Closes #26838. ### Reason for this change In the Provider Framework lambda code,... — committed to aws/aws-cdk by samson-keung 2 years ago
- fix(custom-resource): provider framework lambda missing GetFunction permission (#32904) ### Issue # (if applicable) Closes #26838. ### Reason for this change In the Provider Framework lambda code,... — committed to aws/aws-cdk by samson-keung 2 years ago
- fix(custom-resource): provider framework lambda missing GetFunction permission (#32904) ### Issue # (if applicable) Closes #26838. ### Reason for this change In the Provider Framework lambda code,... — committed to aws/aws-cdk by samson-keung 2 years ago
- fix(custom-resource): provider framework lambda missing GetFunction permission (#32904) ### Issue # (if applicable) Closes #26838. ### Reason for this change In the Provider Framework lambda code,... — committed to aws/aws-cdk by samson-keung 2 years ago
- fix(custom-resource): provider framework lambda missing GetFunction permission (#32904) ### Issue # (if applicable) Closes #26838. ### Reason for this change In the Provider Framework lambda code,... — committed to aws/aws-cdk by samson-keung 2 years ago
- fix(custom-resource): provider framework lambda missing GetFunction permission (#32904) ### Issue # (if applicable) Closes #26838. ### Reason for this change In the Provider Framework lambda code,... — committed to aws/aws-cdk by samson-keung 2 years ago
- fix(custom-resource): provider framework lambda missing GetFunction permission (#32904) ### Issue # (if applicable) Closes #26838. ### Reason for this change In the Provider Framework lambda code,... — committed to aws/aws-cdk by samson-keung 2 years ago
- fix(custom-resource): provider framework lambda missing GetFunction permission (#33315) ### Issue # (if applicable) Closes #26838. ### Reason for this change In the Provider Framework lambda code,... — committed to badmintoncryer/aws-cdk by moelasmar 2 years ago
Encountered same problem with EKS Blueprint, which is using
kubectlProvidercreated from https://github.com/aws/aws-cdk/blob/v2.96.0/packages/aws-cdk-lib/aws-eks/lib/kubectl-provider.ts#L144.The error message on Cloudformation said:
Custom::AWSCDK-EKS-KubernetesResource EKSStackAwsAuthmanifest75D20040 CREATE_FAILED Received response status [FAILED] from custom resource. Message returned: TimeoutError: {"state":"TIMEOUT","reason":"Waiter has timed out"} at checkExceptions ...Also found the following error message in CloudTrail:
User: arn:aws:sts::[redacted]:assumed-role/EKSStack-awscdka-ProviderframeworkonEventS-duAmMCwNZO6z/EKSStack-awscdka-ProviderframeworkonEvent-4NeK4zj7Z6ab is not authorized to perform: lambda:GetFunctionConfiguration on resource: arn:aws:lambda:[redacted]:[redacted]:function:EKSStack-awscdkawseksKube-Handler886CB40B-3fOpwrZnomNI because no identity-based policy allows the lambda:GetFunctionConfiguration actionAfter added
lambda:GetFunctionConfigurationpermision to IAM role of the function, the template can be deployed.I’ve added some more details below, please advise whether it’s still necessary to provide a reproducible code-snippet.
This issue is exacerbated by logic that may run these custom resources on every pipeline run, for instance: https://github.com/awslabs/landing-zone-accelerator-on-aws/issues/237
The following line and the surrounding retry logic requires the as mentioned
lambda:GetFunctionConfigurationpermission to function: https://github.com/aws/aws-cdk/blob/c695b6004219426cf0e67cbb92d916a394ddd594/packages/aws-cdk-lib/custom-resources/lib/provider-framework/runtime/outbound.ts#L66C15-L66C15However only invokeFunction is granted on the onEventHandler Lambda function: https://github.com/aws/aws-cdk/blob/c695b6004219426cf0e67cbb92d916a394ddd594/packages/aws-cdk-lib/custom-resources/lib/provider-framework/provider.ts#L233
I believe the above
provider.tslikely needs afn.addToRolePolicyadded after the above line granting “lambda:GetFunctionConfiguration” onthis.onEventHandler.functionArn.