aws-cdk: [codepipeline] CDK Deploy-Step Fails - Lambda Assets not uploaded to S3 after build -- "Error occurred while GetObject. S3 Error Code: NoSuchKey"

❓ General Issue

I did build a CDK application (let’s call it app “A”) including multiple lambda functions (all single file python code - no dependencies). This CDK application works great and deploys just fine from my local CLI.

Now, I want to deploy app “A” using a CICD Pipeline via a 2nd separate CDK application.

I pull the source from a source commit repo - works fine.
I build app “A” using a codeBuild job with a custom buildspec.yml (including the cdk synth call). This works fine. The output artifact contains the expected cdk.out including the usual content.
Now i want to deploy my application. I pass the output artifact from the build step into a CloudFormationCreateUpdateStackAction() (see code below).

PROBLEM: The CloudFormation deploy step fails. The CloudFormation deploy fails with the error Error occurred while GetObject. S3 Error Code: NoSuchKey. S3 Error Message: The specified key does not exist. (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: 7dbfdd63-58cb-4e58-b005-0a464d1b1055; Proxy: null) After closer inspection of the generated CFN template, I notice that the Lambda functions code property references the cdk S3 Bucket and Key (see below). The problem is that the S3 Bucket is empty. It doesn’t contain the Key!

The Question

How do I get the lambda assets from the build step uploaded to the S3 bucket so that CloudFormationCreateUpdateStackAction() can deploy the stack?

What am I missing here? Or should I deploy my cloudformation using another codebuilld action with a cdk deploy inside?

Environment

CDK CLI Version: 1.69.0 (build 2b474b9)
Module Version: 1.69.0
Node.js Version: v10.15.0
OS: OSX Catalina
Language (Version): Python (3.7)

Other information

Action used to deploy the CloudFormation: The cdk.out content is passed via the artifact source_output_build.

        action = codepipeline_actions.CloudFormationCreateUpdateStackAction(
            action_name=action_name,
            admin_permissions=True,
            stack_name=stack_name,
            replace_on_failure=True,
            template_path= source_output_build.at_path(template_path),
            capabilities=[
                cloudformation.CloudFormationCapabilities.NAMED_IAM,
                cloudformation.CloudFormationCapabilities.ANONYMOUS_IAM,
                cloudformation.CloudFormationCapabilities.AUTO_EXPAND
            ],
            run_order=1,
            region=region,
            extra_inputs=[source_output_build]
     )

Output of template.json with S3Bucket referncnce (the key doesn’t exist - bucket is empty)

"IntegrationTestTriggerXXXX": {
      "Type": "AWS::Lambda::Function",
      "Properties": {
        "Code": {
          "S3Bucket": "cdk-hnb6XXXXXX26-eu-west-1",
          "S3Key": "da663XXXXXXXXbc09fa27e9.zip"
        },

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 15
Comments: 18 (7 by maintainers)

Most upvoted comments

To expand upon @ijemmy 's comments:

If you are using .from_asset() in your Lambda function stack, your CodePipeline will create an extra step Assets.

This is used to upload your asset files from the CDKSynth step to S3.

Looking into the buildspec.yml for that step, you will see something like:
      "commands": [
        "cdk-assets --path \"assembly-STACK/STACK.assets.json\" --verbose publish \"11eecb7b5ccdb7c71db90f9c453ad052b6cdf36b45f7dcfb207bf60e607508c8:ACCOUNT-NUMBER-REGION\""
      ]
With this command, it looks to publish resource 11eecb7b5ccdb7c71db90f9c453ad052b6cdf36b45f7dcfb207bf60e607508c8 in the assembly-STACK/STACK.assets.json file to S3.

On every deployment, that hash value inside the assembly-STACK/STACK.assets.json file (from your Synth step) will change.

HOWEVER, the Assets step’s buildspec.yml file will still reference the old hash value.

This means your local assets will never upload to S3. The step still will succeed with message:
1 assets found
--
Applied selection: 0 assets selected.
Because of this, you must enable SelfMutate in your CodePipeline, in order for the buildspec.yml in your Assets step to update.

Once doing this, you will see the following in your Assets step:
verbose: Applied selection: 1 assets selected.
--
33 | info   : [0%] start: Publishing 95c924c84f5d023be4edee540cb2cb401a49f115d01ed403b288f6cb412771df:ACCOUNT-NUMBER-REGION
34 | verbose: [0%] check: Check s3://cdk-hnb659fds-assets-ACCOUNT-NUMBER-REGION/95c924c84f5d023be4edee540cb2cb401a49f115d01ed403b288f6cb412771df.zip
35 | verbose: [0%] upload: Upload s3://cdk-hnb659fds-assets-ACCOUNT-NUMBER-REGION/95c924c84f5d023be4edee540cb2cb401a49f115d01ed403b288f6cb412771df.zip
36 | info   : [100%] success: Published 95c924c84f5d023be4edee540cb2cb401a49f115d01ed403b288f6cb412771df:ACCOUNT-NUMBER-REGION

are you from this planet ? Did you read the problem ?

+32

scorobogaci on Feb 20, 2022

This happened to me when I forgot to update pipeline (didn’t enable selfMutation). Updating pipeline fixed the issue.

It turns out that some assets building action (in Assets stage) were missing. Redeploying creates those missing actions.

ijemmy on Dec 31, 2020

To expand upon @ijemmy 's comments:

If you are using .from_asset() in your Lambda function stack, your CodePipeline will create an extra step Assets.
This is used to upload your asset files from the CDKSynth step to S3.
Looking into the buildspec.yml for that step, you will see something like:

      "commands": [
        "cdk-assets --path \"assembly-STACK/STACK.assets.json\" --verbose publish \"11eecb7b5ccdb7c71db90f9c453ad052b6cdf36b45f7dcfb207bf60e607508c8:ACCOUNT-NUMBER-REGION\""
      ]

With this command, it looks to publish resource 11eecb7b5ccdb7c71db90f9c453ad052b6cdf36b45f7dcfb207bf60e607508c8 in the assembly-STACK/STACK.assets.json file to S3.
On every deployment, that hash value inside the assembly-STACK/STACK.assets.json file (from your Synth step) will change.
HOWEVER, the Assets step’s buildspec.yml file will still reference the old hash value.
This means your local assets will never upload to S3. The step still will succeed with message:

1 assets found
--
Applied selection: 0 assets selected.

Because of this, you must enable SelfMutate in your CodePipeline, in order for the buildspec.yml in your Assets step to update.

Once doing this, you will see the following in your Assets step:


verbose: Applied selection: 1 assets selected.
--
33 | info   : [0%] start: Publishing 95c924c84f5d023be4edee540cb2cb401a49f115d01ed403b288f6cb412771df:ACCOUNT-NUMBER-REGION
34 | verbose: [0%] check: Check s3://cdk-hnb659fds-assets-ACCOUNT-NUMBER-REGION/95c924c84f5d023be4edee540cb2cb401a49f115d01ed403b288f6cb412771df.zip
35 | verbose: [0%] upload: Upload s3://cdk-hnb659fds-assets-ACCOUNT-NUMBER-REGION/95c924c84f5d023be4edee540cb2cb401a49f115d01ed403b288f6cb412771df.zip
36 | info   : [100%] success: Published 95c924c84f5d023be4edee540cb2cb401a49f115d01ed403b288f6cb412771df:ACCOUNT-NUMBER-REGION

jl178 on Dec 3, 2021

For those that may encounter the same issue, one way around it is allowing the role executing the code build project to assume the CDK file asset publishing role in the account(s) in which the assets need to be published.

If you used cdk bootstrap, the ARN of this role looks like this: arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cdk-${Qualifier}-file-publishing-role-${AWS::AccountId}-${AWS::Region}.

nchylak on Aug 15, 2022

No comments from my side - that looks pretty much exactly what the pipelines.CodePipeline construct does, just “manually”. If this solution works for you, that’s great!

skinny85 on Aug 8, 2022

I just ran into this too. Found this, which I’ll try: https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/aws-codepipeline-actions/test/integ.lambda-deployed-through-codepipeline.lit.ts

I’m assuming the lambda is trying to create itself from the code in the s3 bucket before it’s there yet. Doing it this way with CloudFormation overrides and a separate cdk deployment probably gets around that

seanWLawrence on Dec 8, 2020