aws-cdk: lambda: corrupt zip archive asset produced when using node v15.6

Hi everyone, sharing my conclusions here as well:

• Working: crc32-stream >= 4.0.2 OR Node <= 15.5.0
• Non working: crc32-stream < 4.0.2 AND Node > 15.5.0

I know some folks have been reporting conflicting results, before we dive deeper into that, I wanted to clarify something about the subtleties of asset caches. It might explain some of this behavior.

Note that its not sufficient to remove only the asset your code is creating, but also the assets created by the framework.

For example, the BucketDeployment construct bundles its own asset (that contains the lambda code) inside the cdk.out directory. If you used a faulty combination and deployed the stack, the lambda code asset itself will be corrupted, and any subsequent deployment will suffer from the same problem, regardless of if you removed your own asset. To address this, you need to delete all assets that are included in your stack:

staging_bucket=$(aws cloudformation describe-stack-resources --stack-name CDKToolkit --logical-resource-id StagingBucket --query 'StackResources[].PhysicalResourceId' --output=text)
for asset in $(ls cdk.out | grep asset); do
  hash=$(echo ${asset} | cut -d'.' -f2)
  aws s3 rm s3://${staging_bucket}/assets/${hash}.zip
done

It’s also a good idea to just nuke the entire cdk.out directory instead of specific assets inside it.

I’ve created this reproduction repo that does all that for you and also allows specifying different versions of the relevant components. You can use it for sanity checks as well as plugging in your own stack.

@MHacker9404 @machielg why aren’t you using an active LTS version of node.js?

Good question, brew installs 15.x by default.

Is it possible for aws-cdk to specify the compatible/supported version(s)? So it doesn’t install on node 15.x or gives a warning

You can do it: brew install node@14 && brew link node@14 --overwrite

Downgrading to node 14.15.4 and cleaning cdk.out and s3 zip files made the trick for me.

I had the same issue after upgrading to 15.6.0. Downgrading (and deleting cdk.out/.cache and deleting the zip files on the s3 bucket) worked!

@iliapolo thanks a lot for taking the time for making it sharable, I can confirm it did the trick for me, i.e.:

1. run the above shell script
2. rimraffed cdk.out
3. downgraded Node to v14 via Homebrew
4. cdk deploy

Running into the same issue now with aws_s3_deployment on 1.84.0 and 1.85.0:

//EDIT:

I can see the asset was bundled locally (e.g., asset.90b...), however, when browsing to the s3://cdk-hnb659fds-assets-*-*/assets/ prefix, I can see that the zip corresponding to asset.90b has the correct files in it, but they are entirely empty.

OS: Catalina

This ugly script fixes assets after deploy so you can deploy again. Run it in the root folder of your project so it can find cdk.out. It iterates over assets listed in manifest.json, downloads them from S3, extracts to a temporary folder, re-zips, and finally uploads back to S3.

#!/bin/bash
set -ex

ASSEMBLY_DIRECTORY=`jq -r '.artifacts[] | select(.type == "cdk:cloud-assembly") | .properties.directoryName' cdk.out/manifest.json`
ASSET_MANIFESTS=`jq -r '.artifacts[] | select(.type == "cdk:asset-manifest") | .properties.file' cdk.out/$ASSEMBLY_DIRECTORY/manifest.json`
cd cdk.out/$ASSEMBLY_DIRECTORY
ASSETS=`jq -r '.files[].destinations[] | "s3://" + .bucketName + "/" + .objectKey' $ASSET_MANIFESTS | grep zip`

TMP=`mktemp -d`
cd $TMP

for ASSET in $ASSETS
do
    if aws s3 ls $ASSET; then
        aws s3 cp $ASSET pkg.zip
        mkdir s
        cd s
        if ! unzip ../pkg.zip; then echo bad zip; fi
        rm ../pkg.zip
        zip -r ../pkg.zip * .gitempty
        aws s3 cp ../pkg.zip $ASSET
        cd ..
        rm -rf s
    fi
done

rm -rf $TMP

The fix will be available in the next release.

crc32-stream 4.0.2 fixed it for me with node 15.7.0. Thanks @cmckni3 and @katrinabrock !

Some heroes wear capes. Some heroes wear whatever this is:

Working now! ❤️

I’m using python flavor of CDK, I had to remove this line from @iliapolo 's run.sh https://github.com/iliapolo/aws-cdk-issue12536/blob/main/run.sh#L72

packages.json is this:

{
  "dependencies": {
    "aws-cdk": "1.88.0",
    "yarn": "^1.22.10"
  },
  "resolutions": {
    "archiver": "5.2.0",
    "crc32-stream": "4.0.2"
  }
}

Of course, I also had to make sure the version of the python aws-cdk.* packages matches cdk version (1.88.0 if you use run.sh script as-is). I’m also still clearing out leftover docker images that CDK generates. Not sure if that is needed or not.

I was also experiencing this issue but finally got things working again with NodeJS 12.20.1 and CDK 1.85.0. I suspect it should also work for the latest 14.x LTS.

During initial troubleshooting, I tried several combinations of Node and CDK versions but no to avail, would always get the non-empty zip file error during deployment. I had been clearing out the cdk.out folder but I was NOT clearing the zip files in the CDK bootstrap bucket on S3. Once I did that, it worked.

As may other commenters already pointed out, you NEED to clear BOTH your local cdk.out AND the offending zip files in the CDK bootstrap S3 bucket on AWS (or just deleted everything in the bootstrap bucket which is what I did).

It looks like the root issue is that the zip file names are simple hash checks and CDK is optimized to not bother reuploading the “same file” but “same” is detected just by filename, not by size or modification date. As such, once you have this issue, your bootstrap buckets are “poisoned” for lack of a better term and won’t work again unless you either A) clear the offending files in the S3 bucket, or just delete it all or B) modified your actual code enough to result in a new hash being generated (I suspect this could be a side effect for some troubleshooting).

For future reference, I’m adding my experience with this issue.

Configuration

• CDK: 1.152.0
• CDK CLI: 2.20.0 (build 738ef49)
• Node: 12.22.12
• npm: 6.14.16
• Gradle: 7.4.2
• Java: 11
• OS: Mac OS X 12.3.1 aarch64

I had a Gradle project for each Lambda handler and a Gradle project for CDK code, i.e.,

root
|
--- cdk-code
|
---
    |
    --- cdk.json
    --- cdk.out
|
--- lambda-handler-1
|
--- lambda-handler-2
|
.
.
.
--- gradlew

Each Lambda handler project used the Java application Gradle plugin, so when bundling locally, I had to use

../gradlew clean check distZip

since BundlingOutput.ARCHIVED requires exactly one archive file in the directory (i.e., no .tar archive). Really, only the distZip is necessary, but cleaning and running checks is a safe option.

So what I would try to run is cdk bootstrap, then cdk synth, and then cdk deploy. Bootstrapping and synthesizing would complete successfully, and I could verify that all handler ZIP archives were properly being picked up by CDK and hashed (using AssetHashType.OUTPUT) to cdk.out. However, when I would run cdk deploy, I noticed that an empty ZIP file would appear in cdk.out.

The key observation that ultimately helped me was the following. Upon running cdk synth, I confirmed that each handler aws:asset:path matched what was in cdk.out. However, upon running cdk deploy, the value of aws:asset:path (located under “Metadata”) for all handlers changed to the hash of the empty ZIP archive. From what I could tell, it’s as if CDK didn’t recognize that the handlers were already bundled, and so cdk deploy “bundled the bundled assets.”

To fix the issue, I simply deleted cdk.out after running cdk synth and then ran cdk deploy. I haven’t confirmed, but I think running cdk synth --no-staging would save you the trouble of deleting cdk.out.

This is only the second project I have worked with CDK, so I may be missing some critical detail about how to properly use CDK or the CDK CLI that would’ve avoided this issue, but it didn’t seem to be caused by any partcular version of software I was using.

My currently working configuration is Node v14.16.1 (LTS) and CDK version 1.100.0.

As others have stated, clearing the staging bucket of affected assets is required to fix this problem. After becoming aware of this issue, running cdk deploy ... -vvv indicated that CDK was detecting a pre-existing asset within the staging bucket. With this asset already uploaded, CDK skipped uploading the current copy (leaving a corrupted zip file in place) and proceeded to attempt to use the corrupted zip file in the following Lambda function deployment.

My issue was resolved after updating to the previously mentioned Node and CDK versions, then removing the assets which were indicated in the verbose output produced by the -vvv flag.

@scubbo, are you using a docker container? I was running cdk in the node:latest container and had to, in addition to setting archiver to ~5.1.0 & deleting all objects in the S3 bucket, switch to using node:lts.

@cmckni3 Glad you got it working - I just tried each of those steps after downgrading to CDK 1.74.0 and still get the same errors. 😕

I kept CDK at 1.86.0. Make sure you run an install after updating package.json.

Still no dice on my M1 Mac - thanks though! I’ll keep at it. ⛏️

EDIT: Success! It was an ID10T error on my part…can confirm that CDK 1.86.0 works on an M1 Mac when limiting node-archiver as suggested. 👍

This seems to be caused by: https://github.com/archiverjs/node-archiver/issues/491, which identified the root cause as https://github.com/nodejs/node/issues/37027

Hi everyone!

Sorry about this 😕.

The issue was not automatically assigned to one of the core team members, hence the delay in response. If this ever happens, I asked that you kindly tag one of the core team on the issue, here is the list of GitHub logins: https://github.com/aws/aws-cdk/blob/4871f4a1503bc2d82440e204e1c5b05f2ef26b7b/.mergify.yml#L9

Im going to look into this now.

I am also facing this issue. I managed to make it work by clearing the asset in S3 and deleting cdk.out (not sure what .cache is) but it s failing again.

This issue seems to be fixed on latest version of node. I have tested this with node 15.12.0 “node -v – v15.12.0”. It is working so far for me.

Similar issue with BucketDeployment.

I’m using node 15.8.0: % node -v v15.8.0

CDK 1.89.0: % cdk --version 1.89.0 (build df7253c)

Still not sure why folks use the odd (non-LTS) versions of node.

@cmckni3 In general I’d expect developers to prefer the advertised “Current” version (new features, hopefully less bugs, etc). Also, on macOS when installing with Homebrew (brew install node) you get the latest Node.js version. LTS option seems to be more appealing for system administrators, where stability comes before feature richness, etc.

The same here with

• osx 11.1 (20C69)
• cdk 1.88.0 (build f65009b)
• node v15.8.0

Problem has returned for me on node.js 12.20.0, AWS CDK 1.86.0, and node-archiver 5.1.0.

I know the AssetCode and Code classes in CDK changed recently. 🤔

Issue occurred on CDK 1.74.0 again with node-archiver 5.1.0.

Trying with archiver 5.0.2.

I don’t know if I can be useful, but I’m sharing my experience with this issue.

I was using latest node version and I encountered this issues. I rolled back to the node lts using nvm, removed cdk.out, removed cdk bootstrap folder content on s3 and then it started to work again even with the latest CDK version

@nija-at Just in case you missed it, the root cause is https://github.com/nodejs/node/issues/37027.

And a small nit regarding the changed title: The generated zip files are not actually empty, just corrupt (bad CRC). It’s simply the error message displayed by CDK that reads “Uploaded Zips of Functions are Empty”.

I have downgraded all aws-cdk packages to 1.74.0, set archiver to ~5.1.0 as cmckni3 suggested, deleted uploaded s3 files, run rm -rf cdk.out/*, and run npm install - and I’m still getting this same issue (confirmed that the uploaded files are non-empty, and I get a “bad CRC” error when unzipping).

I’ve also tried all of the above except downgrading from 1.86.0 to 1.74.0 - same error.

Mac, 16-inch 2019, OS 10.15.7

For now, I added a version resolution to my package.json. Yarn locked the offending version of node-archiver, archiver, even after downgrading AWS CDK.

package.json:

  "resolutions": {
    "archiver": "~5.1.0"
  }

I think the best solution is for the cdk cli to pin the archiver dependency to 5.1.0. The latest version seems to have timing issues.

CDK 1.86.0 has timing issues?

I mean archiver 5.2

I can confirm what @duntonr wrote. After deleting the zip files from the bucket and deleting the cdk.out directory, deployment works using NodeJS 14.15.4.

Having the same issue on a new M1 MacBook Air. Older versions (<15.x) of Node don’t work on the new Macs and v15.6.0 is giving me this error when doing cdk deploy

EDIT: I was able to get it working. Steps below…

1. Use nvm for version management
2. Create copy of iTerm app, rename to Rosetta iTerm and update to Open using Rosetta
3. Open Rosetta iTerm, and run nvm install v12.12.0
4. Use Rosetta iTerm to do cdk deploy commands

Thanks @machielg. I downgraded to Node v14.15.4 and to node-archiver 5.1.0. So far it is looking OK now.