aws-cdk: New Docker-based Parcel Build does not work on CircleCI (and likely other CI platforms)
In 1.38.0, the build was moved into a docker container by this PR: https://github.com/aws/aws-cdk/pull/7169
This is causing builds to fail on our CircleCI platform used for continuous delivery of CDK changes.
The issue
You can see the error in the CircleCI interface here: https://app.circleci.com/pipelines/github/blimmer/cdk-parcel-docker-circleci-issue/4/workflows/a4808b45-a04c-48af-851f-960f888518fd/jobs/7
We can easily start up docker-in-docker by using Circle’s setup_remote_docker command.
However, when the parcel image tries to run lscpu, the failure causes the build to break.
#!/bin/bash -eo pipefail
npx cdk synth
Failed to build file at /home/circleci/project/lib/index.ts: Error: [Status 1] stdout: 🚨 No entries found.
at Bundler.bundle (/usr/local/share/.config/yarn/global/node_modules/parcel-bundler/src/Bundler.js:275:17)
stderr: /bin/sh: lscpu: not found
Subprocess exited with error 1
Exited with code exit status 1
CircleCI received exit code 1
There are a number of issues on the parcel issue tracker about this issue on CI platforms: https://github.com/parcel-bundler/parcel/issues?q=is%3Aissue+sort%3Aupdated-desc+lscpu+is%3Aclosed
Reproduction Steps
I’ve created this example repo to show the problem: https://github.com/blimmer/cdk-parcel-docker-circleci-issue
Error Log
Failed to build file at /home/circleci/project/lib/index.ts: Error: [Status 1] stdout: 🚨 No entries found.
at Bundler.bundle (/usr/local/share/.config/yarn/global/node_modules/parcel-bundler/src/Bundler.js:275:17)
stderr: /bin/sh: lscpu: not found
Environment
- CLI Version : 1.38.0
- Framework Version: 1.38.0
- OS : MacOS / Ubuntu (on CircleCI)
- Language : English
Other
A fix
I can confirm that passing the PARCEL_WORKERS environment variable to the docker run parcel-bundler image resolves the problem. I got the idea to try that from this comment on the parcel side: https://github.com/parcel-bundler/parcel/issues/133#issuecomment-619991475
I used CircleCI’s SSH feature to interactively test this.
circleci@0d67793e2a50:~/project$ docker run parcel-bundler
Server running at http://localhost:1234
/bin/sh: lscpu: not found
🚨 No entries found.
at Bundler.bundle (/usr/local/share/.config/yarn/global/node_modules/parcel-bundler/src/Bundler.js:275:17)
at async Bundler.serve (/usr/local/share/.config/yarn/global/node_modules/parcel-bundler/src/Bundler.js:842:7)
at async Command.bundle (/usr/local/share/.config/yarn/global/node_modules/parcel-bundler/src/cli.js:241:20)
no errors when passing the PARCEL_WORKERS param.
circleci@0d67793e2a50:~/project$ docker run parcel-bundler -e PARCEL_WORKERS=2
Suggestion
Most all CI platforms set the CI environment variable to true. Maybe if CI is set, CDK automatically passes the PARCEL_WORKERS parameter in the docker run args?
https://github.com/aws/aws-cdk/pull/7169/files#diff-30cb98bb231179d8900591911d5cc8a4R75-R80
Alternatively, you could try to fix this problem in the docker image itself if lscpu is not present.
This is 🐛 Bug Report
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 17 (16 by maintainers)
Commits related to this issue
- Try setting nodeDockerTag to 12. https://github.com/aws/aws-cdk/issues/7912#issuecomment-626868347 — committed to blimmer/cdk-parcel-docker-circleci-issue by blimmer 4 years ago
- Try setting nodeDockerTag to 12. https://github.com/aws/aws-cdk/issues/7912#issuecomment-626868347 — committed to blimmer/cdk-parcel-docker-circleci-issue by blimmer 4 years ago
- feat(lambda-nodejs): allow passing env vars to container Add a `containerEnvironment` prop to pass environment variables to the container running Parcel. Closes #7912 Closes #8031 — committed to jogold/aws-cdk by jogold 4 years ago
- feat(lambda-nodejs): external and install modules (#8681) Add support for: * external modules: modules that should not be bundled. Defaults to `aws-sdk`. * install modules: modules that should not... — committed to aws/aws-cdk by jogold 4 years ago
For others that might be running into this problem in CircleCI, I highly recommend switching from the Docker executor to the Machine executor. Even with the improvements referenced in this ticket, there are still issues with how CircleCI runs docker-in-docker that causes tons of headaches with lambda bundling.
Simply change your job definition from something like this:
to using a machine executor like this:
Ended up here because I was having trouble getting our cdk deploy running in CircleCI. Can confirm that switching to the machine executor still works.
You can use https://github.com/ds300/patch-package until the env variable workaround gets released, after that I would personally create a
NodejsCircleCiFunction(or something with a broader scope that sets good defaults) that extendsNodejsFunction.@jogold It looks like the issue is resolved in Parcel 2. I tested this out in Circle CI using this Dockerfile:
Then I built and tested on CircleCI (no errors were reported).
It looks like they now have a try/catch in the Parcel source code: https://github.com/parcel-bundler/parcel/blob/db7e3a12105630abc44058bff7a88eb612f12e75/packages/core/workers/src/cpuCount.js
RE: the question about 2 vs. 1, I chose “2” just because it was listed in the original comment thread as a suggestion to fix on Heroku (https://github.com/parcel-bundler/parcel/issues/133#issuecomment-619991475).
It looks like they default to 1 in parcel v2 if they can’t determine how many CPU cores are present: https://github.com/parcel-bundler/parcel/blob/db7e3a12105630abc44058bff7a88eb612f12e75/packages/core/workers/src/cpuCount.js#L55-L58
There are cases where a CircleCI user might want to configure more workers to run based on the machine size they’re using. You can select between 1 - 20 vCPUs: https://circleci.com/docs/2.0/executor-types/#available-docker-resource-classes