ember.js: [2.15.0] - Ember build out of memory errors while using CI tools

Parallel babel transpiling was introduced in ember 2.15. (https://github.com/babel/broccoli-babel-transpiler#number-of-jobs)

By default, broccoli-babel-transpiler uses system resources (cpus) to determine the amount of jobs that can be ran in parallel.

Most modern CI tools use docker to help isolate builds from other builds running on a given server. This allows them to use large VMs to run lots of builds that are largely isolated from each other. For example, the VMs that Circle CI run on usually have 36 CPU cores… But the builds themselves are limited to two.

The issue comes in when attempting to use traditional means to determine what resources are available to the program. For example, to determine the number of CPUs using node you might do node -e "console.log(require('os').cpus().length);" . However, this information actual reports the instance resources, not the limited resources available to the docker container. Resulting in whatever is running thinking it’s got access to 36 cores but really it only has two.

I’ve created an example repo: https://github.com/mwisner/ember-circleci-example in which you can see both passed and failed build history (https://circleci.com/gh/mwisner/ember-circleci-example) (https://travis-ci.org/mwisner/ember-circleci-example/builds) (I have not fixed the travis build yet.)

Tested CI tools – Circle CI 2.0 – Circle CI 1.0 – Travis CI (Failing with default provided Travis CI)

Parallel jobs docs: https://github.com/babel/broccoli-babel-transpiler#number-of-jobs

When errors do come up they look something like like the output below. However in many situations (such as travis CI) it simply times out after 10mins without any information)

#!/bin/bash -eo pipefail
ember test
Could not start watchman
Visit https://ember-cli.com/user-guide/#watchman for more info.
Building
'instrument' is imported from external module 'ember-data/-debug' but never used
/usr/local/bin/node[1116]: ../src/node_file.cc:598:void node::InternalModuleReadFile(const v8::FunctionCallbackInfo<v8::Value>&): Assertion `(numchars) >= (0)' failed.
fs.js:682
  var r = binding.read(fd, buffer, offset, length, position);
                  ^

Error: ENOMEM: not enough memory, read
    at Object.fs.readSync (fs.js:682:19)
    at tryReadSync (fs.js:480:20)
    at Object.fs.readFileSync (fs.js:509:19)
    at Object.Module._extensions..js (module.js:579:20)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)
    at Module.require (module.js:498:17)
    at require (internal/module.js:20:19)
    at /home/circleci/app/node_modules/esutils/lib/utils.js:31:23
fs.js:682
  var r = binding.read(fd, buffer, offset, length, position);
                  ^

Error: ENOMEM: not enough memory, read
    at Object.fs.readSync (fs.js:682:19)
    at tryReadSync (fs.js:480:20)
    at Object.fs.readFileSync (fs.js:509:19)
    at Object.Module._extensions..js (module.js:579:20)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)
    at Module.require (module.js:498:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/home/circleci/app/node_modules/ember-power-select/node_modules/babel-core/lib/transformation/transformers/index.js:43:22)
fs.js:682
  var r = binding.read(fd, buffer, offset, length, position);
                  ^

Error: ENOMEM: not enough memory, read
    at Object.fs.readSync (fs.js:682:19)
    at tryReadSync (fs.js:480:20)
    at Object.fs.readFileSync (fs.js:509:19)
    at Object.Module._extensions..js (module.js:579:20)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)
    at Module.require (module.js:498:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/home/circleci/app/node_modules/debug/src/node.js:14:28)
 1: node::Abort() [/usr/local/bin/node]
 2: node::Assert(char const* const (*) [4]) [/usr/local/bin/node]
 3: 0x12e49fa [/usr/local/bin/node]
 4: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) [/usr/local/bin/node]
 5: 0xb45e2c [/usr/local/bin/node]
 6: v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
 7: 0x1cbf812040c7
fs.js:682
  var r = binding.read(fd, buffer, offset, length, position);
                  ^

Error: ENOMEM: not enough memory, read
    at Object.fs.readSync (fs.js:682:19)
    at tryReadSync (fs.js:480:20)
    at Object.fs.readFileSync (fs.js:509:19)
    at Object.Module._extensions..js (module.js:579:20)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)
    at Module.require (module.js:498:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/home/circleci/app/node_modules/debug/src/node.js:14:28)
fs.js:682
  var r = binding.read(fd, buffer, offset, length, position);
                  ^

Error: ENOMEM: not enough memory, read
    at Object.fs.readSync (fs.js:682:19)
    at tryReadSync (fs.js:480:20)
    at Object.fs.readFileSync (fs.js:509:19)
    at Object.Module._extensions..js (module.js:579:20)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)
    at Module.require (module.js:498:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/home/circleci/app/node_modules/regenerator/node_modules/ast-types/lib/node-path.js:6:12)
cleaning up
cleaning up...
Build failed.
The Broccoli Plugin: [BroccoliMergeTrees: Addon#treeFor (ember-concurrency - addon)] failed with:
Error: Worker terminated unexpectedly
    at ChildProcess.<anonymous> (/home/circleci/app/node_modules/workerpool/lib/WorkerHandler.js:177:17)
    at emitTwo (events.js:106:13)
    at ChildProcess.emit (events.js:194:7)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:215:12)

The broccoli plugin was instantiated at: 
    at BroccoliMergeTrees.Plugin (/home/circleci/app/node_modules/broccoli-plugin/index.js:7:31)
    at new BroccoliMergeTrees (/home/circleci/app/node_modules/broccoli-merge-trees/index.js:16:10)
    at Function.BroccoliMergeTrees [as _upstreamMergeTrees] (/home/circleci/app/node_modules/broccoli-merge-trees/index.js:10:53)
    at mergeTrees (/home/circleci/app/node_modules/ember-cli/lib/broccoli/merge-trees.js:85:33)
    at Class.treeFor (/home/circleci/app/node_modules/ember-cli/lib/models/addon.js:526:30)
    at addons.reduce (/home/circleci/app/node_modules/ember-cli/lib/models/addon.js:383:26)
    at Array.reduce (native)
    at Class.eachAddonInvoke (/home/circleci/app/node_modules/ember-cli/lib/models/addon.js:380:24)
    at Class.treeFor (/home/circleci/app/node_modules/ember-cli/lib/models/addon.js:515:22)
    at project.addons.reduce (/home/circleci/app/node_modules/ember-cli/lib/broccoli/ember-app.js:559:25)


Exited with code 1

The “Workaround” solution, provided by @rwjblue, is to specifically define the number of jobs you wish use for parallel transpiling by taking advantage of the JOBS ENV var. (https://github.com/mwisner/ember-circleci-example/blob/09c63e11c34d4cdfe602b63166b71e6f31e30f3c/.circleci/config.yml#L42)

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 23 (4 by maintainers)

Commits related to this issue

limit travis to 2 jobs see https://github.com/emberjs/ember.js/issues/15641 — committed to czosel/adsycc by czosel 7 years ago
Try jobs override in emberjs/ember.js#15641 — committed to backspace/prison-rideshare-ui by backspace 7 years ago
Try two jobs This is recommended here: https://github.com/emberjs/ember.js/issues/15641#issuecomment-328808660 — committed to backspace/prison-rideshare-ui by backspace 7 years ago
Add override for Babel parallel jobs Thanks to this thread for the solution to builds timing out: emberjs/ember.js#15641 — committed to backspace/prison-rideshare-ui by backspace 7 years ago
Circle CI: Reduce broccoli-babel-transpiler parallelism to avoid hitting memory limits. For details see: https://github.com/emberjs/ember.js/issues/15641 https://github.com/babel/broccoli-babel-trans... — committed to vickychijwani/quill by vickychijwani 7 years ago
fix for broccoli babel transpiler parallelism discussion here: https://github.com/emberjs/ember.js/issues/15641 TLDR circle, using docker, reports the available cores of the hardware, not the VM to ... — committed to nypublicradio/wnyc-web-client by deleted user 7 years ago
Attempt to force JOBS=1, based on https://github.com/emberjs/ember.js/issues/15641 — committed to caxiam/ember-tabular by aaronfischer 7 years ago
set jobs to 1 to avoid memory errors. see https://github.com/emberjs/ember.js/issues/15641#issuecomment-328496408 — committed to forge512/ember-printable-pages by chbonser 5 years ago
#14 make the test suite green (#22) * #14 make the test suite green * don't use named arguments for now... the ember-named-arguments-polyfill is not working and needs more investigation * set j... — committed to forge512/ember-printable-pages by chbonser 5 years ago
Add JOBS=1 env variable before running tests (https://github.com/emberjs/ember.js/issues/15641) — committed to emberobserver/client by pgengler 5 years ago
set JOBS=2 to prevent memory errors; see https://github.com/emberjs/ember.js/issues/15641#issuecomment-328496408 — committed to hashicorp/vault by noelledaley 4 years ago
set JOBS=2 to prevent memory errors; see https://github.com/emberjs/ember.js/issues/15641#issuecomment-328496408 (#8152) — committed to hashicorp/vault by noelledaley 4 years ago
set JOBS=2 to prevent memory errors; see https://github.com/emberjs/ember.js/issues/15641#issuecomment-328496408 (#8152) — committed to hashicorp/vault by noelledaley 4 years ago
set JOBS=2 to prevent memory errors; see https://github.com/emberjs/ember.js/issues/15641#issuecomment-328496408 (#8152) (#8159) — committed to hashicorp/vault by noelledaley 4 years ago
set ci jobs to 1 This is to prevent an out of memory error on circle-ci based on https://github.com/emberjs/ember.js/issues/15641 — committed to Duke-GCB/datadelivery-ui by johnbradley 3 years ago
Upgrade to ember 3.16 and node 14.15 (#83) * upgrade ember 3.16 Fixes requirements. Selecting new delivery recipient is broken. Re-enables jquery * fix tests after 3.16 upgrade Skipping onS... — committed to Duke-GCB/datadelivery-ui by johnbradley 3 years ago

Most upvoted comments

I chatted with @mwisner and @kiwiupover a bit about this over the weekend and we figured out that it has to do with the new parallelism that was added to broccoli-babel-transpiler. It defaults to paralellizing to the number of CPU’s currently present. Unfortunately, on CircleCI this shows as 36 CPU’s but the job itself is limited to 2 concurrent processes (and also limited in available RAM).

The fix here is to set the JOBS environment variable to 1 to essentially disable the parallelism on CI.

+14

rwjblue on Sep 11, 2017

@eric-hu does this also seem like a correct reading to you? Is this something we should expect a fix from Circle on so that the CPUs actually available in the job are shown? Does this prevent any sort of parallelism on Circle’s side (by splitting tests with something like ember-exam for example)?

@JoshSmith there’s two concepts of parallelism to keep in mind for CircleCI 2.0:

A. Per-command parallelism, limited to the number of cores available to a container group. By default, each container group is allocated 2 CPU shares, which guarantees they get 2 CPU cores. There’s a premium Configurable Resources feature that lets you choose a larger/smaller share (1, 4, 8 off the top of my head).

B. CircleCI parallelism, which you can think of as “how many machines [1] do I want to split this across?”. This is useful for test isolation, when you might want to run 2 tests at the same time and both write to a database. This is less useful for, say, transpiling your assets ; you probably want all your tests to run with the transpiled assets.

Regarding A: since you have 2 cores guaranteed available by default, you may be able to run the command you want with JOBS=2. This may speed up execution, but I haven’t checked if it works.

Regarding B: even with JOBS=1, you can still use CircleCI parallelism to speed up your test suite.

Regarding “who should fix this”, I’ve seen this as a long-running issue with multiple containerization tools. The CircleCI 2.0 and 1.0 containerization tools --Docker and LXC respectively-- leak information about the host system for many common Linux commands, like the ones used to check how many cores are available. It’s been this way for a number of years, I think if there were a simple fix it would have been solved by now. Further complicating things, CircleCI changed the CPU core availability model from 1.0 to 2.0. In 1.0, you got a fixed number of cores for a job. In 2.0, you get assigned CPU shares to guarantee your minimum number of cores. If you’re on a fully utilized host, you’ll get at least that many cores. If you’re running on an under-utilized host, you’ll have more cores available to you. Tools like broccoli-babel-transpiler tend to assign the number of cores they’ll use just once, though, and the available resources may change over the life of the program’s execution. It’s best to just hard-code your CI scripts to use the guaranteed resources available.

[1] Your code may not be running on N machines for N parallelism. But you can think of it this way, as they’re effectively isolated from one another.

eric-hu on Sep 12, 2017

I just had to add JOBS=1 to fix this issue on 3.16.0. Was there a regression?

robclancy on Mar 4, 2020

I just ran into this same issue on CircleCI, and JOBS=1 also fixed it for me, thanks all ✌️

bgentry on Sep 12, 2017

Just tried JOBS=2 on my app and the build was successful as well. Should that be the canonical advice here?

No noticeable time difference between those values in my small sample FWIW.

bgentry on Sep 12, 2017

I’m not sure if we can expect a fix form circleci themselves… I am not super 100% but I think it’s more of a Docker problem than a circle problem specifically. Or maybe a Node + Docker issue not being able to accurately detect the cpu/mem limitations imposed by the docker container.

I have created this repo for experimentation purposes: https://github.com/mwisner/ember-circleci-example.

It includes circleci 2.0 (working w/ the JOBS=1 workaround provided by @rwjblue) (https://github.com/mwisner/ember-circleci-example/blob/master/.circleci/config.yml)

Along with the public circle builds: https://circleci.com/gh/mwisner/ember-circleci-example/83

However I’ve also added the repo to travis ci, which also uses Docker for builds, I haven’t fixed the travis config file yet but you can see the travis builds are failing with the provided travis config (https://travis-ci.org/mwisner/ember-circleci-example/builds)

mwisner on Sep 11, 2017

I reopened this issue because after doing some additional testing the provided travisci.yml config that ships with ember is also running into this issue.

While I understand that ember doesn’t support circle, I do think that it would be nice to at least have the issue fixed in the shipped travisci.yml file.

I’ve also updated the title and description to make it a little more generic and not scoped specifically to circleCI

mwisner on Sep 12, 2017

It’s also not specific to testing. The OOM issue is caused during the build phase of ember test. I’ve tested just running ember build as well with the same OOM result.

@JoshSmith I do believe setting JOBS=1 at the env var level would work as well but I haven’t confirmed.

I noticed there is a pattern for setting env vars for ember cli commands for a few addons:

https://github.com/ember-cli/broccoli-viz#usage https://github.com/kategengler/ember-cli-code-coverage#usage

So I was just going off of those usage patterns.

mwisner on Sep 11, 2017

@JoshSmith Yeah if I understand correctly it’s to disable parallelism within broccoli-babel-transpiler. Not circle itself. So in theory using circleci parallelism + jobs=1 would be fine?

But personally, I haven’t experimented with using circleci’s parallelism functionality. So I’m not 100% sure what a configuration file for that would look like.

mwisner on Sep 11, 2017