angular-cli: Angular 7/8 FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

🐞 Bug report

Command (mark with an x)

- [ ] new
- [X] build
- [ ] serve
- [ ] test
- [ ] e2e
- [ ] generate
- [ ] add
- [ ] update
- [ ] lint
- [ ] xi18n
- [ ] run
- [ ] config
- [ ] help
- [ ] version
- [ ] doc

Description

With the last merge project build process is not working. I get failure that is pasted below. Tried to change --max-old-space-size=4096 still not working. Any suggestions what this can be ?

🔬 Minimal Reproduction

run command ng build --prod in angular.json file we have

"production": {
	"optimization": true,
	"outputHashing": "all",
	"sourceMap": false,
	"extractCss": true,
	"namedChunks": false,
	"aot": true,
	"extractLicenses": true,
	"vendorChunk": false,
	"buildOptimizer": true,
	"fileReplacements": [
	{
		"replace": "src/environments/environment.ts",
		"with": "src/environments/environment.prod.ts"
	}
	]
},

🔥 Exception or Error

Here is the log file.

{ Error: Command failed: ng build --prod --configuration=config --output-path=test --base-href=/test/
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8dbaa0 node::Abort() [ng build --prod --configuration=config --output-path=test --base-href=/test/]
 2: 0x8dbaec  [ng build --prod --configuration=config --output-path=test --base-href=/test/]
 3: 0xad83de v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
 4: 0xad8614 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
 5: 0xec5c42  [ng build --prod --configuration=config --output-path=test --base-href=/test/]
 6: 0xec5d48 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
 7: 0xed1e22 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
 8: 0xed2754 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
 9: 0xed53c1 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
10: 0xe9d636  [ng build --prod --configuration=config --output-path=test --base-href=/test/]
11: 0xeafee7 v8::internal::Factory::NewLoadHandler(int) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
12: 0xf2f4db v8::internal::LoadHandler::LoadFromPrototype(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::JSReceiver>, v8::internal::Handle<v8::internal::Smi>, v8::internal::MaybeHandle<v8::internal::Object>, v8::internal::MaybeHandle<v8::internal::Object>) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
13: 0xf36d1f v8::internal::LoadIC::ComputeHandler(v8::internal::LookupIterator*) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
14: 0xf3d94c v8::internal::LoadIC::UpdateCaches(v8::internal::LookupIterator*) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
15: 0xf3dffc v8::internal::LoadIC::Load(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Name>) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
16: 0xf42935 v8::internal::Runtime_LoadIC_Miss(int, v8::internal::Object**, v8::internal::Isolate*) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
17: 0x235e3925be1d

🌍 Your Environment


Angular CLI: 7.3.1
Node: 10.15.1
OS: linux x64
Angular: 7.2.6
... animations, common, compiler, compiler-cli, core, forms
... http, platform-browser, platform-browser-dynamic, router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.13.2
@angular-devkit/build-angular     0.13.2
@angular-devkit/build-optimizer   0.13.2
@angular-devkit/build-webpack     0.13.2
@angular-devkit/core              7.3.2
@angular-devkit/schematics        7.3.1
@angular/cdk                      7.3.3
@angular/cli                      7.3.1
@ngtools/webpack                  7.3.2
@schematics/angular               7.3.1
@schematics/update                0.13.1
rxjs                              6.4.0
typescript                        3.1.6
webpack                           4.29.0

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 171
  • Comments: 205 (38 by maintainers)

Most upvoted comments

same issue on ubuntu server 18.04, but this solved my problem: node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng build --prod

UPDATE :-

I’ve seen this issue only on WINDOWS machine particularly. Anyone else, noticed that? Or have got same issue in other machines?

Note: I tried 2 Windows machine, all tests gave same result. Both machine have same config almost. i.e. 8GB RAM/Admin Access For Bash/CMD/PowerShell/ i3 & i5 Processor (quad core, both).

And, I admit that the project that I’m working on is a MESS in terms of code quality. I tried one clean code, full fledged application, ON WINDOWS, it was slower in build process, but worked fine.

I tried MESSY project’s code, it never compile on default node heap allocation (around 1.2 GB, not sure.). It works if I allocate more heap memory to node process, say 5 GB. But, in that case too, it took more than 30 mins to build. I tried the 5 GB node heap option with and without --aot AND/OR --prod .

Same code when tried on CentOS could instance of 1GB RAM/ Virtual Xenon Processor (1 Core), generates build within 90 Seconds. with and without --aot AND/OR --prod

I’m not sure, but, looks like problem is with Node Service not with Angular CLI itself, coz I’ve seen some performance lag in new NodeJS releases after last update (locally on WINDOWS machine only).

Some observations below.:-

Even though allocating 5 GB of heap, it never used more than 1.9 GB (max) and allocation process was very slow… it was increasing RAM consumption 1-2 MB per 3-5 seconds. And, was not using Disk Resource as it used to be in older version. (Disk access was very high previously on WINDOWS machine, but build process was faster without extra RAM allocation. This time, same machine almost same code with some bug fixes, but node was upgraded, disk consumption was lowered but slow build generation.)

Hope somebody can confirm this.

Hey all,

Just wanted to give you an update on the steps we (and others in the community) took to address this problem in Angular 8:

These efforts are still under way:

Here’s what you can do to take advantage of these fixes:

  • Update to @angular/core@8.2.2 (together with the other framework packages)
  • Update to @angular/cli@8.3.0-rc.0 and @angular-devkit/build-angular@0.803.0-rc.0
  • Update to typescript@3.5.3
  • Install node-sass as a dev dependency if you use Sass
  • Use Node 12.8

One special note is that we completely reworked how Differential Loading works. It should now be much faster, but can use more system memory depending on the numbers of cores you have. This is different from the node memory limit.

When you hit the node memory limit you see the FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory error. But if you hit the system memory limit you should see Fatal process OOM in insufficient memory to create an Isolate.

The new Differential Loading setup makes it rarer to hit the first, but if you have many cores but not a lot of memory, it can be more common to hit the second. We’d like feedback on @angular-devkit/build-angular@0.803.0-rc.0 please, so that we can address this behaviour if it’s a problem, by disabling parallelism in some cases or make it configurable.

Upgrading to 12.x worked for me. Loving Angular less and less each day…

I updated the node.js to the latest LTS (at this moment 12.13.0) version and everything work perfectly. No need anymore set the max-old-space-size.

@noopur-dabhi I don’t think it is fair to say that using node --max_old_space_size=8192 is not best practice.

It is nothing but allowing NODE process (which is going to build your app now) to use more than DEFAULT heap memory (from your RAM or SWAP area).

And this type of allocation is very very common in most of the app development (especially deployment) process. Example, in Java, if we want to restrict our java app heap usage, we add additional flag like --Xmx64m this means maximum heap memory allowed is 64MB.

So if ‘Best Practice’ is the only concern, I think it’s not a problem using node --max_old_space=8192, this means you are allowing NODE process to use maximum of 8192MB from memory. nothing to do with Best Practice Concern.

Same issue here running on a Mac book pro. macOs Mojoave. Any environment with production: true

Build Problem Solved. Suggestion is to change build logic. Instead of angular ng build use node build

set NODE_OPTIONS=–max-old-space-size=30720 command worked for me.

Heya all,

First of all sorry for the delay. I AFK for a good chunk of last week.

I’m trying to give as much detail about what I did to investigate this sort of problem because in the past I have been asked about it (looking at you @wardbell, @johnpapa, and @DanWahlin 👋). I also find it useful for myself to come back to these write-ups in the future because I forget the details of how to do these things, or want to share with colleagues so they can do the same. A lot of work described here is done by fiddling directly in node_modules and changing logic in there to write files that I later debug, or log information, or use different packages, stuff like that. I find that fiddling with the JS sources really is the best way of investigating things because I don’t know what I’m looking for most of the time, so it helps to explore.

At this point we’ve already ascertained that Node 12 has a default 2048MB memory limit (https://github.com/nodejs/node/issues/28202) and is using more memory than Node 10 (https://github.com/nodejs/node/issues/28205). This is unfortunate but a reality at the moment, so we should focus on Node 10 for now while the node folks address memory issues. I’m using Node 10.16.0 myself for testing.

I’ve been working on profiling where memory is used in the problematic builds. The first step to do that is decide on what project to get profiling data from. I chose @johnpapa’s awesome-angular-workshop because in https://github.com/angular/angular-cli/issues/13734#issuecomment-500849058 I came to the conclusion it was the project where we could see the biggest impact:

awesome-angular-workshop suffers a disproportionate increase in memory usage (45%) and build time going (150%) from CLI7 to CLI8DL. This indicates a localized regression.

I made a fork of it at https://github.com/filipesilva/awesome-angular-workshop, commit SHA 9076a3d, with local modifications to get reproducible benchmarks easily. That’s the code I used in https://github.com/filipesilva/angular-cli-perf-benchmark. The build command I use is ng build 5-ngrx-end --prod, since this workspace has several projects and I just want to use this specific one.

My initial idea was to use the Chrome Node DevTools, gather a bunch of profiling data, then work away at trying to explore it to find actionable information.

To open the Chrome Node DevTools, open chrome://inspect/#devices in Chrome and click “Open dedicated DevTools for Node”. When you have a node process started with the --inspect-brk flag, DevTools should automatically attach. An easy way to run Angular CLI like this is to make a npm script: "ng-inspect": "node --inspect-brk node_modules/@angular/cli/bin/ng",. Then if you run npm run ng-inspect -- build --prod and have DevTools open, it will automatically connect. At this point you can set breakpoints and take profiles.

The memory tab has a couple of different profiling tools and descriptions of what they do. When you use these, you will have to go to sources and click the resume script execution (top right) before things happen. I spent some time trialling the different types of profiles on simpler node processes to figure out which one would be better. I was already used to reading the CPU profiles from when I debug build time, so the Allocation sampling (aka heap profile) sounded interesting because it should show how much memory each function is using at a given time. Heap snapshots show whats in memory at a given time and what is retaining those objects, it would also work but isn’t as easy to read and get the big picture of what’s happening.

To be honest capturing profiles with DevTools has been a lot harder than I expected. It actually does not seem to deal well with capturing profiles from large running Node processes. I saw it crashing while taking profiles, or at the end when processing them, for most of the profiles I tried to take.

I tried to explore alternatives. One of them was to profile smaller parts at a time so that DevTools didn’t crash, but it looked like it would always crash on some types of profiles anyway, even on new CLI projects, so that didn’t seem to matter much. I tried ndb but it seemed to suffer from the same problems. automated-chrome-profiling seemed promising byt hadn’t been updated in a while, and only took CPU profiles. @manekinekko suggested but as far as I could tell it wouldn’t give me very granular information on memory usage. node-oom-heapdump looked good for heap snapshots but required native bindings, which I didn’t like much because it would make it harder to later integrate directly into the CLI for user profiling. sampling-heap-profiler looked great for heap profiles, but also needed native bindings. @ofrobots suggested https://github.com/google/pprof-nodejs which I tried and seemed to give me really nice visualizations, but it also needed native bindings and needed a go tool to read it, which would make it harder to integrate.

Ultimately I opted to make some custom profiling code based on the Node experimental profiler support described here. I drew inspiration from code in sampling-heap-profiler and the profiler API to figure out what to use. From this I prototyped some automated profiling code that looks like https://github.com/angular/angular-cli/pull/14936. At the time of writing I manually control flags inside the file but maybe in the future we can expose it somehow. It can take CPU profiles of the whole process, or heap snapshot/profiles at set intervals. This seemed like a good way of doing it because I might want to ask people with build speed or memory problems in the future to take these profiles and send to us.

With this setup I was able to take reproducible profiles with minimal fiddling around. Reducing the fiddling is really important when it takes a long time to gather your information, because the more fiddling you need to do, the higher the likelyhood of you botching a given sample. If you botch a sample and only find it hours later, then you might have to throw away most of your conclusions for that period since your starting points were bad and so conclusions and comparisons weren’t accurate.

Armed with this need tool I managed to capture a few heap profiles of CLI7 and CLI8 (without differential loading) builds to compare them. I opened them on DevTools and tried to figure out what parts of the build pipeline were involved in each section of the profile.

The CLI7 one looked like this: cli7-annotated

The CLI8 one looked like this: cli8-annotated

CLI8 uses around 180mb more memory, for a total of 760mb, versus the 580mb used by CLI7. What stood out the most to me was the difference in CSS processing. It looks like it takes a lot longer in CLI8. In CLI8 we started using dart-sass (published in npm as just sass) by default instead of node-sass (see https://github.com/angular/angular-cli/pull/11791, https://github.com/angular/angular-cli/pull/13950) so it’s a great candidate for investigation.

We expected dart-sass to not be as performant as node-sass so we let you go back to node-sass by installing it in your project (npm install node-sass --save-dev). As long as it’s installed, we’ll pick it up automatically. You can also use dart-sass with the fibers package for better performance by installing it in the same way as node-sass. Both node-sass and fibers use native bindings and might not work on all machines, which is why we don’t install them by default.

Then I compared profiles for CLI8, CLI8 with node-sass, CLI8 with fibers, and CLI 7 side by side to see how they changed. sass-comparisons

So using node-sass seems to return memory usage CLI7-levels, at least on this project. Using just fibers only reduces memory by some 20mb.

As @clydin pointed out, this might not be the full picture since the heap profiles are only for memory managed by V8. node-sass might be using more memory that doesn’t show up in the profile. So I benchmarked the processes themselves:

  • CLI 8
[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build 5-ngrx-end --prod (at D:\sandbox\memory-debug\awesome-angular-workshop)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 157486.80 ms (157292.00, 159517.00, 156464.00, 159606.00, 154555.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 18.31 % (18.68, 18.63, 18.04, 18.76, 17.46)
[benchmark]   Peak CPU usage: 205.60 % (237.50, 204.60, 198.40, 196.90, 190.60)
[benchmark]   Average Memory usage: 1008.24 MB (1048.68, 975.86, 1017.15, 1020.31, 979.21)
[benchmark]   Peak Memory usage: 1560.48 MB (1588.38, 1489.02, 1583.70, 1574.51, 1566.81)
  • CLI 8 with node-sass
[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build 5-ngrx-end --prod (at D:\sandbox\memory-debug\awesome-angular-workshop)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 103204.00 ms (98623.00, 103302.00, 105260.00, 104674.00, 104161.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 16.70 % (15.46, 15.94, 17.36, 17.48, 17.26)
[benchmark]   Peak CPU usage: 152.50 % (160.90, 89.10, 190.60, 128.10, 193.80)
[benchmark]   Average Memory usage: 676.14 MB (681.11, 674.49, 676.65, 675.08, 673.38)
[benchmark]   Peak Memory usage: 1443.15 MB (1444.21, 1439.97, 1435.09, 1449.96, 1446.53)
  • CLI 8 with fibers
[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build 5-ngrx-end --prod (at D:\sandbox\memory-debug\awesome-angular-workshop)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 126098.00 ms (136145.00, 125063.00, 123654.00, 123849.00, 121779.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 14.55 % (16.18, 14.48, 14.25, 14.05, 13.79)
[benchmark]   Peak CPU usage: 190.30 % (215.50, 173.40, 184.40, 170.30, 207.90)
[benchmark]   Average Memory usage: 746.22 MB (752.84, 741.56, 745.46, 747.52, 743.72)
[benchmark]   Peak Memory usage: 1560.88 MB (1603.68, 1555.96, 1575.40, 1569.62, 1499.73)

So it does look like the node-sass really is the one that uses less memory, and takes less time to build. fibers is the second best. The average memory used in the fibers case seems to conflict with the results we got from the heap profiles though, here we see CLI8 with fibers using ~750mb versus the original CLI8 ~1000mb, but in the heap profile using fibers only reduced memory by ~20mb. I don’t know why this is. Maybe it’s because of the nature of averaging used memory, or maybe it’s because there are more garbage collection cycles taking heap profiles. But at the end of the day node-sass seems to be the winner as far as performance goes.

I also found it weird that a big chunk of extra memory in the CLI8 heap profile seems to come from PostCSS processing. I wasn’t expecting PostCSS to need to do anything different. I thought that maybe I just misinterpreted what that section was used for. But maybe the dart-sass output could be different from node-sass and that could cause more work for PostCSS. It was worth looking into so I added some code directly in sass-loader to write the results of sass compilation to disk. The output for dart-sass and node-sass seemed like it was the same really, aside for some whitespace, so I think I just misinterpreted the memory assigned to Sass processing as being assigned to PostCSS.

@johnpapa, as an aside, I did notice that most component CSS files in this project are importing mixin.scss. This file actually ~has no mixins and is just~ includes the whole material theme at 70kb (pre-optimizations) of toplevel CSS definitions. Component CSS is standalone and isolated from other CSS, so there is no deduplication or anything. When it’s imported into component CSS, you’re adding 70kb worth of CSS into that component, which is probably several times over the size of the component itself. @jelbourn has described importing non-mixin Sass this as an anti-pattern but it’s not the sort of information that’s obvious all round. We could do a better job of informing people about it, and warning them when component CSS is very large (cc @vikerman).

Another aside is that I also noticed that compiling just the single project via ng build 5-ngrx-end --prod actually compiles all .ts and .scss files inside src/. This happens because all projects in this workspace use the same ./src/tsconfig.app.json, and this tsconfig includes all ts files (except unit tests). Which is to say, compiling a single project here actually compiles all projects. The only way to truly address this is to have one tsconfig for each app. I wonder how common this mistake is. I wouldn’t be surprised larger projects accidentally include extra TS files. Maybe we could warn users when we see the TS compilation has a lot more files than those that webpack loads (cc @vikerman, @clydin).

So I guess the takeaway here is that if you have a large project, you should install node-sass to your devDependencies. This will make building faster and take less memory.

Although it didn’t seem to change between CLI7 and CLI8, the Angular compilation itself is still by far the largest contributor to the memory footprint, at around 230mb. Maybe we can fully drop all references to it when not building in watch mode, allowing Node to free up that memory. I’ll explore that next.

Has anyone experienced this in a project that doesn’t use sass?

  • new
  • build
  • serve prod
  • test
  • e2e

I am facing same issue while ng serve --prod

Reference Image image

@Flyrell I have the same issue with Nodejs 14.4.0 and Angular 10

I updated the node.js to the latest LTS (at this moment 12.13.0) version and everything work perfectly. No need anymore set the max-old-space-size.

This worked for me!

I updated the node.js to the latest LTS (at this moment 12.13.0) version and everything work perfectly. No need anymore set the max-old-space-size.

This worked for me!

Seems like the issue is back with v10 release. Using @angular-devkit/build-angular v0.1000.0 and @angular/* packages v10.0.0.

Node: v14.1.0 NPM: 6.14.5

Stack trace:

<--- Last few GCs --->

[13013:0x108008000]  2808381 ms: Scavenge 2003.1 (2051.0) -> 2002.5 (2051.5) MB, 4.0 / 0.0 ms  (average mu = 0.117, current mu = 0.045) allocation failure 
[13013:0x108008000]  2809520 ms: Mark-sweep 2003.3 (2051.5) -> 2002.1 (2051.0) MB, 1132.9 / 0.0 ms  (average mu = 0.073, current mu = 0.026) allocation failure scavenge might not succeed
[13013:0x108008000]  2809535 ms: Scavenge 2003.1 (2051.0) -> 2002.5 (2051.5) MB, 3.3 / 0.0 ms  (average mu = 0.073, current mu = 0.026) allocation failure 


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x100bc569b node::Abort() (.cold.1) [/usr/local/bin/node]
 2: 0x1000816b5 node::FatalError(char const*, char const*) [/usr/local/bin/node]
 3: 0x10008181e node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
 4: 0x100180909 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 5: 0x1001808b3 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 6: 0x1002a0bfd v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
 7: 0x1002a1f60 v8::internal::Heap::MarkCompactPrologue() [/usr/local/bin/node]
 8: 0x10029f947 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
 9: 0x10029e112 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
10: 0x1002a60d4 v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
11: 0x1002a612a v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
12: 0x10028417d v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/usr/local/bin/node]
13: 0x1004e7b16 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
14: 0x10074fe99 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/usr/local/bin/node]
zsh: abort      npm run start

After upgrading Node.js to v14.4.0 the issue seems to be resolved.

I updated the node.js to the latest LTS (at this moment 12.13.0) version and everything work perfectly. No need anymore set the max-old-space-size.

Worked for me too

I can confirm, that this is working.

Of note as well is that Node.js 12 automatically configures its memory options based on the available system memory.

Please provide me the solution for this build problem

@filipesilva just install Node 12.8 solved the issue . thanks.

In my case the problem only appears, when building --prod --source-map. The normal --prod build runs just fine.

Using --max_old_space_size=8192 as suggested by @nokhodian it works with memory consumption peaking at ~3.8GiB during source map generation.

Build environment:

Angular CLI: 7.3.8
Node: 10.15.2
OS: linux x64
Angular: 7.2.12
... animations, common, compiler, compiler-cli, core, forms
... http, language-service, platform-browser
... platform-browser-dynamic, router, service-worker

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.10.4
@angular-devkit/build-angular     0.10.4
@angular-devkit/build-optimizer   0.10.4
@angular-devkit/build-webpack     0.10.4
@angular-devkit/core              7.0.4
@angular-devkit/schematics        7.1.0
@angular/cdk                      7.3.7
@angular/cli                      7.3.8
@angular/flex-layout              7.0.0-beta.19
@angular/material                 7.3.7
@angular/pwa                      0.11.0
@ngtools/webpack                  7.0.4
@schematics/angular               7.1.0
@schematics/update                0.13.8
rxjs                              6.4.0
typescript                        3.2.4
webpack                           4.19.1

use “build:prod”: “node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng build --configuration=prod”, in your package.json file 1

this will tell node to exceed default memory allocation by given size in command.

Just happened to me after updating from 9.0.3 to 9.0.5

I updated the node.js to the latest LTS (at this moment 12.13.0) version and everything work perfectly. No need anymore set the max-old-space-size.

Worked for me too

I updated the node.js to the latest LTS (at this moment 12.13.0) version and everything work perfectly. No need anymore set the max-old-space-size.

This worked for me!

Updating to latest node.js worked for me too!

"scripts": {
        "start": "node --max-old-space-size=4096 ./node_modules/@angular/cli/bin/ng serve",
        "start:prod": "node --max-old-space-size=4096 ./node_modules/@angular/cli/bin/ng serve --prod",
}

Above config solves for me

I also have this issue but can resolve it with “NODE_OPTIONS=–max-old-space-size=8192”. Ram usage during build gets up about 14GB.

I have the same error in Angular 8 and this solved my problem:

node --max_old_space_size=X node_modules/@angular/cli/bin/ng build --prod

Where X = (2048 or 4096 or 8192 o..) is the value of memory

Still working on this problem. Notes from the last few days below.

While discussing it with @clydin, he mentioned that the average memory isn’t really what busts the limit. It’s the peak memory. Average is ~750mb while peak is ~1500mb. We should focus on that instead.

First I need to figure out what’s using memory at the peak. I guess it’s uglify and build optimizer but I don’t know. It’s weird that the profiles I take never show peak usage though. I wonder I I’m doing something wrong there. I just started a process without any profiling and see memory at ~1200 during what seems like the ng compilation, then going down to ~900mb when loading files, then ~1300 and ~1500mb at the 90%+ stages. But my profiles show a ~800mb allocation max.

Maybe I need to control the timing of when I take these profiles. If that’s it, then I need to find the exact time when usage is high and take a profile there. Using process.memoryUsage() as described in https://www.valentinog.com/blog/memory-usage-node-js/ might let me identify high memory usage periods. I can set a memory threshold and only take profiles at that time.

It might be that taking a heap profile triggers GC, or just that the increased running time prompts GCs more often. I can try to trigger gc right before each heap profile and see if the result is the same. https://stackoverflow.com/questions/27321997/how-to-request-the-garbage-collector-in-node-js-to-run shows how. Did that, and saw no real difference in the profiles. Whatever the profiles show seems to actually be retained.

I used setInterval with a log for process.memoryUsage() inside, and also kept an eye on the reported task manager memory usage, and noticed a couple of things. When I say that I saw 1234/5678 I mean 1234 was on the task manager and 5678 was on the process.memory logging.

The highest 1300/??? happened during the Angular Compiler phase, but only with dart-sass. With node-sass it lower at 900/700 and saw interval logging. I got almost no logging for that phase with dart-sass, I think because the setInterval never got on the event loop properly, I guess because a lot of things are sync? With node-sass I saw a lot of logging at that time. Maybe it’s a native binding thing, it might free up the event loop. Anyway dart-sass seems to make resource loading in the angular compiler extra bad by piling on memory on an already expensive set of operations.

I saw some peaks of 1200/900 while progress showed loading individual files. I think that’s Build Optimizer since its a js loader.

I barely saw any memory usage during Terser, but I bet that’s because of the cache. I turned off the terser-webpack-plugin cache and saw 1100/940, but also a bunch of spawned processes between 40 and 400mb.

So as far as I could tell, the sass thing was actually a big part of reaching the peak because it runs during the ng compilation and piles on something that’s already memory intensive.

I bet it doesn’t help that we’re doing webpack subcompilations for that either. This also means that even if we freed all references to memory using during the ng compilation, it wouldn’t matter as much because that compilation is still part of the peak.

We already have plans to remove Build Optimizer so that doesn’t bother me much

In regards to terser… Well if we had a smaller main process memory footprint, obviously the total would be reduced. But maybe we can also make that plugin smarter. It seems to run on chunks proper https://github.com/webpack-contrib/terser-webpack-plugin/blob/a0d83170168e786004fa12b94525b2934a17a4f5/src/index.js#L416-L419 But we know terser can’t break out of the webpack module loader so there’s no point is processing whole chunks. Might as well process the individual webpack modules, that should be much smaller.

So important things to follow up:

I was facing same issue for ng serve,

npm cache clean --force resolve this issue

@maxisam I’m on 10.x until AWS decides to use a more recent version or I decide to create a custom Lambda Layer. But this should be working on 8.9+ according to the docs.

From https://www.npmjs.com/package/@angular/cli

Both the CLI and generated project have dependencies that require Node 8.9 or higher, together with NPM 5.5.1 or higher.

I was going to add a rant here, but I’m just going to send it to the bosses with a request to switch frameworks. It seems like this project is just in total chaos.

Here’s something fun to try (8.1.3):

npm install -g @angular/cli ng create new-project cd new-project ng update “rxjs 6.4.0 -> 6.5.2 ng update rxjs”

Why does a brand new project already require a package update? 6.4.0 was in Jan. 6.5.2 was in May.

Angular CLI https://github.com/angular/angular-cli/blob/cae4a9dc4afb53292cb44ebbc644d33b7d8d159e/packages/schematics/angular/utility/latest-versions.ts

RxJs: ‘~6.4.0’,

Angular https://github.com/angular/angular/blob/master/package.json

"rxjs": "^6.4.0",

We are experiencing this problem when we use ng build --prod Sometimes needed because we use different environment settings. node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng serve --prod solves the problem but the build time is exceeded to 3 minutes. I don’t think this issue is solved

We’re using Bitbucket pipelines and we can’t increase the memory limit to 8GB. Also our application takes about 7-10min to build and we need the sourceMaps for our Sentry error reports.

UPDATE: running ng update @angular/cli @angular/core solved the problem.

for me the solution was editing my tsconfig (ionic4 and angular8)

i was playing around adding properties to tsconfig, towards the end i enabled source maps for my prod build inside angular.json.

i had to remove the sourceMap: true and sourceBase: "/" from tsconfig , after that --prod works like a charm

@Senneseph any chance I could see that project? I don’t think a 24 component project should use 16gb of memory. I haven’t seen any project that uses that much memory at all really

There was a severe, but rare, but with typescript 3.4 (https://github.com/microsoft/TypeScript/issues/30050) that caused a lot of extra memory to be used in some cases with union types. We’ve just released support for TS 3.5 in Angular 8.2.0 with @angular-devkit/build-angular@0.802.0-rc.0. That might be what’s affecting your project, you can try updating to those versions to see if it helps.

I have a relatively small app, maybe 24 components, including routes, modals, and a few classes for state management, etc. And on a 16GB machine, even with all of the suggestions posted here, there’s a 3/4 chance it will hit this bug. I don’t understand how it can use this much memory and really don’t understand why it isn’t at least trying to page it out to the SSD (sure, it would be slow, but if it goes from not building at all, to building in 10 minutes, 10x the old build time, that’s still pretty great).

This is with sourcemaps off, node getting the max_old_space_size setting, buildOptimizer off, optimization off.

3 versions ago, this same app could build with a 3GB machine. Should I really roll everything back to Angular 5 or am I better off just switching to Vue?

If your application crash while building scss files try installing those dependencies node-sass, style-loader and sass-loader, this worked for me!

If it helps anyone else, when we switched from “@angular-devkit/build-angular”: “0.803.26” to “@angular-devkit/build-angular”: “0.803.27” it caused memory to run away during prod builds at ~43% build progress. It would get stuck at this step and escalate to >25GB of RAM rather than ~1.5GB.

When we switched back to “@angular-devkit/build-angular”: “0.803.26”, it worked again. 0.803.27 is a recent release around 6/11/20, so it would only explain new occurrences of this issue.

With the latest versions of node is solved (at least for me). I recommend the team node to use in the package.json constructs Engines: https://docs.npmjs.com/files/package.json#Engines

Had the same issue and upgrading node to 12.12 today let me ng build --prod

@filipesilva, I confirm that ng build is working well now with:

  • angular 8.2.2
  • angular/cli 8.2.2
  • node 12.4

ng build --configuration=production

angular.json

					"configurations": {
						"production": {
							"optimization": true,
							"outputHashing": "all",
							"sourceMap": false,
							"extractCss": true,
							"namedChunks": false,
							"aot": true,
							"extractLicenses": true,
							"vendorChunk": false,
							"buildOptimizer": true,
							"fileReplacements": [
								{
									"replace": "src/environments/environment.ts",
									"with": "src/environments/environment.prod.ts"
								}
							],
							"serviceWorker": true
						}

A few months ago when you tested my big fat SPA, the memory consumption of node was over 2GB, so I had to use: node --max_old_space_size=8192 "node_modules/@angular/cli/bin/ng" build --configuration=production

Now with a larger codebase being built with ng build, the memory peak of node is 1.6GB.

Good work. 😃

Last week, after merging a relatively small branch into the master branch, the build of my project started to experience this memory issue reported here. The interesting part is that the problem was happening on the CI server only, none of the developers was having trouble locally, even with --prod.

After reading @filipesilva 's comment above, I decided to update angular as follows:

$ ng update @angular/cli @angular/core
Using package manager: 'npm'
Collecting installed dependencies...
Found 60 dependencies.
Fetching dependency metadata from registry...
    Updating package.json with dependency @angular/cli @ "8.2.1" (was "8.1.2")...
    Updating package.json with dependency @angular/core @ "8.2.1" (was "8.1.2")...
    Updating package.json with dependency @angular/animations @ "8.2.1" (was "8.1.2")...
    Updating package.json with dependency @angular/compiler-cli @ "8.2.1" (was "8.1.2")...
    Updating package.json with dependency @angular/language-service @ "8.2.1" (was "8.1.2")...
    Updating package.json with dependency @angular/router @ "8.2.1" (was "8.1.2")...
    Updating package.json with dependency @angular/forms @ "8.2.1" (was "8.1.2")...
    Updating package.json with dependency @angular/common @ "8.2.1" (was "8.1.2")...
    Updating package.json with dependency @angular/platform-browser-dynamic @ "8.2.1" (was "8.1.2")...
    Updating package.json with dependency @angular-devkit/build-angular @ "0.802.1" (was "0.801.2")...
    Updating package.json with dependency @angular/compiler @ "8.2.1" (was "8.1.2")...
    Updating package.json with dependency @angular/platform-browser @ "8.2.1" (was "8.1.2")...
    Updating package.json with dependency typescript @ "3.5.3" (was "3.4.5")...
UPDATE package.json (2341 bytes)

The problem is now fixed!

My impression is that the extra memory needed for DL dual builds, plus the fact that typescript was bumped to 3.4 which has the aforementioned problems regarding union type edge cases, started causing this extra memory consumption for some projects.

@johannesjo both Build Optimizer and source maps need extra memory to process so that’s expected. It’s just more work to be done, and more intermediate data to store in memory. We are looking at moving the Build Optimizer step from app builds to library builds, but that would be in 9 only at best.

Edit: tagged @JohannesHoppe instead of @johannesjo. sorry about that!

@arayik-yervandyan I’d like to keep this open, as there’s more to this issue than just increasing the memory limit. Some projects are exhibiting disproportionate increases in memory usage, and we also seem to have memory that is not freed on differential loading builds.

My conclusions are this point are:

  • I see increased memory usage just from using Node 12:
    • Super-productivity showed peak 1753 MB on v8, DL, Node 10.10.0 vs 2335 MB v8, DL, Node 12.3.1
  • I see about 15% extra memory usage between v7 and v8 for the same (non DL) case:
    • Super-productivity showed peak 1664 MB on v8, no DL, Node 10.10.0 vs 1456 MB v7, Node 10.10.0
    • Super-productivity showed peak 2064 MB on v8, no DL, Node 12.3.1 vs 1892 MB v7, Node 12.3.1
  • On Windows, the peak process usage is always 1. Our minifier plugin (Terser) is the only thing that uses extra processes. I think it’s not using them on windows, I don’t know why. Maybe I’ll investigate it later. But it’s not super relevant for memory usage because parallelizing something will always use more memory, even if it’s not a lot. Might be related though.

I have gathered a few more example repositories, but after doing all these benchmarks manually I think I have to automate this. I really should have automated it from the start really. So I’ll set up a repository that does this using CI.

Debugging the memory situation in CLI 8

Environment:

kamik@RED-X1C6 MINGW64 /d/sandbox/memory-debug
$ ng version

Angular CLI: 8.0.1
Node: 12.3.1
OS: win32 x64
Angular:
...

Package                      Version
------------------------------------------------------
@angular-devkit/architect    0.800.1
@angular-devkit/core         8.0.1
@angular-devkit/schematics   8.0.1
@schematics/angular          8.0.1
@schematics/update           0.800.1
rxjs                         6.4.0

kamik@RED-X1C6 MINGW64 /d/sandbox/memory-debug
$ npm -v
6.9.0

kamik@RED-X1C6 MINGW64 /d/sandbox/memory-debug
$ yarn -v
1.12.3

Measuring memory usage

We have a private benchmarking package for the CLI https://github.com/angular/angular-cli/tree/master/packages/angular_devkit/benchmark.

To install it, clone the CLI repo, then run follow these commands:

yarn
yarn build
npm pack dist/@angular-devkit/benchmark
npm install -g angular-devkit-benchmark-0.800.0-beta.18.tgz

The filename on the last command might change, depending on the version of the CLI you clone.

To use it, run benchmark -- command. The command will depend on the project you use. To benchmark a prod build, do benchmark -- ng build --prod.

You can also do benchmark --iterations 20 -- ng build --prod to get a larger sample size (20 instead of 5).

Methodology

I have gathered a bunch of projects of various sizes, and have setup a v7 and v8 folder for each.

To get the different versions I first update a project to version 8 following https://update.angular.io/. Thats the v8 variant.

Then I make the v7 variant by replacing:

  • @angular-devkit/build-angular version from ~0.800.0 to 0.13.8
  • @angular/cli version from ~8.0.1 to 7.3.9
  • target in ./tsconfig.json from es2015 to es5.

To change between node versions I used NVM. To disable Differential Loading (DL) I changed the target in ./tsconfig.json from es2015 to es5.

On each I do benchmark -- ng build --prod, then compare the results.

Benchmark matrix:

  • v8, DL, Node 10.10.0
  • v8, DL, Node 12.3.1
  • v8, no DL, Node 10.10.0
  • v8, no DL, Node 12.3.1
  • v7, Node 10.10.0
  • v7, Node 12.3.1

Results

new-project

From ng new new-project

Generated using Angular CLI 8.0.1

  • v8, DL, Node 10.10.0
[benchmark] Benchmarking process over 20 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\new-project\v8)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 46922.00 ms (39800.00, 45773.00, 45097.00, 45295.00, 46447.00, 47061.00, 49343.00, 48787.00, 58921.00, 50950.00, 45493.00, 45809.00, 47836.00, 45142.00, 45632.00, 45060.00, 45956.00, 45688.00, 45292.00, 49058.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 14.31 % (10.82, 12.30, 14.18, 13.79, 13.90, 13.75, 16.09, 15.20, 17.63, 16.20, 14.68, 14.51, 15.48, 14.43, 13.26, 14.56, 13.38, 13.03, 12.87, 16.10)
[benchmark]   Peak CPU usage: 109.84 % (92.30, 89.10, 137.60, 126.60, 103.10, 92.20, 128.20, 103.00, 142.20, 114.00, 90.60, 104.60, 107.80, 96.90, 93.70, 128.20, 123.40, 95.30, 92.20, 135.90)        
[benchmark]   Average Memory usage: 412.94 MB (410.61, 414.42, 412.02, 414.26, 414.95, 413.02, 411.16, 408.59, 408.12, 411.88, 417.27, 416.50, 410.62, 413.21, 416.36, 411.94, 411.58, 414.87, 412.33, 
415.01)
[benchmark]   Peak Memory usage: 972.38 MB (940.09, 1007.36, 939.21, 1014.48, 1012.67, 953.59, 934.79, 944.62, 860.78, 1008.56, 1019.51, 1005.97, 902.56, 1008.87, 999.36, 1005.65, 973.90, 1012.61, 974.84, 928.23)
  • v8, DL, Node 12.3.1
[benchmark] Benchmarking process over 20 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\new-project\v8)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 43714.45 ms (42475.00, 42905.00, 42806.00, 45229.00, 44435.00, 42946.00, 43730.00, 44409.00, 43320.00, 43331.00, 45213.00, 46699.00, 44467.00, 42552.00, 42833.00, 42773.00, 42621.00, 45640.00, 43103.00, 42802.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 15.08 % (13.71, 15.04, 14.23, 15.94, 15.58, 14.19, 14.84, 15.49, 15.26, 14.15, 15.76, 15.63, 16.43, 14.73, 15.51, 15.28, 15.06, 16.25, 14.92, 13.62)
[benchmark]   Peak CPU usage: 114.45 % (103.10, 153.10, 114.10, 98.50, 122.00, 98.40, 121.90, 109.40, 126.60, 109.40, 87.40, 109.30, 112.50, 114.00, 142.10, 120.30, 104.60, 150.00, 109.40, 82.80)    
[benchmark]   Average Memory usage: 443.99 MB (399.76, 469.47, 461.83, 416.02, 425.19, 433.08, 395.28, 459.70, 462.49, 419.98, 462.35, 476.16, 459.91, 463.51, 461.39, 423.32, 461.10, 424.20, 465.06, 
439.96)
[benchmark]   Peak Memory usage: 1041.65 MB (1056.51, 1166.15, 1030.32, 975.46, 1107.44, 920.04, 1028.92, 1062.10, 1067.24, 1075.11, 1040.00, 1087.46, 1005.02, 1073.69, 1040.01, 1034.72, 1059.59, 1003.81, 1081.23, 918.20)
  • v8, no DL, Node 10.10.0
[benchmark] Benchmarking process over 20 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\new-project\v8)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 39715.85 ms (41254.00, 38917.00, 38954.00, 39452.00, 39908.00, 39124.00, 41544.00, 39056.00, 39008.00, 38737.00, 39794.00, 41248.00, 41226.00, 42391.00, 38638.00, 41751.00, 36910.00, 41124.00, 39960.00, 35321.00)
[benchmark]   Average Process usage: 1.02 process(es) (1.37, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.15 process(es) (4.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 19.27 % (20.15, 18.37, 17.48, 19.86, 20.59, 19.83, 20.60, 19.03, 18.46, 18.66, 19.95, 18.84, 20.64, 21.30, 18.69, 20.99, 17.54, 19.45, 18.83, 16.05)
[benchmark]   Peak CPU usage: 118.53 % (170.40, 120.30, 89.00, 109.40, 140.70, 120.30, 132.90, 98.40, 107.80, 86.00, 93.70, 101.50, 132.80, 128.20, 112.60, 154.70, 90.60, 145.40, 134.40, 101.60)     
[benchmark]   Average Memory usage: 313.02 MB (360.81, 308.63, 312.78, 311.23, 310.24, 310.67, 312.26, 309.85, 310.46, 313.62, 310.08, 313.48, 308.39, 306.88, 309.72, 311.82, 311.05, 309.78, 310.82, 
307.85)
[benchmark]   Peak Memory usage: 800.08 MB (774.48, 774.38, 763.67, 807.96, 786.14, 815.06, 835.76, 762.01, 767.66, 861.41, 825.32, 855.64, 794.63, 767.91, 798.90, 805.75, 824.83, 823.76, 791.04, 765.35)
  • v8, no DL, Node 12.3.1
[benchmark] Benchmarking process over 20 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\new-project\v8)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 29947.20 ms (28681.00, 30854.00, 28829.00, 28346.00, 28471.00, 29073.00, 28534.00, 28585.00, 27104.00, 27578.00, 27421.00, 28399.00, 29042.00, 27301.00, 27480.00, 33175.00, 37174.00, 34747.00, 34038.00, 34112.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 15.59 % (14.65, 17.22, 15.86, 14.82, 15.21, 15.75, 15.03, 15.15, 12.67, 14.51, 13.17, 14.72, 14.46, 12.78, 13.69, 17.00, 19.96, 18.69, 17.80, 18.63)
[benchmark]   Peak CPU usage: 97.90 % (90.70, 106.20, 90.70, 90.70, 117.30, 89.00, 96.80, 103.10, 92.20, 75.00, 92.20, 93.80, 96.90, 92.20, 81.20, 134.40, 98.40, 106.20, 95.30, 115.60)
[benchmark]   Average Memory usage: 303.56 MB (294.81, 314.64, 298.23, 310.19, 303.41, 295.35, 293.39, 294.60, 310.87, 311.56, 316.21, 296.54, 305.72, 315.03, 315.93, 313.65, 311.67, 291.89, 283.43, 
294.12)
[benchmark]   Peak Memory usage: 847.21 MB (822.65, 948.96, 819.03, 852.73, 859.57, 764.29, 749.08, 811.35, 887.66, 905.15, 938.75, 799.78, 878.00, 912.18, 929.51, 819.72, 897.77, 769.11, 734.79, 844.19)
  • v7, Node 10.10.0
[benchmark] Benchmarking process over 20 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\new-project\v7)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 34065.15 ms (48589.00, 30736.00, 29772.00, 32748.00, 35670.00, 37796.00, 35251.00, 32773.00, 33065.00, 32493.00, 32957.00, 32942.00, 33630.00, 34067.00, 33241.00, 32454.00, 32997.00, 33366.00, 32509.00, 34247.00)
[benchmark]   Average Process usage: 1.06 process(es) (2.20, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.20 process(es) (5.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 15.93 % (17.74, 15.14, 14.45, 15.67, 16.94, 18.67, 17.13, 14.81, 15.50, 15.68, 15.22, 15.17, 17.00, 15.39, 15.49, 15.34, 15.15, 15.99, 15.94, 16.11)
[benchmark]   Peak CPU usage: 99.83 % (170.30, 98.50, 79.60, 117.20, 117.20, 96.90, 90.70, 82.70, 118.80, 106.20, 79.70, 82.90, 107.80, 93.70, 87.50, 84.30, 84.30, 89.00, 117.20, 92.10)
[benchmark]   Average Memory usage: 307.49 MB (455.09, 299.47, 299.79, 305.62, 300.43, 298.91, 298.40, 301.08, 300.22, 299.42, 298.72, 299.10, 299.30, 301.11, 299.16, 298.94, 300.26, 297.80, 298.66, 
298.40)
[benchmark]   Peak Memory usage: 805.72 MB (847.09, 820.75, 771.20, 851.10, 817.67, 831.61, 782.52, 815.22, 832.90, 798.27, 786.08, 753.38, 779.23, 821.07, 804.25, 824.09, 852.30, 765.94, 754.26, 805.57)
  • v7, Node 12.3.1
[benchmark] Benchmarking process over 20 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\new-project\v7)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 33382.50 ms (33115.00, 33274.00, 31484.00, 30663.00, 30814.00, 30642.00, 31120.00, 31219.00, 33208.00, 39141.00, 36040.00, 36106.00, 35269.00, 34795.00, 33526.00, 33014.00, 34918.00, 33872.00, 32882.00, 32548.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 17.92 % (17.56, 17.77, 16.47, 16.61, 15.86, 16.67, 16.90, 15.90, 18.98, 23.05, 18.36, 19.39, 20.03, 18.58, 17.52, 16.43, 19.60, 17.50, 17.58, 17.72)
[benchmark]   Peak CPU usage: 98.91 % (106.30, 99.90, 90.60, 98.40, 86.00, 87.50, 104.70, 93.70, 115.60, 128.10, 117.20, 103.10, 95.40, 85.90, 96.90, 81.30, 100.00, 85.90, 89.10, 112.60)
[benchmark]   Average Memory usage: 292.19 MB (293.59, 295.74, 293.63, 290.62, 293.48, 291.67, 292.69, 293.37, 293.36, 293.54, 293.94, 292.08, 291.17, 291.83, 292.20, 291.82, 283.89, 292.66, 291.06, 
291.52)
[benchmark]   Peak Memory usage: 846.53 MB (828.64, 886.21, 835.02, 820.26, 837.29, 883.92, 873.06, 882.54, 825.58, 889.52, 880.10, 831.73, 827.94, 868.43, 820.85, 871.18, 763.72, 800.90, 900.42, 803.35)

Super-productivity

From https://github.com/aviabird/angularspree

Provided by @johannesjo in https://github.com/angular/angular-cli/issues/13734#issuecomment-497393904

  • v8, DL, Node 10.10.0
[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\super-productivity\v8)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 269523.80 ms (257464.00, 255811.00, 285836.00, 279557.00, 268951.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 13.37 % (12.17, 12.49, 13.95, 15.03, 13.22)
[benchmark]   Peak CPU usage: 193.76 % (175.00, 179.80, 207.80, 223.40, 182.80)
[benchmark]   Average Memory usage: 1139.50 MB (1125.28, 1128.75, 1165.98, 1146.50, 1131.01)
[benchmark]   Peak Memory usage: 1753.78 MB (1712.68, 1726.49, 1809.78, 1796.38, 1723.58)
  • v8, DL, Node 12.3.1
[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\super-productivity\v8)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 302133.40 ms (379732.00, 271669.00, 295372.00, 289442.00, 274452.00)
[benchmark]   Average Process usage: 1.19 process(es) (1.97, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.80 process(es) (5.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 12.91 % (13.05, 12.47, 13.46, 13.09, 12.45)
[benchmark]   Peak CPU usage: 199.36 % (217.20, 198.50, 203.00, 178.10, 200.00)
[benchmark]   Average Memory usage: 1506.77 MB (1742.04, 1453.37, 1455.98, 1438.85, 1443.59)
[benchmark]   Peak Memory usage: 2335.48 MB (3000.09, 2137.67, 2118.84, 2190.41, 2230.37)
  • v8, no DL, Node 10.10.0
$ benchmark -- ng build --prod
[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\super-productivity\v8)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 137125.80 ms (134088.00, 140877.00, 137668.00, 139447.00, 133549.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 10.91 % (10.27, 11.17, 11.13, 11.15, 10.82)
[benchmark]   Peak CPU usage: 155.30 % (143.80, 154.70, 164.00, 153.10, 160.90)
[benchmark]   Average Memory usage: 944.34 MB (948.63, 948.80, 960.01, 950.97, 913.27)
[benchmark]   Peak Memory usage: 1664.48 MB (1723.53, 1670.94, 1670.71, 1684.96, 1572.24)
  • v8, no DL, Node 12.3.1
[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\super-productivity\v8)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 138378.80 ms (136618.00, 140146.00, 136281.00, 139331.00, 139518.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 10.63 % (10.72, 10.78, 10.38, 10.12, 11.17)
[benchmark]   Peak CPU usage: 154.38 % (162.50, 148.40, 167.20, 145.30, 148.50)
[benchmark]   Average Memory usage: 1173.09 MB (1182.91, 1179.12, 1162.80, 1162.17, 1178.46)
[benchmark]   Peak Memory usage: 2064.48 MB (2104.92, 2113.08, 2071.26, 2007.69, 2025.45)
  • v7, Node 10.10.0
$ benchmark -- ng build --prod
[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\super-productivity\v7)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 126331.00 ms (115769.00, 114542.00, 127353.00, 144113.00, 129878.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 12.75 % (11.16, 10.43, 12.28, 15.91, 13.96)
[benchmark]   Peak CPU usage: 160.92 % (175.00, 139.10, 140.60, 185.90, 164.00)
[benchmark]   Average Memory usage: 850.62 MB (851.26, 851.57, 859.25, 837.64, 853.40)
[benchmark]   Peak Memory usage: 1456.63 MB (1458.19, 1463.49, 1470.42, 1447.42, 1443.61)
  • v7, Node 12.3.1
[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build --prod (at D:\sandbox\memory-debug\super-productivity\v7)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 128415.80 ms (121753.00, 121720.00, 124237.00, 143535.00, 130834.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 12.02 % (11.37, 11.24, 12.02, 13.66, 11.79)
[benchmark]   Peak CPU usage: 155.96 % (160.90, 175.10, 176.60, 186.00, 81.20)
[benchmark]   Average Memory usage: 1126.36 MB (1138.10, 1105.06, 1135.59, 1126.35, 1126.68)
[benchmark]   Peak Memory usage: 1892.42 MB (1864.26, 1773.99, 2055.74, 1777.96, 1990.18)

Not sure if this will help but here is a graph of our memory consumption.

The first spike is a default build with differential loading. Second spike is differential loading using node-sass.

The final spike is a build with differential loading turned off.

Both builds with differential loading failed after they reached the memory limit of the server. The last build was successful.

memoryconsumption

We have about 20 different projects defined in our angular.json. It would normally fail when building the second or third project.

I have run into this issue on a project that does not use sass. I did some environment testing because I originally thought it was an issue relating to Azure Dev Ops.

Using Azure Dev Ops Pipelines:

  • Hosted Linux agent with Node 10: Build succeeds
  • Hosted Windows vs2017 Agent with Node 10: Build fails
  • Hosted Windows vs2017 Agent with Node 6: Build fails
  • Hosted Windows vs2019 Agent with Node 10: Build fails
  • Hosted Windows vs2019 Agent with Node 6: Build fails
  • Hosted Windows Container Agent with Node 10: Build Failed 5/10 attempts
  • Install Build Agent on my dev machine(win10) with Node 10: Build Succeeds
  • Install Build Agent on my dev machine(win10) with Node 6: Build Succeeds
  • Dev machine(win10) using cmd with Node 10: Build Succeeds
  • Dev machine(win10) using cmd with Node 6: Build Succeeds

@Flyrell @gvsakhil Avoid commenting on closed issues that are referring to old versions. There is an opened issue #16860 for NG9, which would be more appropriate.

Note that there is even now an issue #18087 for Angular 10, which got closed so you can expect some improvements with next release.

Heya all, this issue is pretty old by this point and we’re mostly focused on resource usage issues on newer CLI version. This issue also contains several different problems that seem to manifest in similar ways, which makes it hard to zero-in on. If you’re still seeing problems like this, please open a new issue with reproduction details.

Every release I recall a presentation at NGConfg by one of the angular members stating something like:

“We have over 400 applications at google and we dogfood angular before we release it into the wild. If any of the applications have to change in order to adapt the update, we don’t release it”

I think every single release required some type of update to the code.

As far as I remember, they said that they test the applications to make sure nothing breaks after necessary code upgrades. I believe, upgrading the code to take in the enhancements is anyways inevitable.

But this current issue has nothing to do with that. It’s more about application size. If the application size is growing, you will have to increase the heap size for the node process.

Same problem here. Angular 7. Any Node 12.X version.

Works with Node 10.

A Note For Angular/Windows/Docker Users

If you are trying to run ‘npm run build’ or something of the sort inside your Docker file and you are getting this error, as mention above, you want to use the following run command instead, increase Node.js’ access to memory.

node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng build --prod

In addition, however, Docker is running a Linux VM in the background that will also have memory constraints that need to be addressed.

Navigate to ‘C:\Users<profile name>\AppData\Roaming\Docker’ where profile name is the named of your logged in profile. There you will find a ‘settings.json’ file. Open it and adjust the ‘memoryMiB’ value to also be 8192. You will need to restart for the changes to take effect.

WARNING

Increasing the VM memory can have a drastic impact on your machine performance, so ensure that you have enough physical memory to do this before trying. If this solution works for you, you should then try to roll back the values as much as you can, by doing a process something like, halving the value initially, testing, halving it again if it works, or adding half of the new value back if it doesn’t etc. This will help you minimize the impact on your machine.

You will also want to disable Docker as a start up program for Windows 10. Simply search “startup apps” in the task bar and select the corresponding option that appears. This way, you wont have a VM you are not using bogging down your system.

Thanks a lot @filipesilva for the exhaustive benchmark. Installing node-sass didn’t pull us out but upgrading angular to v8.2.6 did the trick.

Still, isn’t it weird that most of people reporting the JavaScript heap out of memory error get stuck at 92% precisely ? I would blame the Terser optimisation process for the sudden memory peak.

Please keep the discussion on topic and review the code of conduct for behavior.

https://github.com/angular/code-of-conduct/blob/master/CODE_OF_CONDUCT.md

I’ve been trying to figure out if there’s really a memory leak in Differential Loading (DL) builds. Below are my notes.

TLDR is that there doesn’t seem to be any memory leak in DL, but when --max_old_space_size is used V8 will keep memory around. A build can finish successfully with --max_old_space_size=300 without erroring but will use up to 2500mb of memory if given --max_old_space_size=10000.


Trying to figure out if DL really stacks memory endlessly. Modified a DL build to double the entries in webpack-browser-config.js. Normal (2) builds uses 480 average/960 peak, 4 builds is 810/1700, 8 builds is 1400/2500. So each build leaves increases average by 200ish.

I don’t quite get why peak also increases so much though. Peak should be average+something… right? Actually that’s wrong. Because we already know the usage is going up over time. So the average during the first build is much smaller than the average during the last build. Kinda like an integral. So the number I should care about really is peak, because it represents things better for the multi-build case.

Also noticed that process.memoryUsage().heapUsed gave me numbers were much lower than what I saw on task manager. I found that odd. I also didn’t disable the terser cache. I should do that to see if there’s also a scaling cost there, after figuring out this initial scaling cost.

Tried doing global.gc every 30 seconds to figure out if memory was truly retained or if node was just very comfortable with the 10gb limit. Did it over 5 iterations. Without GC I got 1200/2150, with GC I got 1200/2050. So only a slightly smaller peak. This shows the memory can’t be freed.

Took heap profiles and heap snapshots every 30s. Also tried the memory leak profile on chrome devtools. The heap profiles show almost no change between them. Very odd. Maybe I need to restart the profiler between samples? The snapshots fail to load too, with an error while building retainers. Will re-make them.

Tried profiling again with a lower sampling interval and didn’t see anything different. All snapshots but the first still failed with the same message. Also tried just taking one snapshot at the end and got only a small snapshot (200mb).

Tried taking a snapshot manually in Chrome Devtools during the 8th (and last) build. Devtools said the process was using ~2000mb memory just before I took the snapshot, and then it showed only ~260mb. The snapshot was only 254mb.

So I thought it might be a GC thing. Tried limiting max_old_space_size to 1gb and it still built. Also built with 600mb and 400mb, but failed with 300 and 200. A single build run failed with 200 but succeeded with 300. A default DL build failed with 300 too.

So as far as I can tell there isn’t any real amount of memory being kept because of DL. Maybe there’s to 100 extra mb. A new app will use roughly the same memory (~300mb on the main process, plus up to 450mb on spawned Terser processes) for producing 1 build or 8 builds in the same command. But if I set the memory limit to a a very large number (10gb) then it will keep more memory around (up to 2.5gb) until it needs to be freed.

@johannesjo @jsgoupil I think the issue with 8 has to do with a memory leak on prod builds with differential loading. Can you try disabling it? See https://angular.io/guide/deployment#opting-out-of-differential-loading

cc @alan-agius4

Yes, I am also facing the same issue

Guys here’s described the “workaround” for this problem: https://github.com/angular/angular-cli/issues/5618

Yes, I was using the dev server for a little more today and the issue still exists.

@kumaresan-subramani do you know about error Npm failed with return code: 3221226505

I am getting this error while build on vsts

We’re using Bitbucket pipelines and we can’t increase the memory limit to 8GB. Also our application takes about 7-10min to build and we need the sourceMaps for our Sentry error reports.

UPDATE: running ng update @angular/cli @angular/core solved the problem.

Running the ng update worked for me as well. One of the minor point versions must have had a memory leak of some sort. Moving from 8.2.8 to 8.2.11 fixed the issue.

I had to upgrade to node 12 AND increase our codebuild instance memory from 3 to 7GB. Just to build an angular app.

@maxisam

I definitely appreciate everyone’s hard work. I initially chose Angular because it seemed to be the best organized and most holistically thought-out project.

The issue is not whether people are working hard on solving this problem. The issue is that it should have been solved before it was released and declared ready for public use.

As for the rxjs thing, there shouldn’t be any breaking changes between the versions. The problem is that the comments read “// These versions should be kept up to date with latest Angular peer dependencies.” and yet someone is just typing whatever they want into the field instead of running a script to extract the exact string. And then no one tested if it was working as intended.

I imagine the Angular team is stretched too thin.

I had same issue with Angular 8. Windows 10 I guess different people have these issues because of different reasons.

My solution was :

node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng build --prod

Now again there are two things in this, either it worked because of max_old_space_size=8192 or may be because ive different local and global cli angular version. and using above command used the local/same version of ng

Ran into this very same issue when calling “ng serve” w/ some custom configuration:

npm run env -s && ng serve --aot --configuration alpha

If you are just looking to get rolling, turning “buildOptimizer” and “optimization” flags in my configuration to false did the trick and I was able to build within seconds.

screen

We were experiecing the same issue. It turned out, we imported the base scss files in most of our components. That was not really necassary. Now without setting the max-old-space-size Node option, we were able to build our apps in production mode without the error, after removing the core file imports in our components, where it was not necessary.

We also switched from dart-sass to node-sass with npm install -DE node-sass and that worked as well, without changing anything.

Below my environment. Maybe this helps 😃


     _                      _                 ____ _     ___
    / \   _ __   __ _ _   _| | __ _ _ __     / ___| |   |_ _|
   / △ \ | '_ \ / _` | | | | |/ _` | '__|   | |   | |    | |
  / ___ \| | | | (_| | |_| | | (_| | |      | |___| |___ | |
 /_/   \_\_| |_|\__, |\__,_|_|\__,_|_|       \____|_____|___|
                |___/


Angular CLI: 8.1.1
Node: 10.16.0
OS: darwin x64
Angular: 8.1.1
... animations, cli, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... platform-server, router

Package                                    Version
--------------------------------------------------------------------
@angular-devkit/architect                  0.800.1
@angular-devkit/build-angular              0.801.1
@angular-devkit/build-optimizer            0.801.1
@angular-devkit/build-webpack              0.801.1
@angular-devkit/core                       8.0.6
@angular-devkit/schematics                 8.0.1
@angular/cdk                               8.0.2
@angular/flex-layout                       8.0.0-beta.26
@angular/material                          8.0.2
@ngtools/webpack                           8.1.1
@nguniversal/express-engine                8.1.1
@nguniversal/module-map-ngfactory-loader   8.1.1
@schematics/angular                        8.0.1
@schematics/update                         0.801.1
rxjs                                       6.5.2
typescript                                 3.4.5
webpack                                    4.35.2

@jsgoupil got in touch and let me have a look at his project. I turned off the terser cache (described in https://github.com/angular/angular-cli/issues/13734#issuecomment-508815407) and benchmarked the “ng build 3cconnect --prod” command:

[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng build 3cconnect --prod (at D:\sandbox\memory-debug\jsgoupil)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 258795.40 ms (280888.00, 250100.00, 241312.00, 256762.00, 264915.00)
[benchmark]   Average Process usage: 3.40 process(es) (3.41, 3.39, 3.42, 3.41, 3.39)
[benchmark]   Peak Process usage: 8.00 process(es) (8.00, 8.00, 8.00, 8.00, 8.00)
[benchmark]   Average CPU usage: 11.59 % (13.45, 10.73, 10.52, 11.53, 11.75)
[benchmark]   Peak CPU usage: 236.56 % (237.60, 325.10, 168.60, 207.80, 243.70)
[benchmark]   Average Memory usage: 1523.76 MB (1547.40, 1471.40, 1544.40, 1562.83, 1492.76)
[benchmark]   Peak Memory usage: 2898.84 MB (2940.99, 2707.18, 2969.51, 3049.01, 2827.53)

This build has AOT and Build Optimizer turned off, and is using Differential Loading. The average memory usage is right at the maximum as well, which means the garbage collector is doing a lot of work to free up memory as the build progresses.

Tried turning on AOT and Build Optimizer and got OOM on the second DL build, during TerserPlugin. Used the node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng build 3cconnect --prod --aot --build-optimizer command to build and it succeeded.

Benchmark for this build shows the following numbers:

$ benchmark -- node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng build 3cconnect --prod --aot --build-optimizer
[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng build 3cconnect --prod --aot --build-optimizer (at D:\sandbox\memory-debug\jsgoupil)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 571885.00 ms (567385.00, 558561.00, 574671.00, 572119.00, 586689.00)
[benchmark]   Average Process usage: 2.93 process(es) (2.84, 2.90, 3.02, 2.89, 3.02)
[benchmark]   Peak Process usage: 8.40 process(es) (8.00, 8.00, 9.00, 8.00, 9.00)
[benchmark]   Average CPU usage: 11.00 % (10.82, 11.15, 11.26, 10.87, 10.89)
[benchmark]   Peak CPU usage: 333.82 % (381.40, 243.80, 272.00, 342.10, 429.80)
[benchmark]   Average Memory usage: 2997.71 MB (3094.71, 2923.23, 2899.82, 2987.55, 3083.24)
[benchmark]   Peak Memory usage: 5613.08 MB (5643.63, 5567.17, 5615.44, 5592.79, 5646.38)

I also tried without Build Optimizer turned on:

[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng build 3cconnect --prod --aot (at D:\sandbox\memory-debug\jsgoupil)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 395413.60 ms (396473.00, 384629.00, 401091.00, 396822.00, 398053.00)
[benchmark]   Average Process usage: 2.56 process(es) (2.61, 2.53, 2.53, 2.52, 2.62)
[benchmark]   Peak Process usage: 8.00 process(es) (8.00, 8.00, 8.00, 8.00, 8.00)
[benchmark]   Average CPU usage: 10.42 % (10.50, 9.93, 10.76, 10.58, 10.32)
[benchmark]   Peak CPU usage: 297.14 % (307.80, 251.40, 293.70, 376.60, 256.20)
[benchmark]   Average Memory usage: 2483.32 MB (2481.63, 2471.55, 2518.96, 2451.15, 2493.35)
[benchmark]   Peak Memory usage: 5238.84 MB (5221.19, 5212.36, 5368.60, 5184.88, 5207.16)

I wasn’t expecting Build Optimizer to impact either average or peak memory usage that much really. 500mb is a fair bit. We were already looking at a way to reduce the number of processed files in https://github.com/angular/angular-cli/pull/14998, which would help reduce the impact.

These numbers look much higher than the initial ones, but bear in mind that we increased the memory from the 1.4gb default to around 8gb. This means that there isn’t as much pressure for the the garbage collector to free up memory, so things stay in memory for longer. For comparison, here is the initial build (without AOT or Build Optimizer) with an increased memory limit:

$ benchmark -- node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng build 3cconnect --prod
[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng build 3cconnect --prod (at D:\sandbox\memory-debug\jsgoupil) 
[benchmark] Process Stats
[benchmark]   Elapsed Time: 253524.00 ms (256206.00, 245034.00, 245636.00, 243953.00, 276791.00)
[benchmark]   Average Process usage: 3.44 process(es) (3.41, 3.45, 3.45, 3.46, 3.42)
[benchmark]   Peak Process usage: 8.00 process(es) (8.00, 8.00, 8.00, 8.00, 8.00)
[benchmark]   Average CPU usage: 11.65 % (12.18, 10.84, 11.67, 11.01, 12.55)
[benchmark]   Peak CPU usage: 200.90 % (200.00, 168.70, 231.20, 217.10, 187.50)
[benchmark]   Average Memory usage: 1972.97 MB (2010.47, 1927.40, 2014.74, 1939.05, 1973.17)
[benchmark]   Peak Memory usage: 3871.67 MB (3942.59, 3815.74, 3944.29, 3851.58, 3804.13)

So when using the same high memory limit, adding turning on AOT and Build Optimizer increased the average memory usage by 1GB and peak memory usage by 1.8GB.

Using just AOT increased build time by 70% (250->400 s), average memory usage by 25% (2000->2500), and peak memory usage by 36% (3800->5200 mb). Adding BO increased build time by 42% (400->570 s), average memory usage by 20% (2500->3000), and peak memory usage by 7% (5200->5600 mb).

I also tried to turn off Differential Loading by setting "target": "es5", in ./tsconfig.json while keeping AOT and Build Optimizer:

[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng build 3cconnect --prod --aot --build-optimizer (at D:\sandbox\memory-debug\jsgoupil)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 332036.80 ms (291110.00, 303767.00, 297923.00, 395777.00, 371607.00)
[benchmark]   Average Process usage: 2.87 process(es) (2.89, 2.86, 2.87, 2.86, 2.85)
[benchmark]   Peak Process usage: 8.00 process(es) (8.00, 8.00, 8.00, 8.00, 8.00)
[benchmark]   Average CPU usage: 12.42 % (10.63, 11.52, 11.41, 14.30, 14.21)
[benchmark]   Peak CPU usage: 283.98 % (250.00, 237.40, 414.00, 207.70, 310.80)
[benchmark]   Average Memory usage: 2496.03 MB (2555.55, 2533.63, 2524.99, 2409.38, 2456.62)
[benchmark]   Peak Memory usage: 5084.00 MB (5386.41, 5283.17, 5257.22, 4586.35, 4906.83)

Without DL, memory used goes does by ~500mb, similar to https://github.com/angular/angular-cli/issues/13734#issuecomment-508815407. We’re looking at improving that.

I tried to figure out how big the project was by listing files in src/ by extension:

$ find -type f | sed -e 's/.*\.//' | sort | uniq -c
      3 eot
      7 gif  
    478 html 
      1 ico  
      2 jpg  
      4 json 
     51 png  
      1 ps1  
    273 scss 
    470 svg  
   1302 ts   
      1 txt  
      3 woff 
      9 woff2
      1 xml  

This does not include third party libs so it doesn’t give us much insight as to the total compilation size. But I’m not surprised that this project needs the memory limit increased, there’s a lot of TS files in here. I had a look at

@jsgoupil in https://github.com/angular/angular-cli/issues/13734#issuecomment-497365534 you mentioned that an Azure machine with 7gb couldn’t build the app, which I find weird because 5.6gb was the maximum I saw on my Windows machine.

Maybe the CI machine is already using 1.4gb? That would make sense. You can probably check how much memory the machine has before the build. But you also said the machine has only 7GB and you are using --max_old_space_size=8192. I’m not really sure how Node behaves when you say it should use more memory than is actually available. So I wonder if that’s a problem: you give it a bigger limit than available, so it thinks it can allocate all this memory, but when it’s needed the system doesn’t have it so you get an error. You do say in https://github.com/angular/angular-cli/issues/13734#issuecomment-497426391 that the error is different with and without the flag so that does point to a different issue. Maybe check how much memory is free and set a limit below that, like 5000 or 6000.

Regardless of the Azure machine you mentioned, it is impressive (in a negative way) that AOT compilation increases memory usage by 1GB average / 1.8GB peak. This is something that we are looking at in the general case. I think in your project it really is just a function of the amount of TS source files that need AOT compilation, I tried taking a profile of the memory usage and it mostly looked like the others I’ve looked at before from other projects.

I also noticed that you have some weird imports to ../../../node_modules/@angular/core instead of just @angular/core, and similar to RxJs. I think this makes no difference because resolution will still use the package.json there. But I thought it was odd.

@alex88 you can find the benchmark tool I used in https://github.com/filipesilva/angular-cli-perf-benchmark#benchmark-package. If you want to provide me a repro via a private github repository, I’d love to have a look too.

@zijianhuang a big thank you for giving me a repro to look at. I had a look at it today using the benchmark package described in https://github.com/filipesilva/angular-cli-perf-benchmark.

One thing to be aware of is that the memory numbers collected by that tool include spawned processes. In the CLI we use a minifier package (terser-webpack-plugin) that spawns multiple processes, so it’s useful to capture the memory the child processes use as well.

But that package also caches the minification results. So the first run does more work than subsequent ones. So when I benchmarked your packages I manually edited ./node_modules/@angular-devkit/build-angular/src/angular-cli-files/models/webpack-configs/common.js and changed cache: true to cache: false to disable that cache and get consistent numbers.

I also added these two npm scripts to help me benchmark things:

		"benchmark": "benchmark -- node --max_old_space_size=8000 node_modules/@angular/cli/bin/ng build --prod",
		"ng-high-memory": "node --max_old_space_size=8000 node_modules/@angular/cli/bin/ng build --prod",

I set the same memory limit for all projects because I know Node will behave differently near the limit. So this way we should get more accurate numbers.

I had to remove the Bounds import from avatar.component.ts because that gave me a build error in NG7. It was just used as a type so it was easy to remove. I replaced "target": "es2015" with "target": "es5" in ./tsconfig.json. This disables Differential Loading as described here. I know it’s an important part of CLI8 applications but we are already looking at a different implementation of it that shouldn’t require as much resources, so I didn’t want to also benchmark it. This way I could compare mostly equal applications.

I noticed that your package.json between the two versions changes a lot of packages that aren’t just the Angular packages. That might have an impact on the benchmarks. After trying to reproduce your initial numbers I’ll also try to alter only Angular packages to see if I get different results.

Dev environment

In your reproduction your said:

Dev Environment:
* Intel Nuc i7, 16 GB memory, SSD
* Windows 10

Benchmarking tool: Process Explorer

I’m using my work machine (Win10, i7, 16GB RAM, SSD) which seems similar to your env. To benchmark I used the tool mentioned in https://github.com/filipesilva/angular-cli-perf-benchmark#benchmark-package. I’m also using Node 10.16.0 and npm 6.9.0.

Your test cases

In your reproduction your said:

Test case 1:
* Node 10.15.3
* npm 6.4.0
* NG 7 and NG CLI 7

Node memory usage peak: 1.3 GB.

I read from the other post that the default limit of memory usage of Node is 1.5 GB. The "NG CLI" command apparently is using such default.


Test case 2:
* Node 10.15.3 / 10.16.0
* npm 6.4.0 / 6.10.0
* NG 8.1 and NG CLI 8.1

Memory usage peak in the DEBUG build: 3.6 GB / 3.5 GB
Memory usage peak in the Production build: 3.3 GB / 3.4 GB


Test case 3:
* Locally installed Node 12.6 through "npm i"
* npm 6.9.0
* NG 8.1 and NG CLI 8.1

Memory usage peak: 5.5 GB

I’m not really going to try Test case 3 because we already know that Node 12 is using more memory in general (https://github.com/nodejs/node/issues/28205). I don’t think there’s much we can do about that.

Your Test case 1 is the CLI7/NG7/NG7deps scenario. I also say NG7deps because those dependencies are different from your NG8 project, which I’ll say has NG8deps. Later on that might make a difference. Ideally we’d only change CLIx/NGx to get numbers.

So for CLI7/NG7/NG7deps (your Test case 1) I got:

$ ng version

Angular CLI: 7.3.9
Node: 10.16.0
OS: win32 x64
Angular: 7.2.15
... animations, common, compiler, compiler-cli, core, forms
... platform-browser, platform-browser-dynamic, platform-server
... router, service-worker

Package                            Version
------------------------------------------------------------
@angular-devkit/architect          0.13.9
@angular-devkit/build-angular      0.13.9
@angular-devkit/build-optimizer    0.13.9
@angular-devkit/build-webpack      0.13.9
@angular-devkit/core               7.3.3
@angular-devkit/schematics         7.3.3
@angular/cdk                       7.3.3
@angular/cli                       7.3.9
@angular/flex-layout               7.0.0-beta.24
@angular/material                  7.3.3
@angular/material-moment-adapter   7.3.3
@angular/pwa                       0.13.3
@ngtools/webpack                   7.3.9
@schematics/angular                7.3.3
@schematics/update                 0.13.9
rxjs                               6.5.2
typescript                         3.2.2
webpack                            4.29.0
[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   node --max_old_space_size=8000 node_modules/@angular/cli/bin/ng build --prod (at D:\sandbox\memory-debug\testngcli8\NG7Source)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 288429.80 ms (297613.00, 292811.00, 282489.00, 284134.00, 285102.00)
[benchmark]   Average Process usage: 2.65 process(es) (2.60, 2.69, 2.66, 2.64, 2.64)
[benchmark]   Peak Process usage: 8.00 process(es) (8.00, 8.00, 8.00, 8.00, 8.00)
[benchmark]   Average CPU usage: 15.64 % (14.82, 16.84, 15.28, 15.27, 16.00)
[benchmark]   Peak CPU usage: 252.80 % (251.60, 300.00, 224.90, 261.00, 226.50)
[benchmark]   Average Memory usage: 1932.92 MB (1885.25, 1947.53, 1937.02, 1953.46, 1941.31)
[benchmark]   Peak Memory usage: 4024.00 MB (4115.83, 4092.19, 3943.85, 4018.98, 3949.16)

And for CLI8/NG8/NG8deps (your Test case 2) I got:

$ ng version

Angular CLI: 8.1.0
Node: 10.16.0
OS: win32 x64
Angular: 8.1.0
... animations, cli, common, compiler, compiler-cli, core, forms
... platform-browser, platform-browser-dynamic, platform-server
... router, service-worker

Package                            Version
------------------------------------------------------------
@angular-devkit/architect          0.801.0
@angular-devkit/build-angular      0.801.0
@angular-devkit/build-optimizer    0.801.0
@angular-devkit/build-webpack      0.801.0
@angular-devkit/core               8.1.0
@angular-devkit/schematics         8.1.0
@angular/cdk                       8.0.2
@angular/flex-layout               8.0.0-beta.26
@angular/material                  8.0.2
@angular/material-moment-adapter   8.0.2
@angular/pwa                       0.801.0
@ngtools/webpack                   8.1.0
@schematics/angular                8.1.0
@schematics/update                 0.801.0
rxjs                               6.5.2
typescript                         3.4.5
webpack                            4.35.2
[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   node --max_old_space_size=8000 node_modules/@angular/cli/bin/ng build --prod (at D:\sandbox\memory-debug\testngcli8\NG8Source)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 253091.40 ms (281761.00, 251054.00, 242851.00, 244676.00, 245115.00)
[benchmark]   Average Process usage: 2.72 process(es) (3.34, 2.52, 2.61, 2.58, 2.54)
[benchmark]   Peak Process usage: 8.80 process(es) (12.00, 8.00, 8.00, 8.00, 8.00)
[benchmark]   Average CPU usage: 12.94 % (15.28, 12.50, 12.20, 12.55, 12.18)
[benchmark]   Peak CPU usage: 210.62 % (223.50, 287.50, 173.50, 189.00, 179.60)
[benchmark]   Average Memory usage: 1997.28 MB (1980.34, 1920.54, 2051.03, 2025.27, 2009.23)
[benchmark]   Peak Memory usage: 4313.95 MB (4445.09, 4162.86, 4258.15, 4380.61, 4323.05)

So for these two cases I actually got very similar numbers.

Variants

Something I know I did different from you was to ensure the minifier output was never cached. That has a big impact on resource usage. On CI machines you won’t get this cache unless you cache node_modules (that’s where it is stored) so I always turn it off manually for benchmarks. If it turn it back on this is what I see in CLI8/NG8/NG8deps:

[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   node --max_old_space_size=8000 node_modules/@angular/cli/bin/ng build --prod (at D:\sandbox\memory-debug\testngcli8\NG8Source)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 184542.00 ms (240928.00, 167820.00, 169525.00, 176387.00, 168050.00)
[benchmark]   Average Process usage: 1.31 process(es) (2.57, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 2.40 process(es) (8.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 11.67 % (11.08, 11.67, 11.74, 12.13, 11.73)
[benchmark]   Peak CPU usage: 173.76 % (268.80, 187.50, 167.20, 121.90, 123.40)
[benchmark]   Average Memory usage: 1569.17 MB (1997.59, 1421.48, 1607.34, 1424.31, 1395.14)
[benchmark]   Peak Memory usage: 2788.43 MB (4277.19, 2291.22, 2705.78, 2327.46, 2340.52)

Here you can see that the first build (first place inside the parenthesis) uses up more resources than the others. Notably build time and peak memory goes down by about 1/3, and average memory down by around 1/4. These numbers are very similar to the ones you reported for Test case 1.

The other thing I did different was disabling Differential Loading (DL) in the CLI8/NG8/NG8deps case, so I also took a benchmark with it turned back on in CLI8/NG8/NG8deps:

[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   node --max_old_space_size=8000 node_modules/@angular/cli/bin/ng build --prod (at D:\sandbox\memory-debug\testngcli8\NG8Source)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 448765.60 ms (450442.00, 441786.00, 453261.00, 442406.00, 455933.00)
[benchmark]   Average Process usage: 2.62 process(es) (2.63, 2.62, 2.65, 2.61, 2.61)
[benchmark]   Peak Process usage: 8.00 process(es) (8.00, 8.00, 8.00, 8.00, 8.00)
[benchmark]   Average CPU usage: 11.07 % (11.65, 11.65, 11.10, 10.18, 10.78)
[benchmark]   Peak CPU usage: 280.28 % (307.70, 356.30, 237.40, 256.20, 243.80)
[benchmark]   Average Memory usage: 2484.00 MB (2467.08, 2548.74, 2480.70, 2471.22, 2452.25)
[benchmark]   Peak Memory usage: 5050.35 MB (5043.85, 5075.19, 5051.67, 5053.34, 5027.68)

So here we see that the build time roughly doubled with DL, which is expected because we do two builds one after the other. We’re planning on doing a single builds with an extra step at the end instead soon.

We also see that average used memory has increased by 500mb, and peak memory by 700mb. In https://github.com/angular/angular-cli/issues/13734#issuecomment-500849058 I said:

Using differential loading seems to add a ~200mb to memory usage. This value does not change much, which seems to indicate that something that doesn’t depend much on project size is being retained in memory between builds.

The DL results from your project contradict that. In your project it increased by more than 200mb. So there is a scaling aspect to the DL cost that we see in your project. I don’t know what it is, but it’s worth investigating. It points to memory being retained between builds, which might also imply that memory is being retained between build steps. If it’s memory used in an earlier step of the build, maybe we can free it and reduce average memory usage in latter build steps.

@alex88 also mentioned that es2015 builds took more resources so I tried leaving "target": "es2015" in tsconfig.json and disabling DL via the browserslist approach, which in your case is in package.json. I removed the not in "not dead", "not IE 9-11" but that still made DL builds (you can see that by the two builds, one after the other). I had to remove all entries in browserslist before I stopped getting DL builds. We should check why these instructions aren’t correct (cc @alan-agius4, I think you worked on this). After I managed to get only ES2015 builds, I took another benchmark:

[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   node --max_old_space_size=8000 node_modules/@angular/cli/bin/ng build --prod (at D:\sandbox\memory-debug\testngcli8\NG8Source)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 246739.80 ms (222235.00, 246069.00, 244327.00, 252636.00, 268432.00)
[benchmark]   Average Process usage: 2.55 process(es) (2.55, 2.59, 2.58, 2.56, 2.46)
[benchmark]   Peak Process usage: 8.00 process(es) (8.00, 8.00, 8.00, 8.00, 8.00)
[benchmark]   Average CPU usage: 12.83 % (11.53, 12.41, 12.92, 13.34, 13.97)
[benchmark]   Peak CPU usage: 229.42 % (287.40, 268.90, 197.00, 212.50, 181.30)
[benchmark]   Average Memory usage: 1947.94 MB (1941.51, 2076.98, 1938.72, 1926.15, 1856.36)
[benchmark]   Peak Memory usage: 4129.88 MB (4141.96, 4347.47, 4064.12, 4071.99, 4023.87)

This doesn’t seem to be very different from the ES5 benchmark. @alex88 maybe when you used es2015 you were actually getting the DL builds? You should also not be using Node 12 for now because of https://github.com/nodejs/node/issues/28205.

@zijianhuang so to wrap up, I think the reason your project uses more resources with NG8/CLI8 than with NG7/CLI7 is that DL was turned on and that some builds were without cache. I really can’t be sure of the second point because I don’t know how you’re caching things locally and in CI, but the numbers I got here seem to point to that.

For us the follow up from investigating your project’s transition from Angular version 7 to 8 is that we really need to figure out what memory is retained between DL builds because that points to a leak, and that we need to have a more performant DL setup (@clydin has some ideas on that).

same issue on ubuntu server 18.04, but this solved my problem: node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng build --prod

Just dropping this in here but if anyone is using AWS codebuild make sure to update your build environment to be configured with more than the minimum 3GB memory, I followed the steps above but had to re-configure my environment with more memory.

If you are on Node 12 and hitting the memory limit, you should try Node 10 instead for now.

We’ll look into the retained memory during Differential Loading builds and the outliers like awesome-angular-workshop.

I think he meant something in the lines of node --max_old_space_size=4096 ./node_modules/@angular/cli/bin/ng build --prod, so the argument would be passed to node environment. But there are other issues linked to this one, like the huge time and memory spent while compiling.

Heya, I’ve put up results and methodology in https://github.com/filipesilva/angular-cli-perf-benchmark. Will be adding more projects there next week.

The only extra detail that I have is that in the awesome-angular-workshop project build times almost doubled from CLI 7 to 8. But they didn’t on the other projects I tested.

If someone wants to follow the methodology there using the scripts, it should be easy enough. If you find something interesting let me know. If you have a large project that you feel could yield interesting benchmarks, let me know (or add a PR) and I’ll add it there.

Upgrading from node 10 to node 12 resolved my compiling memory issue when using Angular 8.

However, when I try to run the compiled code from ng build --prod, it just displays a blank screen 😕 Works fine with ng serve though

I think the increased memory requirement might be related to sass processing then. We moved from node-sass to sass in 8.

Can you follow the instructions in @angular-devkit/build-angular: use sass instead of node-sass (ce15899) to force usage of node-sass instead?

@vindemi you are going to need to edit your Build Pipeline. Remove the npm build task and replace it with a command line task with the following script node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng build --prod make sure to set your working directory to the folder with your package.json.

This issue is the same as https://github.com/angular/angular-cli/issues/5618#issuecomment-450151214, as far as I can tell.

I don’t see a lot of things I can follow up and investigate though. @themanojshukla mentioned Windows was taking a lot longer. @j3gb3rt is asking about sass, and I have seen reports in the past about sass making things slower.

Does anyone have a reproduction that we can look at and try to debug?