angular-cli: Angular 7/8 FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
š Bug report
Command (mark with an x)
- [ ] new
- [X] build
- [ ] serve
- [ ] test
- [ ] e2e
- [ ] generate
- [ ] add
- [ ] update
- [ ] lint
- [ ] xi18n
- [ ] run
- [ ] config
- [ ] help
- [ ] version
- [ ] doc
Description
With the last merge project build process is not working. I get failure that is pasted below. Tried to change --max-old-space-size=4096 still not working. Any suggestions what this can be ?
š¬ Minimal Reproduction
run command ng build --prod
in angular.json file we have
"production": {
"optimization": true,
"outputHashing": "all",
"sourceMap": false,
"extractCss": true,
"namedChunks": false,
"aot": true,
"extractLicenses": true,
"vendorChunk": false,
"buildOptimizer": true,
"fileReplacements": [
{
"replace": "src/environments/environment.ts",
"with": "src/environments/environment.prod.ts"
}
]
},
š„ Exception or Error
Here is the log file.
{ Error: Command failed: ng build --prod --configuration=config --output-path=test --base-href=/test/
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0x8dbaa0 node::Abort() [ng build --prod --configuration=config --output-path=test --base-href=/test/]
2: 0x8dbaec [ng build --prod --configuration=config --output-path=test --base-href=/test/]
3: 0xad83de v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
4: 0xad8614 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
5: 0xec5c42 [ng build --prod --configuration=config --output-path=test --base-href=/test/]
6: 0xec5d48 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
7: 0xed1e22 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
8: 0xed2754 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
9: 0xed53c1 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
10: 0xe9d636 [ng build --prod --configuration=config --output-path=test --base-href=/test/]
11: 0xeafee7 v8::internal::Factory::NewLoadHandler(int) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
12: 0xf2f4db v8::internal::LoadHandler::LoadFromPrototype(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::JSReceiver>, v8::internal::Handle<v8::internal::Smi>, v8::internal::MaybeHandle<v8::internal::Object>, v8::internal::MaybeHandle<v8::internal::Object>) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
13: 0xf36d1f v8::internal::LoadIC::ComputeHandler(v8::internal::LookupIterator*) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
14: 0xf3d94c v8::internal::LoadIC::UpdateCaches(v8::internal::LookupIterator*) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
15: 0xf3dffc v8::internal::LoadIC::Load(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Name>) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
16: 0xf42935 v8::internal::Runtime_LoadIC_Miss(int, v8::internal::Object**, v8::internal::Isolate*) [ng build --prod --configuration=config --output-path=test --base-href=/test/]
17: 0x235e3925be1d
š Your Environment
Angular CLI: 7.3.1
Node: 10.15.1
OS: linux x64
Angular: 7.2.6
... animations, common, compiler, compiler-cli, core, forms
... http, platform-browser, platform-browser-dynamic, router
Package Version
-----------------------------------------------------------
@angular-devkit/architect 0.13.2
@angular-devkit/build-angular 0.13.2
@angular-devkit/build-optimizer 0.13.2
@angular-devkit/build-webpack 0.13.2
@angular-devkit/core 7.3.2
@angular-devkit/schematics 7.3.1
@angular/cdk 7.3.3
@angular/cli 7.3.1
@ngtools/webpack 7.3.2
@schematics/angular 7.3.1
@schematics/update 0.13.1
rxjs 6.4.0
typescript 3.1.6
webpack 4.29.0
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 171
- Comments: 205 (38 by maintainers)
same issue on ubuntu server 18.04, but this solved my problem: node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng build --prod
UPDATE :-
Iāve seen this issue only on WINDOWS machine particularly. Anyone else, noticed that? Or have got same issue in other machines?
Note: I tried 2 Windows machine, all tests gave same result. Both machine have same config almost. i.e. 8GB RAM/Admin Access For Bash/CMD/PowerShell/ i3 & i5 Processor (quad core, both).
And, I admit that the project that Iām working on is a MESS in terms of code quality. I tried one clean code, full fledged application, ON WINDOWS, it was slower in build process, but worked fine.
I tried MESSY projectās code, it never compile on default node heap allocation (around 1.2 GB, not sure.). It works if I allocate more heap memory to node process, say 5 GB. But, in that case too, it took more than 30 mins to build. I tried the
5 GBnode heap option with and without--aotAND/OR--prod.Same code when tried on
CentOScould instance of 1GB RAM/ Virtual Xenon Processor (1 Core), generates build within 90 Seconds. with and without--aotAND/OR--prodIām not sure, but, looks like problem is with Node Service not with Angular CLI itself, coz Iāve seen some performance lag in new NodeJS releases after last update (locally on WINDOWS machine only).
Some observations below.:-
Even though allocating 5 GB of heap, it never used more than 1.9 GB (max) and allocation process was very slowā¦ it was increasing RAM consumption 1-2 MB per 3-5 seconds. And, was not using Disk Resource as it used to be in older version. (Disk access was very high previously on WINDOWS machine, but build process was faster without extra RAM allocation. This time, same machine almost same code with some bug fixes, but node was upgraded, disk consumption was lowered but slow build generation.)
Hope somebody can confirm this.
Hey all,
Just wanted to give you an update on the steps we (and others in the community) took to address this problem in Angular 8:
These efforts are still under way:
Hereās what you can do to take advantage of these fixes:
@angular/core@8.2.2(together with the other framework packages)@angular/cli@8.3.0-rc.0and@angular-devkit/build-angular@0.803.0-rc.0typescript@3.5.3node-sassas a dev dependency if you use SassOne special note is that we completely reworked how Differential Loading works. It should now be much faster, but can use more system memory depending on the numbers of cores you have. This is different from the node memory limit.
When you hit the node memory limit you see the
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memoryerror. But if you hit the system memory limit you should seeFatal process OOM in insufficient memory to create an Isolate.The new Differential Loading setup makes it rarer to hit the first, but if you have many cores but not a lot of memory, it can be more common to hit the second. Weād like feedback on
@angular-devkit/build-angular@0.803.0-rc.0please, so that we can address this behaviour if itās a problem, by disabling parallelism in some cases or make it configurable.Upgrading to 12.x worked for me. Loving Angular less and less each dayā¦
I updated the node.js to the latest
LTS(at this moment12.13.0) version and everything work perfectly. No need anymore set themax-old-space-size.@noopur-dabhi I donāt think it is fair to say that using
node --max_old_space_size=8192is not best practice.It is nothing but allowing NODE process (which is going to build your app now) to use more than DEFAULT heap memory (from your RAM or SWAP area).
And this type of allocation is very very common in most of the app development (especially deployment) process. Example, in Java, if we want to restrict our java app heap usage, we add additional flag like
--Xmx64mthis means maximum heap memory allowed is 64MB.So if āBest Practiceā is the only concern, I think itās not a problem using
node --max_old_space=8192, this means you are allowing NODE process to use maximum of 8192MB from memory. nothing to do withBest Practice Concern.Same issue here running on a Mac book pro. macOs Mojoave. Any environment with production: true
Build Problem Solved. Suggestion is to change build logic. Instead of angular
ng buildusenode buildset NODE_OPTIONS=āmax-old-space-size=30720 command worked for me.
Heya all,
First of all sorry for the delay. I AFK for a good chunk of last week.
Iām trying to give as much detail about what I did to investigate this sort of problem because in the past I have been asked about it (looking at you @wardbell, @johnpapa, and @DanWahlin š). I also find it useful for myself to come back to these write-ups in the future because I forget the details of how to do these things, or want to share with colleagues so they can do the same. A lot of work described here is done by fiddling directly in node_modules and changing logic in there to write files that I later debug, or log information, or use different packages, stuff like that. I find that fiddling with the JS sources really is the best way of investigating things because I donāt know what Iām looking for most of the time, so it helps to explore.
At this point weāve already ascertained that Node 12 has a default 2048MB memory limit (https://github.com/nodejs/node/issues/28202) and is using more memory than Node 10 (https://github.com/nodejs/node/issues/28205). This is unfortunate but a reality at the moment, so we should focus on Node 10 for now while the node folks address memory issues. Iām using Node 10.16.0 myself for testing.
Iāve been working on profiling where memory is used in the problematic builds. The first step to do that is decide on what project to get profiling data from. I chose @johnpapaās awesome-angular-workshop because in https://github.com/angular/angular-cli/issues/13734#issuecomment-500849058 I came to the conclusion it was the project where we could see the biggest impact:
I made a fork of it at https://github.com/filipesilva/awesome-angular-workshop, commit SHA 9076a3d, with local modifications to get reproducible benchmarks easily. Thatās the code I used in https://github.com/filipesilva/angular-cli-perf-benchmark. The build command I use is
ng build 5-ngrx-end --prod, since this workspace has several projects and I just want to use this specific one.My initial idea was to use the Chrome Node DevTools, gather a bunch of profiling data, then work away at trying to explore it to find actionable information.
To open the Chrome Node DevTools, open chrome://inspect/#devices in Chrome and click āOpen dedicated DevTools for Nodeā. When you have a node process started with the
--inspect-brkflag, DevTools should automatically attach. An easy way to run Angular CLI like this is to make a npm script:"ng-inspect": "node --inspect-brk node_modules/@angular/cli/bin/ng",. Then if you runnpm run ng-inspect -- build --prodand have DevTools open, it will automatically connect. At this point you can set breakpoints and take profiles.The memory tab has a couple of different profiling tools and descriptions of what they do. When you use these, you will have to go to sources and click the resume script execution (top right) before things happen. I spent some time trialling the different types of profiles on simpler node processes to figure out which one would be better. I was already used to reading the CPU profiles from when I debug build time, so the Allocation sampling (aka heap profile) sounded interesting because it should show how much memory each function is using at a given time. Heap snapshots show whats in memory at a given time and what is retaining those objects, it would also work but isnāt as easy to read and get the big picture of whatās happening.
To be honest capturing profiles with DevTools has been a lot harder than I expected. It actually does not seem to deal well with capturing profiles from large running Node processes. I saw it crashing while taking profiles, or at the end when processing them, for most of the profiles I tried to take.
I tried to explore alternatives. One of them was to profile smaller parts at a time so that DevTools didnāt crash, but it looked like it would always crash on some types of profiles anyway, even on new CLI projects, so that didnāt seem to matter much. I tried ndb but it seemed to suffer from the same problems. automated-chrome-profiling seemed promising byt hadnāt been updated in a while, and only took CPU profiles. @manekinekko suggested but as far as I could tell it wouldnāt give me very granular information on memory usage. node-oom-heapdump looked good for heap snapshots but required native bindings, which I didnāt like much because it would make it harder to later integrate directly into the CLI for user profiling. sampling-heap-profiler looked great for heap profiles, but also needed native bindings. @ofrobots suggested https://github.com/google/pprof-nodejs which I tried and seemed to give me really nice visualizations, but it also needed native bindings and needed a go tool to read it, which would make it harder to integrate.
Ultimately I opted to make some custom profiling code based on the Node experimental profiler support described here. I drew inspiration from code in sampling-heap-profiler and the profiler API to figure out what to use. From this I prototyped some automated profiling code that looks like https://github.com/angular/angular-cli/pull/14936. At the time of writing I manually control flags inside the file but maybe in the future we can expose it somehow. It can take CPU profiles of the whole process, or heap snapshot/profiles at set intervals. This seemed like a good way of doing it because I might want to ask people with build speed or memory problems in the future to take these profiles and send to us.
With this setup I was able to take reproducible profiles with minimal fiddling around. Reducing the fiddling is really important when it takes a long time to gather your information, because the more fiddling you need to do, the higher the likelyhood of you botching a given sample. If you botch a sample and only find it hours later, then you might have to throw away most of your conclusions for that period since your starting points were bad and so conclusions and comparisons werenāt accurate.
Armed with this need tool I managed to capture a few heap profiles of CLI7 and CLI8 (without differential loading) builds to compare them. I opened them on DevTools and tried to figure out what parts of the build pipeline were involved in each section of the profile.
The CLI7 one looked like this:
The CLI8 one looked like this:
CLI8 uses around 180mb more memory, for a total of 760mb, versus the 580mb used by CLI7. What stood out the most to me was the difference in CSS processing. It looks like it takes a lot longer in CLI8. In CLI8 we started using dart-sass (published in npm as just
sass) by default instead of node-sass (see https://github.com/angular/angular-cli/pull/11791, https://github.com/angular/angular-cli/pull/13950) so itās a great candidate for investigation.We expected
dart-sassto not be as performant asnode-sassso we let you go back tonode-sassby installing it in your project (npm install node-sass --save-dev). As long as itās installed, weāll pick it up automatically. You can also usedart-sasswith the fibers package for better performance by installing it in the same way asnode-sass. Bothnode-sassandfibersuse native bindings and might not work on all machines, which is why we donāt install them by default.Then I compared profiles for CLI8, CLI8 with
node-sass, CLI8 withfibers, and CLI 7 side by side to see how they changed.So using
node-sassseems to return memory usage CLI7-levels, at least on this project. Using justfibersonly reduces memory by some 20mb.As @clydin pointed out, this might not be the full picture since the heap profiles are only for memory managed by V8.
node-sassmight be using more memory that doesnāt show up in the profile. So I benchmarked the processes themselves:node-sassfibersSo it does look like the
node-sassreally is the one that uses less memory, and takes less time to build.fibersis the second best. The average memory used in thefiberscase seems to conflict with the results we got from the heap profiles though, here we see CLI8 withfibersusing ~750mb versus the original CLI8 ~1000mb, but in the heap profile usingfibersonly reduced memory by ~20mb. I donāt know why this is. Maybe itās because of the nature of averaging used memory, or maybe itās because there are more garbage collection cycles taking heap profiles. But at the end of the daynode-sassseems to be the winner as far as performance goes.I also found it weird that a big chunk of extra memory in the CLI8 heap profile seems to come from PostCSS processing. I wasnāt expecting PostCSS to need to do anything different. I thought that maybe I just misinterpreted what that section was used for. But maybe the
dart-sassoutput could be different fromnode-sassand that could cause more work for PostCSS. It was worth looking into so I added some code directly insass-loaderto write the results of sass compilation to disk. The output fordart-sassandnode-sassseemed like it was the same really, aside for some whitespace, so I think I just misinterpreted the memory assigned to Sass processing as being assigned to PostCSS.@johnpapa, as an aside, I did notice that most component CSS files in this project are importing
mixin.scss. This file actually ~has no mixins and is just~ includes the whole material theme at 70kb (pre-optimizations) of toplevel CSS definitions. Component CSS is standalone and isolated from other CSS, so there is no deduplication or anything. When itās imported into component CSS, youāre adding 70kb worth of CSS into that component, which is probably several times over the size of the component itself. @jelbourn has described importing non-mixin Sass this as an anti-pattern but itās not the sort of information thatās obvious all round. We could do a better job of informing people about it, and warning them when component CSS is very large (cc @vikerman).Another aside is that I also noticed that compiling just the single project via
ng build 5-ngrx-end --prodactually compiles all.tsand.scssfiles insidesrc/. This happens because all projects in this workspace use the same./src/tsconfig.app.json, and this tsconfig includes all ts files (except unit tests). Which is to say, compiling a single project here actually compiles all projects. The only way to truly address this is to have one tsconfig for each app. I wonder how common this mistake is. I wouldnāt be surprised larger projects accidentally include extra TS files. Maybe we could warn users when we see the TS compilation has a lot more files than those that webpack loads (cc @vikerman, @clydin).So I guess the takeaway here is that if you have a large project, you should install
node-sassto your devDependencies. This will make building faster and take less memory.Although it didnāt seem to change between CLI7 and CLI8, the Angular compilation itself is still by far the largest contributor to the memory footprint, at around 230mb. Maybe we can fully drop all references to it when not building in watch mode, allowing Node to free up that memory. Iāll explore that next.
Has anyone experienced this in a project that doesnāt use sass?
I am facing same issue while
ng serve --prodReference Image
@Flyrell I have the same issue with Nodejs 14.4.0 and Angular 10
This worked for me!
This worked for me!
Seems like the issue is back with v10 release. Using
@angular-devkit/build-angularv0.1000.0 and@angular/*packages v10.0.0.Node: v14.1.0 NPM: 6.14.5
Stack trace:
After upgrading Node.js to v14.4.0 the issue seems to be resolved.I can confirm, that this is working.
Of note as well is that Node.js 12 automatically configures its memory options based on the available system memory.
Please provide me the solution for this build problem
@filipesilva just install Node 12.8 solved the issue . thanks.
In my case the problem only appears, when building
--prod --source-map. The normal--prodbuild runs just fine.Using
--max_old_space_size=8192as suggested by @nokhodian it works with memory consumption peaking at ~3.8GiB during source map generation.Build environment:
use ābuild:prodā: ānode --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng build --configuration=prodā, in your package.json file
this will tell node to exceed default memory allocation by given size in command.
Just happened to me after updating from 9.0.3 to 9.0.5
Worked for me too
Updating to latest node.js worked for me too!
I also have this issue but can resolve it with āNODE_OPTIONS=āmax-old-space-size=8192ā. Ram usage during build gets up about 14GB.
I have the same error in Angular 8 and this solved my problem:
node --max_old_space_size=X node_modules/@angular/cli/bin/ng build --prodWhere
X = (2048 or 4096 or 8192 o..)is the value of memoryStill working on this problem. Notes from the last few days below.
While discussing it with @clydin, he mentioned that the average memory isnāt really what busts the limit. Itās the peak memory. Average is ~750mb while peak is ~1500mb. We should focus on that instead.
First I need to figure out whatās using memory at the peak. I guess itās uglify and build optimizer but I donāt know. Itās weird that the profiles I take never show peak usage though. I wonder I Iām doing something wrong there. I just started a process without any profiling and see memory at ~1200 during what seems like the ng compilation, then going down to ~900mb when loading files, then ~1300 and ~1500mb at the 90%+ stages. But my profiles show a ~800mb allocation max.
Maybe I need to control the timing of when I take these profiles. If thatās it, then I need to find the exact time when usage is high and take a profile there. Using
process.memoryUsage()as described in https://www.valentinog.com/blog/memory-usage-node-js/ might let me identify high memory usage periods. I can set a memory threshold and only take profiles at that time.It might be that taking a heap profile triggers GC, or just that the increased running time prompts GCs more often. I can try to trigger gc right before each heap profile and see if the result is the same. https://stackoverflow.com/questions/27321997/how-to-request-the-garbage-collector-in-node-js-to-run shows how. Did that, and saw no real difference in the profiles. Whatever the profiles show seems to actually be retained.
I used setInterval with a log for process.memoryUsage() inside, and also kept an eye on the reported task manager memory usage, and noticed a couple of things. When I say that I saw 1234/5678 I mean 1234 was on the task manager and 5678 was on the process.memory logging.
The highest 1300/??? happened during the Angular Compiler phase, but only with dart-sass. With node-sass it lower at 900/700 and saw interval logging. I got almost no logging for that phase with dart-sass, I think because the setInterval never got on the event loop properly, I guess because a lot of things are sync? With node-sass I saw a lot of logging at that time. Maybe itās a native binding thing, it might free up the event loop. Anyway dart-sass seems to make resource loading in the angular compiler extra bad by piling on memory on an already expensive set of operations.
I saw some peaks of 1200/900 while progress showed loading individual files. I think thatās Build Optimizer since its a js loader.
I barely saw any memory usage during Terser, but I bet thatās because of the cache. I turned off the terser-webpack-plugin cache and saw 1100/940, but also a bunch of spawned processes between 40 and 400mb.
So as far as I could tell, the sass thing was actually a big part of reaching the peak because it runs during the ng compilation and piles on something thatās already memory intensive.
I bet it doesnāt help that weāre doing webpack subcompilations for that either. This also means that even if we freed all references to memory using during the ng compilation, it wouldnāt matter as much because that compilation is still part of the peak.
We already have plans to remove Build Optimizer so that doesnāt bother me much
In regards to terserā¦ Well if we had a smaller main process memory footprint, obviously the total would be reduced. But maybe we can also make that plugin smarter. It seems to run on chunks proper https://github.com/webpack-contrib/terser-webpack-plugin/blob/a0d83170168e786004fa12b94525b2934a17a4f5/src/index.js#L416-L419 But we know terser canāt break out of the webpack module loader so thereās no point is processing whole chunks. Might as well process the individual webpack modules, that should be much smaller.
So important things to follow up:
I was facing same issue for ng serve,
npm cache clean --forceresolve this issue@maxisam Iām on 10.x until AWS decides to use a more recent version or I decide to create a custom Lambda Layer. But this should be working on 8.9+ according to the docs.
From https://www.npmjs.com/package/@angular/cli
I was going to add a rant here, but Iām just going to send it to the bosses with a request to switch frameworks. It seems like this project is just in total chaos.
Hereās something fun to try (8.1.3):
Why does a brand new project already require a package update? 6.4.0 was in Jan. 6.5.2 was in May.
Angular CLI https://github.com/angular/angular-cli/blob/cae4a9dc4afb53292cb44ebbc644d33b7d8d159e/packages/schematics/angular/utility/latest-versions.ts
Angular https://github.com/angular/angular/blob/master/package.json
We are experiencing this problem when we use
ng build --prodSometimes needed because we use different environment settings.node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng serve --prodsolves the problem but the build time is exceeded to 3 minutes. I donāt think this issue is solvedWeāre using Bitbucket pipelines and we canāt increase the memory limit to 8GB. Also our application takes about 7-10min to build and we need the sourceMaps for our Sentry error reports.
UPDATE: running
ng update @angular/cli @angular/coresolved the problem.for me the solution was editing my tsconfig (ionic4 and angular8)
i was playing around adding properties to tsconfig, towards the end i enabled source maps for my prod build inside
angular.json.i had to remove the
sourceMap: trueandsourceBase: "/"from tsconfig , after that --prod works like a charm@Senneseph any chance I could see that project? I donāt think a 24 component project should use 16gb of memory. I havenāt seen any project that uses that much memory at all really
There was a severe, but rare, but with typescript 3.4 (https://github.com/microsoft/TypeScript/issues/30050) that caused a lot of extra memory to be used in some cases with union types. Weāve just released support for TS 3.5 in Angular 8.2.0 with
@angular-devkit/build-angular@0.802.0-rc.0. That might be whatās affecting your project, you can try updating to those versions to see if it helps.I have a relatively small app, maybe 24 components, including routes, modals, and a few classes for state management, etc. And on a 16GB machine, even with all of the suggestions posted here, thereās a 3/4 chance it will hit this bug. I donāt understand how it can use this much memory and really donāt understand why it isnāt at least trying to page it out to the SSD (sure, it would be slow, but if it goes from not building at all, to building in 10 minutes, 10x the old build time, thatās still pretty great).
This is with sourcemaps off, node getting the max_old_space_size setting, buildOptimizer off, optimization off.
3 versions ago, this same app could build with a 3GB machine. Should I really roll everything back to Angular 5 or am I better off just switching to Vue?
If your application crash while building scss files try installing those dependencies node-sass, style-loader and sass-loader, this worked for me!
If it helps anyone else, when we switched from ā@angular-devkit/build-angularā: ā0.803.26ā to ā@angular-devkit/build-angularā: ā0.803.27ā it caused memory to run away during prod builds at ~43% build progress. It would get stuck at this step and escalate to >25GB of RAM rather than ~1.5GB.
When we switched back to ā@angular-devkit/build-angularā: ā0.803.26ā, it worked again. 0.803.27 is a recent release around 6/11/20, so it would only explain new occurrences of this issue.
With the latest versions of node is solved (at least for me). I recommend the team node to use in the package.json constructs Engines: https://docs.npmjs.com/files/package.json#Engines
Had the same issue and upgrading node to 12.12 today let me
ng build --prod@filipesilva, I confirm that
ng buildis working well now with:ng build --configuration=productionangular.json
A few months ago when you tested my big fat SPA, the memory consumption of node was over 2GB, so I had to use:
node --max_old_space_size=8192 "node_modules/@angular/cli/bin/ng" build --configuration=productionNow with a larger codebase being built with
ng build, the memory peak of node is 1.6GB.Good work. š
Last week, after merging a relatively small branch into the master branch, the build of my project started to experience this memory issue reported here. The interesting part is that the problem was happening on the CI server only, none of the developers was having trouble locally, even with
--prod.After reading @filipesilva 's comment above, I decided to update angular as follows:
The problem is now fixed!
My impression is that the extra memory needed for DL dual builds, plus the fact that typescript was bumped to 3.4 which has the aforementioned problems regarding union type edge cases, started causing this extra memory consumption for some projects.
@johannesjo both Build Optimizer and source maps need extra memory to process so thatās expected. Itās just more work to be done, and more intermediate data to store in memory. We are looking at moving the Build Optimizer step from app builds to library builds, but that would be in 9 only at best.
Edit: tagged @JohannesHoppe instead of @johannesjo. sorry about that!
@arayik-yervandyan Iād like to keep this open, as thereās more to this issue than just increasing the memory limit. Some projects are exhibiting disproportionate increases in memory usage, and we also seem to have memory that is not freed on differential loading builds.
My conclusions are this point are:
v8, DL, Node 10.10.0vs 2335 MBv8, DL, Node 12.3.1v8, no DL, Node 10.10.0vs 1456 MBv7, Node 10.10.0v8, no DL, Node 12.3.1vs 1892 MBv7, Node 12.3.1I have gathered a few more example repositories, but after doing all these benchmarks manually I think I have to automate this. I really should have automated it from the start really. So Iāll set up a repository that does this using CI.
Debugging the memory situation in CLI 8
Environment:
Measuring memory usage
We have a private benchmarking package for the CLI https://github.com/angular/angular-cli/tree/master/packages/angular_devkit/benchmark.
To install it, clone the CLI repo, then run follow these commands:
The filename on the last command might change, depending on the version of the CLI you clone.
To use it, run
benchmark -- command. The command will depend on the project you use. To benchmark a prod build, dobenchmark -- ng build --prod.You can also do
benchmark --iterations 20 -- ng build --prodto get a larger sample size (20 instead of 5).Methodology
I have gathered a bunch of projects of various sizes, and have setup a v7 and v8 folder for each.
To get the different versions I first update a project to version 8 following https://update.angular.io/. Thats the v8 variant.
Then I make the v7 variant by replacing:
@angular-devkit/build-angularversion from~0.800.0to0.13.8@angular/cliversion from~8.0.1to7.3.9targetin./tsconfig.jsonfromes2015toes5.To change between node versions I used NVM. To disable Differential Loading (DL) I changed the
targetin./tsconfig.jsonfromes2015toes5.On each I do
benchmark -- ng build --prod, then compare the results.Benchmark matrix:
Results
new-project
From
ng new new-projectGenerated using Angular CLI 8.0.1
Super-productivity
From https://github.com/aviabird/angularspree
Provided by @johannesjo in https://github.com/angular/angular-cli/issues/13734#issuecomment-497393904
Not sure if this will help but here is a graph of our memory consumption.
The first spike is a default build with differential loading. Second spike is differential loading using node-sass.
The final spike is a build with differential loading turned off.
Both builds with differential loading failed after they reached the memory limit of the server. The last build was successful.
We have about 20 different projects defined in our angular.json. It would normally fail when building the second or third project.
I have run into this issue on a project that does not use sass. I did some environment testing because I originally thought it was an issue relating to Azure Dev Ops.
Using Azure Dev Ops Pipelines:
@Flyrell @gvsakhil Avoid commenting on closed issues that are referring to old versions. There is an opened issue #16860 for NG9, which would be more appropriate.
Note that there is even now an issue #18087 for Angular 10, which got closed so you can expect some improvements with next release.
Heya all, this issue is pretty old by this point and weāre mostly focused on resource usage issues on newer CLI version. This issue also contains several different problems that seem to manifest in similar ways, which makes it hard to zero-in on. If youāre still seeing problems like this, please open a new issue with reproduction details.
As far as I remember, they said that they test the applications to make sure nothing breaks after necessary code upgrades. I believe, upgrading the code to take in the enhancements is anyways inevitable.
But this current issue has nothing to do with that. Itās more about application size. If the application size is growing, you will have to increase the heap size for the node process.
Same problem here. Angular 7. Any Node 12.X version.
Works with Node 10.
A Note For Angular/Windows/Docker Users
If you are trying to run ānpm run buildā or something of the sort inside your Docker file and you are getting this error, as mention above, you want to use the following run command instead, increase Node.jsā access to memory.
node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng build --prodIn addition, however, Docker is running a Linux VM in the background that will also have memory constraints that need to be addressed.
Navigate to āC:\Users<profile name>\AppData\Roaming\Dockerā where profile name is the named of your logged in profile. There you will find a āsettings.jsonā file. Open it and adjust the āmemoryMiBā value to also be 8192. You will need to restart for the changes to take effect.
WARNING
Increasing the VM memory can have a drastic impact on your machine performance, so ensure that you have enough physical memory to do this before trying. If this solution works for you, you should then try to roll back the values as much as you can, by doing a process something like, halving the value initially, testing, halving it again if it works, or adding half of the new value back if it doesnāt etc. This will help you minimize the impact on your machine.
You will also want to disable Docker as a start up program for Windows 10. Simply search āstartup appsā in the task bar and select the corresponding option that appears. This way, you wont have a VM you are not using bogging down your system.
Thanks a lot @filipesilva for the exhaustive benchmark. Installing node-sass didnāt pull us out but upgrading angular to v8.2.6 did the trick.
Still, isnāt it weird that most of people reporting the JavaScript heap out of memory error get stuck at 92% precisely ? I would blame the Terser optimisation process for the sudden memory peak.
Please keep the discussion on topic and review the code of conduct for behavior.
https://github.com/angular/code-of-conduct/blob/master/CODE_OF_CONDUCT.md
Iāve been trying to figure out if thereās really a memory leak in Differential Loading (DL) builds. Below are my notes.
TLDR is that there doesnāt seem to be any memory leak in DL, but when
--max_old_space_sizeis used V8 will keep memory around. A build can finish successfully with--max_old_space_size=300without erroring but will use up to 2500mb of memory if given--max_old_space_size=10000.Trying to figure out if DL really stacks memory endlessly. Modified a DL build to double the entries in webpack-browser-config.js. Normal (2) builds uses 480 average/960 peak, 4 builds is 810/1700, 8 builds is 1400/2500. So each build leaves increases average by 200ish.
I donāt quite get why peak also increases so much though. Peak should be average+somethingā¦ right? Actually thatās wrong. Because we already know the usage is going up over time. So the average during the first build is much smaller than the average during the last build. Kinda like an integral. So the number I should care about really is peak, because it represents things better for the multi-build case.
Also noticed that process.memoryUsage().heapUsed gave me numbers were much lower than what I saw on task manager. I found that odd. I also didnāt disable the terser cache. I should do that to see if thereās also a scaling cost there, after figuring out this initial scaling cost.
Tried doing global.gc every 30 seconds to figure out if memory was truly retained or if node was just very comfortable with the 10gb limit. Did it over 5 iterations. Without GC I got 1200/2150, with GC I got 1200/2050. So only a slightly smaller peak. This shows the memory canāt be freed.
Took heap profiles and heap snapshots every 30s. Also tried the memory leak profile on chrome devtools. The heap profiles show almost no change between them. Very odd. Maybe I need to restart the profiler between samples? The snapshots fail to load too, with an error while building retainers. Will re-make them.
Tried profiling again with a lower sampling interval and didnāt see anything different. All snapshots but the first still failed with the same message. Also tried just taking one snapshot at the end and got only a small snapshot (200mb).
Tried taking a snapshot manually in Chrome Devtools during the 8th (and last) build. Devtools said the process was using ~2000mb memory just before I took the snapshot, and then it showed only ~260mb. The snapshot was only 254mb.
So I thought it might be a GC thing. Tried limiting max_old_space_size to 1gb and it still built. Also built with 600mb and 400mb, but failed with 300 and 200. A single build run failed with 200 but succeeded with 300. A default DL build failed with 300 too.
So as far as I can tell there isnāt any real amount of memory being kept because of DL. Maybe thereās to 100 extra mb. A new app will use roughly the same memory (~300mb on the main process, plus up to 450mb on spawned Terser processes) for producing 1 build or 8 builds in the same command. But if I set the memory limit to a a very large number (10gb) then it will keep more memory around (up to 2.5gb) until it needs to be freed.
@johannesjo @jsgoupil I think the issue with 8 has to do with a memory leak on prod builds with differential loading. Can you try disabling it? See https://angular.io/guide/deployment#opting-out-of-differential-loading
cc @alan-agius4
Yes, I am also facing the same issue
Guys hereās described the āworkaroundā for this problem: https://github.com/angular/angular-cli/issues/5618
Yes, I was using the dev server for a little more today and the issue still exists.
@kumaresan-subramani do you know about error Npm failed with return code: 3221226505
I am getting this error while build on vsts
Running the ng update worked for me as well. One of the minor point versions must have had a memory leak of some sort. Moving from 8.2.8 to 8.2.11 fixed the issue.
I had to upgrade to node 12 AND increase our codebuild instance memory from 3 to 7GB. Just to build an angular app.
@maxisam
I definitely appreciate everyoneās hard work. I initially chose Angular because it seemed to be the best organized and most holistically thought-out project.
The issue is not whether people are working hard on solving this problem. The issue is that it should have been solved before it was released and declared ready for public use.
As for the rxjs thing, there shouldnāt be any breaking changes between the versions. The problem is that the comments read ā// These versions should be kept up to date with latest Angular peer dependencies.ā and yet someone is just typing whatever they want into the field instead of running a script to extract the exact string. And then no one tested if it was working as intended.
I imagine the Angular team is stretched too thin.
I had same issue with Angular 8. Windows 10 I guess different people have these issues because of different reasons.
My solution was :
Now again there are two things in this, either it worked because of max_old_space_size=8192 or may be because ive different local and global cli angular version. and using above command used the local/same version of ng
Ran into this very same issue when calling āng serveā w/ some custom configuration:
npm run env -s && ng serve --aot --configuration alphaIf you are just looking to get rolling, turning ābuildOptimizerā and āoptimizationā flags in my configuration to false did the trick and I was able to build within seconds.
We were experiecing the same issue. It turned out, we imported the base scss files in most of our components. That was not really necassary. Now without setting the
max-old-space-sizeNode option, we were able to build our apps in production mode without the error, after removing the core file imports in our components, where it was not necessary.We also switched from dart-sass to node-sass with
npm install -DE node-sassand that worked as well, without changing anything.Below my environment. Maybe this helps š
@jsgoupil got in touch and let me have a look at his project. I turned off the terser cache (described in https://github.com/angular/angular-cli/issues/13734#issuecomment-508815407) and benchmarked the āng build 3cconnect --prodā command:
This build has AOT and Build Optimizer turned off, and is using Differential Loading. The average memory usage is right at the maximum as well, which means the garbage collector is doing a lot of work to free up memory as the build progresses.
Tried turning on AOT and Build Optimizer and got OOM on the second DL build, during TerserPlugin. Used the
node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng build 3cconnect --prod --aot --build-optimizercommand to build and it succeeded.Benchmark for this build shows the following numbers:
I also tried without Build Optimizer turned on:
I wasnāt expecting Build Optimizer to impact either average or peak memory usage that much really. 500mb is a fair bit. We were already looking at a way to reduce the number of processed files in https://github.com/angular/angular-cli/pull/14998, which would help reduce the impact.
These numbers look much higher than the initial ones, but bear in mind that we increased the memory from the 1.4gb default to around 8gb. This means that there isnāt as much pressure for the the garbage collector to free up memory, so things stay in memory for longer. For comparison, here is the initial build (without AOT or Build Optimizer) with an increased memory limit:
So when using the same high memory limit, adding turning on AOT and Build Optimizer increased the average memory usage by 1GB and peak memory usage by 1.8GB.
Using just AOT increased build time by 70% (250->400 s), average memory usage by 25% (2000->2500), and peak memory usage by 36% (3800->5200 mb). Adding BO increased build time by 42% (400->570 s), average memory usage by 20% (2500->3000), and peak memory usage by 7% (5200->5600 mb).
I also tried to turn off Differential Loading by setting
"target": "es5",in./tsconfig.jsonwhile keeping AOT and Build Optimizer:Without DL, memory used goes does by ~500mb, similar to https://github.com/angular/angular-cli/issues/13734#issuecomment-508815407. Weāre looking at improving that.
I tried to figure out how big the project was by listing files in
src/by extension:This does not include third party libs so it doesnāt give us much insight as to the total compilation size. But Iām not surprised that this project needs the memory limit increased, thereās a lot of TS files in here. I had a look at
@jsgoupil in https://github.com/angular/angular-cli/issues/13734#issuecomment-497365534 you mentioned that an Azure machine with 7gb couldnāt build the app, which I find weird because 5.6gb was the maximum I saw on my Windows machine.
Maybe the CI machine is already using 1.4gb? That would make sense. You can probably check how much memory the machine has before the build. But you also said the machine has only 7GB and you are using
--max_old_space_size=8192. Iām not really sure how Node behaves when you say it should use more memory than is actually available. So I wonder if thatās a problem: you give it a bigger limit than available, so it thinks it can allocate all this memory, but when itās needed the system doesnāt have it so you get an error. You do say in https://github.com/angular/angular-cli/issues/13734#issuecomment-497426391 that the error is different with and without the flag so that does point to a different issue. Maybe check how much memory is free and set a limit below that, like 5000 or 6000.Regardless of the Azure machine you mentioned, it is impressive (in a negative way) that AOT compilation increases memory usage by 1GB average / 1.8GB peak. This is something that we are looking at in the general case. I think in your project it really is just a function of the amount of TS source files that need AOT compilation, I tried taking a profile of the memory usage and it mostly looked like the others Iāve looked at before from other projects.
I also noticed that you have some weird imports to
../../../node_modules/@angular/coreinstead of just@angular/core, and similar to RxJs. I think this makes no difference because resolution will still use thepackage.jsonthere. But I thought it was odd.@alex88 you can find the benchmark tool I used in https://github.com/filipesilva/angular-cli-perf-benchmark#benchmark-package. If you want to provide me a repro via a private github repository, Iād love to have a look too.
@zijianhuang a big thank you for giving me a repro to look at. I had a look at it today using the benchmark package described in https://github.com/filipesilva/angular-cli-perf-benchmark.
One thing to be aware of is that the memory numbers collected by that tool include spawned processes. In the CLI we use a minifier package (
terser-webpack-plugin) that spawns multiple processes, so itās useful to capture the memory the child processes use as well.But that package also caches the minification results. So the first run does more work than subsequent ones. So when I benchmarked your packages I manually edited
./node_modules/@angular-devkit/build-angular/src/angular-cli-files/models/webpack-configs/common.jsand changedcache: truetocache: falseto disable that cache and get consistent numbers.I also added these two npm scripts to help me benchmark things:
I set the same memory limit for all projects because I know Node will behave differently near the limit. So this way we should get more accurate numbers.
I had to remove the
Boundsimport fromavatar.component.tsbecause that gave me a build error in NG7. It was just used as a type so it was easy to remove. I replaced"target": "es2015"with"target": "es5"in./tsconfig.json. This disables Differential Loading as described here. I know itās an important part of CLI8 applications but we are already looking at a different implementation of it that shouldnāt require as much resources, so I didnāt want to also benchmark it. This way I could compare mostly equal applications.I noticed that your package.json between the two versions changes a lot of packages that arenāt just the Angular packages. That might have an impact on the benchmarks. After trying to reproduce your initial numbers Iāll also try to alter only Angular packages to see if I get different results.
Dev environment
In your reproduction your said:
Iām using my work machine (Win10, i7, 16GB RAM, SSD) which seems similar to your env. To benchmark I used the tool mentioned in https://github.com/filipesilva/angular-cli-perf-benchmark#benchmark-package. Iām also using Node 10.16.0 and npm 6.9.0.
Your test cases
In your reproduction your said:
Iām not really going to try
Test case 3because we already know that Node 12 is using more memory in general (https://github.com/nodejs/node/issues/28205). I donāt think thereās much we can do about that.Your
Test case 1is theCLI7/NG7/NG7depsscenario. I also say NG7deps because those dependencies are different from your NG8 project, which Iāll say has NG8deps. Later on that might make a difference. Ideally weād only change CLIx/NGx to get numbers.So for
CLI7/NG7/NG7deps(yourTest case 1) I got:And for
CLI8/NG8/NG8deps(yourTest case 2) I got:So for these two cases I actually got very similar numbers.
Variants
Something I know I did different from you was to ensure the minifier output was never cached. That has a big impact on resource usage. On CI machines you wonāt get this cache unless you cache
node_modules(thatās where it is stored) so I always turn it off manually for benchmarks. If it turn it back on this is what I see inCLI8/NG8/NG8deps:Here you can see that the first build (first place inside the parenthesis) uses up more resources than the others. Notably build time and peak memory goes down by about 1/3, and average memory down by around 1/4. These numbers are very similar to the ones you reported for
Test case 1.The other thing I did different was disabling Differential Loading (DL) in the
CLI8/NG8/NG8depscase, so I also took a benchmark with it turned back on inCLI8/NG8/NG8deps:So here we see that the build time roughly doubled with DL, which is expected because we do two builds one after the other. Weāre planning on doing a single builds with an extra step at the end instead soon.
We also see that average used memory has increased by 500mb, and peak memory by 700mb. In https://github.com/angular/angular-cli/issues/13734#issuecomment-500849058 I said:
The DL results from your project contradict that. In your project it increased by more than 200mb. So there is a scaling aspect to the DL cost that we see in your project. I donāt know what it is, but itās worth investigating. It points to memory being retained between builds, which might also imply that memory is being retained between build steps. If itās memory used in an earlier step of the build, maybe we can free it and reduce average memory usage in latter build steps.
@alex88 also mentioned that
es2015builds took more resources so I tried leaving"target": "es2015"intsconfig.jsonand disabling DL via the browserslist approach, which in your case is inpackage.json. I removed thenotin"not dead", "not IE 9-11"but that still made DL builds (you can see that by the two builds, one after the other). I had to remove all entries in browserslist before I stopped getting DL builds. We should check why these instructions arenāt correct (cc @alan-agius4, I think you worked on this). After I managed to get only ES2015 builds, I took another benchmark:This doesnāt seem to be very different from the ES5 benchmark. @alex88 maybe when you used
es2015you were actually getting the DL builds? You should also not be using Node 12 for now because of https://github.com/nodejs/node/issues/28205.@zijianhuang so to wrap up, I think the reason your project uses more resources with NG8/CLI8 than with NG7/CLI7 is that DL was turned on and that some builds were without cache. I really canāt be sure of the second point because I donāt know how youāre caching things locally and in CI, but the numbers I got here seem to point to that.
For us the follow up from investigating your projectās transition from Angular version 7 to 8 is that we really need to figure out what memory is retained between DL builds because that points to a leak, and that we need to have a more performant DL setup (@clydin has some ideas on that).
Just dropping this in here but if anyone is using AWS codebuild make sure to update your build environment to be configured with more than the minimum 3GB memory, I followed the steps above but had to re-configure my environment with more memory.
If you are on Node 12 and hitting the memory limit, you should try Node 10 instead for now.
Weāll look into the retained memory during Differential Loading builds and the outliers like
awesome-angular-workshop.I think he meant something in the lines of
node --max_old_space_size=4096 ./node_modules/@angular/cli/bin/ng build --prod, so the argument would be passed to node environment. But there are other issues linked to this one, like the huge time and memory spent while compiling.Heya, Iāve put up results and methodology in https://github.com/filipesilva/angular-cli-perf-benchmark. Will be adding more projects there next week.
The only extra detail that I have is that in the
awesome-angular-workshopproject build times almost doubled from CLI 7 to 8. But they didnāt on the other projects I tested.If someone wants to follow the methodology there using the scripts, it should be easy enough. If you find something interesting let me know. If you have a large project that you feel could yield interesting benchmarks, let me know (or add a PR) and Iāll add it there.
Upgrading from node 10 to node 12 resolved my compiling memory issue when using Angular 8.
However, when I try to run the compiled code from ng build --prod, it just displays a blank screen š Works fine with ng serve though
@johannesjo Yes
I think the increased memory requirement might be related to sass processing then. We moved from
node-sasstosassin 8.Can you follow the instructions in
@angular-devkit/build-angular: use sass instead of node-sass (ce15899)to force usage ofnode-sassinstead?@vindemi you are going to need to edit your Build Pipeline. Remove the npm build task and replace it with a command line task with the following script
node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng build --prodmake sure to set your working directory to the folder with your package.json.This issue is the same as https://github.com/angular/angular-cli/issues/5618#issuecomment-450151214, as far as I can tell.
I donāt see a lot of things I can follow up and investigate though. @themanojshukla mentioned Windows was taking a lot longer. @j3gb3rt is asking about sass, and I have seen reports in the past about sass making things slower.
Does anyone have a reproduction that we can look at and try to debug?