angular-cli: Angular 12 RC 2 - Compilation is enormously slow and crashes on large build.

🐞 Bug report

Command (mark with an x)

  • new
  • [ x] build
  • serve
  • test
  • e2e
  • generate
  • add
  • update
  • lint
  • extract-i18n
  • run
  • config
  • help
  • version
  • doc

The build time for for angular 12 RC 2 is very slow - was also slow in RC0. I have quite a big project with multiple lazy loaded modules. If i remove most of these and run its on a base set the compilation takes about 4-5minutes. When i include the lazy loaded modules it aborts after 5 minutes with a JavaScript heap memory.

With small code changes building/sealing takes 2-3 minutes.

[22280:00000215DB856250]  1371502 ms: Scavenge 2017.3 (2061.6) -> 2005.4 (2059.4) MB, 15.0 / 0.0 ms  (average mu = 0.373, current mu = 0.365) allocation failure
[22280:00000215DB856250]  1371545 ms: Scavenge 2020.7 (2061.9) -> 2008.2 (2059.4) MB, 13.9 / 0.0 ms  (average mu = 0.373, current mu = 0.365) allocation failure
[22280:00000215DB856250]  1371582 ms: Scavenge 2023.4 (2061.9) -> 2010.9 (2060.2) MB, 16.2 / 0.0 ms  (average mu = 0.373, current mu = 0.365) allocation failure


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 00007FF6C293978D]
    1: StubFrame [pc: 00007FF6C29A56B5]
Security context: 0x011ffec408d1 <JSObject>
    2: LimitedMapView$blocklist0 [00000090B0991EF9] [C:\Development\R9-core\business-fed-r12\node_modules\sass\sass.dart.js:19422] [bytecode=000001A84E877181 offset=60](this=0x023f71f290a9 <Object map = 0000003DDD0B6219>,0x023c27a9bae9 <PrefixedMapView0 map = 00000237AEFEDC59>,0x03138c835f49 <UnmodifiableSetView map ...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 00007FF6C1D251EF napi_wrap+114271
 2: 00007FF6C1CCFE66 v8::base::CPU::has_sse+66630
 3: 00007FF6C1CD0C66 v8::base::CPU::has_sse+70214
 4: 00007FF6C24E522E v8::Isolate::ReportExternalAllocationLimitReached+94
 5: 00007FF6C24CD301 v8::SharedArrayBuffer::Externalize+833
 6: 00007FF6C239983C v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1436
 7: 00007FF6C23A4A70 v8::internal::Heap::ProtectUnprotectedMemoryChunks+1312
 8: 00007FF6C23A1594 v8::internal::Heap::PageFlagsAreConsistent+3204
 9: 00007FF6C2396D93 v8::internal::Heap::CollectGarbage+1283
10: 00007FF6C239D624 v8::internal::Heap::GlobalSizeOfObjects+212
11: 00007FF6C23D344B v8::internal::StackGuard::HandleInterrupts+907
12: 00007FF6C211E459 v8::internal::interpreter::JumpTableTargetOffsets::iterator::operator=+7737
13: 00007FF6C293978D v8::internal::SetupIsolateDelegate::SetupHeap+546637
14: 00007FF6C29A56B5 v8::internal::SetupIsolateDelegate::SetupHeap+988789
15: 00007FF6C28BE73C v8::internal::SetupIsolateDelegate::SetupHeap+42748
16: 000000F5DDB88F1E

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 18
  • Comments: 47 (3 by maintainers)

Most upvoted comments

@hadrien-toma thanks a lot for sharing the project and all the details, reproductions like these are super helpful and crucial to get to the bottom of issue.

Myself and @JoostK are looking at this and it appears that this performance issue is only manifested during watch mode.

We’ll keep you posted.

PS: indeed your issue seems to be unrelated to this.

@alan-agius4 very minor improvement for me. My build times + ng lint version 11 — 7 minutes version 12.0.1 — 26 minutes version 12.0.3 — 23 minutes

I have a very large mono-repo using nx.dev so excuse the dramatic numbers

Hi @ganySA,

Here are what I found;

  • It appears that the project was manually updated and not using ng update. This caused several migrations that need to be run to amend you project not to be executed.

The left your project is a broken state were now "development’ builds are using “production” settings and hence cause increase in memory usage and slow rebuild times.

You can run the migration by using the below command (Note, don’t run the ng-update command multiple times as one of the migrations is non-idempotent)

ng update @angular/cli --migrate-only --from 11 --to 12
  • From the CPU profile, I can also see that dart-sass is taking most of the time of the compilation, one of your files tailwind.scss contains 121836 lines with a size of 2.44mb. Realistically I don’t think SASS can handle files this big and I am not surprised with the amount of memory needed.

Thanks @alan-agius4 I have confirmed using the following script did work for HMR:

"start": "ng serve --hmr --configuration development",

And to follow up, no I’m not using any other custom bundlers. Thanks!

You need to opt-in into HMR. Using —hmr.

if you are experiencing performance issues please share a reproduction and/or CPU profile, unfortunately without any of these we will not be able to Investigate.

@NateRadebaugh, are you using third party builders?

What I did to get around this issue:

  1. Make sure git is set up and everything is checked in.
  2. Delete the existing app folder/contents
  3. Using angular CLI to create a new blank app in the spot where the old one was
  4. Looking at the git status (eg on vs code) One by one, revert the deletes for non-build related files like your app components, services, directives, pipes, etc.
  5. Slowly restore everything except the custom HRM config, angular.json, tsconfig, and package.json/lock files. (Manually npm install dependencies until your package.json is mostly back to how it was)
    • Custom HMR setups like @angularclass/hmr seem to conflict with the “out of the box” supported --hmr in the latest version of angular
  6. Rebuild. Somehow magically the updates to your tsconfig and angular.json has made the build take ~4 minutes.

Angular 11: 8 minutes Ng update to 12: 24 minutes Recreate as 12: 4 minutes

Why/how? My guess is the automatic migrations aren’t working correctly, and so far what others are reporting is that the cache stuff in 12 with webpack 5 is busted with the migrations as is. Migrations also aren’t able to detect unsupported dependencies.

@hadrien-toma, do you mind opening a new issue for visibility and to keep this issue on topic.

If you have some time, can you try running them with the environment variable NG_BUILD_CACHE=0?

@alan-agius4 I updated with ng update and was still seeing very (frankly) depressingly longer serve times.

Running:

ng update @angular/cli --migrate-only --from 11 --to 12

took me back down from 120 seconds to 74 seconds which is around where I was before.

However it turns out that if you run the migrations multiple time it can actually disable aot which then gives the illusion of speeding everything up. The migrations file from Angular 11 to 12 will remove the aot setting if it is set to true (because it removes redundant default values by design). If you then run it a second time it will actually set the aot value to false (due to a bug).

@clydin it looks like my issue may be the same as this thread or related to #20792

My issue with for production builds in the CI, not development builds. In fact, local production builds are not impacted as much, but the CI builds are 2x.

@NateRadebaugh Are those production or development build times?

Sass processing was greatly improved in 12.0.1. However, in an unrelated change, the CSS optimizer plugin was also switched to use the Webpack provided plugin. This plugin has the unfortunate behavior of potentially creating an unbounded number of worker threads. In the event of this happening, performance can become significantly worse due to the large volume of context switches and can also lead to memory problems. A fix (PR https://github.com/angular/angular-cli/pull/20886) to limit the number of worker threads that the Webpack plugin can use has been merged and will be included in 12.0.2.

Yesterday release (12.0.1) contains several performance improvements. Please give it a try!

Sorry taken some time to work out best way to emulate a crash. Steps to reproduce the crash on compilation:

  1. ng serve
  2. open app in browser
  3. edit app.routing.ts as follows:
  4. comment out lines 171 to 183 and let it compile (a few lazy loaded modules)
  5. uncomment back lines 171 to 183 and let it compile – it crashes on my end

sometimes you need to do step 3 and 4 twice.

I did notice that due to import path src/@calendar5/styles the build is slower this appears to be related to @import '~@angular/material/theming'; in treo.scss.

This is above, is import is quite expensive and in your case it needs to be resolved around 65 times because you import this file that many times. Previously @angular/material/theming was a flattened into a single file not now it appears that something in version 12 changed is spit among a large number of files which indeed in this particular case is causing a significant performance degradation. //cc @jelbourn

Ps: I see don’t see the tailwind.config.js configuration file.

Glad we seem to have found the issue!

I did notice that due to import path src/@calendar5/styles the build is slower this appears to be related to @import '~@angular/material/theming'; in treo.scss.

This is above, is import is quite expensive and in your case it needs to be resolved around 65 times because you import this file that many times. Previously @angular/material/theming was a flattened into a single file not now it appears that something in version 12 changed is spit among a large number of files which indeed in this particular case is causing a significant performance degradation. //cc @jelbourn

Ps: I see don’t see the tailwind.config.js configuration file.

I have reset my configuration to a simple configuration from a new angular 12 project and slowly added back settings. I can further add that it seems a lot happier when i remove my styles folder from preprocessors “stylePreprocessorOptions”: { “includePaths”: [ “src/@calendar5/styles” ] },

Even with a basic ‘reset’ configuration build takes about 4 minutes. I have not pushed will do so now. I would say on 11 it takes about 1minute.

node --max_old_space_size=6144 ./node_modules/@angular/cli/bin/ng serve --open

Allows it to run longer than 5 minutes. Sitting at over 20 minutes at (phase: building)