angular-cli: Rebuild is way slower and stop working after several rebuilds

We discussed this issue in our weekly triage meeting. We also did a bit of digging and found the original PR (#6813) and it seems the main motivation was to fix a compiler bug (#6309). That bug only occurred with the “magic import string” which is no longer supported. The compiler has also evolved a lot since then, so this is unlikely to still be an issue. The circular dependency check is also only a warning, so I don’t think it really fixed the compiler problem, just gave a helpful error message about the problem.

General consensus is that the circular dependency checker doesn’t provide a whole lot of value. While circular dependencies can cause issues, most of the Node toolchain supports them so it isn’t a major problem for most web projects. Angular also supports circular dependencies^{[citation needed]}. The compiler throws errors in specific, problematic cases but generally allows circular dependencies. This is only an issue if a project owner decides it is an issue. As a result, it is reasonable for them to integrate such a check without necessarily expecting one out of the box. ESLint has one such check, and there are likely other implementations as well.

We could just run this check for production builds, however prod builds are usually done by CI and the circular dependency check is only a warning, so it is unlikely many humans would see this log and take any meaningful action from it. Since the check provides little value now, limiting it to production builds will only reduce that value further. As a result, we’re looking into removing the circular dependency check altogether. This will speed up build times (as shown above), reduce third party dependencies, and make node_modules/ a bit smaller.

Two quick things to confirm before we commit to removing the plugin:

1. @filipesilva, you originally added the circular dependency check in #6813 (thanks for the detailed commit message!), is there any other context or considerations we should have before removing it? Any objections to these arguments?
2. We should also include a proper circular dependency test case with our new dynamic import infrastructure if it does not already exist. This would also confirm that the previous compiler issue no longer exists and that it is safe to remove the circular dependency check.

@montella1507 Refresh once more 😄

So just to summarize: you’ll need @angular/compiler-cli (and its other framework friends) at version >= 11.2.5, with 11.2.6 being today’s release. Additionally, you’ll need @angular/cli 11.2.5, i.e. today’s release. Furthermore, it would be beneficial to use TS 4.1 if combined with AOT, but I don’t expect this to matter much in incremental rebuilds (only initial builds).

Thanks. Yeah profiling a long session will take down the Chrome Inspector at some point, it’s not uncommon for me to see Chrome take 10GB of memory to show a profiling session.

The last profile you shared does appear to still use the new Ivy pipeline, not the legacy one. Not sure why that would be the case, your set NG_BUILD_IVY_LEGACY=1 should be adequate.

In the meantime we have an idea of what is happening; it’s indeed on the CLI side where we can do some experiments to see if things can be improved.

I’m calling it a day now 👋

We could just run this check for production builds, however prod builds are usually done by CI and the circular dependency check is only a warning, so it is unlikely many humans would see this log and take any meaningful action from it.

we have modified CI configuration that it runs build from commits WITHOUT Source-maps and WITHOUT circular dependency check + AUTO night builds (any periodic builds) runs WITH SM and WITH CDCH.

It changed time required to 1/10 (for CI commit builds + develeper experience) and still viable option to detect circular dependencies (it is very rare situation for us - we have good barel files strategies + NRWL tags + linters).

I will be honest here… FInding out that Circular dependency plugin is the cause of very big rebuild times and disabling it is ABSOLUTE GAME CHANGER for us - because if you wait 30sec every 2mins or wait 3sec every 2 mins is big difference in DX + economy of development in angular.

I think THE TIME WHEN IS DEVELOPER WAITING(during rebuilds) is little bit underestimated and may be the reason why IS / IS NOT angular selected for larger projects.

Can you share an AOT profile as well? Thanks for all the insights btw, super helpful 👍

@montella1507 Thanks for the profile. It’s mostly what I was expecting from the earlier profiles and the changes that have been made since.

If you feel like it, can you run another experiment for me? Based on your profiling output I have been experimenting with cycle analysis (not the expensive step that you can disable, but an internal Angular compiler analysis step needed for proper output) and I’d like to know what the impact of that would be.

The build artifacts are here. In particular, you’ll need @angular/compiler and @angular/compiler-cli. These are based on 12.0 preview builds and only TS 4.2 is supported so you could either update TS to 4.2.x or set disableTypeScriptVersionCheck: true in angularCompilerOptions in a tsconfig.json file. If you encounter too much trouble with this I can probably cherry-pick it to 11.2.x, please let me know if that’s desired.

Using those build artifacts you should get slightly better AOT times (I estimate ~400ms less). Next, I’m interested in a more extensive optimization of which I have no idea how effective it is. This is hidden behind the NG_EXPERIMENTAL_CYCLES environment variable; if you set that to 1 it will take a different approach for cycle detection. I’d also like how that compares. If you can, please share AOT / NO CIRC / NO SM profiles using those build artifacts with and without NG_EXPERIMENTAL_CYCLES=1.

Instructions for installing the build artifacts can be found here.

Unrelated to the above, the CLI is ramping up for the adoption of Webpack 5 in v12. Your project is a gem in terms of the insights it provides, so if you could have a go at Webpack 5 with the CLI’s latest next releases that would also be super valuable. I think that using Webpack 5 does require a Yarn resolutions rule, though, so I’m not sure if testing that is feasible for you.

Just noticed that you are in a v11 project but the plugin is for a v12 prelease. For a v11 version of the snapshot with the commits you can use:

npm install git+https://github.com/angular/ngtools-webpack-builds.git#ba719cff4bab883b91684738b04bea0145ec2a81

As an aside, the v12 JIT mode should definitely be faster since support for the deprecated string format for lazy routes has been removed.

@montella1507 We think to have found the culprit, it was fixed on master in angular/angular-cli#20236 and will be ported to patch so should be part of next week’s release. I’ll share a snapshot build for patch once that’s available.

Will do at weekend.

That’s more like it 😃

The AOT improvement to deep-shared component is due to angular/angular#40947. The general AOT decrease from ~10s to ~4s is likely due to angular/angular#40948, although angular/angular#40947 also helps.

Which version of TS was this? And could you share an AOT / NO CIRC / NO SM profile as well?

@montella1507 CLI 11.2.5 just released, so you should be unblocked there.

The circular-dependency-plugin default is definitely something the team will consider. It is, however, a breaking change to modify the default so timing of the change will need to be carefully considered.

That snapshot build includes fixes for the repeat JIT build crash issue (fix via #20237) and the rebuild performance (fix via #20247). If you have the opportunity, please give the snapshot build a try and let us know if you are still encountering issues.

Also, the asset issue is due to a third-party dependency used by the Angular CLI so a separate issue is appropriate. Some more details regarding the asset performance in serve can be found in a comment in that issue.

@JoostK hi, btw i have made additional benchmarks on 11.0.9 to measure circular-dependency-plugin + sourcmap plugin effects on build time… here are the results:

When circular-dependency detection is enabled, build and rebuild takes 5-8x more time. I think it has serious issues with async circular dependencies or something…

And yes, “showCircularDependencies”: false is enough. Will do the same benchmarks on current version of angular / compiler later (when this issue is closed or requested by anyone explicitely)

Hi @montella1507,

thanks for the report. We have landed a large change in 11.2.5 with respect to incremental rebuilds, however the outcome should be that it’s now faster than before, not slower. It could be that you’re hitting a case where the compiler is unable to prove that work can be reused, in which case it may request more code to be regenerated than before.

I’ll have a look at the profile dump, thanks for sharing 👍

In the meantime we have also addressed another slow path that became clear from @montella1507’s profile, albeit only for v12. Closing this issue as not further actionable.

Hi I have a similar issue our build went from in angular 11.0.3 ng serve started in around 1 min is a big project after migrating to 11.2.9 the ng serves takes 3mins to start, I started downgrading libs to understand what libs was causing the issue, in the end the problem seems to be related to @angular-devkit/build-angular, I manage to run the project with everything updated to 11.2.9 except to that library which I set to version 0.1100.7 and runs again in 1 min, if I change to any version after that one the time is 3mins again. I college tried with 12 and said that was faster but still slower than before. I also generated chrome-profiler-events.json took around 30 min to generate is 500mb and chrome://tracing fails to load it, I could post it here if interested. Also Im thinking as a workaround keep angular 11.2.9 with angular-devkit/build-angular@0.1100.7 till 12 is release, could there be a problem with this?, it seems to run fine

Oh and one more thing that we noticed in the profile that deserves further research: in AOT mode the Angular compiler does template type-checking by feeding the Angular templates into TypeScript. This is done by injecting additional source files into a temporary TypeScript program. Creating that temporary program should be efficient as it’s mostly identical to the original. In your profile however, we’re seeing a codepath that indicates that TypeScript has determined that it can’t reuse everything from the original program, which incurs an extra cost (circa 500ms as taken from your profiles.). So to verify: are the edits you’re making entirely redundant, such as whitespace only changes? They probably are, in which case we’d need to do some experimenting to figure out if we can avoid that extra cost (we have tests that tell us this should work, but perhaps it’s missing a case).

i do everytime the same - editing string / number value. eg. return “george”; >> return “georgedcjwekwejkfk”; etc. 😃 I can do bigger changes, if you wish. But NO, not at all, my changes are NOT redundant.

Oh and one more thing that we noticed in the profile that deserves further research: in AOT mode the Angular compiler does template type-checking by feeding the Angular templates into TypeScript. This is done by injecting additional source files into a temporary TypeScript program. Creating that temporary program should be efficient as it’s mostly identical to the original. In your profile however, we’re seeing a codepath that indicates that TypeScript has determined that it can’t reuse everything from the original program, which incurs an extra cost (circa 500ms as taken from your profiles.). So to verify: are the edits you’re making entirely redundant, such as whitespace only changes? They probably are, in which case we’d need to do some experimenting to figure out if we can avoid that extra cost (we have tests that tell us this should work, but perhaps it’s missing a case).

Thanks, those are all JIT though. A single variant would do, they should be similar enough.

1. maybe it is bug that i see 0 bytes in rebuilt chunk size ? 😃

Lazy Chunk Files                                        | Names                                                |    Size      
components-src-lib-user-management-components-module.js | components-src-lib-user-management-components-module | 0 bytes  

1. Let me know when you got to Czech Republic, i owe you a beer

NOTES:

• in 11.0.9, disabling Sourcemaps means not big improvement in JIT mode, in 11.2.6 it means often 50%
• 11.2.6 improved ALL times speciallz AOT rebuild time in deep-shared component

Don’t worry too much about the TS update. I meant TS 4.1.x in general, so that would be the latest 4.1.5. You can use ~4.1.5 as version specifier to get the latest 4.1.x but not 4.2.x (TS does not follow semver). The ~4.2.3 dependency version is for the prereleases of Angular 12.

OK. Just let me know when everything is OUT. I will:

1. pull latest source code
2. ng update
3. rm package-lock
4. npm i
5. provide correct results with everything up to date
   (will attach more CPU profiles when anything not cool 😃 ) 😃

(fury refreshing 😃 )

Another observation is that you’re not yet on TypeScript 4.1. This affects source-map performance in AOT compilations, as there was a severe performance penalty for source mapping into external templates.

@montella1507 I’d be okay to keep it open until after the next release (likely today), so that you can check whether it’s back to normal then.

@montella1507 If you have some additional time, could you try the latest snapshot build?

npm install git+https://github.com/angular/ngtools-webpack-builds.git#a2c144fbeeef9c9584b7848acdb0d1f5b4d9e20b

You’ll need to do the same manual change to node modules again to force the @ngtools/webpack usage.

Angular CLI 11.2.5 will be released this week but feedback before the release would be helpful. Thank you again for all the profiling information.

Okey, just let me know if i can help you with anything. (remote desktop / more profiles / test / adding any CONSOLE.log() lol anywhere) 😃 We will keep 11.0.9 in main branch for a while, however you made big progress here in last day, TYVM guys.

Results - AOT - no circular: App.component.ts rebuild time 1st 11 994ms 2nd 10 900ms 3rd 9 000ms Thats very impressive for AOT i think anyway 🤟

Results - JIT - no circular:

App.component.ts rebuild time BEST 5 500ms now / 3 600ms on 11.0.9

Col.component.ts (deep shared component) BEST 6 200 ms now / 3700md on 11.0.9

User-role.component.ts (page component) BEST 4300 ms now / 1950ms on 11.0.9

Stock.component.ts BEST 4000ms now / 2100ms on 11.0.9

So i can confirm 3 things:

• it is like 6-8x faster than stock 11.2.5
• ± 2x slower than 11.0.9
• it works (no broken rebuild)

Here is the profile from JIT 11.2.5-next.4

vscode-profile-2021-03-11-16-41-25.zip

Unfortunately, the @ngtools/webpack package is a direct dependency of @angular-devkit/build-angular so it won’t be used if installed directly in the project. If using yarn, a resolutions addition in the project’s package.json would force the version used. For npm, you could try manually removing any instances of @ngtools/webpack from node_modules that are not the snapshot version (there should be one inside the node modules directory of @angular-devkit/build-angular).

The following is the output of npm ls @ngtools/webpack in a test project:

├─┬ @angular-devkit/build-angular@0.1200.0-next.3
│ └── @ngtools/webpack@12.0.0-next.3
└── @ngtools/webpack@12.0.0-next.4  (git+https://github.com/angular/ngtools-webpack-builds.git#dafcd5969c31e41f7a1893ebb83cc8ebe53cf6da)

Thank you will try soon (1-2 days - propably weekend). BTW are you willing to consider to disable circular-dependency-plugin as default? TBH it is cause of serious DX issues on medium - large projects. I tried to disable that on medium projects without dynamic cross references (in comments above) and it affects rebuild time too (30% in medium project, 80% on large with dynamic cross-references).

Or would you accept my PR about “rebuild-time performance” article in documentation?

Snapshot build is here: https://github.com/angular/ngtools-webpack-builds/tree/11.2.x (see top of readme)

Thanks for the additional information. I suspect that it’s just Webpack’s dependency graph that makes it believe that everything needs to be reemitted, which is introducing a huge bottleneck as emit is the slowest of all phases. The improvements to 11.2.5 are only concerning Angular’s involvement in the process, which used to be suboptimal, but given that your JIT recompiles were also slow means that this is not the relevant here (my initial comment about JIT mode being affected by the AOT compiler turned out to be inaccurate). So that leaves a new problem to be investigated with respect to the dependency graph that Webpack is using to drive recompiles.

Ok, thanks for the additional profile. I do indeed see a lot more emit activity than I’d expect, but I can’t tell why that would be the case.

Another observation is that the cyclic import check takes 17s, so you may want to disable that in devbuilds (there’s a setting in angular.json, not sure what it’s called from the top of my head—edit: it’s showCircularDependencies).

Ah, gotcha. You’re using JIT compilation and the new Ivy Webpack integration that shipped in 11.1 is incorrectly using the AOT compiler’s judgement on which files need to be emitted. But since AOT output is never requested, the AOT compiler will continue to indicate that everything needs to be emitted.

If you can enable AOT, then I suggest you do! Otherwise, I’d suggest to disable the new Ivy Webpack plugin in the CLI for now by setting environment variable NG_BUILD_IVY_LEGACY=1.

I’ll transfer this to the CLI repo for now.