angular: AOT Dev builds in Angular v9+ are much slower than JIT
š Bug report
Command (mark with an x
)
- new
- build
- serve
- test
- e2e
- generate
- add
- update
- lint
- xi18n
- run
- config
- help
- version
- doc
Is this a regression?
Yes, not present in 8.2.
Description
Dev builds in Angular 9 are very slow. Builds of 8.2 took on average 30 seconds or so, and recompiles took 2-5 seconds. This same app in Angular 9, however, takes ~45-60 seconds to build initially, and recompiles take ~15-25 seconds even if nothing changes at all. Production builds are roughly 5-10 seconds faster in 9, so those are better at least. (Note that none of these times include ngcc
, thatās taken care of before I attempt to build.) Additionally, the speed is slow regardless of whether Ivy is used or not.
The Angular 9 build always seems to freeze for a bit at a certain point. It frequently lists the same file, but itās not always the same file that it seems to get stuck on. Thereās nothing special about the file it usually lists, itās actually quite small and has no advanced Angular or TypeScript features.
I found #34699 (and #33532 by extension), but those are unrelated. Iām using TS 3.8 and it happens regardless of the value of that option in tsconfig.json
.
Is there a way I can audit whatās causing it to take so long? I tried ng build --watch --verbose
, but it provides way more information than I can feasibly parse (and also doesnāt include timings from what I can tell).
š¬ Minimal Reproduction
This repo has a reproduction. Instructions are in the readme.
š„ Exception or Error
N/A
š Your Environment
Angular CLI: 9.1.3
Node: 13.12.0
OS: win32 x64
Angular: 9.1.3
... animations, cli, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... router
Ivy Workspace: Yes
Package Version
------------------------------------------------------------
@angular-devkit/architect 0.901.3
@angular-devkit/build-angular 0.901.3
@angular-devkit/build-ng-packagr 0.901.3
@angular-devkit/build-optimizer 0.901.3
@angular-devkit/build-webpack 0.901.3
@angular-devkit/core 9.1.3
@angular-devkit/schematics 9.1.3
@ngtools/webpack 9.1.3
@schematics/angular 9.1.3
@schematics/update 0.901.3
ng-packagr 9.0.3
rxjs 6.5.4
typescript 3.8.3
webpack 4.42.0
Anything else relevant? N/A
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 123
- Comments: 90 (32 by maintainers)
While the idea of AOT compilation is great, this issue is too much to ignore (if you ask me). Development times have skyrocketed for our team:
This has to be a higher priority for the Angular team
In version 9, AOT and JIT compilations time should be similar.
Unfortunately, since you cannot provide a reproduction not even privately. I donāt see how we can investigate this any further.
optimization: false
didnāt make much difference for me. Thought maybe that would help, shame.I did some work on an Angular 8 project over the weekend that I hadnāt touched in a while, and Iād totally forgotten how quick the dev experience used to be. Conservatively it was, I dunno, like, 10x-20x faster? Really used to scream.
This needs to be a high priority, and itās weird that it doesnāt appear to be (maybe it is already, I dunno!), given how much thought and care goes into so much of the tooling. The cli is great, stuff like
ng update
works amazingly well more often than not. But then you have stuff likeng serve
builds just being slow as molasses now, and Angular 9 completely breaking the HMR boilerplate config that everyone used for yearsā¦itās frustrating. I was very disappointed after making the (relatively painless) switch to 9 to find out that everything just got a LOT slower for development.@alan-agius4 I was able to find a project that exhibits the same behavior. Could this issue be reopened?
Hereās the repo. Instructions are in the readme, but the short version is that an AOT build of the
extra
branch takes ~81s, whereas a JIT build takes ~32s.upgrade to
angular 9.1
will help so much sir. Itās will faster alot @vaindil @Serginho https://blog.angular.io/version-9-1-of-angular-now-available-typescript-3-8-faster-builds-and-more-eb292f989428sass-loader seems to be the most time-consuming process! Is there any way to replace the sass-loader with the fast-sass-loader in an Angular project??
Itās me again. Angular 10.1.2 was released yesterday which contains several optimizations to improve the performance of AOT compilations. I did some tests with the ng9-aot-build-times repo (using the
extra
branch) withtypescript@next
(to benefit from microsoft/TypeScript#40055, which isnāt being backported to 4.0) and the overhead of AOT has decreases quite a bit.The below table gives an overview of approximated times as reported by the CLI. Itās by no means a scientific measurement or anything and thereās quite some fluctuations across runs, but is should give an idea of the improvements.
* This the version I started measuring on, so taking that as the baseline.
Overall, the AOT build now completes ~25s quicker than it did in the baseline test. Most of this improvement (~15s) is found in the TypeScript fix, which is unfortunately not yet available.
I am quite interested to learn about how these improvements work out on real projects. Please note that your mileage may vary depending on hardware and operating system.
I updated to Angular 10 and its even slower. Production Build times taking 5 - 6 minutes! Before angular 9 it never used to take so much time.
Angular 8
Angular 10
This is the most I can share from my project. It might give you insight into the scope and size.
Itās time for a change!
Any update on this issue?
I appreciate the feedback, but I am already on 9.1. The problem remains.
Another update:
I have done a deep-dive in the flame-graph to analyse differences between JIT and AOT compiles. One striking difference was found in TypeScriptās emit phase, where actual JavaScript output is written. In JIT, this was just 4s, whereas in AOT it was about 25s. This was surprising, as the Angularās compiler impact on the emit phase should not be that large.
Upon further investigation, AOT compilations go through TypeScriptās ability to create source maps that map into external sources, namely the external template html files that Angular uses. This turned out to be a performance cliff responsible for approximately 16s of overhead. Without this bottleneck, emit time reduced to 9s which is a lot more reasonable. Note that these are times when running with a profiler, the real production improvement was about 11s.
I opened microsoft/TypeScript#40054 (issue) and microsoft/TypeScript#40055 (PR) to address the bottleneck in TypeScript itself. Note that the timings reported in microsoft/TypeScript#40054 are when using raw
ngc
, which experienced a lower overhead compared to the CLI.Hello folks,
Over the past couple weeks I have done some profiling on the ng-speed-rebuild repo to get an insight in performance bottlenecks and experiment with changes that may improve the situation. This has led to #38418 which optimises the template type-checker, offering a significant improvement of template type-checking time. The particular repo showed a 5x improvement in just the time-checking time, from 35 seconds to 7 seconds, but your mileage may vary quite substantially depending on the number of templates and their complexity. I expect this change to land in 10.1.
I am fully aware that this wonāt address the performance issue that many of you are facing when AOT is enabled. In particular, the details provided in #35906 have not been closely looked at; this is another area where performance gains could be achieved in some way.
@vaindil I only just discovered this repo. I ran #38418 on it and it reduces the type-checking time by ~7 seconds, going from 15.5s to 8.5s (roughly a 2x improvement, not nearly the 5x in the repo I mentioned above). This is a minor improvement in your case, and I see itās not nearly approaching the JIT build time (for reference, JIT takes ~40s whereas AOT is 92s, reduced to ~85s with quite large variances between runs). The demo app provided in that repo appears to have 2845 components, which is an enormous number of components to compile. This works out to be ~16ms per component, so for typical compilations itās not too much overhead compared to JIT compiles. Note that AOT compilation also does template type-checking, which is not done at all in JIT mode, and JIT mode would still require compilation in the browser (that would probably be quicker; although I donāt know what the JIT compile time per component would be). Looking at the profiling flame graph I do think there is still some room for improvement to be made, so your repo is an interesting candidate to experiment with optimisations. To be clear, though, donāt expect wonders in this scenario as the compilation unit is absolutely enormous.
Looks like the problem comes down to AOT builds being turned on for all builds by default with Angular 9. Disabling it for dev builds brings times roughly back to where they were before (just slightly slower, but acceptably so). I would normally expect this behavior, but the docs are written in such a way as to imply that AOT builds are faster than JIT builds with Ivy, which is clearly not the case (since thereās no difference in speed between a View Engine AOT build and an Ivy AOT build of this codebase).
Looks like the solution therefore is to disable AOT for dev builds. If thatās the answer, feel free to close this issue. Iāll leave this open though in case this is a bug.
In fact Iām currently observing increase of build time (both initial and incremental in watch mode) after upgrading to Angular 11.1, comparing to Angular 10.1.
Iām currently in a process of benchmarking and analyzing possible bottlenecks, so probably will file a separate issue once I have more results to share.
EDIT: https://github.com/angular/angular/issues/40635
Hello folks,
with the release of Angular 11.2.5 just now, a significant improvement to incremental rebuilds has landed. The initial builds have been improved in the course of this issue; with several improvements to template parsing, template-type checking, source-map generation etc. and I think weāve reached a point where this issue is no longer actionable. Iām definitely interested in investigating performance bottlenecks if you have any, but from what Iāve seen in most cases the AOT compilation is no longer the primary contributor. Instead, things like SASS compiles, source-mapping and bundling/optimization passes are where most time is spent. Since these areas are outside of the scope of the Angular compiler itself, I believe this is not the right place to track those bottlenecks so Iāll close this issue.
@alan-agius4 I am facing the same issue as everyone else here. Angular 9,10,11 are all much slower than Angular 8 when running ng serve. Iād be happy to share the project offline.
For a simple app with lazy loaded modules, counting up to roughly 7 screens for whole app, the production build takes 7-8 minutes to complete. Each time you need to push to the staging app and test the changes, you are completely stalled for 10 mins, waiting for build to finish before testing. Donāt know what is the state of other competitor frameworks (React, Vue, Ember etc.) for build performance, but it is becoming frustrating to lose 1-2 hours per day waiting for the build to finish. Angular team should put priority to this issue above all others.
As much as I love speedy prod builds, this thread is primarily about dev builds and itād be nice to keep any discussion on that topic
Same problem. Angular cli is extremely slow in version 9.0.1.
Command:
ng serve --host 0.0.0.0
Step1, the cli prints:
This step take at least 4 minutes in my Macbook PRO 13 (2017)
Step2: the percent starts to increase. This step takes 1 minute.
Total time: 5 minutes. Itās insane.
PD: Iām using
ngcc
in yarnpostinstall
so dependencies are compiled.Looks like
fast-sass-loader
is un-maintained, doesnāt support source-maps, and only supports node-sass which support is deprecated in Angular CLI, therefore it cannot be considered as a valid alternative tosass-loader
.If you are noticing some performance regressions with sass-loader, I suggest that you create a reproduction and file an issue in their tracker.
Any chance of an update on this issue? We just upgraded to Angular 10 and builds (even JIT) take ~20% longer for the same codebase than they did in 9.
Iāve just upgraded an angular 8 app to 11 and the build is sooo much slower. After all Iāve read about Ivy being faster to build I have to say Iām very disappointed.
Updated to that version and is a lot better, not great cause our project is pretty big. But a lot better than before.
Folks, letās take this strategically. This is serious issue in our work flow. It takes our life-time ale energy on CI servers. Is there anyone who really understands the issue and the code that is responsible for this? I understand that support open source for free is not for everyone. Letās consider fact, that we would literally buy this functionality (paid over Patreon or whatever for certain price). If you are the person you can handle this, just say the price and watch if the comunity is willing to pay it. I offer $10. If there is more people like me, I trust, we will be able to afford it.
Sincerely, A person who donāt like to wait
I rerun the tests, but this time using cli instead of configuration, just to be 100% sure
Seems like you were correct about JIT and JIT w/o source maps, though I did it twice it was strange to me too that the difference is so small.
I get impression that the biggest part of building process in AOT is taken by
sass-loader
, at least the cli output gets stuck there the longest.ngc
&tsc
in 10.1.4:The build slowness is as reported - on my machine
ngc
takes ~4x the time thattsc
does.This is partly to be expected. The project in question is a worst-case scenario for this comparison, with ~750 TS files, ~700 of which are Angular components. This is not a ātypicalā Angular project structure - typical Angular projects tend to have more services, pipes, directives, NgModules, and business logic files.
Effectively, this means that ngc sees almost 2x the number of inputs as tsc: 750 TS files, plus 700 template files to compile. Whatās more, templates are effectively compiled twice: once to generated component definitions for the compilation output, and once to TS code used internally for template type-checking. So the cost of compiling ~700 template files is amplified compared to the TS files in the program.
That said, I think ngc is still too slow here. One issue and area for improvement might be the cost of cycle detection on a project of this size. We should be able to do better.
ngc will never approach tscās performance on a project of this shape, but we should be able to improve on these numbers somewhat.
You can use the
NG_BUILD_PROFILING=1
environment variable which outputs profile events for Chrome profiler and a JSON file with plugins stats.@stagefright5 I would expect #41448 to not affect most applications, as it only occurs in specific
node_modules
structures. it is however an all or nothing improvement for incremental compilations; if you would have been affected by #41448 then youād never have gotten incremental rebuilds, making them a lot slower. I donāt have any measurements here, but Iāve seen incremental rebuilds take ~1-2 seconds vs >10s for a full rebuild.@doublemcz I think you should start at $100. While this may seem strange, in the end, it could be an interesting idea if it was conceived and eventually implemented as an additional mechanism for speeding up development in certain areas. The main problem, however, would probably be who would be the guarantor.
Hello folks, with the release of Angular 11.1 we now support TS 4.1 which contains the performance improvement for AOT builds with source-maps enabled. Larger projects may see a build time reduction of several seconds, if source maps are enabled.
I am using AOT and I checked the build when building a second time and I am getting almost identical timings for a prod build and ng serve. Itās around a minute. I will assume that that is expected timing?
Same issue here after upgrading to Angular 9.0.2.
It does help a lot when changing optimization to false in angular.json
For those who want to disable AOT meanwhile go to
angular.json
and change there"projects": { "architect": { "build": { "options": { "aot": false, ....} }}}
@vaindil, would you be able to share the application privately? unfortunately, it will be hard to track down the issue without a reproduction.
Also can you use try Node 12 instead of 13?
@turnerguo those differences look insignificant to me, apart from
yarn install
but we donāt have a lot of opportunities to improve there and itās unrelated to AOT vs JIT. This issue is/was primarily about theyarn build
step, which as you can see is the same (which is to be expected in this case, as the AOT improvements only matter for large projects)I think as an ecosystem we need to compare ourselves to our competitors (so to say) I know itās very hard to measure but how long would it take to compile a react app of the same size and complexity?
I tried one my app in Ng 11 with experimental support of Webpack 5. That caching helps speed-up process from 120s to 68s in this case.
@rodolfojnn Your proposal should be probably raised here: https://github.com/angular/angular-cli
@vsavovski The
ngc
vstsc
results look great when compared to the 10.0.8 run, as AOT compiles usingngc
went from 61s to just 35s, saving 26s! This means that the overhead has decreased from 44s to just 18s, or relatively speaking from 258% to just 105% compared totsc
. It is unfortunate that a full build still takes a lot of time, but it does looks like itās not the AOT compiler anymore.@vsavovski The timing difference in the last two columns is the most interesting to me, as that indicates that AOT overhead is only 12s. I am wondering if that is correct, given that the overhead in
ngc
vstsc
using 10.0.8 was 44s (61s - 17s). I sure hope it is correct, as that means that the performance improvements have really paid off here. To be certain, you may want to comparengc
vstsc
using10.1.4
like you did earlier (with source maps disabled in the tsconfig as before).Also, is it really the case that the JIT column went from 256s to just 150s by just upgrading to 10.1.4? That looks incorrect to me, but I might be comparing numbers which shouldnāt be compared. The same is true for the Ivy column, which is also significantly lower than before.
Excellent, thank you! That is some serious hardware and Iād expected better results. The run with just
ngc
will probably have produced source maps, though, asngc
only takes its configuration from the tsconfig file. You may want to set"sourceMap": false
under"compilerOptions"
in the tsconfig and run thengc
test again. Additionally, you can get an insight into the overhead of the Angular compilation by comparing it with the raw TypeScript compiler usingtime ./node_modules/.bin/tsc -p src/tsconfig.app.json
. Iād recommend to disable source maps in both instances as they will have quite a significant impact when usingngc
(see details in https://github.com/angular/angular/issues/37293#issuecomment-674279162)@vsavovski Thanks for posting these numbers. I have a couple questions:
Additionally, could you try updating to latest Angular (10.1.3 as of right now) as there have been a few performance improvements to the Ivy AOT compiler.
I meant I found a library that also exhibits the issue when it is built, not included. This is the repo (linked in the OP).
For what itās worth, I gave Node like 12GB of memory and just left the build there for 45 minutes or so and it eventually finished. I tried to interpret the profiler results but couldnāt get anywhere with it, which is why I just tried to find a library that could reproduce the problem (and eventually did).
The root issue for me came down to AOT being on by default for dev builds in Angular 9, despite the documentation saying their build times should be the same as non-AOT builds. If you change
"aot": true
tofalse
for your dev configuration inangular.json
, your dev build times should go right back down more or less to where they were in 8.Same issue for me. With Angular 9, things got worse in my project buildsā¦My CI jobs take forever. ngc in CI takes 5 mins and the remaining part of build takes 2-3 mins. Partially because of my cpu/mem constrained CI runner but it used to better before. I enabled all caching options to avoid running ngc, but still itās not good enough. Turning off the optimization is not an option in production builds. This became huge blocker for me to choose Angular for any new projects.
Transferring to the FW repo for a better triage, as this is more of a compiler issue.
I see this issue too, and also while you are watching for changes, recompiles take much longer with aot than with jit
Mine also getās stuck on the 71% chunk graph