While the idea of AOT compilation is great, this issue is too much to ignore (if you ask me). Development times have skyrocketed for our team:

• ng serve (avg time)
  • w/o aot: 55s
  • w/ aot: 1m 45s
• ng build --prod (avg time)
  • w/o aot: 1m 26s
  • w/ aot: 3m 13s

This has to be a higher priority for the Angular team

In version 9, AOT and JIT compilations time should be similar.

Unfortunately, since you cannot provide a reproduction not even privately. I don’t see how we can investigate this any further.

optimization: false didn’t make much difference for me. Thought maybe that would help, shame.

I did some work on an Angular 8 project over the weekend that I hadn’t touched in a while, and I’d totally forgotten how quick the dev experience used to be. Conservatively it was, I dunno, like, 10x-20x faster? Really used to scream.

This needs to be a high priority, and it’s weird that it doesn’t appear to be (maybe it is already, I dunno!), given how much thought and care goes into so much of the tooling. The cli is great, stuff like ng update works amazingly well more often than not. But then you have stuff like ng serve builds just being slow as molasses now, and Angular 9 completely breaking the HMR boilerplate config that everyone used for years…it’s frustrating. I was very disappointed after making the (relatively painless) switch to 9 to find out that everything just got a LOT slower for development.

@alan-agius4 I was able to find a project that exhibits the same behavior. Could this issue be reopened?

Here’s the repo. Instructions are in the readme, but the short version is that an AOT build of the extra branch takes ~81s, whereas a JIT build takes ~32s.

upgrade to angular 9.1 will help so much sir. It’s will faster alot @vaindil @Serginho https://blog.angular.io/version-9-1-of-angular-now-available-typescript-3-8-faster-builds-and-more-eb292f989428

sass-loader seems to be the most time-consuming process! Is there any way to replace the sass-loader with the fast-sass-loader in an Angular project??

It’s me again. Angular 10.1.2 was released yesterday which contains several optimizations to improve the performance of AOT compilations. I did some tests with the ng9-aot-build-times repo (using the extra branch) with typescript@next (to benefit from microsoft/TypeScript#40055, which isn’t being backported to 4.0) and the overhead of AOT has decreases quite a bit.

The below table gives an overview of approximated times as reported by the CLI. It’s by no means a scientific measurement or anything and there’s quite some fluctuations across runs, but is should give an idea of the improvements.

^{* This the version I started measuring on, so taking that as the baseline.}

Overall, the AOT build now completes ~25s quicker than it did in the baseline test. Most of this improvement (~15s) is found in the TypeScript fix, which is unfortunately not yet available.

I am quite interested to learn about how these improvements work out on real projects. Please note that your mileage may vary depending on hardware and operating system.

I updated to Angular 10 and its even slower. Production Build times taking 5 - 6 minutes! Before angular 9 it never used to take so much time.

Angular 8

Angular 10

This is the most I can share from my project. It might give you insight into the scope and size.

It’s time for a change!

Any update on this issue?

I appreciate the feedback, but I am already on 9.1. The problem remains.

Another update:

I have done a deep-dive in the flame-graph to analyse differences between JIT and AOT compiles. One striking difference was found in TypeScript’s emit phase, where actual JavaScript output is written. In JIT, this was just 4s, whereas in AOT it was about 25s. This was surprising, as the Angular’s compiler impact on the emit phase should not be that large.

Upon further investigation, AOT compilations go through TypeScript’s ability to create source maps that map into external sources, namely the external template html files that Angular uses. This turned out to be a performance cliff responsible for approximately 16s of overhead. Without this bottleneck, emit time reduced to 9s which is a lot more reasonable. Note that these are times when running with a profiler, the real production improvement was about 11s.

I opened microsoft/TypeScript#40054 (issue) and microsoft/TypeScript#40055 (PR) to address the bottleneck in TypeScript itself. Note that the timings reported in microsoft/TypeScript#40054 are when using raw ngc, which experienced a lower overhead compared to the CLI.

Hello folks,

Over the past couple weeks I have done some profiling on the ng-speed-rebuild repo to get an insight in performance bottlenecks and experiment with changes that may improve the situation. This has led to #38418 which optimises the template type-checker, offering a significant improvement of template type-checking time. The particular repo showed a 5x improvement in just the time-checking time, from 35 seconds to 7 seconds, but your mileage may vary quite substantially depending on the number of templates and their complexity. I expect this change to land in 10.1.

I am fully aware that this won’t address the performance issue that many of you are facing when AOT is enabled. In particular, the details provided in #35906 have not been closely looked at; this is another area where performance gains could be achieved in some way.

Here’s the repo. Instructions are in the readme, but the short version is that an AOT build of the extra branch takes ~81s, whereas a JIT build takes ~32s.

@vaindil I only just discovered this repo. I ran #38418 on it and it reduces the type-checking time by ~7 seconds, going from 15.5s to 8.5s (roughly a 2x improvement, not nearly the 5x in the repo I mentioned above). This is a minor improvement in your case, and I see it’s not nearly approaching the JIT build time (for reference, JIT takes ~40s whereas AOT is 92s, reduced to ~85s with quite large variances between runs). The demo app provided in that repo appears to have 2845 components, which is an enormous number of components to compile. This works out to be ~16ms per component, so for typical compilations it’s not too much overhead compared to JIT compiles. Note that AOT compilation also does template type-checking, which is not done at all in JIT mode, and JIT mode would still require compilation in the browser (that would probably be quicker; although I don’t know what the JIT compile time per component would be). Looking at the profiling flame graph I do think there is still some room for improvement to be made, so your repo is an interesting candidate to experiment with optimisations. To be clear, though, don’t expect wonders in this scenario as the compilation unit is absolutely enormous.

Looks like the problem comes down to AOT builds being turned on for all builds by default with Angular 9. Disabling it for dev builds brings times roughly back to where they were before (just slightly slower, but acceptably so). I would normally expect this behavior, but the docs are written in such a way as to imply that AOT builds are faster than JIT builds with Ivy, which is clearly not the case (since there’s no difference in speed between a View Engine AOT build and an Ivy AOT build of this codebase).

Looks like the solution therefore is to disable AOT for dev builds. If that’s the answer, feel free to close this issue. I’ll leave this open though in case this is a bug.

In fact I’m currently observing increase of build time (both initial and incremental in watch mode) after upgrading to Angular 11.1, comparing to Angular 10.1.

I’m currently in a process of benchmarking and analyzing possible bottlenecks, so probably will file a separate issue once I have more results to share.

EDIT: https://github.com/angular/angular/issues/40635

Hello folks,

with the release of Angular 11.2.5 just now, a significant improvement to incremental rebuilds has landed. The initial builds have been improved in the course of this issue; with several improvements to template parsing, template-type checking, source-map generation etc. and I think we’ve reached a point where this issue is no longer actionable. I’m definitely interested in investigating performance bottlenecks if you have any, but from what I’ve seen in most cases the AOT compilation is no longer the primary contributor. Instead, things like SASS compiles, source-mapping and bundling/optimization passes are where most time is spent. Since these areas are outside of the scope of the Angular compiler itself, I believe this is not the right place to track those bottlenecks so I’ll close this issue.

@alan-agius4 I am facing the same issue as everyone else here. Angular 9,10,11 are all much slower than Angular 8 when running ng serve. I’d be happy to share the project offline.

For a simple app with lazy loaded modules, counting up to roughly 7 screens for whole app, the production build takes 7-8 minutes to complete. Each time you need to push to the staging app and test the changes, you are completely stalled for 10 mins, waiting for build to finish before testing. Don’t know what is the state of other competitor frameworks (React, Vue, Ember etc.) for build performance, but it is becoming frustrating to lose 1-2 hours per day waiting for the build to finish. Angular team should put priority to this issue above all others.

As much as I love speedy prod builds, this thread is primarily about dev builds and it’d be nice to keep any discussion on that topic

Same problem. Angular cli is extremely slow in version 9.0.1.

Command: ng serve --host 0.0.0.0

Step1, the cli prints:

WARNING: This is a simple server for use in testing or debugging Angular applications
locally. It hasn't been reviewed for security issues.

Binding this server to an open connection can result in compromising your application or
computer. Using a different host than the one passed to the "--host" flag might result in
websocket connection issues. You might need to use "--disableHostCheck" if that's the
case.
0% compiling

This step take at least 4 minutes in my Macbook PRO 13 (2017)

Step2: the percent starts to increase. This step takes 1 minute.

Total time: 5 minutes. It’s insane.

PD: I’m using ngcc in yarn postinstall so dependencies are compiled.

Looks like fast-sass-loader is un-maintained, doesn’t support source-maps, and only supports node-sass which support is deprecated in Angular CLI, therefore it cannot be considered as a valid alternative to sass-loader.

If you are noticing some performance regressions with sass-loader, I suggest that you create a reproduction and file an issue in their tracker.

Any chance of an update on this issue? We just upgraded to Angular 10 and builds (even JIT) take ~20% longer for the same codebase than they did in 9.

I’ve just upgraded an angular 8 app to 11 and the build is sooo much slower. After all I’ve read about Ivy being faster to build I have to say I’m very disappointed.

Updated to that version and is a lot better, not great cause our project is pretty big. But a lot better than before.

Folks, let’s take this strategically. This is serious issue in our work flow. It takes our life-time ale energy on CI servers. Is there anyone who really understands the issue and the code that is responsible for this? I understand that support open source for free is not for everyone. Let’s consider fact, that we would literally buy this functionality (paid over Patreon or whatever for certain price). If you are the person you can handle this, just say the price and watch if the comunity is willing to pay it. I offer $10. If there is more people like me, I trust, we will be able to afford it.

Sincerely, A person who don’t like to wait

I rerun the tests, but this time using cli instead of configuration, just to be 100% sure

Seems like you were correct about JIT and JIT w/o source maps, though I did it twice it was strange to me too that the difference is so small.

I get impression that the biggest part of building process in AOT is taken by sass-loader, at least the cli output gets stuck there the longest.

ngc & tsc in 10.1.4:

time ./node_modules/.bin/ngc -p src/tsconfig.app.json

real	0m22.773s
user	0m35.429s
sys	0m1.555s

time ./node_modules/.bin/tsc -p src/tsconfig.app.json

real	0m10.052s
user	0m16.953s
sys	0m0.686s

The build slowness is as reported - on my machine ngc takes ~4x the time that tsc does.

This is partly to be expected. The project in question is a worst-case scenario for this comparison, with ~750 TS files, ~700 of which are Angular components. This is not a “typical” Angular project structure - typical Angular projects tend to have more services, pipes, directives, NgModules, and business logic files.

Effectively, this means that ngc sees almost 2x the number of inputs as tsc: 750 TS files, plus 700 template files to compile. What’s more, templates are effectively compiled twice: once to generated component definitions for the compilation output, and once to TS code used internally for template type-checking. So the cost of compiling ~700 template files is amplified compared to the TS files in the program.

That said, I think ngc is still too slow here. One issue and area for improvement might be the cost of cycle detection on a project of this size. We should be able to do better.

ngc will never approach tsc’s performance on a project of this shape, but we should be able to improve on these numbers somewhat.

You can use the NG_BUILD_PROFILING=1 environment variable which outputs profile events for Chrome profiler and a JSON file with plugins stats.

@stagefright5 I would expect #41448 to not affect most applications, as it only occurs in specific node_modules structures. it is however an all or nothing improvement for incremental compilations; if you would have been affected by #41448 then you’d never have gotten incremental rebuilds, making them a lot slower. I don’t have any measurements here, but I’ve seen incremental rebuilds take ~1-2 seconds vs >10s for a full rebuild.

@doublemcz I think you should start at $100. While this may seem strange, in the end, it could be an interesting idea if it was conceived and eventually implemented as an additional mechanism for speeding up development in certain areas. The main problem, however, would probably be who would be the guarantor.

Hello folks, with the release of Angular 11.1 we now support TS 4.1 which contains the performance improvement for AOT builds with source-maps enabled. Larger projects may see a build time reduction of several seconds, if source maps are enabled.

I am using AOT and I checked the build when building a second time and I am getting almost identical timings for a prod build and ng serve. It’s around a minute. I will assume that that is expected timing?

Same issue here after upgrading to Angular 9.0.2.

It does help a lot when changing optimization to false in angular.json

          "builder": "@angular-devkit/build-angular:dev-server",
          "options": {
            ...
            "sourceMap": true,
            "optimization": false
            ...
          }
        },

For those who want to disable AOT meanwhile go to angular.json and change there "projects": { "architect": { "build": { "options": { "aot": false, ....} }}}

@vaindil, would you be able to share the application privately? unfortunately, it will be hard to track down the issue without a reproduction.

Also can you use try Node 12 instead of 13?

@turnerguo those differences look insignificant to me, apart from yarn install but we don’t have a lot of opportunities to improve there and it’s unrelated to AOT vs JIT. This issue is/was primarily about the yarn build step, which as you can see is the same (which is to be expected in this case, as the AOT improvements only matter for large projects)

I think as an ecosystem we need to compare ourselves to our competitors (so to say) I know it’s very hard to measure but how long would it take to compile a react app of the same size and complexity?

I tried one my app in Ng 11 with experimental support of Webpack 5. That caching helps speed-up process from 120s to 68s in this case.

@rodolfojnn Your proposal should be probably raised here: https://github.com/angular/angular-cli

@vsavovski The ngc vs tsc results look great when compared to the 10.0.8 run, as AOT compiles using ngc went from 61s to just 35s, saving 26s! This means that the overhead has decreased from 44s to just 18s, or relatively speaking from 258% to just 105% compared to tsc. It is unfortunate that a full build still takes a lot of time, but it does looks like it’s not the AOT compiler anymore.

@vsavovski The timing difference in the last two columns is the most interesting to me, as that indicates that AOT overhead is only 12s. I am wondering if that is correct, given that the overhead in ngc vs tsc using 10.0.8 was 44s (61s - 17s). I sure hope it is correct, as that means that the performance improvements have really paid off here. To be certain, you may want to compare ngc vs tsc using 10.1.4 like you did earlier (with source maps disabled in the tsconfig as before).

Also, is it really the case that the JIT column went from 256s to just 150s by just upgrading to 10.1.4? That looks incorrect to me, but I might be comparing numbers which shouldn’t be compared. The same is true for the Ivy column, which is also significantly lower than before.

Excellent, thank you! That is some serious hardware and I’d expected better results. The run with just ngc will probably have produced source maps, though, as ngc only takes its configuration from the tsconfig file. You may want to set "sourceMap": false under "compilerOptions" in the tsconfig and run the ngc test again. Additionally, you can get an insight into the overhead of the Angular compilation by comparing it with the raw TypeScript compiler using time ./node_modules/.bin/tsc -p src/tsconfig.app.json. I’d recommend to disable source maps in both instances as they will have quite a significant impact when using ngc (see details in https://github.com/angular/angular/issues/37293#issuecomment-674279162)

@vsavovski Thanks for posting these numbers. I have a couple questions:

1. Is the Ivy measurement with AOT enabled (and therefore the AOT labeled column indicating a View Engine build?)
2. Do you have source maps enabled? Source maps and AOT have a large overhead which has been fixed in the upcoming TypeScript 4.1, but that’s not yet available unfortunately.
3. What is roughly the number of TypeScript files/Angular components/directives/services in your compilation?
4. Are these numbers development builds, or optimized production builds?

Additionally, could you try updating to latest Angular (10.1.3 as of right now) as there have been a few performance improvements to the Ivy AOT compiler.

Is the library CommonJS or ES import?

I meant I found a library that also exhibits the issue when it is built, not included. This is the repo (linked in the OP).

I followed the same instructions on a project that went from a 3second build to a 40 second build and also got stuck at 71% chunk graph with a zero byte chrome-profiler-events.json. I will be comparing my application configuration with vanilla ng app.

For what it’s worth, I gave Node like 12GB of memory and just left the build there for 45 minutes or so and it eventually finished. I tried to interpret the profiler results but couldn’t get anywhere with it, which is why I just tried to find a library that could reproduce the problem (and eventually did).

The root issue for me came down to AOT being on by default for dev builds in Angular 9, despite the documentation saying their build times should be the same as non-AOT builds. If you change "aot": true to false for your dev configuration in angular.json, your dev build times should go right back down more or less to where they were in 8.

Same issue for me. With Angular 9, things got worse in my project builds…My CI jobs take forever. ngc in CI takes 5 mins and the remaining part of build takes 2-3 mins. Partially because of my cpu/mem constrained CI runner but it used to better before. I enabled all caching options to avoid running ngc, but still it’s not good enough. Turning off the optimization is not an option in production builds. This became huge blocker for me to choose Angular for any new projects.

Transferring to the FW repo for a better triage, as this is more of a compiler issue.

 tsc --project demo/src/tsconfig.json
$ /workspace/ng9-aot-build-times/node_modules/.bin/tsc --project demo/src/tsconfig.json
Done in 16.32s.

 ngc --project demo/src/tsconfig.json
$ /workspace/ng9-aot-build-times/node_modules/.bin/ngc --project demo/src/tsconfig.json
Done in 99.39s.

I see this issue too, and also while you are watching for changes, recompiles take much longer with aot than with jit

Mine also get’s stuck on the 71% chunk graph