angular-cli: v7.0.0 bugs and 4/5x slower serve & compile AOT

Hi all,

Just wanted to give another update. With a lot of help from @istiti and @crisbeto, I think I’ve identified what causes compiler to be included and builds to take more memory, time and be bigger.

This is related to https://github.com/angular/angular/issues/25645. In Angular v7, there is a direct dependency from @angular/core to @angular/compiler.

This should not be a problem because treeshaking should eliminate that dependency as it is not used anymore in the final code. The most advanced treeshaking optimizations are enabled through the joint work of Webpack’s Module Concatenation, Terser/Uglify and Build Optimizer over a projects source code.

There are cases where module concatenation can be prevented, called bailouts. One such case is importing a module through require().

Specifically in this issue, whenever @angular/core is imported through require() anywhere in a project (including in dependencies), it will not be correctly optimized. Although this was still true of Angular v6, now with the new @angular/compiler import the consequences are greater and more code is retained.

More code being retained means bigger bundles, but also means that processing the bundles takes longer and needs more memory.

I’ve prepared two repos that show this:

• without the CLI just importing a module that uses require: https://github.com/filipesilva/cli-12646-custom-config
• with the CLI importing a library that uses require: https://github.com/filipesilva/cli-12646-cli-app

Now that we have identified the root cause of this problem, we are looking at solutions.

I really want to give a huge shoutout and thanks to both @istiti and @crisbeto. They helped a lot over the last few days by running tests, diagnostics and experiments on their code bases that lead to identifying the root cause.

Without that we still wouldn’t have a reproduction, and without a reproduction there is little hope of fixing problems. Personally I believe making reproductions is the most important part of open source, because once a reproduction is identified anyone can try to work out a solution. But without a reproduction there’s no hope of fixing truly a problem. So in cases where the reproduction is not trivial and projects in the wild are exhibiting issues, we really have to rely on help from those projects to figure out what’s wrong.

We discussed a plan yesterday with @mhevery that would allow us to cut the dependency from @angular/core to @angular/compiler by publishing a well-known symbol on window from the latter, and ensuring that either platform-browser-dynamic or maybe a new @angular/core/jit module was loaded first so that the compiler is present before the first Ivy component bootstraps. We are working to prioritize that.

A private project I tried to upgrade did increase in bundle size due to the compiler it looks like being bundled into the production output. I am hoping to try to recreate this is in a isolated small repo. The project is fairly large 60,000+ lines of TS. Here are the images of the webpack bundle analyzer before upgrading to 7 and after.

https://github.com/angular/angular/pull/26734 will fix this from framework side. We are figuring out if that can go in the next 7.0.x release of Angular to resolve this issue next week.

Proposal:

• ng build --prod will check if @angular/compiler got retained in your optimized prod bundle
• if so, it prints something like "We detected a de-optimization affecting your production bundle size, run ng build --prod --verbose=optimizations
• This command uses information known to Webpack to find CJS imports that bring in @angular/core (could parse the stats file or add a new plugin after the graph is built)
• It prints something that helps the user take action to file an issue against the right library which caused the de-optimization

To be honest I expected terser to require less memory than uglify-es, because of existing bugs that were fixed.

Are you using the additional memory flag? You can use it on a npm script like this:

"ng-high-memory": "node --max_old_space_size=8000 ./node_modules/@angular/cli/bin/ng",

Increasing the memory limit is something you should expect to do at some point. Node processes have a default memory limit of about 1.7gigs. When a node process starts getting close to the memory limit it needs to spend more and more time doing garbage collection to free up memory, which in turn makes it run more slowly.

I wouldn’t be terribly surprised if the projects that are now need more memory were nearly at the maximum before, and different code structures ended up demanding more memory…

Also what can happen is that the inner module closures (the module concatenation thing I mentioned before) managed to get more code in a single module and thus require more memory to optimize, with potential better savings.

Same here, I too have a bunch of dependencies that have require() statements in them (e.g. https://github.com/tjoskar/ng-lazyload-image)…

@stevethemacguy I did this:

externals: {
    "@angular/compiler": JSON.stringify("@angular/compiler")
},

which essentially tells webpack to replace an import of “@angular/compiler” with a string. You might also use JSON.stringify("")

Filipesilva… Please accept some Angular love

I just came across your comment to fix that issue but it actually fixed way more than that.

Working on a huge project, I had to turn off source maps when building in prod mode (which was a shame because we’re reporting our errors to sentry and before we were able to point out where in our TS code the error occurred instead of compiled JS!).

But that’s not it. Recently, I got crashes when building prod so I also had to turn off optimization…

And build, serve, linting… Pretty much everything was slow. It’s waaaay faster now ❤️ 🎉 🔥

Thanks!

Same here.

Tested using the same versions described above.

Anyway, our project must be very close to that point where the build can either fail or succeed though. I’d say that it fails 90% of the cases vs 10% of successes.

The question is: is there anything (probably with the TerserPlugin) that can make our build jump the red line of default memory needs?

What has changed in v7 that can make this build cross that line?

Is it expected that we need to allocate more memory for our v7 prod builds?

Back to normal build times with the new CLI 7.0.4 and TypeScript.

“JavaScript heap out of memory” issue is still present in production builds though…

Just wanted to mention there was a regression in 7.1 (https://github.com/angular/angular-cli/issues/13102), and we have a fix on the way for it.

This issue was specifically about bundle size and performance, which we think are fixed, and I’ve noted this in the release notes for https://github.com/angular/angular-cli/releases/tag/v7.0.4

I think we should close this issue, thanks everyone! If there’s a memory leak, that’s a separate issue and should be opened separately.

@coryrylan you mentioned you were trying to get a repro, and I know that’s not easy starting at a large application so I wanted to give you an approach that might help:

• make a new git branch to keep debugging changes
• set "outputHashing": "none", and "sourceMap": true, in angular.json
• npm install -g source-map-explorer
• make a new npm script that runs build and source-map-explorer (like "test": "ng build --prod && source-map-explorer dist/latest-project/main.js",)
• try removing certain parts of the application and then running npm test, if @angular/compiler is still in the sourcemap then that part of the app is not relevant to the repro.

Starting from a full application, these are the parts I would try to remove in order:

• comment out lazy routes, as they are individual compilation units
• comment out non-lazy routes and delete their component imports
• comment out NgModule imports
• replace all templateUrl and template with template: '<div></div> to remove template compilation

This order should hopefully help eliminate big chunks of source code until the compiler stops appearing in the source-map-explorer visualization.

@filipesilva

Angular v6 and CLI v6 1.36 MB Angular v6 and CLI v7 1.36 MB Angular v7 and CLI v7 1.59 MB

So I’m assuming that means it’s an angular issue rather than cli.

@istiti did the size of you main bundle increase as well after the update? I’ve read a report that the angular compiler was being included in bundles which can cause longer builds (this is unintended).

Also cc @ocombe about the i18n error.

Also have slower live reloading with angular 7. Was way quicker with angular 6.

In my case everything was fine after ng update to v7. But once I updated devDependencies:

the production build time increased about 3 times:

The @angular/core@7.0.2 releaseincludes https://github.com/angular/angular/pull/26734 and I could verify with the repros in https://github.com/angular/angular-cli/issues/12646#issuecomment-432292721 that the compiler is not retained anymore. This should address compiler being retained and part of the longer build time.

On the TS side, Microsoft/TypeScript#28028 is merged but not yet released. We are waiting for it to be released in 3.1.x. When that is out, it should overall address longer build and rebuild times.

@filipesilva In typescript version 3.1 and more, was found a bug with performance regression. I think the problem is the same.

Issue: Microsoft/TypeScript#28025 PR: Microsoft/TypeScript#28028

Has anyone got a definite fix for this yet? I created a new Angular 7 project (version details below) and running any ng commands while in the app folder is taking a ridiculous about of time, for example, running “ng --version” outside the app folder takes about 5 seconds, running it INSIDE the app folder took around 10 minutes.

VERSION DETAILS (ng --version): Angular CLI: 7.2.1 Node: 8.11.3 OS: win32 x64 Angular: 7.2.0 … animations, common, compiler, compiler-cli, core, forms … language-service, platform-browser, platform-browser-dynamic … router

Package Version

@angular-devkit/architect 0.12.1 @angular-devkit/build-angular 0.12.1 @angular-devkit/build-optimizer 0.12.1 @angular-devkit/build-webpack 0.12.1 @angular-devkit/core 7.2.1 @angular-devkit/schematics 7.2.1 @angular/cli 7.2.1 @ngtools/webpack 7.2.1 @schematics/angular 7.2.1 @schematics/update 0.12.1 rxjs 6.3.3 typescript 3.2.2 webpack 4.23.1

UPDATE Through trial and error I have discovered my problem is something to do with the IDE I am using, VS Code. When it is closed I can run any ng command, i.e. --version or serve, and it runs as quickly as I expect. When VS Code is open, AND my angular apps folder is opened in it, it slows down again.

Currently debugging trying to pinpoint what VS Code is doing to slow it down.

@filipesilva getting ERROR in Internal Error: The structure of the program changed during codegen.. while creating build using ng build --prod. it is working fine for ng build or ng serve. while upgrading my application to angular 7. “@angular/cli”: “^7.1.0”, “@angular/compiler-cli”: “~7.1.0” “@angular/core”: “~7.1.0”

angular.json configuration “production”: { “optimization”: true, “outputHashing”: “all”, “sourceMap”: false, “extractCss”: true, “namedChunks”: false, “aot”: true, “extractLicenses”: false, “vendorChunk”: false, “buildOptimizer”: true, “fileReplacements”: [ { “replace”: “src/environments/environment.ts”, “with”: “src/environments/environment.prod.ts” } ] }

Here’s an example of a webpack process being very close to the Node heap threshold and a webpack + minifier upgrade just put it over the limit: https://github.com/webpack-contrib/uglifyjs-webpack-plugin/issues/188

I kinda assumed that the memory usage came hand in hand with the CPU consumption since as one gets closer to the memory limit more and more time is spent on garbage collection.

True, CPU and memory use are often correlated but minifiers (and probably most optimizing compilers in general) spend the majority of their time looking for optimization opportunities in the AST that may or may not exist. These AST searches don’t usually create much in the way of long lived heap objects.

Recent V8 versions reportedly have better GC algorithms, but there also could be regressions in some cases. So it would be worth testing different Node versions when encountering a memory issue.

Also keep in mind that one must take into account the aggregate memory consumed by angular-cli, webpack and the minifier. Because the minifier is the last to run it often is blamed for problems occurring in earlier phases of the build pipeline. Webpack had several memory problems related to chunks in the past as I recall.

with ng build --prod --vendor-chunk --progress false “portal”

Angular CLI: 7.0.4
Node: 11.1.0
OS: linux x64
Angular: 7.0.2
... animations, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... router, service-worker

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.10.4
@angular-devkit/build-angular     0.10.4
@angular-devkit/build-optimizer   0.10.4
@angular-devkit/build-webpack     0.10.4
@angular-devkit/core              7.0.4
@angular-devkit/schematics        7.0.4
@angular/cli                      7.0.4
@ngtools/webpack                  7.0.4
@schematics/angular               7.0.4
@schematics/update                0.10.4
rxjs                              6.3.3
typescript                        3.1.6
webpack                           4.19.1

Still JavaScript heap out of memory

I agree with @mko. We have a larger app which fails to rebuild when running using ng serve. I identified the problem that compiler.js is included in the bundle. I expected that this PR angular/angular#26734 will fix it but it’s merged and as of version 7.0.2 the issue still remains - compiler.js is included in the bundle. This blocks our upgrade to Angular 7.

As a note: we have noticed that production builds aren’t the only thing that suffer from this. With a sufficiently large application, ~this~ a JS Heap out of memory error (which was how I got to this issue page from #12645 ) happens 100% reliably with the application watcher. Increasing the heap limit as suggested in a few places definitely delays the error, but it looks like there might be a memory leak in the build->watch->detect change->rebuild loop.

It should be noted that it appears to happen mostly in the SourceMapDevToolPlugin step of the rebuild. Not sure how to provide a reproduction as the issue appears to be purely related to the size of the application and number of independent modules. We have multiple projects that this happens on and the smaller (~1650 modules) fails after more rebuilds than our larger (~3400 modules) project with equivalent memory settings.

@dhardtke Nice. Seems like a sensible workaround for now. Thanks!

@filipesilva I see require(“@angular/core”) in a number of my dependencies (mostly in the included UMD files). Is there something I should do about that? I added “node: false” to the Webpack config, but it had no effect. If a more general solution is on it’s way, then I can wait.

@dhardtke, How did you mark the compiler as external? I was able to exclude the compiler using the following, but the build fails:

externals: { ‘@angular/compiler’: ‘’, },

I’ve been having a look at @crisbeto’s reproduction. I’ve reduced it a bit further (https://github.com/filipesilva/cli-12646-custom-config) and think I have something that can help some configurations.

If you are using a custom webpack configuration, make sure you have turned off the node builtins (see https://github.com/filipesilva/cli-12646-custom-config/commit/c9f28d0632a5da114c57d3ff6667b870734b5622).

This is something we do on the CLI since v6. Having them on can prevent @angular/core from being concatenated via https://webpack.js.org/plugins/module-concatenation-plugin/.

Doing this prevents most of @angular/core and @angular/compiler from being retained. But not all. Still investigating why some is still retained.

This advice has no effect on CLI projects since they already have it turned off.

@filipesilva no problem will try and get that info together will be a bit later on though.