sdk: Slow build with msbuild and dotnet cli

I’m seeing build times over a minute, where previously (project.json) I saw 20ish seconds worst case. A full build of my not-very-large at all project with ZERO changes takes nearly 20 seconds now. So that’s all overhead, and poking around it seems to all be due to ResolveAssemblyReferences/ResolveProjectReferences.

I think this is the sort of thing that can permanently hurt .net core’s reputation. I don’t understand why it was acceptable to replace project.json when important regressions like this are present.

The decision to switch away from the new project format, which had numerous advantages, was already a contentious one. The argument I heard was frequently that it was just XML vs json, and it wasn’t a huge deal.

It seems that in reality that much like when DNX went away, cross-platform developers have lost a lot and gained nothing with the shift back to csproj. If cross-platform (i.e. not using Visual Studio) development truly is important to MS, I think they need to consider the impact of these sorts of regressions more carefully, and with more weight.

In my opinion, the compile/run/test cycle is one of the most important to get right for developer happiness and productivity. Microsoft doesn’t seem to agree, when you consider deeds vs words.

@jeffhube and folks - performance is very top of mind at the moment, and we’re actively working on it, we’ve got https://github.com/dotnet/project-system/issues/2789 which we’re using as the uber issue.

We’re first focusing on evaluation/target performance - which affects all builds, then we’ll peel off and start digging into individual scenarios such as command-line and VS performance - which will be approached differently due to the way they do builds.

Some recent progress, for example, is: https://github.com/dotnet/standard/issues/442 and https://github.com/Microsoft/msbuild/issues/2392.

My concern is that 2.0 will be a HUGE adoption release and people will be surprised/disappointed when they see how bad the CLI performance is. I guess this will be a reason for many to step back once again.

For us, the great DNX performance was one reason to do the early switch to ASP.NET Core. I fully understand the reasons for all changes that happened since then but it would also have been great to see more performance-related effort for the 2.0 release.

We’re focusing on performance for the next couple of updates, you can see some of the things we’ve already tackled for 15.5/2.1 here: https://github.com/dotnet/project-system/issues/2789.

For Visual Studio 15.5/CLI 2.1, we’ve been mainly focused on improving evaluation time. Slow evaluation time is what what results in large build times for the ResolveProjectReferences/_GetProjectReferenceTargetFrameworkProperties targets, results in delays when adding/removing files from a project and results in UI delays when referencing .NET Standard/.NET Core projects from .NET Framework projects. We saw a 30% - 60% build time improvement for some large solutions from this work. Make note, small projects or projects without project references won’t see much improvement from this, that will be focused in Visual Studio 15.6/CLI 2.2.

For Visual Studio 15.6/CLI 2.2, we’ll be focusing on improving what we call the inner loop; basically all the things you do over and over again while developing an application. This includes, but is not limited to, looking at both up-to-date and out-of-date CLI builds/restores, the dreaded ResolveAssemblyReference target (which is significantly worse in .NET Core) and improving the startup time of the CLI.

Folks, we’re still making some pretty good progress - this change https://github.com/NuGet/NuGet.Client/pull/1866 significantly reduces the overhead of implicit restore. We saw up-to-date dotnet build for https://github.com/OrchardCMS/OrchardCore reduce by about 25%.

@dan-i-am yes the node_modules folder is excluded at every level, but at non-root level you are hitting a perf issue in msbuild: https://github.com/Microsoft/msbuild/issues/2000 (TL;DR msbuild scans the whole directory structure and then ignores it, but could do better since it knows it is going to ignore it).

A good workaround is to explicitly specify the full folder name that you are going to exclude by adding this to the csproj file:

  <PropertyGroup>
    <DefaultItemExcludes>ClientApp\node_modules\**;$(DefaultItemExcludes)</DefaultItemExcludes>
  </PropertyGroup>

Note that the path must be before $(DefaultItemExcludes) which is unlike most similar re-definitions seen in msbuild project files. (originating from https://github.com/Microsoft/msbuild/issues/2453#issuecomment-323466096)

I’m giving ASP.NET MVC Core a try and I personally use dotnet watch run. Every time I make the smallest change in code, for example a letter in a string somewhere, I hit save and it’s a minimum of 6s for the app to be up again.

It’s been a tedious development experience so far.

On the latest bits .net core 2.0 preview 2, mac os x (mac air), dotnet run times against a hello, world! console app take 4+ seconds to run, consistently. Roughly the same times on a Mac Pro i tried at work. I was assuming that since I didn’t change anything, the second run should have been blazing fast.

Here’s the output of dotnet --version and dotnet --info

As a comparison, I generated a hello, world! app with Rust, and ran run 3 times. First compile time was 2.1 seconds, and run times were consistently ~100ms. Run times for rust console app should be pretty close to the theoretical fastet time, since it’s compiled to native code. cargo build is incremental - if no files have changed, cargo build finishes in ~100ms.

I was expecting a similar behaviour from the .net runtime, although I would have expected a little overhead due to Rosyln / .NET CLR.

And as another comparison, I re-ran the dotnet core code, but did a build, publish, followed by 3 invocations of the compiled dll.

Running the dotnet dll’s is on par with machine code generated by Rust. But the build time was 6.5 seconds, and the publish time was 4.7 seconds.

Timing the dotnet core console app might not be a primary use case, but if the simplest possible dotnet core app is slow compiling, it might give the wrong impression.

Is someone looking at speeding this up? Having to wait seconds for an application to start that has been built before is poor user experience.

As a heads up, more improvements have been made in Preview 2 - https://blogs.msdn.microsoft.com/dotnet/2018/04/11/announcing-net-core-2-1-preview-2/.

@mikeharder I can’t share this code sadly (it’s not open source), however I’ve tried the 2.1.300 preview1 and it solved things 😃 The total build time is now ~22 seconds (retested) on the guest (VM), so I see the problem I had solved, as the upcoming release contains the fix. No further action required 😃 (On the host, the 2.1.300 preview1 takes 15 seconds, with 8 core parallel build). Nice progress! 😃

Incremental build times are improving using the 15.5 msbuild/cli version and the current latest bits (containing additional improvements). Using https://github.com/dasMulli/cli-incremental-perf-testbed, unscientific measurements for incremental builds (macOS 10.13.1, time dotnet build --no-restore) are:

run / version	2.0.3	2.1.3-preview-007232	2.2.0-preview1-007736
initial	1m17s	56s	41s
1st inremental	55s	36s	22s
2nd incremental	1m3s	37s	23s
3rd incremental	52s	34s	22s

@dasMulli I just did a quick test on a rather small repo of mine: https://github.com/c3-ls/ServiceFabric-Http

dotnet --version
1.0.4
(Measure-Command { git clean -fxd }).TotalSeconds
0.0301536
(Measure-Command { dotnet restore }).TotalSeconds # First restore
1.7914825
(Measure-Command { dotnet build }).TotalSeconds # First build
7.1661289
(Measure-Command { dotnet build }).TotalSeconds # build without any change
3.9407937
(Measure-Command { dotnet build }).TotalSeconds # build without any change
3.9514562
(Measure-Command { dotnet build }).TotalSeconds # build without any change
3.934135

dotnet --version
2.1.0-preview1-006984
(Measure-Command { git clean -fxd }).TotalSeconds
0.196593
(Measure-Command { dotnet build }).TotalSeconds # First build (includes restore)
10.1680267
(Measure-Command { dotnet build }).TotalSeconds # build without any change
7.5226035
(Measure-Command { dotnet build }).TotalSeconds # build without any change
6.8356463
(Measure-Command { dotnet build }).TotalSeconds # build without any change
6.5224566
(Measure-Command { dotnet build --no-restore }).TotalSeconds # build without any change (skip restore)
5.2879741
(Measure-Command { dotnet build --no-restore }).TotalSeconds # build without any change (skip restore)
5.7895082
(Measure-Command { dotnet build --no-restore }).TotalSeconds # build without any change (skip restore)
4.9253091

As you can see, the current daily CLI is a lot slower - even if you do the build without a restore.

I would like to add that this also really hurts the productivity with unit tests. We use VS Code on macOS, so it lacks all Visual Studio caching. Changing a test => run test, results in a compile time of 13 seconds, while only the unit test project should rebuild. Add to the 13 seconds, the Xunit test discovery and running the test, it’s a big regression.

I like MSBuild and I am familiar with the reasons why MSBuild is the foundation for building .Net Core now. But really, none of it matters if it’s this slow. Performance should have been on top the list in my opinion.

I really hope when 2.0.0 is out, performance will be as important as it is for WebKit:

We have a zero-tolerance policy for performance regressions. If a patch lands that regresses performance according to our benchmarks, then the person responsible must either back the patch out of the tree or drop everything immediately and fix the regression.

The way to make a program faster is to never let it get slower.

Edit: The 13 seconds build time is when I test a class library with only three project references. When I test an Mvc project, the build time is approaching 30 seconds, although nothing changed, except a unit test. These times are for building a 35K lines product on an iMac with Core i7 4Ghz.

@davidfowl This started in RC4 timeframe. Is this handled in 2.0?

Even hello world is brutally slow 😕

This is definitely something we want to improve. We develop cross-plat nearly 100% of our time ourselves, so we feel this pain. We have it in the books to go look into performance in general for the CLI.

@nguerrera ‘soon’, great… I’m not really happy I cannot even find a beta build of it anywhere for the other platforms. Many .net core users are not using the standard windows + vs stack which usually is why they’re a .net core user at all. Your logic to push these things first to VS is flawed and seems to again lack the willingness to keep the other platforms on the same level of quality and support. Sorry to say but people not using vs are hit the hardest by slow build times as they don’t have vs’s up-to-date checks etc, for us it’s rebuild all day everyday. We’d like to finally see some improvement.

@plaisted

You could create an MSBuild file for testing. Maybe this helps: https://github.com/Microsoft/vstest/issues/411#issuecomment-322383204

While it’s a lot faster for 2.0 due to the smaller package graphs 🎉 , it’s slow again once you add Microsoft.AspNetCore.All to some projects 😢

Dramatic improvement here. Excellent work 👍

@FransBouma: We haven’t seen significant perf differences between a Hyper-V VM and the host. My first guess would be a difference in disk perf. If you can share your solution, I can measure the perf on a physical machine and a Hyper-V VM to see how they compare.

You can also try a daily build of 2.1.300-preview1 which has many performance improvements to .NET Core build. I would expect the relative performance to be the same between the VM and host, but both should be significantly faster.

@livarcocc is someone working on this issue?