project-system: UpToDate "set of project items was changed more recently" infinite loop (DLL stays the same, LastItemsChangedAtUtc stays the same)

Visual Studio Version: 16.7.3

Summary: I have a very large solution and some projects can somehow get into this weird state where they get built over and over even when they don’t have any changes. For example, let’s say MyProject is stuck in this state. I build it and get this log message:

FastUpToDate: The set of project items was changed more recently (2020-10-02 13:01:00) than the earliest output 'xxx\obj\Debug\MyProject.dll' (2020-10-02 13:00:58), not up to date. (MyProject)

Then Visual Studio builds the project, but the build doesn’t actually end up doing anything, it only copies the already existing DLL that has the same 2020-10-02 13:00:58 timestamp. So next time I build it, the DLL timestamp is still the same, it’ll still be older than the timestamp 2020-10-02 13:01:00 that’s remembered by Visual Studio.

The log message comes from:

https://github.com/dotnet/project-system/blob/519340c5d37ab96aaddadb5eca93a5955831a869/src/Microsoft.VisualStudio.ProjectSystem.Managed/ProjectSystem/UpToDate/BuildUpToDateCheck.cs#L242

And by reading the code, I just think that this process is faulty. The LastItemsChangedAtUtc never gets updated, the project gets built, but the DLL is only copied, so the DLL timestamp never gets updated, and I get stuck in a loop. I don’t know exactly how it is that the LastItemsChangedAtUtc timestamp is 2 seconds after the timestamp of the DLL, but I guess it can happen in some way (maybe the items are loaded with some delay if the solution is big? maybe something caused some “reload”?). Anyway, there needs to be some circuit breaker to get out of this endless loop. Both timestamps stay the same forever, and the build just happens over and over. Maybe a solution would be that the LastItemsChangedAtUtc is somehow reset when the project starts building?

Steps to Reproduce: Unfortunately, I don’t have a reliable way of reproducing this, it doesn’t happen every time. On my big solution, it happens often, but for different projects each time, and I don’t exactly know what determines that. But it’s still a situation with those 2 timestamps that can happen, and there should be a way how to break out of the loop.

Some notes regarding reproducing this: I didn’t modify any code files, only opened a solution, clicked Build, and then watched how some projects get stuck with those timestamps. I have automated restore of NuGet packages enabled (maybe it’s somehow related to the strange LastItemsChangedAtUtc value).

Expected Behavior: If a project is not up-to-date, then gets built, it should be up-to-date from that point.

Actual Behavior: A project can get stuck not being up-to-date forever, which makes it (and all of it dependants) built forever.

User Impact: The builds are a lot slower on a very large solution (over 100 projects) if a project somewhere deep in the dependency tree gets stuck in this loop. This just destroys the development experience making developers wait a long time for every tiny change.

---

Could be related to #5794, #5968, #6129.