wpf: Building WPF projects is not deterministic with dotnet.exe

So, today I had some time to look into it again and I might have finally found the cause of the different output.

It seemed that my first intuition was not that wrong. While it is correct that the Dictionary class keeps track of the insert order, the WPF code often uses its non-generic counterpart HashTable (and HybridDictionary), which do not keep track of the insert order, which results in different entry order between multiple runs when iterating over it.

After doing some debugging I came to the XmlnsCache class, which is more or less responsible for mapping XAML namespaces to assemblies.

The problematic code is where the reference assemblies are being scanned:

private void AddReferencedAssemblies()
{
    if (_assemblyPathTable == null || _assemblyPathTable.Count == 0)
    {
        return;
    }
    List<Assembly> interestingAssemblies = new List<Assembly>();
    // Load all the assemblies into a list.
    foreach(string assemblyName in _assemblyPathTable.Keys) // <-- here
    {
        bool hasCacheInfo = true;
        Assembly assy;
        if (_assemblyHasCacheInfo[assemblyName] != null)
        {
           hasCacheInfo = (bool)_assemblyHasCacheInfo[assemblyName];
        }
        ...

(_assemblyPathTable is here a HybridDictionary)

After patching the foreach loop using a simple OrderBy, I was no longer able to reproduce the issue.

I hope I made no mistake. Maybe someone could verify this by cherry picking this small commit: https://github.com/MichaeIDietrich/wpf/commit/cff984aaf851ba126a822190bf815d0bf5c4d40e

Here again the steps to reproduce, since the steps from my first comment used different output paths which seem to have also an effect on the output for some reason:

1. dotnet new wpf --name deterministic_bug
2. cd deterministic_bug
3. dotnet build
4. Move obj and bin folder to some location outside the project (project is now clean again)
5. dotnet build

Compare the output of both builds.

To test against a local build of the PresentationBuildTasks.dll, one can simply add this line to the csproj file: <_PresentationBuildTasksAssembly>path\to\PresentationBuildTasks.dll</_PresentationBuildTasksAssembly>

The AssemblyReference.cache file is still different, but it does not seem to have a negative effect here.

@MichaeIDietrich: Thank you for this. I’ve confirmed your findings. The hash produced is indeed different, and your solution fixes the incremental build issue on WPF for .NET Core. As is, the first comparison against the hashed Reference list is failing every time, causing WPF to always compile.

https://github.com/dotnet/wpf/blob/master/src/Microsoft.DotNet.Wpf/src/PresentationBuildTasks/MS/Internal/Tasks/IncrementalCompileAnalyzer.cs#L98

PresentationBuildTasks.dll!MS.Internal.Tasks.IncrementalCompileAnalyzer.AnalyzeInputFiles() Line 100 C# PresentationBuildTasks.dll!Microsoft.Build.Tasks.Windows.MarkupCompilePass1.AnalyzeInputsAndSetting() Line 1110 C# PresentationBuildTasks.dll!Microsoft.Build.Tasks.Windows.MarkupCompilePass1.Execute() Line 155 C#

https://github.com/dotnet/wpf/blob/master/src/Microsoft.DotNet.Wpf/src/PresentationBuildTasks/Microsoft/Build/Tasks/Windows/MarkupCompilePass1.cs#L1094

https://github.com/dotnet/wpf/blob/master/src/Microsoft.DotNet.Wpf/src/PresentationBuildTasks/MS/Internal/Tasks/CompilerState.cs#L218

Two possible options for next steps are:

1. You can submit a PR with this change and we can review and merge it ASAP.
2. Someone from WPF can submit a fix and we will add you as the coauthor.

Thanks again for the analysis and fix. This is an extremely impactful contribution.

(This will go in to .NET 6.0, but we also want to include this in the next servicing releases for 5.0 and 3.1.)

/cc @dotnet/wpf-developers

Dictionary order in .NET Framework (and I believe .NET Core too), if memory serves, is based on insertion order rather than a function of the key.

OK, the bug still reproduces with .NET 5.0.102.

After digging deeper into the issue with the WPF compiler cache, I realised that the reason for this issue is most probably a difference in the behavior of .NET Core compared to .NET Framework. As I stated in my initial comment, the cache uses GetHashCode() on strings to calculated cache values. What I was not aware of until recently is that string.GetHashCode() is in fact not stable on .NET Core for subsequent program runs and in that way is not meant to be persisted.

The hash code itself is not guaranteed to be stable. Hash codes for identical strings can differ across .NET implementations, across .NET versions, and across .NET platforms (such as 32-bit and 64-bit) for a single version of .NET. In some cases, they can even differ by application domain. This implies that two subsequent runs of the same program may return different hash codes.

As a result, hash codes should never be used outside of the application domain in which they were created, they should never be used as key fields in a collection, and they should never be persisted.

source: documentation (see Remarks)

This behavior is intentional for security reasons, so we might simply switch to a custom implementation to generate the hashes for the cache. The easiest approach might probably to just copy the hash code generation of .NET Framework.

Please have a look at this. Thank you in advance.

OK, I have good news and bad news.

Good news first, changing the hashing algorithm fixes indeed the incremental build for WPF projects. I will make a PR for this change.

The bad news are that as expected this does not fix the issue with the non-deterministic output DLLs.

Comparing the obj-folder including the fix still leaves some files with different content:

A difference in the order of assembly references can be found in all the 4 files: (diff of both .resources files)

If I could make a guess, I could imagine that this also might come from string hashing and that there is some code that for-eaches over the values of a dictionary that contains assembly references where a string is used as key, so that the order varies depending on the hash value of the string. But I might be wrong, maybe someone with knowledge about these files could have a look at this.

Hi @ryalanms

I cloned the wpf repo yesterday, but I had trouble making the whole solution to compile, to test the fix locally.

While this might fix the incremental build for wpf projects, I’m currently not quite sure, whether this also fixes my initial problem with the non-deterministic output DLLs, if dotnet.exe is used instead of msbuild.exe, since the repro steps I posted should usually be independent from incremental build.

But I will give it another try tomorrow and will report back about the result.