testfx: WinUI 3 Tests fail to run after upgrade to Windows App SDK 1.1
Description
After upgrading my test project to Windows App SDK 1.1 the execution of tests fails.
Steps to reproduce
I followed https://devblogs.microsoft.com/ifdef-windows/winui-desktop-unit-tests/
UITestMethod attributed tests, fail to run after I upgrade to Windows App SDK 1.1
with error message:
Exception thrown while executing test. If using extension of TestMethodAttribute then please contact vendor.
Error message:
UITestMethodAttribute.DispatcherQueue should not be null.
To use UITestMethodAttribute within a WinUI Desktop App, remember to set the static UITestMethodAttribute.DispatcherQueue during the test initialization.,
Stack trace:
at Microsoft.VisualStudio.TestTools.UnitTesting.AppContainer.UITestMethodAttribute.Execute(ITestMethod testMethod)
at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestMethodRunner.ExecuteTest(TestMethodInfo testMethodInfo)
The exact same project work with Microsoft.WindowsAppSDK 1.0.3.
I tried to add [assembly: WinUITestTarget(typeof(App))] but that doesn’t work either.
The DispatcherQueue should get set in the OnLaunched method of my App class:
UITestMethodAttribute.DispatcherQueue = m_window.DispatcherQueue;
Expected behavior
Test should get executed.
Actual behavior
Running tests fails with message above. No window ever pops up and the App class seems to be not created.
Environment
WinAppSDK 1.1.0 VS 17.2.3 Microsoft.TestPlatform.TestHost 17.2.0 MSTest.TestFramework 2.2.10 MSTest.TestAdapter 2.2.10
.csProj (click to expand)
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>WinExe</OutputType>
<TargetFramework>net6.0-windows10.0.22000.0</TargetFramework>
<TargetPlatformMinVersion>10.0.18363.0</TargetPlatformMinVersion>
<RootNamespace>Telani.UITests</RootNamespace>
<ApplicationManifest>app.manifest</ApplicationManifest>
<Platforms>x86;x64;arm64</Platforms>
<RuntimeIdentifiers>win10-x86;win10-x64;win10-arm64</RuntimeIdentifiers>
<EnablePreviewMsixTooling>true</EnablePreviewMsixTooling>
<ImplicitUsings>enable</ImplicitUsings>
<UseWinUI>true</UseWinUI>
</PropertyGroup>
<ItemGroup>
<ProjectCapability Include="TestContainer" />
</ItemGroup>
<ItemGroup>
<AppxManifest Include="Package.appxmanifest">
<SubType>Designer</SubType>
</AppxManifest>
</ItemGroup>
<ItemGroup>
<Content Include="Images\LockScreenLogo.scale-200.png" />
<Content Include="Images\SplashScreen.scale-200.png" />
<Content Include="Images\Square150x150Logo.scale-200.png" />
<Content Include="Images\Square44x44Logo.scale-200.png" />
<Content Include="Images\Square44x44Logo.targetsize-24_altform-unplated.png" />
<Content Include="Images\StoreLogo.png" />
<Content Include="Images\Wide310x150Logo.scale-200.png" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="Microsoft.WindowsAppSDK" Version="1.0.3" />
<Manifest Include="$(ApplicationManifest)" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="MSTest.TestAdapter">
<Version>2.2.10</Version>
</PackageReference>
<PackageReference Include="MSTest.TestFramework">
<Version>2.2.10</Version>
</PackageReference>
<PackageReference Include="Microsoft.TestPlatform.TestHost">
<Version>17.2.0</Version>
<ExcludeAssets>build</ExcludeAssets>
</PackageReference>
</ItemGroup>
</Project>
App.xaml.cs (click to expand)
using Microsoft.UI.Xaml;
using Microsoft.UI.Xaml.Controls;
using Microsoft.UI.Xaml.Controls.Primitives;
using Microsoft.UI.Xaml.Data;
using Microsoft.UI.Xaml.Input;
using Microsoft.UI.Xaml.Media;
using Microsoft.UI.Xaml.Navigation;
using Microsoft.UI.Xaml.Shapes;
using Microsoft.VisualStudio.TestTools.UnitTesting.AppContainer;
using System.Runtime.InteropServices.WindowsRuntime;
using Telani.UITests;
using Windows.ApplicationModel;
using Windows.ApplicationModel.Activation;
using Windows.Foundation;
using Windows.Foundation.Collections;
[assembly: System.Runtime.Versioning.SupportedOSPlatform("windows10.0.18363.0")]
//[assembly: WinUITestTarget(typeof(App))]
namespace Telani.UITests;
/// <summary>
/// Provides application-specific behavior to supplement the default Application class.
/// </summary>
public partial class App : Application
{
/// <summary>
/// Initializes the singleton application object. This is the first line of authored code
/// executed, and as such is the logical equivalent of main() or WinMain().
/// </summary>
public App()
{
this.InitializeComponent();
}
/// <summary>
/// Invoked when the application is launched normally by the end user. Other entry points
/// will be used such as when the application is launched to open a specific file.
/// </summary>
/// <param name="args">Details about the launch request and process.</param>
protected override void OnLaunched(Microsoft.UI.Xaml.LaunchActivatedEventArgs args)
{
Microsoft.VisualStudio.TestPlatform.TestExecutor.UnitTestClient.CreateDefaultUI();
m_window = new MainWindow();
// Ensure the current window is active
m_window.Activate();
UITestMethodAttribute.DispatcherQueue = m_window.DispatcherQueue;
// Replace back with e.Arguments when https://github.com/microsoft/microsoft-ui-xaml/issues/3368 is fixed
Microsoft.VisualStudio.TestPlatform.TestExecutor.UnitTestClient.Run(Environment.CommandLine);
}
private Window m_window;
}
Tests.cs (click to expand)
using Microsoft.UI.Xaml.Controls;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Microsoft.VisualStudio.TestTools.UnitTesting.AppContainer;
using System.Text;
namespace Telani.UITests;
[TestClass]
public class QuickTest
{
[TestMethod]
public void TestMethod1()
{
Assert.AreEqual(0, 0);
}
[UITestMethod]
public void TestMethod2()
{
var grid = new Grid();
Assert.AreEqual(0, grid.ActualWidth);
}
}
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 1
- Comments: 30 (13 by maintainers)
Commits related to this issue
- Making the tests actually run in a UI process w/ a dispatcher See https://github.com/microsoft/testfx/issues/1127 — committed to grork/StoryvoidDotNet by grork 2 years ago
- Consolidated ProjectHeads and supporting infrastructure (#360) * Moved project heads to common folder * Fixed broken reference to props file * Fix broken path references * Fixed broken file ... — committed to CommunityToolkit/Labs-Windows by Arlodotexe a year ago
- Consolidated ProjectHeads and supporting infrastructure (#360) * Moved project heads to common folder * Fixed broken reference to props file * Fix broken path references * Fixed broken file ... — committed to Martin1994/Labs-Windows by Arlodotexe a year ago
Just a small message to let you know that we have seen this ticket and we will handle as fast as we can (we are a little short staffed with holidays and some other high priority tasks we need to handle).
Hey there! We identified a root cause of this error and filled a Dev Community bug for a fix. We also identified some improvements on WinUI and MSTest sides and we also discussed a way to improve our testing matrix to reduce likelihood of such issue in the future.
Thanks again for your patience.
From what I can tell, MSTest.TestFramework still has bugs relevant to this (described below) but the problem that starts everything off does not originate with MSTest.TestAdapter. Its
ITestDiscoverer, MSTestDiscoverer, never finds the test container for the WinUI app (regardless of SDK version) and thus also finds none of its tests. Instead, it’s found by “test container discoverer executor://projectoutputcontainerdiscoverer/v1” which comes fromMicrosoft.VisualStudio.TestWindow.Client.TestContainer.ProjectOutputContainerDiscoverer. I cannot find that source for that type on GitHub or anywhere else so I’m guessing that it is an internal part of Visual Studio that isn’t public.I’m attaching a repro project. In addition to Debug and Release configurations, it also has Test10 and Test11 configurations. When using x86, those will expose the tests and swap between WindowsAppSDK 1.0.4 (Test10) and 1.1.3 (Test11). In any other configuration the SDK is 1.1.3. The PackageReference to the correct SDK for the configuration is included using conditional ItemGroup markup in the .csproj file. Because of this, Solution Explorer will not always show the correct value for the SDK package version that is currently being used. It doesn’t seem to expect that different versions of a nuget package will be conditionally included. But as long as you do a Rebuild after switching configurations before attempting to run the tests, the proper SDK package will be used.
The
Tests.MainWindowContentIsStackPaneltest method in the repro will still cause unexpected behavior (either a failure or a crash) in MSTest.TestFramework (testhost) when trying to run it using Test11 (SDK 1.1.3) so this is still an MSTest bug.If
[assembly:WinUITestTarget(typeof(MSTestWinAppSDK11BugRepro.App))]in App.xaml.cs in the repro is commented out, thenTests.MainWindowContentIsStackPanel(the test that uses[UITestAttribute]) will fail due to an exception with error message “UITestMethodAttribute.DispatcherQueue should not be null” despite the value being assigned properly inApp.OnLaunchedsince the test host never attempts to create an instance of the app.If
[assembly:WinUITestTarget(typeof(MSTestWinAppSDK11BugRepro.App))]in App.xaml.cs in the repro is not commented out, thenTests.MainWindowContentIsStackPanel(the test that uses[UITestAttribute]) will crash the test host due to a COMException “Class not registered (0x80040154 (REGDB_E_CLASSNOTREG))”, which aborts the active test run (thus not running any of the remaining tests). Specifically it results from "[Error] The active test run was aborted. Reason: Test host process crashed : Unhandled exception. System.TypeInitializationException: The type initializer for 'WinRT.ActivationFactory1' threw an exception." which occurs when an instance of the app is attempted to be launched since MSTest.TestFramework knows about the WinUI app due to theWinUITestTargetAttribute` being set.Ultimately,
Microsoft.VisualStudio.TestWindow.Client.TestContainer.ProjectOutputContainerDiscovererdetects and reports “IsAppContainer: False” when using SDK 1.1 but (correctly) detects and reports “IsAppContainer: True” when using SDK 1.0. Why its behavior differs and what all results from it is something I’d need to be able to see its source to tell. Some change to the WindowsAppSDK between 1.0 and 1.1 is ultimately the source of the problem. Since I was able to get UI tests to work using SDK 1.1 when using a customized version ofUITestMethodAttributethat runs the app as unpackaged as described in my earlier post, it’s probably not something unfixable. It’s just that for whatever reason, that discoverer’s behavior difference results in the Win32 API functionGetCurrentPackageFullNamefinding a package name using SDK 1.0 but not finding one using SDK 1.1.This was all tested using Windows 11 Pro 10.0.22622 Build 22622 with Visual Studio 2022 Community Version 17.3 Preview 5.0.
MSTestWinAppSDK11BugRepro.zip
Hopefully you can file an internal ticket about the
Microsoft.VisualStudio.TestWindow.Client.TestContainer.ProjectOutputContainerDiscovererproblem to bring it to the attention of the people who maintain that. If it is a problem with internal code in Visual Studio, perhaps there’s a chance it can be fixed before 17.3 (Stable) is released.@Evangelink, it’s still broken, it’s ok if I set app as packaged app, but my case I want to run in unpackaged mode.
Just a quick message to say that we (with
WinUIteam) are still investigating this issue.That essentially matches what I have, yes. I define both
<EnableMsixTooling>true</EnableMsixTooling>and
<EnablePreviewMsixTooling>true</EnablePreviewMsixTooling>for all configurations.
WindowsAppSDK 1.0 does not reference
EnableMsixToolingsince the feature was still in preview then so including it does nothing. WindowsAppSDK 1.1 addedEnableMsixTooling. SDK 1.1 checks forEnablePreviewMsixToolingorEnableMsixToolingand builds and runs correctly with either or both defined. I expect that in some future version WindowsAppSDK will probably deprecate and removeEnablePreviewMsixTooling. Since SDK 1.1 did not generate any warnings or errors when both are defined I included both for all configurations in caseEnablePreviewMsixToolingis removed in the future.I too can only get
Test11to work when<WindowsAppContainer>true</WindowsAppContainer>is included along with either or both ofEnablePreviewMsixToolingandEnableMsixTooling.I found that doing a Clean followed by Rebuild would display the correct behavior when switching to or from
Test11. I did not need to explicitly uninstall the app or restart VS, but that just saves a bit of time.I have managed to reproduce the issue consistently! I have tweaked a little your csproj (see below) so that I can test the various combinations of
x86,x64,Test10andTest11.For Test10 (x86 and x64), the only way to have the test working (for me) was to add
<EnablePreviewMsixTooling>true</EnablePreviewMsixTooling>.For Test11 (x86 and x64), the only way to have the test working (for me) was to have either
or
Could you confirm this is what you have on your side?
Note that for
Test11, I had to uninstall app and restart VS between changes to be sure of what was really working or not. Simple update/rebuild after a successful run were often resulting in the new setup to also work while if I uninstall and restart VS it was not working anymore.I will start back discussion with WinUI team now that I have a consistent reproduceable scenario.
@Evangelink I do still encounter it when using MSTestWinAppSDK11BugRepro above with Test11|x86 configuration.
(Get-AppxPackage *windowsapprun*).PackageFullNameMicrosoft.WindowsAppRuntime.1.0_4.528.1755.0_x64__8wekyb3d8bbwe Microsoft.WindowsAppRuntime.1.0_4.528.1755.0_x86__8wekyb3d8bbwe Microsoft.WindowsAppRuntime.1.2-experimental1_2000.572.710.0_x86__8wekyb3d8bbwe Microsoft.WindowsAppRuntime.1.1_1005.616.1651.0_x64__8wekyb3d8bbwe Microsoft.WindowsAppRuntime.1.1_1005.616.1651.0_x86__8wekyb3d8bbweWhen using MSTestWinAppSDK11BugSolved above, which includes
<WindowsAppContainer>true</WindowsAppContainer>in the first<PropertyGroup>for the project, then the issue disappears. What I determined was that Microsoft.WindowsAppSDK versions 1.0.X defined this property but versions 1.1.X do not. MSTestDiscoverer seems to rely on that property being set to determine whether or not the project is an Appx container and thus whether it should run it using the test host code path that’s meant for WinUI or the default test host.I found that sometimes VSTest will not pickup changes in the repro after changing the solution configuration just by doing a Rebuild. Instead I needed to do a Clean first followed by a Build or Rebuild. I think it’s because of the conditional package inclusion in the .csproj that enables testing against WindowsAppSDK 1.0.X and 1.1.X versions.
Tested with Visual Studio 2022 Community 17.3.6 on Windows 11 Pro 10.0.22623 Build 22623.
Edit: Setting
<WindowsSdkPackageVersion>is not required. I ran a test after setting<WindowsAppContainer>and it seems that despite doing a rebuild of the project, Test Explorer was using some cached data. Unloading and reloading the project (and presumably closing and reopening the Solution) is required to get Test Explorer to clear its cache after you’ve added that property. I’m attaching a new version of the fixed repro that also includes more information about these three properties in its Readme.md file.I solved the problem. What happened was that WindowsAppSDK stopped defining certain MSBuild properties beginning in version 1.1 that it defined in 1.0. There are also certain properties that changed. In total, there are four properties that need to be added to your .csproj now rather than the one property that the article mentions. Once these are added, if you followed the rest of the directions in the article you will be able to use MSTest with Test Explorer in Visual Studio (the same as before). I’m attaching the fixed version of the repro project I posted earlier. But here are the key properties that need to be in a
<PropertyGroup>, it’s probably best to put them in the first one but I don’t think it matters:<EnablePreviewMsixTooling>true</EnablePreviewMsixTooling>(You probably have this already if you followed the article but I’m putting it here just in case.)<EnableMsixTooling>true</EnableMsixTooling>(The Msix Tooling is no longer in preview as of SDK 1.1 so this is the new property to use, but there is no harm in keeping the previous Preview version and it might even be needed for some corner cases.)<WindowsAppContainer>true</WindowsAppContainer>(WindowsAppSDK 1.0 defined this but it stopped defining it in 1.1+ so you need to define it now. There’s no harm in defining it in a 1.0 project.)The last one,<WindowsSdkPackageVersion>depends on what your<TargetFramework>property is set to. The value below assumes a project with<TargetFramework>net6.0-windows10.0.19041.0</TargetFramework><WindowsSdkPackageVersion>10.0.19041.26</WindowsSdkPackageVersion>The key to getting a correct value for that is that the build number (19041 here) should match the build number of your target framework. Also, the version number, including the revision number (26 here), must exactly match a Microsoft.Windows.SDK.NET.Ref version otherwise you will get an error. You do not need to add that package to your project. You will most likely get an error if you try. The reason for the link to NuGet is just because it is a convenient source of proper version numbers. Also, it’s recommended that you use the most recent revision but you can use earlier revisions if you have some reason to (e.g. version 10.0.19041.24).Making sure you have those
fourthree properties correctly defined in your project’s .csproj file solves the problem. I’m not aware of any side effects, though you may want to take care to keep the WindowsSdkPackageVersion up-to-date unless you need to maintain a specific version.The problem with the entire test run crashing if there’s a COMException when trying to start the app still should probably be fixed, but I think that’s an issue with VSTest. Either way, the above changes avoid that because the app will launch correctly and testing will work the same as was described in the original article!
(Note: This is the updated version mentioned in the edit note at the top.) MSTestWinAppSDK11BugSolved.zip