bUnit: Deadlock when using FindComponent(s) with WaitForX

Today we started to see random hang-ups while running bUnit tests. After some debugging, it boiled down to these two threads mutually locking each other:

Worker thread @33878491
	Monitor.Enter() at /Users/rmihael/Library/Application Support/JetBrains/Rider2021.2/resharper-host/SourcesCache/3da0403ae6e540a7a79283e509f3945b8d7e00/18E/Monitor.cs:line 48
	TestRenderer.FindComponents<SolidCube.Components.Shared.TreeItem>() at /Users/rmihael/Library/Application Support/JetBrains/Rider2021.2/resharper-host/SourcesCache/8adb50b4d399477b881862854de9b0cb13c00/26/TestRenderer.cs:line 251
	TestRenderer.FindComponents<SolidCube.Components.Shared.TreeItem>() at /Users/rmihael/Library/Application Support/JetBrains/Rider2021.2/resharper-host/SourcesCache/8adb50b4d399477b881862854de9b0cb13c00/26/TestRenderer.cs:line 111
	RenderedFragmentExtensions.FindComponents<SolidCube.Components.Shared.TreeItem>() at /Users/rmihael/Library/Application Support/JetBrains/Rider2021.2/resharper-host/SourcesCache/b1aaada6d19e4b1a9a6620aa89b26c4b2aa00/27/RenderedFragmentExtensions.cs:line 83
	RenderedComponentExtensions.<>c__DisplayClass1_0<TreeItem>.<WaitForComponents>b__0()
	WaitForAssertionHelper.<>c__DisplayClass6_0.<.ctor>b__0()
	WaitForHelper<object>.OnAfterRender() at /Users/rmihael/Library/Application Support/JetBrains/Rider2021.2/resharper-host/SourcesCache/8adb50b4d399477b881862854de9b0cb13c00/14/WaitForHelper.cs:line 106
	new WaitForHelper<object>() at /Users/rmihael/Library/Application Support/JetBrains/Rider2021.2/resharper-host/SourcesCache/8adb50b4d399477b881862854de9b0cb13c00/14/WaitForHelper.cs:line 73
	new WaitForAssertionHelper()
	RenderedFragmentWaitForHelperExtensions.WaitForAssertion()
	RenderedComponentExtensions.WaitForComponents<SolidCube.Components.Shared.TreeItem>()

Worker thread @33878451
	Monitor.Enter() at /Users/rmihael/Library/Application Support/JetBrains/Rider2021.2/resharper-host/SourcesCache/3da0403ae6e540a7a79283e509f3945b8d7e00/18E/Monitor.cs:line 48
	WaitForHelper<object>.OnAfterRender() at /Users/rmihael/Library/Application Support/JetBrains/Rider2021.2/resharper-host/SourcesCache/8adb50b4d399477b881862854de9b0cb13c00/14/WaitForHelper.cs:line 97
	RenderedFragment.Bunit.IRenderedFragmentBase.OnRender() at /Users/rmihael/Library/Application Support/JetBrains/Rider2021.2/resharper-host/SourcesCache/b1aaada6d19e4b1a9a6620aa89b26c4b2aa00/4A/RenderedFragment.cs:line 158
	TestRenderer.UpdateDisplayAsync()
	Renderer.ProcessRenderQueue()
	Renderer.ProcessPendingRender() at /Users/rmihael/Library/Application Support/JetBrains/Rider2021.2/resharper-host/SourcesCache/28489e7a98534715a55deed8cf7572df6b200/3E/Renderer.cs:line 438
	TestRenderer.ProcessPendingRender() at /Users/rmihael/Library/Application Support/JetBrains/Rider2021.2/resharper-host/SourcesCache/8adb50b4d399477b881862854de9b0cb13c00/26/TestRenderer.cs:line 125
	Renderer.DispatchEventAsync() at /Users/rmihael/Library/Application Support/JetBrains/Rider2021.2/resharper-host/SourcesCache/28489e7a98534715a55deed8cf7572df6b200/3E/Renderer.cs:line 285
	TestRenderer.<>n__0()
	TestRenderer.<>c__DisplayClass15_0.<DispatchEventAsync>b__0()
	RendererSynchronizationContext.<>c.<<InvokeAsync>b__9_0>d.MoveNext()
	AsyncMethodBuilderCore.Start<Microsoft.AspNetCore.Components.Rendering.RendererSynchronizationContext.<>c.<<InvokeAsync>b__9_0>d>()
	AsyncVoidMethodBuilder.Start<Microsoft.AspNetCore.Components.Rendering.RendererSynchronizationContext.<>c.<<InvokeAsync>b__9_0>d>()
	RendererSynchronizationContext.<>c.<InvokeAsync>b__9_0()
	RendererSynchronizationContext.ExecuteSynchronously()
	RendererSynchronizationContext.<>c.<.cctor>b__23_0()
	ExecutionContext.RunInternal()
	ExecutionContext.Run()
	RendererSynchronizationContext.ExecuteBackground()
	RendererSynchronizationContext.<>c.<.cctor>b__23_1()
	ContinuationTaskFromTask.InnerInvoke()
	Task.<>c.<.cctor>b__277_0()
	ExecutionContext.RunFromThreadPoolDispatchLoop()
	Task.ExecuteWithThreadLocal()
	Task.ExecuteEntryUnsafe()
	Task.ExecuteFromThreadPool()
	ThreadPoolWorkQueue.Dispatch()
	_ThreadPoolWaitCallback.PerformWaitCallback()
	[Native to Managed Transition]

It seems that Renderer.DispatchEventAsync calls TestRenderer.ProcessPendingRender which locks TestRenderer.renderTreeAccessLock and then proceed to WaitForHelper.OnAfterRender where it waits on WaitForHelder.lockObject. On the other thread WaitForHelper.OnAfterRender locks WaitForHelper.lockObject and then proceeds to TestRenderer.FindComponents where it waits for TestRenderer.renderTreeAccessLock.

Version info:

bUnit version: 1.3.42
.NET Runtime and Blazor version: .NET 5.0.402, AspNetCore 5.0.8
OS type and version: MacOS 11.5.2

Additional context:

We saw hang-ups sporadically before, but now they are very consistent. It happened after we replaced Blazor.EventsAggregator dependency with native C# events. Many methods that were async before now sync, possibly resulting in a more deterministic flow.

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 29 (17 by maintainers)

Commits related to this issue

fix: race condition between renders, FindComponents, and WaitForHelper fixes #577. The problem is that FindComponents traverses down the render tree when invoked, and this ensures that no renders ha... — committed to bUnit-dev/bUnit by egil 2 years ago
fix: race condition between renders, FindComponents, and WaitForHelper fixes #577. The problem is that FindComponents traverses down the render tree when invoked, and this ensures that no renders ha... — committed to bUnit-dev/bUnit by egil 2 years ago
fix: race condition between renders, FindComponents, and WaitForHelper fixes #577. The problem is that FindComponents traverses down the render tree when invoked, and this ensures that no renders ha... — committed to bUnit-dev/bUnit by egil 2 years ago
fix: race condition between renders, FindComponents, and WaitForHelper fixes #577. The problem is that FindComponents traverses down the render tree when invoked, and this ensures that no renders ha... — committed to bUnit-dev/bUnit by egil 2 years ago
fix: race condition between renders, FindComponents, and WaitForHelper fixes #577. The problem is that FindComponents traverses down the render tree when invoked, and this ensures that no renders ha... — committed to bUnit-dev/bUnit by egil 2 years ago
fix: race condition between renders, FindComponents, and WaitForHelper fixes #577. The problem is that FindComponents traverses down the render tree when invoked, and this ensures that no renders ha... — committed to bUnit-dev/bUnit by egil 2 years ago

Most upvoted comments

Hi, We also have this problem with the tests deadlocking.

Hi @vedion, can you share a minimal example from your end that causes the deadlock? That would really help find a solution. Also, does this happens consistently in your tests, or is it just once in a while?

At the moment we have around 120 BUnit tests and it happens almost 50 % of the time. It happens when disposing TestContext. We haven’t found a consistent way to reproduce the problem but we have come up with two workarounds:

Skip disposing of TestContext. Not a big issue right now to skip it, but it is not pretty.
When disposing “TestContext” the “TestContext.TestServiceProvider” is disposed. “TestServiceProvider.Dispose()” uses: “AsTask().GetAwaiter().GetResult()” to sync dispose with an async method. This is known to cause deadlocks (https://www.nikouusitalo.com/blog/why-is-getawaiter-getresult-bad-in-c/). It works for us when we with reflection changes the dispose to this:

public abstract class TestBase : IAsyncLifetime
{
	protected TestBase()
	{
		TestContext = new TestContext();
	}

	protected TestContext TestContext { get; }

	public Task InitializeAsync()
	{
		return Task.CompletedTask;
	}

	public async Task DisposeAsync()
	{
		var rootServiceProvider = (IAsyncDisposable)TestContext.Services.GetFieldValue<IServiceProvider>("rootServiceProvider");
		if (rootServiceProvider != null)
		{
			await rootServiceProvider.DisposeAsync();
		}

		var serviceScope = (IAsyncDisposable)TestContext.Services.GetFieldValue<IServiceScope>("serviceScope");
		if (serviceScope != null)
		{
			await serviceScope.DisposeAsync();
		}

		TestContext.Dispose();
	}
}

vedion on Apr 8, 2022

Hi Egil! I’m working with @rmihael on the same project. Sorry for such a long reply. It’s been difficult to allocate time to work on the sample project between upcoming release and the started war… Finally we did it! Please, find the sample project in the attachment TestExample.zip

SviatoslavK on Mar 21, 2022

Worked for us too. Thank you!

rmihael on May 26, 2022

yep. works for us as well

SimonCropp on May 25, 2022

It works, thank you!

On 22 May 2022, at 03.54, Simon Cropp @.***> wrote:

thanks. i will apply it on our codebase now, and let you know how it goes

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

vedion on May 25, 2022

Hello everyone. I think this is sorted. There are nightly builds dropping in the GitHub package repository, and I will also be pushing a nightly build to NuGet, to make it easy for you all to try it out.

egil on May 21, 2022

@egil thanks for the clarification. is there anything i can help with? given we have this on my current project, perhaps i could help by running a beta nuget to verify any change?

SimonCropp on May 20, 2022

Hi again,

I have not yet been able to reproduce with a simple test yet. Would it be possible to let TestServiceProvider (and also TextContext) support IAsyncDisposable so we can dispose async? That would solve our problem.

Technically there is nothing preventing it, other than it would be a breaking change (esp for TestContext). I’m not ready to move to V2, but this would definitely be something I would change.

Can you think of another change i can make that will make your hack feel less …hacky?

egil on Apr 10, 2022

Let me explain what the problem is:

In the TestRenderer, we currently have a lock in place to prevent test code in thread 1 from traversing it in thread 2, which can lead to other race conditions, where users are not able to get their assertions passing before the render tree changes again (e.g. if a component renders, then waits a few milliseconds, and then renders again). FindComponents is how we traverse the render tree.

When a render is triggered/queued, the ProcessPendingRender method is called and the lock is entered when possible.
base.ProcessPendingRender() is called inside the lock, which starts rendering cycle.
When the render cycle completes, base.ProcessPendingRender() calls UpdateDisplayAsync.
UpdateDisplayAsync then call all IRenderedFragment the user has a handle on and tells them to update themselves with the latest changes to the DOM tree.
UpdateDisplayAsync then triggers all WaitForXXX predicate/assertions that gets a chance to inspect component or DOM state before a new render cycle can start.
After UpdateDisplayAsync finishes, control returns to ProcessPendingRender which release the lock.

The deadlock can happen if FindComponents is used in step 5 in a “WaitForXXX” predicate/assertion, since that would be blocked by the render lock, since that lock is still entered. until UpdateDisplayAsync completes.

egil on Apr 6, 2022

I tried to reproduce your example… besides the 26 minutes of runtime (due to the 1000 elements) the tests went green:

linkdotnet on Mar 23, 2022