bUnit: Test Runs Failing on net5.0

Describe the bug

Firstly this may not be a bug with bUnit. These are some observations and I would like some advice on how to log repro these issues. Since moving our library to net5.0 our test pipeline has started giving seemingly random failures. We almost exclusively do component testing using bUnit. I am really not sure where to start. Could it possibly be a timing bug with bUnit on Linux (the build server)? I am at a loss since it seems different test fail every time so finding a pattern is very hard as you can imagine. The below is just an example although it probably doesn’t help much sorry. This test will pass normally and will almost certainly pass if I just re-run the pipeline.

Example: Testing this component:

<MudButton Variant="Variant.Filled" OnClick="()=>visible=true">Open</MudButton>

<MudDialog @bind-IsVisible="visible">
    <DialogContent>
        <MudText>Wabalabadubdub!</MudText>
    </DialogContent>
    <DialogActions>
        <MudButton Color="Color.Primary" OnClick="()=>visible=false">Close</MudButton>
    </DialogActions>
</MudDialog>

@code {
    public static string __description__ = "Click on open will open the inlined dialog";

    bool visible;
}

With this test:

  [TestFixture]
  public class DialogTests
  {
        private Bunit.TestContext ctx;

        [SetUp]
        public void Setup()
        {
            ctx = new Bunit.TestContext();
            ctx.AddTestServices();
        }

        [TearDown]
        public void TearDown() => ctx.Dispose();

           /// <summary>
        /// Click outside the dialog (or any other method) must update the IsVisible parameter two-way binding on close
        /// </summary>
        /// <returns></returns>
        [Test]
        public async Task InlineDialog_Should_UpdateIsVisibleOnClose()
        {
            var comp = ctx.RenderComponent<MudDialogProvider>();
            comp.Markup.Trim().Should().BeEmpty();
            var service = ctx.Services.GetService<IDialogService>() as DialogService;
            service.Should().NotBe(null);
            // displaying the component with the inline dialog only renders the open button
            var comp1 = ctx.RenderComponent<InlineDialogIsVisibleStateTest>();
            // open the dialog
            comp1.Find("button").Click();
            Console.WriteLine("\nOpened dialog: " + comp.Markup);
            comp.Find("div.mud-dialog-container").Should().NotBe(null);
            // close by click outside
            comp.Find("div.mud-overlay").Click();
            comp.Markup.Trim().Should().BeEmpty();
            // open again
            comp1.Find("button").Click();
            comp.Find("div.mud-dialog-container").Should().NotBe(null);
            // close again by click outside
            comp.Find("div.mud-overlay").Click();
            comp.Markup.Trim().Should().BeEmpty();
        }

    }

  public static class TestContextExtensions
    {
        public static void AddTestServices(this Bunit.TestContext ctx)
        {
            ctx.JSInterop.Mode = JSRuntimeMode.Loose;
            ctx.Services.AddSingleton<NavigationManager>(new MockNavigationManager());
            ctx.Services.AddMudServices(options =>
            {
                options.SnackbarConfiguration.ShowTransitionDuration = 0;
                options.SnackbarConfiguration.HideTransitionDuration = 0;
            });
            ctx.Services.AddScoped(sp => new HttpClient());
            ctx.Services.AddOptions();
        }
    }

Results in this output:


  Failed InlineDialog_Should_UpdateIsVisibleOnClose [25 ms]
  Error Message:
   Bunit.ElementNotFoundException : No elements were found that matches the selector 'div.mud-dialog-container'
  Stack Trace:
     at Bunit.RenderedFragmentExtensions.Find(IRenderedFragment renderedFragment, String cssSelector) in /_/src/bunit.web/Extensions/RenderedFragmentExtensions.cs:line 30
   at MudBlazor.UnitTests.DialogTests.InlineDialog_Should_UpdateIsVisibleOnClose() in /home/vsts/work/1/s/src/MudBlazor.UnitTests/Components/DialogTests.cs:line 123
   at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter`1.GetResult()
   at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(Func`1 invoke)
   at NUnit.Framework.Internal.Commands.TestMethodCommand.RunTestMethod(TestExecutionContext context)
   at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context)
   at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.<>c__DisplayClass1_0.<Execute>b__0()
NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action)
  Standard Output Messages:
 
 Opened dialog: <div class="mud-dialog-container mud-dialog-center"><div class="mud-overlay mud-overlay-dialog" style="" blazor:onclick="2" blazor:onclick:stopPropagation><div class="mud-overlay-scrim mud-overlay-dark mud-overlay-dialog"></div>
         <div class="mud-overlay-content"></div></div>
     <div id="_9e7cbd63e65248b7b09ec03ebe80ff58" class="mud-dialog mud-dialog-width-sm"><div class="mud-dialog-title"><h6 class="mud-typography mud-typography-h6 mud-inherit-text"><svg class="mud-icon-root mud-svg-icon mud-inherit-text mud-icon-size-medium mr-3" focusable="false" viewBox="0 0 24 24" aria-hidden="true"><path d="M0 0h24v24H0z" fill="none"/><path d="M3 17.25V21h3.75L17.81 9.94l-3.75-3.75L3 17.25zM20.71 7.04c.39-.39.39-1.02 0-1.41l-2.34-2.34c-.39-.39-1.02-.39-1.41 0l-1.83 1.83 3.75 3.75 1.83-1.83z"/></svg> Edit rating
         </h6></div><div style="outline-style: none;" blazor:onkeydown="3" blazor:onkeyup="4" blazor:onfocus="5" tabindex="-1" blazor:elementReference=""><div style="pointer-events:none; position:fixed;" tabindex="0" blazor:onfocus="6"></div>
 
     <div style="pointer-events:none; position:fixed;" tabindex="0" blazor:onfocus="7" blazor:elementReference=""></div>
 
     <div style="pointer-events:none; position:fixed;" tabindex="-1" blazor:elementReference=""></div>
 
     <div class="mud-dialog-content"><p>Close and re-open me by clicking out of the dialog. </p></div><div class="mud-dialog-actions"></div>
 
     <div style="pointer-events:none; position:fixed;" tabindex="0" blazor:onfocus="8" blazor:elementReference=""></div>
 
     <div style="pointer-events:none; position:fixed;" tabindex="0" blazor:onfocus="9"></div></div></div></div>

Expected behavior: Passes (It does usually)

Version info:

bUnit version: preview 01
.NET Runtime and Blazor version: 5.0.3
OS type and version: buiild server Ubuntu 20.04.2

Additional context: Passes nearly all of the time. This is just one of seeming random failures. All tests that have passed in the past many times. One Thing that has chnaged that seems to have triggered these random failures is our library moving to net 5.0

cc @henon

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 28 (12 by maintainers)

Most upvoted comments

Or maybe this is expected if I dont use waitfor?

No, this should ideally show up no matter what in the test output. It is the plan to fix that in #319.

egil on Mar 9, 2021

@egil Is this the same as #319 ?

Exactly.

egil on Mar 6, 2021

Thanks for taking the time to report this. I am aware of the issue, and will be attempting to fix it in the next release.

egil on Mar 6, 2021

I had another seemingly random failure but it was this. We render all our docs examples to check for errors. We don’t test any assertions. We mock api for http content retrieval using a MockHttpHandler, Every now and then I got an exception . I didn’t know why. But when I tested the MockHttpHandler directly it failed every time. I had an incorrect content type. Why does the thread not synchronise with the await call and bubble the exception back to the test thread?

 var response = await HttpClient.GetAsync("https://raw.githubusercontent.com/Garderoben/MudBlazor/master/LICENSE");
        LicenseText = await response.Content.ReadAsStringAsync();

This is the example

@using System.Net
@using System.Text
@namespace MudBlazor.Docs.Examples

<MudDialog DisableSidePadding="true">
    <DialogContent>
        <MudContainer Style="max-height: 300px; overflow-y: scroll">
            @if (Loading)
            {
                <MudProgressCircular Indeterminate="true"></MudProgressCircular>
            }
            else
            {
                <MudText Style="white-space: pre-wrap;">@LicenseText</MudText>
            }
        </MudContainer>
    </DialogContent>
    <DialogActions>
        <MudButton Color="Color.Primary" OnClick="Ok">Accept</MudButton>
    </DialogActions>
</MudDialog>


@code {
    [CascadingParameter] MudDialogInstance MudDialog { get; set; }

    [Inject] HttpClient HttpClient { get; set; }

    protected override async Task OnInitializedAsync()
    {
        await base.OnInitializedAsync();
        Loading = true;
        var response = await HttpClient.GetAsync("https://raw.githubusercontent.com/Garderoben/MudBlazor/master/LICENSE");
        LicenseText = await response.Content.ReadAsStringAsync();
        Loading = false;
    }

    private string LicenseText;
    private bool Loading = false;

    private void Ok()
    {
        MudDialog.Close(DialogResult.Ok(true));
    }
}

And the test which is auto generated just makes sure the example renders without error

        [Test]
        public void DialogScrollableExample_Test()
        {
            ctx.RenderComponent<DialogScrollableExample>();
        }

mikes-gh on Mar 5, 2021

Yes I will close. When I have another seemingly random failure I will do my best to tie it down. Certainly there is something going on on the build server that is not happening locally. Whether it be through poorly written tests or some complex threading issues I am not sure.

mikes-gh on Feb 19, 2021

Thanks for clarifying @mikes-gh.

As for the WaitFor methods… you might need to use them when you have async operations that trigger the renders in the component, otherwise you are likely to see this issue that randomly shows up. That is expected, because the test code runs in its own thread, and the renderer runs in another. And if you trigger something from the test thread, e.g. by clicking a button, that causes the renderer to schedule a async render, e.g. due to a Task.Delay call, then you most definently need to use one of the WaitFor methods.

egil on Feb 19, 2021