aspnetcore: Integration testing is hard to set up and may break on shared framework servicing

Sort of a follow-up to @poke’s https://github.com/dotnet/sdk/issues/2253 but orthogonal.

TL;DR: Unit tests should have an option to run on the asp.net core shared framework using the same servicing strategy that the main subject-under-test application uses, which means referencing the shared framework using the latest known version of the web SDK and applying NuGet and assembly resolution accordingly.

Story: As a developer, I want to add a new test project with a reference to my asp.net core project so that I can write integration tests.

Naive steps:

  • dotnet new razor -o …/RazorPagesProject
  • dotnet new xunit
  • dotnet add reference …/RazorPagesProject
  • dotnet add package Microsoft.AspNetCore.Mvc.Testing -v 2.1.0-rc1-final
  • add C# code

    public class SampleIntegrationTests : IClassFixture<WebApplicationFactory<Startup>>
    {
        public HttpClient Client { get; }

        public SampleIntegrationTests(WebApplicationFactory<Startup> fixture)
        {
            Client = fixture.CreateDefaultClient();
        }

        [Fact]
        public async Task ItShallWork()
        {
            var response = await Client.GetAsync("/");
            Assert.Equal(HttpStatusCode.OK, response.StatusCode);
        }
    }
  • dotnet test

Expected: Test passes

Actual:

Starting test execution, please wait...
[xUnit.net 00:00:00.5209160]   Discovering: SampleTests
[xUnit.net 00:00:00.5793000]   Discovered:  SampleTests
[xUnit.net 00:00:00.5854810]   Starting:    SampleTests
[xUnit.net 00:00:00.7291840]     SampleTests.SampleIntegrationTests.ItShallWork [FAIL]
[xUnit.net 00:00:00.7305700]       System.IO.FileNotFoundException : Could not load file or assembly 'Microsoft.AspNetCore, Version=2.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.
[xUnit.net 00:00:00.7306390]
[xUnit.net 00:00:00.7461290]   Finished:    SampleTests
Failed   SampleTests.SampleIntegrationTests.ItShallWork
Error Message:
 System.IO.FileNotFoundException : Could not load file or assembly 'Microsoft.AspNetCore, Version=2.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. Thesystem cannot find the file specified.

Workaround: Add a Package reference to the test project (!):

<ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.All" Version="2.1.0-rc1-final" />
</ItemGroup>

What I really want:

  • It should just work ™️
  • Integration tests shall run on the asp.net core shared framework
  • Integration tests shall use the same patch level of asp.net core components that dotnet run or dotnet publish + dotnet myapp.dll would use

What could be done here? a web-test-SDK maybe?

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 9
  • Comments: 23 (22 by maintainers)

Commits related to this issue

Most upvoted comments

I agree with @poke, the VS UI gets awkward (makes it look like a web project, adds connected services node, and adds launchsettings.json, I can “publish” my unit tests to Azure, etc.). Improved documentation and a template does helps people get it right though. It just would be nice if the VS UI wasn’t misleading too.

@DamianEdwards Naive suggestion: why not have <Project Sdk="Microsoft.NET.Sdk.Test"> and <Project Sdk="Microsoft.NET.Sdk.Web.Test"> so that project nodes “definitively” look like test projects and work in their respective environments, netcoreapp testing and aspnetcoreapp testing? Additionally, the hypothetical test SDK could make <PackageReference Include="Microsoft.NET.Test.Sdk" Version="x.x.x" /> automagical, so users don’t have to reference it from NuGet themselves. It would ideally just match the version of Visual Studio you are running.

Side note: It’s not perfectly resolved since testing the app should ideally pull in the needed references transitively, so a reference to the shared framework should not be necessary.

But if I understand it correctly, with 3.0’s framework references, the dependencies can be pulled transitively, so this should work out of the box again.

After working with this for a while now, there are two things that are really bothering me about having to make the test project a Web SDK project:

  • In Visual Studio, the test project has a normal web application icon; so I cannot quickly see that this is a test project.
  • This being a web project makes Visual Studio add a launchSettings.json to it by default, which is really unneded there. So one has to use <NoDefaultLaunchSettingsFile>True</NoDefaultLaunchSettingsFile> explicitly to disable that.

We’re going to look at adding a project template for this in 2.2.0. Thanks for the feedback folks.

It could be. If you are running from the package cache, you don’t get the same roll-forward and security-patch semantics as a shared framework would.

There are lots of problems here to consider. The shared framework aspect is one of them, but there are more. Nuspec generation, target framework compatibility, NuGet UI, minor version rollforward for shared frameworks, etc. I appreciate your feedback, and it’s something we’ll continue to consider. But I agree with @dasMulli - this is a problematic and tricky area in which there do not seem to be good options.

FYI adding a template for tests is being tracked in https://github.com/aspnet/templating/issues/511 though it has not yet been triaged.