coverlet: [BUG] Empty cobertura.xml in coverlet.collector 6.0.3 that is not reproduced in 6.0.2

Describe the bug I recently upgraded to 6.0.3 and I noticed the pipelines failing. It seems that cobertura.xml comes out empty. I am using a runsettings file with exclude/include. I notice that if I remove these sections the cobertura gets populated.

That said, coverage works fine and cobertura gets entries on 6.0.2 of coverlet.collector nuget package.

To Reproduce Check out https://github.com/kotsaris/repro-coverlet-empty-cobertura cd into ClassLibrary1 folder and run tests via

dotnet test --collect:"XPlat Code Coverage" --settings .\coverage.runsettings

Expected behavior It should populate cobertura.xml with something.

Actual behavior Cobertura.xml is empty.

<?xml version="1.0" encoding="utf-8"?>
<coverage line-rate="0" branch-rate="0" version="1.9" timestamp="1735730899" lines-covered="0" lines-valid="0" branches-covered="0" branches-valid="0">
  <sources />
  <packages />
</coverage>
Restore complete (0.3s)
  ClassLibrary1 succeeded (0.1s) → ClassLibrary1\bin\Debug\net9.0\ClassLibrary1.dll
  ClassLibrary1.Tests succeeded (0.0s) → ClassLibrary1.Tests\bin\Debug\net9.0\ClassLibrary1.Tests.dll
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v3.0.0+e341b939fe (64-bit .NET 9.0.0)
[xUnit.net 00:00:00.03]   Discovering: ClassLibrary1.Tests
[xUnit.net 00:00:00.05]   Discovered:  ClassLibrary1.Tests
[xUnit.net 00:00:00.06]   Starting:    ClassLibrary1.Tests
[xUnit.net 00:00:00.08]   Finished:    ClassLibrary1.Tests
  ClassLibrary1.Tests test succeeded (0.7s)

Test summary: total: 1, failed: 0, succeeded: 1, skipped: 0, duration: 0.7s
Build succeeded in 1.2s

Workload updates are available. Run `dotnet workload list` for more information.

Attachments:
  C:\dev\kotsaris\repro-coverlet-empty-cobertura\ClassLibrary1\ClassLibrary1.Tests\TestResults\5e5adf4e-c517-478b-a81e-ccff2099c832\coverage.cobertura.xml

Configuration (please complete the following information): Run Settings:

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
    <DataCollectionRunSettings>
        <DataCollectors>
            <DataCollector friendlyName="XPlat code coverage">
                <Configuration>
                    <Format>cobertura</Format>
                    <Exclude>
                        [*]ClassLibrary1.Tests.*
                    </Exclude>
                    <Include>[*]ClassLibrary1*</Include>
                    <ExcludeByAttribute>Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute</ExcludeByAttribute>
                    <SingleHit>false</SingleHit>
                    <UseSourceLink>false</UseSourceLink>
                    <IncludeTestAssembly>false</IncludeTestAssembly>
                    <SkipAutoProps>true</SkipAutoProps>
                    <DeterministicReport>false</DeterministicReport>
                    <ExcludeAssembliesWithoutSources>MissingAll,MissingAny,None</ExcludeAssembliesWithoutSources>
                </Configuration>
            </DataCollector>
        </DataCollectors>
    </DataCollectionRunSettings>
</RunSettings>

Please provide more information on your .NET configuration: * Which coverlet package and version was used? 6.0.3 * Which version of .NET is the code running on? 9 * What OS and version, and what distro if applicable? Windows 11 * What is the architecture (x64, x86, ARM, ARM64)? x64 * Do you know whether it is specific to that configuration? 6.0.2 does not exhibit this

Additional context Downgrade to 6.0.2 and the problem goes away.

❗ Please also read Known Issues

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 16
  • Comments: 27

Commits related to this issue

Most upvoted comments

Good news, we now officially released the fix with version 6.0.4.

yes, it is working again with 8.0.0-preview.2.ge3717da4a7

@mu88 Sorry for the misunderstanding. The issue was automatically closed when I merged the PR. We planned some breaking changes for the v8 version but nothing of this is merged yet. So I can easily bump back to e.g. 6.0.4. The plan is most likely to release a 6.0.4 with the fix in the next days. But I need to discuss this with the other maintainers. Also I would appreciate a bit more feedback 😃. As this fix is now merged you can also consume our nightly build to test it.

ps: The nightly build is also called something with 8.0.0-preview... for the same reason as described above. Just ignore that. There is no breaking change yet.

Good news, we now officially released the fix with version 6.0.4.

Nice work @daveMueller. @dependabot updated all our projects last night, and I can confirm the fix has resolved our issue.

use 8.0.0-preview.2.ge3717da4a7

@daveMueller I have tested the latest nightly build package version 8.0.0-preview.2.ge3717da4a7 and it is working as expected against my exclusion rules.

Thanks a lot guys 🙏 . I understand it much better now and think my changes should work. Could somebody try out the version I compiled locally?

coverlet.8.0.0-preview.3.g13af8e390b.zip

@daveMueller Yes, these updated packages solve the problem for us.

I meant to report this issue months ago and forgot =(. Per my notes it was introduced with 6.0.3-preview.3

You asked what our Include/Excludes look like…

<ExcludeByAttribute>ExcludeFromCodeCoverage,Obsolete,GeneratedCode,DebuggerHidden,DebuggerNonUserCode,Migration</ExcludeByAttribute>
<Exclude>[.*.tests?]*,[*.Tests]*,[xunit.*]*,[*]*.Entities,[My.Company.Analyzers]*,[My.Company.Kubernetes]*,[*]My.Company.Namespace.*</Exclude>

It’s been a while, but I believe:

  • [.*.tests?]* and [*.Tests]* exclude assemblies ending in “test” or “tests”
    • I’m not sure why the second one isn’t included in the first, but I also think I got these from coverlet documentation at some point
  • [xunit.*]* - I have no idea why we’d need to exclude xUnit assemblies
  • that the first two are to exclude test projects from coverage. I’m not sure why xunit is there.
  • [*]*.Entities - exclude anything in any assembly in a namespace ending in ‘Entities’
  • [My.Company.Analyzers]* and [My.Company.Kubernetes]* - exclude these two assemblies entirely
  • [*]My.Company.Namespace.* - exclude the My.Company.Namespace in any assembly

Another project has (very similar):

<ExcludeByAttribute>ExcludeFromCodeCoverage,GeneratedCode,DebuggerHidden,DebuggerNonUserCode,Migration,DbContext</ExcludeByAttribute>
<Exclude>[.*.tests?]*,[*.Tests]*,[xunit.*]*,[*]*.Migrations.*,[*.Test.Shared]*</Exclude>
  • First three same as above
  • [*]*.Migrations.* - Exclude any namespace with “Migrations” as one of its components
  • [*.Test.Shared]* - Exclude the entirety of any assembly that ends in “.Test.Shared”

Thanks a lot guys 🙏 . I understand it much better now and think my changes should work. Could somebody try out the version I compiled locally?

coverlet.8.0.0-preview.3.g13af8e390b.zip

Hi guys, could you help out by providing a bit more input on how you are using the Include and Exclude filters in combination in your projects and what is the result you expect.

We run our test builds from a bash script. Our problem is not specifically cobertura output but any output. Where we previous received coverage, some of our builds get no coverage.

Here’s the GitHub actions output where we no longer receive results.

Broken coverage
$ dotnet test --no-restore --no-build -c Release \
  /home/runner/work/my-product/my-product/my-service/test/Abc.MyService.Tests/Abc.MyService.Tests.csproj \
  --results-directory /home/runner/work/my-product/my-product/my-service/test/TestResults/output/ \
  --logger trx;LogFileName=Abc.MyService.Tests.trx \
  --logger html;LogFileName=Abc.MyService.Tests.html \
  -p:CollectCoverage=true \
  -p:Include=[Abc.MyService*]* \
  -p:Exclude="[*.Tests]*,[Abc.*]*.Migrations.*,[Abc.*]*.SourceGenerationContext" \
  -p:CoverletOutput=/home/runner/work/my-product/my-product/my-service/test/TestResults/output/ \
  -p:CoverletOutputFormat="json,opencover,cobertura,lcov"

  [coverlet] _mapping file name: 'CoverletSourceRootsMapping_Abc.MyService.Tests'
Test run for /home/runner/work/my-product/my-product/my-service/test/Abc.MyService.Tests/bin/Release/net9.0/Abc.MyService.Tests.dll (.NETCoreApp,Version=v9.0)
VSTest version 17.12.0 (x64)

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Results File: /home/runner/work/my-product/my-product/my-service/test/TestResults/output/Abc.MyService.Tests.trx
Html test results file : /home/runner/work/my-product/my-product/my-service/test/TestResults/output/Abc.MyService.Tests.html

Passed!  - Failed:     0, Passed:   687, Skipped:     0, Total:   687, Duration: 1 m 20 s - Abc.MyService.Tests.dll (net9.0)
  [coverlet] 
  Calculating coverage result...
   Generating report '/home/runner/work/my-product/my-product/my-service/test/TestResults/output/coverage.json'
   Generating report '/home/runner/work/my-product/my-product/my-service/test/TestResults/output/coverage.opencover.xml'
   Generating report '/home/runner/work/my-product/my-product/my-service/test/TestResults/output/coverage.cobertura.xml'
   Generating report '/home/runner/work/my-product/my-product/my-service/test/TestResults/output/coverage.info'

+--------+------+--------+--------+
| Module | Line | Branch | Method |
+--------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 0%   | 0%     | 0%     |
+---------+------+--------+--------+
| Average | 0%   | 0%     | 0%     |
+---------+------+--------+--------+

Though we have another service where we still receive the output.

Still working coverage
$ dotnet test --no-restore --no-build -c Release \
  /home/runner/work/my-product/my-product/other-service/test/Abc.OtherService.Tests/Abc.OtherService.Tests.csproj \
  --results-directory /home/runner/work/my-product/my-product/other-service/test/TestResults/output/ \
  --logger trx;LogFileName=Abc.OtherService.Tests.trx \
  --logger html;LogFileName=Abc.OtherService.Tests.html \
  -p:CollectCoverage=true \
  -p:Include=[Abc.*]* \
  -p:Exclude="[*.Tests]*,[Abc.Testing.*]*,[Abc.Platform.Testing.*]*" \
  -p:CoverletOutput=/home/runner/work/my-product/my-product/other-service/test/TestResults/output/Abc.OtherService.Tests.coverage.json

  [coverlet] _mapping file name: 'CoverletSourceRootsMapping_Abc.OtherService.Tests'
Test run for /home/runner/work/my-product/my-product/other-service/test/Abc.OtherService.Tests/bin/Release/net9.0/Abc.OtherService.Tests.dll (.NETCoreApp,Version=v9.0)
VSTest version 17.12.0 (x64)

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Results File: /home/runner/work/my-product/my-product/other-service/test/TestResults/output/Abc.OtherService.Tests.trx
Html test results file : /home/runner/work/my-product/my-product/other-service/test/TestResults/output/Abc.OtherService.Tests.html

Passed!  - Failed:     0, Passed:   776, Skipped:     0, Total:   776, Duration: 50 s - Abc.OtherService.Tests.dll (net9.0)
  [coverlet] 
  Calculating coverage result...
   Generating report '/home/runner/work/my-product/my-product/other-service/test/TestResults/output/Abc.OtherService.Tests.coverage.json'

+-----------------------+--------+--------+--------+
| Module                | Line   | Branch | Method |
+-----------------------+--------+--------+--------+
| Abc.OtherService      | 100%   | 100%   | 100%   |
+-----------------------+--------+--------+--------+
| Abc.OtherService.Apps | 99.32% | 94.58% | 100%   |
+-----------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 99.36% | 94.76% | 100%   |
+---------+--------+--------+--------+
| Average | 99.66% | 97.29% | 100%   |
+---------+--------+--------+--------+

@JohnYoungers not sure about a new release just for this, I need to discuss this with the other maintainers. There will be the possibility to consume the fix in our nightly once we have merged it.

I understand the hesitation, but I think as time goes on, this will catch more and more consumers with automated nuget update processes for patch releases.

This would be a sample of our usage:

  /p:CollectCoverage=true \
  /p:CoverletOutputFormat=cobertura%2copencover \
  /p:CoverletOutput=${{ inputs.output_path }}/${{ inputs.test_run_name }} \
  /p:Exclude="Company.App/Subnamespace/**/*,Company.App.Api/Program.cs,database/**" \

And we’re doing something like this:

dotnet test 
  --configuration Release 
  --no-build 
  --no-restore 
  --logger:trx 
  --verbosity minimal 
  --filter RunOnBuildServer!=false&Category=Unit 
  --results-directory test-outputs/component 
  /p:CollectCoverage=true 
  /p:ExcludeByAttribute="GeneratedCodeAttribute,ExcludeFromUnitTestCoverageAttribute" 
  /p:Exclude=[*]*.Migrations.* 
  /p:CoverletOutputFormat=opencover 
  /p:CoverletOutput=coverage.opencover.Unit.xml 
  Tests/Tests.csproj

Hi guys, could you help out by providing a bit more input on how you are using the Include and Exclude filters in combination in your projects and what is the result you expect.

Here’s what I’ve got:

          <Exclude>[coverlet.*.tests?]*,[*]Coverlet.Core*</Exclude> 
          <Include>*C.D*</Include>

The intention is to include assemblies such as:

A.C.D.E
B.C.D.E
A.C.D.E.F

and obviously exclude the corverlet assemblies. I honestly don’t recall why the exclude looks the way it does other than to say that it took longer than I expected and I couldn’t spend any more time on it. Without the exclude, coverlet assemblies were throwing off the code coverage percentage.

Thanks a lot for reporting and for the repro @kotsaris 🙏. I did some quick analysis and think the problem came in here #1645. To be even more precise I think this condition here was missed: https://github.com/coverlet-coverage/coverlet/pull/1645/files#diff-be914c2ef40b4434ccb0927c453da5f7f2a4f828d601367f585bd7284349ba85L354-L357

I locally added it back and it seems to work. But maybe we should consider putting in a bit more work because I think the whole Include and Exclude filter logic is a bit difficult to understand? cc @Bertk @MarcoRossignoli

(btw @kotsaris in your repro it is also sufficient if you only remove the Exclude filter from the runsettings file)