coverlet: [BUG] Empty cobertura.xml in coverlet.collector 6.0.3 that is not reproduced in 6.0.2
Describe the bug I recently upgraded to 6.0.3 and I noticed the pipelines failing. It seems that cobertura.xml comes out empty. I am using a runsettings file with exclude/include. I notice that if I remove these sections the cobertura gets populated.
That said, coverage works fine and cobertura gets entries on 6.0.2 of coverlet.collector nuget package.
To Reproduce Check out https://github.com/kotsaris/repro-coverlet-empty-cobertura cd into ClassLibrary1 folder and run tests via
dotnet test --collect:"XPlat Code Coverage" --settings .\coverage.runsettings
Expected behavior It should populate cobertura.xml with something.
Actual behavior Cobertura.xml is empty.
<?xml version="1.0" encoding="utf-8"?>
<coverage line-rate="0" branch-rate="0" version="1.9" timestamp="1735730899" lines-covered="0" lines-valid="0" branches-covered="0" branches-valid="0">
<sources />
<packages />
</coverage>
Restore complete (0.3s)
ClassLibrary1 succeeded (0.1s) → ClassLibrary1\bin\Debug\net9.0\ClassLibrary1.dll
ClassLibrary1.Tests succeeded (0.0s) → ClassLibrary1.Tests\bin\Debug\net9.0\ClassLibrary1.Tests.dll
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v3.0.0+e341b939fe (64-bit .NET 9.0.0)
[xUnit.net 00:00:00.03] Discovering: ClassLibrary1.Tests
[xUnit.net 00:00:00.05] Discovered: ClassLibrary1.Tests
[xUnit.net 00:00:00.06] Starting: ClassLibrary1.Tests
[xUnit.net 00:00:00.08] Finished: ClassLibrary1.Tests
ClassLibrary1.Tests test succeeded (0.7s)
Test summary: total: 1, failed: 0, succeeded: 1, skipped: 0, duration: 0.7s
Build succeeded in 1.2s
Workload updates are available. Run `dotnet workload list` for more information.
Attachments:
C:\dev\kotsaris\repro-coverlet-empty-cobertura\ClassLibrary1\ClassLibrary1.Tests\TestResults\5e5adf4e-c517-478b-a81e-ccff2099c832\coverage.cobertura.xml
Configuration (please complete the following information): Run Settings:
<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="XPlat code coverage">
<Configuration>
<Format>cobertura</Format>
<Exclude>
[*]ClassLibrary1.Tests.*
</Exclude>
<Include>[*]ClassLibrary1*</Include>
<ExcludeByAttribute>Obsolete,GeneratedCodeAttribute,CompilerGeneratedAttribute</ExcludeByAttribute>
<SingleHit>false</SingleHit>
<UseSourceLink>false</UseSourceLink>
<IncludeTestAssembly>false</IncludeTestAssembly>
<SkipAutoProps>true</SkipAutoProps>
<DeterministicReport>false</DeterministicReport>
<ExcludeAssembliesWithoutSources>MissingAll,MissingAny,None</ExcludeAssembliesWithoutSources>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>
Please provide more information on your .NET configuration: * Which coverlet package and version was used? 6.0.3 * Which version of .NET is the code running on? 9 * What OS and version, and what distro if applicable? Windows 11 * What is the architecture (x64, x86, ARM, ARM64)? x64 * Do you know whether it is specific to that configuration? 6.0.2 does not exhibit this
Additional context Downgrade to 6.0.2 and the problem goes away.
❗ Please also read Known Issues
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 16
- Comments: 27
Commits related to this issue
- Upgrade coverlet.msbuild to overcome the problem described here: https://github.com/coverlet-coverage/coverlet/issues/1726 — committed to Jericho/ZoomNet by Jericho 2 years ago
Good news, we now officially released the fix with version
6.0.4.yes, it is working again with
8.0.0-preview.2.ge3717da4a7@mu88 Sorry for the misunderstanding. The issue was automatically closed when I merged the PR. We planned some breaking changes for the
v8version but nothing of this is merged yet. So I can easily bump back to e.g.6.0.4. The plan is most likely to release a6.0.4with the fix in the next days. But I need to discuss this with the other maintainers. Also I would appreciate a bit more feedback 😃. As this fix is now merged you can also consume our nightly build to test it.ps: The nightly build is also called something with
8.0.0-preview...for the same reason as described above. Just ignore that. There is no breaking change yet.Nice work @daveMueller. @dependabot updated all our projects last night, and I can confirm the fix has resolved our issue.
use 8.0.0-preview.2.ge3717da4a7
@daveMueller I have tested the latest nightly build package version 8.0.0-preview.2.ge3717da4a7 and it is working as expected against my exclusion rules.
@daveMueller Yes, these updated packages solve the problem for us.
I meant to report this issue months ago and forgot =(. Per my notes it was introduced with
6.0.3-preview.3You asked what our Include/Excludes look like…
It’s been a while, but I believe:
[.*.tests?]*and[*.Tests]*exclude assemblies ending in “test” or “tests”[xunit.*]*- I have no idea why we’d need to exclude xUnit assembliesxunitis there.[*]*.Entities- exclude anything in any assembly in a namespace ending in ‘Entities’[My.Company.Analyzers]*and[My.Company.Kubernetes]*- exclude these two assemblies entirely[*]My.Company.Namespace.*- exclude theMy.Company.Namespacein any assemblyAnother project has (very similar):
[*]*.Migrations.*- Exclude any namespace with “Migrations” as one of its components[*.Test.Shared]*- Exclude the entirety of any assembly that ends in “.Test.Shared”Thanks a lot guys 🙏 . I understand it much better now and think my changes should work. Could somebody try out the version I compiled locally?
coverlet.8.0.0-preview.3.g13af8e390b.zip
We run our test builds from a bash script. Our problem is not specifically
coberturaoutput but any output. Where we previous received coverage, some of our builds get no coverage.Here’s the GitHub actions output where we no longer receive results.
Broken coverage
Though we have another service where we still receive the output.
Still working coverage
I understand the hesitation, but I think as time goes on, this will catch more and more consumers with automated nuget update processes for patch releases.
This would be a sample of our usage:
And we’re doing something like this:
Here’s what I’ve got:
The intention is to include assemblies such as:
and obviously exclude the corverlet assemblies. I honestly don’t recall why the exclude looks the way it does other than to say that it took longer than I expected and I couldn’t spend any more time on it. Without the exclude, coverlet assemblies were throwing off the code coverage percentage.
Thanks a lot for reporting and for the repro @kotsaris 🙏. I did some quick analysis and think the problem came in here #1645. To be even more precise I think this condition here was missed: https://github.com/coverlet-coverage/coverlet/pull/1645/files#diff-be914c2ef40b4434ccb0927c453da5f7f2a4f828d601367f585bd7284349ba85L354-L357
I locally added it back and it seems to work. But maybe we should consider putting in a bit more work because I think the whole
IncludeandExcludefilter logic is a bit difficult to understand? cc @Bertk @MarcoRossignoli(btw @kotsaris in your repro it is also sufficient if you only remove the
Excludefilter from therunsettingsfile)