runtime: Code coverage for System.Private.CoreLib is broken in CI

Reopening as System.Private.CoreLib numbers are busted again. With my recent work I overlooked some details and we use the native System.Private.CoreLib.dll again. I’ll try to solve this.

Since there is some discussion on top with some outdated/incorrect information it seems useful to record the history of the issues affecting coverage and for System.Private.CoreLib in particular:

1. When S.P.C.ni.dll was renamed to S.P.C.dll coverage was broken due to two issues:

   a. At that time the Profiler API had a bug that prevented OpenCover from instrumenting any crossgen assembly. That’s why Stephen’s attempts here failed.

   b. Once the Profiler API was fixed OpenCover behavior regarding symbols, prevented it from instrumenting the methods in S.P.C.dll. OpenCover (current sources and version 4.6.519) bails on any assembly if it finds a pdb with same name and it fails to load it, i.e.: on firt mismatch it doesn’t look in extra search paths (not typical symbol search algo). In this case S.P.C.pdb was for the IL version, so it didn’t match S.P.C.dll.

2. Around 10/2017 pdbs became portable by default and since version 4.6.519 of OpenCover doesn’t support portable PDBs a workaround was put in place to generate respective WindowsPDB. However, the workaround just target AssemblyBeingTestedName not any additional assemblies, like S.P.C.dll (this is a simple fix on CodeCoverage.targets).

One related issue that tripped me in the beginning of my investigation was the fact that current master sources of OpenCover w/ latest Mono.Cecil (that supports portable PDBs) failed with an unchanged CoreFx repo due to a bug (already fixed) in Mono.Cecil that prevented it from working with WindowsPDBs.

In summary: at the present, to make CoreLib work with OpenCover we can use either the native or IL S.P.C.dll, but, the S.P.C.pdb first found by OpenCover must match the version being loaded.

np, first step for automatic corelib coverage is to include IL version on the transport package.

Jan already indicated which file can be used to ensure that it doesn’t get included in shipping bits, but 😄 @danmosemsft @safern can you save me some grepping to figure out where the contents of the transport package are defined? I suspect the the report won’t be able to reach corelib sources at first, but let’s get the IL corelib as first step.

I also started looking at the alternatives to get code coverage in Unix, I’ll report in a few days about it.

BTW @ahsonkhan @pjanotti if either of you have manually generated coverage for corelib from all the corefx tests, I would be interested to take a look if you could share the folder out to me.

Unfortunately not. At least OpenCover should soon support portable PDB clearly there’s other issues to root cause here.