c8: Incorrect branch coverage when loaders used

c8 shows uncovered lines, but the whole file is covered.

Version: output of node -v 16.9.0 Platform: output of uname -a Darwin

Version: latest
Platform: mac os

Repository https://github.com/coderaiser/c8-reproduce

When I’m using loaders to mock imports the coverage I see is broken.

Code:

import {
    readFile,
} from 'fs/promises';

import {
    execSync,
} from 'child_process';

export default (a, b, c) => {
    if (a)
        return readFile();

      if (c)
          return execSync();

      return 'd';
};

Tests:

import {createMockImport} from 'mock-import';
import {
    test,
    stub,
} from 'supertape';

const {mockImport, reImport, stopAll} = createMockImport(import.meta.url);

test('changelog: a', async (t) => {
    mockImport('fs/promises', {
        readFile: stub().returns('a'),
    });
    const fn = await reImport('./changelog.js');
    stopAll();

    t.equal(fn.default(1), 'a');
});

test('changelog: c', async (t) => {
    mockImport('child_process', {
        execSync: stub().returns('c'),
    });

    const fn = await reImport('./changelog.js');
    stopAll();

    t.equal(fn.default(0, 0, 1), 'c');
});

test('changelog: d', async (t) => {
    const fn = await import('./changelog.js?count=4');

    t.equal(fn.default(0, 0, 0, 1), 'd');
});

What mock-import does is converts source to:

const {
    readFile: readFile
} = global.__mockImportCache.get('fs/promises');

import {
    execSync,
} from 'child_process';

export default (a, b, c) => {
    if (a)
        return readFile();
     
      if (c)
          return execSync();
      
      return 'd';
};

And imports it as ./changelog.js?count=1 on first test, then on second test:

import {
    readFile
} from 'fs/promises'

const {
    execSync,
} = global.__mockImportCache.get('fs/promises');

export default (a, b, c) => {
    if (a)
        return readFile();
     
      if (c)
          return execSync();
      
      return 'd';
};

File imported as ./changelog.js?count=2, and then on third assertion code isn’t changed.

About this issue

Original URL
State: open
Created 3 years ago
Reactions: 2
Comments: 19 (15 by maintainers)

Commits related to this issue

feature(mock-import) add support of sourcemaps (bcoe/c8#325) — committed to coderaiser/mock-import by coderaiser 3 years ago
feature: report: ability to use dynamic source maps for `?count=x` imported files (#325) — committed to coderaiser/c8 by coderaiser 2 years ago

Most upvoted comments

This is amazing! Would be great if you add this to c8 😃

Yes I’d be open to adding this functionality, the only issue is it does draw attention to the fact that merging multiple istanbul reports is a bit buggy.

The function and line coverage should be pretty accurate, I think, but as demonstrated by the yellow blocks in the report I shared, branch coverage is a bit off.

All I did was stop dropping the ? suffix:

diff --git a/lib/report.js b/lib/report.js
index d3c8806..41edabf 100644
--- a/lib/report.js
+++ b/lib/report.js
@@ -117,6 +117,7 @@ class Report {
           map.merge(converter.toIstanbul())
         }
       } catch (err) {
+        console.info(err)
         debuglog(`file: ${v8ScriptCov.url} error: ${err.stack}`)
       }
     }
@@ -143,7 +144,12 @@ class Report {
    */
   _getSourceMap (v8ScriptCov) {
     const sources = {}
-    const sourceMapAndLineLengths = this.sourceMapCache[pathToFileURL(v8ScriptCov.url).href]
+    let suffix = '';
+    const match = v8ScriptCov.url.match(/(?<query>\?.*)$/)
+    if (match) {
+      suffix = match.groups.query;
+    }
+    const sourceMapAndLineLengths = this.sourceMapCache['file://' + v8ScriptCov.url]
     if (sourceMapAndLineLengths) {
       // See: https://github.com/nodejs/node/pull/34305
       if (!sourceMapAndLineLengths.data) return
@@ -279,15 +285,22 @@ class Report {
       }
       if (/^file:\/\//.test(v8ScriptCov.url)) {
         try {
-          v8ScriptCov.url = fileURLToPath(v8ScriptCov.url)
-          fileIndex.add(v8ScriptCov.url)
+          let suffix = '';
+          const match = v8ScriptCov.url.match(/(?<query>\?.*)$/)
+          if (match) {
+            suffix = match.groups.query;
+          }
+          const normalized = fileURLToPath(v8ScriptCov.url)
+          v8ScriptCov.url = normalized + suffix;
+          fileIndex.add(normalized)
         } catch (err) {
           debuglog(`${err.stack}`)
           continue
         }
       }
       if ((!this.omitRelative || isAbsolute(v8ScriptCov.url))) {
-        if (this.excludeAfterRemap || this.exclude.shouldInstrument(v8ScriptCov.url)) {
+        const url = v8ScriptCov.url.split('?')[0]
+        if (this.excludeAfterRemap || this.exclude.shouldInstrument(url)) {
           result.push(v8ScriptCov)
         }
       }
@@ -307,7 +320,12 @@ class Report {
   _normalizeSourceMapCache (v8SourceMapCache) {
     const cache = {}
     for (const fileURL of Object.keys(v8SourceMapCache)) {
-      cache[pathToFileURL(fileURLToPath(fileURL)).href] = v8SourceMapCache[fileURL]
+      let suffix = '';
+      const match = fileURL.match(/(?<query>\?.*)$/)
+      if (match) {
+        suffix = match.groups.query;
+      }
+      cache[pathToFileURL(fileURLToPath(fileURL)).href + suffix] = v8SourceMapCache[fileURL]
     }
     return cache
   }

The approach should be fleshed out more with tests, and using a helper rather than repeated code – also it fails if no source is found right now, since v8-to-istanbul will try to load coverage.js?=foo.bar from disk (this should probably be fixed in v8-to-istanbul, with it dropping the suffix perhaps there?).

bcoe on Jan 3, 2022

It would be outstanding if c8 would differentiate moduleIds requested with query string to provide accurate coverage results.

@iambumblehead agreed. Would happily accept a patch for this, or even a failing test.

@bcoe https://github.com/bcoe/c8/pull/501

iambumblehead on Nov 21, 2023

I think we’re running into this issue (or something very similar) trying to run tests with TypeScript / ESM / tsx as well: it seems to incorrectly report import statements as branches:

alecgibson on Jun 9, 2023

I hacked together an approach that merges three coverage reports at the end, rather than merging the v8 output at the start, it kind of works:

However, it has trouble merging branches, this isn’t unexpected because SourceMaps provide sparse data, so trying to combine two source maps can be like comparing apples and oranges.

A better way to fix your problem, I think, would be to figure out where the 30 byte discrepancy is happening in loader-3, vs., loader-2 and loader-1. If you can get these byte ranges to correlate, c8 will start working – my guess is a header is being injected by 1/2 but is not being injected for 3 … perhaps a different function wrapper?

bcoe on Jan 2, 2022