nyc: Merging coverage files doesn't produce accurate cumulative result

I execute the following nyc commands, after generating individual coverage reports on the same codebase using different test executions run by Jest and Cypress.

nyc merge coverage-jest combined/coverage-jest.json nyc merge coverage-cypress combined/coverage-cypress.json nyc report combined --reporter html --reporter text

Expected Behavior

I’d expect this to show a correct cumulative coverage report of .ts and .html files.

Observed Behavior

  1. Coverage of .html files is not shown in merged report, while they are show in individial report.
  2. Coverage of some .ts files is shown correctly, but for others a Cannot read property 'locations' of undefined TypeError is shown.

Troubleshooting steps

  • still occurring when I put cache: false in my nyc config

Environment Information

  System:
    OS: Windows 10 10.0.18363
    CPU: (8) x64 Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz
    Memory: 7.97 GB / 31.88 GB
  Binaries:
    Node: 13.6.0 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.0 - C:\Program Files\nodejs\yarn.CMD
    npm: 6.13.4 - C:\Program Files\nodejs\npm.CMD
  npmPackages:
    istanbul-instrumenter-loader: 1.2.0 => 1.2.0
    istanbul-lib-coverage: 1.2.1 => 1.2.1
    nyc: ^15.0.1 => 15.0.1
    source-map-support: ^0.5.16 => 0.5.16
    ts-node: ^7.0.1 => 7.0.1
    typescript: ~3.7.4 => 3.7.5

For Cypress coverage I use above dependencies as well as @cypress/code-coverage:^3.0.2. I’m using jest: 24.1.0 to produce unit test coverage.

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 4
  • Comments: 22 (1 by maintainers)

Commits related to this issue

Most upvoted comments

I’m experiencing this same problem.

My coverage results look like so:

cypress/coverage-final.json:

{
  "path/to/file.ts": {
    "path": "path/to/file.ts",
    "statementMap": {
      "0": {
        "start": { "line": 4, "column": 35 },
        "end": { "line": 4, "column": 66 }
      },
      "1": {
        "start": { "line": 5, "column": 2 },
        "end": { "line": 7, "column": 3 }
      },
      "2": {
        "start": { "line": 6, "column": 4 },
        "end": { "line": 6, "column": 29 }
      },
      "3": {
        "start": { "line": 8, "column": 2 },
        "end": { "line": 8, "column": 14 }
      },
      "4": {
        "start": { "line": 1, "column": 0 },
        "end": { "line": 1, "column": 65 }
      },
      "5": {
        "start": { "line": 5, "column": 2 },
        "end": { "line": 7, "column": null }
      },
      "6": {
        "start": { "line": 3, "column": 0 },
        "end": { "line": 3, "column": 24 }
      }
    },
    "fnMap": {
      "0": {
        "name": "getRoot",
        "decl": {
          "start": { "line": 3, "column": 24 },
          "end": { "line": 3, "column": 31 }
        },
        "loc": {
          "start": { "line": 3, "column": 47 },
          "end": { "line": 9, "column": 1 }
        },
        "line": 3
      },
      "1": {
        "name": "getRoot",
        "decl": {
          "start": { "line": 3, "column": 24 },
          "end": { "line": 3, "column": 31 }
        },
        "loc": {
          "start": { "line": 3, "column": 31 },
          "end": { "line": 9, "column": 1 }
        }
      }
    },
    "branchMap": {
      "0": {
        "loc": {
          "start": { "line": 5, "column": 2 },
          "end": { "line": 7, "column": 3 }
        },
        "type": "if",
        "locations": [
          {
            "start": { "line": 5, "column": 2 },
            "end": { "line": 7, "column": 3 }
          },
          { "start": {}, "end": {} }
        ],
        "line": 5
      },
      "1": {
        "loc": {
          "start": { "line": 5, "column": 2 },
          "end": { "line": 7, "column": null }
        },
        "type": "if",
        "locations": [
          {
            "start": { "line": 5, "column": 2 },
            "end": { "line": 7, "column": null }
          }
        ]
      }
    },
    "s": { "0": 6, "1": 4, "2": 2, "3": 4, "4": 2, "5": 2, "6": 2 },
    "f": { "0": 4, "1": 2 },
    "b": { "0": [0, 4], "1": [2] },
    "_coverageSchema": "1a1c01bbd47fc00a2c39e90264f33305004495a9",
    "hash": "81128de9166cac88dff871fa724f973dcd80167a"
  }
}

jest/coverage-final.json:

{
  "path/to/file.ts": {
    "path": "path/to/file.ts",
    "statementMap": {
      "0": {
        "start": { "line": 1, "column": 0 },
        "end": { "line": 1, "column": 65 }
      },
      "1": {
        "start": { "line": 4, "column": 35 },
        "end": { "line": 4, "column": 66 }
      },
      "2": {
        "start": { "line": 5, "column": 2 },
        "end": { "line": 7, "column": null }
      },
      "3": {
        "start": { "line": 6, "column": 4 },
        "end": { "line": 6, "column": 29 }
      },
      "4": {
        "start": { "line": 8, "column": 2 },
        "end": { "line": 8, "column": 14 }
      },
      "5": {
        "start": { "line": 3, "column": 0 },
        "end": { "line": 3, "column": 24 }
      }
    },
    "fnMap": {
      "0": {
        "name": "getRoot",
        "decl": {
          "start": { "line": 3, "column": 24 },
          "end": { "line": 3, "column": 31 }
        },
        "loc": {
          "start": { "line": 3, "column": 31 },
          "end": { "line": 9, "column": 1 }
        }
      }
    },
    "branchMap": {
      "0": {
        "loc": {
          "start": { "line": 5, "column": 2 },
          "end": { "line": 7, "column": null }
        },
        "type": "if",
        "locations": [
          {
            "start": { "line": 5, "column": 2 },
            "end": { "line": 7, "column": null }
          }
        ]
      }
    },
    "s": { "0": 1, "1": 1, "2": 1, "3": 1, "4": 0, "5": 1 },
    "f": { "0": 1 },
    "b": { "0": [1] }
  }
}

The expected outcome would be path/to/file.ts having 100% coverage, since the gaps on one test are covered by the other test. However, instead, what you see is the gaps being reported:

combined/coverage-final.json:

{
  "path/to/file.ts": {
    "path": "path/to/file.ts",
    "statementMap": {
      "0": {
        "start": { "line": 4, "column": 35 },
        "end": { "line": 4, "column": 66 }
      },
      "1": {
        "start": { "line": 5, "column": 2 },
        "end": { "line": 7, "column": 3 }
      },
      "2": {
        "start": { "line": 6, "column": 4 },
        "end": { "line": 6, "column": 29 }
      },
      "3": {
        "start": { "line": 8, "column": 2 },
        "end": { "line": 8, "column": 14 }
      },
      "4": {
        "start": { "line": 1, "column": 0 },
        "end": { "line": 1, "column": 65 }
      },
      "5": {
        "start": { "line": 5, "column": 2 },
        "end": { "line": 7, "column": null }
      },
      "6": {
        "start": { "line": 3, "column": 0 },
        "end": { "line": 3, "column": 24 }
      }
    },
    "fnMap": {
      "0": {
        "name": "getRoot",
        "decl": {
          "start": { "line": 3, "column": 24 },
          "end": { "line": 3, "column": 31 }
        },
        "loc": {
          "start": { "line": 3, "column": 47 },
          "end": { "line": 9, "column": 1 }
        },
        "line": 3
      },
      "1": {
        "name": "getRoot",
        "decl": {
          "start": { "line": 3, "column": 24 },
          "end": { "line": 3, "column": 31 }
        },
        "loc": {
          "start": { "line": 3, "column": 31 },
          "end": { "line": 9, "column": 1 }
        }
      }
    },
    "branchMap": {
      "0": {
        "loc": {
          "start": { "line": 5, "column": 2 },
          "end": { "line": 7, "column": 3 }
        },
        "type": "if",
        "locations": [
          {
            "start": { "line": 5, "column": 2 },
            "end": { "line": 7, "column": 3 }
          },
          { "start": {}, "end": {} }
        ],
        "line": 5
      },
      "1": {
        "loc": {
          "start": { "line": 5, "column": 2 },
          "end": { "line": 7, "column": null }
        },
        "type": "if",
        "locations": [
          {
            "start": { "line": 5, "column": 2 },
            "end": { "line": 7, "column": null }
          }
        ]
      }
    },
    "s": { "0": 7, "1": 4, "2": 3, "3": 4, "4": 3, "5": 3, "6": 3 },
    "f": { "0": 4, "1": 3 },
    "b": { "0": [0, 4], "1": [3] },
    "_coverageSchema": "1a1c01bbd47fc00a2c39e90264f33305004495a9",
    "hash": "81128de9166cac88dff871fa724f973dcd80167a"
  }
}

The “coverage” merge is really just concatenated instead of merging.

It appears that: line 1 characters 1-10 uncovered + line 1 characters 1-5 covered + line 1 characters 5-10 covered = line 1 characters 1-10 uncovered (wrong) + line 1 characters 1-10 covered (right).

The uncovered characters are still shown as uncovered even though their character-span is covered by a different part of the map. Just guessing.

Here is a nice visual LCOV report for the above JSON:

Cypress (fully covered except for 1 branch): image

Jest (specifically covering the above branch): image

Combined report (expected to be fully covered): image

Any advice to unblock this? I don’t want to have to write a JSON merger manually. 😓

Maybe this is just a weird issue with TypeScript.

Below, how are imports and exports not covered? The export statement isn’t covered, but the inside of the function somehow is. The inside references an import, but the import isn’t covered. Nonsensical. 😵

image

+30
quisido on Nov 4, 2021

I have the same issue just by merging multiple reports from various cypress runs in parallel. The merge is just taking the last report base on the order:

coverage-final-A.json

{
  "global.component.ts": {
    // ...same value in both files
    "s": {
      "0": 0,
      "1": 0,
      "2": 0,
      "3": 0,
      "4": 0,
      "5": 0,
      "6": 0,
      "7": 0,
      "8": 0,
      "9": 0,
      "10": 0,
      "11": 0,
      "12": 0,
      "13": 0,
      "14": 0,
      "15": 0,
      "16": 0,
      "17": 0,
      "18": 0,
      "19": 0,
      "20": 64,
      "21": 0
    },
    "f": { "0": 0 },
    "b": { "0": [0, 0] }
  }
}

coverage-final-B.json

{
  "global.component.ts": {
    // ...same value in both files
    "s": {
      "0": 77,
      "1": 77,
      "2": 5151,
      "3": 77,
      "4": 77,
      "5": 5151,
      "6": 77,
      "7": 77,
      "8": 5151,
      "9": 77,
      "10": 77,
      "11": 5151,
      "12": 77,
      "13": 77,
      "14": 5151,
      "15": 77,
      "16": 77,
      "17": 77,
      "18": 77,
      "19": 77,
      "20": 154,
      "21": 77
    },
    "f": { "0": 77 },
    "b": { "0": [77, 77] }
}

In this case, only the result of coverage-final-A.json will be taken, so without coverage. If I rename coverage-final-A.json to coverage-final-Z.json, then the order is different and the result of coverage-final-B.jsonwill be taken.

Any ideas? Should rewrite a merge myself?

+4
jogelin on Jul 20, 2022

Hi all, Maybe it will help someone: I had the same problem (merging of cypress and jest coverage wasn’t right) and I got that fixed after adding coverageProvider into jest config: coverageProvider: 'v8',

+2
mmisty on Nov 22, 2022

@D0rmouse did you finally find a good configuration to combine multiple coverage reports?

I have the same type of stack with Nx + Cypress + Storybook + Jest, generating reports separately works fine but when I want to merge, it is not good.

It makes sense because each tool are using different versions for each generation.

+2
jogelin on Aug 31, 2022

I’m encountering the same problem. When I try to merge two coverage-final.json files, the output is erroneous. I have posted a Stack Overflow question with the details here .

+1
KissBalazs on Sep 15, 2023

In my case, I fixed this issue by using the babel coverage provider in my Jest config coverageProvider: 'babel',. In my babel config I added a enviroment variable to check if the Code should be instrumented or not.

const shouldInstrumentCode = 'INSTRUMENT_CODE' in process.env

//console.log('shouldInstrumentCode', shouldInstrumentCode)



module.exports = {

    presets: ["next/babel"],

    plugins: shouldInstrumentCode? ["istanbul"]: []

  };

To run this in instrument mode, add a script like this to your package.json "dev:test": "set INSTRUMENT_CODE=1 && next dev",

This finally provides accurate coverage results for me.

+1
janow25 on Apr 24, 2023

Same issue. Any updates. Screenshots below:

Cypress Coverage - correct: Screenshot 2023-04-18 at 1 35 30 PM

Jest Coverage - correct: Screenshot 2023-04-18 at 1 35 24 PM

Combined Coverage - wrong: Screenshot 2023-04-18 at 1 35 15 PM

I’m using nyc merge and nyc report just like the original reporter of this issue. The final JSON just doesn’t appear correct.

@mmisty I tried your suggestion and it didn’t work. I end up with this instead:

Screenshot 2023-04-18 at 1 43 39 PM

+1
sscaff1 on Apr 19, 2023
Read more comments on GitHub
← nyc: JavaScript heap out of memory in v15.0.0.
nyc: `nyc` + `esm` is broken in latest NodeJS versions →