angular-cli: Angular Cli: 6.0.1 code coverage thresholds not failing ng test task

After upgrade to Angular Cli 6.x the code coverage thresholds enforced by the karma-coverage-istanbul-reporter do not make the testing command fail with emitWarning set to false as they used to when using Angular Cli 6.0.1.

Versions

Angular CLI: 6.0.1
Node: 8.9.1
OS: win32 x64
Angular: 6.0.1
... animations, cli, common, compiler, compiler-cli, core, forms
... http, platform-browser, platform-browser-dynamic
... platform-server, router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.6.1
@angular-devkit/build-angular     0.6.1
@angular-devkit/build-optimizer   0.6.1
@angular-devkit/core              0.6.1
@angular-devkit/schematics        0.6.1
@ngtools/webpack                  6.0.1
@schematics/angular               0.6.1
@schematics/update                0.6.1
rxjs                              6.1.0
typescript                        2.7.2
webpack                           4.6.0

Repro steps

Create a repo using Angular Cli 1.7.x
Enforce thresholds
See the test command failing
Upgrade Angular Cli and migrate

Observed behavior

The error is emitted by the reporter, however seems ignored by the angular client.

Desired behavior

The test command should be failing when thresholds not met as it used to.

Mention any other details that might be useful (optional)

Tried with both karma@1.7.x and 2.x, reporter version 0.x and 1.x. Seems that stops working as soon as the upgrade of Angular Client is done. Same repo with Angular Client 1.7 behaves correctly.

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 16
Comments: 23 (1 by maintainers)

Most upvoted comments

Found a fix for this:

Remove coverage-istanbul from your reporters array in karma.conf and it will use the correct exit code.

It seems that coverage-istanbul is registered automatically when codeCoverage is set to true in your configuration and when it was being called twice it was causing issues.

SlashmanX on Nov 6, 2018

Same issue here, currently preventing our regular CI workflow from running as expected

bennyadam on Jun 11, 2018

After trying everything, I was able to make it work by upgrading @angular-devkit/build-angular to 0.7.5 as per @alexw10 suggestion. My config looks like this:

coverageIstanbulReporter: {
  verbose: true,
  dir: require('path').join(__dirname, 'coverage'),
  reports: ['html', 'lcovonly'],
  fixWebpackSourcePaths: true,
  thresholds: {
    emitWarning: false, // <- this is important to make karma fail
    global: {
      statements: 85,
      lines: 85,
      branches: 85,
      functions: 85
    }
  }
}

Also, don’t forget to set singleRun to false when running the tests with --code-coverage.

singleRun: config.angularCli && config.angularCli.codeCoverage

croso-df on Oct 16, 2018

@codeNoobie you need to install latest version of the: "@angular-devkit/build-angular": "~0.7.1",

alexw10 on Jul 30, 2018

Same issue with the exit code being 0. The output:

⇒  ng test --browsers Chrome --code-coverage=true --watch=false; echo $?
11 07 2018 14:34:44.604:INFO [karma]: Karma v2.0.4 server started at http://0.0.0.0:9876/
11 07 2018 14:34:44.606:INFO [launcher]: Launching browser Chrome with unlimited concurrency
11 07 2018 14:34:44.611:INFO [launcher]: Starting browser Chrome
11 07 2018 14:34:51.569:INFO [Chrome 67.0.3396 (Mac OS X 10.12.6)]: Connected on socket hvsiXwNm6-QYSfFIAAAA with id 90762867
Chrome 67.0.3396 (Mac OS X 10.12.6): Executed 26 of 28 (skipped 2) SUCCESS (0.4 secs / 0.332 secs)
TOTAL: 26 SUCCESS

=============================== Coverage summary ===============================
Statements   : 56.86% ( 348/612 )
Branches     : 12.67% ( 19/150 )
Functions    : 34.22% ( 64/187 )
Lines        : 56.89% ( 293/515 )
================================================================================
11 07 2018 14:34:54.042:ERROR [reporter.coverage-istanbul]: Coverage for statements (56.86%) does not meet global threshold (100%)
11 07 2018 14:34:54.042:ERROR [reporter.coverage-istanbul]: Coverage for lines (56.89%) does not meet global threshold (100%)
11 07 2018 14:34:54.042:ERROR [reporter.coverage-istanbul]: Coverage for branches (12.67%) does not meet global threshold (100%)
11 07 2018 14:34:54.042:ERROR [reporter.coverage-istanbul]: Coverage for functions (34.22%) does not meet global threshold (100%)
0

And the ng --version output:

Angular CLI: 6.0.8
Node: 8.9.4
OS: darwin x64
Angular: 6.0.3
... animations, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.6.8
@angular-devkit/build-angular     0.6.8
@angular-devkit/build-optimizer   0.6.8
@angular-devkit/core              0.0.29
@angular-devkit/schematics        0.0.52
@angular/cli                      6.0.8
@ngtools/json-schema              1.1.0
@ngtools/webpack                  6.0.8
@schematics/angular               0.6.8
@schematics/update                0.6.8
ng-packagr                        2.2.0
rxjs                              6.1.0
typescript                        2.7.2
webpack                           4.8.3

cheshirsky on Jul 11, 2018

We are running into the same issue. Status code of the ng test --code-coverage will always be 0. We are heavily relying on this feature in our local and CI environment.

mboughaba on May 29, 2018