jest-junit: "Test suite failed to run" does not report as a test failure in the xml output

Considering the following console output

 PASS  __tests__\src\common\services\index.test.js
 PASS  __tests__\src\common\services\support.test.js
 FAIL  __tests__\src\features\product\routers\product.js
  ● Test suite failed to run

    brand-engine\src\common\react-app\src\common\templates\cta.mustache:1
    ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,global,jest){<a class="hello" href="{{ href }}">{{ hrefText }}</a>
                                                                                             ^

    SyntaxError: Unexpected token <

      4 | import FrameworkTemplate from '../../../common/framework-template';
      5 |
    > 6 | import * as cta from '../../../common/templates/cta.mustache';
      7 |
      8 | class Home extends React.Component {
      9 |       render() {

      at ScriptTransformer._transformAndBuildScript (node_modules/jest-runtime/build/script_transformer.js:307:17)
      at Object.<anonymous> (src/common/react-app/src/features/test/components/home.js:6:38)
      at Object.<anonymous> (src/common/react-app/src/app.js:6:39)
 PASS  __tests__\src\common\framework-engine\index.test.js

There is no output in the junit.xml referencing product.js test suite providing no indication it failed. Everything else is in there as expected.

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 29 (4 by maintainers)

Most upvoted comments

I’m seeing this issue rearing it’s ugly head again in a new way. Tests with errors are simply not recorded.

Did you set reportTestSuiteErrors?

"jest-junit": {
    "reportTestSuiteErrors": "true"
  }

SamTheisens on Apr 15, 2022

It’s not exactly just a jest issue and not exactly just a jest-junit issue. A combination of both.

If your test suite fails to run and I write that information to a junit.xml file what would I write exactly? Jest does is not able to tell me what test suite(s) failed or what individual tests contain syntax errors (for example).

Only details I have as a reporter is what file path(s) failed to run.

And given that you can configure jest-junit to write junit.xml files in different ways, not necessarily containing file path, if I attempt to write anything it could just confuse whatever reports and records your tests.

This is why I don’t do it. But if you can think of a way to approach this while retaining the users configuration then feel free to submit a PR.

How about making this an option for people who don’t do any complicated configuration. Just drop the failed test suite name into the junit xml. If people don’t like it, then they don’t have to enable it.

Blacktiger on May 21, 2020

@SamTheisens No, we sure didn’t. Thank you.

matthewoden on Apr 15, 2022

There isn’t an issue with jest that I’m aware of filing this as a bug or a feature request. But the issue I’m linking here is probably the most relevant to the core of the problem.

https://github.com/facebook/jest/issues/6616

palmerj3 on Dec 12, 2019

Ah, sorry I misunderstood the timeline.

It’s certainly true that it’s impossible to generate a junit.xml file that says “Test suite x,y,z failed” by the very nature of the bug.

However, I’m not sure that this is a necessary product requirement for this bug; users wish for their CI to fail and provide a clue as to where to look, and are likely to accept that a list of test suite names can’t be provided if the test suite setup fails.

Internally, we have a workaround that generates a single testsuite, each with a single testcase, for each testResult with a testExecError containing the failureMessage as CDATA. From what I understand of a quick skim of the PR, it does a much more elegant job of a similar approach.

It may be uncomfortable to invent a "system-err" testsuite out of thin air, especially since it won’t mesh with CI failure history (a feature we also use), but given that https://github.com/facebook/jest/issues/5958 hasn’t made progress, it may be a better tradeoff for the time being. Users at least may be able to see a history of test suites which fail to run!

rattrayalex-stripe on Dec 5, 2019