cypress: 3.0 - custom reporters broken now that tests are run in isolation

If anyone tries running from the command line (or through package.json script), you cannot pass the [hash] param as --reporter-options argument. Instead, you need to add the following to your cypress.json:

 "reporter": "junit",
    "reporterOptions": {
        "mochaFile": "cypress/results/results-[hash].xml"
    }

@abataub how do you know when the run is finished? You use our module API, which will yield you all of the results or you simply chain something off of cypress run like cypress run && echo "finished"

@joshmccure nothing for this will be done with 3.0.2 because other than mochawesome we’re not seeing anything that is a fire that needs putting out. We have talked internally about exposing a service which will aggregate the results for you and spit out junit or mochawesome or whatever it is you want.

The reason this likely needs to be moved to the service level is because of parallelization releasing in 3.1.0. In that case the reports will not only be split out by spec file but they will be split out across machines and there won’t be an easy way to aggregate the results except if you have a service that knows everything about the run. That’s the logical place to put it. For now you can also see all of the results aggregated correctly in the Dashboard - although I do understand the desire to want to have junit or mochawesome reports.

@knoopx did you find what you were looking for? With the junit reporter you can pass [hash] in the filename so that it will not overwrite the previous ones. Then you can use a tool to combine them if necessary.

Well, testcafe is $499 per developer, this is not an option for everyone out there.

Incorrect. Testcafé is free and open source: https://devexpress.github.io/testcafe/faq/#what-is-the-difference-between-a-paid-and-an-open-source-testcafe-version-what-is-testcafe-studio

It also has proper TypeScript support, supports multiple browsers and supports parallel testing.

@Hollander I would start another issue that focuses around not being able to use [hash] in the module API. The issue identified in this thread is more general, thus the Cypress team’s rationale for marking it wontfix makes sense.

Ah, I’ve got it. It works as an option in cypress.json.

We could use mochawesome’s reporterFilename, but Cypress plugins don’t support changing config per file/test suite.

A quick win would be implementing in mochawesome a [file] wildcard just like jUnit’s [hash].

I have mochawesome working here -> https://github.com/testdrivenio/cypress-mochawesome-s3

Essentially-

1. cypress runs
2. each spec generates a new report
3. a custom script combines each mochawesome json file and calculates the stats
4. html report is then generated

Thanks everyone. Is there a way to pass the name of the given spec instead of the hash into the file name?

@HugoGiraudel Couldn’t you do something like this…

// support/index.js

after(() => {
  if (Cypress.env('COVERAGE') {
    cy.window().its('__coverage__').then(cov) => {
      const spec = window.location.hash
      cy.writeFile(`cypress/coverage/${spec}-coverage.json`, cov)
    })
  }
})

And then after the run just combine all the coverage reports with standard node code.

@adubatl Hi there. Any chance you’d share your code regarding coverage aggregation? 😃

@artemshoiko You can just exec rm -rf mochawesome-report to delete the report folder, or use rimraf npm package for this if you wish to stay cross-platform.

I personally use fs-extra to remove a folder from a JS module.

Hey kids, I may have code to help those not using Mochaawesome.

At work, I need to generate a report of all end to end tests that passed/failed for audit purposes. In Protractor, I’d just put resultJsonOutputFile: 'report.json' in the configuration, and it “just worked”. With Cypress, this would only work if you had all your files in 1 file.

So I created a custom reporter to combine them. It’s not a library (yet?) but for those of you struggling to combine the reports a single reporter generates, perhaps the code can help you provide a reference on how you’d do it yourself.

https://github.com/JesterXL/cypress-report-combine-example

Specifically the combine function: https://github.com/JesterXL/cypress-report-combine-example/blob/master/reporters/combine-reports.js#L88

It:

1. nukes the reports folder rm -rf reports
2. then run cypress and our custom reporter writes a uniquely named report (for now just status, test name, and time it took to run)cypress run
3. when done, you’ll have a reports folder full of files that have the test results for each test in JSON. To combine them, you just pass the folder name, and this script will combine 'em then delete the interim report files: node reporters/combine-reports.js --reportsFolder=reports

Hopefully this is helpful. I know many people are using existing reporters that broken once Cypress went isolated, but they mentioned in #1946 that they may fix this in the future.

Any news regarding to mochawesome report? I am getting only the last report out of the whole suite.

@Hollander the modules API does not currently support the [hash] option - you need to provide it in your cypress.json. It should be picked up when you run through the module API, however. This is the solution my team is currently using for testing during CI.

The module API is here. You can console.log the .then() so you can see what is output.

• https://on.cypress.io/module-api
• https://github.com/cypress-io/cypress/issues/1248

@SeriousM

this feature is not new, it’s broken instead.

…

A mode to run the tests in serial on one machine to return a single report should be added.

It seems like you’ve been convinced this is a feature. I don’t think people would mind if you opened up an issue for that to be implemented.

@Bkucera

or have a stable behavior

I expect to have a stable execution of one test or 100, on one machine or on 30. Stability shouldn’t be an option.

@SeriousM you couldn’t be more wrong, and nobody appreciates your presumptive comments.

@brian-mann I guess you missed my point and I hope I can change that: I’m a new user to cypress and the first-time experience was not quite that good as “advertised”. The first thing to do when I look at new technology is to read the documentation and try it in the described way.

The docker-support is very bad documented, the docker images don’t have cypress in it. The “mochawesome” (as mentioned in the documentation) reporter can’t work as it supposed to be (since 3.x) and junit reports need the special “[hash]” in the filename. These points aren’t documented and I worked them out in many hours.

Don’t get me wrong, I thing cypress is a very nice tool and I very like the way it changes the way how e2e tests are done but the documentation and get-started-pages does not really help you after the “start ide and try it”-point.

it was effectively communicated and the idea of the reporters being split up was even accounted for as a downside of doing this

how should I know that something was communicated before when I’m a new user?

You could submit a PR to mochawesome to have them do this by default.

Already discussing the problem.

or you could just not upgrade and stay on 2.1.0 forever if you’d like.

I see your “go f*** yourself” here, thank you.

With the demand for parallelization and faster CI runs, no matter what - we will have to solve this problem

A problem that a small team or new project never will have for months or years.

because a single machine will not have all of the details on its physical drive to stitch together the results.

I think the decision to enforce parallel execution (thus, split reports) was not well thought thru in terms to support all users, not just the big ones with your dashboard. A mode to run the tests in serial on one machine to return a single report should be added.

@brian-mann: I would like to discuss these points, not to offend nor get offended with strong words. I hope you want the same.

@brian-mann Whoops, that is what I intended to write.

I’m not having any luck. Trying on a different computer using the last three example specs. Running npx cypress run --reporter junit --reporter-options "mochaFile=results/my-test-output.[hash].xml,toConsole=true" on the command line and the resulting file is `results/my-test-output.hash.xml’

On the previous computer, I attempted using the module api with no success.