cypress-parallel: Question about: Some test suites likely terminated without passing on results
Hi,
First off, thanks for the repo and work š
So i want to use this package in our tests, but am seeing the following error randomly on my test runs:
Some test suites likely terminated without passing on results
I know that itās a validation, but do we know why this is happening?
Thanks, -arnon
About this issue
- Original URL
- State: open
- Created 5 years ago
- Reactions: 7
- Comments: 25
Since we donāt get answers here Iāve been digging through the code because I think this plugin is really nice and I think I found what the issue was for usā¦
First of all, the error I was getting is:
So, it was false failing even if all the tests passed and the statement triggering the error is this: https://github.com/tnicola/cypress-parallel/blob/710dd191692c8f11c2823183fcfa36fa81fcd91e/lib/cli.js#L99
To make shorter the story, after debugging it I realized that for āsomeā unknown reason none of the tests was generating the
.jsonfile needed and created automatically by this plugin in order to output the results, it also creates a runner-results folder for that but basically it was empty after any execution and thatās whyTest suite results: 0 (timeMap.size). I continued debugging and found where this issue started to happen:We have mochawesome, mocha-junit-reporter and cypress-multi-reporters, so our code looked like this:
cypress v10.11 mochawesome v6.2.2 cypress-multi-reporters v1.6.1 mocha-junit-reporter v2.0.2 cypress-parallel v0.13.0
^ That way, I was getting the error⦠So, after almost a full day of trial and error, I found that when I was executing this way then the
multi-reporter-config.jsonfile generated by default by the plugin was writing this lines:So, it immediately made me think that my
reporterandreporterOptionsflag was not being read⦠why? I donāt know the answer, I thought it should have worked but looks like itās not working in that way, maybe a bug?? anyway⦠this statement confirmed my theory: https://github.com/tnicola/cypress-parallel/blob/710dd191692c8f11c2823183fcfa36fa81fcd91e/lib/thread.js#L36 effectively it was not recognizing my reporting flags.Note: I tried passing
-rand-oand a lot of ways/combinations but canāt make it work.The solution? ⦠I made it work by doing this:
reporterandreporterOptionsflags from thecm-desktop.config.jsfile (yes, deleted those lines from there)reporter-config.jsonfile now looks like this:package.jsonnow look like this:I found that
-pflag in the settings file: https://github.com/tnicola/cypress-parallel/blob/710dd191692c8f11c2823183fcfa36fa81fcd91e/lib/settings.js#L50 which is not documented in README but it was the correct for us, now Iām not passing a reporter flag and letting thereporter-config.jsonto drive the enabled reporters.Hope it helps anyone to solve the issue, I donāt know if itās a bug or a miss by me while following instructions but thatās what it is.
cc @tnicola
Any update on this one ? Facing this issue and there is no information on why this is happening
@tnicola is this package abandoned? Maybe someone has working fork?
This script should using glob/pattern for spec files, there is no reason to do this differently than Cypress itself. Iām using Cypress for component testing, and like many other people I have spec files together with component files. All my specs are passing, but Iām still getting false error from cypress-parallel š
The issue āSome test suites likely terminated without passing on resultsā can happen for valid reasons, such as the reason in the error message. However, this is a bug and essentially makes this library unusable without forking. We can verify that the test suite output results and the JSON file exist, but in the mocha folder, there is no JSON for it. As mentioned by another poster above, the tests are overwritten and they had to modify the source to use different folders. It would be nice to get a bug fix, but from what I gather, weāre on our own if we decide to use this library.
hi @guychienll in lib/cli.js you can find such line of code
const resultsPath = path.join(process.cwd(), 'runner-results');So here the name of directory for results is hardcoded. Iāve added here parameter that could be added from command line, so now if I use this plugin for different types of tests run in one workspace it wrote results in different directories. After this we donāt have problems with empty or not full resultsSo changes in settings.js
.option('runnerResults', { alias: 'x', type: 'string', description: 'Path where cypress results will be located' }runnerResults: argv.runnerResults ? argv.runnerResults : 'runner-default',in cli.js:
const resultsPath = path.join(process.cwd(), settings.runnerResults);in shared-config.js:
const resultsPath = path.join(process.cwd(),${settings.runnerResults});For us it was an issue with different threads that used the same results folder. After small update to put results in different folders for different threads (unfortunately itās hardcoded in source lib) we didnāt meet these problem. For now 2nd month without this issue
Hi, Just in case anyone is still having an issue with the error @olga-govor and @aram-bigid mentioned. I was getting the junit and mochawesome reports, but was only having the issue with the exit code / error message. After reading everything here and then re-reading the README document I tried setting
--strictModeto false and my tests correctly failed or passed regardless of the total number of tests found.Example package.json script:
cypress-parallel -s cy:run -t 4 -d cypress/e2e -r 'cypress-multi-reporters' -o 'configFile=reporter-config.json' --strictMode false",Hope this helps others.
I passed correct tests directory with correct regexp and in 95% tests pass without any issues. But sometimes randomly Iāve got results like
Found test suites does not match results. Test suites found: 71 Test suite results: 61 Some test suites likely terminated without passing on results, exit with errorIt mentions specific suites that were not found, but I see results for these suites. Any ideas why it could happen?
Mahanthi-arun-kumar, I believe your problem is in the command line for cypress-parallel.
ā-d ā./srcāā tells cypress-parallel to look in your src directory for cypress test suite files. cypress-parallel treats every file it finds as a test suite file. Hence āTest suites found: 1036ā that is every code file in your src directory. in your -a arguments passed to cypress itself, you declare testFiles=**/integration-test/*cypress.js" which tells cypress where to find test suite files. further, cypress can distinguish between files that contain tests and files which do not. This accounts for Test suite results: 44 You likely only have 44 test suites. But cypress-parallel thinks you have 1036 test suites, and so expects 1036 test suite results. Upon getting less than 1036 results, it throws an error. If you only pass directories which contain only cypress test suite files to cypress-parallel, that should fix your problem. I ran into this same issue and that fixed it for me.