cypress-parallel: Question about: Some test suites likely terminated without passing on results

Any update on this one ? Facing this issue and there is no information on why this is happening

Since we don’t get answers here I’ve been digging through the code because I think this plugin is really nice and I think I found what the issue was for us…

First of all, the error I was getting is:

Found test suites does not match results. 
Test suites found: 68 
Test suite results: 0 
Some test suites likely terminated without passing on results, exit with error
The following tests suites are missing results: { test1, test2, ... all 68 files }

So, it was false failing even if all the tests passed and the statement triggering the error is this: https://github.com/tnicola/cypress-parallel/blob/710dd191692c8f11c2823183fcfa36fa81fcd91e/lib/cli.js#L99

To make shorter the story, after debugging it I realized that for “some” unknown reason none of the tests was generating the .json file needed and created automatically by this plugin in order to output the results, it also creates a runner-results folder for that but basically it was empty after any execution and that’s why Test suite results: 0 (timeMap.size). I continued debugging and found where this issue started to happen:

We have mochawesome, mocha-junit-reporter and cypress-multi-reporters, so our code looked like this:

cypress v10.11 mochawesome v6.2.2 cypress-multi-reporters v1.6.1 mocha-junit-reporter v2.0.2 cypress-parallel v0.13.0

Piece of our cm-desktop.config.js file
module.exports = defineConfig({
	e2e: {
	        specPattern: 'cypress/user/*.test.js',
		reporter: 'cypress-multi-reporters',
		reporterOptions: {
			configFile: 'cypress/config/user/reporter-config.json',
		}
	}
});

Our reporter-config.json file
{
  "reporterEnabled":"mochawesome, mocha-junit-reporter",
  "mochaJunitReporterReporterOptions":{
    "mochaFile":"cypress/results/user/dsk/user-[hash].xml"
  },
  "mochawesomeReporterOptions":{
    "reportDir":"cypress/results/user/dsk",
    "overwrite":false,
    "html":false,
    "json":true
  }
}

Our package.json scripts
"cm:dsk:user": "cypress run --config-file ./cypress/config/user/cm-desktop.config.js --browser chrome",
"cm:run:dsk:user:parallel": "cypress-parallel -s cm:dsk:user -t 4 -d ./cypress/user",

^ That way, I was getting the error… So, after almost a full day of trial and error, I found that when I was executing this way then the multi-reporter-config.json file generated by default by the plugin was writing this lines:

{
  "reporterEnabled": "cypress-parallel/json-stream.reporter.js, cypress-parallel/simple-spec.reporter.js"
}

So, it immediately made me think that my reporter and reporterOptions flag was not being read… why? I don’t know the answer, I thought it should have worked but looks like it’s not working in that way, maybe a bug?? anyway… this statement confirmed my theory: https://github.com/tnicola/cypress-parallel/blob/710dd191692c8f11c2823183fcfa36fa81fcd91e/lib/thread.js#L36 effectively it was not recognizing my reporting flags.

Note: I tried passing -r and -o and a lot of ways/combinations but can’t make it work.

The solution? … I made it work by doing this:

1. Removed the reporter and reporterOptions flags from the cm-desktop.config.js file (yes, deleted those lines from there)
2. Our reporter-config.json file now looks like this:

{
  "reporterEnabled":"mochawesome, cypress-parallel/json-stream.reporter.js, cypress-parallel/simple-spec.reporter.js, mocha-junit-reporter",
    "mochawesomeReporterOptions":{
      "reportDir":"cypress/results/user/dsk",
      "overwrite":false,
      "html":false,
      "json":true
    },
  "cypressParallelSimpleSpecReporterJsReporterOptions": {
    "reportDir": "runner-results"
  },
  "mochaJunitReporterReporterOptions":{
    "mochaFile":"cypress/results/user/dsk/user-[hash].xml"
  }
}

1. Our package.json now look like this:

"cm:dsk:user": "cypress run --config-file ./cypress/config/user/cm-desktop.config.js --browser chrome",
"cm:run:dsk:user:parallel": "cypress-parallel -s cm:dsk:user -t 4 -d ./cypress/user -p 'cypress/config/user/reporter-config.json'",

I found that -p flag in the settings file: https://github.com/tnicola/cypress-parallel/blob/710dd191692c8f11c2823183fcfa36fa81fcd91e/lib/settings.js#L50 which is not documented in README but it was the correct for us, now I’m not passing a reporter flag and letting the reporter-config.json to drive the enabled reporters.

Hope it helps anyone to solve the issue, I don’t know if it’s a bug or a miss by me while following instructions but that’s what it is.

cc @tnicola

@tnicola is this package abandoned? Maybe someone has working fork?

This script should using glob/pattern for spec files, there is no reason to do this differently than Cypress itself. I’m using Cypress for component testing, and like many other people I have spec files together with component files. All my specs are passing, but I’m still getting false error from cypress-parallel 😕

The issue “Some test suites likely terminated without passing on results” can happen for valid reasons, such as the reason in the error message. However, this is a bug and essentially makes this library unusable without forking. We can verify that the test suite output results and the JSON file exist, but in the mocha folder, there is no JSON for it. As mentioned by another poster above, the tests are overwritten and they had to modify the source to use different folders. It would be nice to get a bug fix, but from what I gather, we’re on our own if we decide to use this library.

hi @guychienll in lib/cli.js you can find such line of code const resultsPath = path.join(process.cwd(), 'runner-results'); So here the name of directory for results is hardcoded. I’ve added here parameter that could be added from command line, so now if I use this plugin for different types of tests run in one workspace it wrote results in different directories. After this we don’t have problems with empty or not full results

So changes in settings.js

• add option for CMD .option('runnerResults', { alias: 'x', type: 'string', description: 'Path where cypress results will be located' }
• add read and default value for it runnerResults: argv.runnerResults ? argv.runnerResults : 'runner-default',

in cli.js: const resultsPath = path.join(process.cwd(), settings.runnerResults);

in shared-config.js: const resultsPath = path.join(process.cwd(), ${settings.runnerResults});

For us it was an issue with different threads that used the same results folder. After small update to put results in different folders for different threads (unfortunately it’s hardcoded in source lib) we didn’t meet these problem. For now 2nd month without this issue

I passed correct tests directory with correct regexp and in 95% tests pass without any issues. But sometimes randomly I’ve got results like Found test suites does not match results. Test suites found: 71 Test suite results: 61 Some test suites likely terminated without passing on results, exit with error

It mentions specific suites that were not found, but I see results for these suites. Any ideas why it could happen?