stryker-js: Stryker run succeeds when something goes wrong in the initial test run
Scenario: The one where something went wrong in the initial test run
Given something went wrong in the initial test run
When running mutation tests with stryker run
Then …
Personally, I would expect stryker run to fail but to my surprise it succeeds.
stryker.conf.js
module.exports = function(config) {
config.set({
logLevel: 'error',
mutate: [
'{src,lib}/**/*.ts?(x)',
'!{src,lib}/**/?(*.)+(spec|fixture).ts?(x)',
],
mutator: 'typescript',
transpilers: ['typescript'],
packageManager: 'npm',
reporters: ['clear-text', 'progress', 'html'],
testRunner: 'jest',
jest: {
enableFindRelatedTests: false,
config: {
preset: 'ts-jest',
testEnvironment: 'node',
resetModules: true,
},
},
thresholds: { high: 100, low: 100, break: 100 },
coverageAnalysis: 'off',
tsconfigFile: 'tsconfig.json',
htmlReporter: {
baseDir: 'reports/coverage/mutation',
},
timeoutMS: 60000,
})
}
jest.config.js
module.exports = {
preset: 'ts-jest',
testEnvironment: 'node',
collectCoverageFrom: ['**/*.ts', '!**/fixtures/**'],
coverageDirectory: 'reports/coverage/unit',
coverageThreshold: {
global: {
branches: 100,
functions: 100,
lines: 100,
statements: 100,
},
},
reporters: [
'default',
[
'jest-junit',
{
outputDirectory: '<rootDir>/reports/tests',
outputName: 'unit-test-evidence.xml',
},
],
[
'jest-html-reporter',
{
outputPath: '<rootDir>/reports/cucumber/requirements-evidence.html',
},
],
],
}
Stryker environment
├── @stryker-mutator/core@2.5.0
├── @stryker-mutator/html-reporter@2.5.0
├── @stryker-mutator/jest-runner@2.5.0
├── @stryker-mutator/typescript@2.5.0
├── jest@24.9.0
Test command
stryker run
Environment
| software | version(s) |
|---|---|
| node | v12.14.1 |
| npm | 6.13.6 |
| Operating System | macOS 10.14.6 (Mojave) |
stryker.log
Starting task 'mutation test'
+ npm run test:mutation
> stryker run
INFO ConfigReader Using stryker.conf.js in the current working directory.
INFO TypescriptConfigEditor Loading tsconfig file /path/to/project/tsconfig.json
INFO BroadcastReporter Detected that current console does not support the "progress" reporter, downgrading to "progress-append-only" reporter
INFO InputFileResolver Found 9 of 37 file(s) to be mutated.
INFO InitialTestExecutor Starting initial test run. This may take a while.
ERROR StrykerCli an error occurred Error: Something went wrong in the initial test run
at InitialTestExecutor.validateResult (/path/to/project/node_modules/@stryker-mutator/core/src/process/InitialTestExecutor.js:86:15)
at InitialTestExecutor.run (/path/to/project/node_modules/@stryker-mutator/core/src/process/InitialTestExecutor.js:41:14)
at process._tickCallback (internal/process/next_tick.js:68:7)
INFO StrykerCli Trouble figuring out what went wrong? Try `npx stryker run --fileLogLevel trace --logLevel debug` to get some more info.
Finished task 'mutation test' with result: Success
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 24
- Comments: 19 (7 by maintainers)
I’ve pushed the fix to master by accident 🙈
(I’ve restricted the master branch to prevent this from now on)
Anyway, fixed with https://github.com/stryker-mutator/stryker/commit/49c5162461b5240a6c4204305cb21a7dd74d5172
Thank you Nico! I am sorry I wasn’t able to find the time to fix this myself. We appreciate the help.
Willing to add this but my employment requires approval before I can contribute to open source projects. I’ll open a PR once that gets through (mid next week?) unless one of y’all adds it first.
Looks like the Stryker CLI tries to kill its own process with 1, but something is ignoring that: https://github.com/stryker-mutator/stryker/blob/master/packages/core/src/StrykerCli.ts#L155-L160
I run into this issue a lot. Stryker will throw an error during the initial run and my build will pass because it exits successfully. This means you have to look at the logs to know it failed.
Yeah, it would be nice to fail out with something along the lines of “Mutation Coverage Testing requires a green suite” an an exit code of NOT SUCCESS.
Hi! This has been released with @stryker-mutator version 3! You can read all about it here: https://stryker-mutator.io/blog/2020-03-11/stryker-version-3
@nicojs @simondel Is there any chance we could get a patch version released that includes these changes?
@simondel are you the mole? 🙈
Your
exit-codebranch on the robobar example actually always executes both commands, regardless of exit code. This is because you used&to separate the commands instead of&&😅I found the issue:
https://github.com/stryker-mutator/stryker/blob/fbb8102dc890521a80ef1f87236193e506ab9922/packages/core/src/StrykerCli.ts#L160
Don’t know why we’re using
process.kill, since that overrides the exit code (which we’ve just set). removing it fixes the issue.@nicojs You’re right, I forgot that was the reason we do it that way! We should add that as a comment.
@anthony-telljohann Since your teams are using the same process, I assume they’re also using similar hardware (i.e. macOS). Could you try to reproduce your problem using this demo-app?
Reproduction:
This will run mutation testing, which will fail and then Stryker should exit. If the demo-app reproduces your problem (i.e. it keeps on going), it should open a browser with the app which should look like: https://stryker-mutator.io/robobar-example/#!/
We all experienced the same issue on our own projects throughout the last year. Recently, I made a change to all of our projects and it happened to everyone. Once I submitted the issue, I shared it with every team that may be interested in supporting the fix.
Great find! Thanks for all the reactions here.
Please don’t kill the process with
process.exit. Instead useprocess.exitCode = 1. Simply killing the current process is not nice for use cases where Stryker is run as a plugin of something else. We’ve found this quite frustrating ourselves if test runners we support simply exit all of a sudden.It’s also described in the nodejs docs: https://nodejs.org/api/process.html#process_process_exit_code
Agreed. It is not helpful to have a passing message with error indicators. This is misleading.