infection: Problem with "Skipped"
| Question | Answer |
|---|---|
| Infection version | 0.17.2@31e4af4be7e7508bed086d8254f010a56ff79bea |
| Test Framework version | 9.2.6 |
| PHP version | 7.4.3 |
| Platform | Ubuntu |
| Github Repo | - |
In version 0.16.4 most mutations do not run in timeouts. However, after updating to version 0.17.2, some of these mutants will be “skipped”.
I also checked that the mutants with “skipped” are not generated by the new “instanceof” mutator.
I don’t understand exactly why this happens? Is this a bug or intentional behavior?
Output Infection 0.17.2:
____ ____ __ _
/ _/___ / __/__ _____/ /_(_)___ ____
/ // __ \/ /_/ _ \/ ___/ __/ / __ \/ __ \
_/ // / / / __/ __/ /__/ /_/ / /_/ / / / /
/___/_/ /_/_/ \___/\___/\__/_/\____/_/ /_/
Infection - PHP Mutation Testing Framework version 0.17.2@31e4af4be7e7508bed086d8254f010a56ff79bea
[notice] You are running Infection with PCOV enabled.
#!/usr/bin/env php
Running initial test suite...
PHPUnit version: 9.2.6
1739 [============================] 21 secs
Generate mutants...
Processing source code files: 943/943
.: killed, M: escaped, U: uncovered, E: fatal error, T: timed out, S: skipped
.................................................. ( 50 / 4095)
...............S.................................. ( 100 / 4095)
<skip>
.................................................. (4050 / 4095)
............................................. (4095 / 4095)
4095 mutations were generated:
4055 mutants were killed
0 mutants were not covered by tests
0 covered mutants were not detected
0 errors were encountered
2 time outs were encountered
38 mutants required more time than configured
Metrics:
Mutation Score Indicator (MSI): 100%
Mutation Code Coverage: 100%
Covered Code MSI: 100%
Please note that some mutants will inevitably be harmless (i.e. false positives).
Time: 12m 58s. Memory: 0.15GB
Output Infection 0.16.4:
You are running Infection with PCOV enabled.
____ ____ __ _
/ _/___ / __/__ _____/ /_(_)___ ____
/ // __ \/ /_/ _ \/ ___/ __/ / __ \/ __ \
_/ // / / / __/ __/ /__/ /_/ / /_/ / / / /
/___/_/ /_/_/ \___/\___/\__/_/\____/_/ /_/
Infection - PHP Mutation Testing Framework 0.16.4@c1ef0b16b3e8d171f10ba8bfe8b6c93d482a68b4
#!/usr/bin/env php
Running initial test suite...
PHPUnit version: 9.2.6
1741 [============================] 22 secs
Generate mutants...
Processing source code files: 943/943
.: killed, M: escaped, S: uncovered, E: fatal error, T: timed out
.................................................. ( 50 / 3950)
..................E............................... ( 100 / 3950)
<skip>
.................................................. (3900 / 3950)
.................................................. (3950 / 3950)
3950 mutations were generated:
3947 mutants were killed
0 mutants were not covered by tests
0 covered mutants were not detected
1 errors were encountered
2 time outs were encountered
Metrics:
Mutation Score Indicator (MSI): 100%
Mutation Code Coverage: 100%
Covered Code MSI: 100%
Please note that some mutants will inevitably be harmless (i.e. false positives).
Time: 12m 8s. Memory: 136.01MB
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 33 (18 by maintainers)
Commits related to this issue
- Attempt to fix #1304 — committed to sanmai/infection by sanmai 4 years ago
- Attempt to fix #1304 TestLocator returns non-unique tests, and JUnitTestCaseSorter works around that; we have to do that too. — committed to sanmai/infection by sanmai 4 years ago
- TestLocator returns non-unique tests, but Mutation didn't know that Fixes #1304 TestLocator returns non-unique tests, and JUnitTestCaseSorter works around that; we have to do that too. — committed to sanmai/infection by sanmai 4 years ago
- TestLocator returns non-unique tests, but Mutation didn't know that (#1310) * TestLocator returns non-unique tests, but Mutation didn't know that Fixes #1304 TestLocator returns non-unique test... — committed to infection/infection by sanmai 4 years ago
- TestLocator returns non-unique tests, but Mutation didn't know that (#1310) * TestLocator returns non-unique tests, but Mutation didn't know that Fixes #1304 TestLocator returns non-unique test... — committed to sanmai/infection by sanmai 4 years ago
- TestLocator returns non-unique tests, but Mutation didn't know that (#1310) * TestLocator returns non-unique tests, but Mutation didn't know that Fixes #1304 TestLocator returns non-unique test... — committed to infection/infection by sanmai 4 years ago
I didn’t mean to push, I just didn’t know how the process worked. Thanks for the explanation.
Thanks for the great work on Infection! I wish you a nice day
@DaveLiddament Could you try with the latest from #1310?
So far I’m getting 0 skipped with timeout of 1 second.
Thank you guys for providing examples, especially nice to see Playground example 😉
Thank you @sanmai for such a quick fix.
Released: https://github.com/infection/infection/releases/tag/0.17.3
Yeah! Thanks for your work @sanmai!
@sanmai yes that works!
Fantastic work, both on this new feature and getting this fix in. Thanks very much for all you effort today.
For my testsuite I can confirm that this pull-request eliminates the unwanted “skipped” cases.
OK, does this recreate the problem: https://infection-php.dev/r/608
Assuming playground has a timeout of 10s. This test should only run 5s. So no reason for any to be skipped?
I might be having a similar problem…
My project’s complete test suite runs in 5 seconds (see output below using this commit).
I believe the infection timeout is defaulted to 10 seconds. If I understand this functionality correctly then surely no test takes more than 10 seconds to run, and therefore none should be marked as skipped?
Output:
Increasing timeout to 90 seconds does decrease the number of mutants marked as skipped. My best guess is that this functionality is working but the timing is wrong.