infection: Infection posts no message about executionOrder="random" upon initial test suite failure
I think Infection should follow the standards of the testing framework it is build on top. The afford to refactor massive testing suites to work in random execution often does not pay off.
It took me time to figure out that I had to set PHPUnit’s defaults executionOrder="default" inside phpunit.xml to have the default behaviour. Seriously, most developers might not expect this!
BTW: I never got this error message - the Infection process exist without showing up an error.
Please keep this issue open and wait for up / downvotes. Thank you!
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 15 (10 by maintainers)
See, the whole point in running tests in a random order is to verify that your tests have no hidden dependencies, and that Infection can properly detect all mutations. (I think we should post a warning with this description if
executionOrderis set todefault.)For example, say you have a line covered by tests A, B, and C. Say, Infection changes something in that line, and runs those three tests to see if they caught the change. Now, if any of these tests have a hidden dependency (undeclared, without
@dependsclause), they’ll fail, and Infection would think that these tests are good enough to catch the mutation. But really they’re not. And you would think you have, say, 90% MSI, but really you’re could be very far from this number.This is why it is paramount to have all tests dependencies declared properly. And if you have them as such, then even with
executionOrder="random"everything will proceed as it should because we specifically ask PHPUnit to respect intertest dependencies.Really you don’t need to refactor anything, all you need is to specify
@dependsbefore every test that depends on other test, and that’s it.(With all said I can add that I feel your pain. For a very large project it may worth to start small, and validate with Infection only parts of a project. For example, core abstraction layers could be a good candidate. Yeah, you would have to walk around and tag tests and suites with groups, but it’s doable. You only need to tag some suites to make Infection work on them, not every suite out there. Yet another possibility is to tag all and every test with
@coversNothing(PHP CS Fixer hasphp_unit_test_class_requires_coversto help you), then gradually tag specific test suites as covering specific things. Infection will happily ignore tests that cover nothing, so they will be less of an issue.)@redaxmedia the thing is, we are not asking this to be purist or strict, but out of necessity. Infection will run your tests in a random order (that’s just how the tool works). Sorry that this cannot be addressed differently for now.
A workaround though is as @sanmai mentioned, to manage to isolate those tests to avoid them being run by infection.