phpunit: Execution order "by-defect" does not support parameterized tests with @dataProvider
Q | A |
---|---|
PHPUnit version | 7.3.1 |
PHP version | 7.2.8 |
Installation Method | PHAR |
In my test set (executed via XML configuration) there is (coincidentally) (1) one incomplete, (2) one skipped and (3) one failing test. I just started playing around with the new cool feature of running tests first that failed in a previous run, but it does not behave as expected:
acmedev@dev56w1:~$ rm .phpunit.result.cache
rm: remove regular file ‘.phpunit.result.cache’? y
acmedev@dev56w1:~$ php /mnt/data/Kunden/acme/components/phpunit/7.3.1/phpunit.phar --configuration /home/data/acmedev/unittests/AllPlainPHPUnitTests.xml --order-by=defects --cache-result
Execution according to PHPUnit XML configuration file '/home/data/acmedev/unittests/AllPlainPHPUnitTests.xml'...
PHPUnit 7.3.1 by Sebastian Bergmann and contributors.
.................I............................................. 63 / 524 ( 12%)
............................................................... 126 / 524 ( 24%)
............................................................... 189 / 524 ( 36%)
............................................................... 252 / 524 ( 48%)
............................................................... 315 / 524 ( 60%)
............................................................... 378 / 524 ( 72%)
..........................................................F.... 441 / 524 ( 84%)
............................................................... 504 / 524 ( 96%)
.S.................. 524 / 524 (100%)
Time: 6.41 seconds, Memory: 94.25MB
There was 1 failure:
1) Acme_Phone_FormatNumberTest::formatNumberForMobileTelLink with data set #8 (' 123-456-789', '123456789')
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'123456789'
+' 123456789'
/home/data/acmedev/unittests/unit/app/admin/classes/phone/formatnumberTest.php:40
FAILURES!
Tests: 524, Assertions: 1031, Failures: 1, Skipped: 1, Incomplete: 1.
acmedev@dev56w1:~$ php /mnt/data/Kunden/acme/components/phpunit/7.3.1/phpunit.phar --configuration /home/data/acmedev/unittests/AllPlainPHPUnitTests.xml --order-by=defects --cache-result
Execution according to PHPUnit XML configuration file '/home/data/acmedev/unittests/AllPlainPHPUnitTests.xml'...
PHPUnit 7.3.1 by Sebastian Bergmann and contributors.
I.............................................................. 63 / 524 ( 12%)
............................................................... 126 / 524 ( 24%)
............................................................... 189 / 524 ( 36%)
............................................................... 252 / 524 ( 48%)
............................................................... 315 / 524 ( 60%)
............................................................... 378 / 524 ( 72%)
..................................................F............ 441 / 524 ( 84%)
.....................................................S......... 504 / 524 ( 96%)
.................... 524 / 524 (100%)
Time: 6.63 seconds, Memory: 94.25MB
There was 1 failure:
1) Acme_Phone_FormatNumberTest::formatNumberForMobileTelLink with data set #8 (' 123-456-789', '123456789')
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'123456789'
+' 123456789'
/home/data/acmedev/unittests/unit/app/admin/classes/phone/formatnumberTest.php:40
FAILURES!
Tests: 524, Assertions: 1031, Failures: 1, Skipped: 1, Incomplete: 1.
=> My expectation would have been, that it starts with the single failing test (or in general all failing test cases or maybe all test classes with failing test cases?); and executing the other tests (including incomplete or skipped ones) in the previous/default order.
Not sure if it is needed or helpful, but the cache result file starts like this:
C:30:"PHPUnit\Runner\TestResultCache":35938:{a:2:{s:7:"defects";a:3:{s:20:"sendQuoteRequestTest";i:2;s:45:"formatNumberForMobileTelLink with data set #8";i:3;s:44:"testPhpWarningIsConvertedToPhpUnitException3";i:1;}...
…whereas:
sendQuoteRequestTest
is the incomplete testformatNumberForMobileTelLink with data set #8
is the failing test (see above)testPhpWarningIsConvertedToPhpUnitException3
is the skipped test
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 17 (11 by maintainers)
Commits related to this issue
- Reproduce #3246. Tests with failing individual dataprovider rows do not mark the parent test failed. — committed to sebastianbergmann/phpunit by epdenouden 6 years ago
@reinholdfuereder thank you for the super detailed issue! I will look into it later this week, it is currently not impeding anything, right?