phpunit: Execution order "by-defect" does not support parameterized tests with @dataProvider

Q A
PHPUnit version 7.3.1
PHP version 7.2.8
Installation Method PHAR

In my test set (executed via XML configuration) there is (coincidentally) (1) one incomplete, (2) one skipped and (3) one failing test. I just started playing around with the new cool feature of running tests first that failed in a previous run, but it does not behave as expected:

acmedev@dev56w1:~$ rm .phpunit.result.cache
rm: remove regular file ‘.phpunit.result.cache’? y
acmedev@dev56w1:~$ php /mnt/data/Kunden/acme/components/phpunit/7.3.1/phpunit.phar --configuration /home/data/acmedev/unittests/AllPlainPHPUnitTests.xml --order-by=defects --cache-result
Execution according to PHPUnit XML configuration file '/home/data/acmedev/unittests/AllPlainPHPUnitTests.xml'...
PHPUnit 7.3.1 by Sebastian Bergmann and contributors.

.................I.............................................  63 / 524 ( 12%)
............................................................... 126 / 524 ( 24%)
............................................................... 189 / 524 ( 36%)
............................................................... 252 / 524 ( 48%)
............................................................... 315 / 524 ( 60%)
............................................................... 378 / 524 ( 72%)
..........................................................F.... 441 / 524 ( 84%)
............................................................... 504 / 524 ( 96%)
.S..................                                            524 / 524 (100%)

Time: 6.41 seconds, Memory: 94.25MB

There was 1 failure:

1) Acme_Phone_FormatNumberTest::formatNumberForMobileTelLink with data set #8 ('  123-456-789', '123456789')
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'123456789'
+'      123456789'

/home/data/acmedev/unittests/unit/app/admin/classes/phone/formatnumberTest.php:40

FAILURES!
Tests: 524, Assertions: 1031, Failures: 1, Skipped: 1, Incomplete: 1.
acmedev@dev56w1:~$ php /mnt/data/Kunden/acme/components/phpunit/7.3.1/phpunit.phar --configuration /home/data/acmedev/unittests/AllPlainPHPUnitTests.xml --order-by=defects --cache-result
Execution according to PHPUnit XML configuration file '/home/data/acmedev/unittests/AllPlainPHPUnitTests.xml'...
PHPUnit 7.3.1 by Sebastian Bergmann and contributors.

I..............................................................  63 / 524 ( 12%)
............................................................... 126 / 524 ( 24%)
............................................................... 189 / 524 ( 36%)
............................................................... 252 / 524 ( 48%)
............................................................... 315 / 524 ( 60%)
............................................................... 378 / 524 ( 72%)
..................................................F............ 441 / 524 ( 84%)
.....................................................S......... 504 / 524 ( 96%)
....................                                            524 / 524 (100%)

Time: 6.63 seconds, Memory: 94.25MB

There was 1 failure:

1) Acme_Phone_FormatNumberTest::formatNumberForMobileTelLink with data set #8 ('  123-456-789', '123456789')
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'123456789'
+'      123456789'

/home/data/acmedev/unittests/unit/app/admin/classes/phone/formatnumberTest.php:40

FAILURES!
Tests: 524, Assertions: 1031, Failures: 1, Skipped: 1, Incomplete: 1.

=> My expectation would have been, that it starts with the single failing test (or in general all failing test cases or maybe all test classes with failing test cases?); and executing the other tests (including incomplete or skipped ones) in the previous/default order.

Not sure if it is needed or helpful, but the cache result file starts like this:

C:30:"PHPUnit\Runner\TestResultCache":35938:{a:2:{s:7:"defects";a:3:{s:20:"sendQuoteRequestTest";i:2;s:45:"formatNumberForMobileTelLink with data set #8";i:3;s:44:"testPhpWarningIsConvertedToPhpUnitException3";i:1;}...

…whereas:

  1. sendQuoteRequestTest is the incomplete test
  2. formatNumberForMobileTelLink with data set #8 is the failing test (see above)
  3. testPhpWarningIsConvertedToPhpUnitException3 is the skipped test

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 17 (11 by maintainers)

Commits related to this issue

Most upvoted comments

@reinholdfuereder thank you for the super detailed issue! I will look into it later this week, it is currently not impeding anything, right?