paratest: No info on failing / risky tests

Fixed in master:

#~/repos/paratest (master %=)$ vendor/bin/phpunit test/fixtures/warning_tests/
PHPUnit 9.3.7 by Sebastian Bergmann and contributors.

W.WW                                                                4 / 4 (100%)

Time: 00:00.008, Memory: 8.00 MB

There were 3 warnings:

1) ParaTest\Tests\fixtures\warning_tests\HasOtherWarningsTest::testMockingNonExistentMethod
Trying to configure method "nonExistentMethodMock" which cannot be configured because it does not exist, has not been specified, is final, or is static

2) Warning
Test method "testPrivateTest" in test class "ParaTest\Tests\fixtures\warning_tests\HasWarningsTest" is not public.

3) Warning
Test method "testMissingDataProvider" in test class "ParaTest\Tests\fixtures\warning_tests\HasWarningsTest" is not public.

WARNINGS!
Tests: 4, Assertions: 1, Warnings: 3.



#~/repos/paratest (master %=)$ bin/paratest test/fixtures/warning_tests/

Running phpunit in 4 processes with /paratest/vendor/phpunit/phpunit/phpunit

Configuration read from /paratest/phpunit.xml.dist

W.WW

Time: 00:00.196, Memory: 8.00 MB

There were 3 warnings:

1) ParaTest\Tests\fixtures\warning_tests\HasOtherWarningsTest::testMockingNonExistentMethod
Trying to configure method "nonExistentMethodMock" which cannot be configured because it does not exist, has not been specified, is final, or is static

2) Warning
Test method "testPrivateTest" in test class "ParaTest\Tests\fixtures\warning_tests\HasWarningsTest" is not public.

3) Warning
Test method "testMissingDataProvider" in test class "ParaTest\Tests\fixtures\warning_tests\HasWarningsTest" is not public.
OK (4 tests, 1 assertion)



#~/repos/paratest (master %=)$ bin/paratest --runner WrapperRunner test/fixtures/warning_tests/

Running phpunit in 4 processes with /paratest/vendor/phpunit/phpunit/phpunit

Configuration read from /paratest/phpunit.xml.dist

W.WW

Time: 00:00.216, Memory: 8.00 MB

There were 3 warnings:

1) ParaTest\Tests\fixtures\warning_tests\HasOtherWarningsTest::testMockingNonExistentMethod
Trying to configure method "nonExistentMethodMock" which cannot be configured because it does not exist, has not been specified, is final, or is static

2) Warning
Test method "testPrivateTest" in test class "ParaTest\Tests\fixtures\warning_tests\HasWarningsTest" is not public.

3) Warning
Test method "testMissingDataProvider" in test class "ParaTest\Tests\fixtures\warning_tests\HasWarningsTest" is not public.
OK (4 tests, 1 assertion)



#~/repos/paratest (master %=)$ bin/paratest --runner SqliteRunner test/fixtures/warning_tests/

Running phpunit in 4 processes with /paratest/vendor/phpunit/phpunit/phpunit

Configuration read from /paratest/phpunit.xml.dist

W.WW

Time: 00:01.011, Memory: 8.00 MB

There were 3 warnings:

1) ParaTest\Tests\fixtures\warning_tests\HasOtherWarningsTest::testMockingNonExistentMethod
Trying to configure method "nonExistentMethodMock" which cannot be configured because it does not exist, has not been specified, is final, or is static

2) Warning
Test method "testPrivateTest" in test class "ParaTest\Tests\fixtures\warning_tests\HasWarningsTest" is not public.

3) Warning
Test method "testMissingDataProvider" in test class "ParaTest\Tests\fixtures\warning_tests\HasWarningsTest" is not public.
OK (4 tests, 1 assertion)

That’s correct, because until the end they cannot be distinguished by errors due to a JUnit XML schema.

By now risky tests are reported correctly in ParaTest too I meant that now they are verbose and you can debug them.

Good news: https://github.com/sebastianbergmann/phpunit/pull/4438 has been merged, and together with https://github.com/paratestphp/paratest/pull/523 now risky tests are reported correctly in ParaTest too:

There were 2 risky tests:

1) ParaTest\Tests\fixtures\failing_tests\UnitTestWithMethodAnnotationsTest::testRisky
This test did not perform any assertions

./paratest/test/fixtures/failing_tests/UnitTestWithMethodAnnotationsTest.php:68

2) ParaTest\Tests\fixtures\failing_tests\UnitTestWithErrorTest::testRisky
This test did not perform any assertions

./paratest/test/fixtures/failing_tests/UnitTestWithMethodAnnotationsTest.php:68

@andreasschroth we can send you some Mollie swag or stuff from your Amazon wishlist instead, please contact me at willem@mollie.com.

@willemstuursma Thank you for your generous offer. I gave this issue the label “most-upvotes” now (which basically means “often requested for”). I’ll take a look at this once I have some more spare time (but I don’t accept any money 😃 ).

We are willing to sponsor support for Warnings and Risky tests.

I am still having this issue with the latest paratest and PHPUnit 7.3, but for warnings.

Time: 26.25 seconds, Memory: 90.00MB
FAILURES!
Tests: 14851, Assertions: 44497, Failures: 2, Errors: 0.```

PHPUnit gives the warnings:

Time: 2.81 minutes, Memory: 728.00MB

There were 2 warnings:

1) Warning
The data provider specified for zzzTest:xxx is invalid.
...
2) Warning
The data provider specified for zzzTest::yyy is invalid.
...

I’ve faced same issue. Paratest doesn’t show warnings triggered by phpunit. How to reproduce:

1. Create a test and mock some class with non-existent method.

2. Run phpunit with default settings. It will show warning: Trying to configure method "XXX" which cannot be configured because it does not exist, has not been specified, is final, or is static

3. Run paratest and you will not see a warning.