pest: Not seeing exception when running in PhpStorm

@olivernybroe Thank you for your valuable hints.

I think I found a way to fix the issues and reflect this within the tests.

cc @nunomaduro

I digged deeper into this issue and it seems to be a tricky one. My findings so far:

The problem is definitively related to a change in PHPUnit in the latest version (10.1.2). Before this change a test throwing an exception only triggered a PHPUnit\Event\Test\Errored event. But now it also triggers a PHPUnit\Event\Test\ConsideredRisky event in case if the test did not any assertions before the exception was thrown.

This is also visible in the default output of PHPUnit: Test

    public function test_error_behaviour(): void
    {
        throw new Exception('test error');
        $this->assertTrue(true);
    }

Output PHPUnit 10.1.1

PHPUnit 10.1.1 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.1.14
Configuration: /Users/sandro/Code/phpunit-test/phpunit.xml

E                                                                   1 / 1 (100%)

Time: 00:00.011, Memory: 8.00 MB

There was 1 error:

1) Tests\Unit\ExampleTest::test_error_behaviour
Exception: test error

/Users/sandro/Code/phpunit-test/tests/Unit/ExampleTest.php:15

ERRORS!
Tests: 1, Assertions: 0, Errors: 1.

Output PHPUnit 10.1.2

PHPUnit 10.1.2 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.1.14
Configuration: /Users/sandro/Code/phpunit-test/phpunit.xml

E                                                                   1 / 1 (100%)

Time: 00:00.009, Memory: 8.00 MB

There was 1 error:

1) Tests\Unit\ExampleTest::test_error_behaviour
Exception: test error

/Users/sandro/Code/phpunit-test/tests/Unit/ExampleTest.php:15

--

There was 1 risky test:

1) Tests\Unit\ExampleTest::test_error_behaviour
This test did not perform any assertions

/Users/sandro/Code/phpunit-test/tests/Unit/ExampleTest.php:13

ERRORS!
Tests: 1, Assertions: 0, Errors: 1, Risky: 1.

Having this extra ConsideredRisky event we have two issues in the Teamcity output:

1. The count of passed tests is too low, because the same tests is subtracted twice:

        $numberOfPassedTests = $result->numberOfTestsRun()
            - $result->numberOfTestErroredEvents() // <-- here
            - $result->numberOfTestFailedEvents()
            - $result->numberOfTestSkippedEvents()
            - $result->numberOfTestsWithTestConsideredRiskyEvents() // <-- AND here
            - $result->numberOfTestMarkedIncompleteEvents();

https://github.com/pestphp/pest/blob/2.x/src/Logging/TeamCity/Converter.php#L191

1. The exception is not shown by PHPStorm because there are two entries for the same test (name=“error”) in the TeamCity output and only the second one is shown:

...
##teamcity[testFailed name='error' message='Exception: test error' details='at tests/Unit/ExampleTest.php:5|nat vendor/pestphp/pest/src/Factories/TestCaseMethodFactory.php:100|nat vendor/pestphp/pest/src/Concerns/Testable.php:302|nat vendor/pestphp/pest/src/Support/ExceptionTrace.php:28|nat vendor/pestphp/pest/src/Concerns/Testable.php:302|nat vendor/pestphp/pest/src/Concerns/Testable.php:221|nat vendor/pestphp/pest/src/Kernel.php:86' flowId='78739']
##teamcity[testIgnored name='error' message='This test did not perform any assertions' details='' flowId='78739']
...

---

To be honest I am not sure how to proceed here. My thoughts so far:

Maybe triggering the two events for the same test is not intended in PHPUnit and it is a bug on their end, but I don’t think so.

If we want to fix this on the Pest side we have to calculate the number of successful tests differently by checking every event if there is another event for the same test. This seems doable to me.

For the second problem we would have to store all the events instead of writing them to the output directly and then filter them to only pass the most important event to the output. But this looks like a massive change compared to the current behaviour and I would like to hear your thought before continuing.

Last but not least it would also be possible that the second problem is on PHPStorm side and they should show two or more messages for the same test. I am not familiar with the TeamCity specifications.

@nunomaduro @olivernybroe

The issue comes from this file: https://github.com/pestphp/pest/blob/2.x/src/Logging/TeamCity/TeamCityLogger.php.