paratest: Worker crashing after all tests doesn't produce any meaningful output

Q	A
ParaTest version	7.2.6
PHPUnit version	10.3
PHP version	8.1.9

Summary

If a worker crashes for whatever reason after executing all of the assigned tests (i.e. ApplicationForWrapperWorker::end() method), then ParaTest doesn’t acknowledge the worker has crashed and hence doesn’t print any meaningful debugging output.

Current behavior

If a worker crashes after all tests, this is the output produced (in this case - with coverage):

........................................................... 17936 / 17964 ( 99%)
......R
Time: 59:16.353, Memory: 439.50 MB
No tests executed!
Generating code coverage report in Clover XML format ... done [01:16.524]
Generating code coverage report in Cobertura XML format ... done [00:13.068]
Generating code coverage report in HTML format ... done [01:11.969]
Code Coverage Report:       
  2023-09-04 16:39:47       
                            
 Summary:                   
  Lines:    0.00% (0/123123123)

There’s no “100%” line; it states that no tests were executed (although 18k were indeed executed) and the coverage is empty. It’s all due to silent discarding of missing output files from the wrapper worker.

How to reproduce: command, code and error stack trace

Add a throw new \RuntimeException(); into ApplicationForWrapperWorker::end() and run ParaTest.

Expected behavior

I’d expect a “WorkerCrashedException” with a full STDOUT/STDERR output printed the same way it does when a worker crashes when executing a test.

I’d also expect a stack trace if it was due to an exception. Right now, error_reporting() or ini_set('display_errors', 'On'); is never called anywhere in ParaTest and there are no try-catch blocks inside of phpunit-wrapper.php. That means that even if WorkerCrashedException was thrown in this case, it’d still have an empty output and would still be harder to debug than it should be.

About this issue

Original URL
State: closed
Created 10 months ago
Comments: 16 (5 by maintainers)

Most upvoted comments

@oprypkhantc Thank you for the detailed information! I’ll look into it 👍

CanVuralStudocu on Jan 29, 2024

@CanVuralStudocu This is the extension we ended up using: https://github.com/oprypkhantc/phpunit-auto-covers

Keep in mind that it’s not a composer package, it’s not “production ready” and I’m not going to support it. But I specifically added an MIT license to that repo so you can hard fork it and do whatever you want with it.

The algorithm is as follows:

look for coverage metadata (attributes, annotations) from PHPUnit. If there’s any already specified, skip to the next test;
look for @see annotation above the test case. We use that to reference the class that’s being tested, which is pretty much #[CoversClass]. If there are any, then add #[CoversClass] or #[CoversFunction] and skip to the next test;
check the namespace. If it’s a unit test (namespace contains Tests\Unit\ ), then guess the source class name by removing the Tests\Unit\ from the class name. So Modules\ABC\Tests\Unit\SomeService test will get a #[CoversClass(\Modules\ABC\SomeService::class)] and skip to the next test;
otherwise treat it as an integration test and use code coverage driver to get classes that the test covers. I only capture the covered files between the end of setUp hook and the end of the test to avoid a lot of clutter in covered classes. I also filter covered classes by a “root namespace” - i.e. a test in Modules\ABC\Tests\Integration\SomethingTest can only cover source classes from Modules\ABC\ namespace

This successfully covered 98.5% of our tests, leaving just 61 test case files to fix by hand. They were unit tests without @see annotations and with non-matching namespace structure. After that I added a requireCoverageMetadata=true that was it. In the end, the coverage dropped by about 13% in lines, functions, methods and classes 🙃 But that’s alright, at least now generated coverage better represents the real picture.

This change has also resolved performance/resources usage problems with coverage:

paratest now uses 3GB of RAM to merge and generate reports at the end of the run instead of 53GB
paratest now generates an HTML report in about 20 seconds instead of 8+ minutes

oprypkhantc on Jan 27, 2024

I’ve started automating #[Covers] attributes through a PHPUnit extension and will open-source the solution if it comes out okay. I’m hoping to at least save 50% on efforts from our team.

@oprypkhantc Just out of curiosity, what was the experience with the extension? Was it successful?

Unfortunately I had other tasks, this one is on pause. It looked like the extension solution may work for Integration tests.

For unit tests though it doesn’t seem like that’s a good solution, so my plan for them is to write a Rector rule that matches a class and it’s test case based on namespace / class name.

oprypkhantc on Dec 23, 2023

You seems someone who did your research, but I’ll tell you anyway: #[CoversClass(MyClass::class)] annotations are definitively a must for sane CodeCoverage object sizes (see https://github.com/sebastianbergmann/php-code-coverage/blob/d74598007cc2cd4094b6185ba9fcaf0e784120ac/src/CodeCoverage.php#L237-L242)

Then a --max-batch-size=1 process should be no more than few MB (on top of your application memory footprint).

Slamdunk on Sep 6, 2023