pest: `junit` report is broken

How to reproduce:

Source class:

# src/SourceFile.php

namespace PestExample;

final class SourceFile
{
    public function add(int $a, int $b): int
    {
        return $a + $b;
    }
}

test file:

# tests/SourceFileTest.php

use PestExample\SourceFile;

it('can add two integers', function (): void {
    $source = new SourceFile();

    $result = $source->add(3, 2);

    $this->assertGreaterThan(0, $result);
});

Command line:

vendor/bin/pest --coverage-xml=pest-coverage-xml --log-junit="pest-coverage-xml/junit.xml"

generated files:

tree pest-coverage-xml

pest-coverage-xml
├── SourceFile.php.xml
├── index.xml
└── junit.xml

And the incorrect junit.xml report looks like:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="" tests="1" assertions="1" errors="0" warnings="0" failures="0" skipped="0" time="0.006614">
    <testsuite name="P\Tests\SourceFileTest" file="/pest-example/vendor/pestphp/pest/src/Factories/TestCaseFactory.php(185) : eval()'d code" tests="1" assertions="1" errors="0" warnings="0" failures="0" skipped="0" time="0.006614">
      <testcase name="it can add two integers" assertions="1" time="0.006614"/>
    </testsuite>
  </testsuite>
</testsuites>

What’s wrong here? look at the following line:

<testsuite name="P\Tests\SourceFileTest" file="pest-example/vendor/pestphp/pest/src/Factories/TestCaseFactory.php(185) : eval()'d code" tests="1" assertions="1" errors="0" warnings="0" failures="0" skipped="0" time="0.006614">

and in particular look at the file attribute:

file="/pest-example/vendor/pestphp/pest/src/Factories/TestCaseFactory.php(185) : eval()'d code"

It’s not a valid test file.

❗❗

Expected: /pest-example/tests/SourceFileTest.php Actual: /pest-example/vendor/pestphp/pest/src/Factories/TestCaseFactory.php(185) : eval()'d code

If I run the same test but written in PHPUnit style, I get the correct junit report:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="" tests="1" assertions="1" errors="0" warnings="0" failures="0" skipped="0" time="0.007011">
    <testsuite name="Test Suite" tests="1" assertions="1" errors="0" warnings="0" failures="0" skipped="0" time="0.007011">
      <testsuite name="PestExample\Test\PhpUnitSourceFileTest" file="/pest-example/tests/PhpUnitSourceFileTest.php" tests="1" assertions="1" errors="0" warnings="0" failures="0" skipped="0" time="0.007011">
        <testcase name="test_it_can_add_two_integers" class="PestExample\Test\PhpUnitSourceFileTest" classname="PestExample.Test.PhpUnitSourceFileTest" file="/pest-example/tests/PhpUnitSourceFileTest.php" line="14" assertions="1" time="0.007011"/>
      </testsuite>
    </testsuite>
  </testsuite>
</testsuites>

Related to and blocks https://github.com/pestphp/pest/issues/4

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 3
  • Comments: 16 (9 by maintainers)

Most upvoted comments

@ArondeParon issue was resolved back in pest V1, hence why it was closed.

@nunomaduro chose to not implement it in pest V2, but I am pretty sure that he is okay with someone adding support.

The best workaround would be to use team city output as that is supported. There might exist converters between those formats. Better workaround is to submit a PR 😉

+1
olivernybroe on Aug 7, 2023

@olivernybroe 😄 For some time after upgrading to v2 I was sure my code is “ideal” because Infection was reporting 100% when I was at 100% coverage. Incredible fail.

+1
michael-rubel on Jun 27, 2023

This issue is actually related to the issues we had with TeamCity printer.

+1
olivernybroe on Oct 4, 2020

@nunomaduro with the following 2 changes in your branch, I was able to generate a correct junit

diff --git a/native/phpunit/phpunit/src/Util/Log/JUnit.php b/native/phpunit/phpunit/src/Util/Log/JUnit.php
index 3a21ead..87afb77 100644
--- a/native/phpunit/phpunit/src/Util/Log/JUnit.php
+++ b/native/phpunit/phpunit/src/Util/Log/JUnit.php
@@ -198,8 +198,7 @@ final class JUnit extends Printer implements TestListener
                 if ($suite instanceof \Pest\Contracts\Test) {
                     $testSuite->setAttribute('file', $suite->getFilename());
                 } else {
-                    $class = new \ReflectionClass($suite->getName());
-                    $testSuite->setAttribute('file', $class->getFileName());
+                    $testSuite->setAttribute('file', $suite->tests()[0]->getFilename());
                 }
             } catch (\ReflectionException $e) {
             }
@@ -303,7 +302,7 @@ final class JUnit extends Printer implements TestListener
         // @codeCoverageIgnoreEnd

         if ($test instanceof \Pest\Contracts\Test) {
-            $testCase->setAttribute('class', $test->getPrintableTestCaseName());
+            $testCase->setAttribute('class', $class->getName());
             $testCase->setAttribute('classname', \str_replace('\\', '.', $test->getPrintableTestCaseName()));
             $testCase->setAttribute('file', $test->getFilename());
         } else {

I’ve created 2 pest files (tests) with 2 cases (test() functions) inside just to quickly check it.

Resulting junit report is:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="" tests="4" assertions="4" errors="0" warnings="0" failures="0" skipped="0" time="0.028040">
    <testsuite name="P\Tests\FirstSourceFileTest" file="/pest-example/tests/FirstSourceFileTest.php" tests="2" assertions="2" errors="0" warnings="0" failures="0" skipped="0" time="0.021014">
      <testcase name="it First file - first case" class="P\Tests\FirstSourceFileTest" classname="Tests.FirstSourceFileTest" file="/pest-example/tests/FirstSourceFileTest.php" assertions="1" time="0.019044"/>
      <testcase name="it First file - second case" class="P\Tests\FirstSourceFileTest" classname="Tests.FirstSourceFileTest" file="/pest-example/tests/FirstSourceFileTest.php" assertions="1" time="0.001970"/>
    </testsuite>
    <testsuite name="P\Tests\SecondSourceFileTest" file="/pest-example/tests/SecondSourceFileTest.php" tests="2" assertions="2" errors="0" warnings="0" failures="0" skipped="0" time="0.007026">
      <testcase name="it Second file - first case" class="P\Tests\SecondSourceFileTest" classname="Tests.SecondSourceFileTest" file="/pest-example/tests/SecondSourceFileTest.php" assertions="1" time="0.003728"/>
      <testcase name="it Second file - second case" class="P\Tests\SecondSourceFileTest" classname="Tests.SecondSourceFileTest" file="/pest-example/tests/SecondSourceFileTest.php" assertions="1" time="0.003297"/>
    </testsuite>
  </testsuite>
</testsuites>

For sure, this is not a ready solution, but probably will speed up your investigation on this topic. What do you think we need to do else to move it forward?

+1
maks-rafalko on Jun 4, 2020

It’s much better! But still not enough.

Currently, it generates such junit report:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="" tests="1" assertions="1" errors="0" warnings="0" failures="0" skipped="0" time="0.022643">
    <testsuite name="P\Tests\SourceFileTest" file="/pest/src/Factories/TestCaseFactory.php(187) : eval()'d code" tests="1" assertions="1" errors="0" warnings="0" failures="0" skipped="0" time="0.022643">
      <testcase name="it can add two integers" class="Tests\SourceFileTest" classname="Tests.SourceFileTest" file="/pest-example/tests/SourceFileTest.php" assertions="1" time="0.022643"/>
    </testsuite>
  </testsuite>
</testsuites>

where <testcase node contains the correct file=/pest-example/tests/SourceFileTest.php, but with incorrect FQCN Tests\SourceFileTest

but node <testsuite contains correct name="P\Tests\SourceFileTest" but incorrect file file="/pest/src/Factories/TestCaseFactory.php(187) : eval()'d code"

Can we somehow have this:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="" tests="1" assertions="1" errors="0" warnings="0" failures="0" skipped="0" time="0.022643">
-    <testsuite name="P\Tests\SourceFileTest" file="/pest/src/Factories/TestCaseFactory.php(187) : eval()'d code" tests="1" assertions="1" errors="0" warnings="0" failures="0" skipped="0" time="0.022643">
+    <testsuite name="P\Tests\SourceFileTest" file="/pest-example/tests/SourceFileTest.php" tests="1" assertions="1" errors="0" warnings="0" failures="0" skipped="0" time="0.022643">
      <testcase name="it can add two integers" class="Tests\SourceFileTest" classname="Tests.SourceFileTest" file="/pest-example/tests/SourceFileTest.php" assertions="1" time="0.022643"/>
    </testsuite>
  </testsuite>
</testsuites>

or at least this

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="" tests="1" assertions="1" errors="0" warnings="0" failures="0" skipped="0" time="0.022643">
    <testsuite name="P\Tests\SourceFileTest" file="/pest/src/Factories/TestCaseFactory.php(187) : eval()'d code" tests="1" assertions="1" errors="0" warnings="0" failures="0" skipped="0" time="0.022643">
-      <testcase name="it can add two integers" class="Tests\SourceFileTest" classname="Tests.SourceFileTest" file="/pest-example/tests/SourceFileTest.php" assertions="1" time="0.022643"/>
+      <testcase name="it can add two integers" class="P\Tests\SourceFileTest" classname="Tests.SourceFileTest" file="/pest-example/tests/SourceFileTest.php" assertions="1" time="0.022643"/>

    </testsuite>
  </testsuite>
</testsuites>

In other words, we need to have a node where we both have P\Test\SourceFileTest as FQCL and /pest-example/tests/SourceFileTest.php as a file

Explanation:

  • P\Test\SourceFileTest is used in coverage-xml report as a FQCN, but not Test\SourceFileTest (not the P\ prefix, I guess this is what Pest adds)
+1
maks-rafalko on Jun 1, 2020

Just pushed a very native solution for this: https://github.com/pestphp/pest/tree/feat/poc-junit.

Can you try it out and tell me if something is missing?

+1
nunomaduro on May 31, 2020

I will take a better look at this tomorrow.

+1
nunomaduro on May 30, 2020
Read more comments on GitHub
← pest: Can't install on Laravel 10
pest: Not seeing exception when running in PhpStorm →