phpunit: Test Failures with PHP 8

I get the following test failures with PHP 8.0.0-dev (cb95175d6c9f36637818142989648ea74af4cf3d):

PHPUnit 8.1-g2ac6c9466 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.0.0-dev
Configuration: /usr/local/src/phpunit/phpunit.xml

.............................................................   61 / 1938 (  3%)
.............................................................  122 / 1938 (  6%)
.............................................................  183 / 1938 (  9%)
.............................................................  244 / 1938 ( 12%)
.............................................................  305 / 1938 ( 15%)
.............................................................  366 / 1938 ( 18%)
.............................................................  427 / 1938 ( 22%)
.............................................................  488 / 1938 ( 25%)
.............................................................  549 / 1938 ( 28%)
............................S................................  610 / 1938 ( 31%)
.............................................................  671 / 1938 ( 34%)
.............................................................  732 / 1938 ( 37%)
.............................................................  793 / 1938 ( 40%)
.............................................................  854 / 1938 ( 44%)
.............................................................  915 / 1938 ( 47%)
.............................................................  976 / 1938 ( 50%)
............................................................. 1037 / 1938 ( 53%)
............................................................. 1098 / 1938 ( 56%)
............................................................. 1159 / 1938 ( 59%)
............................................................. 1220 / 1938 ( 62%)
............................................................. 1281 / 1938 ( 66%)
............................................................. 1342 / 1938 ( 69%)
............................................................. 1403 / 1938 ( 72%)
............................................................. 1464 / 1938 ( 75%)
............................................................. 1525 / 1938 ( 78%)
............................................................. 1586 / 1938 ( 81%)
............................................................. 1647 / 1938 ( 84%)
..............................S.....SS..........SS....F..F... 1708 / 1938 ( 88%)
.F.............F............................................. 1769 / 1938 ( 91%)
.........F................................................... 1830 / 1938 ( 94%)
.....................................................SS...... 1891 / 1938 ( 97%)
........SS.............F.......................               1938 / 1938 (100%)

Time: 11.86 seconds, Memory: 24.00MB

There were 6 failures:

1) /usr/local/src/phpunit/tests/end-to-end/execution-order/cache-result.phpt
Failed asserting that string matches format description.
--- Expected
+++ Actual
@@ @@
 PHPUnit 8.1-g2ac6c9466 by Sebastian Bergmann and contributors.
 
-.SS..                                                               5 / 5 (100%)
+.WW..                                                               5 / 5 (100%)
 
 Time: 21 ms, Memory: 4.00MB
 
-OK, but incomplete, skipped, or risky tests!
-Tests: 5, Assertions: 3, Skipped: 2.
-C:30:"PHPUnit\Runner\TestResultCache":%d:{a:2:{s:7:"defects";a:2:{s:29:"MultiDependencyTest::testFour";i:1;s:30:"MultiDependencyTest::testThree";i:1;}s:5:"times";a:5:{s:29:"MultiDependencyTest::testFive";d:%f;s:29:"MultiDependencyTest::testFour";d:%f;s:30:"MultiDependencyTest::testThree";d:%f;s:28:"MultiDependencyTest::testTwo";d:%f;s:28:"MultiDependencyTest::testOne";d:%f;}}}
+There were 2 warnings:
+
+1) MultiDependencyTest::testFour
+This test depends on "MultiDependencyTest::testThree" which does not exist.
+
+2) MultiDependencyTest::testThree
+This test depends on "MultiDependencyTest::testOne" which does not exist.
+
+WARNINGS!
+Tests: 5, Assertions: 3, Warnings: 2.
+C:30:"PHPUnit\Runner\TestResultCache":335:{a:2:{s:7:"defects";a:2:{s:29:"MultiDependencyTest::testFour";i:6;s:30:"MultiDependencyTest::testThree";i:6;}s:5:"times";a:5:{s:29:"MultiDependencyTest::testFive";d:0.001;s:29:"MultiDependencyTest::testFour";d:0;s:30:"MultiDependencyTest::testThree";d:0;s:28:"MultiDependencyTest::testTwo";d:0;s:28:"MultiDependencyTest::testOne";d:0;}}}

/usr/local/src/phpunit/tests/end-to-end/execution-order/cache-result.phpt:27
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/TextUI/TestRunner.php:632
/usr/local/src/phpunit/src/TextUI/Command.php:208
/usr/local/src/phpunit/src/TextUI/Command.php:164

2) /usr/local/src/phpunit/tests/end-to-end/execution-order/dependencies-isolation.phpt
Failed asserting that string matches format description.
--- Expected
+++ Actual
@@ @@
 
 Runtime:       PHP 8.0.0-dev
 
-...FSSSW                                                            8 / 8 (100%)
+...FW.WW                                                            8 / 8 (100%)
 
 Time: 127 ms, Memory: 6.00MB
 
-There was 1 warning:
+There were 3 warnings:
 
-1) DependencyFailureTest::testHandlesDependsAnnotationForNonexistentTests
+1) DependencyFailureTest::testTwo
+This test depends on "DependencyFailureTest::testOne" which does not exist.
+
+2) DependencyFailureTest::testFour
+This test depends on "DependencyFailureTest::testOne" which does not exist.
+
+3) DependencyFailureTest::testHandlesDependsAnnotationForNonexistentTests
 This test depends on "DependencyFailureTest::doesNotExist" which does not exist.
 
 --
@@ @@
 
 1) DependencyFailureTest::testOne
 
-%s:%i
+/usr/local/src/phpunit/tests/_files/DependencyFailureTest.php:16
 
---
-
-There were 3 skipped tests:
-
-1) DependencyFailureTest::testTwo
-This test depends on "DependencyFailureTest::testOne" to pass.
-
-2) DependencyFailureTest::testThree
-This test depends on "DependencyFailureTest::testTwo" to pass.
-
-3) DependencyFailureTest::testFour
-This test depends on "DependencyFailureTest::testOne" to pass.
-
 FAILURES!
-Tests: 8, Assertions: 4, Failures: 1, Warnings: 1, Skipped: 3.
+Tests: 8, Assertions: 5, Failures: 1, Warnings: 3.

/usr/local/src/phpunit/tests/end-to-end/execution-order/dependencies-isolation.phpt:21
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/TextUI/TestRunner.php:632
/usr/local/src/phpunit/src/TextUI/Command.php:208
/usr/local/src/phpunit/src/TextUI/Command.php:164

3) /usr/local/src/phpunit/tests/end-to-end/execution-order/execution-order-options-via-config.phpt
Failed asserting that string matches format description.
--- Expected
+++ Actual
@@ @@
 
 Time: 23 ms, Memory: 4.00MB
 
-OK, but incomplete, skipped, or risky tests!
-Tests: 5, Assertions: 3, Skipped: 2.
+There were 2 warnings:
+
+1) MultiDependencyTest::testFour
+This test depends on "MultiDependencyTest::testThree" which does not exist.
+
+2) MultiDependencyTest::testThree
+This test depends on "MultiDependencyTest::testOne" which does not exist.
+
+WARNINGS!
+Tests: 5, Assertions: 3, Warnings: 2.

/usr/local/src/phpunit/tests/end-to-end/execution-order/execution-order-options-via-config.phpt:33
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/TextUI/TestRunner.php:632
/usr/local/src/phpunit/src/TextUI/Command.php:208
/usr/local/src/phpunit/src/TextUI/Command.php:164

4) /usr/local/src/phpunit/tests/end-to-end/execution-order/test-order-reversed-without-dependency-resolution.phpt
Failed asserting that string matches format description.
--- Expected
+++ Actual
@@ @@
 
 Time: 22 ms, Memory: 4.00MB
 
-There were 2 skipped tests:
+There were 2 warnings:
 
 1) MultiDependencyTest::testFour
-This test depends on "MultiDependencyTest::testThree" to pass.
+This test depends on "MultiDependencyTest::testThree" which does not exist.
 
 2) MultiDependencyTest::testThree
-This test depends on "MultiDependencyTest::testOne" to pass.
+This test depends on "MultiDependencyTest::testOne" which does not exist.
 
-OK, but incomplete, skipped, or risky tests!
-Tests: 5, Assertions: 3, Skipped: 2.
+WARNINGS!
+Tests: 5, Assertions: 3, Warnings: 2.

/usr/local/src/phpunit/tests/end-to-end/execution-order/test-order-reversed-without-dependency-resolution.phpt:37
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/TextUI/TestRunner.php:632
/usr/local/src/phpunit/src/TextUI/Command.php:208
/usr/local/src/phpunit/src/TextUI/Command.php:164

5) /usr/local/src/phpunit/tests/end-to-end/mock-objects/generator/3154_namespaced_constant_resolving.phpt
Failed asserting that two strings are equal.

/usr/local/src/phpunit/tests/end-to-end/mock-objects/generator/3154_namespaced_constant_resolving.phpt:1
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/TextUI/TestRunner.php:632
/usr/local/src/phpunit/src/TextUI/Command.php:208
/usr/local/src/phpunit/src/TextUI/Command.php:164

6) /usr/local/src/phpunit/tests/end-to-end/regression/GitHub/3379.phpt
Failed asserting that string matches format description.
--- Expected
+++ Actual
@@ @@
 PHPUnit 8.1-g2ac6c9466 by Sebastian Bergmann and contributors.
 
 Skipped test testOne, status: 1
-SSkipped test testTwo, status: 1
-S                                                                  2 / 2 (100%)
+SW                                                                  2 / 2 (100%)
 
-Time: %s, Memory: %s
+Time: 25 ms, Memory: 4.00MB
 
-OK, but incomplete, skipped, or risky tests!
-Tests: 2, Assertions: 0, Skipped: 2.
+There was 1 warning:
+
+1) Test\Issue3379Test::testTwo
+This test depends on "Test\Issue3379Test::testOne" which does not exist.
+
+WARNINGS!
+Tests: 2, Assertions: 0, Warnings: 1, Skipped: 1.

/usr/local/src/phpunit/tests/end-to-end/regression/GitHub/3379.phpt:14
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/TextUI/TestRunner.php:632
/usr/local/src/phpunit/src/TextUI/Command.php:208
/usr/local/src/phpunit/src/TextUI/Command.php:164

--

There were 10 skipped tests:

1) PHPUnit\Framework\Constraint\ExceptionMessageRegExpTest::testMessageXdebugScreamCompatibility
Extension xdebug is required.

/usr/local/src/phpunit/tests/unit/Framework/Constraint/ExceptionMessageRegExpTest.php:34
/usr/local/src/phpunit/tests/unit/Framework/Constraint/ExceptionMessageRegExpTest.php:34
/usr/local/src/phpunit/src/Framework/TestCase.php:1687
/usr/local/src/phpunit/src/Framework/TestCase.php:836
/usr/local/src/phpunit/src/Framework/TestResult.php:685
/usr/local/src/phpunit/src/Framework/TestCase.php:807

2) /usr/local/src/phpunit/tests/end-to-end/assertion.phpt
assert.exception=1 is required

/usr/local/src/phpunit/tests/end-to-end/assertion.phpt:14
/usr/local/src/phpunit/src/Runner/PhptTestCase.php:141
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/TextUI/TestRunner.php:632
/usr/local/src/phpunit/src/TextUI/Command.php:208
/usr/local/src/phpunit/src/TextUI/Command.php:164

3) /usr/local/src/phpunit/tests/end-to-end/code-coverage-ignore.phpt
Extension xdebug is required.

/usr/local/src/phpunit/tests/end-to-end/code-coverage-ignore.phpt:6
/usr/local/src/phpunit/src/Runner/PhptTestCase.php:141
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/TextUI/TestRunner.php:632
/usr/local/src/phpunit/src/TextUI/Command.php:208
/usr/local/src/phpunit/src/TextUI/Command.php:164

4) /usr/local/src/phpunit/tests/end-to-end/code-coverage-phpt.phpt
Extension xdebug is required.

/usr/local/src/phpunit/tests/end-to-end/code-coverage-phpt.phpt:6
/usr/local/src/phpunit/src/Runner/PhptTestCase.php:141
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/TextUI/TestRunner.php:632
/usr/local/src/phpunit/src/TextUI/Command.php:208
/usr/local/src/phpunit/src/TextUI/Command.php:164

5) /usr/local/src/phpunit/tests/end-to-end/disable-code-coverage-ignore.phpt
Extension xdebug is required.

/usr/local/src/phpunit/tests/end-to-end/disable-code-coverage-ignore.phpt:6
/usr/local/src/phpunit/src/Runner/PhptTestCase.php:141
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/TextUI/TestRunner.php:632
/usr/local/src/phpunit/src/TextUI/Command.php:208
/usr/local/src/phpunit/src/TextUI/Command.php:164

6) /usr/local/src/phpunit/tests/end-to-end/dump-xdebug-filter.phpt
xdebug not loaded

/usr/local/src/phpunit/tests/end-to-end/dump-xdebug-filter.phpt:6
/usr/local/src/phpunit/src/Runner/PhptTestCase.php:141
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/TextUI/TestRunner.php:632
/usr/local/src/phpunit/src/TextUI/Command.php:208
/usr/local/src/phpunit/src/TextUI/Command.php:164

7) /usr/local/src/phpunit/tests/end-to-end/regression/GitHub/2085-enforce-time-limit-options-via-config-without-invoker.phpt
package phpunit/php-invoker is installed

/usr/local/src/phpunit/tests/end-to-end/regression/GitHub/2085-enforce-time-limit-options-via-config-without-invoker.phpt:8
/usr/local/src/phpunit/src/Runner/PhptTestCase.php:141
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/TextUI/TestRunner.php:632
/usr/local/src/phpunit/src/TextUI/Command.php:208
/usr/local/src/phpunit/src/TextUI/Command.php:164

8) /usr/local/src/phpunit/tests/end-to-end/regression/GitHub/2085-without-invoker.phpt
package phpunit/php-invoker is installed

/usr/local/src/phpunit/tests/end-to-end/regression/GitHub/2085-without-invoker.phpt:10
/usr/local/src/phpunit/src/Runner/PhptTestCase.php:141
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/TextUI/TestRunner.php:632
/usr/local/src/phpunit/src/TextUI/Command.php:208
/usr/local/src/phpunit/src/TextUI/Command.php:164

9) /usr/local/src/phpunit/tests/end-to-end/regression/GitHub/2591-separate-function-no-preserve-no-bootstrap-xdebug.phpt
xdebug not loaded

/usr/local/src/phpunit/tests/end-to-end/regression/GitHub/2591-separate-function-no-preserve-no-bootstrap-xdebug.phpt:7
/usr/local/src/phpunit/src/Runner/PhptTestCase.php:141
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/TextUI/TestRunner.php:632
/usr/local/src/phpunit/src/TextUI/Command.php:208
/usr/local/src/phpunit/src/TextUI/Command.php:164

10) /usr/local/src/phpunit/tests/end-to-end/regression/GitHub/2591-separate-function-no-preserve-no-bootstrap.phpt
PHP < 7.3 required

/usr/local/src/phpunit/tests/end-to-end/regression/GitHub/2591-separate-function-no-preserve-no-bootstrap.phpt:9
/usr/local/src/phpunit/src/Runner/PhptTestCase.php:141
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/Framework/TestSuite.php:761
/usr/local/src/phpunit/src/TextUI/TestRunner.php:632
/usr/local/src/phpunit/src/TextUI/Command.php:208
/usr/local/src/phpunit/src/TextUI/Command.php:164

FAILURES!
Tests: 1938, Assertions: 3292, Failures: 6, Skipped: 10.

Now PHPUnit does not have to be compatible with PHP 8 just yet, but these failures could potentially reveal code mistakes that have no effect in PHP 7. I will investigate this, either I find something interesting in PHPUnit or in PHP 8.

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 15 (7 by maintainers)

Commits related to this issue

Do not use is_callable() here as it behaves differently in PHP 8 (see #3526) — committed to sebastianbergmann/phpunit by sebastianbergmann 5 years ago
確保 travis 在 php 8.x 能裝到 phpunit 7 疑似關聯 issue： - https://github.com/sebastianbergmann/phpunit/issues/3526 — committed to CQD/qlurk by CQD 3 years ago
確保 travis 在 php 8.x 能裝到 phpunit 7 疑似關聯 issue： - https://github.com/sebastianbergmann/phpunit/issues/3526 跑 php 8 的測試時，告訴 composer 無視平台版本限制。 — committed to CQD/qlurk by CQD 3 years ago
確保 travis 在 php 8.x 能裝到 phpunit 7 疑似關聯 issue： - https://github.com/sebastianbergmann/phpunit/issues/3526 跑 php 8 的測試時，告訴 composer 無視平台版本限制。這個 commit 還不正常，會導致 phpunit 的相依套件裝錯版本整包壞了了。 — committed to CQD/qlurk by CQD 3 years ago

Most upvoted comments

By instructing Composer to ignore the PHP version requirement.

sebastianbergmann on May 14, 2020

!\defined($value) in https://github.com/sebastianbergmann/phpunit/blob/c5577642a33691832e27c9b28d29c2c26625867f/src/Framework/MockObject/MockMethod.php#L354 is true in PHP 7.3.4 (and in PHP 7.4-dev) but false in PHP 8.0-dev.

sebastianbergmann on Apr 22, 2019

You can use the code using the “–ignore-platform-reqs” parameter. I tested it and it worked perfectly!

composer global require phpunit/phpunit --ignore-platform-reqs

I hope it helps 👍

oviniciusfeitosa on Dec 8, 2020

Notes dump:

<dump> The main problem appears to be changes to how is_callable behaves:

https://github.com/sebastianbergmann/phpunit/blob/aed4efca869c86f7e118e47120ae18a2e8aac944/src/Framework/TestCase.php#L1780-L1784

In PHP 8, is_callable(class::method, …) can only work on static methods, something like:


diff --git a/src/Framework/TestCase.php b/src/Framework/TestCase.php
index 54e9866a3..62d661e77 100644
--- a/src/Framework/TestCase.php
+++ b/src/Framework/TestCase.php
@@ -1775,9 +1775,21 @@ private function handleDependencies(): bool
                 if (\strpos($dependency, '::') === false) {
                     $dependency = $className . '::' . $dependency;
                 }
-
+                
+                if (!preg_match('~([^:]+)::(.*)~', $dependency, $parts)) {
+                    /* error */
+                }
+                
+                /*
+                if ($parts[1] != get_class(this)) {
+                    you will need a reference to an object of type parts[1]
+                    or to enforce the rule that you may not depend on a test in another
+                    class/suite/whatever
+                } 
+                */
+                
                 if (!isset($passedKeys[$dependency])) {
-                    if (!\is_callable($dependency, false, $callableName) || $dependency !== $callableName) {
+                    if (!\is_callable([$this, $parts[2]], false, $callableName) || $dependency !== $callableName) {
                         $this->markWarningForUncallableDependency($dependency);
                     } else {
                         $this->markSkippedForMissingDependecy($dependency);

Get’s it down to one failure …

method_exists may be faster there, because you know you are testing for the existence of a public method, and you know you’re not checking for static callability (because it 's not allowed if the function wasn’t declared static) … just a thought …

While you’re refactoring that function, I’d probably use a regex, maybe something like:


<?php
$input = [
    "clone class::method",
    "!clone class::method",
    "shallowClone class::method",
    "!shallowClone class::method",
    "function",
    "clone function",
    "!clone function",
    "shallowClone function",
    "!shallowClone function"
];

foreach ($input as $next) {
    preg_match('~(?<negate>\!?)'.
                '(?<type>clone|shallowClone)?\s?'.
                '((?<class>[a-zA-Z0-9_]+)::)?'. /* this should probably be a better pattern */
                '(?<function>([a-zA-Z0-9\_]+))~si', /* this should probably be a better pattern */
                $next, $matches); /* no match is malformed */
    
    var_dump([
        "negate"   => $matches['negate'] === '!',
        "type"     => $matches['type'],
        "class"    => $matches['class'],
        "function" => $matches['function']
    ]);
}

If I understood it properly …

</dump>

That should get you somewhere …

krakjoe on Apr 2, 2019

I suggest to use php: 7.4snapshot instead of php: nightly on travis ci.

henryruhs on Feb 16, 2019

i will not remove php: nightly; still want to know what’s happening with upcoming versions. it’s in allow_failure list, so not marking whole pipeline failed.

btw, travis have pre-installed phpunit 8.0.2 on their php 8.0 image.

glensc on Feb 11, 2019

If you use travis-ci do not use php: nightly in your .travis.yml.

fenric on Feb 10, 2019

PHPUnit is not yet supported on PHP 8.

sebastianbergmann on Feb 9, 2019