symfony: [HttpClient] Failed to open stream: Too many open files

Same as @javer I can confirm that any of those commits are the culprit.

I tested it on PHP 8.0.13 and PHP 8.0.14.

Can confirm the issue with PHP 8.1 also. Downgrading symfony/http-client to 5.4.1 fixes the issue.

Git bisect between 5.4.1 and 5.4.2 shows that the first bad commit could be any of:

• https://github.com/symfony/http-client/commit/d05b20ee7e0214f50a4d7a5ba132dc56dd60547b
• https://github.com/symfony/http-client/commit/d8efd751261e65d34659c12d6e3279c426f44d75
• https://github.com/symfony/http-client/commit/7c2cb862412add1263ee21ec641cbdc65327da01
• https://github.com/symfony/http-client/commit/ec58f58534e23138db709bb77abadf5a2b63174b
• https://github.com/symfony/http-client/commit/343f824ccc666777a535791f29ada7a81e4c0281

More precise bisecting (with fixing incompatible types in HttpClientDataCollector in intermediate commits) led me to the single first bad commit: https://github.com/symfony/http-client/commit/d05b20ee7e0214f50a4d7a5ba132dc56dd60547b

@nicolas-grekas Unfortunately the patch didn’t help. But my reproducer shows that we even don’t need to make any curl requests, because CurlClientState::reset() is called by ServicesResetter, so just having any HttpClient in the container is enough to get an error after many calls to ServicesResetter::reset(), see backtrace:

array:52 [
  0 => array:5 [
    "file" => "vendor/symfony/http-client/CurlHttpClient.php"
    "line" => 325
    "function" => "reset"
    "class" => "Symfony\Component\HttpClient\Internal\CurlClientState"
    "type" => "->"
  ]
  1 => array:5 [
    "file" => "vendor/symfony/http-client/TraceableHttpClient.php"
    "line" => 94
    "function" => "reset"
    "class" => "Symfony\Component\HttpClient\CurlHttpClient"
    "type" => "->"
  ]
  2 => array:5 [
    "file" => "vendor/symfony/http-kernel/DependencyInjection/ServicesResetter.php"
    "line" => 47
    "function" => "reset"
    "class" => "Symfony\Component\HttpClient\TraceableHttpClient"
    "type" => "->"
  ]
  3 => array:5 [
    "file" => "vendor/symfony/http-kernel/Kernel.php"
    "line" => 116
    "function" => "reset"
    "class" => "Symfony\Component\HttpKernel\DependencyInjection\ServicesResetter"
    "type" => "->"
  ]
  4 => array:5 [
    "file" => "vendor/symfony/http-kernel/Kernel.php"
    "line" => 197
    "function" => "boot"
    "class" => "Symfony\Component\HttpKernel\Kernel"
    "type" => "->"
  ]
  5 => array:5 [
    "file" => "vendor/symfony/http-kernel/HttpKernelBrowser.php"
    "line" => 65
    "function" => "handle"
    "class" => "Symfony\Component\HttpKernel\Kernel"
    "type" => "->"
  ]
  6 => array:5 [
    "file" => "vendor/symfony/framework-bundle/KernelBrowser.php"
    "line" => 182
    "function" => "doRequest"
    "class" => "Symfony\Component\HttpKernel\HttpKernelBrowser"
    "type" => "->"
  ]
  7 => array:5 [
    "file" => "vendor/symfony/browser-kit/AbstractBrowser.php"
    "line" => 398
    "function" => "doRequest"
    "class" => "Symfony\Bundle\FrameworkBundle\KernelBrowser"
    "type" => "->"
  ]
  8 => array:5 [
    "file" => "vendor/symfony/browser-kit/AbstractBrowser.php"
    "line" => 606
    "function" => "request"
    "class" => "Symfony\Component\BrowserKit\AbstractBrowser"
    "type" => "->"
  ]
...
]

For example, in my application (test run) ServicesResetter::reset() is called 502 times before getting the error Failed to open stream: Too many open files, at the same time there weren’t any calls to CurlResponse constructor, so it seems we cannot rely on the response to free the resources.

@nicolas-grekas Here is a small isolated script to reproduce the issue (need lsof util and pcntl extension for verbose output):

<?php

use Symfony\Component\HttpClient\Internal\CurlClientState;

require dirname(__DIR__) . '/vendor/autoload.php';

$verbose = false;
$iterations = ceil(shell_exec('ulimit -n') / 2);

$state = new CurlClientState(6, 50);

for ($i = 0; $i < $iterations; $i++) {
    $state->reset();

    if ($verbose) {
        $output = [];
        exec('lsof -p ' . posix_getpid(), $output);
        var_dump(count($output));
    }
}

exec('ls');

results in:

PHP Warning:  exec(): Unable to fork [ls] in http_client_test.php on line 22

Changing $verbose to true exposes the stream handles leak 2 per each reset call:

int(136)
int(138)
...
int(1144)
int(1146)
int(1148)
PHP Warning:  exec(): Unable to fork [lsof -p 180109] in http_client_test.php on line 17
int(0)
PHP Warning:  exec(): Unable to fork [lsof -p 180109] in http_client_test.php on line 17
int(0)
PHP Warning:  exec(): Unable to fork [lsof -p 180109] in http_client_test.php on line 17
int(0)
PHP Warning:  exec(): Unable to fork [lsof -p 180109] in http_client_test.php on line 17
int(0)
PHP Warning:  exec(): Unable to fork [lsof -p 180109] in http_client_test.php on line 17
int(0)
PHP Warning:  exec(): Unable to fork [ls] in http_client_test.php on line 22

@jtojnar ok, so you’re experiencing something else. Please open an issue with as many details as possible when you have some hints.