guzzle: using GuzzleHttp\Pool doesn't free up memory after request is fulfilled

I need to download a lot of remote packages. I am using the Pool function according to the documentation here: http://docs.guzzlephp.org/en/latest/quickstart.html#making-a-request

I’m not sure how I can free up the used memory. The packages are all small but several tens of thousands. How can I properly unset/free $response? unset(), $response->getBody()->close(); all seem not to do what I want.


use GuzzleHttp\Pool;
use GuzzleHttp\Client;
use GuzzleHttp\Psr7\Request;

require __DIR__ . '/vendor/autoload.php';

$client = new Client(['base_uri' => '...']);

$requests = function ()
{
    ...

    yield $index => new Request('GET', ...);
};

$pool = new Pool($client, $requests(), [
    'concurrency' => 50,
    'fulfilled' => function ($response, $index)
    {
        $content = $response->getBody()
                            ->getContents();

        file_put_contents('storage/' . $index, $content);

        print 'fulfilled index:' . $index . PHP_EOL;
    },
    'rejected' => function ($reason, $index)
    {
        print 'rejected index:' . $index . PHP_EOL;
    },
]);

$promise = $pool->promise();

$promise->wait();

About this issue

Original URL
State: closed
Created 8 years ago
Reactions: 6
Comments: 36 (10 by maintainers)

Most upvoted comments

Hey guys, I was experiencing the same issue. Every guzzle request was adding 0.07 - 0.2 MB, so 10K requests were causing memory crash.

Then I realized that Guzzle was dev mode because Symfony commands are working in dev mode by default. Adding “–env=prod” to the command line reduced 1GB of consumed memory down to 50MB 😃

istrof on Aug 18, 2017

Подтверждаю, действительно жрёт память в цикле. Даже если unset основного объекта выполнить. Это печаль (((

craftsman-expert on Mar 11, 2019

Also same problem here. CPU going crazy when making lots of calls. @alekseykuleshov we understand that environments are a framework feature, but how can we directly set Guzzle to production mode? There is no reference at all to “dev mode” in Guzzle docs.

pablolemosassis on Aug 22, 2017

I seem to have the same problem. On every call I make the memory increases. The problem is that my scripts extracts 20.000 values from an excel file and uploads them with the api. In the end everything is very slow because every cycle adds 0.15 mb.

PMassoels on Jul 14, 2017

To add more clarification to this issue. From what I was able to debug in our setup. We use Guzzle to perform more than 1000 request per second to multiple upstream server. We recently experienced a massive memory leak within PHP and a corresponding CPU peak. We were able to downtrack to issue to the usage of cURL multi handle. Because Guzzle is very widespread and the default HTTP handler chosen by Guzzle is CurlMultiHandler this problem looks like it Guzzle related. We where able to resolve the issue by choosing the dedicated StreamHandler. To avoid curl in any way we disabled curl_init, curl_multi_init and curl_exec by adding this functions to php.ini disable_functions directive. Currently Guzzle assumes that curl or multi curl is the best default handler. This might be up for discussion until the memory leak in the libcurl PHP core bug is fixed.

See https://www.drupal.org/project/drupal/issues/2595927 for a similar conclusion.

dol on May 27, 2019

Yea, cool, after update on eightpoints/guzzle-bundle 7.4 in my main project the problem is gone facepalm. In my post above with the simple example I have changed the APP_ENV to the prod only. This brought the code to the constant memory usage.

Maniumn on Nov 19, 2018

I think the needs-more-information tag is obsolete currently, isn’t it?

bong0 on Oct 9, 2017

So… how does one actually get Guzzle out of this this mystical ‘dev mode’ to prevent the memory leak? We are seeing the same .15 MB leak/call… not great when dealing with large numbers of requests…

celloman on Jun 26, 2018

I too have a similar problem. I’m using Guzzle to loop through a product inventory (between 100 and 20,000 items) and API calls (limit 100 at a time). I only do GET requests, quickly take the response as string, parse it and dump it in a file. However, the peak memory usage keeps increasing.

I also tried using request sink (in case the php://temp was saving the quest in memory), but the total memory used easily gets in the 25MB just after 2-3 calls. Tried to run the script with 20,000 items and it passed 540MB of memory.

Is there a way to release resources used by the client after a successful GET requests?

vallieres on Apr 13, 2016

I have same problem ( PHP 7.3.20 or PHP 7.4.8, Symfony, RoadRunner, curl 7.29.0 , CentOS 7, guzzle 6.5.5) When i send any Guzzle request or just create Promise(without real send request) i have leak(many gc_status->roots, mem). The tip from above(HTTP 1.0) actually works partially, it removes the memory problem, but does not remove the problem with the roots that fill the garbage collector pool. It looks like a crutch solution.

Has anyone found a completely correct solution? Maybe someone knows why it works like this?

Upd: @ekojs solution with GuzzleHttp\Pool and Iterator works too

codemistake on Jul 23, 2020

I think i get a memory leak like this when repeating the same call from my client

kgrosvenor on May 8, 2018

@mtdowling Thanks for the suggestion! however, I don’t see any noticeable drop in memory usage. I call gc_collect_cycles() after a request has finished.

drsect0r on Mar 7, 2016