mongodb-odm: Memory leak when useing embedded documents

Bug Report

| Version | 2.0.3

Summary

I ran into a memory leak and it occurs when using EmbedMany annotation.

/**
 * @ODM\Document(collection="file_keyfames_hash")
 */
class FileKeyframesHash
{
    /**
     * @var Collection<FileKeyframesHash>
     * @ODM\EmbedMany(name="hashes_4", targetDocument=KeyframesHash::class)
     */
    private $hashes4;
}

/**
 * @ODM\EmbeddedDocument()
 */
class KeyframesHash
{
    /**
     * @var string
     * @ODM\Field(name="hash", type="bin")
     */
    private $hash;

    // ... getters/setters
}

        $cursor = $dm->createQueryBuilder(FileKeyframesHash::class)
            ->getQuery()
            ->execute();

        $count = 0;
        foreach ($cursor as $e) {
            /** @var FileKeyframesHash $e */
            $count++;
            if ($count % 10 === 0) {
                echo "Processed: {$count}", PHP_EOL;
                echo "Managed: {$dm->getUnitOfWork()->size()}", PHP_EOL;
                echo Helper::bytesHumanize(memory_get_usage()), PHP_EOL;
                echo PHP_EOL;
            }
            $dm->detach($e);
            unset($e);
        }

The problem occurs event when used $dm->clear() instead of $dm->detach($e)

UPD: HERE I PROVIDED TEST https://github.com/doctrine/mongodb-odm/issues/2162#issuecomment-584599533

example of my script output:


Processed: 100
Managed: 101
472.7 MB

Processed: 110
Managed: 111
538.9 MB

Processed: 120
Managed: 121
592.3 MB

Processed: 130
Managed: 131
684.9 MB

Processed: 140
Managed: 141
766.0 MB

Processed: 150
Managed: 151
851.1 MB

Processed: 160
Managed: 161
903.2 MB

Processed: 170
Managed: 171
967.1 MB

PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20480 bytes) in /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Iterator/HydratingIterator.php on line 106
PHP Stack trace:
PHP   1. {main}() /usr/local/app/yii:0
PHP   2. yii\console\Application->run() /usr/local/app/yii:32
PHP   3. yii\console\Application->handleRequest() /usr/local/app/vendor/yiisoft/yii2/base/Application.php:386
PHP   4. yii\console\Application->runAction() /usr/local/app/vendor/yiisoft/yii2/console/Application.php:147
PHP   5. yii\console\Application->runAction() /usr/local/app/vendor/yiisoft/yii2/console/Application.php:180
PHP   6. app\commands\TestController->runAction() /usr/local/app/vendor/yiisoft/yii2/base/Module.php:528
PHP   7. app\commands\TestController->runAction() /usr/local/app/vendor/yiisoft/yii2/console/Controller.php:164
PHP   8. yii\base\InlineAction->runWithParams() /usr/local/app/vendor/yiisoft/yii2/base/Controller.php:157
PHP   9. call_user_func_array:{/usr/local/app/vendor/yiisoft/yii2/base/InlineAction.php:57}() /usr/local/app/vendor/yiisoft/yii2/base/InlineAction.php:57
PHP  10. app\commands\TestController->actionExportHashes() /usr/local/app/vendor/yiisoft/yii2/base/InlineAction.php:57
PHP  11. Doctrine\ODM\MongoDB\Iterator\CachingIterator->next() /usr/local/app/commands/TestController.php:121
PHP  12. Generator->next() /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Iterator/CachingIterator.php:89
PHP  13. Doctrine\ODM\MongoDB\Iterator\CachingIterator->wrapTraversable() /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Iterator/CachingIterator.php:89
PHP  14. Doctrine\ODM\MongoDB\Iterator\HydratingIterator->next() /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Iterator/CachingIterator.php:156
PHP  15. Generator->next() /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Iterator/HydratingIterator.php:71
PHP  16. Doctrine\ODM\MongoDB\Iterator\HydratingIterator->wrapTraversable() /usr/local/app/vendor/doctrine/mongodb-odm/lib/Doctrine/ODM/MongoDB/Iterator/HydratingIterator.php:71

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 1
Comments: 49 (27 by maintainers)

Most upvoted comments

Had the same issue here, we do have a 3 million documents collection and using hydration true when having big embedded documents just explodes the memory, no matter what kind of clear mechanism we try (clear, detach, unset, gb, gc_collect_cycles…)

douglasjam on Nov 10, 2020

I spent a lot of time last week investigating these leaks, and it turns out this has nothing to do with CachingIterator or embedded documents. I was able to observe the leak, but did not observe any difference with respect to using a non-caching iterator, or using clear vs. using detach. I did however observe that we didn’t properly clean up references of an object as soon as HydratingIterator was involved. However, I wasn’t able to pinpoint where the object sticks around, I only observed that the refCount I could check with xdebug was higher than it should’ve been, and that removing HydratingIterator fixes the issue.

That said, I’ll have to spend some more time on this, which I currently don’t have. I’m moving this to the 2.0.7 milestone and hope to be able to take a closer look once I’ve got more time on my hands. In the meantime, please use clear or detach in combination with the non-caching (non-rewindable) iterator introduced in 2.1.0. The want to release 2.1.0 is one of the reasons why I don’t have time to investigate this, so I hope that this at least works around the problem for the time being, even though it’s not a full solution for it. Thanks for understanding.

alcaeus on May 28, 2020

@StalkAlex I think mongodb-odm team already fixed the issue, at least I do not face this issue anymore. Our practice here is to always use rewindable(false) readOnly(true) and keep clearing after batches.

douglasjam on Dec 1, 2021

I little change test of @TheHett and what I got

Also I change access modifier for \Doctrine\ODM\MongoDB\Iterator\CachingIterator::getIterator from private on public for test him

Test

<?php declare(strict_types=1);

namespace Doctrine\ODM\MongoDB;

use Doctrine\ODM\MongoDB\Iterator\CachingIterator;
use Doctrine\ODM\MongoDB\Tests\BaseTest;
use Documents\Bars\Bar;

class MemoryLeakTest extends BaseTest
{
    /**
     * @throws MongoDBException
     */
    public function setUp(): void
    {
        parent::setUp();
        for ($j = 0; $j < 1000; $j++) {
            $bar = new Bar($this->generateRandomString(10000));
            $this->dm->persist($bar);
        }
        $this->dm->flush();
    }

    public function tearDown(): void
    {
        $this->dm->clear();
        parent::tearDown();
    }

    private function memUsage(): void
    {
        gc_collect_cycles();
        echo memory_get_usage(), PHP_EOL;
    }

    /**
     * @throws MongoDBException
     */
    public function testCacheIteratorWithDetach(): void
    {
        $query = $this->dm->createQueryBuilder(Bar::class)
            ->readOnly(true)
            ->getQuery();

        $iterator = $query->execute();

        $this->assertInstanceOf(CachingIterator::class, $iterator);

        $counter = 0;
        $this->memUsage();
        foreach ($iterator as $bar) {
            if ($counter % 100 === 0) {
                $this->memUsage();
            }
            $counter++;
            $this->dm->detach($bar);
        }
        $this->memUsage();
    }

    /**
     * @throws MongoDBException
     */
    public function testIteratorFromCacheIteratorWithDetach(): void
    {
        $query = $this->dm->createQueryBuilder(Bar::class)
            ->readOnly(true)
            ->getQuery();

        $iterator = $query->execute();

        $this->assertInstanceOf(CachingIterator::class, $iterator);

        $iterator = $iterator->getIterator();

        $counter = 0;
        $this->memUsage();
        foreach ($iterator as $bar) {
            if ($counter % 100 === 0) {
                $this->memUsage();
            }
            $counter++;
            $this->dm->detach($bar);
        }
        $this->memUsage();
    }

    function generateRandomString(int $length): string {
        $characters = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ';
        $charactersLength = strlen($characters);
        $randomString = '';
        for ($i = 0; $i < $length; $i++) {
            $randomString .= $characters[random_int(0, $charactersLength - 1)];
        }
        return $randomString;
    }
}

Result

Memory Leak (Doctrine\ODM\MongoDB\MemoryLeak)
 ✔ Cache iterator with detach
20976440
20976584
22256584
23532808
24813128
26085256
27357384
28645896
29918024
31190152
32462280
33719216
 ✔ Iterator from cache iterator with detach
21342632
21342728
21355576
21355704
21355832
21355960
21356088
21356216
21356344
21356472
21356600
21341064

serieznyi on Apr 5, 2020

Thanks for providing the test case. I’ll take a deeper look at it later.

alcaeus on Feb 11, 2020