composer: Fatal Error : out of memory with Composer Update

There may actually be a use case I think to have a composer optimize command, which takes root constraints like this and makes them a ton more efficient by rewriting them from the lock file. So if, in this specific case, doctrine/doctrine-cache-bundle was resolved to 1.3.5 in a successful solution, we could replace the constraint safely with ^1.3.5 and it would by definition produce the same solution.

Potential risk would be people running it on libraries as well which could be dangerous, but we could print some nasty warnings if composer.json type is not project. Also we would have to check that the new constraint is only narrower on the low end, so that we don’t replace ^5.6 || ^6.1 with ^5.6.35 just because it resolved to 5.6.35 in a specific situation. It’ll be hard to optimize anything other than single caret constraints in practice, but again more complex constraints belong in libraries, not applications.

@Seldaek whatcha think of this?

Yep, that’s what I would expect to see, as the constraint covers 3 more versions it causes a few more rules to be generated, but as all 4 have the same dependencies themselves it shouldn’t really impact the whole process a lot.l

I decided to continue the test. After the above solution, I checked which versions of root packages were resolved:

dantleech/phpcr-migrations-bundle    0.1                PHPCR migrations bundle
doctrine/doctrine-bundle             1.10.0             Symfony DoctrineBundle
doctrine/doctrine-cache-bundle       1.3.5              Symfony Bundle for Doctrine Cache
doctrine/doctrine-fixtures-bundle    v2.2.1             Symfony DoctrineFixturesBundle
incenteev/composer-parameter-handler v2.1.3             Composer script handling your ignored parameter file
jackalope/jackalope-doctrine-dbal    1.3.2              Jackalope Transport library for Doctrine DBAL
jackalope/jackalope-jackrabbit       1.3.2              Jackalope Transport library for Jackrabbit
massive/build-bundle                 0.3.1              Massive Art Build Bundle
matthiasmullie/minify                1.3.61             CSS & JavaScript minifier, in PHP. Removes whitespace, strips comments, combines files (incl. @import statements and small assets in CSS files), and optimizes/shortens a few commo...
oro/doctrine-extensions              1.2.2              Doctrine Extensions for MySQL and PostgreSQL.
sensio/distribution-bundle           v5.0.24            Base bundle for Symfony Distributions
sensio/framework-extra-bundle        v3.0.12            This bundle provides a way to configure your controllers with annotations
sensio/generator-bundle              v3.1.7             This bundle generates code for you
sulu/sulu                            1.6.23             Sulu core distribution
sulu/theme-bundle                    1.3.2              The Sulu Bundle which provides themes
symfony/assetic-bundle               v2.8.2             Integrates Assetic into Symfony2
symfony/monolog-bundle               v2.12.1            Symfony MonologBundle
twig/extensions                      v1.5.4             Common additional features for Twig that do not directly belong in core
zendframework/zend-stdlib            2.7.7             
zendframework/zendsearch             dev-master c8de250 a general purpose text search engine written entirely in PHP 5

And adapted my composer.json accordingly, ,following the “rules” I outlined above only replacing pure caret constraints with the resolved version:

"require": {
    "php": "^7.0",
    "sulu/sulu": "^1.6.23",
    "twig/extensions": "^1.5.4",
    "symfony/monolog-bundle": "^2.12.1",
    "sensio/distribution-bundle": "^5.0.24",
    "sensio/framework-extra-bundle": "^3.0.12",
    "incenteev/composer-parameter-handler": "^2.1.3",
    "dantleech/phpcr-migrations-bundle": "0.1.*",
    "zendframework/zend-stdlib": "^2.7.7",
    "zendframework/zendsearch": "@dev",
    "massive/build-bundle": "^0.3.1",
    "jackalope/jackalope-doctrine-dbal": "^1.3.2",
    "jackalope/jackalope-jackrabbit": "^1.3.2",
    "doctrine/doctrine-fixtures-bundle": "2.2.*",
    "doctrine/doctrine-bundle": "^1.10.0",
    "doctrine/doctrine-cache-bundle": "^1.3.5",
    "oro/doctrine-extensions": "^1.2.2",
    "sensio/generator-bundle": "^3.1.7",

    "sulu/theme-bundle": "^1.3.2",
    "symfony/assetic-bundle": "^2.8.2",
    "matthiasmullie/minify": "^1.3.61"
}

It got a lot faster, but indeed not much less RAM:

[374.6MB/113.96s] Memory usage: 374.59MB (peak: 1741.24MB), time: 113.96s

At which point I realized something that was bugging me before about the composer.json you provided - there was no explicit mention of Symfony anywhere, you’re depending on a full stack Symfony version being decided on recursively, which eventually leads to symfony/symfony:2.8.49 being installed. This is highly dangerous, as many of those libraries are compatible with a lot of Symfony versions (ie. ^2.8 || ^3.2 || ^4.0, or >2.8 or similar incredibly wide constraints, given the amount of releases Symfony has. This is more or less the exact reason Symfony Flex was developed, to battle the sheer amount of Symfony packages Composer has to consider that are compatible with hundreds of other packages.

So I added a single line to your require list, to disqualify all 3.x and 4.x releases of all components explicitly:

    "symfony/symfony": "^2.8.49"

And ran composer update --profile again:

[341.0MB/12.23s] Memory usage: 340.96MB (peak: 1172.99MB), time: 12.23s

Achievement unlocked - from 380 seconds down to 12 with 2 simple optimizations aimed at making Composer’s life easier 😉

This leaves the question why in 12 seconds it’s still using 1.2GB of RAM. The answer to that is also simple actually - Composer is optimized towards up to date development. It downloads and processes package info for the last 3 months first, and starts to load older data incrementally as it turns out to be needed by the solver. This way in the case of Symfony for example it can skip years of outdated versions if you’re just working with the current 4.2 release. In your case you are using 3 year old dependencies, giving it little other choice than to “waste” over a GB of RAM on more or less the past 5 years of Packagist releases. Hence why my much bigger project only used 778MB of RAM - it’s on Symfony 4.2 with cutting edge dependencies.

Hope this all explains a lot, and happy coding now your composer update only takes 10~12 seconds 😆

Would a compiled SAT solver PHP extension help?

It would likely make the process more than 10 times faster with 10% of the RAM. Feel free to issue a PR 😉

@curry684 Your suggestion reminds me of #7273.

Hi Guys, I have spent couple of hours to solve memory issue during composer upate on my projects. I have the latest version of composer 1.7.3, run PHP 7.2, OS win7 x64, RAM 16GB and have no luck at all. On every project it gives me out of memory. I have changed memory limit to unlimited, no luck, which is strange. I have checked the code inside composer.phar and find out this code:

// Increase memory_limit if it is lower than 1.5GB if ($memoryLimit != -1 && $memoryInBytes($memoryLimit) < 1024 * 1024 * 1536) { @ini_set(‘memory_limit’, ‘1536M’); }

So the memory limit is set to 1.5GB minimum, which is not enough in my case.

The solution on my installation was to switch from 32bit to 64bit version of PHP. Reason is simple, to get more than 2GB memory available. When I run composer with command:

php -d memory_limit=-1 composer.phar update --profile -vvv

It gives me at the end of success run this result: Memory usage: 322.37MB (peak: 1804.41MB), time: 178.78s

So memory usage is really high and maybe it is time to talk about necessity to use composer with PHP 64bit as 32bit has 2GB memory limitation. I went through so many discussion about this issue but mostly there is no solution. As this issue is here so long time and obviously I am not alone should be nice to update official documentation to mention this option.

Thank you, love composer so much!

Best regards, Lukas