foundry: Performance Problem with random

While diagnosing why some of our tests are so slow, I discovered the culprit for us is with random(). We have a function:

public static function randomApiUser(): ApiUser
    {
        $user = self::random()->object();
        return new ApiUser($user);
    }

And it was making tests take five times as long as when we didn’t use it.

After poking around, I noticed that rather than getting X random records, it’s getting ALL of them and then getting random ones from that.

public function randomRange(int $min, int $max, array $attributes = []): array
    {
        if ($min < 0) {
            throw new \InvalidArgumentException(\sprintf('$min must be positive (%d given).', $min));
        }

        if ($max < $min) {
            throw new \InvalidArgumentException(\sprintf('$max (%d) cannot be less than $min (%d).', $max, $min));
        }

        $all = \array_values($this->findBy($attributes));

        \shuffle($all);

        if (\count($all) < $max) {
            throw new \RuntimeException(\sprintf('At least %d "%s" object(s) must have been persisted (%d persisted).', $max, $this->getClassName(), \count($all)));
        }

        return \array_slice($all, 0, \random_int($min, $max)); // @phpstan-ignore-line
    }

I think it would be more performant if the code used something like this to just get random records from the database (with the appropriate attribute filters):

SELECT * 
FROM table_name
ORDER BY RAND()
LIMIT 1;

I am willing to do a PR for this if you would like.

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 23 (22 by maintainers)

Most upvoted comments

Alright, got some time tonight to do it 👍🏻

There we go, I created two PRs. I see that at least one of them has some psalm issues but one of the reasons being that phpbench is not installed there. But alright, maybe best to discuss that there and not pollute this issue.

about randomRange(): I’d really like that we keep only relying on doctrine/persistence.

Based on this constraint, I can only see half solutions, both being trade-offs:

  • we can use what you’ve done with random() and perform only one query, and change the offset… This would be kinda fake random, but I think this is ok-ish for testing purpose
  • we can perform as many queries as requested objects, at the price of performance (but still be better than the current implementation), but this would give real randomness

Hi @mdeboer

this is very cool! We were actually talking with @kbond the we would like to add a CI check about performances, using PHPBench, maybe based on this example

Would you mind to create two PRs? one which just fixes the random stuff, and another one with your work on PHPBench?

If you’re willing to introduce the GitHub Action, this is super cool, but we can also handle this ourselves, based on your work

Alright, so fixing this for random() (one random entity) is easy and yields a nice improvement. But I haven’t spend much time working on randomSet() and randomRange(), they seem a bit harder to implement at first glance.

We can work on randomRange() another time.

Another thing to consider is that for example we use this in our tests and we usually seed the RNG to get random yet consistent results. This would not be possible with ORDER BY RAND() as far as I know

makes totally sense.

thanks!

Alright, so fixing this for random() (one random entity) is easy and yields a nice improvement. But I haven’t spend much time working on randomSet() and randomRange(), they seem a bit harder to implement at first glance.

Most time went to setting up phpbench but I got it working. Writing benchmarks is pretty similar to writing either unit or integration tests now.

Anyway, here are the results for random() I got with ORM so far. Comparing the original implementation in the 2.x repository with my implementation I explained before (using the random offset):

PHPBench (1.4.1) running benchmarks... #standwithukraine
with configuration file: /home/maarten/Projects/Forks/foundry/phpbench.json
with PHP version 8.4.5, xdebug ❌, opcache ❌
comparing [actual vs. before]

\Zenstruck\Foundry\Tests\Benchmark\ORM\GenericEntityFactoryBench

    bench_random # 1........................I9 - [Mo2.111ms vs. Mo2.020ms] +4.48% (±3.89%)
    bench_random # 50.......................I9 - [Mo2.464ms vs. Mo3.504ms] -29.68% (±6.08%)
    bench_random # 500......................I9 - [Mo2.738ms vs. Mo11.773ms] -76.75% (±9.23%)
    bench_random # 1000.....................I9 - [Mo3.515ms vs. Mo35.876ms] -90.20% (±12.87%)

Subjects: 1, Assertions: 0, Failures: 0, Errors: 0
+---------------------------+--------------+------+------+-----+------------------+-----------------+-----------------+
| benchmark                 | subject      | set  | revs | its | mem_peak         | mode            | rstdev          |
+---------------------------+--------------+------+------+-----+------------------+-----------------+-----------------+
| GenericEntityFactoryBench | bench_random | 1    | 1    | 10  | 36.310mb +0.00%  | 2.111ms +4.48%  | ±3.89% -2.19%   |
| GenericEntityFactoryBench | bench_random | 50   | 1    | 10  | 37.108mb +0.00%  | 2.464ms -29.68% | ±6.08% -59.30%  |
| GenericEntityFactoryBench | bench_random | 500  | 1    | 10  | 45.293mb -1.22%  | 2.738ms -76.75% | ±9.23% -74.94%  |
| GenericEntityFactoryBench | bench_random | 1000 | 1    | 10  | 59.239mb -10.32% | 3.515ms -90.20% | ±12.87% -47.94% |
+---------------------------+--------------+------+------+-----+------------------+-----------------+-----------------+

Depending on the number of items in the repository, performance is increased by 30 to 90%! 😍 Also the stability in speed is improved considerable (50-75%).

You can find my changes here: https://github.com/mdeboer/foundry/tree/feat/speedup-random

If you like I can make a PR 👍🏻

Another thing to consider is that for example we use this in our tests and we usually seed the RNG to get random yet consistent results. This would not be possible with ORDER BY RAND() as far as I know.

I’m pulling up the code now to have a look at random() and randomRange() 😃

I just saw your edit: isn’t doctrine/persistence + dbal helps with this? here is the prototype of ObjectRepositroy::findBy():

    public function findBy(
        array $criteria,
        ?array $orderBy = null,
        ?int $limit = null,
        ?int $offset = null
    );

it actually has limit + offset parameters, I think your approach is ok for any DBs!

@nikophil I will have a look this weekend! 👍🏻

@nikophil - OK. I will pull the 2.0 branch and play locally. Hoping I can find a way to do it in a database agnostic way.