tenancy: [4.x] Testing tenants is slow

I’ve been able to use Parallel Testing with php artisan test -p --recreate-databases

<?php

namespace Tests;

use App\Models\Tenant;
use Illuminate\Foundation\Testing\RefreshDatabase;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\ParallelTesting;
use Illuminate\Support\Facades\URL;

trait RefreshDatabaseWithTenant
{
    use RefreshDatabase {
        beginDatabaseTransaction as parentBeginDatabaseTransaction;
    }

    /**
     * The database connections that should have transactions.
     *
     * `null` is the default landlord connection
     * `tenant` is the tenant connection
     */
    protected array $connectionsToTransact = [null, 'tenant'];

    /**
     * We need to hook initialize tenancy _before_ we start the database
     * transaction, otherwise it cannot find the tenant connection.
     */
    public function beginDatabaseTransaction()
    {
        $this->initializeTenant();

        $this->parentBeginDatabaseTransaction();
    }

    public function initializeTenant()
    {
        $tenantId = 'acme';

        $tenant = Tenant::firstOr(function () use ($tenantId) {
            config(['tenancy.database.prefix' => config('tenancy.database.prefix') . ParallelTesting::token() . '_']);

            $dbName = config('tenancy.database.prefix') . $tenantId;

            DB::unprepared("DROP DATABASE IF EXISTS `{$dbName}`");

            $t = Tenant::create(['id' => $tenantId]);

            if ( ! $t->domains()->count()) {
                $t->domains()->create(['domain' => $tenantId . '.localhost']);
            }

            return $t;
        });

        tenancy()->initialize($tenant);

        URL::forceRootUrl('http://acme.localhost');
    }
}

However, for some tests DatabaseMigrations Trait will be needed instead of RefreshDatabase. As soon as I need it and get it ready, will also post here.

UPD: Also adding this container to docker-compose.yml speeded up tests significantly (I use Laravel Sail)

    mysqlt:
        image: 'mysql:8.0'
        ports:
            - '${FORWARD_DB_PORT:-3307}:3306'
        environment:
            MYSQL_ROOT_PASSWORD: '${DB_PASSWORD}'
            MYSQL_DATABASE: '${DB_DATABASE}'
        tmpfs:
            - /var/lib/mysql
        networks:
            - sail
        healthcheck:
          test: ["CMD", "mysqladmin", "ping", "-p${DB_PASSWORD}"]
          retries: 3
          timeout: 5s

Please note, that you’ll also need this update DB_HOST=mysqlt in `.env.testing’.

Yeah, it’s on my roadmap to make a few traits like those provided by Laravel. (e.g. to only run migrations on phpunit setup rather than every single test setup).

There could be a trait for tests that don’t need to know about tenant creation (because they don’t test the central app) and only test parts of the tenant app. That trait could create a tenant & DB on phpunit setup and then cleanup after all tests are run.

I was running into this issue as well. I’ve made a slightly adapted RefreshDatabase trait that works for me. Tests were previously running for more than a second each, with this trait it was reduced back to 300ms (as it was before using multi-database tenancy).

<?php

namespace Tests;

use App\Tenant\Models\Tenant;
use Illuminate\Support\Facades\URL;
use Illuminate\Foundation\Testing\RefreshDatabase;

trait RefreshDatabaseWithTenant
{
    use RefreshDatabase {
        beginDatabaseTransaction as parentBeginDatabaseTransaction;
    }

    /**
     * The database connections that should have transactions.
     *
     * `null` is the default landlord connection
     * `tenant` is the tenant connection
     */
    protected array $connectionsToTransact = [null, 'tenant'];

    /**
     * We need to hook initialize tenancy _before_ we start the database
     * transaction, otherwise it cannot find the tenant connection.
     */
    public function beginDatabaseTransaction()
    {
        $this->initializeTenant();

        $this->parentBeginDatabaseTransaction();
    }

    public function initializeTenant()
    {
        $tenant = Tenant::firstOr(fn () => Tenant::factory()->name('Acme')->create());

        tenancy()->initialize($tenant);

        URL::forceRootUrl('http://acme.localhost');
    }
}

@viicslen can you share a complete setup / config? Meaning Pest.php, TestCase, tenant aware test, and a central aware test?

I’ll reopen this since testing is something we want to focus on in v4 👍🏻

In my case, I was able to run tests much faster using schema:dump command. To create a tenant schema dump you can run - artisan tenants:run schema:dump

There are 3 types of tests:

• central only - these test purely central functionality, completely separate from any tenancy things. These tests can use Laravel’s testing traits.
• central & tenant - tests that interact with both parts of the application, e.g. a signup form that creates tenants. The tenant creation is up to the test, so that part can’t be made more performant, but we could at least store the central DB empty post-migration state in a transaction and roll back any changes to central DB after each test (this would delete tenants, domains, but only assuming the DB storage driver is used)
• tenant only - these tests don’t care about central anything, they run purely in the tenant app

The third category is what I expect to be the biggest group of tests for all apps that use this package, so let’s focus on that.

Assuming the DB storage driver is used, we can:

• hold the state of an empty database seeded with one tenant and one domain for the central database. Roll back to that after every test
• hold the state of an empty, post-migration tenant database in another transaction (on the tenant connection) and use that as the base for each test

Later we can do something like Snipe migrations for even faster tests, but this is a good start.

I will have to look into how the Laravel migration traits work, because Application state is not persisted between tests (for obvious reasons), but the transaction persists. My concern is that Application being destroyed between tests could also destroy the tenant connection, which would probably break the transaction. But that shouldn’t be a big issue, if RefreshDatabase works, we should be able to make a similar trait.

Ended up writing my own static method that creates a test tenant once when the first tenant-related test is run. Importantly, this means only one tenant is created.

Additionally, to help with cleanup, this method also keeps track of how many tests need to run (phpunit --list-tests | grep '-' | wc -l), and what the current test is. Then on the last test, it deletes the test tenant (which also triggers the jobs for deleting S3 bucket and Stripe customer).

The trade-off of speed vs. testing “best practice” (i.e. creating a fresh environment/application for each test) is very clear here. The code I use is super specific to my use case, but happy to answer questions on how it works.

If there’s enough interest, I might be able to package it up or create a PR to include as a testing helper trait.

Here’s what I have been using so far and it has been working perfectly

trait WithTenancy
{
    protected function setUpTraits(): array
    {
        $uses = parent::setUpTraits();

        if (isset($uses[WithTenancy::class])) {
            $this->initializeTenancy($uses);
        }

        return $uses;
    }

    protected function initializeTenancy(array $uses): void
    {
        $organization = Organization::firstOr(static fn () => Organization::factory()->create());
        tenancy()->initialize($organization);

        if (isset($uses[DatabaseTransactions::class]) || isset($uses[RefreshDatabase::class])) {
            $this->beginTenantDatabaseTransaction();
        }

        if (isset($uses[DatabaseMigrations::class]) || isset($uses[RefreshDatabase::class])) {
            $this->beforeApplicationDestroyed(function () use ($organization) {
                $organization->delete();
            });
        }
    }

    public function beginTenantDatabaseTransaction(): void
    {
        $database = $this->app->make('db');

        $connection = $database->connection('tenant');
        $dispatcher = $connection->getEventDispatcher();

        $connection->unsetEventDispatcher();
        $connection->beginTransaction();
        $connection->setEventDispatcher($dispatcher);

        $this->beforeApplicationDestroyed(function () use ($database) {
            $connection = $database->connection('tenant');
            $dispatcher = $connection->getEventDispatcher();

            $connection->unsetEventDispatcher();
            $connection->rollBack();
            $connection->setEventDispatcher($dispatcher);
            $connection->disconnect();
        });
    }
}

Then on the tests that need it I just have to reference it like so:

PestPHP

use Illuminate\Foundation\Testing\DatabaseTransactions;
use Tests\WithTenancy;

uses(WithTenancy::class);
uses(DatabaseTransactions::class);

// ...

PHPUnit

namespace Tests\Feature;

use Illuminate\Foundation\Testing\DatabaseTransactions;
use Tests\TestCase;
use Tests\WithTenancy;

class ExampleTest extends TestCase
{
	use WithTenancy;
	use DatabaseTransactions;

	// ...
}

Seems like a very clean implementation, thanks a lot Erik! I’ll test it when I start working on v4.

BTW, did someone find a way to use php artisan schema:dump with multi-database multi-tenancy?

With v3 coming soon, I’d like to make this part of 3.0 or 3.1.

Actually I was thinking about it a bit differently. My thought was to have 1 or more databases for testing tenants. Maybe like tenant_1_db and tenant_2_db. These would be set up beforehand by the developer. Like SnipeMigrations, it will run the migrations once then take a snapshot of the databases in the “clean” state. After initial db intialization by importing the snapshot, the databases would use database transactions for each test so that no actual data gets saved to the test tenant databases. If a tenant migration file changes the app would know about it, re-run the tenant migrations and take a new mysql snapshot for future imports. The “central” database would be pre-seeded with the tenants belonging to the 2 pre-created databases, using custom db names. Snipe already allows a seeder to run before the snapshot is taken.

This way we can avoid the slow part which is database creation and running migrations. Of course if a developer needed more than the pre-provisioned number of tenants for testing they could create a new one in the test. Since this isn’t a requirement most of the time the tests should run really fast.

Is there anything that would prevent us from going in this direction?

Fixed in v4

When will it be available to the general public?

It depends, just try in your project

any update ? and maybe migrate to using LazilyRefreshDatabase

@stancl I wrote a package the hijacks the Refresh Database trait. I’m not sure, but you may be able to find some useful code in there to make this happen. If I get some free time in the upcoming weeks I might just add it myself. This is a major pain point for us right now too.

https://laravel-news.com/snipe-migrations-laravel-package