playwright: [Regression]: Can't call test.use() inside a describe() suite of a different test

We are experiencing difficulties due to this recent update, and we’d like to share our code snippet to highlight its repercussions. Adapting to this change requires significant effort, as it’s disrupting the functionality of nearly all our tests.

The error we get is

Error: Can’t call test() inside a describe() suite of a different test type. Make sure to use the same “test” function (created by the test.extend() call) for all declarations inside a suite.

Our setup involves categorizing shows based on user access or the presence of a paywall, or sometimes both. This feature is critical not only for our organization but also for every other organization using Playwright. As others begin upgrading to the latest version, the absence of this feature will undoubtedly have a widespread impact on test organization for all users. Therefore, we kindly request the restoration of this feature to ensure smooth operations for all users.

import freeContentShow from 'path/to/freeContentMultipleSeasons.json';
import { expect, test } from 'path/to/test';

const { platform } = config;
const fixtures = {
  free: ['show1', 'show2', 'Season 5'],
  premium: ['showA', 'showB', 'Season 2']
};
const [freeShowUrl, premiumShowUrl, seasonName] = fixtures[platform];

test.describe('Test Description 1', () => {
  test.beforeEach(async ({ page, platform }) => {
    await page.route(`**/catalog/v2/${platform}/show/${freeShowUrl}*`, async (route) => {
      const json = freeContentShow[platform];
      await route.fulfill({ json });
    });
  });
  test('Test Name 1', async ({ page, platform }) => {
    await page.goto(`/${freeShowUrl}`);
    await expect(page.getByLabel(seasonName)).toHaveScreenshot(`free-season-${platform}.png`);
  });
});

test.describe('Test Description 2', () => {
  test.beforeEach(async ({ page, platform }) => {
    await page.route(`**/catalog/v2/${platform}/show/${premiumShowUrl}*`, async (route) => {
      const json = require(`path/to/showAllMediaPremium.json`);
      await route.fulfill({ json });
    });
  });
  test('Test Name 2', async ({ page, platform }) => {
    await page.goto(`/${premiumShowUrl}`);
    await expect(page.getByLabel(seasonName)).toHaveScreenshot(`premium-season-${platform}.png`);
  });
});

@pavelfeldman Any updates as to whether this breaking change will be reverted? Our playwright testing framework is built around this functionality, and we would very much like to maintain the current way that it is working, At the moment, we’re stuck on version 1.41, and we’re in a bit of uncertainty as to what the future holds (whether we will have to change our architectural approach completely or whether we should wait to see if this breaking change will be reverted).

We are affected by this change as well.

In our case we have test groupings which, after consuming dynamically the current combination under test, determines a particular version of the test group to be executed.

We’re affected by this change as well

not sure if it is related or not, but we’re on version 1.40

And getting this error:

Cannot use({ fixture1 }) in a describe group, because it forces a new worker.
Make it top-level in the test file or put in the configuration file.

where we try to do something like the code below. We use fixtures quite heavily and they set up a lot of our entities, but lately we’ve come to scenarios where we would potentially not want them invoked conditionally per test (so can’t use a global setting such as auto: true/false.

test.describe(
    'hello world',
    () => {
        if(condition) test.use({ fixture1: '' });
        test.only( 'just testing', async ({ fixture2, fixture1 }) => {
            // ...do some things
        })
    }
)

I updated the release notes to include this as a breaking change. Is there a reason you need to mix test instances? Can you fix the code?

@pavelfeldman There is no particular reason, I just thought it looked neat the way I wrote it. Since it wasn’t mentioned in the release notes I wasn’t sure if it was an intentional change or not.

I can also do this, it was only to avoid having fixtures like adminItem and normalItem without splitting into seperate files, so I nested the fixtures to have them both be item.

(If it is still wrong, since I have different test.use per describe block, please let me know.)

import { test as base } from '@playwright/test';

const test = base.extend({
  adminItem: async ({}, use) => {},
  normalItem: async ({}, use) => {},
});

test.describe('Admin user', () => {
  test.use({ storageState: '/admin/user/path' });

  test('1', async ({ page, adminItem }) => {});
  test('2', async ({ page, adminItem }) => {});
});

test.describe('Normal user', () => {
  test.use({ storageState: '/normal/user/path' });

  test('1', async ({ page, normalItem}) => {});
  test('2', async ({ page, normalItem }) => {});
});

Also affected here. I’ve popped a question on the Discord group but cross posting for visibility:

I have the following setup for visual testing:

describe('goes to confirm page', () => {
    beforeEach(async ({ page }) => {
        await formToConfirmation(page);
        await form.titlePage.waitFor();
    });
    
    test('@visual desktop', async ({ captureScreenshot }) => {
        await captureScreenshot('aml-contacts-desktop-confirm');
    });

    testMobile('@visual mobile', async ({ captureScreenshot }) => {
        await captureScreenshot('aml-contacts-mobile-confirm');
    });
});

testMobile comes from our playwright.config.ts file:

export const testMobile = test.extend<PlaywrightTestArgs>({
    viewport: { width: 414, height: 896 },
});

We’ve found this a nice way to manage our tests, however is the expectation now to have a separate describe block when the test instances change? It’s a shame as this seemed to offer quite a maintainable approach to our testing at scale.

Allowing the following code was an oversight on our side. Mixing the test instances (and hooks called on those instances) in the same suite prevents us from keeping the strict semantic.

const test = base.extend({ item: async ({}, use) => {} });
base.describe('Admin user', () => {
  test('1', async ({ page, item }) => {});
  test('2', async ({ page, item }) => {});
});

I’ll leave this change open to see how many people are affected by this.

I am also having issues with this change. I was using this to set up an electron app or browser within the same test suite.

I updated the release notes to include this as a breaking change. Is there a reason you need to mix test instances? Can you fix the code?