playwright: [BUG] Test runner hangs during regression run in v1.35

System info

• Playwright Version: [v1.35.1]
• Operating System: [macOS 13.4]
• Browser: [All]
• VSCode: 1.79.2
• node: 16.15.1

Source code

Config file

import { defineConfig, devices } from "@playwright/test";
import dotenv from "dotenv";

/**
 * See https://playwright.dev/docs/test-configuration.
 */

/**
 * Read environment variables from file.
 * https://github.com/motdotla/dotenv
 */
dotenv.config();

export default defineConfig({
  testDir: "./tests",
  projects: [
    { name: "setup", testMatch: ["**/*.setup.ts"], teardown: "teardown" },
    { name: "teardown", testMatch: ["**/*.setup.ts"] },
    {
      name: "chromium",
      use: {
        ...devices["Desktop Chrome"],
        viewport: { width: 1280, height: 768 }
      },
      testMatch: ["**/*.spec.ts"],
      dependencies: ["setup"]
    },
    {
      name: "firefox",
      use: {
        ...devices["Desktop Firefox"],
        viewport: { width: 1280, height: 768 }
      },
      testMatch: ["**/*.spec.ts"],
      dependencies: ["setup"]
    },
    {
      name: "webkit",
      use: {
        ...devices["Desktop Safari"],
        viewport: { width: 1280, height: 768 }
      },
      testMatch: ["**/*.spec.ts"],
      dependencies: ["setup"]
    }
  ],
  /* Shared settings for all projects. See https://playwright.dev/docs/api/class-testoptions. */
  use: {
    /* Maximum time each action such as `click()` can take. Defaults to 0 (no limit). */
    actionTimeout: 0,
    /* Base URL to use in actions like `await page.goto('/')`. */
    baseURL: process.env.BASE_URL,
    trace: "retain-on-failure",
    headless: true
    /*launchOptions: {
      slowMo: 100 //Slow test execution down by 100ms, helpful for debugging
    }*/
  },
  /* Maximum time one test can run for. */
  timeout: 2 * 60 * 1000,
  globalTimeout: 25 * 60 * 1000,
  expect: {
    /**
     * Maximum time expect() should wait for the condition to be met.
     * For example in `await expect(locator).toHaveText();`
     */
    timeout: 5000
  },
  maxFailures: process.env.CI ? 16 : undefined,
  /* Run tests in files in parallel */
  fullyParallel: false,
  /* Fail the build on CI if you accidentally left test.only in the source code. */
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 3 : 1,
  /* Use 3 works locally. Use workers equal to 50% of available cores on CI - UNTESTED */
  workers: process.env.CI ? "50%" : 3,
  /* Reporters to use. See https://playwright.dev/docs/test-reporters */
  reporter: [
    ["html", { open: "never" }],
    ["./reporters/resultChecker"], //This outputs a file with the current branch and overall run result, to be consumed by GitHub Actions
    ["./reporters/failed-test-reporter.ts"], //This outputs a log of which tests have failed the last time they were executed, so just the failures can be rerun while debugging
    ["./reporters/better-dot-reporter.ts"]    
  ]
});

Steps

• Upgrade to Playwright 1.35 (previously using 1.34)
• Run our full regression suite via a VSCode terminal using pnpx playwright test --project chromium (ours is a monorepo managed with Rush so we don’t use npx)

Expected Test run will complete as per usual

Actual The test run stalls part way through, usually close to the end, and doesn’t complete. We didn’t previously have a globalTimeout value set in our config so I added one to help with trying to diagnose this. Once this was set, the runner will abort when that value is exceeded (set to 25 minutes as shown in the config above, a successful run usually takes about 10 minutes) but without the timeout configured I would have to cancel it manually using ctrl+C in the VSCode terminal once it become obvious that the runner was no longer responding.

I appreciate this will be a hard one to reproduce as it seems specific to something in our suite. Unfortunately I’m not able to provide our entire repo due to the proprietary information it contains about our application. The problem has occurred for at least two team members (me and one of our developers) so it seems to be common across the board when using 1.35.

I have completed a successful run with 1.34 and a hung/aborted run with 1.35 both using DEBUG=pw:api and saved output the logs to file. The files include bearer tokens so I would rather not attach them here for security reasons but would be very happy to email them direct if they will potentially help with identifying a cause.

Dot reporter output from most recent run which was aborted by the global timeout:

Running 158 tests using 3 workers
··········································································°·····
···················°······°°······································°··°×T
Timed out waiting 1500s for the test suite to run

Timed out waiting 1500s for the teardown for start workers to run


  1) [chromium] › 04_managedentities.spec.ts:894:9 › Managed Entities › Create & manage new Legal Entity › ME15 - Deleting a business entity with ownership stake in another business entity @notvisual 

    "afterAll" hook timeout of 120000ms exceeded.

      969 |     });
      970 |
    > 971 |     test.afterAll(async () => {
          |          ^
      972 |       await deleteAllLegalEntities();
      973 |     });
      974 |   });

        at /Users/sarahwoodhouse/Documents/repos/core-platform/apps/gelt-frontend-autotests/tests/04_managedentities.spec.ts:971:10

    Retry #1 ───────────────────────────────────────────────────────────────────────────────────────

    "afterAll" hook timeout of 120000ms exceeded.

      969 |     });
      970 |
    > 971 |     test.afterAll(async () => {
          |          ^
      972 |       await deleteAllLegalEntities();
      973 |     });
      974 |   });

        at /Users/sarahwoodhouse/Documents/repos/core-platform/apps/gelt-frontend-autotests/tests/04_managedentities.spec.ts:971:10

  Slow test file: [chromium] › 01_taxprofile.spec.ts (4.7m)
  Slow test file: [chromium] › 03_docvault.spec.ts (3.7m)
  Slow test file: [chromium] › 05_assets.cryptoaccount.spec.ts (2.2m)
  Slow test file: [chromium] › 02_assets.privateinvestment.spec.ts (2.0m)
  Slow test file: [chromium] › 04_managedentities.spec.ts (1.4m)
  Consider splitting slow test files to speed up parallel execution
  1 failed
    [chromium] › 04_managedentities.spec.ts:894:9 › Managed Entities › Create & manage new Legal Entity › ME15 - Deleting a business entity with ownership stake in another business entity @notvisual 
  13 skipped
  144 passed (25.0m)
  2 errors were not a part of any test, see above for details