playwright-bdd: Random test failures when running >40 tests in parallel

I have a test suite with about 45 Scenarios, split across about 30 feature files. When I run them individually, all is good. When I run less than 40, fully parallel, all is good When I run more than 40, fully parallel, I experience random failures, always from the tests executed towards the end If I shard them and run them in batches of 20 or 30, no failure happens

This is the error being thrown:

    TypeError: Cannot read properties of null (reading 'includes')

        at World.invokeStep (/Users/myName/workspace/myProject/node_modules/playwright-bdd/src/run/world.ts:53:41)
        at /Users/myName/workspace/myProject/starter-kit/async /Users/myName/workspace/myProject/starter-kit/tests/pwFeature/.feature-gen/tests/pwFeature/textInput.feature.spec.js:19:5
        at /Users/myName/workspace/myProject/starter-kit/async /Users/myName/workspace/myProject/node_modules/@playwright/test/lib/worker/workerMain.js:351:9
        at /Users/myName/workspace/myProject/starter-kit/async /Users/myName/workspace/myProject/node_modules/@playwright/test/lib/worker/workerMain.js:347:7

My configuration: OS: Mac OSx Ventura i7 32GB RAM playwright-bdd: 3.2.1 @playwright/test: 1.33.0 cucumber: 9.1.0 chrome: 114 nodejs: 16.16.0 ts-node: 10.9.1

To write my step definitions, I am using cucumber style:

example:

Given('I am on the {string} page', async function(pageName: string) {
  const currentPage: PageInterface = globalThis[`${pageName}Page`];
  return currentPage.open();
});

To run my tests, I first run bddgen (there is nothing wrong with the generated spec files), then I use npx playwright test --config path/to/my/config

This is what my feature files look like:

Feature: Switch

  Background:
    Given create "switch" page

  Scenario: Switch displayed correctly
    Given I am on the "switch" page
    Then Switch is displayed correctly
    And Switch label is displayed correctly

They all include a background and are written in exactly the same way

About this issue

Original URL
State: closed
Created a year ago
Reactions: 1
Comments: 22 (7 by maintainers)

Commits related to this issue

fix getting location from stacktrace, see #31 — committed to vitalets/playwright-bdd by vitalets a year ago

Most upvoted comments

I’m also seeing these. Haven’t managed to create a reproduction to post here yet

garrlker on Jul 6, 2023

Thank you @paolodg

vitalets on Jul 14, 2023

Fixed on both versions: node 16.16.0 node 18.16.1

paolodg on Jul 14, 2023

Fixed in v3.3.0. Feel free to reopen if there are any issues.

@paolodg if possible, could you re-check on both node 16 and node 18 with all your test files?

vitalets on Jul 11, 2023

hi @paolodg thanks for the feedback! I’ve also reproduced the issue on node v16. Going deeper I found that the reason is in stacktrace capturing that is used for detecting step call locations.

Normally step call stacktrace is like following:

/playwright-bdd-example/node_modules/playwright-bdd/dist/playwright/getLocationByStacktrace.js 36
/playwright-bdd-example/node_modules/playwright-bdd/dist/run/world.js 27
/playwright-bdd-example/.features-gen/features/sample-27.feature.spec.js 17     <-- this frame (with depthIndex = 2) is used for step call reporting
/playwright-bdd-example/node_modules/@playwright/test/lib/worker/workerMain.js 345
...

But when running large number of files on Node v16 stacktrace sometimes gets microtasks injected and fileName = null:

/playwright-bdd-example/node_modules/playwright-bdd/dist/playwright/getLocationByStacktrace.js 36
/playwright-bdd-example/node_modules/playwright-bdd/dist/run/world.js 27
null null        <-- this actually causes "Cannot read properties of null (reading 'includes')"
node:internal/process/task_queues 61
node:internal/timers 437
/playwright-bdd-example/.features-gen/features/sample-27.feature.spec.js 17
/playwright-bdd-example/node_modules/@playwright/test/lib/worker/workerMain.js 345
...

And on Node v18 microtask calls also appears (but no null filenames):

/playwright-bdd-example/node_modules/playwright-bdd/dist/playwright/getLocationByStacktrace.js 36
/playwright-bdd-example/node_modules/playwright-bdd/dist/run/world.js 27
node:internal/process/task_queues 61      <-- this frame will be incorrectly used for step call reporting
node:internal/timers 437
/playwright-bdd-example/.features-gen/features/sample-27.feature.spec.js 17
/playwright-bdd-example/node_modules/@playwright/test/lib/worker/workerMain.js 345
...

That means that even though it’s passing on Node v18, step locations in report can be incorrect.

What I’m going to do:

emulate the same conditions on raw Playwright tests to check how it deals with it
fix stacktrace handling so it will filter out nulls and node: internal calls to find appropriate userland frame.

Will post here the updates.

vitalets on Jul 11, 2023

Could you please let us know what node version you are using?

I was testing on node v20.3.1

try to downgrade to node 16.18.X and run tests again to see if we can replicate

Will do and share the results.

Hopefully we are at the end of tunnel! 😃

Yes ))

vitalets on Jul 10, 2023

sure, I will try, but I am not passing the World object to my steps, so this setup looks a bit different from the one I have First I’ll try with your config, then I’ll try to amend the one I currently have to use the world

paolodg on Jul 7, 2023