test-runner: [Bug]ReferenceError: __test is not defined

I seem to get this error seemingly at random when running the tests. Following the stack trace just takes me to the very bottom of my story file.

 FAIL   browser: chromium  src/Tag.stories.tsx (26.779 s)
  ● Base Components/Tag › Variants › smoke-test

    page.evaluate: ReferenceError: __test is not defined

      at eval (eval at evaluate (:3:2389), <anonymous>:4:17)
      at t.default.evaluate (<anonymous>:3:2412)
      at t.default.<anonymous> (<anonymous>:1:44)
      at testFn (src/Tag.stories.tsx:84:37)
      at Object.<anonymous> (src/Tag.stories.tsx:99:17)

About this issue

Original URL
State: open
Created 2 years ago
Reactions: 10
Comments: 27 (10 by maintainers)

Commits related to this issue

fix: Flaky "__test is not defined" issue fixes #68 — committed to storybookjs/test-runner by deleted user 2 years ago
fix: Flaky "__test is not defined" issue fixes #68 — committed to storybookjs/test-runner by deleted user 2 years ago

Most upvoted comments

It seems the race condition is resolved by running that initialization script not with evaluate, but with addInitScript.

diff --git a/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.js b/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.js
index f6b3bdf..328350c 100644
--- a/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.js
+++ b/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.js
@@ -57,7 +57,7 @@ const testPrefixer = (0, import_template.default)(`
           await globalThis.__sbPreRender(page, context);
         }

-        const result = await page.evaluate(({ id, hasPlayFn }) => __test(id, hasPlayFn), {
+        const result = await page.addInitScript(({ id, hasPlayFn }) => __test(id, hasPlayFn), {
           id: %%id%%,
         });

diff --git a/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.mjs b/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.mjs
index 08d0006..dfaec94 100644
--- a/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.mjs
+++ b/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.mjs
@@ -25,7 +25,7 @@ const testPrefixer = template(`
           await globalThis.__sbPreRender(page, context);
         }

-        const result = await page.evaluate(({ id, hasPlayFn }) => __test(id, hasPlayFn), {
+        const result = await page.addInitScript(({ id, hasPlayFn }) => __test(id, hasPlayFn), {
           id: %%id%%,
         });

throw5way on Mar 7, 2024

Hey there @throw5way thanks for the suggestion! I made a canary with the fix at #445, please try it out and give some feedback if you can. Thanks!

yarn add @storybook/test-runner@0.17.1--canary.445.f7d2785.0

yannbf on Mar 26, 2024

Looking at the source code, this smells like a race condition to me.

The __test invocation is defined here:

https://github.com/storybookjs/test-runner/blob/6624c5e6b7958e05f7c9dbeb57b581e374202c2b/src/playwright/transformPlaywright.ts#L28-L30

And the __test definition is in setup-page-script.ts: https://github.com/storybookjs/test-runner/blob/6624c5e6b7958e05f7c9dbeb57b581e374202c2b/src/setup-page-script.ts#L266

Which in turn is loaded in setupPage: https://github.com/storybookjs/test-runner/blob/6624c5e6b7958e05f7c9dbeb57b581e374202c2b/src/setup-page.ts#L64-L75

Could the explanation be that there is a race between the invocation and loading of the setup page?

kasper573 on Dec 10, 2023

For me, this is caused when I navigate to a different URL trying to capture snapshots during postRender(). Probably the test runner injects something into the page that is lost by navigating to another URL.

(The use case for navigating for me, is to put different arg values in the URL and as such capture different permutations of the Story).

joriswitteman on Jul 10, 2023

These errors were definitely mixed with other ones, but I’m not sure in which order they came.

I’ll see if I have time to get some more info about this when back at work on Monday.

VictorGlindasPaf on Feb 25, 2022