test-runner: [Bug]ReferenceError: __test is not defined

I seem to get this error seemingly at random when running the tests. Following the stack trace just takes me to the very bottom of my story file.

 FAIL   browser: chromium  src/Tag.stories.tsx (26.779 s)
  ● Base Components/Tag › Variants › smoke-test

    page.evaluate: ReferenceError: __test is not defined

      at eval (eval at evaluate (:3:2389), <anonymous>:4:17)
      at t.default.evaluate (<anonymous>:3:2412)
      at t.default.<anonymous> (<anonymous>:1:44)
      at testFn (src/Tag.stories.tsx:84:37)
      at Object.<anonymous> (src/Tag.stories.tsx:99:17)

About this issue

  • Original URL
  • State: open
  • Created 5 years ago
  • Reactions: 26
  • Comments: 63 (19 by maintainers)

Commits related to this issue

Most upvoted comments

EDIT: see this update

I have done some more research into this, and here’s what I’ve found:

The problem seems to lie in the fact that a URL change is used to load a story into the page for testing. Specifically, what happens is this:

I believe that the last part is the navigation that causes “An error occurred in the following story, most likely because of a navigation.”

How do I know that this navigation is causing a problem? Well, if you set the environment variable DEBUG="pw:api" in your terminal before running a Storybook test, Playwright logs a bunch of stuff that’s happening, and you can see the navigation happen right before the problem does:

  pw:api => page.evaluate started +7ms
  pw:api   navigated to "http://127.0.0.1:6006/iframe.html?id=pages-storefront-singleproductpage--one-product-one-variant&viewMode=%7B%7BviewMode%7D%7D" +3ms                                                                                                                         
  pw:api   "commit" event fired +5ms
  pw:api   navigated to "http://127.0.0.1:6006/iframe.html?id=pages-storefront-singleproductpage--one-product-one-variant&viewMode=%7B%7BviewMode%7D%7D" +1ms                                                                                                                         
  pw:api <= page.evaluate failed +1ms
  console.log                                                                                                                              
    An error occurred in the following story, most likely because of a navigation: "pages/Storefront/SingleProductPage/One Product One Variant". Retrying...

      at Object.<anonymous> (tc-components/components/storefront/SingleProductPage/stories/SingleProductPage.stories.tsx:830:37)

In fact, as you can see, the navigation actually happens twice. However, it’s not consistent. Sometimes the navigation destroys the execution context and causes a problem, as above. Sometimes it doesn’t:

  pw:api => page.evaluate started +0ms
  pw:api   navigated to "http://127.0.0.1:6006/iframe.html?id=pages-storefront-singleproductpage--one-product-one-variant&viewMode=%7B%7BviewMode%7D%7D" +10ms                                                                                                                        
  pw:api <= page.evaluate succeeded +233ms

As far as I can tell, every time a test fails, it looks like the first example: there’s a navigation logged, then a “commit” event logged, then the navigation logged again, and then page.evaluate fails. On the other hand, things sometimes instead look like the second example, where there’s only one “navigated to” that is logged, and no “commit” event, and page.evaluate succeeds.

I don’t know why the browser sometimes navigates to the specific story once, and sometimes navigates to it twice and crashes the test. It seems to treat it differently based on how fast or slow the test runs; if I slow the test way down with Playwright’s debug mode (PWDEBUG=1), the issue never happens. But, in theory, it does seem a bit weird to run history.replaceState during a Playwright test at all, as the Storybook code does, since Playwright provides page.goto for navigation. I don’t know what the solution is there.

Any update?

Hi all, I am just starting to set up or component library project with the test-runner with Storybook v8.6.4, @storybook/test-runner v0.22.0, @storybook/react-vite v8.6.4 I am still running into this issue with the __test not being defined randomly. It happens inconsistently on firefox, on almost every test on webkit (along with some Execution context was destroyed, most likely because of a navigation errors), and on many tests in chromium (along with some page.evaluate: ReferenceError: Cannot access 'StorybookTestRunnerError' before initialization). Firefox is the only browser that I can get any test to pass at all. All but one of our tests, right now, are basic smoke tests. The one play function that we are testing (as we get things set up) works perfectly when running in Storybook, but is inconsistent (with the above errors) in test-runner. We are also getting a bunch of TypeError: error loading dynamically imported module: on all browsers, again randomly.

We were using Playwright with Storybook for component testing and it was working fine until about three weeks ago when we started having these flaky tests breaking, both on Github CI and when running locally. We have no navigation happening in any of the stories and no purposefully dynamically importing components that I can find.

I am very new to this sort of testing and build/compile configuration stuff and find myself a bit out of my depth. I’m picking this up from a more senior engineer who left our company not long ago. I’m not sure how related my issue is to this thread, but I thought it’s a good place to start. I’ve found nothing helpful in many hours on various docs sites, stack overflow threads, and repo issues threads. Any advice or help is welcome. Thank you.

I can confirm that the attempted patch causes all tests to pass, even those that should fail. From a quick reading of the code, that canary version tries to make the test function run earlier, which is the opposite of what you would want; you want the setup function to run earlier, and/or the actual test to run later.

It looks to me like if you have too many stories, you will inevitably start seeing tests fail with “An error occurred in the following story, most likely because of a navigation.” It looks like a lot of people who have been reporting this problem have large repos with a lot of stories. At work, where we have a lot of stories, we have been hitting this error (and some random timeouts, and occasionally the __test is not defined error) intermittently. It appears that you can get these problems just by having Too Many Stories.

I have a repo here that reliably reproduces these errors for me; it has a simple test page and 400 copies of that test page’s story. In this repo, I reliably get “An error occurred in the following story, most likely because of a navigation” when running tests. (There is definitely no navigation taking place in the stories.)

This is kind of a contrived example, but like I said, I (and presumably other people in this thread) have been hitting the exact same problems in real life. In general, I don’t see any real reason why there should be a limit for the number of stories the test runner should be able to support; it should be possible to run each of the tests more or less in isolation, so if you can run each test, you should be able to run them all (it’s not like there’s any shared state between them.)

I notice that the memory and threads used by the test runner reliably increase as the test runner progresses in a large project. It seems likely to me that the underlying cause of the issue is some kind of memory leak and resource starvation that’s taking place; probably the execution context for the tests is being destroyed because the underlying browser is killing tabs that take too much memory, or something along those lines.

The latest test runner 0.21.3 contains the fix!

Same issue here on @storybook/test-runner@0.19.1, using Storybook 8 & vite. We’re having an issue when running the tests in CI, the first one tends to fail due to timeout but the error in the console is the same:

console.log
  An error occurred in the following story, most likely because of a navigation: […]

For us, it only happens in the first test that runs. It does not have any navigation in the test itself.

I tried applying the patch directly from https://github.com/storybookjs/test-runner/issues/68#issuecomment-2224900181 or installing the canary version. Both result in the pipeline passing but I tried breaking the tests on purpose and they continue passing (as false positives).

I encountered this issue at 0.18.2 and got a workaround with this patch-package file. This is same as #445.

patches/@storybook+test-runner+0.18.2.patch

diff --git a/node_modules/@storybook/test-runner/dist/index.js b/node_modules/@storybook/test-runner/dist/index.js
index e1bc5a2..1aba18e 100644
--- a/node_modules/@storybook/test-runner/dist/index.js
+++ b/node_modules/@storybook/test-runner/dist/index.js
@@ -13556,7 +13556,7 @@ var testPrefixer = /* @__PURE__ */ __name((context) => {
           await globalThis.__sbPreVisit(page, context);
         }
 
-        const result = await page.evaluate(({ id, hasPlayFn }) => __test(id, hasPlayFn), {
+        const result = await page.addInitScript(({ id, hasPlayFn }) => __test(id, hasPlayFn), {
           id: %%id%%,
         });
 
diff --git a/node_modules/@storybook/test-runner/dist/index.mjs b/node_modules/@storybook/test-runner/dist/index.mjs
index c32e886..b8ca561 100644
--- a/node_modules/@storybook/test-runner/dist/index.mjs
+++ b/node_modules/@storybook/test-runner/dist/index.mjs
@@ -13539,7 +13539,7 @@ var testPrefixer = /* @__PURE__ */ __name((context) => {
           await globalThis.__sbPreVisit(page, context);
         }
 
-        const result = await page.evaluate(({ id, hasPlayFn }) => __test(id, hasPlayFn), {
+        const result = await page.addInitScript(({ id, hasPlayFn }) => __test(id, hasPlayFn), {
           id: %%id%%,
         });
 
diff --git a/node_modules/@storybook/test-runner/dist/test-storybook.js b/node_modules/@storybook/test-runner/dist/test-storybook.js
index 32bbbd8..c59380a 100755
--- a/node_modules/@storybook/test-runner/dist/test-storybook.js
+++ b/node_modules/@storybook/test-runner/dist/test-storybook.js
@@ -17621,7 +17621,7 @@ var testPrefixer = /* @__PURE__ */ __name((context) => {
           await globalThis.__sbPreVisit(page, context);
         }
 
-        const result = await page.evaluate(({ id, hasPlayFn }) => __test(id, hasPlayFn), {
+        const result = await page.addInitScript(({ id, hasPlayFn }) => __test(id, hasPlayFn), {
           id: %%id%%,
         });
 
diff --git a/node_modules/@storybook/test-runner/dist/test-storybook.mjs b/node_modules/@storybook/test-runner/dist/test-storybook.mjs
index da7fd85..0ac585b 100755
--- a/node_modules/@storybook/test-runner/dist/test-storybook.mjs
+++ b/node_modules/@storybook/test-runner/dist/test-storybook.mjs
@@ -17627,7 +17627,7 @@ var testPrefixer = /* @__PURE__ */ __name((context) => {
           await globalThis.__sbPreVisit(page, context);
         }
 
-        const result = await page.evaluate(({ id, hasPlayFn }) => __test(id, hasPlayFn), {
+        const result = await page.addInitScript(({ id, hasPlayFn }) => __test(id, hasPlayFn), {
           id: %%id%%,
         });

It seems the race condition is resolved by running that initialization script not with evaluate, but with addInitScript.

diff --git a/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.js b/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.js
index f6b3bdf..328350c 100644
--- a/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.js
+++ b/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.js
@@ -57,7 +57,7 @@ const testPrefixer = (0, import_template.default)(`
           await globalThis.__sbPreRender(page, context);
         }

-        const result = await page.evaluate(({ id, hasPlayFn }) => __test(id, hasPlayFn), {
+        const result = await page.addInitScript(({ id, hasPlayFn }) => __test(id, hasPlayFn), {
           id: %%id%%,
         });

diff --git a/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.mjs b/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.mjs
index 08d0006..dfaec94 100644
--- a/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.mjs
+++ b/node_modules/@storybook/test-runner/dist/playwright/transformPlaywright.mjs
@@ -25,7 +25,7 @@ const testPrefixer = template(`
           await globalThis.__sbPreRender(page, context);
         }

-        const result = await page.evaluate(({ id, hasPlayFn }) => __test(id, hasPlayFn), {
+        const result = await page.addInitScript(({ id, hasPlayFn }) => __test(id, hasPlayFn), {
           id: %%id%%,
         });

With Storybook 8.5.0 version upgrade, I am not starting to get An error occurred in the following story, most likely because of a navigation issue. Is there an incoming fix?

@vanessayuenn @yannbf seems like a good thing to fix if we can squeeze it in. WDYT?

for us, after upgrading the the canary version, all test pass even though they should be failing.

also we are trying todo some individual retries to deflake test runs.

We start to see this error after we add the setupFilesAfterEnv config in jest

// .storybook/test-runner-jest.config.js
import {getJestConfig} from '@storybook/test-runner';

// The default Jest configuration comes from @storybook/test-runner
const testRunnerConfig = getJestConfig();

// eslint-disable-next-line import/no-default-export
export default {
  ...testRunnerConfig,
  /** Add your own overrides below, and make sure
   *  to merge testRunnerConfig properties with your own
   * @see https://jestjs.io/docs/configuration
   */
  setupFilesAfterEnv: [`<rootDir>/.storybook/test-global-setup.ts`],
};

// global setup.ts
import {beforeEach, jest} from '@jest/globals';

beforeEach(() => {
  jest.retryTimes(3, {logErrorsBeforeRetry: true});
});

Hey there @throw5way thanks for the suggestion! I made a canary with the fix at #445, please try it out and give some feedback if you can. Thanks!

yarn add @storybook/test-runner@0.17.1--canary.445.f7d2785.0

The latest test runner 0.21.3 contains the fix!

As usual you hit the nail @valentinpalkovic ! Thank you very much for this fix 😺

@nick-lanners you might consider the new @storybook/experimental-addon-test since you’re using Vite. https://storybook.js.org/docs/writing-tests/test-addon

Your guess is correct! We correctly await afterEach calls before processing the next test.

After many sleepless nights and immolated brain cells, I have found the problem, at least as it occurs in our setup. There is a call to window.location.reload() in the teardown method of the StoryRender class here. This reloads the window if the story is in one of these phases when the current story is being torn down and the next test needs to run. When debugging this issue, the problem seemed to be that the story was in the “afterEach” phase for too long. This reload is what destroys the Playwright execution context, and is the “navigation” mentioned in the error message.

This is the patch that fixes the problem, which can be applied with patch-package. It waits indefinitely for the story to finish tearing down, producing a console error if it takes too long. The teardown will only be delayed for that long if there is some kind of callback returned from the beforeEach of a story’s meta or some experimental_afterEach added by the user or an add-on that takes a really long time.

diff --git a/node_modules/@storybook/core/dist/preview/runtime.js b/node_modules/@storybook/core/dist/preview/runtime.js
index 0b1ef63..0662f03 100644
--- a/node_modules/@storybook/core/dist/preview/runtime.js
+++ b/node_modules/@storybook/core/dist/preview/runtime.js
@@ -6105,13 +6105,19 @@ var Sn = class Sn {
     this.abortController?.abort();
   }
   async teardown() {
     this.torndown = !0, this.cancelRender(), this.story && await this.store.cleanupStory(this.story);
-    for (let e = 0; e < 3; e += 1) {
+    let index = 0;
+    while (true) {
       if (!this.isPending()) {
         await this.teardownRender();
         return;
       }
-      await new Promise((t) => setTimeout(t, 0));
+      ++index;
+      if (index > 5) {
+        console.error("storybook teardown taking a really long time!");
+      }
+      await new Promise((t) => setTimeout(t, 10));
     }
     window.location.reload(), await new Promise(() => {
     });

I solved the issue on my end. The reasons were bad decisions made by previous developers. They did reloads to get rid of cache. I made proper redux query invalidators in decorators i preview.tsx instead, so reloads werent neccessay. My take away is, if youre getting this issue, you’re likely doing something in a way it shouldnt be done. 😃

FYI just hit this issue. Its been tricky to repro, because it only happens for us when using Vite and the dependency optimization step has not run. Only in Storybook 8 in our case (even though it looks like this issue predates it by a fair bit). Eg, setting the vite config to have:

export default defineConfig({
	plugins: [react()],
	optimizeDeps: {
		force: true,
	},
});

Otherwise, first run fails, second run pass. But using a production build of storybook to test against bypasses this optimize dep step, so the tests always work. Not a great solution when running locally, though.

I added code in a preVisit step to wait until the setup-page-script functions are available on the global object. In the event of dep optimization, they NEVER become available (but on second run they are). It’s really strange.

I notice this still occurs on 0.18.1. Is the canary change included in 0.18?

And I’m running visual snapshot testing.

Hey @yannbf The canary build solves our issues as reproduced by https://github.com/storybookjs/test-runner/issues/68#issuecomment-1837872661 all the tests succeed after upgrading from 0.17.0. I’m awaiting the final release before upgrading our pileline though.

@yannbf that fixes my issue! I have been scratching my head over this for a week.

Looking at the source code, this smells like a race condition to me.

The __test invocation is defined here:

https://github.com/storybookjs/test-runner/blob/6624c5e6b7958e05f7c9dbeb57b581e374202c2b/src/playwright/transformPlaywright.ts#L28-L30

And the __test definition is in setup-page-script.ts: https://github.com/storybookjs/test-runner/blob/6624c5e6b7958e05f7c9dbeb57b581e374202c2b/src/setup-page-script.ts#L266

Which in turn is loaded in setupPage: https://github.com/storybookjs/test-runner/blob/6624c5e6b7958e05f7c9dbeb57b581e374202c2b/src/setup-page.ts#L64-L75

Could the explanation be that there is a race between the invocation and loading of the setup page?

getting same error, following the docs, it says zero config, but do we need extra config?

For me, this is caused when I navigate to a different URL trying to capture snapshots during postRender(). Probably the test runner injects something into the page that is lost by navigating to another URL.

(The use case for navigating for me, is to put different arg values in the URL and as such capture different permutations of the Story).

I’m having the same problem. The error seems to occur at random. I’m using the test-runner to take snapshots of stories of web components. Because the web components are not hydrated at the time the postRender hook runs, I added a waitForFunction to wait for my components to have the .hydrated class.

This is my test-runner.js file:

module.exports = {
  async postRender(page, context) {

    await page.waitForFunction(
      () => {
        const inoElements = Array.from(
          document.getElementsByTagName('*')
        ).filter((el) => el.tagName.startsWith('INO'));

        return inoElements.every((el) => el.classList.contains('hydrated'));
      },
      { polling: 50 }
    );

    await page.$('#root');

    const elementHandler = await page.$('#root');
    const innerHTML = await elementHandler.innerHTML();
    expect(innerHTML).toMatchSnapshot();
  },
};

The introduction of this snippet led to this error. Any ideas why this is happening?

@VictorGlindasPaf I had the same issue. In my case, the issue was a redirection happening in the components by using a decorator, redirecting to an SSO page when a Token was not available in the localstorage. Maybe you have a similar issue and a redirection happens in special cases or you are triggering a page reload. This would explain, why the initial script, which contains the __test function, is not available in the head section anymore.

These errors were definitely mixed with other ones, but I’m not sure in which order they came.

I’ll see if I have time to get some more info about this when back at work on Monday.