test-runner: [Bug]ReferenceError: __test is not defined
I seem to get this error seemingly at random when running the tests. Following the stack trace just takes me to the very bottom of my story file.
FAIL browser: chromium src/Tag.stories.tsx (26.779 s)
● Base Components/Tag › Variants › smoke-test
page.evaluate: ReferenceError: __test is not defined
at eval (eval at evaluate (:3:2389), <anonymous>:4:17)
at t.default.evaluate (<anonymous>:3:2412)
at t.default.<anonymous> (<anonymous>:1:44)
at testFn (src/Tag.stories.tsx:84:37)
at Object.<anonymous> (src/Tag.stories.tsx:99:17)
About this issue
- Original URL
- State: open
- Created 5 years ago
- Reactions: 26
- Comments: 63 (19 by maintainers)
Commits related to this issue
- fix: Flaky "__test is not defined" issue fixes #68 — committed to storybookjs/test-runner by deleted user 4 years ago
- fix: Flaky "__test is not defined" issue fixes #68 — committed to storybookjs/test-runner by deleted user 4 years ago
EDIT: see this update
I have done some more research into this, and here’s what I’ve found:
The problem seems to lie in the fact that a URL change is used to load a story into the page for testing. Specifically, what happens is this:
__testfunction in this repo emits asetCurrentStoryevent.onSetCurrentStoryevent listener in the main Storybook repo fires.setSelectionmethod that is implemented by theUrlStoreclass, and that calls itssetPathmethod.setPath,history.replaceStateis called to give the current URL theidquery parameter that specifies which story is being run.I believe that the last part is the navigation that causes “An error occurred in the following story, most likely because of a navigation.”
How do I know that this navigation is causing a problem? Well, if you set the environment variable
DEBUG="pw:api"in your terminal before running a Storybook test, Playwright logs a bunch of stuff that’s happening, and you can see the navigation happen right before the problem does:In fact, as you can see, the navigation actually happens twice. However, it’s not consistent. Sometimes the navigation destroys the execution context and causes a problem, as above. Sometimes it doesn’t:
As far as I can tell, every time a test fails, it looks like the first example: there’s a navigation logged, then a “commit” event logged, then the navigation logged again, and then page.evaluate fails. On the other hand, things sometimes instead look like the second example, where there’s only one “navigated to” that is logged, and no “commit” event, and page.evaluate succeeds.
I don’t know why the browser sometimes navigates to the specific story once, and sometimes navigates to it twice and crashes the test. It seems to treat it differently based on how fast or slow the test runs; if I slow the test way down with Playwright’s debug mode (
PWDEBUG=1), the issue never happens. But, in theory, it does seem a bit weird to runhistory.replaceStateduring a Playwright test at all, as the Storybook code does, since Playwright providespage.gotofor navigation. I don’t know what the solution is there.Any update?
Hi all, I am just starting to set up or component library project with the test-runner with Storybook v8.6.4, @storybook/test-runner v0.22.0, @storybook/react-vite v8.6.4 I am still running into this issue with the __test not being defined randomly. It happens inconsistently on firefox, on almost every test on webkit (along with some
Execution context was destroyed, most likely because of a navigationerrors), and on many tests in chromium (along with somepage.evaluate: ReferenceError: Cannot access 'StorybookTestRunnerError' before initialization). Firefox is the only browser that I can get any test to pass at all. All but one of our tests, right now, are basic smoke tests. The oneplayfunction that we are testing (as we get things set up) works perfectly when running in Storybook, but is inconsistent (with the above errors) in test-runner. We are also getting a bunch ofTypeError: error loading dynamically imported module:on all browsers, again randomly.We were using Playwright with Storybook for component testing and it was working fine until about three weeks ago when we started having these flaky tests breaking, both on Github CI and when running locally. We have no navigation happening in any of the stories and no purposefully dynamically importing components that I can find.
I am very new to this sort of testing and build/compile configuration stuff and find myself a bit out of my depth. I’m picking this up from a more senior engineer who left our company not long ago. I’m not sure how related my issue is to this thread, but I thought it’s a good place to start. I’ve found nothing helpful in many hours on various docs sites, stack overflow threads, and repo issues threads. Any advice or help is welcome. Thank you.
I can confirm that the attempted patch causes all tests to pass, even those that should fail. From a quick reading of the code, that canary version tries to make the test function run earlier, which is the opposite of what you would want; you want the setup function to run earlier, and/or the actual test to run later.
It looks to me like if you have too many stories, you will inevitably start seeing tests fail with “An error occurred in the following story, most likely because of a navigation.” It looks like a lot of people who have been reporting this problem have large repos with a lot of stories. At work, where we have a lot of stories, we have been hitting this error (and some random timeouts, and occasionally the
__test is not definederror) intermittently. It appears that you can get these problems just by having Too Many Stories.I have a repo here that reliably reproduces these errors for me; it has a simple test page and 400 copies of that test page’s story. In this repo, I reliably get “An error occurred in the following story, most likely because of a navigation” when running tests. (There is definitely no navigation taking place in the stories.)
This is kind of a contrived example, but like I said, I (and presumably other people in this thread) have been hitting the exact same problems in real life. In general, I don’t see any real reason why there should be a limit for the number of stories the test runner should be able to support; it should be possible to run each of the tests more or less in isolation, so if you can run each test, you should be able to run them all (it’s not like there’s any shared state between them.)
I notice that the memory and threads used by the test runner reliably increase as the test runner progresses in a large project. It seems likely to me that the underlying cause of the issue is some kind of memory leak and resource starvation that’s taking place; probably the execution context for the tests is being destroyed because the underlying browser is killing tabs that take too much memory, or something along those lines.
The latest test runner 0.21.3 contains the fix!
Same issue here on
@storybook/test-runner@0.19.1, using Storybook 8 & vite. We’re having an issue when running the tests in CI, the first one tends to fail due to timeout but the error in the console is the same:For us, it only happens in the first test that runs. It does not have any navigation in the test itself.
I tried applying the patch directly from https://github.com/storybookjs/test-runner/issues/68#issuecomment-2224900181 or installing the canary version. Both result in the pipeline passing but I tried breaking the tests on purpose and they continue passing (as false positives).
I encountered this issue at 0.18.2 and got a workaround with this patch-package file. This is same as #445.
patches/@storybook+test-runner+0.18.2.patchIt seems the race condition is resolved by running that initialization script not with
evaluate, but withaddInitScript.With Storybook 8.5.0 version upgrade, I am not starting to get
An error occurred in the following story, most likely because of a navigationissue. Is there an incoming fix?@vanessayuenn @yannbf seems like a good thing to fix if we can squeeze it in. WDYT?
for us, after upgrading the the canary version, all test pass even though they should be failing.
also we are trying todo some individual retries to deflake test runs.
We start to see this error after we add the
setupFilesAfterEnvconfig in jestHey there @throw5way thanks for the suggestion! I made a canary with the fix at #445, please try it out and give some feedback if you can. Thanks!
As usual you hit the nail @valentinpalkovic ! Thank you very much for this fix 😺
@nick-lanners you might consider the new
@storybook/experimental-addon-testsince you’re using Vite. https://storybook.js.org/docs/writing-tests/test-addonYour guess is correct! We correctly await afterEach calls before processing the next test.
After many sleepless nights and immolated brain cells, I have found the problem, at least as it occurs in our setup. There is a call to
window.location.reload()in theteardownmethod of theStoryRenderclass here. This reloads the window if the story is in one of these phases when the current story is being torn down and the next test needs to run. When debugging this issue, the problem seemed to be that the story was in the “afterEach” phase for too long. This reload is what destroys the Playwright execution context, and is the “navigation” mentioned in the error message.This is the patch that fixes the problem, which can be applied with
patch-package. It waits indefinitely for the story to finish tearing down, producing a console error if it takes too long. The teardown will only be delayed for that long if there is some kind of callback returned from thebeforeEachof a story’s meta or someexperimental_afterEachadded by the user or an add-on that takes a really long time.I solved the issue on my end. The reasons were bad decisions made by previous developers. They did reloads to get rid of cache. I made proper redux query invalidators in decorators i preview.tsx instead, so reloads werent neccessay. My take away is, if youre getting this issue, you’re likely doing something in a way it shouldnt be done. 😃
FYI just hit this issue. Its been tricky to repro, because it only happens for us when using Vite and the dependency optimization step has not run. Only in Storybook 8 in our case (even though it looks like this issue predates it by a fair bit). Eg, setting the vite config to have:
Otherwise, first run fails, second run pass. But using a production build of storybook to test against bypasses this optimize dep step, so the tests always work. Not a great solution when running locally, though.
I added code in a preVisit step to wait until the setup-page-script functions are available on the global object. In the event of dep optimization, they NEVER become available (but on second run they are). It’s really strange.
I notice this still occurs on 0.18.1. Is the canary change included in 0.18?
And I’m running visual snapshot testing.
Hey @yannbf The canary build solves our issues as reproduced by https://github.com/storybookjs/test-runner/issues/68#issuecomment-1837872661 all the tests succeed after upgrading from 0.17.0. I’m awaiting the final release before upgrading our pileline though.
@yannbf that fixes my issue! I have been scratching my head over this for a week.
Looking at the source code, this smells like a race condition to me.
The
__testinvocation is defined here:https://github.com/storybookjs/test-runner/blob/6624c5e6b7958e05f7c9dbeb57b581e374202c2b/src/playwright/transformPlaywright.ts#L28-L30
And the
__testdefinition is insetup-page-script.ts: https://github.com/storybookjs/test-runner/blob/6624c5e6b7958e05f7c9dbeb57b581e374202c2b/src/setup-page-script.ts#L266Which in turn is loaded in
setupPage: https://github.com/storybookjs/test-runner/blob/6624c5e6b7958e05f7c9dbeb57b581e374202c2b/src/setup-page.ts#L64-L75Could the explanation be that there is a race between the invocation and loading of the setup page?
getting same error, following the docs, it says zero config, but do we need extra config?
For me, this is caused when I navigate to a different URL trying to capture snapshots during
postRender(). Probably the test runner injects something into the page that is lost by navigating to another URL.(The use case for navigating for me, is to put different arg values in the URL and as such capture different permutations of the Story).
I’m having the same problem. The error seems to occur at random. I’m using the test-runner to take snapshots of stories of web components. Because the web components are not hydrated at the time the
postRenderhook runs, I added awaitForFunctionto wait for my components to have the.hydratedclass.This is my
test-runner.jsfile:The introduction of this snippet led to this error. Any ideas why this is happening?
@VictorGlindasPaf I had the same issue. In my case, the issue was a redirection happening in the components by using a decorator, redirecting to an SSO page when a Token was not available in the localstorage. Maybe you have a similar issue and a redirection happens in special cases or you are triggering a page reload. This would explain, why the initial script, which contains the
__testfunction, is not available in the head section anymore.These errors were definitely mixed with other ones, but I’m not sure in which order they came.
I’ll see if I have time to get some more info about this when back at work on Monday.