test-runner: [bug] preVisit accesses the story store before the index is ready
Describe the bug
We have the test-runner.js implement with the preVisit to apply the different viewports when running tests, per the README.
After upgrading to Storybook 8, our tests fail in a flaky way, e.g.:
FAIL src/components/PostCompletionViews/PostCompletionView.stories.js
● Private API / Post completion views › Generic › smoke-test
page.evaluate: SB_PREVIEW_API_0011 (StoryStoreAccessedBeforeInitializationError): Cannot access the Story Store until the index is ready.
It is not recommended to use methods directly on the Story Store anyway, in Storybook 9 we will
remove access to the store entirely
at Object.get (http:/127.0.0.1:6006/sb-preview/runtime.js:118:5542)
at __getContext (<anonymous>:215:54)
at eval (eval at evaluate (:226:30), <anonymous>:1:29)
at UtilityScript.evaluate (<anonymous>:228:17)
at UtilityScript.<anonymous> (<anonymous>:1:44)
at getStoryContext (node_modules/@storybook/test-runner/dist/index.js:15306:15)
at getStoryContext (.storybook/test-runner.js:9:27)
at src/components/PostCompletionViews/PostCompletionView.stories.js:151:64
at step (src/components/PostCompletionViews/PostCompletionView.stories.js:109:23)
at Object.next (src/components/PostCompletionViews/PostCompletionView.stories.js:50:20)
at asyncGeneratorStep (src/components/PostCompletionViews/PostCompletionView.stories.js:4:28)
at _next (src/components/PostCompletionViews/PostCompletionView.stories.js:22:17)
at src/components/PostCompletionViews/PostCompletionView.stories.js:27:13
at src/components/PostCompletionViews/PostCompletionView.stories.js:19:16
at testFn (src/components/PostCompletionViews/PostCompletionView.stories.js:211:49)
at Object.<anonymous> (src/components/PostCompletionViews/PostCompletionView.stories.js:224:33)
at step (src/components/PostCompletionViews/PostCompletionView.stories.js:109:23)
at Object.next (src/components/PostCompletionViews/PostCompletionView.stories.js:50:20)
at asyncGeneratorStep (src/components/PostCompletionViews/PostCompletionView.stories.js:4:28)
at _next (src/components/PostCompletionViews/PostCompletionView.stories.js:22:17)
at src/components/PostCompletionViews/PostCompletionView.stories.js:27:13
at Object.<anonymous> (src/components/PostCompletionViews/PostCompletionView.stories.js:19:16)
In the browser, there are no problems, only happens when using the test-runner.
To Reproduce
Unfortunately I don’t see a way to reproduce this reliably, but our repository is public: https://github.com/open-formulieren/open-forms-sdk/tree/chore/662-sb-upgrade
# clone repo
npm ci
npm run storybook
and in a different tab/shell:
npm run test:storybook
Expected behavior
Tests only run when the story story is ready
Screenshots n/a
System
Storybook Environment Info:
System:
OS: Linux 6.7 Arch Linux
CPU: (16) x64 AMD Ryzen 7 PRO 4750U with Radeon Graphics
Shell: 5.9 - /bin/zsh
Binaries:
Node: 20.10.0 - ~/.nvm/versions/node/v20.10.0/bin/node
Yarn: 1.22.21 - /usr/bin/yarn
npm: 10.2.3 - ~/.nvm/versions/node/v20.10.0/bin/npm <----- active
pnpm: 8.15.2 - /usr/bin/pnpm
npmPackages:
@storybook/addon-actions: ^8.0.0 => 8.0.0
@storybook/addon-coverage: ^1.0.1 => 1.0.1
@storybook/addon-essentials: ^8.0.0 => 8.0.0
@storybook/addon-interactions: ^8.0.0 => 8.0.0
@storybook/addon-links: ^8.0.0 => 8.0.0
@storybook/addon-webpack5-compiler-babel: ^3.0.3 => 3.0.3
@storybook/components: ^8.0.0 => 8.0.0
@storybook/manager-api: ^8.0.0 => 8.0.0
@storybook/preview-api: ^8.0.0 => 8.0.0
@storybook/react: ^8.0.0 => 8.0.0
@storybook/react-webpack5: ^8.0.0 => 8.0.0
@storybook/test: ^8.0.0 => 8.0.0
@storybook/test-runner: ^0.17.0 => 0.17.0
@storybook/types: ^8.0.0 => 8.0.0
msw-storybook-addon: ^1.10.0 => 1.10.0
storybook: ^8.0.0 => 8.0.0
storybook-addon-remix-react-router: ^3.0.0 => 3.0.0
storybook-addon-themes: ^6.1.0 => 6.1.0
storybook-react-intl: ^3.0.1 => 3.0.1
Additional context
-
I have an outdated theme addon that needs to be replaced still, no idea if that could be messing things up 🤔
-
The relevant code throwing the error appears to be this: https://github.com/storybookjs/storybook/blob/718ae4ad3352d2bc69f95c0cfab25b4b075e2308/code/lib/preview-api/src/modules/preview-web/Preview.tsx#L106
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 27
- Comments: 31 (3 by maintainers)
@VizTheWiz indeed we are putting a lot of work into the new Vitest stuff, but we will continue to maintain the test runner for the foreseeable future since it’s used by a lot of teams, does not require Vite, etc. I’ll try to get the team to look into this soon.
Hey @yannbf! Thanks for taking a look here!
I cannot speak for OG, but my team is having flaky tests with the exact same error code. We have not further investigated on our end because of time limitations and since one rerun always seems to fix this for a couple of runs but I had my eyes on this thread in the meantime.
I can unfortunately not give you a reliable reproduction, as this error only happens occasionally for us, but I might be able to help you pinpoint it by providing details of our setup (most of this has been mentioned before):
webkitfor us, never withchromiumorfirefox.test-runnerand not in the Storybook UI.preVisithook of thetest-runnerconfig (see image below), where we set up viewports with thestoryContextandinjectAxe(see code snippet).test-storybook -c src --url http://127.0.0.1:6006 --browsers webkit --junit --ci --maxWorkers=1.8.3.5in a monorepo setup.I hope this is helpful, let me know if there is anything else I can provide!
Do you have any update on this? I’m having the same issue here, but even worse is that if I add
await waitForPageReady(page)all my tests fail with a timeout:Increasing the timeout doesn’t help.
We have the same problem. If I add
await waitForPageReady(page);i runs less flakky.Currently this hacky approach seems te reduce the impact for me:
The idea is to try again but to delay if the first attempt did not work. A. bit hacky but seems to pass for now.
Same here! Same problem.
I’m doing the same thing where I’m using preVisit to set the viewport in some of our tests. In development mode, we see these errors when using
test-runner. When building statically, we don’t see the errors.I had the same issue in a project at work.
It happened very randomly, but at least every other time I ran the test.
Adding await waitForPageReady(page); only made it worse and made every test fail.
But extending the timeout by adding –testTimeout=60000 to all npm run test-storybook commands in my package.json file fixed the issue for me:
"test-storybook": "test-storybook --testTimeout=60000",I hope this helps anyone.
We faced the same issue after upgrading to Storybook v8. We set the viewport size in
preVisit. However, in some conditions (on CI env or not as part of a complex docker-compose), tests in the same docker image work fine.With a very small set of stories,
test-storybook --maxWorkers=1worked for me locally. Haven’t tried yet on CI.Any updates on this? I am having the exact same problem.