storybook: [Bug]: some tests failing on first run, but work seconds later (flakey, but consistent)
Describe the bug
I’ve been upgrading my CI from storybook test-runner to use the new vitest integration and noticed that my tests cause the ci to fail. When running locally I notice that when the node_module is freshly installed then i get the same failure, but when running the ci again the test now pass.
Looking at storybook and using the Test Widget, i get the exact same behaviour where the same components fail the first time, but will pass the second:
Reproduction link
https://github.com/lloydrichards/shadcn-storybook-registry
Reproduction steps
- install deps:
bun install - run storybook:
bun storybook - run Testing Widget
- ❌ get some failed tests
- run Testing Widget again
- ✅ get all passing tests
System
Storybook Environment Info:
System:
OS: macOS 15.5
CPU: (12) arm64 Apple M2 Max
Shell: 5.9 - /bin/zsh
Binaries:
Node: 22.13.1 - ~/.nvm/versions/node/v22.13.1/bin/node
Yarn: 1.22.22 - ~/.nvm/versions/node/v22.13.1/bin/yarn
npm: 10.9.2 - ~/Documents/GitHub/shadcn-storybook-registry/node_modules/.bin/npm
pnpm: 10.10.0 - /opt/homebrew/bin/pnpm
Browsers:
Chrome: 137.0.7151.55
Safari: 18.5
npmPackages:
@storybook/addon-a11y: ^9.0.0 => 9.0.0
@storybook/addon-docs: ^9.0.0 => 9.0.0
@storybook/addon-vitest: ^9.0.0 => 9.0.0
@storybook/nextjs-vite: 9.0.0 => 9.0.0
eslint-plugin-storybook: ^9.0.0 => 9.0.0
storybook: ^9.0.0 => 9.0.0
storybook-addon-test-codegen: ^1.3.4 => 1.3.4
Additional context
Terminal output for first rest run:
Test Files 18 failed | 29 passed (47)
Tests 38 failed | 72 passed (110)
Start at 11:51:55
Duration 15.74s (transform 0ms, setup 19.37s, collect 30.17s, tests 5.95s, environment 0ms, prepare 1.66s)
And second run:
storybook/test: Test Files 47 passed (47)
Tests 123 passed (123)
Start at 11:51:55
storybook/test: Duration 92.17s (transform 0ms, setup 9.56s, collect 8.77s, tests 3.92s, environment 0ms, prepare 961ms)
About this issue
- Original URL
- State: closed
- Created a year ago
- Reactions: 5
- Comments: 29 (7 by maintainers)
Commits related to this issue
- fix(ci): storybook cache issue in the first run https://github.com/storybookjs/storybook/issues/31599 — committed to soroush-tech/soroush.tech by soroushm a year ago
Similar issue since upgrading to storybook 9 and using the vitest project as per docs to run tests
cannot read properties of null (reading useContext, useState)Doing a
vitest run --project storybook > /dev/null 2>&1 || vitest run --project storybookfor my CI script is a current workaround.@NagaiKoki Could you make a separate issue for that and include a repo?
I also encountered this error, but it disappeared after adding the following configuration to
vitest.workspace.ts. Ideally, this error should not occur in the first place, so I hope it will be fixed in the future.@kenny-f I agree, but let’s start a new issue for your repro.
@eatyourgreens @yynnooot I would need a reproduction. Because I verified that the PR with this canary fixed the issue in @lloydrichards repro:
npx storybook@0.0.0-pr-31764-sha-f043b02d upgradeI’ve done some more digging and it seems the flakes appear whenever we see this message
It seems that this is happening during the test run so files have changed when the tests are running.
I was able to make this go away by adding all the dependencies in the
optimizeDeps.includebut that is not a long term solution and it can and will change regularly.We also have regular unit tests using vitest, testing the same application and not have had to add any of these
optimizeDeps.includecompared to running storybook with vitest addon.I’m seeing the same issue when in the terminal when running headless.
I can reliably reproduce it when there initial no cache in node_modules.
First I delete
node_modules/.cache/storybook1st run:
Error: Failed to import test file /........./node_modules/@storybook/addon-vitest/dist/vitest-plugin/setup-file.mjs2nd run:
The file does not exist at "/......../node_modules/.cache/storybook/387356c5301985278b69755b01eff3e43f747fbc8f67b52cdea22a3b61b0d12e/sb-vitest/deps/react-18-XN45J6DR.js?v=176eccab" which is in the optimize deps directory. The dependency might be incompatible with the dep optimizer. Try adding it to `optimizeDeps.exclude3rd run: Runs successfully without any errors.
glad i’m not the only one 🥲
I just upgraded to vitest v3.2.0 and still getting the same issue, but i appreciate the workaround 🙏
** Disclaimer** This information might be inaccurate, due to it being generated automatically This appears to be a Vite dependency optimization issue. Add the following to your
.storybook/main.tsconfiguration:ts export default { viteFinal: (config) => { config.optimizeDeps = { ...config.optimizeDeps, exclude: [ '@radix-ui/react-tooltip', 'sonner', 'next-themes' ] }; return config; } };This will prevent Vite from attempting to optimize these dependencies that are causing the initial test failures.About Greptile
This response provides a starting point for your research, not a precise solution.
Help us improve! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
Edit Issue Bot Settings · Greptile