storybook: [Bug]: some tests failing on first run, but work seconds later (flakey, but consistent)

Describe the bug

I’ve been upgrading my CI from storybook test-runner to use the new vitest integration and noticed that my tests cause the ci to fail. When running locally I notice that when the node_module is freshly installed then i get the same failure, but when running the ci again the test now pass.

Github Action

Looking at storybook and using the Test Widget, i get the exact same behaviour where the same components fail the first time, but will pass the second:

Image

Reproduction link

https://github.com/lloydrichards/shadcn-storybook-registry

Reproduction steps

  1. install deps: bun install
  2. run storybook: bun storybook
  3. run Testing Widget
  4. ❌ get some failed tests
  5. run Testing Widget again
  6. ✅ get all passing tests

System

Storybook Environment Info:

  System:
    OS: macOS 15.5
    CPU: (12) arm64 Apple M2 Max
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 22.13.1 - ~/.nvm/versions/node/v22.13.1/bin/node
    Yarn: 1.22.22 - ~/.nvm/versions/node/v22.13.1/bin/yarn
    npm: 10.9.2 - ~/Documents/GitHub/shadcn-storybook-registry/node_modules/.bin/npm
    pnpm: 10.10.0 - /opt/homebrew/bin/pnpm
  Browsers:
    Chrome: 137.0.7151.55
    Safari: 18.5
  npmPackages:
    @storybook/addon-a11y: ^9.0.0 => 9.0.0 
    @storybook/addon-docs: ^9.0.0 => 9.0.0 
    @storybook/addon-vitest: ^9.0.0 => 9.0.0 
    @storybook/nextjs-vite: 9.0.0 => 9.0.0 
    eslint-plugin-storybook: ^9.0.0 => 9.0.0 
    storybook: ^9.0.0 => 9.0.0 
    storybook-addon-test-codegen: ^1.3.4 => 1.3.4

Additional context

Terminal output for first rest run:

 Test Files  18 failed | 29 passed (47)
      Tests  38 failed | 72 passed (110)
   Start at  11:51:55
   Duration  15.74s (transform 0ms, setup 19.37s, collect 30.17s, tests 5.95s, environment 0ms, prepare 1.66s)

And second run:

storybook/test: Test Files  47 passed (47)
      Tests  123 passed (123)
   Start at  11:51:55
storybook/test: Duration  92.17s (transform 0ms, setup 9.56s, collect 8.77s, tests 3.92s, environment 0ms, prepare 961ms)

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 5
  • Comments: 29 (7 by maintainers)

Commits related to this issue

Most upvoted comments

Similar issue since upgrading to storybook 9 and using the vitest project as per docs to run tests

  1. build storybook
  2. serve storybook on url
  3. vitest run --project storybook -> many tests fail with errors like cannot read properties of null (reading useContext, useState)
  4. Run it again all passes

Doing a vitest run --project storybook > /dev/null 2>&1 || vitest run --project storybook for my CI script is a current workaround.

@NagaiKoki Could you make a separate issue for that and include a repo?

I also encountered this error, but it disappeared after adding the following configuration to vitest.workspace.ts. Ideally, this error should not occur in the first place, so I hope it will be fixed in the future.

export default defineWorkspace([
    // ...
+  optimizeDeps: {
+     include: ['react/jsx-dev-runtime'],
+   },
  },
])

@kenny-f I agree, but let’s start a new issue for your repro.

@eatyourgreens @yynnooot I would need a reproduction. Because I verified that the PR with this canary fixed the issue in @lloydrichards repro: npx storybook@0.0.0-pr-31764-sha-f043b02d upgrade

I’ve done some more digging and it seems the flakes appear whenever we see this message

[vite] (client) ✨ new dependencies optimized: {a list of dependencies...}
[vite] (client) ✨ optimized dependencies changed. reloading
[vitest] Vite unexpectedly reloaded a test. This may cause tests to fail, lead to flaky behaviour or duplicated test runs.
For a stable experience, please add mentioned dependencies to your config's `optimizeDeps.include` field manually.

It seems that this is happening during the test run so files have changed when the tests are running.

I was able to make this go away by adding all the dependencies in the optimizeDeps.include but that is not a long term solution and it can and will change regularly.

We also have regular unit tests using vitest, testing the same application and not have had to add any of these optimizeDeps.include compared to running storybook with vitest addon.

I’m seeing the same issue when in the terminal when running headless.

I can reliably reproduce it when there initial no cache in node_modules.

First I delete node_modules/.cache/storybook

1st run: Error: Failed to import test file /........./node_modules/@storybook/addon-vitest/dist/vitest-plugin/setup-file.mjs

2nd run: The file does not exist at "/......../node_modules/.cache/storybook/387356c5301985278b69755b01eff3e43f747fbc8f67b52cdea22a3b61b0d12e/sb-vitest/deps/react-18-XN45J6DR.js?v=176eccab" which is in the optimize deps directory. The dependency might be incompatible with the dep optimizer. Try adding it to `optimizeDeps.exclude

3rd run: Runs successfully without any errors.

glad i’m not the only one 🥲

I just upgraded to vitest v3.2.0 and still getting the same issue, but i appreciate the workaround 🙏

** Disclaimer** This information might be inaccurate, due to it being generated automatically This appears to be a Vite dependency optimization issue. Add the following to your .storybook/main.ts configuration: ts export default { viteFinal: (config) => { config.optimizeDeps = { ...config.optimizeDeps, exclude: [ '@radix-ui/react-tooltip', 'sonner', 'next-themes' ] }; return config; } }; This will prevent Vite from attempting to optimize these dependencies that are causing the initial test failures.

About Greptile

This response provides a starting point for your research, not a precise solution.

Help us improve! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

Edit Issue Bot Settings · Greptile