test-runner: [bug] preVisit accesses the story store before the index is ready

Describe the bug

We have the test-runner.js implement with the preVisit to apply the different viewports when running tests, per the README.

After upgrading to Storybook 8, our tests fail in a flaky way, e.g.:

 FAIL  src/components/PostCompletionViews/PostCompletionView.stories.js
  ● Private API / Post completion views  › Generic › smoke-test

    page.evaluate: SB_PREVIEW_API_0011 (StoryStoreAccessedBeforeInitializationError): Cannot access the Story Store until the index is ready.

    It is not recommended to use methods directly on the Story Store anyway, in Storybook 9 we will
    remove access to the store entirely

      at Object.get (http:/127.0.0.1:6006/sb-preview/runtime.js:118:5542)
      at __getContext (<anonymous>:215:54)
      at eval (eval at evaluate (:226:30), <anonymous>:1:29)
      at UtilityScript.evaluate (<anonymous>:228:17)
      at UtilityScript.<anonymous> (<anonymous>:1:44)
      at getStoryContext (node_modules/@storybook/test-runner/dist/index.js:15306:15)
      at getStoryContext (.storybook/test-runner.js:9:27)
      at src/components/PostCompletionViews/PostCompletionView.stories.js:151:64
      at step (src/components/PostCompletionViews/PostCompletionView.stories.js:109:23)
      at Object.next (src/components/PostCompletionViews/PostCompletionView.stories.js:50:20)
      at asyncGeneratorStep (src/components/PostCompletionViews/PostCompletionView.stories.js:4:28)
      at _next (src/components/PostCompletionViews/PostCompletionView.stories.js:22:17)
      at src/components/PostCompletionViews/PostCompletionView.stories.js:27:13
      at src/components/PostCompletionViews/PostCompletionView.stories.js:19:16
      at testFn (src/components/PostCompletionViews/PostCompletionView.stories.js:211:49)
      at Object.<anonymous> (src/components/PostCompletionViews/PostCompletionView.stories.js:224:33)
      at step (src/components/PostCompletionViews/PostCompletionView.stories.js:109:23)
      at Object.next (src/components/PostCompletionViews/PostCompletionView.stories.js:50:20)
      at asyncGeneratorStep (src/components/PostCompletionViews/PostCompletionView.stories.js:4:28)
      at _next (src/components/PostCompletionViews/PostCompletionView.stories.js:22:17)
      at src/components/PostCompletionViews/PostCompletionView.stories.js:27:13
      at Object.<anonymous> (src/components/PostCompletionViews/PostCompletionView.stories.js:19:16)

In the browser, there are no problems, only happens when using the test-runner.

To Reproduce

Unfortunately I don’t see a way to reproduce this reliably, but our repository is public: https://github.com/open-formulieren/open-forms-sdk/tree/chore/662-sb-upgrade

# clone repo
npm ci
npm run storybook

and in a different tab/shell:

npm run test:storybook

Expected behavior

Tests only run when the story story is ready

Screenshots n/a

System

Storybook Environment Info:

  System:
    OS: Linux 6.7 Arch Linux
    CPU: (16) x64 AMD Ryzen 7 PRO 4750U with Radeon Graphics
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 20.10.0 - ~/.nvm/versions/node/v20.10.0/bin/node
    Yarn: 1.22.21 - /usr/bin/yarn
    npm: 10.2.3 - ~/.nvm/versions/node/v20.10.0/bin/npm <----- active
    pnpm: 8.15.2 - /usr/bin/pnpm
  npmPackages:
    @storybook/addon-actions: ^8.0.0 => 8.0.0 
    @storybook/addon-coverage: ^1.0.1 => 1.0.1 
    @storybook/addon-essentials: ^8.0.0 => 8.0.0 
    @storybook/addon-interactions: ^8.0.0 => 8.0.0 
    @storybook/addon-links: ^8.0.0 => 8.0.0 
    @storybook/addon-webpack5-compiler-babel: ^3.0.3 => 3.0.3 
    @storybook/components: ^8.0.0 => 8.0.0 
    @storybook/manager-api: ^8.0.0 => 8.0.0 
    @storybook/preview-api: ^8.0.0 => 8.0.0 
    @storybook/react: ^8.0.0 => 8.0.0 
    @storybook/react-webpack5: ^8.0.0 => 8.0.0 
    @storybook/test: ^8.0.0 => 8.0.0 
    @storybook/test-runner: ^0.17.0 => 0.17.0 
    @storybook/types: ^8.0.0 => 8.0.0 
    msw-storybook-addon: ^1.10.0 => 1.10.0 
    storybook: ^8.0.0 => 8.0.0 
    storybook-addon-remix-react-router: ^3.0.0 => 3.0.0 
    storybook-addon-themes: ^6.1.0 => 6.1.0 
    storybook-react-intl: ^3.0.1 => 3.0.1 

Additional context

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 27
  • Comments: 31 (3 by maintainers)

Most upvoted comments

@VizTheWiz indeed we are putting a lot of work into the new Vitest stuff, but we will continue to maintain the test runner for the foreseeable future since it’s used by a lot of teams, does not require Vite, etc. I’ll try to get the team to look into this soon.

Hey @yannbf! Thanks for taking a look here!

I cannot speak for OG, but my team is having flaky tests with the exact same error code. We have not further investigated on our end because of time limitations and since one rerun always seems to fix this for a couple of runs but I had my eyes on this thread in the meantime.

I can unfortunately not give you a reliable reproduction, as this error only happens occasionally for us, but I might be able to help you pinpoint it by providing details of our setup (most of this has been mentioned before):

  1. This specific issue only happens when testing on webkit for us, never with chromium or firefox.
  2. Checking our CI history, this happens about every 4th to 6th run (not very helpful but still 😅).
  3. The issue only happens with the test-runner and not in the Storybook UI.
  4. The error itself always originates in the preVisit hook of the test-runner config (see image below), where we set up viewports with the storyContext and injectAxe (see code snippet).
  5. The command we use for the CI looks as follows test-storybook -c src --url http://127.0.0.1:6006 --browsers webkit --junit --ci --maxWorkers=1.
  6. We are currently running Storybook 8.3.5 in a monorepo setup.

I hope this is helpful, let me know if there is anything else I can provide!

Image

async preVisit(page, context) {
    await waitForPageReady(page);

    const storyContext = await getStoryContext(page, context);

    // Accesses the story's parameters and retrieves the viewport used to render it
    // https://storybook.js.org/docs/writing-tests/test-runner#accessing-story-information-with-the-test-runner
    const storyViewport =
        storyContext.parameters?.viewport?.defaultViewport;

    const viewportName: ViewportBreakpointToken =
        storyViewport in VIEWPORTS ? storyViewport : DEFAULT_VIEWPORT;

    const { width, height } = VIEWPORTS[viewportName].styles;

    // Configures the Playwright page to use the viewport size
    await page.setViewportSize({
        width: parseFloat(width.replace('px', '')),
        height: parseFloat(height.replace('px', '')),
    });

    await injectAxe(page);
},

Do you have any update on this? I’m having the same issue here, but even worse is that if I add await waitForPageReady(page) all my tests fail with a timeout:

`thrown: "Exceeded timeout of 15000 ms for a test. Add a timeout value to this test to increase the timeout, if this is a long-running test. See https://jestjs.io/docs/api#testname-fn-timeout." `

Increasing the timeout doesn’t help.

We have the same problem. If I add await waitForPageReady(page); i runs less flakky.

const config: TestRunnerConfig = {
  async preVisit(page, story) {
    await waitForPageReady(page);
    }
}

Currently this hacky approach seems te reduce the impact for me:

async function waitForStoryContext(
  page: Page,
  story: StoryFn,
  attempt = 1,
  maxAttempts = 20,
) {
  try {
    return await getStoryContext(page, story);
  } catch (e) {
    if (attempt > maxAttempts) {
      throw e;
    }
    // ¯\_(ツ)_/¯ - If this is not the first attempt: add a timeout.
    await new Promise(resolve => setTimeout(resolve, 600));
    return waitForStoryContext(page, story, attempt + 1);
  }
}

async preVisit(page: Page, story: StoryFn) {
    const context = await waitForStoryContext(page, story);
   // ...
};

The idea is to try again but to delay if the first attempt did not work. A. bit hacky but seems to pass for now.

Same here! Same problem.

I’m doing the same thing where I’m using preVisit to set the viewport in some of our tests. In development mode, we see these errors when using test-runner. When building statically, we don’t see the errors.

I had the same issue in a project at work.

It happened very randomly, but at least every other time I ran the test.

Adding await waitForPageReady(page); only made it worse and made every test fail.

But extending the timeout by adding –testTimeout=60000 to all npm run test-storybook commands in my package.json file fixed the issue for me:

"test-storybook": "test-storybook --testTimeout=60000",

I hope this helps anyone.

We faced the same issue after upgrading to Storybook v8. We set the viewport size in preVisit. However, in some conditions (on CI env or not as part of a complex docker-compose), tests in the same docker image work fine.

 page.evaluate: SB_PREVIEW_API_0011 (StoryStoreAccessedBeforeInitializationError): Cannot access the Story Store until the index is ready.

    It is not recommended to use methods directly on the Story Store anyway, in Storybook 9 we will
    remove access to the store entirely

      at Object.get (http:/localhost:6006/sb-preview/runtime.js:8396:17)
      at __getContext (<anonymous>:219:54)
      at eval (eval at evaluate (:234:30), <anonymous>:1:29)
      at UtilityScript.evaluate (<anonymous>:236:17)
      at UtilityScript.<anonymous> (<anonymous>:1:44)
      at getStoryContext (node_modules/@storybook/test-runner/dist/index.js:12861:15)
      at preVisit (.storybook/test-runner.ts:43:42)

With a very small set of stories, test-storybook --maxWorkers=1 worked for me locally. Haven’t tried yet on CI.

Any updates on this? I am having the exact same problem.