cypress: ChunkLoadError > Loading chunk cypress-support-file failed

Current behavior

After updating from cypress@12.16.0 to cypress@13.6.2, we are seeing this error occasionally on our CI runs.

ChunkLoadError: The following error originated from your test code, not from Cypress.
> Loading chunk cypress-support-file failed.
[(error: http://localhost:8080/__cypress/src/cypress-support-file.js)]
When Cypress detects uncaught errors originating from your test code it will automatically fail the current test.
Cypress could not associate this error to any specific test.
We dynamically generated a new test to display this failure.
[at __webpack_require__.f.j (http://localhost:8080/__cypress/src/runtime.js:272:29)]
[at <unknown> (http://localhost:8080/__cypress/src/runtime.js:126:40)]
at Array.reduce (<anonymous>)
[at __webpack_require__.e (http://localhost:8080/__cypress/src/runtime.js:125:67)]
[at Object.load (http://localhost:8080/__cypress/src/cypress-entry.js:33:192)]
[at <unknown> (http://localhost:8080/__cypress/runner/cypress_runner.js:110819:21)]
[at tryCatcher (http://localhost:8080/__cypress/runner/cypress_runner.js:1807:23)]
[at Object.gotValue (http://localhost:8080/__cypress/runner/cypress_runner.js:6476:18)]
[at Object.gotAccum (http://localhost:8080/__cypress/runner/cypress_runner.js:6465:25)]
[at Object.tryCatcher (http://localhost:8080/__cypress/runner/cypress_runner.js:1807:23)]

Unfortunately it is not reproducable locally, so I can not give you a repository to check. I know that this might make it hard to investigate but maybe somebody already fell over the same issue or others are experiencing it as well.

The test fails around 30% of the time.

Desired behavior

No response

Test code to reproduce

no reproduction repository

Cypress Version

13.6.2

Node version

18.19.0

Operating System

Ubuntu 22.04.3

Debug Logs

No response

Other

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 37
  • Comments: 147 (68 by maintainers)

Most upvoted comments

Any progress on the possible fix (by providing configuration options) according to #29073 ? We still encounter this issue quite often and a (possible) fix would be highly appreciated 🤗

We’re planning to set aside time over the next few months to try to find a resolution to this. Hopefully nothing pulls those plans off track.

We have had good results in turning off splitChunks.chunks = 'all' and leveraging splitChunks.chunks = 'initial' to avoid async chunks. However, this isn’t easy to do currently with @cypress/webpack-dev-server since the optimization option is overwritten . I created a feature request https://github.com/cypress-io/cypress/issues/29073 to allow users to configure this option. There are more details on a possible work around for this issue on the feature request and interested to see who gives it a try!

We’re spiking into this to see if we can find what’s causing this.

@mirobo We intend to release this as an experimental feature to enable with a newer Cypress version. Hopefully the next release if all goes well.

Hey all. I have been working on a solution to the chunk load errors, in particular the just-in-time compiling solution proposed in https://github.com/cypress-io/cypress/issues/29244. Effectively, the experimental flag experimentalJustInTimeCompile will only compile assets that are currently under test (the current spec) as opposed to all related CT specs in the project. The idea here is less entries should mean smaller dev server which in some cases might mean smaller chunks. We have seen with our own testing that memory usage is much lower (in our case about half the memory used) and we have not run into chuck load errors in testing.

However, since this issue is unique in the fact that everyones set up is different, I would like to see if this experiment alleviates the problem for you all and understand possible cases where it may not. I have some pre release binaries build on this commit (note: this is based on 13.13.3).

In particular I am curious about the cypress-support-file chunk failing, since I don’t believe this chunk really can get much smaller for users since the support file is somewhat constant. As we get closer to releasing the feature I will post updates here, but the prerelease is available if anyone wants to give it a try! You can turn it on like this:

// cypress.config.js
{
  component: {
    experimentalJustInTimeCompile: true
  }
}

Sorry for the delay in an update. We are continuing to experiment and try a few things on our end. @anilanar’s comment of

This is a problem with async chunks triggering builds and those builds not finishing before test times out.

I think is likely the correct speculation. We are also wondering if the async chunks are being rebuilt on the fly in the dev-server cache and that is also causing more problems.

Either way, on our end we are experimenting with turning async chunking off and opting for initial chunking. The issue is @cypress/webpack-dev-server overrides this configuration for users (likely when it shouldn’t), so we needed to patch the package in order to try this.

We are gathering more data with this change. It has caused a spike in some of our resource usage, but I think we have that ironed out. Once we have more information on whether the configuration changes help alleviate the symptoms I will report back!

Hey all. Sorry for the delay on an update here. I’ve been investigating the issue heads down for about a week now. I’m able to reproduce the error on my end but only at the start of the run and sparingly in the middle of the run. I think there are two issues possibly going on here

At the start of the run

At the end of webpack compilation, the dev-server:compile:success event is emitted by the cypress webpack dev server, but nothing is actively listening to this event. The only exception is watchForChanges being enabled inside the SocketCT handler inside the Cypress server, which is unset in run mode and isn’t applicable here.

In other words, we can get ourselves into a race condition where webpack takes a long time to build all assets, but the cypress run has begun and is able to resolve the index.html chunk, but not other chunks in the test because they are not present in the output directory. That is what is happening in this example:

support_file_fallover_with_retries

In the middle of a run

Adding to what @benpatterson mentioned and a bit more complicated to reproduce, but does happen, is webpack-dev-server recompiling in the middle of a test. We can see something like this here with some logging I added in this diagnostic binary.

Test starts. Something triggers webpack-dev-server to recompile. test fails with chunk error

1

Just in time recompile (just by chance) while the browser is relaunching for the second test to run. second test passes

2

For this, it seems like the dev-server is watching assets and recompiling. What I need to further investigate is

  • what is being compiled here and does it need to be compiled? Maybe something configuration-wise changed in the webpack.config from version 4 to 5.

Waiting for the dev server to compile is a bit tricky as it can trigger a compilation at any time, but if we can answer the above question, it puts us in a better position to solve the problem correctly.

Some questions that might be helpful for us in fixing the issue

  • Does the error happen at the beggining of the cypress run or in the middle? It sounds like the first case for @TomaszG where compilation fails in the beginning of the test and the second case for @benpatterson where the test fails in the middle. I am curious with others running into this issue where it is happening for you.
  • What are the stats on your webpack build? Any idea how long it is taking for webpack to recompile?
  • if willing, we have a diagnostic binary build here (this is not a fix) that you can install and run with the following flags CYPRESS_INTERNAL_BROWSER_CONNECT_TIMEOUT=120000 DEBUG='cypress:server:open_project,cypress:webpack-dev-server:CypressCTWebpackPlugin npx cypress run --component ... which might give us an idea what webpack is doing under your test.

I will give more updates as soon as I have them. I appreciate everyones patience with the issue resolution.

It’s a big chunk of work, where we do understand the problem well now. We’re prioritizing some of the support for new component testing frameworks atm since that’s quicker work to get out and hoping to get to this work after.

Just an update after 2 weeks: we don’t have ChunkLoadErrors anymore with a custom “dev server” that serves prebundled static files.

Closing as this will be considered done with Cypress 14 upcoming release

This makes it difficult to depend on Cypress component tests for validation in CI. These chunk failures are making CI flakey. If we could get some official focus on this, that’d be appreciated. It’s a real issue right now for us early adopters of Cypress component tests.

We are getting the same error also intermittently. I don’t think we had it as far back as 12.17.4 but I can’t say that for certain.

It only seems to happen toward the end of our Component Test runs. We have 1500+ component tests that we are running.

The following error originated from your test code, not from Cypress.

  > Loading chunk cypress-support-file failed.
(error: http://localhost:8080/__cypress/src/cypress-support-file.js)

When Cypress detects uncaught errors originating from your test code it will automatically fail the current test.

Cypress could not associate this error to any specific test.

We dynamically generated a new test to display this failure.
ChunkLoadError: The following error originated from your test code, not from Cypress.
  > Loading chunk cypress-support-file failed.
(error: http://localhost:8080/__cypress/src/cypress-support-file.js
When Cypress detects uncaught errors originating from your test code it will automatically fail the current test.
Cypress could not associate this error to any specific test.
We dynamically generated a new test to display this failure.
    at __webpack_require__.f.j (http://localhost:8080/__cypress/src/main.js:8702:29
    at <unknown> (http://localhost:8080/__cypress/src/main.js:8552:40
    at Array.reduce (<anonymous>)
    at __webpack_require__.e (http://localhost:8080/__cypress/src/main.js:8551:67
    at Object.load (http://localhost:8080/__cypress/src/main.js:1510:582
    at <unknown> (http://localhost:8080/__cypress/runner/cypress_runner.js:110819:21
    at tryCatcher (http://localhost:8080/__cypress/runner/cypress_runner.js:1807:23
    at Object.gotValue (http://localhost:8080/__cypress/runner/cypress_runner.js:6476:18
    at Object.gotAccum (http://localhost:8080/__cypress/runner/cypress_runner.js:6465:25
    at Object.tryCatcher (http://localhost:8080/__cypress/runner/cypress_runner.js:1807:23
[Open in build log](https://teamcity.haivision.com/buildConfiguration/CineNet4x_Projects_CypressComponents/690401?hideProblemsFromDependencies=false&expandBuildTestsSection=true&hideTestsFromDependencies=false&expandBuildChangesSection=true&showLog=690401_2000000247_2000000247&logFilter=debug)

I’m confused as to why this was closed? The flag “experimentalJustInTimeCompile” made things worse for us and I wonder what solution is going to come with Cypress 14?

It’s a big chunk of work, where we do understand the problem well now. We’re prioritizing some of the support for new component testing frameworks atm since that’s quicker work to get out and hoping to get to this work after.

@jennifer-shehane you’re prioritizing adding support for new component testing frameworks? That sucks. This is a pretty big bug in an existing component testing framework. It’s impactful for companies doubling-down on Cypress component tests (like us). Can’t supporting existing users be a higher priority?

Any updates? This is blocking our upgrade to Cypress v13.

We’re seeing this on 13.2

We ended up reverting and pinning to 13.5.1 to avoid the crashes. It’s not a solid perm fix since we eventually will need to upgrade and pull in other critical bug fixes, but helps us keep our tests running.

Adding these event handlers to the support / component file got our CI passing reliably

if (window.Cypress) {
  // Prevent chunk loading errors from failing tests
  const originalOnError = window.onerror
  window.onerror = (msg, source, lineno, colno, err) => {
    if (err && err.message && err.message.includes('Loading chunk')) {
      console.warn('Chunk loading error intercepted:', err)
      return false
    }
    return originalOnError?.(msg, source, lineno, colno, err)
  }

  window.addEventListener('unhandledrejection', (event) => {
    if (event.reason && event.reason.message && event.reason.message.includes('Loading chunk')) {
      event.preventDefault()
      console.warn('Chunk loading rejection intercepted:', event.reason)
    }
  })

  window.addEventListener('error', (event) => {
    if (event.error?.message?.includes('Loading chunk')) {
      event.preventDefault()
      cy.log('Chunk loading error detected - reloading page')
      window.location.reload()
    }
  })
}

With larger codebases, the experimentalJustInTimeCompile=true makes the whole execution time even 3x longer. This is a case in our project where we test larger components with multiple specs. Because now webpack compilation runs before each spec, the total execution time growth from ~550s to ~1400s for the whole suite.

@TomaszG Execution time is going to vary on several factors, but in some cases it is completely reasonable to expect longer execution times. Are you leveraging any type of smart orchestration features, like parallelization to speed up the total run time? Most importantly, are the chunk load errors still present with experimentalJustInTimeCompile=true

Yes, we do run the tests in parallel and component tests run takes significantly longer, especially on CI. Given that webpack bundle doesn’t change during the test execution, I don’t see a reason of involving webpack bundle for each spec, as that’s a major performance drawback, especially when testing complex enterprise apps. I recon that a simple wait (even static if there’s no milestone on which you could rely) for webpack to finish or a one-time retry in case of ChunkLoadError would make the error gone for most of the users.

Looks like @AlonMiz is seeing similar results. 25% is in the margin of expectation. @AlonMiz are you seeing chunk load errors and are the tests completing as expected?

No more errors 🤞👏 Stability is more important than speed. I’ll stick to that setting for now. I definitely think performance should be improved in the future as i dont want to move to playwright for now

seeing a decrease in performance after applying this flag experimentalJustInTimeCompile AFTER:4m 40s BEFORE:3m 5s

This looks very promising. 4 different smoke tests in CI are working correctly. We will continue to monitor the behavior. Thanks for the release so far

The experimentalJustInTimeCompile feature released today with Cypress 13.14.0. Please give this a try and give us your feedback here or in this github discussion on whether the experiment improves the chunk load error outcomes.

Any news on this, we are still pinned on v12 until this can be fixed?

@AtofStryker, great investigation and summary. I can confirm that the error happens only at the beginning of the run for me; I haven’t noticed it in the middle. I also haven’t noticed any webpack re-compilation in the middle of the run.

The compilation times on CI are around 19 seconds for the first run (which is allowed to fail) and 14 seconds for the second run.

We made an update from webpack 4 to webpack 5 in 12.17.4, I wonder if it has anything to do with that upgrade. https://docs.cypress.io/guides/references/changelog#12-17-4

Hi, I’m currently seeing this issue on my own project, but instead with a random chunk (not the support file) like #30521 and other posts above.

I don’t have much to contribute to the conversation, but I know my specific issue isn’t rooted in chrome since we don’t use it for running. We do use webpack

I appreciate the work looking into this and look forward to when this issue can be reopened. Thanks.

we are still experiencing this issue quite a lot and it causes PR pipelines to fail - even multiple times in a row. @jennifer-shehane - as multiple other people above still run into the same issue, can you please re-open it and give us a status?

we tried updating to 14.x using default settings

  • error still happened
  • tests take longer overall
  • some test suites failed to start locally with the watch parameter, they would start and then idle indefinitely

mitigations we tried

I get that webpack is theoretically not a hard requirement (mentioned that in a comment earlier) except that it is because without it the mocking/stubing functionality is not working with Vite due to the nature of esm. At least that was a problem a few months ago when I last tried to migrate away from using the webpack dev server. That’s actually the only reason why we still have webpack in our project - to support Cypress component tests. The app itself is built with Vite. If anything changed in that regard, I’m happy to try migrating again.

I would not rule out a memory leak somewhere – the project consists of almost 2000 Vue components and a similar number of ts and js files. But investigating that is not an easy task as you might imagine.

Don’t get me wrong, I’ve been a proponent of Cypress pretty much since its inception. But the amount of frustration it has been generating for our team over the last years is beginning to be hard to ignore. Of course, a lot of that might come from the pains of a rapidly growing codebase. So please do take my rant with a grain of salt. 😅

edit: we are also using the code coverage plugin for component tests which is contributing to the memory consumption, though I haven’t measured exactly by how much.

We are hoping with experimentalJustInTimeCompile that we can allow users to better horizontally scale with the option on where users can leverage smart orchestration for really large enterprise projects so the time is not a significant factor. We have seen significant improvements on our end when leveraging this.

@AtofStryker That’s something that CI providers often provide out of the box (load balancing tests based on timings). It might help with the overall time, but it does nothing to help with the budget side of things. Throwing more machines at the problem should not be the only solution to a performance problem like this. Even if it is the simplest way to go around it, compute is ultimately not free. Unless of course, Cypress starts offering alternative hosted CI runners where clients can pay on a per test basis rather than per compute minutes.

Bill and I found that when intending to upgrade to v13.15.0, I must have typed in 13.5.0, that is whay it seemed like the ChunkLoadError has come back. We used 13.15.0 and the error went away with the flag experimentalJustInTimeCompile

However, we uncovered a new issue where time to time, the last component test does not crash, but hangs there indefinately.

What kind of logs or debugging can I give to aid with diagnosing this one?

image

@AtofStryker

regarding experimentalJustInTimeCompile We do see it in console.log, but we do not actually see it in the runner, there is no entry for it anymore, perhaps since a recent version.

I sent you a link of the zipped repo on Discord, if you can log in within 24 hours.

@jennifer-shehane @AtofStryker I have reported previously that experimentalJustInTimeCompile made things better for the chunkLoad error. However, as the component suite grows, we still sporadically see it, with a failure rate of 16% (the Cypress Cloud link should be accessible by Cypress employees). We are somewhat desperate for a solution, as we are vested into component testing with over 500 files/1600 it blocks. Unless there is a solution, the only plausible way forward for us could be to execute tests based on source code changes. We would be missing a lot of issues, but we would not be alerted falsely (and cost 1600+ runs from our credit) at a 16% rate.

Hey @muratkeremozcan.

Thank you for the feedback! I am taking a look at the failures and can see the chunk load error. I am seeing something like :

The following error originated from your test code, not from Cypress.


  > Loading chunk vendors~cypress-support-file~spec-1~spec-10~spec-100~spec-101~spec-102~spec-104~spec-105~spec-106~sp~41b558d7 failed.

(error: http://localhost:3000/__cypress/src/vendors~cypress-support-file~spec-1~spec-10~spec-100~spec-101~spec-102~spec-104~spec-105~spec-106~sp~41b558d7.js)

When Cypress detects uncaught errors originating from your test code it will automatically fail the current test.

Cypress could not associate this error to any specific test.

We dynamically generated a new test to display this failure.

Is there something specific about compiling vendor code in your webpack config? The reason I ask is each spec should only compile assets related to itself, so there should be no chunking of the specs like what we see here. It almost looks like the experimentalJustInTimeCompile flag is not turned on here or something custom is happening. Are you able to send a reproduction to the support team?

I am sending the zipped repo to support right away.

Worst case scenario, is there a way we can flag the ChunLoadError a false positive, and not have things fail but perhaps just skip when we see it?

https://files.slack.com/files-pri/T047EH9G6G2-F07RK079PV0/image.png

@jennifer-shehane @AtofStryker

I have reported previously that experimentalJustInTimeCompile made things better for the chunkLoad error.

However, as the component suite grows, we still sporadically see it, with a failure rate of 16% (the Cypress Cloud link should be accessible by Cypress employees).

We are somewhat desperate for a solution, as we are vested into component testing with over 500 files/1600 it blocks.

Unless there is a solution, the only plausible way forward for us could be to execute tests based on source code changes. We would be missing a lot of issues, but we would not be alerted falsely (and cost 1600+ runs from our credit) at a 16% rate.

Any updates? This problem makes components testing completely unusable.

  Running:  DisplaySlackChannel.cy.tsx                                                      (4 of 4)


  1) An uncaught error was detected outside of a test

  0 passing (84ms)
  1 failing

  1) An uncaught error was detected outside of a test:
     ChunkLoadError: The following error originated from your test code, not from Cypress.

  > Loading chunk cypress-support-file failed.
(missing: http://localhost:8080/__cypress/src/cypress-support-file.js)

When Cypress detects uncaught errors originating from your test code it will automatically fail the current test.

Cypress could not associate this error to any specific test.

We dynamically generated a new test to display this failure.
  ChunkLoadError
      at __webpack_require__.f.j (http://localhost:8080/__cypress/src/webpack.js:326:29)
      at <unknown> (http://localhost:8080/__cypress/src/webpack.js:137:40)
      at Array.reduce (<anonymous>)
      at __webpack_require__.e (http://localhost:8080/__cypress/src/webpack.js:136:67)
      at Object.load (http://localhost:8080/__cypress/src/main.js:3396:447)
      at <unknown> (http://localhost:8080/__cypress/runner/cypress_runner.js:111334:21)
      at tryCatcher (http://localhost:8080/__cypress/runner/cypress_runner.js:1807:23)
      at Object.gotValue (http://localhost:8080/__cypress/runner/cypress_runner.js:6476:18)
      at Object.gotAccum (http://localhost:8080/__cypress/runner/cypress_runner.js:6465:25)
      at Object.tryCatcher (http://localhost:8080/__cypress/runner/cypress_runner.js:1807:23)

  (Results)

I came across this issue today using Cypress 13.11.0. Not sure if this will help others, but for my particular test this was caused by me trying to set a env variable in the test to a dummy value in the beforeEach().

process.env.MY_CUSTOM_VAR = Cypress.env('MY_CUSTOM_VAR');

The error went away when I commented out the variable in my .env.local file. Therefore, I updated the test to check if the variable is already set before trying to set it in the beforeEach().

if (!process.env.MY_CUSTOM_VAR) {
      process.env.MY_CUSTOM_VAR = Cypress.env('MY_CUSTOM_VAR');
}

I think memory problems are caused by either source maps (webpackConfig.devtools) or webpack cache (webpackConfig.cache) or both.

I have a feeling chunk load errors happen due to swallowed OOM errors though. Did people try using high NODE_OPTIONS=--max-old-space-size? I noticed we started getting chunk load errors more often when I reduced CI machine size to a lower RAM class.

Nevertheless, I’ve given up on running a “dev server” on CI because that doesn’t make sense to me as I previously expressed. Why run a long running build process and be prone to potential memory errors and harder to debug problems? Webpack dev server stores all generated assets in memory with a lot of overhead due to internal stats. Running webpack build and then serving static files from a simple static http server is much simpler, or it would’ve been simpler if cypress didn’t pass spec name with a query parameter/header. Even then, serving them with an express server isn’t too much trouble. Still, I wish cypress team took this path instead of potentially fighting an uphill battle.

So my implementation of a “custom dev server” still uses high memory during build but significantly reduced memory usage during test run, so I can even use different machine sizes for building and for running and save $$$.

Example custom dev server:

const { createCustomDevServer } = require('cypress-ct-custom-devserver');

exports.prebundleDevServer = createCustomDevServer(async ({ serveStatic }) => {
  serveStatic('./build/cypress', '/__cypress/src');

  return {
    loadTest: async (spec, { loadBundle }) => {
      const testPath = spec.relative.replace(/.(js|jsx|ts|tsx)$/, '.js');
      loadBundle(testPath);
    },
    devServerPort: 8080,
    onClose: () => {
      compiler.close(err => {
        console.error(err);
      });
    },
  };
});

This requires running webpack beforehand with a config like:

module.exports = {
  entry: fs
      .readdirSync('src', { recursive: true })
      .filter(name => minimatch(name, '**/*.cy.{js,jsx,ts,tsx}'))
      .map(name => `src/${name}`)
      .reduce((acc, name) => {
        acc[name.replace(/\.(js|jsx|ts|tsx)$/, '')] = ['./src/cypress/support.ts', `./${name}`];
        return acc;
      }, {}),
  output: {
    filename: '[name].js',
    path: path.resolve('../build/cypress'),
    publicPath: '/__cypress/src/',
  },
 // ...rest of webpack config
}

Good luck everyone!

We noticed that service workers are leaking memory unless cypress is ran with experimentalSingleTabRunMode: true config.

We noticed that we get ChunkLoadErrors whenever CI machines run out of memory.

We don’t know what’s causing cypress to leak memory and whether if that’s on our side or not. We use msw, therefore there’s a chance that the service worker is leaking memory. Nevertheless, memory usage keeps increasing with no boundary and eventually hits 100% and crashes regardless of amount of memory, unless tests finish before being able to hit 100%.

Ah ha – we’re also seeing the error and using monaco! We’re using it via @monaco-react/loader which loads the webworkers (and the majority of the code) from a CDN, so it doesn’t get built via the webpack dev server by default. Not sure if monaco itself might be the issue, or if it’s just super heavyweight and triggers a race in webpack.

We don’t use React nor “monaco” (we use Angular) and we have the same issue with component tests with the ChunkLoadError… 🤔

We have been running into this for a while now in our component tests. Specifically with the monaco editor loading web workers.

Loading chunk vendors-node_modules_yaml_browser_index_js failed.

Our fix is to pin to Cypress version 13.5.1.

Cypress 13.5.1 works consistently. Upgrading to new versions after that, up to version 13.6.6 give us random failures in our component tests.

Looking forward to a fix.

Update: Still seeing the issue sometimes on version 13.5.1

Try this workaround, as I previously suggested:

// cypress.config.js

const { defineConfig } = require('cypress');
const { devServer } = require('@cypress/webpack-dev-server');

module.exports = defineConfig({
    ...
    devServer: async ({ specs, cypressConfig, devServerEvents }) => {
      const result = await devServer({
        specs,
        cypressConfig,
        devServerEvents,
        webpackConfig,
      });
      // prebuild files that can cause test failures when they take too long
      // during runs
      await Promise.all([
        fetch('http://localhost:8080/__cypress/src/cypress-support-file.js'),
        // prebuild any other file, such as service workers here 
      ]);
      return result;
    },
});

If other async chunks are causing middle-of-the-run builds, then try using https://webpack.js.org/plugins/limit-chunk-count-plugin/ and limit your chunk count to 1. Or try turning off chunk splitting, see https://webpack.js.org/plugins/split-chunks-plugin/#optimizationsplitchunks .

Again, I doubt this is a misconfiguration in webpack dev server. Async chunks are normal things that optimize initial bundle size by lazy loading things later. They are usually caused by async imports, but if you don’t have an async import on your side or if you don’t know what that even means, then it must be used by one of the libraries you use.

This is a problem with async chunks triggering builds and those builds not finishing before test times out.

The solution is one of:

  1. disable all async chunks (see https://webpack.js.org/plugins/split-chunks-plugin/#optimizationsplitchunks). Do a build with the same webpack config and make sure webpack produces a single js file.
  2. cypress’s dev server integration reads async chunks found by webpack dev server and trigger a build for them (with webpack compiler api or simply by triggering a fetch). Webpack dev server might find new async chunks in those async chunks, so they must be prebuilt too until no async chunk is left.
  3. cypress allows us to prebundle everything (e.g. do a build for all components) and use those prebuilt assets, e.g. by replacing webpack dev server with a simple http server that serves those static files.

Edit: typo

Hey all. Just a few updates.

Mid recompilation with webpack-dev-server has stopped with the update from v3 to v4, which is great! However, we are still hitting ChunkLoadErrors 🙁 .

This issue has been extremely challenging to reproduce, but we have a few theories. My guess is that there are chunk loading issues associated with the main bundle mainly due to us bundling every component test, regardless of what is used under test. The only exception to this is if the --spec option is leveraged to pass in specs. For example:

  • if cypress open is used to to develop a single component test and there are 39 other component tests, ALL 40 component tests are compiled when the app is launched.
  • If cypress run is used to test all specs, ALL component tests are compiled before the tests are initialized.
  • If cypress run is used to test multiple specs via the --parallel flag in CI, ALL component tests are compiled before the tests are initialized, regardless of what tests are run.
  • If cypress run is used to test a single spec via the spec argument, ONE single spec is compiled before the test is run.
  • If cypress run is used to test multiple specs via the spec argument, ONLY THE SPECS PASSED IN are compiled before the test is run.

The larger N is in this case (our example above was 40), the worse the dev server scales as far as building is concerned. Each spec is considered its own entry point in webpack. We don’t have proof that reducing the bundle size would fix the chunk errors, but I think this is in large part of the problem as this seems to happen mostly in projects where there are dozens of CT tests.

In most cases, even if the build takes minutes, cypress waits for the dev server to return a response before executing tests. @anilanar In other words, we are pre-bundling everything needed under test already but the assets are not loaded into the browser all at once, which might explain the service worker issue you are running into and I’m glad you were able to find a solution! Something is causing the chunk to fail to load meaning it either doesn’t exist, or it timed out loading. Since the assets are held in memory, it is difficult to figure out which has occurred. Either way, I am not convinced flaky tests are due to builds taking too long.

What we are trying internally with one of our projects where this error happens frequently, is to leverage the --spec option to group our tests under execution and use circleci to split the tests per container (@cwillsea-mtg this might be something you can try depending on your setup and would be great feedback for us if it lessens or resolve the issue for you). This will only bundle the assets needed under test, reducing the chunk and bundle sizes under test and likely the stress to load them into memory. This could give us direction on if we need to change some aspects of the dev server depending on whether ChunkLoadErrors continue to surface or not and our ability to tune them. I will report back when I gather more data/information.

My team is using the default dev server that comes with cypress:

  component: {
    devServer: {
      bundler: 'webpack',
      framework: 'next',
    },
  },

The errors seem to always happen at the start of a test file, but pop up in random places in our test suite. For example, here is a recent failure in the 73th test file out of a suite of 90 where the other 89 runs all passed (edited to remove component/file names):

  Running:  components/COMPONENT_NAME.cy.tsx                          (73 of 90)


  1) An uncaught error was detected outside of a test

  0 passing (665ms)
  1 failing

  1) An uncaught error was detected outside of a test:
     ChunkLoadError: The following error originated from your test code, not from Cypress.

  > Loading chunk vendors-node_modules_SOME_STRING.js failed.
(error: http://localhost:8080/__cypress/src/vendors-SOME_STRING.js)

When Cypress detects uncaught errors originating from your test code it will automatically fail the current test.

Cypress could not associate this error to any specific test.

We dynamically generated a new test to display this failure.
      at __webpack_require__.f.j (http://localhost:8080/__cypress/src/webpack.js:340:29)
      at <unknown> (http://localhost:8080/__cypress/src/webpack.js:137:40)
      at Array.reduce (<anonymous>)
      at __webpack_require__.e (http://localhost:8080/__cypress/src/webpack.js:136:67)
      at Object.load (http://localhost:8080/__cypress/src/main.js:485:577)
      at <unknown> (http://localhost:8080/__cypress/runner/cypress_runner.js:110819:21)
      at tryCatcher (http://localhost:8080/__cypress/runner/cypress_runner.js:1807:23)
      at Object.gotValue (http://localhost:8080/__cypress/runner/cypress_runner.js:6476:18)
      at Object.gotAccum (http://localhost:8080/__cypress/runner/cypress_runner.js:6465:25)
      at Object.tryCatcher (http://localhost:8080/__cypress/runner/cypress_runner.js:1807:23)


  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        1                                                                                │
  │ Passing:      0                                                                                │
  │ Failing:      1                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  1                                                                                │
  │ Video:        false                                                                            │
  │ Duration:     0 seconds                                                                        │
  │ Spec Ran:     components/COMPONENT_NAME.cy.tsx                             │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

The frequency with which these errors occur has been increasing as we add more test files. If there’s any debug/diagnostic config we can set to get more info happy to try as this issue is happening frequently in our CI

I’m leaving behind some notes after digging into this further.

We are experiencing this error on ~9% of our component test builds. (I can’t remember offhand how many tests that entails.)

Indeed error messages above are consistent with what we see, here’s an excerpt (with a slight edit on chunk name tbh):

  1) An uncaught error was detected outside of a test:
     ChunkLoadError: The following error originated from your test code, not from Cypress.

  > Loading chunk name1-name2-file~spec-11~spec-14~spec-15~spec-17~spec-22~spec-23~spec-25~spec-27~spec-28~43d59dbf failed.

It includes the localhost path as well on the next line (which I left out here).

Noteworthy…going up a few lines in the build, we are seeing something like this (I changed more file names out of caution):

  Running:  dir/myComponentSpec.spec.tsx                                                  
  Estimated: 1 second


ℹ 「wdm」: Compiling...
  (Attempt 1 of 3) An uncaught error was detected outside of a test
ℹ 「wdm」: wait until bundle finished: /<root+directory>/related-file~spec-34.js
ℹ 「wdm」: wait until bundle finished: /<root+directory>/related-file.js
  (Attempt 2 of 3) An uncaught error was detected outside of a test
  1) An uncaught error was detected outside of a test

It appears to me that the ChunkLoadError occurs because the next test is running before webpack compilation is completed on the devserver. I think this includes related CSS compilations, but honestly I don’t know if that’s all the time or a subset of examples.

Ultimately I’m convinced the next test needs to wait until compilation completes and the devserver has updated the cache. I attempted to set devserver settings around hot reload, etc, but those didn’t work – likely because the test is still trying to serve the page and is not waiting for the devserver to signal it’s up to date.

When it comes to waiting for compilation to complete, we do have logic around this in CypressCTWebpackPlugin in this function. As-written, it’s awaiting specs, and this is why I’m wondering if we need something to wait for related assets, too. Regardless, I think we need a way of ensuring all compilation completes, including any styling, before proceeding to the next test.

There had been a previous PR that leveraged the webpack devserver’s index HTML file, which I believe would yield the result I’m thinking of (change here). That PR was intended to fix a situation specific to MacOS version, but I’m optimistic it could resolve the ChunkLoadError. The PR was merged but the code in the file today is older than the PR, so I don’t know if it was rolled back at a later time.

We have been running into this sporadically when Component testing after migrating from CY 12.x to 13.6.2. Similarly to cases above, it happens towards the end of running a batch of suites (files) together.

Screenshot 2024-01-08 at 5 44 43 PM

Resorted to running cypress with DEBUG='cypress:*'. This is an instance of a crash:

 cypress: server: reporter got mocha event 'test'
with args: [u {
    title: 'An uncaught error was detected outside of a test',
    fn: [Function: o] {
        toString: [Function(anonymous)]
    },
    body: '() => {\n        throw err;\n      }',
    async: undefined,
    sync: undefined,
    _timeout: 2000,
    _slow: 250,
    _enableTimeouts: true,
    timedOut: undefined,
    _retries: -1,
    _currentRetry: 0,
    pending: false,
    type: 'test',
    duration: undefined,
    state: 'skipped',
    parent: p {
        title: '',
        ctx: {},
        suites: [],
        tests: [Array],
        pending: false,
        _beforeEach: [],
        _beforeAll: [],
        _afterEach: [],
        _afterAll: [],
        root: true,
        _timeout: 2000,
        _enableTimeouts: true,
        _slow: 250,
        _bail: false,
        _retries: -1,
        _onlyTests: [],
        _onlySuites: [],
        delayed: false,
        _events: [Object: null prototype],
        _eventsCount: 1,
        file: 'VALID_PATH_TO_FILE_DEDACTED',
        id: 'r1',
        type: 'suite'
    },
    id: 'r2',
    _testConfig: {
        testConfigList: [],
        unverifiedTestConfig: {}
    },
    order: 1,
    invocationDetails: {
        function: 'Object.getInvocationDetails',
        fileUrl: 'http://localhost:8080/__cypress/runner/cypress_runner.js',
        originalFile: 'http://localhost:8080/__cypress/runner/cypress_runner.js',
        line: 97377,
        column: 17,
        whitespace: '    ',
        stack: 'Error\n' + '    at Object.getInvocationDetails 
(http://localhost:8080/__cypress/runner/cypress_runner.js:97377:17)\n' + '    at Suite.addTest 
(http://localhost:8080/__cypress/runner/cypress_runner.js:145737:85)\n' + '    at Object.createRootTest 
(http://localhost:8080/__cypress/runner/cypress_runner.js:145872:21)\n' + '    at Object.normalizeAll 
(http://localhost:8080/__cypress/runner/cypress_runner.js:163115:17)\n' + '    at CDPBrowserSocket.<anonymous> 
(http://localhost:8080/__/assets/index-caf2649c.js:146264:46)\n' + '    at CDPBrowserSocket.on2 
(http://localhost:8080/__/assets/index-caf2649c.js:34575:11)\n' + '    at Emitter2.emit 
(http://localhost:8080/__/assets/index-caf2649c.js:34616:24)\n' + '    at http://localhost:8080/__/assets/index-caf2649c.js:37709:15'

    }
}] + 9 ms

There is a lot in the logs that makes it hard to share but please do let us know if we should be looking for or can share anything useful to help unraveling this.