@pmdarrow I’m investigating switching to vm.runInNewContext from vm.runInThisContext. This, hopefully, will allow us to drop costly isolation using Workers. We will essentially have the same isolation as Jest.

But I did not have time to work on this recently (there is the war going on)

So is the summary that we’re stuck with the performance where it’s at right now, unless we want to sacrifice isolation? If so, I think this should be published in the docs somewhere. I know the docs say disabling isolation can improve performance, but it doesn’t communicate that comparable tools (i.e. Jest) are 3x faster even with isolation. It would help to know up front why vite is slower by default and what is blocking it from becoming faster (vm module bugs?)

Just a heads up. The next Vitest version should enable profiling workers (thanks to @AriPerkkio in #2702), so it might be easier to find the problem. In the meantime you can enable a few environmental variables to debug performance:

• VITE_NODE_DEBUG_RUNNER=true will log modules that take a lot of time to load
• DEBUG=vite-node:* will show resolve and load time
• VITE_NODE_DEBUG_DUMP=load enables debug cache (which is PERSISTENT between runs, so don’t use it for your regular tests), so you can see what files were transformed and how they look like

My team in the process of migrating a large test suite (6073 tests) from jest to vitest. With jest, the full suite runs in 3m47s. With the latest vitest (0.18.0) it runs in 10m7s. We really want to continue with vitest but this kind of a slow down is making it challenging. We don’t want to disable isolation, as it’s crucial for reliability in a test suite this large. Is there any plan to improve the performance vitest, or is jest doing something fundamentally different that vitest won’t be able to replicate? @antfu @sheremet-va @Demivan

I trimmed a bit locally. Flow + Less is a bit less common tbh, but yes with your tests I can confirm it surfaced some bottleneck of Vitest, where I am still investigating.

Has there been an attempt to improve Vitest’s performance? I also experienced slower test-performance with Vitest compared to Jest (about 1.5x slower) as documented here. It would be unfortunate to see this project fail after being heralded as superior to the existing options.

SUMMARY

I’ve written up a summary of all the advice given in this GitHub Issue as a time saver for those coming here in the future.

Improving Vitest Performance

@antfu is there anything that we could do to help move this forward?

I tested on my end with a different setup.

Many test files (5000) 1 test per test file

Repo: https://github.com/aurelienbottazini/test-js-test-runners

Jest finishes in 70 seconds on my computer Vitest is very very slow (i end up canceling the task, tired of waiting) and test are progressively running slower and slower.

toward the end i even see slowness reports/warnings for each spec:

Running with --isolate=false is also very very slow --isolate=false --single-thread=true is also very very slow

@AriPerkkio Really great to see the performance improvements from https://github.com/vitest-dev/vitest/pull/3006. Moving the 5000 test above from 5667.68s to 4.63s is huge! Thanks for the effort on that.

Please, to all the people who basically just add “+1” - stop. A lot of people are following this thread, and it doesn’t help anyone, including maintainers.

If you have real examples, please provide it. Otherwise it’s just an annoying notification.

@EvHaus @antfu that’s not an apples-to-apples comparison. Jest has fully isolated tests, so setting isolate: false is an unfair comparison. And it’s clear most devs want their tests isolated.

FYI – I updated my benchmarks to get rid of Flow, LESS and switched to more modern ways of doing things: TypeScript, Testing-Library, etc… and that had a drastic impact on the results.

Vitest is now consistently the fastest, with Jasmine just behind and Jest still consistently 4x slower than both (details).

NOTE: I’m using isolate: false in Vitest to ensure it performs at its best.

Summary
  'yarn workspace vitest test' ran
    1.17 ± 0.01 times faster than 'yarn workspace jasmine test'
    4.64 ± 0.03 times faster than 'yarn workspace jest test'
    4.64 ± 0.06 times faster than 'yarn workspace jest-swc test'
   10.11 ± 0.06 times faster than 'yarn workspace fastest-jest-runner test'

What this tells me is that there’s some other external factor at play here. Perhaps a third-party library, or an extra preprocessor, or something.

Also interestingly, swc had 0 effect on the performance of jest in my case.

Ok, I can have a look at this. Can anyone set up a minimal repo (2~3 test files, or the less possible) of how the slowdown is being observed (as I personally do not face such a slowdown when doing unit testing or testing Vue)? Thanks

Is there anyway of debugging vitest performance? Maybe list what takes time on each run, or which dependencies are loaded? After switching a React Testing Library application (built with Vite) from Jest to Vitest, the test time CI (Azure DevOps) went from around 50s with code coverage using Jest, to 2 minutes without code coverage in Vitest. It could very well be related to dependencies we are using (like Material UI), but it’s really hard to debug.

Vitest 0.34.0 is released with the new experimentalVmThreads flag which should make tests run as fast as they do in Jest. If you have any issues with the new pool, please open a separate issue instead of commenting here.

After investigating it more and checking with @sheremet-va i don’t think it can be implemented with current state of vm module. Fixing all the issues will mean getting all Jest bugs too.

Lets hope ShadowRealm implementation will be better then vm module.

SUMMARY

I’ve written up a summary of all the advice given in this GitHub Issue as a time saver for those coming here in the future.

Improving Vitest Performance

--no-threads is outdated. If you want your tests to run in sequence without firing up new workers use --single-thread

--no-threads now uses child_process instead of workers, so your code might run even slower with it.

Any feedback concerning this flag should be published via new issues, NOT inside this issue. Thank you.

@marpro200 @dmitriybo @sheremet-va lets continue this specific topic (“ViteNodeServer: ${id} not found. This is a bug, please report it.”) in #2940

@cbxp User time will be higher because Vitest is multi-threaded. User time is the time processor spent running a process. If you have multiple cores, user time will be higher for multi-threaded apps. Because it aggregates all the work that all cores did.

I also noticed that after moving from Jest to Vitest, my tests were taking 2-3x more time to run.

After doing some experiments lately, one thing that drastically reduced the time to run all specs was to disable experimental fetch (NODE_OPTIONS='--no-experimental-fetch' yarn test). We currently use whatwg-fetch.

Before: 126.28s After disabling experimental fetch: 53.05s

Same thing for me.

I just finished migrating a project with 3,000+ tests (in 450+ test files).

With Jest, it takes 30 to 40 seconds. With Vitest, it takes 130 to 150 seconds.

Roughly 3 to 4 times slower, which is a bummer, because we’ll probably have to abort the migration at this point.

Vitest runs tests 3x slower than Jest

That is an incorrect statement. It really deps on what and how your tests are written. You can’t say a man walks faster than cars without the context (in a traffic jam or something it might be true). The reason why this is happening commonly due to you are using some giant dependencies, where in isolation, they will be executed multiple times for each tests. Vitest is on-demand like Vite, while Jest will bundle the code, so in some scenario it would be more efficient than Vitest.

While indeed there is some space for optimization like pre-bundle the deps, or waiting for vm to support ESM. But in general, it’s still trade-offs between speed and correctness. For performance and also for good practice, I would suggest to make the tests as “unit” as possible, or mocking dependencies whenever you can.

Running with --isolate=false is also very very slow

@aurelienbottazini this reproduction case seems to hit performance issue related to test reporter. See https://github.com/vitest-dev/vitest/issues/2602.

If you use --isolate false --reporter tap in your reproduction case the tests will execute fast.

~/x/test-js-test-runners/vitest  $ time npm exec vitest -- run --reporter tap --isolate false

TAP version 13
1..5000
ok 1 - test/sum.0.test.js # time=2.00ms {
    1..1
    ok 1 - adds 1 + 2 to equal 3 # time=0.00ms
}

...

ok 5000 - test/sum.999.test.js # time=0.00ms {
    1..1
    ok 1 - adds 1 + 2 to equal 3 # time=0.00ms
}

real    0m3.530s
user    0m7.545s
sys     0m1.565s

Thanks for the great advice @sheremet-va 🙇. Aliasing element-plus to a dist build(for example element-plus/dist/index.full.mjs) instead of element-plus/lib/index.js sped up things even further: ~7.30s (compared to the initial ~18.66s)

On a bigger real world project (couple hundred test files with couple thousand tests… ~30% of those using element-plus)… I managed to shave off 25s (~112s initially, 87s now ). And on top of that, switching to Node 18 brought it down to 70s. And the upcoming “~400 ms for each file” improvement sound brilliant! 🥳

I was actually tried @AriPerkkio’s PR and did some profiling before I came here but couldn’t quite understand what’s going on 😃 https://github.com/vitest-dev/vitest/pull/2702 and https://github.com/vitest-dev/vitest/pull/2721 - great stuff guys 🙌 Can’t wait to get my hands on the next release.

Speaking from experience, enabling isolate: false in a large test suite will cause all sorts of unexpected behaviour and flaky tests. In theory, it’s possible, but if you add more than a couple developers on a project, it will get out of hand pretty quickly.

Like @thebuilder and @pmdarrow we have also experienced a 3x slowdown of tests after migrating to vitest from jest, our large CI unit test suite now takes 15 mins instead of 5 when running on GHA.

If a test that takes 30 seconds locally takes over 30 minutes on your server, the first thing you need to check is that the test is written correctly. There seems to be something causing an infinite loop or timeout.

After your comment, I realized there was a test with using vi.useFakeTimers, but it’s strange that I didn’t have any issues locally.

I ended up being able to make it work without any changes to the tests, using the option threads: false thanks to the issue I mentioned earlier.

If a test that takes 30 seconds locally takes over 30 minutes on your server, the first thing you need to check is that the test is written correctly. There seems to be something causing an infinite loop or timeout.

Why not import jsdom or happy-dom once and reuse for all test files in need of it?

@adrian-gierakowski

Yep, it depends on the environment. You can check out my previous comment https://github.com/vitest-dev/vitest/issues/579#issuecomment-1406519409

I’ve noticed that one of the reasons why there is a difference between Jest and Vitest is the fact that Vitest imports jsdom for each test file which takes a lot of time (300ms on my M1 Mac to simply import jsdom). So one of my recommended solutions is to try to use happy-dom, which takes 3 times less to import.

0.29.0 is out. This release adds deps.experimentalOptimizer option. Please, try this flag if you have any performance issues. Don’t forget to enable threads, if you used --no-threads before!

Any feedback concerning this flag should be published via new issues, NOT inside this issue. Thank you.

I would highly argue that this flag is a solution vitest is still simply slow. Our previous setup with jest was taking around 30-40s and here, we’re facing over 200s just to finish the same tests - and we don’t really have any huge dependencies besides vue, vuex and moveable. We have around ~1500 tests to be run in 360 files, where we use snapshots & expectations. We really were pretty excited about moving onto vite with vitest but this seems simply unrealistic.

What’s even more funny in this case -> with the flag enabled the tests are running even slower 😃 The results with this flag on/off presented below.

With experimentalOptimizer: true -> Duration 218.01s (transform 23.50s, setup 47.16s, collect 1850.54s, tests 48.56s) With experimentalOptimizer: false -> Duration 210.45s (transform 21.67s, setup 44.51s, collect 1764.43s, tests 49.67s)

0.29.0 is out. This release adds deps.experimentalOptimizer option. Please, try this flag if you have any performance issues. Don’t forget to enable threads, if you used --no-threads before!

Any feedback concerning this flag should be published via new issues, NOT inside this issue. Thank you.

Perhaps element-plus is a barrel import(?) and importing one component imports the whole library? (even though tree-shaking works for builds, it slows down vitest significantly)

Yes, when you import the component, it imports the whole library. Vitest also needs to do this 100 times for each test file. Excluding it from inline might help. Also, it might be a good idea to alias element-plus to its CJS version, since this is what jest is importing. My tests show ~10 seconds improvements with this alias (inlining disabled):

      {
        find: /^element-plus$/,
        replacement: resolve(
          __dirname,
          "node_modules",
          "element-plus/lib/index.js"
        ),
      },

What does the “collect” duration in the Vitest results mean?

Basically, it means the time it took for Vitest to do this:

await import('/path-to-test-file.js')

There is also a bottleneck for just importing vitest (~400 ms for each file), which should be fixed in #2721.

After investigating it more and checking with @sheremet-va i don’t think it can be implemented with current state of vm module. Fixing all the issues will mean getting all Jest bugs too.

Lets hope ShadowRealm implementation will be better then vm module.

Do https://github.com/justjake/quickjs-emscripten will resolve this?

At my last job we used Jasmine (which I believe doesn’t have isolation) with a huge number of UI test suites, and more recently we’re using Vitest with isolation off. We’ve never had any issues or flakiness or cases where we pick up DOM elements from previous renders.

AFAIK, Jasmine doesn’t even have watch mode, so I’m guessing you are comparing/referring to full runs with isolation vs full runs w/o isolation.

In watch mode it’s easy to introduce some unwanted state/mocking or polluting global/process state to the nature of various incremental changes (that are not always good or final). Without proper isolation watch mode becomes unusable and unreliable, it’s not even a choice between isolation or performance it’s between working as expected vs restarting watch mode every few minutes.

@EvHaus As you can see from the benchmark above, there was a performance problem when isolate was true.

• https://github.com/vitest-dev/vitest/issues/579#issuecomment-1154571206
• https://github.com/vitest-dev/vitest/issues/579#issuecomment-1197227773

Glad to hear that, thanks for the update @EvHaus!

I think there is indeed some more space for us to optimize further.

@dmLinker @pmdarrow We also want to use vitest as our modern test tool but for now we are using @swc/jest with good performance gain (3x)

Same experience here. Jest ran my test-suite in about 3-5 minutes, after migrating to vitest it now takes 10-17 minutes 😞

I’m sure it would be a lot faster if I disabled threads, but it’s not worth dealing with all the potential issues that comes with that.

I do have .less files in my project as well, but they are not called .module.less so I’m not sure if that would affect anything?

@phifa I don’t think that’s related to this discussion - that’s just vitest using as many cores as possible. If you don’t want this behaviour, you can turn it off: https://vitest.dev/guide/features.html#threads.

My team in the process of migrating a large test suite (6073 tests) from jest to vitest. With jest, the full suite runs in 3m47s. With the latest vitest (0.18.0) it runs in 10m7s. We really want to continue with vitest but this kind of a slow down is making it challenging. We don’t want to disable isolation, as it’s crucial for reliability in a test suite this large. Is there any plan to improve the performance vitest, or is jest doing something fundamentally different that vitest won’t be able to replicate? @antfu @sheremet-va @Demivan

You can get speed boost by disabling isolation. Bear in mind tho, that it will make globalThis object sharable between tests.