jest: Jest performance is at best 2x slower than Jasmine, in our case 7x slower

Same issue here on 25.2.2, file resolution takes too long. Is there any plan to speed it up?

Just updated my benchmarks with a new player in town – Vitest. I have good news. It has a compatible API to Jest but in my benchmarks it ran 2x faster than Jest and even outperformed Jasmine. 😮

I’m going to try migrating a larger real world codebase to it early in the new year and report back on the experience for those curious.

i was trying to do migration from mocha to jest… and… mocha is finishing all tests before jest starts first one… i think there is somewhere issue with resolving/reading files -> my project contains ~70k files, and i’m running ~19k tests.

after some digging its looks like jest is trying to import all files from all folders before he starts tests, i’m providing explicit match for test file: testMatch: ['<rootDir>/dist/alignment.spec.js'].

i was able to run tests by adding to jest.config

modulePathIgnorePatterns: ['<rootDir>/fixtures/.*'],

but it’s still 11m… as opposed to mocha ~1m and without test framework (try/catch assert) ~40-50s

turning off transformation helped to

transform: {}

so far my configuration looks like this:

module.exports = {
  testEnvironment: 'node',
  testMatch: ['<rootDir>/dist/alignment.spec.js'],
  moduleFileExtensions: ['js'],
  transform: {},
  modulePathIgnorePatterns: ['<rootDir>/projects/.*', '<rootDir>/node_modules/.*']
};

its still slow, ~4min

now i’m looking for way to turn of prettier, i don’t care about formatting errors…

To “fix” imports overhead, I’ve written custom test runner. It’s using posix fork() to clone processes (more docs are in link) In our case, we reduced our test run from 18 to 4.5 minutes, from which 1 minute is warmup and could be speed up by moving to swc/esbuild.

https://github.com/goloveychuk/fastest-jest-runner

Am I right in saying the problem is that jasmine loads all specs into one process and runs it, where as jest creates a new mini-environment per test suite? We see exactly the same issue and profiling seems to show a significant amount of time resolving files and parsing javascript - unfortunately the multi-core aspect can’t make up for this. I have no idea why resolving is so slow. We made significant speed increases by trying to make suites import the least number of files, but we’ve hit a wall on going further in that direction as we in many cases want to test multiple components running together and not to mock every dependency. I planned to do some more profiling and it would be great if anyone on the core jest team can point in any directions to things they would like to see.

@goloveychuk Interesting idea, but your solution didn’t seem to make a significant difference in my benchmark. 😢 I’ve added it to https://github.com/EvHaus/jest-vs-jasmine/.

Native Jest

Your approach

Jasmine (for comparison)

I’ve updated my repo with the latest benchmarks, latest version of Jest, latest version of Node and a more reproducible benchmarking tool (via hyperfine). Overall, I’m still seeing Jest performing at least 3x slower than Jasmine. So nothing has really changed since the original post.

FYI: I’m not complaining. Just want to ensure those subscribed to the thread know that no significant advancements have been made here yet in the latest versions.

Hey folks, I’ve done an investigation run on my own with a no-op test and a lot of imports (requireModuleOrMock() is being called ~12500 times!). Most of my files in this test are TypeScript.

Ignoring jest init time (by strictly measuring the 2nd test of a jest --watch ...), this no-op test takes ~1.5s. Here’s what I’m seeing that’s causing that:

• ~625ms is spent doing getModuleID() which does some expensive FS work to find the absolute location of the module - iteratively check dirs for package.json, check if there’s aliasing properties in it (resolveExports), then find the actual module itself, and do some resolve calls as well. Since getModuleID() is called once per module (`~26000 calls), these FS operations add up.
• ~450ms is spent in _execModule() (excluding the actual invocation of the module, of course).
  • ~60ms is spent in transformFile: read the file, hash it, check if hash matches local cache
  • ~350ms is spent in createScriptFromCode: I think this is Node VM shenanigans requiring a bunch of work to happen on the script before it can be interpreted “for real”.
• There’s roughly ~400ms leftover, but I think that it can be explained by interpretation time of the imported modules themselves - there may be other wins in here, but they’re going to have depreciating returns.

So, a couple recommendations for things to look into next:

1. Perhaps we can extend file-watching to be more intelligent in getModuleID to see if a file’s been changed? Of course, if file watching isn’t available (no watchman, etc), then fall back to the current slow mode
2. It might be worth hashing and storing information about package.json aliasing, so we don’t need to load and parse it from scratch every time. Of course, this would still need to be invalidated if the package.json is changed (hopefully file watching can help us with this)
3. As we build an understanding of the file system, we might be able to get more clever to avoid some FS work (e.g. if we know that package.json doesn’t exist at <subdirectory> in a previous import, perhaps we can avoid a stat when importing a future module. Maybe this is already happening and I missed it 😃
4. Can we reuse createScriptFromCode output between test invocations somehow? I wonder if it is Node-VM-specific (so, each time we create a new sandbox, we need to createScriptFromCode again).
5. Smaller thing: we initialize our hastemaps and watchman, even if we aren’t in --watch mode. Maybe this can be skipped?
6. Would it be possible to remove the amount of resolve()/realpath/general FS operations in getModuleID()? Perhaps some of them are redundant 🤞
7. (Inspired by a workaround discovered by my colleague, Patrik) in watch mode, we can probably lean on mtime to determine if a file is changed on re-runs. This sidesteps hashing all input files, which is a big win.

If I have time, I may be able to dig into some of these potential perf-gain options in the next few months, but no guarantees. I wanted to brain-dump here in case any other Jesters and Fools got inspired 😃

(note that I do have some low-hanging-fruit PRs that I’ll be upstreaming, but none of them address the remaining code hotspots mentioned above).

Things we’ve done to increase the performance of jest in our setup:

1. Changed the reporter to not verbose and a dot reporter. For us this 2.5x speed increases
2. Implemented our own custom resolver and instead of just caching imports with a cache key of the current directory and the import, cache based on current directory only if its relative or within node_modules, otherwise cache globally no matter what folder we are in - this appeared to save about 10% for us
3. reduce the files imported per suite - for instance even a static import of ten json files across every suite, when removed saved several seconds. Removing some lazy imports where too much was imported took some suites from 10 seconds to 5 seconds.

I think it’s a fair assumption to say it’s the module resolution that’s taking time. While require('foo'); is an in-memory cache lookup for jasmine (after the first one), every single test file in jest will have to do full resolution, and execution, of foo and all its dependencies. I doubt it’s the resolution itself that takes significant time (we should have the fs in memory (after the first run, at least)), but executing the files probably takes up a significant chunk of time.

Another difference is that jest executes your code inside the jsdom vm, while with jasmine you’ve just copied over all the globals to the node runtime (https://github.com/jsdom/jsdom/wiki/Don’t-stuff-jsdom-globals-onto-the-Node-global), which will always be quicker as you skip an entire abstraction layer (https://nodejs.org/api/vm.html).

That said, I agree it’s really not ideal (to put it mildly) that Jest is about twice as slow as jasmine. I’m not really sure what we an do, though. We could try to cache the resolution (although we’d still have to run through the entire tree in case there’s been any module mocking) which might allow us to not resolve modules by looking around, but again the FS should be in memory, so I doubt it’d have much impact.

@cpojer @mjesun @aaronabramov @rickhanlonii do you think there’s anything clever we can do here? Or any awesome ways of profiling what we spend our time on?

Also, thank you so much for setting up a great reproduction case @EvHaus!

I have simular performance issues, our tests are running at least 5x slower.

Mocha takes one second. Jest takes 12 seconds. So I removed the Jest from my project.

I doubt it’s the resolution itself that takes significant time (we should have the fs in memory (after the first run, at least)), but executing the files probably takes up a significant chunk of time.

I did some profiling of the node processes while running Jest on my projects it seemed like requiring was one of the most time consuming tasks. At least that was the case on Windows (not WSL), which I found to be substantially slower than Linux, especially in watch mode. Granted, I’m not particularly confident in my understanding of the node profiler’s output, but that’s what it looked like. I saw the same thing with this reproduction.

Any news on this?

@EvHaus yea, I think it won’t have a difference in this benchmark. More info about my setup/project.

1. we had leaks
2. 600 test files, 8000 tests total
3. many dependencies - createScriptFromCode is called for 8000 unique files.
4. number of resolved files - 20000.
5. memory used after test file - up to 1gb. (per each worker). If don’t clean - up to 2gb (node max old space)
6. workers=8.
7. pc - amd 3700x, 8/16 cores, 16gb memory.
8. 26.6 jest, jasmine test runner.

In this setup I have 531s default, and 226s with above optimisations. Most important trick is to clean memory before each test file, and it will help only if you’re using many workers and your tests are taking much memory. In my example - 8 workers * 2 memory, with 16gb total, system is going to swap. And you have this “effect”, when at start jest is running pretty fast, and after several test files it’s slowed down. If you have such a symptoms - mb gc clean will help you.

So answering on your comment - those optimisations could help on real world heavy projects, it cannot make jest same speed as jasmine, since jest have expensive runtime (think of all those features/overhead: mocks, transformers, reporters, error formatting, tests isolation, caching etc)

also interesting is this, watch mode is three times slower than non watch mode even with the same amount of workers. (35s vs 11s)

tracked it down to the passing of to rawModuleMap in _createParallelTestRun of jest-runner, it seems like not passing the rawModuleMap is faster for some reason, note that in my case, test.context.moduleMap.getRawModuleMap() always returns { duplicates: {}, map: {}, mocks: {} }

I was intrigued by the 2.5x speed increase mentioned from using a dot reporter, so I gave it a go.

Added verbose: false and reporters: ['jest-dot-reporter'] to the config. On our giant main repo it only offered about a 15% performance improvement (260s instead of 300s to run all tests). That’s small but something. And on the test repo it didn’t seem to make any difference at all (probably because it doesn’t have enough specs for the reporter change to make an impact).