core: [Bug report] Compiling & rendering too too too much slower than VitePress

@meteorlxy I have a deeper dig in to our code, pageData only changes during router navigation, but SSG doesn’t need those at all, so maybe it could be possible for us to stop loading the whole @internal/pageData, and just load that page Data instead during SSG, I think that will probably reduce memory usage and speed up SSG. Loading all page data through promise when rending any page surely has negative effects during SSG

I used to bring up the issue of VuePress scalability on its GitHub repository (https://github.com/vuejs/vuepress/issues/2689). In my project (https://github.com/openupm/openupm-next), I have around 3000 pages generated by a custom VuePress plugin’s onInitialized method. The API allows you to provide all pages through app.pages.push(page) instead of using a generator that yields items individually. The intentional design of VuePress leads to keeping everything in memory. With each page consuming approximately 1 MB of memory (the estimation could be totally wrong), the total memory usage for 3000 pages amounts to about 3 GB, which may not sound excessive.

My project was initially built with VuePress v1, but I encountered some difficulties during a partial migration to VuePress v2 using --max_old_space_size=14000. It’s worth noting that a significant number of efforts have been made to address such issues in VuePress v2, which drives my curiosity about the bottleneck.

For illustration, let’s consider @lslzl3000’s repository (https://github.com/lslzl3000/vuepressVSvitepress), which contains 914 markdown files, each having an average file size of 22KB. Conducting a quick test yielded the following results:

Profiling 1: 914 pages	Peak Memory	Time
vuepress dev	868 MB	10s
vitepress dev	209 MB	Instantly
vuepress build	12.2 GB + swap	Out of memory during compiling with vite
vitepress build	5.9 GB	1m59s

A few additional notes:

• Memory usage is manually captured using htop’s RES (resident size) column.
• The virtual machine (VM) has a total of 12 GB of memory, and all tests were run with --max_old_space_size=12000.
• The Git plugin has been disabled for vuepress.
• Vuepress offers more features with zero configuration compared to vitepress, including auto sidebar functionality.

Let’s double the page numbers by duplicating pages under the docs folder.

Profiling 2: 1828 pages	Peak Memory	Time
vuepress dev	1.3 GB	20s
vitepress dev	209 MB	Instantly
vuepress build	12.2 GB + swap	Out of memory during compiling with vite
vitepress build	8.6 GB	4m51s

My major takeaways are two points:

1. Vuepress build costs much more memory than Vitepress. But it’s comparing apples and oranges for different feature sets.

2. The memory usage of both Vuepress and Vitepress grows almost linearly with the number of pages. For vitepress, it costs 4.8 MB/page.

That is a bit counterintuitive - I assume the build time grows linearly with the number of pages, not the memory usage. The behavior is not friendly with large websites that rely on cloud-based build services (GitHub actions for example). We can suffer with more build time, but a large memory VM is usually very expensive or unnecessary for generating a static website with thousands of pages, think about a 10K page website that requires a 48 GB memory VM.

After reading https://github.com/vuepress/vuepress-next/issues/1262’s proposal, it seems hard to improve:

@Mister-Hope:

During the build process, this cannot be done, webpack and vite all need to load and analyze all codes to pack the application, this means the more you put inside a project, the more codes they need to handle, so the compiling stage needs a linear space of memory comparing to content. These memory are taken by bundler, and it’s not quite related to VuePress itselfs, unless we do not require a bundler to pack our spa anymore.

On the vite community it’s also a hot topic: https://github.com/vitejs/vite/issues/2433, I’m processing it slowly and hopefully find some clues.

It seems a pitfall you’re going hit when you grow to a larger scale, and the solution is not very obvious here.

Refs #994, #1262

Updates here, we have a new pr which avoids heavy cost on router.resolve

https://github.com/vuepress/vuepress-next/pull/1433

This would speed up build and webpaage performance, but little on memory cost.

@favoyang Thanks for your investigation.

Some problems of performance IMO:

1. Transforming all md files to vue files and collect page data in prepare stage, and store page data in memory. This is what make our dev server start slowly and consume more memory. If we change to only transform md -> vue on demand when visiting the page, it would be much faster in dev. But the drawback is we’ll lose our current auto sidebar functionality.

2. vue-router might be the killer of build. According to #1353, vue-router is too expensive for static site. That should be the reason why VitePress implements a custom lightweight router instead. If we change to use a lightweight router, it might be much faster in build.

3. SPA might not be the solution of static site, and that’s why we have to bundle all files together and causing linear memory usage increase in build.

---

I’ve been wanting to try these things for a while, but don’t have enough time. TBH it should be faster after those changes, but it would also make VuePress more like VitePress. Thus I also don’t have much motivation to do so. 😞

I have put a lot of effort into this project, although it is not perfect. I am grateful to have you who are still using this project. ❤️

I also conducted a quick test on @hustcer’s repository (https://github.com/nushell/nushell.github.io), which comprises 676 markdown files, each with an average file size of 6 KB.

Here are the profiling results for the test:

Profile	Pages	Peak Memory	Time
vuepress build	676 pages	1.7 GB	1m16s (with Git plugin on)

The outcome is entirely acceptable. The primary distinction when compared with lslzl3000/vuepressVSvitepress is that the average markdown file size of nutshell.github.io is significantly smaller (22KB vs. 6 KB).

This implies that before delving into vite bundle options, the most immediate area to address is reducing the overall project size. While this approach may not be suitable for a purely markdown-based VuePress site, it might prove beneficial for some dynamically generated pages. One potential strategy could be deferring the loading of heavy data to runtime, such as fetching a JSON file from the public folder.

@meteorlxy I have created a simple repo to test VitePress and VuePress https://github.com/lslzl3000/vuepressVSvitepress

It has about 1000 .md files, just simple doc generated from babylon.js you can try build with VitePress and VuePress

1. You will see how much longer the VuePress will take
   In my Macbook Pro 2022 with M2, VuePress: 13mins+ vs VitePress：60-70s

2. VuePress also use much much more memory than VitePress It has to build with NODE_OPTIONS=--max_old_space_size=20480, otherwise node process will exit by JavaScript heap out of memory

If build with full VuePress theme and other plugins, it will take more time