eleventy: large data sets: 1.x has issues that 0.11.1, 0.12.1 do not

Describe the bug I’ve a semi-open site generator project that squishes gigabytes of data sources down to about 6.9k pages to be processed by eleventy in two ways:

the legacy markdown repo
the more up-to-date json data source (pagination ftw!)

0.11.1 handles the 6.9k documents & 15.6mb json file without issue, 1.0.0 falls over in a similar fashion to that described in #695

To Reproduce The site generator is semi-open in that the source is available at https://github.com/Satisfactory-Clips-Archive/Media-Search-Archive, but it’s not feasible to stash 2.7gb+ of source data into the repo, so the repro steps aren’t readily reproduceable by anyone that doesn’t have the data set.

While the method mentioned in #695 of specifying --max-old-space-size does move the goalposts somewhat, it still falls over with 8gb assigned.

Steps to reproduce the behaviour:

npm run build or ./node_modules/.bin/elevent --config=./.eleventy.pages.js
watch & wait

Expected behaviour 1.x to handle 6.9k markdown documents or 6.9k json data file entries as reliably as 0.11.x does

Screenshots

<--- Last few GCs --->

[16544:000001D55D359230]   168029 ms: Mark-sweep 4038.5 (4130.3) -> 4024.3 (4130.3) MB, 3347.8 / 0.0 ms  (average mu = 0.138, current mu = 0.017) task scavenge might not succeed
[16544:000001D55D359230]   171431 ms: Mark-sweep 4039.7 (4131.5) -> 4025.8 (4132.0) MB, 3345.7 / 0.0 ms  (average mu = 0.081, current mu = 0.016) task scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 00007FF70641DF0F v8::internal::CodeObjectRegistry::~CodeObjectRegistry+113567
 2: 00007FF7063AD736 v8::internal::MicrotaskQueue::GetMicrotasksScopeDepth+67398
 3: 00007FF7063AE5ED node::OnFatalError+301
 4: 00007FF706DA0CAE v8::Isolate::ReportExternalAllocationLimitReached+94
 5: 00007FF706D8B2FD v8::Isolate::Exit+653
 6: 00007FF706C2EC5C v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1468
 7: 00007FF706C3AC57 v8::internal::Heap::PublishPendingAllocations+1159
 8: 00007FF706C37C3A v8::internal::Heap::PageFlagsAreConsistent+2874
 9: 00007FF706C2B919 v8::internal::Heap::CollectGarbage+2153
10: 00007FF706BDC315 v8::internal::IndexGenerator::~IndexGenerator+22133
11: 00007FF70633F0AF X509_STORE_CTX_get_lookup_certs+4847
12: 00007FF70633DA46 v8::CFunctionInfo::HasOptions+16150
13: 00007FF70647C27B uv_async_send+331
14: 00007FF70647BA0C uv_loop_init+1292
15: 00007FF70647BBAA uv_run+202
16: 00007FF70644ABD5 node::SpinEventLoop+309
17: 00007FF706365BC3 v8::internal::UnoptimizedCompilationInfo::feedback_vector_spec+52419
18: 00007FF7063E3598 node::Start+232
19: 00007FF70620F88C CRYPTO_memcmp+342300
20: 00007FF707322AC8 v8::internal::compiler::RepresentationChanger::Uint32OverflowOperatorFor+14488
21: 00007FFB71217034 BaseThreadInitThunk+20
22: 00007FFB71402651 RtlUserThreadStart+33

Environment:

OS and Version: Win 10, running tool via git bash
Eleventy Version: 1.0.0

Additional context

6.9k markdown files (still falls over if the path is excluded due to the json file)
6.9k objects in a pretty-printed json file, totalling 15.6mb spread over 280k lines

About this issue

Original URL
State: open
Created 2 years ago
Reactions: 3
Comments: 31 (7 by maintainers)

Most upvoted comments

On another topic: do you have any tips on importing this amount without breaking? Is an external database a better idea?

Thank you!

The most terrible option would be to duplicate templates & split the data up.

SignpostMarv on Jul 20, 2023

Howdy y’all, there are a few issues to organize here so just to keep things concise I am opening #2360 to coordinate this one. Please follow along there!

zachleat on May 3, 2022

/summon @zachleat Possible performance regression between 0.12.x and 1.0.

Thanks @SignpostMarv, I’ll try fetching the ZIP file from https://github.com/11ty/eleventy/issues/2226#issuecomment-1059791740 and see if it will build on my laptop locally (disclaimer: it’s a higher end M1 MacBook Pro, so results may differ).

@esheehan-gsl How complex is your content from your database? (is it Liquid or Markdown? etc) I’ve toyed with creating a “11ty lorem ipsum” blog generator in the past which just creates X pages based on some global variable so I can poke at performance issues like this w/ bigger sites. But sometimes it comes down to more of how many other filters and plugins and the general complexity of the site instead of just 600 pages vs 6000 pages (which can be frustrating).

pdehaan on Mar 30, 2022

<--- Last few GCs --->

[16756:0000024C7C0F9B80]   144598 ms: Mark-sweep (reduce) 4067.7 (4143.4) -> 4067.1 (4143.9) MB, 7589.3 / 0.0 ms  (average mu = 0.141, current mu = 0.001) allocation failure scavenge might not succeed
[16756:0000024C7C0F9B80]   151443 ms: Mark-sweep (reduce) 4068.3 (4144.1) -> 4067.8 (4144.6) MB, 6831.9 / 0.1 ms  (average mu = 0.080, current mu = 0.002) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 00007FF70641DF0F v8::internal::CodeObjectRegistry::~CodeObjectRegistry+113567
 2: 00007FF7063AD736 v8::internal::MicrotaskQueue::GetMicrotasksScopeDepth+67398
 3: 00007FF7063AE5ED node::OnFatalError+301
 4: 00007FF706DA0CAE v8::Isolate::ReportExternalAllocationLimitReached+94
 5: 00007FF706D8B2FD v8::Isolate::Exit+653
 6: 00007FF706C2EC5C v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1468
 7: 00007FF706C2C151 v8::internal::Heap::CollectGarbage+4257
 8: 00007FF706C29AC0 v8::internal::Heap::AllocateExternalBackingStore+1904
 9: 00007FF706C464E0 v8::internal::FreeListManyCached::Reset+1408
10: 00007FF706C46B95 v8::internal::Factory::AllocateRaw+37
11: 00007FF706C5AB7A v8::internal::FactoryBase<v8::internal::Factory>::NewFixedArrayWithFiller+90
12: 00007FF706C5AE63 v8::internal::FactoryBase<v8::internal::Factory>::NewFixedArrayWithMap+35
13: 00007FF706A689A6 v8::internal::HashTable<v8::internal::NameDictionary,v8::internal::NameDictionaryShape>::EnsureCapacity<v8::internal::Isolate>+246
14: 00007FF706A6E88E v8::internal::BaseNameDictionary<v8::internal::NameDictionary,v8::internal::NameDictionaryShape>::Add+110
15: 00007FF70697AE68 v8::internal::Runtime::GetObjectProperty+1624
16: 00007FF706E33281 v8::internal::SetupIsolateDelegate::SetupHeap+513585
17: 0000024C0028643A

$ ./node_modules/.bin/eleventy --version
1.0.1-canary.3

SignpostMarv on Feb 18, 2022

that completes as expected, although I haven’t diffed the output to see if there are any changes/bugs etc.

So, i think you’re saying: ✔️ 0.11.1
✔️ 0.12.1 ❌ 1.0.0 ❌ 1.0.1-canary.3

Doubting this has already been fixed in 1.0.1-canary builds, but if you were looking to try the sharpest of cutting edge builds, you could try <kbd>npm i @11ty/eleventy @canary</kbd>. 🔪

npm info @11ty/eleventy dist-tags --json

{
  "latest": "1.0.0",
  "beta": "1.0.0-beta.10",
  "canary": "1.0.1-canary.3"
}

pdehaan on Feb 18, 2022