svgo: FATAL ERROR out of memory after using 2.8GB RAM (convertPathData, defaultoption)
Prozessing the olderst version of File:Germany (+districts +municipalities) location map.svg (~73MB) fails afer needing more than 2,8GB RAM (I have 16GB RAM total, and during processing always more than 8GB free)
_location_map_2013.svg#filehistory){kind=link}
$ svgo -i “20180507223736!Germany_(+districts_+municipalities)_location_map_2013.svg” -o output.svg
<--- Last few GCs --->
[14628:000001C5D93EBC40] 1345238 ms: Mark-sweep 1384.2 (1457.6) -> 1384.1 (1458.1) MB, 3403.4 / 0.0 ms allocation failure GC in old space requested
[14628:000001C5D93EBC40] 1347898 ms: Mark-sweep 1384.1 (1458.1) -> 1384.1 (1427.1) MB, 2659.4 / 0.0 ms last resort GC in old space requested
[14628:000001C5D93EBC40] 1350475 ms: Mark-sweep 1384.1 (1427.1) -> 1384.1 (1427.1) MB, 2576.5 / 0.0 ms last resort GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 00000248B9D257C1 <JSObject>
2: split(this=000000CF890516F9 <Very long string[12093]>)
3: fn [C:\Users\jkalliau\AppData\Roaming\npm\node_modules\svgo\plugins\convertPathData.js:~56] [pc=000001C934FF3D35](this=00000175BEC37051 <Object map = 000000B31E6A17E1>,item=0000027C01EF5009 <JSObject>,params=00000175BEC37E89 <Object map = 000000B31E6BA439>)
4: arguments adaptor frame: 3->2
5: /* anonymous */(aka /* an...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node_module_register
2: v8::internal::FatalProcessOutOfMemory
3: v8::internal::FatalProcessOutOfMemory
4: v8::internal::Factory::NewRawTwoByteString
5: v8::internal::Smi::SmiPrint
6: v8::internal::StackGuard::HandleInterrupts
7: v8::internal::RegExpImpl::Exec
8: v8::internal::RegExpImpl::Exec
9: v8::internal::interpreter::BytecodeArrayRandomIterator::UpdateOffsetFromIndex
10: 000001C934E843C1
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 1
- Comments: 19 (10 by maintainers)
Thanks for sharing, I figured on some environments it’d still be possible.
In my opinion, it is worth the time. It’s not top priority, but making SVGO work with large files and be better equip to operate on smaller environments like phones or CI is favorable.
For example, I wouldn’t want OhMySVG crashing on my phone (PinePhone/Linux) because SVGO is using too much memory. If we can find better ways to store and process data that reduces our memory footprint, it’s just better for everyone.
Allocating memory takes time, which we’re trying to get down, and reserving resources that other processes or the system memory cache could be using instead isn’t considerate of our users or the other software installed on their machine.
TL;DR: Large files not being our primary use case, or the assumptions that clients have the resources to spare, isn’t a good reason not to try to use resources more efficiently.
Edit: Sorry I’m less active right not btw, I’m sick. 😷
I retried it on another computer, and I can’t reproduce the bug either.
I consider it as fixed or not reproducable report.Thanks for your efford.Edit: Others can still reproduce the bug.
I’ll close this issue as it’s quite open-ended, and we’re actively trying to improve the performance of SVGO as we go. I haven’t personally been able to reproduce it under the same conditions, but appreciate that given a large enough SVG or low enough available memory this will still occur. My hope is that the situation here will improve over the next few releases.
Here are some stats showing how SVGO is performing with the
mainbranch and other recent releases:main(8d6385bd9ab49d1d300a10268930238baa5eb269)Overall, time has significant improved, memory usage has marginally improved, and compression has negligibly improved.
The rationale for the time and memory improvements are mostly for https://github.com/svg/svgo/pull/1918, and once merged will be used as the benchmark to ensure we understand the performance implications of new contributions.
Meanwhile, we may explore solutions like chunking large SVG files or clearing caches when we’re using too much memory, etc. That would undermine speed ofc, but may improve stability if users are having OOM issues.
Rather than take action, since it’s unclear when we’ve done enough, we’ll just ensure all future work keeps memory/performance in mind, and that we profile SVGO and upstream dependencies to see where we can improve.
@JoKalliauer
I believe you need to set the environment variable for NODE_OPTIONS.
On Windows via:
set NODE_OPTIONS=--max_old_space_size=8192On Linux via:
export NODE_OPTIONS=--max_old_space_size=8192Alternatively, you can try svgcleaner. I found this reduces size more and is extremely fast, as it is written in Rust.
You can try replacing
svgowithNODE_OPTIONS=--max_old_space_size=8192 node /usr/local/bin/svgo. I had to process a 120MB file which took 5GB of RAM.