remark: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed
Subject of the issue
Error at simple remark run.
Your environment
- OS: Arch Linux
- Packages:
"remark-cli": "^7.0.0" - Env:
node v11.9.0,npm 6.12.0,yarn 1.19.1
Steps to reproduce
> yarn run remark .
Expected behaviour
Result of linting (or nothing).
Actual behaviour
Pause about 2 minutes and:
> yarn run remark .
yarn run v1.19.1
$ /home/alex/Projects/foo/node_modules/.bin/remark .
<--- Last few GCs --->
[16692:0x3b3e640] 62016 ms: Mark-sweep 1326.5 (1441.8) -> 1314.4 (1441.8) MB, 494.6 / 2.3 ms (average mu = 0.116, current mu = 0.039) allocation failure scavenge might not succeed
[16692:0x3b3e640] 62542 ms: Mark-sweep 1328.3 (1441.8) -> 1316.2 (1442.3) MB, 504.9 / 2.3 ms (average mu = 0.079, current mu = 0.041) allocation failure scavenge might not succeed
<--- JS stacktrace --->
==== JS stack trace =========================================
0: ExitFrame [pc: 0x69bea9cfc5d]
1: StubFrame [pc: 0x69bea9d0f81]
Security context: 0x0123ce91d949 <JSObject>
2: resolve(aka resolve) [0x13031b81fcb9] [path.js:~1068] [pc=0x69bea9e0c8d](this=0x3f09ae0025b1 <undefined>)
3: arguments adaptor frame: 2->0
4: statAndIgnore(aka statAndIgnore) [0x14708d324701] [/home/alex/Projects/foo/node_modules/unified-engine/lib/finder.js:~219] [pc=0x69beaa2fd68](this=0x3f0...
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Writing Node.js report to file: report.20191102.222326.16692.001.json
Node.js report completed
1: 0x9518b0 node::Abort() [/home/alex/.nodenv/versions/11.9.0/bin/node]
2: 0x9525f9 node::OnFatalError(char const*, char const*) [/home/alex/.nodenv/versions/11.9.0/bin/node]
3: 0xb28c2e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/alex/.nodenv/versions/11.9.0/bin/node]
4: 0xb28e64 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/alex/.nodenv/versions/11.9.0/bin/node]
5: 0xf25bd2 [/home/alex/.nodenv/versions/11.9.0/bin/node]
6: 0xf25cd8 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/home/alex/.nodenv/versions/11.9.0/bin/node]
7: 0xf323f8 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/home/alex/.nodenv/versions/11.9.0/bin/node]
8: 0xf32f0b v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/alex/.nodenv/versions/11.9.0/bin/node]
9: 0xf35c41 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/home/alex/.nodenv/versions/11.9.0/bin/node]
10: 0xeffe74 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/home/alex/.nodenv/versions/11.9.0/bin/node]
11: 0x11b582e v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/home/alex/.nodenv/versions/11.9.0/bin/node]
12: 0x69bea9cfc5d
error Command failed with signal "SIGABRT".
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 34 (17 by maintainers)
I don’t think there is any issue. After
No files foundyou end up here: https://github.com/unifiedjs/unified-engine/blob/2f48feaf3483edce7321edb57eed1c67216009a8/lib/find-up.js#L64-L69 , where the path to each folder where remark is looking for files gets added tocache. In any sensible setup it would reach the end of all paths and exit. In your setup, the circular symlinks generate so many paths that the cache gets too heavy for node to handle.It’s a configuration issue on your end (you should ignore the circular symlinks), there’s not much that can be done on remark’s end in my opinion. I mean, there’s certainly room for optimization but there’s hardly a way of preventing this OOM error from happening. We could probably get remark to visit 500000 paths before running out of memory instead of visiting “only” 100000.
Symlinks can be nice, but recursive…
OK, I got your position. Thank you for help, I appreciate it.
I agree with @vhf that the out of memory exception is probably out of scope of
unified-engine. Adding a conditional for the number of files in cache would add overhead, complexity, and could cause issues with projects which do want to handle large numbers of files.One idea to improve
unified-enginecould be to add a--[no-]follow-symlinksoption, giving an automatic way to ignore symlinks from processing. 💭I’m not sure there is much to fix here. In your situation, ignoring
lib/*/lib/should solve your specific issue.unified-engineseems to handle circular symlinks just fine:In your situation, node becomes out of memory because backtracking into all of your files and directories and circular symlinks is just too many paths to keep track of.
I was able to reproduce OP’s issue.
@ChristianMurphy Good call regarding symlink loops: