micromark: unravelLinkedTokens RangeError: Maximum call stack size exceeded

Subject of the issue

#18 fed with https://github.com/remarkjs/remark/blob/8108fe54e04640dda119aad366d70e6edf2602f1/test/fixtures/input/title-attributes.text can trigger a call stack exceeded issue in unravelLinkedTokens. These files are pretty large 1mb and around 30k lines a piece, a more minimal example, at 105kb is also included.

It seems the be related to unterminated links, but more research is needed.

Your environment

OS: Ubuntu
Packages: micromark 2.6.1
Env: node 14

Steps to reproduce

var fs = require('fs')
var micromark = require('./index')

// var doc = fs.readFileSync('crash-395a731d55c510f1338b8c9911c159ab56329d18bc3a12a26b826b750d0b1253.txt')
// var doc = fs.readFileSync('crash-4bf6a4882505b11dea88b5e16e6f0d3766252601ae704e42ebe606d270f9f26f.txt')
var doc = fs.readFileSync('crash-7182fa3e89e1b8fb28bda27b6da6b3769f05b1ce68551d96c46acd0931d95004.txt')

var result = micromark(doc)

console.log(result)

crash-7182fa3e89e1b8fb28bda27b6da6b3769f05b1ce68551d96c46acd0931d95004.txt crash-4bf6a4882505b11dea88b5e16e6f0d3766252601ae704e42ebe606d270f9f26f.txt crash-395a731d55c510f1338b8c9911c159ab56329d18bc3a12a26b826b750d0b1253.txt

a more minimal example of what may be the same issue ([]( repeated 35k times in a 105kb file)

repeated-unterminated-links.txt

Expected behavior

If possible no error, alternatively a better error message could help.

Actual behavior

RangeError: Maximum call stack size exceeded
    at unravelLinkedTokens (micromark/dist/util/subtokenize.js:1:16585)
    at unravelLinkedTokens (micromark/dist/util/subtokenize.js:1:17944)
    at unravelLinkedTokens (micromark/dist/util/subtokenize.js:1:17944)
    at unravelLinkedTokens (micromark/dist/util/subtokenize.js:1:17944)
    at unravelLinkedTokens (micromark/dist/util/subtokenize.js:1:17944)
    at unravelLinkedTokens (micromark/dist/util/subtokenize.js:1:17944)
    at unravelLinkedTokens (micromark/dist/util/subtokenize.js:1:17944)
    at unravelLinkedTokens (micromark/dist/util/subtokenize.js:1:17944)
    at unravelLinkedTokens (micromark/dist/util/subtokenize.js:1:17944)
    at unravelLinkedTokens (micromark/dist/util/subtokenize.js:1:17944)

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 18 (18 by maintainers)

Most upvoted comments

Testing the new code from #21 , It seems to reach it’s limit at: crash-d40a5f23182756de9477e2782ba15dd2be71457d6cd8d0c7637795b5f9ecd6ba.txt 15 minutes in, it hasn’t crashed yet (progress!), but it also seems to be hung/stalled.

I’m running a profiler on it to see if it’s related to unravelLinkedTokens, or something else.

edit: it did eventually finish, including flame graph of run flamegraph.html.log

ChristianMurphy on Sep 22, 2020

the unclosed (3) case mentioned before is still slow, but the rest are fine:

base
24ms
strong
1s
strong/emphasis?
1s
unclosed links
365ms
unclosed links (2)
417ms
tons of definitions
746ms

wooorm on Sep 23, 2020

I managed to get the definitions case down to .7s by adding one line ✨ haha, pfew

wooorm on Sep 23, 2020

Merging tokens is certainly a good idea. I hope that it will speed things up !

fazouane-marouane on Sep 22, 2020

Seems to be my node/mac version!

$ node test/perf.js 
base
25ms
strong
4s
strong/emphasis?
4s
unclosed links
291ms
unclosed links (2)
919ms
tons of definitions
24s

wooorm on Sep 22, 2020

Hello 👋

I’ve provided a fix for the three crash reports in #21

However, we still have an issue with the following and it’s unrelated to unravelLinkedTokens

var micromark = require('./lib')
var doc = Array(35_000).fill('[](').join('')

var result = micromark(doc)
console.log(result)

It seems to be caused by the presence of a call to main() in createTokenizer().write()

fazouane-marouane on Sep 22, 2020