simdjson_nodejs: Slower than JSON.parse
Hi @luizperes
I know this library was made to handle large JSON files, but there i occurred to some performance stranges when tried to parse my json and benchmarked, this library is slow, up to 6-x slower.
Here result:
json.parse - large: 985.011ms
simdjson - large: 4.756s
Also, lazyParse
does not return expected result for me or i’m doing something wrong, and even using lazyParse, performance still slow. How we can improve this?
Code to test
const simdJson = require("simdjson");
const bench = (name, fn) => {
console.time(name);
for (let i = 0; i < 200000; i++) {
fn();
}
console.timeEnd(name);
};
// Create large JSON file
let JSON_BUFF_LARGE = {};
for (let i = 0; i < 20; i++) { // 20 is close to 0.5Kb which very small, but you can increase this value
JSON_BUFF_LARGE["key_" + i] = Math.round(Math.random() * 1e16).toString(16);
}
JSON_BUFF_LARGE = JSON.stringify(JSON_BUFF_LARGE);
console.log(
"JSON buffer LARGE size is ",
parseFloat(JSON_BUFF_LARGE.length / 1024).toFixed(2),
"Kb"
);
bench("json.parse - large", () => JSON.parse(JSON_BUFF_LARGE));
bench("simdjson - large", () => simdJson.parse(JSON_BUFF_LARGE));
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 1
- Comments: 20 (10 by maintainers)
That is correct @RichardWright. My idea, as mentioned in other threads, would be to have simdjson implemented as the native json parser directly into the engine e.g V8. That would possibly speed up the process. At this moment I am finishing my masters thesis and don’t have time to try it, so we will have to wait a little bit on that. 😃
CC @jkeiser @lemire
Passing around a buffer and using key access is the preferred method then?
@RichardWright I cannot speak specifically for this library but one general concern is that constructing the full JSON representation in JavaScript, with all the objects, strings, arrays… is remarkably expensive. In some respect, that’s independent from JSON.
cc @jkeiser
So just to confirm, if you want to get the entire object from a string(eg lazy usage isn’t possible), this probably isn’t the library to use in it’s current state?
@luizperes Thanks, i’ll wait 😃
cc @croteaucarine @jkeiser