got: The `merge` function is slow

What would you like to discuss?

Hi, we are using got heavily in my company to send several thousand requests/s so I’m doing some profiling to see in which places our app spends its time the most.

I’ve recently upgraded to Got 10 and I’ve seen that the library spends a significant (though, not alarming) percentage of the time in the normalize arguments phase, which at the same time calls the aforementioned merge function.

Captura de pantalla 2020-01-08 a las 15 50 59

It looks like this single slice call to clone the array seems to spend quite a lot of time. So, my question is, do we really need to clone the array or could it be just referenced? If we need a copy, is slice() the fastest method?

I tried https://jsperf.com/cloning-arrays/3 in Chrome 79 and it seems that there’re faster (not so much tbh) alternatives.

Checklist

  • I have read the documentation.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 30 (4 by maintainers)

Most upvoted comments

I also noticed that got can be “relatively slow” to create an instance, If you have any improvements I’m interested to try them 😃

In my case the request is done on localhost so the network is fast.

Thank you very much, if you provide also profiling for mergeOptions and normalizeArguments (it’s in the normalize-arguments.js file) that will be great! I’ve got a few ideas on how to improve the performance.

Wops, I pasted the pre normalize function before, sorry.

mergeOptions: Captura de pantalla 2020-01-13 a las 11 17 21

normalizeArguments: Captura de pantalla 2020-01-13 a las 11 17 42

Also shouldn’t we deep clone arrays too? For example you can do:

Yes

I think it depends if you’re running a very very very low-spec machine / VPS and make thousands of requests…

Sure. My point was that standalone micro-benchmarks doesn’t mean anything without the context. It’s better to profile the actual running code on the actual system.

@cesarfd What Node.js version and OS?

Hi, we are using got heavily in my company to send several thousand requests/s so I’m doing some profiling to see in which places our app spends its time the most.

Are there any other heavy places like merge.ts? This is very interesting work.

Here’s a more detailed tree where we can see what the code does when we invoke got:

got normalize options

This comes from a 70 second profiling. There are more places where we call got with a different option object each time as we are performing POST json requests.

It seems the normalizeArguments section (and the whole merging process) is the only place where got spends a noticeable amount of time, at least with our use case.