ramda: R.clone is really slow

@Fi3: If you want only a shallow clone, then you can use merge({}).

clone(obj);  // takes ~650ms on an ancient machine

merge({}, obj); // takes ~5ms on that machine

I just made a pool request with a modified _clone function 2985, with this modification the ramda.clone() goes from 93.228ms to 6.829ms on average after 100 iterations cloning this object.

comparison between ramda lodash and JSON.parse(JSON.stringify(obj)) before:

max/min/avg time in ms after 100 runs ramda clone(): max: 124.229 min: 90.160 avg: 93.228 lodash cloneDeep(): max: 26.819 min: 10.101 avg: 11.643 JSON.parse(JSON.stringify(obj)): max: 17.482 min: 9.535 avg: 10.851

and after :

max/min/avg time in ms after 100 runs ramda clone(): max: 35.512 min: 4.916 avg: 6.829 lodash cloneDeep(): max: 26.919 min: 9.832 avg: 10.937 JSON.parse(JSON.stringify(obj)): max: 19.063 min: 9.525 avg: 10.542

We just came across this issue, was using ramda to clone state objects in redux. For now we have switched to JSON.parse(JSON.stringify(o)) but it would be nice to know if an enhancement is on the horizon.

For us, R.clone was taking close to 1s.

May be, but this problem is definitely not specific to Ramda, and seen in various projects using Lua. Many are looking for a solution and there’s still no common satisfactory one (some solutions truing to profiles the use cases are even adding their own cost, in the profiler itself, a sign that Lua still does not properly support independant instrumentation like there’s in other languages or VM like Java, Python, PHP…). The solutions proposed are then hard to evaluate, we never know if this will optimize things or if it will be stable over time or with other updates of Lua or in the underlying OS as the measured performances are extremely unstable and hard to reproduce.

My purpose was moregeneric than that, and in fact what I was exposing is a problem in Lua itself for not providing a correct or sufficient resource management system. First its implementation of tables has known issues (some of them very severe and easily reproducible, allow unbound use of resources, in terms of both CPU time and memory space), but also the way Lua can be integrated within other existing environment is not well studied: it has been only tuned when running from a single process, and its intiialization is very slow (causing major problems for other kinds of integrations, notably inside other VMs or containers). And we already see security issues, and each attempt to fix them causes a major degradation of performances (notably in CPU time: the fixes are still focusing only the memory space, ignoring all other metrics). So recent changes in all exeting intergations causes sudden spikes in CPU time, and there’s no ezasy way to monitor the execution time. In any shared environment, there’s a hard limit, a quota in execution time: we already see numerous reports in many Lua projects, about the time taken only for the isolation layer: starting a Lua instance, or passing data to Lua or from Lua in memory or via any other storage space takes considerably longer time and its CPU time is constantly growing (and not by small factors: we can see such factors like 2 or 5. This means that any modest update of Lua make a working application or service written in Lua broken (and it’s really difficult to find other ways to optimize the Lua code: when we try, we just save some space, but actually no CPU time or very small improvements (and this is done by a real complexification of the Lua code, tracking to find anything that could be saved: we study the execution profiles, we only optimize small spots, but the major hotspot is not optimizable at all and not acesssible directly in Lua, and not even from the containing environment). This is a symptom that Lua lacks a resources manager and its own integration API is insufficient to correctly track the problem (even attempts to profile apps cause additional costs, that are also growing over time without even changing any thing in this integration code). This raises questions about the usafe and long term usage of Lua in shared environments.

May be it’s time to think about rewrite Lua not on native code (with its base code written in C/C++), with help of an isolation/sandboxing layer, but directly in the hosting VM code itself (several candidates are possible: Python seems a good choice, as well as Java or Javascript, PHP apparently does not. And then provide a resources manager in the hosting environment rather than Lua itself (visibly the attempt to to that in a separate integration layer, like Scribunto for MediaWiki, is failing, and no one knows for now how to fix that, given the too simplist model used by Lua, where resource management is simply completely ignored).

Note that I found rthis thread when lokking at varuous discussions about slow R.clone: this is not a problem specific to Ramda here, but it is found in lot of other projects using other integration methods: Lua is still very difficult to integrate correctly, and as a result the Lua code must be written specifically for each environment, and we see that reusable libraries for Lua are almost inexistant (we just find some small “codelets” working in limited cases and specific environments, but adding their own problems in others, and there’s still no good practices). There’s an attempt in “Lua Rocks”, but we are very far of what would be needed, and I see in fact no one using “Lua Rocks” solutions directly without needing to patch it; and worse, most environment are reluctant to update Lua itself: it requires really too much work to assert again the code and fix lot of new perfomance problems (some former fixes in Lua code have to be reverted or inverted, and other solutiosn need to be tested to see if they have a notable positive impact).

There’s no such major problem with Python, Rust or Javascript (even if there are still works needed to secure them) but both allow these languages to support virtual hosting (e.g. running a WebAssembly in a Javascript VM, or Node.js, and the common libaries for Python, or Perl are also very large now: they are easy to integrate, easy to update for security fixes, or upgrade for new features, without causing major degradation of performance or exhaustion of reasonable quotas.

Your talk here in Ramda is still interesting, and I am still looking for solutions that other projects would have found and what can be reproduced or ported in other environment if this can solve their existing integration problems.

I have also seen that more recent version of Mediawiki (or Scribunto) is gowiong slower and slower over time, with pages in Wikimedia now exploding their rendering time limitation of 10 seconds, only because recursiveClone (mwInit.lua:41) can take more than 5 seconds alone, when in the past it took just a few hundreds of milliseconds.

It seems that this is caused not by Lua itself, but by Scribunto on the interface between Lua and PHP/Mediawiki when passing parameters or returning values. I don’t know what it does not, but a “deep clone” is really overkill, when this interface is just supposed to pass strings of text. But it seems that Scribunto needs it to build temporary arrays. The same occurs in the Scribunto module for Wikibase. Scribunto does not properly reuse the arrays and string buffer it needs to isolate each other’s runtime environment. I understand that such isolation is needed, but Scribunto does not seem to use any pool of objects.

So if a wiki page makes frequent calls to Lua modules via Scribunto (mwInit.lua), or if Scributo modules make frequent calls to the MediaWiki API to use “mw.preprocess()”, this is visibly where most of the time is lost now.

In Wikimedia, some pages may work… sometimes… but not later when refreshed, depending on which server will run the rendering (not all servers have the same performances, but there’s no way to predict which one will do that and visibly they are not properly balanced, so the execution time varies really a lot, notably for mwInit.lua !). And a page may be forced rendered at any unpredictable time (without any change in the source). I’ve seen that servers in the “mw13xx” range are much slower (and will break these pages, then storing in the cache broken renderings) than servers in “mw14xx” range.

Scributo now urgently needs a performance analysis (this is already needed for Wikibase/Wikidata infoboxes, but as well in various statistics pages making many different calls to Scribunto modules, which cannot be optimized and cached because these are calls with many different parameters, so the template/module expansion cache does not help… and in fact iot may even worsen the rendering performance of these pages).

All we can do now is to limit the number of Lua/MediaWiki transitions:

• we can keep more cached values in Lua itself (in varaibles that are kept in the Lua VM across multiple invokations)
• we should limit the number of calls to mw.parse* functions
• we should have a way to get the $wgPageLanguage (built in the PHP API) without having to expand {{int:Lang}} from Lua with a call to the Mediawiki parser (for now it’s impossible, there’s still no mw.* variables for that in the mw package for Scribunto, and no builtin array of strings containing $w* variables exposed in the PHP API).
• Scribunto must extend what can be safely exposed from the Mediawiki PHP API (notably all variables that won’t change during when rendering the same target Wiki page with the same context). For now this Scribunto interface is too weak and generates huge costs if script have to rely on performing a recursive call to the Mediawiki parser.
• And this will become even more important when we’ll start to use Wikifunctions (including for the Abstract Wikipedia starting next year): without this extension, Wikifunctions will NOT be launchable at all and will cause severe performance problems on ALL wikimedia servers for all wikis.

The same problems occurs with all other sorts of integration of Lua with other runtime environments: PHP, Ruby, C#, Java, Javascript, … and even with C/C++ (this is not “Scribunto” in all these, as Scribunto is just the adapter written for MediaWiki in PHP, but there’s the same kind of integration for isolation, i.e. marshalling objects and managing the lifecycle of memory allocations, and type conversions).

On webservers for user-generated contents, or any “cloud computing” service on shared servers (or containers) where there are very strict quotas of resource usage, such interface needs improvement (may be Lua should natively offer a dynamic datastore suppporting caches with “weak pointers”, but I think that the Lua garbage collector is TOO much agressive and not enough flexible).

And what I fear in Lua’s integration in Mediawiki (via Scribunto) for the coming Wikifunctions, will very likely affect as well the integration of Lua as a VM engine for us in other container-based services hosting “functions”: these services criotically need the facility and speed of deployment and an extremely fast instanciation of functions and very low cost of instanciation (so that many small functions can be created, derived, and updated (there’s now a trend for using such functions for low-code/no-code and versatility, not just for cooperative projects but as well in Enterprise, notably for customer services, as they allow a very high degree of customization and adaptation to always changing environments and business rules with very low cost: see for example Amazon’s Functions service, Microsoft Azure’s offers for functions : Wikifunctions will appear there but will be free, and it will critically depend on a fast interface for integrating Lua into another VM and within a highly scalable farm of servers.

Accelerating Lua startup, implementing worker threads and reusable (but still secure) resource pools is a real need. But the isolation interface for now is really too slow and too costly; the very slow and costly array cloning is a severe symptom that such integration is difficult and still not satisfied (independantly of the speed of the Lua engine itself): Lua needs a good resource manager and most probably a better allocator and garbage collector, supporting also on-demand caches and “weak pointers” (somthing still not in Lua but well integrated in Java, .Net/CLR, Ruby, Python or Javascript (note: Wikifunctions initially will not start of Lua implementations, its first releases will use Javascript, as it will scale not only on Wikimedia servers but also on visitor’s web browsers! It is likely that Wikifunctions will integrate easily Amazon Function implementations, and most probably without Lua, Wikifunctions will just be a cataloging index for metadescriptions of functions, but not for hosting the implemenations in Lua in Wiki server farms; there’s also the risk of seeing Lua abandoned by Wikimedia in favor of Javascript, or even Java, or its new replacement now in Android, using the new opensourced VM created by Google).

We’re certainly open to PRs designed to speed this up. I think lodash’s cloneDeep has similar functionality, so we could look to its implementation for inspiration. But I don’t recall if there are substantive behavior differences that Ramda wants to support that are responsible for this speed issue.