txiki.js: TextDecoding pretty slow
I recently wondered why TextDecoding was so slow. Several seconds for 50-100 Kbytes (I am running on a low power embedded system with single core performance comparable to RPI 1).
I saw it currently uses a JS only implementation. I am wondering why, the JS_NewStringLen of quickjs already handles utf8 doesn’t it?
As a workaround I am currently using
import FFI from 'tjs:ffi';
export function quickDecodeUtf8(buf) {
return FFI.bufferToString(buf);
}
Which also just does a JS_NewString.
The opposite case of Encoding isn’t quite as slow, but it seems JS_ToCString would also handle this case.
Though maybe I am missing something, e.g. some feature TextEncoder/TextDecoder which would not be happening/handled in the quickjs functions.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 18 (16 by maintainers)
Sorry, I got carried away… The use case for the source code is not pertinent, as we could use
JS_NewStringLen()to create the string from a fragment of the source code kept in its original UTF-8 encoded format.I am not familiar with the issue you mention regarding
TextDecoderandJS_NewStringLen, let me investigate this later today.Efficient conversions between raw byte strings and
JSStringwould still be a useful addition. I shall propose an API later today too.Hello, late to the discussion…
Where does the byte stream come from? It looks like we should support raw byte strings in the QuickJS engine and add some methods to convert between JSStrings and UTF-8 byte data.
Uint8Arraymight be a good candidate, I am not sure.There is a use case in the engine for this: function source code. We should keep the source code as raw 8-bit data and implement a getter for
Function.prototype.sourcethat performs the conversion to JSString. This would allow us to share the source code for functions, nested functions and classes / modules. The source is currently duplicated, using extra memory and sizeable overhead in the serialized data.I will look into this tomorrow, trying to make a C implementation.