kvdex: DENO_CORE.serialize bug on Deno Deploy?

What would be the right way to benchmark superserial against these two?

You can check out the benchmarks I made here: https://github.com/oliver-oloughlin/kvdex/tree/main/benchmarks/utils

Just for comparison, here are the benchmark results for the json serializer and core serializer (roughly 58MB when serialized):

What would be the right way to benchmark superserial against these two?

By the way, I ended up using these basic json one-liners, as in my case I only handle pure json anyway (as it’s data coming and going to the front):

      serialized: {
            serialize: (obj) => new TextEncoder().encode(JSON.stringify(obj)),
            deserialize: (data) => JSON.parse(new TextDecoder().decode(data)),
        },

I guess that they should be faster that your more complex enhanced json serializer?

It will definitely be faster than my custom json stringify/parse as it handles checks for every possible KvValue type. But I think it will still be way slower than a lot of other methods, like v8’s serialize (which is what Deno core uses), that serializes to Uint8Array. If you just have JSON data anyway, and it’s not ridiculously large (many 10s or even 100s of MB), I think you’ll be fine, the speed difference won’t be too noticeable. In a previous version of kvdex I used the basic JSON.parse/stringify for large objects, and I was using it in another project where I had data that reached about 150kB, and writing/reading was still done in just a few ms (roughly 5-15ms).

I might actually go for a compromise solution:

• serialize = true => uses deno core by default (json when on deploy)
• serialize = “core” => always use core
• serialize = “json” => always use json
• serialize = object => use whatever serialize/deserialize/compress/decompress functions that are provided

I’m still biased against a magic switch on Deploy, as it will have surprising effects for devs. It would be more consistent and as easy to always be explicit ie. deprecate the boolean.

Replace the true boolean by a “auto” option and I think it would be just perfect!

I have released a new version (v0.24.0) with the discussed changes (+ a few more smaller changes). Serialized collections should now work by default on Deploy, and be configurable between using the default chosen serializer, core, json or custom.

Just for comparison, here are the benchmark results for the json serializer and core serializer (roughly 58MB when serialized):

I have a potentially working solution now. Would you be so kind to see if importing from this branch directly fixes all the issues you are seeing? Like so:

import { kvdex, collection, model } from "https://raw.githubusercontent.com/oliver-oloughlin/kvdex/patch/deploy-compatible-serialize/mod.ts"

Yes it works! 🙏 🥳

Enabling your custom function with a flag would be useful! In the meantime maybe an alert in the readme that serialize won’t work as-is on Deno Deploy would be useful for others that might work on local and be surprised when deploying (as Deno Deploy is the go-to deploy option of the ecosystem)

I’ll test using Superserial and see if it could be a recommendation for other devs

JSON.parse and JSON.stringify by itself won’t work, because it only supports JSON values, which are a small subset of all values that can be stored in KV. I do however have a custom parse and stringify function which uses JSON.parse and JSON.stringify with custom replacer/reviver functions that works with all KvValue types. Actually, it’s a solution I used for kvdex before I changed it to use Deno.core instead. The downside is that it’s A LOT slower. I’m currently working on adding it as an extension, so that it available for use, but optional.

Superserial may work, but it’s a third-party library, so I don’t want to make it a default. You can however try using it, check my first answer to see how to set custom serialize/deserialize functions for any collection.

After a quick look it seems like kvfs just saves Uint8Array data as chunks, which I already do in kvdex. That’s not really the issue. The problem that needs solving is being able to take any JavaScript object and serialize it as a Uint8Array, and to do the opposite when deserializing.