datafusion: DataFusion does not support wasm32-unknown-unknown target

Good news, fellow WebAssembly enthusiasts! It looks like the stars are finally aligning, and with relatively minimal patching, I successfully compiled the code from the gist (create, insert and query a MemTable) to wasm32-wasi and wasm32-unknown-unknown, and ran it in wasmedge and the browser (via wasmpack):

❯ docker run --rm -it -v $(pwd)/target/wasm32-wasi/debug:/app wasmedge/slim:0.11.2-rc.1 wasmedge --reactor dfwasm.wasm _start
+---+----+
| a | b  |
+---+----+
| b | 10 |
| c | 10 |
+---+----+
0

I pushed the proof-of-concept to a public repository at splitgraph/experimental-datafusion-webassembly. There are two branches:

• wasm32-wasi
  • This is the target I got working first. The readme on this branch contains all the details and you should be able to reproduce it yourself.
• wasm32-unknown-unknown
  • This is branched from wasm32-wasi and the diff of wasm32-wasi..wasm32-unknown-unknown shows the changes
  • The top of the readme includes instructions for running this in the browser, but the patch is still very messy and might not be easily reproducible. Make sure you check Cargo.toml for any patched crates that you need to have checked out at a local path.

In the near future, I intend to cleanup these changes and submit a PR to DataFusion feature-flagging WebAssembly support.

In general, the summary of requirements for wasm-wasi:

• Patching arrow to use its upstream Git repository fixed some initial compilation issues, but otherwise no changes were required. It seems like the latest release on master of arrow-rs (v26.0.0) can compile to both webassembly targets, and so DataFusion just needs to upgrade to that. (There was also one minor change to datafusion/physical-expr required to support the upgraded Arrow package).
• Compiling with RUSTFLAGS="--cfg tokio_unstable" is necessary, and will benefit from recently stabilized wasm support in Tokio
• Removing cloud providers for object_store, as they are not compatible with the wasm32-unknown-unknown target (they may be with wasm32-wasi in some runtimes, but I disabled them).
• Removing local file system calls/writes (at least where necessary)
• Removing bzip2
• Using a forked reqwest to add some wasm compatibility (note: I’m not sure how much of this was/is necessary, and/or if it forces tokio to resolve to a version that other packages incidentally benefit from)
• Replacing two calls to spawn_blocking in sort with spawn, making the compiler happy but possibly causing runtime/logic errors
• …and some other stuff in the patch

for wasm32-unknown-unknown, in addition to all those requirements, it was also necessary to:

• Replace usage of std::time with Instant, in both datafusion and arrow
• Make sure every library that calls getrandom is also passing it the js feature flag, which I did by just patching getrandom and making that the default

To get it to run (without a runtime error related to std::time being unreachable), a few more changes were made:

• Don’t run the demo code in a Tokio main runtime, even with flavor = current-thread. Instead, use wasm-bindgen-futures to await a future that performs the asynchronous task that calls datafusion

This is all very messy. I will clean it up and submit a PR to DataFusion once I have a better sense of the most minimal changes required and the proper way to feature flag them. Also, general disclaimer that I’m new to Rust and YMMV, especially on the wasm-unknown-unknown patch - after all, I barely got it to run. But it does compile and create and query a small in-memory table, which is pretty good!

Hello, folks.

I’m trying to add WASM support to DataFusion’s dependencies. Started with bzip2-rs https://github.com/alexcrichton/bzip2-rs/pull/93

This sounds very cool @milesrichardson - DataFusion should be upgraded to arrow 26.0.0 shortly: https://github.com/apache/arrow-datafusion/pull/4039. I think @Jimexist is in the process of making bzip support optional https://github.com/apache/arrow-datafusion/pull/3993

In terms of being messy / submitting a PR – if it is possible I suggest trying to do it incrementally – like for example we can probably sort out the calls to spawn_blocking in a separate PR

But all in all this is pretty exciting

Thanks @alamb . I will do some experiments but seems like a good solution.

@REASY In my experiment (the one linked above), I put bzip behind a configuration flag and disabled it for the wasm targets. Datafusion still compiled. I don’t know enough about DF to say how important bzip is, or which parts of DF would be broken without it, however. It seemed limited in scope, since it should only affect files that are encoded with bzip.

@seddonm1 compile and run?

I have experimented with that yesterday. I tried wasm32-wasi first and a simple sample works in single threaded mode after disabling some parquet features. See this gist for the example: https://gist.github.com/roee88/91f2b67c3e180fa0dfb688ba8d923dae

For wasm32-unknown-unknown adding getrandom with js as a dependency of the sample makes it compile IIRC, but actually running it is a different story. I tried to get a sample working with wasm-pack and it stops execution on the datafusion context creation, I suspect that it uses some sync primitives that are unsupported in wasm32-unknown-unknown but I didn’t investigate further.

I didn’t try wasm32-unknown-emscripten yet since my local rust version is incompatible with my installed emcc version (both latest at the time of this writing).

Edit: re tokio, the sample above worked on wasm32-wasi with other executors in single threaded mode including futures 0.3, https://github.com/richardanaya/executor, and async-global-executor. As long as you don’t hit code paths that use things like tokio::spawn (used in hash aggregate) it might be fine to use another executor. I’m not sure what’s the best approach for library code that needs to spawn tasks. I have seen opinions for 1) a library should never spawn, 2) futures should be universally supported, 3) a library should accept an executor trait (as implemented in https://github.com/najamelan/async_executors). I didn’t check the state of futures and WebAssembly recently. I didn’t try wasmbindgen-futures because it’s officially no longer compatible with wasi and emscripten and as I said I couldn’t get anything running with wasm32-unknown-unknown.

dirs-rs

Note that https://github.com/apache/arrow-rs/pull/656 from @PsiACE has removed the pretty-table dependency in arrow-rs upstream. This will be included in the 6.0 arrow release (in 2ish months); I am not sure if/how this affects your decision

lz4:

I think lz4 is an optional dependency of parquet: https://github.com/apache/arrow-rs/blob/master/parquet/Cargo.toml#L40 thus perhaps we could just have a lz4 feature flag for datafusion?

Polars proof of concept (shows that arrow-rs and datafusion like API can work): https://github.com/ritchie46/polars/blob/master/js-polars/app.js