watcher: Does not work in stackblitz

There is now an alpha version of a WASM version available in the @parcel/watcher-wasm package. Docs. I verified that all of the tests pass in the Stackblitz environment. I think the Stackblitz team should now be able to make this work as an automatic fallback for @parcel/watcher as they do for lightningcss and esbuild’s wasm versions.

in https://github.com/parcel-bundler/watcher/pull/134 i’ve added support for a chokidar backend 🎊

This should now work in Stackblitz as of v2.3.0! https://stackblitz.com/edit/node-bpupxc?file=index.js

I would not recommend using it except in unsupported environments (like browsers). It is significantly less efficient than the native watcher implementations that deeply integrate with the operating system. It must recursively traverse the directory hierarchy and watch each nested directory separately, whereas most of the native implementations use operating system APIs that avoid the need for this.

As an example of the difference, the Nuxt team tested their previous implementation using chokidar (which also uses node’s watcher API under the hood) and the native Parcel watcher and saw startup time decrease from ~30s to 300ms https://github.com/nuxt/nuxt/pull/20179#issuecomment-1502103899. That’s probably almost entirely due to not needing to walk the entire directory heirarchy, which can be avoided by using better OS APIs.

So unless you’re running in a browser or on an unsupported operating system, I’d keep using @parcel/watcher. You could also implement a fallback where you use @parcel/watcher-wasm automatically when building for the browser and @parcel/watcher otherwise. I believe Stackblitz itself will start doing this automatically soon as well, so using @parcel/watcher there would begin working with no changes on your side.

Hey ya all 👋

Maybe I can chime in on some of this stuff.

I think the solution that has been discussed here, a pure JS implementation that simply uses Node’s fs module, sounds reasonable to me. Compiling to WASM and providing the syscalls via imports is just overhead which can be avoided. Also it means data constantly has to be copied from and to linear memory.

Yeah try/catch seems fine.

@devongovett Yep, you can use a try/catch or you leverage conditional exports. Loading a native addon will fail in WebContainer because they are disabled by default (via the new CLI option --no-addons). This also gets inherited to worker threads and child processes. If you try to load a native addon while it’s disabled you’ll get a descriptive error.

The other option would be to use conditional exports. In your package.json you could define your entry point to use a different one if native addons have been disabled. For example:

{
  "name": "@parcel/watcher",
  "exports": {
    ".": {
      "node-addons": "./index.js",
      "default": "./no-addons-main.js",
    }
  }
}

With this in place, it would use a different entry point (./no-addons-main.js) when native addons are disabled via --no-addons and otherwise load index.js, which can be the entry point that relies on native code.

In this particular case, it’s recommended to treat the default condition as the progressive enhancement for environments where native addons may not be supported (e.g. the browser).

I would personally recommend the approach using conditional exports because it’s good practice to define a clear interface that is exported as well as taking different environments into account.

Yeah try/catch seems fine.

To be 100% transparent, I do not personally have time to work on this anytime soon. But, if someone wants to do it, please feel free to send a PR! Basically we’d expect a JS implementation of the same API exposed by the module currently, but written in JS using the Node fs APIs, as a fallback in case loading the native module fails.

Is there a more technical overview available that describes the exact interface that is available and some of the implementation details?

Beyond what’s in the readme and the blog post not really, as it hasn’t been an issue to date. I think the parcel filewatcher might be a unique case simply because it’s purpose is a very specific syscall level integration, whereas other projects (like next.js/swc/etc) have implementations that are a bit higher level and/or syscall agnostic.

Is it shimming the Node.js API, or something lower level?

Lower level, largely the syscall layer where Node.js interacts with the host OS.

For WASM, what POSIX APIs (if any) are available? You describe a WebContainer as a “WebAssembly-based operating system”, but what is the API?

The shell in WebContainer exposes a lot of the usual POSIX built-ins (ls, cd, cat, etc). Beyond that, the only interfaces to the Wasm OS today are exposed via the Node.js API surface. The reason for this is that we do not intend to expose a non-standardized WebAssembly system interface to end users, largely because WASI already exists. However, WASI isn’t yet capable of providing all the syscalls we need to run Node.js wholesale on Wasm, but bridging this gap is one of our primary objectives in being a part of the Bytecode Alliance.

For Wasm applications that need syscall access, in the near future the solution would be leveraging Node.js’ implementation of WASI. We haven’t enabled access to this in WebContainer due to some upcoming changes I can’t speak to yet. For the time being, you can instead leverage the Node.js API surface to perform syscalls. This is similar to how emscripten compiles down to the Node.js APIs for filesystem, networking, etc.

What NAPI apis are available?

None. WebContainer is an operating system designed to run inside a browser engine, and therefore is subject to the same limitations as any other browser context: if you want execute code from a language other than JS, it needs to be precompiled as a WASM.

There has been work from Node.js core on enabling Wasm support w/ NAPI but AFAIK has not yet come to fruition. Last I heard there was interest in WASI to help bridge the syscall gaps, but it needs to be sufficiently matured before it could be viable.

Can you only run Node within a WebContainer, or do other processes work? What if we implemented a CLI entirely in Rust for example?

Node.js is the only runtime we ship with WebContainer OOTB atm, but any language/runtime provided as a Wasm module can also run inside WebContainer today.

If you have an entirely Rust-based CLI that compiles to a Wasm module, the only thing you’d need to do is wrap it with some JS code that wires it up to the interfaces it requires (command line args, fs, network, etc).

For file watching specifically, what is the native API? There is no standard POSIX API for file watching, each OS implements their own (e.g. Darwin’s FSEvents/kqueue, Linux inotify, etc.). So if a WebContainer provides such an API, what is it? Or is it only available to JS via a Node.js fs.watch shim?

Yes, only available to JS via Node.js’ fs.watch/etc until WASI support is landed.

As we move more of Parcel away from JavaScript and more toward native Rust modules, having an understanding of this would be helpful.

Totally. I think the main thing is to include Wasm as a compile target alongside your existing native addons. This does require a little bit of extra work in loading those native bindings- here’s how Next.js does this w/ SWC as an example.

Another quick way to test if this will work for existing (and future) sandboxed Node.js environments is by passing the --no-addons flag to any Node.js command. I think @padmaia was using this when developing the Next.js SWC Wasm and might have additional color.

Hey all- Eric from StackBlitz here, happy to provide some context.

StackBlitz uses WebContainers (a WebAssembly-based operating system) to run Node.js inside your browser tab. This version of Node.js is pulled from Node.js core itself (w/ some minor tweaks) so it has broad support for nearly all Node.js APIs. Importantly, WebContainer includes a UNIX compliant filesystem, so all Node.js FS APIs you would expect (including synchronous operations, watching, etc) work exactly as they do on local.

The main limitation of this approach is that native addons must be available as Wasm binaries otherwise they will not work. This seems to be the issue happening here for the @parcel/watcher package. Shipping Wasm binaries alongside native binaries is the direction most packages in the ecosystem are heading towards, and is the impetus of folks like Next.js/etc pushing for the npm Package Distributions RFC. The ability to restrict native binary addons in Node.js is also important from a security PoV as progress is made towards complete sandboxing of Node.js code execution.

For filesystem operations in StackBlitz/WebContainer, you can use the normal fs.watch and fs.watchFile from Node.js’ core fs module. This is what Chokidar uses under the hood when running on non-Darwin platforms. Perhaps @parcel/watcher could do something similar in this regard, where the Node.js fs events could be normalized, handed to your Wasm module, which then emits the appropriate results to listeners?

I’m very interested in using @parcel/watch in some of my projects (which are currently using Chokidar). For some of those projects it’s important to me that they also work on Stackblitz - for those I’ll hold off switching over to @parcel/watch for now until support has landed.