godot: Compress HTML5 export for hosts that don't support on-the-fly compression like itch.io

Usually you’d rely on your server to gzip those files but for instance https://itch.io/ (which is really popular for distributing web games) does not do it because other engines (e.g. Unity) already compress their data during the export.

This makes me sad… I think some engines started to do that because not all hosting providers enabled serverside compression. If major websites like itch.io are not doing it because engines already do then that’s a really bad feedback loop. The engine doing it will never be as fast as the browser + webserver working together in the standard way 😢

If you do want to do this in Godot, then if the pck file is a normal file from emscripten’s point of view, you can compress it using the LZ4 option, with just -s LZ4 (but obviously LZ4 is not the most compact compression format - the decompressor is tiny though, and very fast).

For the wasm file, you’d need something external to emscripten. That may have different tradeoffs, though, as if you compress it manually you may be preventing the browser from doing streaming compilation (compile during the download). So that might be worth measuring - a bigger download might start up faster in some cases if it’s streaming. There are a bunch of decompressors in JS that could be experimented with (like pako and zee).

This would be good to have for itch.io or any server you dont control that doesnt have server side compression enabled. Although it should be an option.

I did some comparisons between compression algorithms

All compression algorithms used the maximum settings, only the .pak and the .wasm were compressed. the pak was always 35kb. Files were compressed using https://peazip.github.io/

Custom export template with most everything striped the .wasm was 9.75mb. Release build

7z lzma2 - 1.8mb Zstd - 2mb brotili - 2.11mb gzip - 2.55mb

with the default export template, release build. the size of the .wasm is 13.4mb

7z lzma2 - 2.65mb Zstd - 2.94mb brotili - 3.12mb gzip - 3.75mb

Based on these results I think adding compression to html5 export would be good. its seems the best would be zstd as it maintains good compression and is built into godot.

Can you suggest some simple way to decompress the .wasm file of the engine itself in the body of the main html file of the games (or so), which does not require deep knowledge in the field of rocket science and quantum physics, for simple workers like me?

Look into using pako as mentioned above, but this won’t be trivial. @paulloz’s branch no longer appears to be available so this is something you’d have to implement yourself.

Has anything changed in three years?

No, although we have support for Brotli decompression in Godot now. That said, using Zstandard from Godot would still be a better choice if we want to support compressed PCKs that are decompressed at load-time (Zstandard is more efficient, faster, or both).

The .wasm file itself requires a different approach (decompression from JavaScript) as Godot isn’t initialized at the time the browser reads that file.

I don’t think this should be an option, it should always be enabled or it’ll just cause confusion. A web frontend fallback must be added for webgame hosts that deliver without Accept-Encoding headers.

We should also consider Brotli, Calinou tested this and it seems there is a reasonable difference in size.

Not sure if compressing the main pack makes sense, doesn’t .pck already use some compression?

Has anything changed in three years?

If you’re interested in this, there’s a WIP branch on my fork repo with a partial implementation.