compress: Default Brotli compression level is too slow

I stand corrected—I’ve discovered that it is in fact possible to set brotli compression level (1-11). A little obscure, but there you are:

        const options: zlib.BrotliOptions = {
            params: {
                [zlib.constants.BROTLI_PARAM_QUALITY]: 7
            }
        }

For koa:

    koa.use(koaCompress({
        br: {
            params: {
                [zlib.constants.BROTLI_PARAM_QUALITY]: 7
            }
        }
    }))

With a 12MB json payload typical for my app:

BROTLI Level 1 Compression 88.7% Time 36ms Level 2 Compression 89.3% Time 72ms Level 3 Compression 89.6% Time 74ms Level 4 Compression 91.1% Time 112ms Level 5 Compression 92.0% Time 180ms Level 6 Compression 92.1% Time 222ms Level 7 Compression 92.2% Time 302ms Level 8 Compression 92.3% Time 397ms Level 9 Compression 92.4% Time 545ms Level 10 Compression 93.1% Time 6441ms Level 11 Compression 93.6% Time 33956ms

GZIP Level 1 Compression 87.4% Time 75ms Level 2 Compression 87.8% Time 79ms Level 3 Compression 88.2% Time 91ms Level 4 Compression 89.1% Time 115ms Level 5 Compression 89.4% Time 135ms Level 6 Compression 89.9% Time 169ms Level 7 Compression 90.0% Time 215ms Level 8 Compression 90.2% Time 505ms Level 9 Compression 90.3% Time 557ms

So this makes things a lot more interesting. For the kind of json data I work with, there seems to be a large sweet spot where brotli has the edge: both faster and smaller file size!

You may need to run some experiments with your own data to tune the compression level (1-11 for brotli, 1-9 for gzip).

I think koa-compress can use brotli by default so long as it sets the compression level to something suitable for doing realtime compression.

NodeJS default is Brotli L11, which is great for offline compression, but woeful for realtime compression. Since koa-compress is all about realtime compression, it must not use the NodeJS default.

I suggest Brotli L5. From my experiments (above), this works out 3x faster then default GZIP L9 compression. Of course your mileage will vary.

I noticed this as well, running in node 14 and using streams. Enabling brotli compression causes a 3+ second delay before transfer of bytes begins on requests that experience <20ms of latency when using gzip.

This suggested to me that it was buffering the entire file into memory before sending it. While this may be the case, it is clear this would be a change in node’s zlib, as this library seems very sensible and treats the different encoders the same.

Given the disastrous performance here, it may be best to change the default such that brotli is disabled.

+1, please change the default back to gzip.

According to https://www.opencpu.org/posts/brotli-benchmarks/, brotli has an edge of <25% in compression ratio, but the compression speed is 70 times slower than gzip. Definitely not sensible to choose brotli for on-the-fly compression.

The brotli implementation in NodeJS does not seem to allow the compression level to be specified (1-11). This is unfortunate. Given the time taken to compress, it appears that the default level is high (11?).

Some experiments with a 3MB csv file on 2.9GHz i7:

NodeJS.zlib.brotliCompress Level - Compression 81% Speed 5924ms

NodeJS.zlib.gzip Level 1 Compression 67% Speed 45ms Level 2 Compression 69% Speed 48ms Level 3 Compression 70% Speed 55ms Level 4 Compression 71% Speed 67ms Level 5 Compression 72% Speed 90ms Level 6 Compression 73% Speed 113ms Level 7 Compression 73% Speed 131ms Level 8 Compression 73% Speed 164ms Level 9 Compression 73% Speed 169ms

https://github.com/foliojs/brotli.js I assume these times cannot be directly compare to above (JS vs NodeJS.native?). Level 1 Compression 71% Speed 159ms Level 2 Compression 72% Speed 142ms Level 3 Compression 73% Speed 126ms Level 4 Compression 74% Speed 174ms Level 5 Compression 76% Speed 216ms Level 6 Compression 77% Speed 256ms Level 7 Compression 77% Speed 373ms Level 8 Compression 78% Speed 428ms Level 9 Compression 78% Speed 548ms Level 10 Compression 81% Speed 5801ms Level 11 Compression 81% Speed 13457ms

I conclude that brotli is a poor choice for runtime compression. On the other hand, it is ideal for build-time compression!

Further Reading: https://blogs.akamai.com/2016/02/understanding-brotlis-potential.html

I noticed this as well, running in node 14 and using streams. Enabling brotli compression causes a 3+ second delay before transfer of bytes begins on requests that experience <20ms of latency when using gzip.

This suggested to me that it was buffering the entire file into memory before sending it. While this may be the case, it is clear this would be a change in node’s zlib, as this library seems very sensible and treats the different encoders the same.

Given the disastrous performance here, it may be best to change the default such that brotli is disabled.