fastify-compress: Unicode emoji converted to broken chars in certain extreme long string responses

Prerequisites

  • I have written a descriptive issue title
  • I have searched existing issues to ensure the bug has not already been reported

Fastify version

4.26.2

Plugin version

7.0.2

Node.js version

18.x

Operating system

Windows

Description

Download a 200kb yaml plain text file with fastify/compress. Certain lines of the yaml contains illegal chars.

image

As you can see, on the left side (without compress), icons in the small green box are same. But on the right side, one of the icon becomes 2 strange question marks.

The content encoding is br (I tried with gzip, same error)

image

Any thoughts, what can be the root cause?

Steps to Reproduce

const fastify = new Fastify({})
fastify.register(fastifyCompress)

fastify.get('/subscription', async (req, reply) => {
  const subscription = await handler.downloadClashSubscriptionAsync(config)

  // await require('fs').promises
  //   .writeFile('/out.txt', subscription, 'utf-8')  // <--- the file content is okay

  return reply.send(subscription)  // <--- the file content is corrupt
})

Expected Behavior

Dont expect the two question marks appear in the content.

About this issue

  • Original URL
  • State: closed
  • Created 3 months ago
  • Comments: 24 (24 by maintainers)

Most upvoted comments

I can reproduce. Note that /good has no compression enabled in the example, so that’s why it works.

It is impossible for anyone to assist if you are unable to provide a known good input file that results in a bad output file after being compressed.

const data = fs.readFileSync('/tmp/known_good.txt')
const compressedData = zlib.brotliCompressSync(data)
fs.writeFileSync('/tmp/compressed.txt.brotli', compressedData)
$ brotli -d /tmp/compressed.txt.brotli

Please reopen the issue, until #291 is tested successfully. Thanks

Can you provide your mvce as a github repo, please?