franz-go: lz4 decompression causes a lot of allocations due to ioutil.ReadAll

While using a consumer group where messages were lz4 compressed I noticed most of the allocations were due to ioutil.ReadAll. Should we consider using io.Copy (fixed buffer length) instead of ioutil.ReadAll (dynamically resized array)? I can provide benchmark results if needed

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 16 (15 by maintainers)

Most upvoted comments

I’m open to a PR. You’ll need to use the third option because decompressors are used concurrently – for example, the client can be processing responses from multiple brokers at once.

Let me know if you can get a PR open in the next day or two – I have two other PRs ready that can go into a patch release and this can fit as well.