reqwest: issue when decompressing zstd-compressed response bodies

When using zstd feature to decompress HTTP response bodies, if the response body is quite large, then I get the following error:

reqwest::Error { kind: Decode, source: "there are extra bytes after body has been decompressed" }

For the same query, when using gzip, it’s working. The same query with curl (curl -v --compressed -Haccept-encoding:zstd ....) is working.

In my experience, this issue only appears when the response size is quite large. For smaller payload, reqwest can decompress the HTTP response body without issues.

I cannot give any reproducer for now, as I can only reproduce it with a public API that needs an OAuth token.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 26 (11 by maintainers)

Commits related to this issue

Most upvoted comments

But the test ab0f9ff never completes.

Because your ZSTD encoded data (in the test) is not full

I notice that in async-compression, the ZstdDecoder has an option for multiple_members(bool). I imagine setting that would fix this. My question is, is that the right place to do this? Is it expected in zstd to have multiple members? I believe in other algorithms that’s supported but atypical. If you want to investigate that, it’d be good to have a solid answer before we blindly flip that switch.