reqwest: issue when decompressing zstd-compressed response bodies
When using zstd feature to decompress HTTP response bodies, if the response body is quite large, then I get the following error:
reqwest::Error { kind: Decode, source: "there are extra bytes after body has been decompressed" }
For the same query, when using gzip, it’s working.
The same query with curl (curl -v --compressed -Haccept-encoding:zstd ....) is working.
In my experience, this issue only appears when the response size is quite large. For smaller payload, reqwest can decompress the HTTP response body without issues.
I cannot give any reproducer for now, as I can only reproduce it with a public API that needs an OAuth token.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 26 (11 by maintainers)
Commits related to this issue
- fix: support HTTP responses containing multiple ZSTD frames (#2583) Closes #2574 — committed to CodeWeaver13/reqwest by Andrey36652 2 years ago
Because your ZSTD encoded data (in the test) is not full
I notice that in async-compression, the
ZstdDecoderhas an option formultiple_members(bool). I imagine setting that would fix this. My question is, is that the right place to do this? Is it expected in zstd to have multiple members? I believe in other algorithms that’s supported but atypical. If you want to investigate that, it’d be good to have a solid answer before we blindly flip that switch.