hyper: TLS server corrupts large client responses
Got a strange one. I’m using hyper as a simple proxy. I have an EC2 instance running on HTTPS. When I contact the server, it uses the hyper client to download data from another (external) source, and sends it back as a response.
My service function has the signature
fn service(
mut req: Request<hyper::Body>,
) -> impl Future<Item = Response<Body>, Error = Error> + Send
And since the client ResponseFuture implements Future<Item=Response<Body> I was just returning the future directly. However I found that large payloads (~1MB) were getting corrupted in strange ways. Certain sections of the payload were being duplicated and/or sent out-of-order (but usually the first 100KB or so were correct). This problem only seemed to occur when both the server and the external server were contacted through TLS - if either used HTTP the problem went away.
I worked around the problem by concatenating the response body before returning it.
resp.and_then(|resp| {
let (parts, body) = resp.into_parts();
body.concat2().map(|body| {
let body = Body::from(body);
Response::from_parts(parts, body)
})
})
That seemed to fix it.
Stack:
hyper = "0.12.8"
hyper-tls = "0.3.0"
rustls = "0.13.0"
tokio = "0.1.7"
tokio-rustls = "0.7.1"
tokio-tcp = "0.1.1"
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 3
- Comments: 21 (11 by maintainers)
I just published tokio-rustls 0.7.2, which should fix the problem. By the way, it also has writev support.
I’ve just published rustls 0.13.1 too.
OK, I now understand what’s happening. It revolves around use of
rustls::Streamwith non-blocking transports. This wasn’t really supported, and indeed the documentation (though alas not the std::io type system) for the type states the requirement for a blocking transport. This can be made to work, and I’ll push a version of rustls which supports this shortly.I can reproduce this, and echo the difficulty getting a reproduction over loopback. The discontinuities seem to be non-deterministic, too. So I’m currently thinking the bug is happening only when TCP buffers get filled, and then somewhere backpressure goes wrong. Still looking.
Thanks for the repro, I’ll look at this asap
I see two TLS dependencies, does it happen with both? If so, that’d imply the bug is in hyper. If only one, that’d imply the bug is in that TLS lib.