reqwest: Connection reset while connecting `https://storage.googleapis.com` with rustls

Hi, I came across a strange issue where sending a request via rustls with the host header to https://storage.googleapis.com results in the connection being reset.

Related to https://github.com/apache/incubator-opendal/issues/2125#issuecomment-1523431970

Reproduce

Cargo.toml

[package]
name = "demo"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[features]
default = ["reqwest/rustls-tls-native-roots"]
native-tls = ["reqwest/native-tls"]

[dependencies]
reqwest = { version = "0.11.16", default-features = false }
tokio = { version = "1.28.0", features = ["full"] }

main.rs

use reqwest::Client;

#[tokio::main]
async fn main() {
    let client = Client::new();

    let url = "https://storage.googleapis.com/";
    let content = client
        .get(url)
        .body("")
        .header("host", "storage.googleapis.com")
        .send()
        .await
        .expect("Failed to send request");

    println!("{}", content.text().await.unwrap())
}

With rustls:

:) cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.04s
     Running `target/debug/demo`
thread 'main' panicked at 'Failed to send request: reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("storage.googleapis.com")), port: None, path: "/", query: None, fragment: None }, source: hyper::Error(Http2, Error { kind: Reset(StreamId(1), PROTOCOL_ERROR, Remote) }) }', src/main.rs:14:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

With native-tls:

:( cargo run --features=native-tls
    Finished dev [unoptimized + debuginfo] target(s) in 0.04s
     Running `target/debug/demo`
<?xml version='1.0' encoding='UTF-8'?><Error><Code>MissingSecurityHeader</Code><Message>Your request was missing a required header.</Message><Details>Authorization</Details></Error>

If we remove the host header, it works too:

:) cargo run
   Compiling demo v0.1.0 (/tmp/demo)
    Finished dev [unoptimized + debuginfo] target(s) in 0.30s
     Running `target/debug/demo`
<?xml version='1.0' encoding='UTF-8'?><Error><Code>MissingSecurityHeader</Code><Message>Your request was missing a required header.</Message><Details>Authorization</Details></Error>

Question

How do rustls and native-tls interact with the host header? Is this behavior expected?

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 3
  • Comments: 22 (16 by maintainers)

Most upvoted comments

I know what happens!

The package-capture tools change the behavior. Damn…

Oh, they’re sending a RST_STREAM of PROTOCOL_ERROR? I think that interpretation is incorrect. It’s not a protocol error, the protocol says it MAY happen.

I assumed it was the server replying with a 400, which is fine, a server can say anything is bad in that sense.

The HTTP/2 spec (RFC 9113) doesn’t make the Host header illegal (like it does a few others, such as connection). But it does say that :authority is the correct one, and that a client should not make a request where the :authority and host differ.

I can also see the same behaviour when trying to fetch https://www.youtube.com. That suggests that the issue may be common to using any Google web service, not just Google Cloud Storage. I haven’t tested this against a Google Cloud L7 load balancer.

It looks like nginx has a similar issue when serving HTTP/2, but it returns HTTP 400 Bad Request. nginx.org appears to be configured to serve over HTTP/1.1 only (which isn’t affected), but I could reproduce this behaviour on a local nginx instance which was configured to serve HTTP/2.

In applications, this issue can be surprising, because it’ll only break once a server accepts HTTP/2. This could lead to a similar situation to the January 2022 Firefox outage triggered[^1] by Google enabling HTTP/3 serving by default on GCP load balancers. So I can see the argument for the library taking care of this.

However, sometimes you need to sign requests including the Host and/or :authority header(s), and your application needs to know if the Host header is used (HTTP/1.1), the :authority header is used (HTTP/2), or both (HTTP/2) – regardless of server-side limitations. A HTTP library silently altering those requests would invalidate those signatures.

In any case, the simple answer for applications not signing requests is “don’t explicitly set the Host header”. 😄

[^1]: The word “triggered” has a different meaning to the word “caused”. Mozilla’s public write-up goes into lots of detail about all the contributing factors.

Well, between the two, only rustls supports ALPN, which means it will use HTTP/2 automatically (if the server supports it). The native-tls backend doesn’t have ALPN support, the request is using HTTP/1 (unless you use http2_prior_knowledge()).

It shouldn’t be an error, at least the spec doesn’t say it should.