async-http-client: `HTTPClient.remoteConnectionClosed` errors

Current Behavior

After a few hours, e.g. 2.5 hours, my app starts to throw HTTPClient.remoteConnectionClosed errors.

Expected Behavior

No HTTPClient errors should be thrown.

What am i doing?

In Vapor, I have a piece of code like so:

let clientRequest = ClientRequest(url: uri, headers: headers)
return app.client.send(clientRequest)

This method is called 2.5/s, and basically crawls another API. Though, when the errors start happening, the throwing happens even outside this code, and with every eligible API call.

More Info

I’ve tested with v1.7.0 and latest commit and 1.6.4.
Everything works fine on 1.6.4, but using the other two results in the error.
Happens with heavier response bodies (~100kb +). I haven’t seen it happen with a req with res body of like 10/20 kb, probably due to it having less chance to happen.
Happens only after a few hours of my app’s start. Something in range of 2-4, usually like 2.5.
I haven’t tested without the code that is calling another api 2.5 times per second. So not sure if the errors start to happen due to some kind of resource being exhausted, or other reasons.
Haven’t tested with setting httpVersion from .automatic to .http1Only. Can try this if you need more info.

I can attempt to make a repro project if it seems necessary.

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 33 (31 by maintainers)

Commits related to this issue

Use SwiftWasm 5.5.0, bump version to 0.12.0 (#266) `async-http-client` package is now pinned to 1.6.4, as 1.7.0 and later versions caused issues with toolchain downloads (see https://github.com/swift... — committed to swiftwasm/carton by MaxDesiatov 3 years ago
Forward all reads before stream channel inactive (#317) ### Motivation In the AsyncHTTPClient the read event is not always forwarded right away. We have seen instances, in which we see a `HTTPClie... — committed to apple/swift-nio-http2 by fabianfett 3 years ago

Most upvoted comments

@MaxDesiatov I could reproduce your issue. Thanks for the report. A fix has been proposed to nio-http2: apple/swift-nio-http2#317

fabianfett on Nov 30, 2021

.http1Only is the best workaround for now.

fabianfett on Nov 23, 2021

Interesting quotes from the RFC:

Activity on streams numbered lower or equal to the last stream identifier might still complete successfully. The sender of a GOAWAY frame might gracefully shut down a connection by sending a GOAWAY frame, maintaining the connection in an “open” state until all in- progress streams complete.

A server that is attempting to gracefully shut down a connection SHOULD send an initial GOAWAY frame with the last stream identifier set to 2^31-1 and a NO_ERROR code. This signals to the client that a shutdown is imminent and that initiating further requests is prohibited. After allowing time for any in-flight stream creation (at least one round-trip time), the server can send another GOAWAY frame with an updated last stream identifier. This ensures that a connection can be cleanly shut down without losing requests.

This is NOT what the AWS ALB does.

On streams with lower- or equal-numbered identifiers that were not closed completely prior to the connection being closed, reattempting requests, transactions, or any protocol activity is not possible, with the exception of idempotent actions like HTTP GET, PUT, or DELETE. Any protocol activity that uses higher-numbered streams can be safely retried using a new connection.

We could auto retry requests.

https://datatracker.ietf.org/doc/html/rfc7540#section-6.8

fabianfett on Nov 21, 2021

More context in the AWS dev forums: https://forums.aws.amazon.com/message.jspa?messageID=967466

fabianfett on Nov 21, 2021

Interestingly the Go client seems to have the same issue: https://github.com/golang/go/issues/18639

fabianfett on Nov 21, 2021

@robipresotto Would you mind checking if your problem got fixed with 1.8.1 as well?

fabianfett on Dec 1, 2021

@MaxDesiatov we just shipped 1.8.1. Would you mind verifying that we fixed your issue?

fabianfett on Dec 1, 2021

I’m personally doing fine pinning my AHC to 1.6.4, though since i saw someone else having the same problem in Vapor Discord server, i wanted to ask what is a good workaround in your opinion for now? Setting httpVersion to .http1Only should be the better way to make sure the AHC can receive updates, but i just went with the lazier fix … Are those two the expected workarounds, or are there better ways?

MahdiBM on Nov 23, 2021