falcon: errors on production, socket fail
Hi there, I have migrated some months ago to Falcon on rails app,
suddenly I’ve started to see weird errors which are not being notified on my reporting error, but users noted that some realtime (socket connection) failed.
the logs is like this:
{"time":"2024-10-17T02:38:15+00:00","severity":"warn","oid":3028920,"pid":1,"subject":"Async::Task","annotation":"Reading HTTP/1.1 requests for Async::HTTP::Protocol::HTTP1::Server.","message":["Task may have ended with unhandled exception.","undefined method `write' for nil:NilClass"],"error":{"kind":"NoMethodError","message":"undefined method `write' for nil:NilClass","stack":"/usr/local/bundle/gems/protocol-http1-0.18.0/lib/protocol/http1/connection.rb:134:in `write_response'\n/usr/local/bundle/gems/async-http-0.63.0/lib/async/http/protocol/http1/server.rb:63:in `each'\n/usr/local/bundle/gems/async-http-0.63.0/lib/async/http/server.rb:40:in `accept'\n/usr/local/bundle/gems/async-io-1.41.0/lib/async/io/server.rb:15:in `block in accept_each'\n/usr/local/bundle/gems/async-io-1.41.0/lib/async/io/socket.rb:58:in `block in accept'\n/usr/local/bundle/gems/async-2.8.2/lib/async/task.rb:161:in `block in run'\n/usr/local/bundle/gems/async-2.8.2/lib/async/task.rb:331:in `block in schedule'\n"}}
not sure what could happen here but seems to be async http gems
the installed dependences were:
async (2.8.2)
console (~> 1.10)
fiber-annotation
io-event (~> 1.1)
timers (~> 4.1)
async-container (0.16.13)
async
async-io
async-http (0.63.0)
async (>= 1.25)
async-io (>= 1.28)
async-pool (>= 0.2)
protocol-http (~> 0.26.0)
protocol-http1 (~> 0.18.0)
protocol-http2 (~> 0.16.0)
traces (>= 0.10.0)
async-http-cache (0.4.3)
async-http (~> 0.56)
async-io (1.41.0)
async
async-pool (0.4.0)
async (>= 1.25)
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 20 (10 by maintainers)
Thanks for all the logs and data I will review this week.
Tracking here: https://github.com/socketry/falcon/issues/289
The initial issues reported by @jakeonfire and @travisbell are caused by clients disconnecting before receiving the full response body. This leads to a “broken pipe”
Errno::EPIPEerror when the server attempts to write to a socket that the remote end has closed.In more detail: An
Errno::EPIPE(“Broken pipe”) error simply indicates that the client (remote peer) closed the TCP connection on their side mid-response. This typically happens if the client disconnects (e.g. by sending a TCP FIN or triggering a TCP reset) before it has received the full HTTP/1.1 response. In other words, the server attempted to write to the socket, but the underlying TCP stack reported that the remote side has gone away (the socket was closed by the peer, not by the server). Such broken-pipe errors are common in real-world web environments – for example, when a user navigates away from a page early or a proxy times out a slow request – and they generally do not indicate a bug in the server.Falcon mixes the
acceptcall with the event loop. If the event loop gets saturated, the accept call will be invoked less frequently. This leads to the least loaded falcon worker typically accepting the inbound connection first. This generally distributes connections evenly across a single container of falcon servers. However, you’ll still be depending on your load balancer to do the right thing if you have multiple containers on different network interfaces. In principle, falcon will eventually accept every connection that is given to it (so you are in principle correct), so it would be on the infrastructure to scale up once saturated. We now report a metric for this in the process title:The string is documented here. But in our case, we are interested in
L=...which indicates the scheduler load. If you see this getting above 0.8 for a sustained amount of time (e.g. more than a few seconds) you may be running close to saturation.On good hardware, Falcon can easily handle ~50,000 requests per second per event loop. So it’s unlikely that Falcon itself is going to cause saturation in typical scenarios, it will be your application code that is essentially getting overloaded. In that case, we recently introduced the
fiber-profilerto help detect poorly performing code which can stall the event loop. This isn’t the only class of bugs but it would be the biggest one and perhaps one of the easiest ones to address. Be aware that in it’s full tracing mode it can incur significant overhead, but if you can run it in a staging environment or shadow environment it will tell you any code paths which are causing contention on the event loop.It does sound like there are issues here but it’s not entirely clear the original issue and the follow up discussion are related. Therefore, as the original issue has been addressed, I’m going to close this issue, but if you feel that you are still having problems, please feel free to open a new issue and I am happy to continue the discussion.
(Also you might be interested in https://github.com/socketry/falcon/issues/154#issuecomment-2832911337 too).
The original issue posted by @michelson is with this code:
In the latest version of this code, we now have a state machine which validates the behaviour:
So the original issue should now at worst fail with a
Protocol::HTTP1::ProtocolErrorand ideally the issue is actually handled correctly earlier on as the state machine is much more comprehensive at dealing with client shenanigans.I think what @jakeonfire and I found while troubleshooting it further is that these errors come up when the server making the inbound request to Falcon closes the connection on a timeout. In our case we were catching external web hooks and doing an unreasonable amount of processing before responding with a 200. Since the processing was of no interest to the caller we moved it to an async block and started responding right away. I believe the errors stopped immediately after that.
It turns out there is often also the following error during these spikes:
It’s kind of hard to tell which error comes first though, or if they are even truly connected. I thought it might be helpful to see it as well though.
We just had one of these events I was referring to. For us, it looks like this:
And request times during this:
fwiw ours are fairly steady:
and there is no evidence that they are causing a bigger issue, strangely.
no, just Falcon on Gemfile. It also happened to me on a new rails app. steps: