asgiref: Exceptions from `send`, and expected behaviors on closed connections.
Before digging into the meat of this topic I want to start with this assumption:
- Exceptions from
send
andreceive
should be treated as opaque, and should be allowed to bubble back up to the server in order to be logged. They may be intercepted by logging or 500 technical response middlewares, and subsequently re-raised, but applications shouldn’t have any expectations about the type that may be raised, since these will differ from implementation to implementation.
That’s pretty much how the spec is currently worded, and I think that’s all as it should be. An alternative design could choose to have exceptions be used in a limited capacity to communicate information back to the application, but I don’t think that’s likely to be desirable. An exception raised by the server in send
or receive
is a black box indicating an error condition with an associated traceback, and we shouldn’t expect to catch or handle particular subclasses in differing ways.
The core question in this issue is: 1. What expectations (if any) should the application have about the behavior when it makes a send()
to a closed connection? 2. What behavior is most desirable from servers in that case?
There are three possible cases for the application here:
- The application should expect an exception to be raised, indicating that the connection is closed.
- The application should expect that no exception be raised.
- An exception may or may not be raised, but the application cannot rely on either behavior.
In the case of HTTP, we’ve adopted “Note messages received by a server after the connection has been closed are not considered errors. In this case the send awaitable callable should act as a no-op.” See https://github.com/django/asgiref/pull/49
I don’t recall where the conversation that lead to #49 is, but I think it’s a reasonable behaviour because premature HTTP client disconnects once receiving the start of the response are valid behaviour, and not an error condition. If we raise an exception in that case then we end up with lots of erroneous exceptions being raised in response to perfectly valid server and client behavior.
If we’re happy with that decision then there’s two questions that it leads on to:
- What do we expect to happen in long-polling / SSE connections if
send
doesn’t raise an exception once the connection has been closed? - What should we expect in the case of websockets?
For the case of (1) I think that #49 means we have to have the expectation that the disconnect is communicated through the receive
disconnect message. No exception should be raised from the server on send
to the closed connection because that’d pollute server logs with false errors in the standard case, and we can’t differentiate between long-polling connections and regular HTTP requests/responses.
Mature server implementations probably would want to enforce a timeout limit on the task once the connection has been closed, and that is the guardrail against applications that fail to properly listen for and act on the disconnect.
For the case of (2) it’s less obvious. We could perfectly well raise exceptions in response to send
on a disconnected connection. (Which I assume is what all current implementations do.) The issue here is that we’d be raising logged error conditions on what is actually perfectly correct behavior… there’s no guarantee that an application will have yet “seen” the disconnect message at the point it sends any given outgoing message.
The upshot of all this is that I think the following would be the right sort of recommendations to make:
- Applications must not assume that an exception will necessarily be raised when sending to a closed connection, or rely on the server doing so. They should listen to a disconnect message and terminate the task when that occurs.
- Mature server implementations should prefer not to raise exceptions when a send occurs against a closed websocket, but should instead ignore the message, ensure that
disconnect
is sent on the “receive” channel, and should forcibly kill the task if it has not completed after a reasonable timeout. Server implementations that do raise exceptions in response tosend
on a closed channel are valid, but may end up falsely logging server errors in response to standard client disconnects.
Anyways, thorny issue, and more than happy to talk this through more, but I’m moderately sure that’s the conclusion I’d come to.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 3
- Comments: 49 (49 by maintainers)
Oh absolutely yes. Sending invalid state transitions should raise errors.
Good point, yes.
I don’t think we necessarily need to strongarm “servers must not do this”, more that “server implementations should prefer not to do this, and instead prefer to guard against poor app behavior with an enforced timeout after disconnect” (A server implementation could validly raise an error if tasks fail to terminate within the timeframe after a disconnect.)
At some point I’ll aim to have a little dig into daphne, hypercorn, and uvicorn’s current behavior here, since I doubt it’s in line with this proposal. (Paging @pgjones)
I totally agree that it’s just going to happen, due to the async nature of the code in question, that you’re going to send just after the socket is disconnected in any case - even if you have a perfect disconnect handling loop. I tell client code writers to write code that expects disconnects at any time, we should be thinking the same way.
My questioning of the problem before was around safety - is it better for us to suppress those errors and maybe let bugs go uncaught? Initially I was erring on the side of not suppressing errors, which tends to be my default, but now I think I would agree that if we didn’t suppress them, people would just see these errors randomly and they’d become ignored noise, which is perhaps even more dangerous.
Given that, I’m more than happy to enshrine in the spec that “send should not raise errors if used after disconnect”. The only exception I might carve out is if the server sent a disconnect packet and then sent something else, but that’s more a nice thing that mature servers could maybe raise a warning over rather than an edge case I want to specify out.
Well, the problem with that originally was that I wanted the entire body of asgi messages to be JSON-safe, as I had intended to do a process model where the process terminating the sockets was separate to the one running Django/etc.
I think that’s still a valuable property to try and maintain - it means you can prefork and have warm worker processes more easily, for example.
I don’t really know if we can get actually good user research on this, so I’m OK going with the backwards compatible solution if we can correctly specify what the exception should be. I’d suggest “a unique subclass of IOError specified by the server”, so that user applications can catch IOError but the server can catch its own specific subclass.
@andrewgodwin we can make the HTTP version non-breaking if the server raises and catches the exception. That is, given any of these scenarios:
An application could, in theory, still try to
send()
after a client disconnection has happened, so I think it can happen with HTTP as well?Yes, and I appreciate you helping me work through this!