urllib3: trying to close a response hangs if a blocking read is happening on another thread

Subject

Thread A starts Thread B to read a chunked HTTP response. At some point, regardless of whether the server has more data, I want to break the connection from the client side. Thread B is blocked trying to read the socket, so I’m trying to call HTTPResponse.close() from Thread A. This call hangs.

I’m not sure if this is a urllib3 issue, or a limitation of the underlying Python synchronous I/O model, or something I am doing wrong.

Environment

  • Python 3.7.13, 3.10.6
  • platform: Darwin-22.2.0-x86_64-i386-64bit, Linux-5.10.104-linuxkit-x86_64-with-debian-10.13
  • urllib3 1.26.13

Steps to Reproduce

Unzip this archive and run main.py.

urllib3-close.zip

This starts a simple HTTP server on a daemon thread; makes a request to it from another thread, which receives a little bit of data and then blocks waiting for more; then, from the main thread, tries to close the response.

Expected Behavior

I hoped that closing the response would cause the underlying socket to be closed, causing the client thread to get a read error and terminate. For this demo app, the output would look like this:

main [MainThread] starting server
  server [Thread-1] listening on port 10000
main [MainThread] starting client
    client [Thread-2] opening connection
main [MainThread] sleeping a while
  server [Thread-1] starting stream
127.0.0.1 - - [05/Jan/2023 13:38:11] "GET / HTTP/1.1" 200 -
  server [Thread-1] wrote 10 bytes
    client [Thread-2] reading data
    client [Thread-2] read 10 bytes
main [MainThread] about to close response
main [MainThread] closed response
    client [Thread-2] failed: <some kind of I/O error>
    client [Thread-2] no longer reading
main [MainThread] about to clear connection pool
main [MainThread] cleared connection pool
main [MainThread] exiting

Actual Behavior

The HTTPResponse.close() call hangs. The output looks like this, and then the process waits indefinitely:

main [MainThread] starting server
  server [Thread-1] listening on port 10000
main [MainThread] starting client
    client [Thread-2] opening connection
main [MainThread] sleeping a while
  server [Thread-1] starting stream
127.0.0.1 - - [05/Jan/2023 13:38:11] "GET / HTTP/1.1" 200 -
  server [Thread-1] wrote 10 bytes
    client [Thread-2] reading data
    client [Thread-2] read 10 bytes
main [MainThread] about to close response

The only way I have found to make it not hang is to make the server send some more data; then, the blocking read on the client read unblocks, and the client is able to notice that the HTTPResponse is no longer in a valid state. But then that no longer corresponds to the use case I’m talking about, where the server does not have more data yet but I still want to break the connection.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 1
  • Comments: 18 (4 by maintainers)

Commits related to this issue

Most upvoted comments

Right, I read your comment when you posted it but then mostly ignored the thread and started again from first principles. It’s a good sign that we landed on the same solution!

@eight04, @pquentin et al.

I did some more testing, and it looks like the problem is the underlying io.BufferReader isn’t realizing the socket is closed, so it’s waiting on that.

In previous comments, we had mentioned trying

    socket.close(resp._fp.fileno())

but that didn’t allow us to re-use the connection pool. I think calling

resp._fp.fp.raw._sock.shutdown(socket.SHUT_RDWR)

works. At least my local testing seems to indicate it is working. However, I have no idea what sort of ramifications there might be to doing something like this. Also I’m reaching deep into implementation details, so this is also clearly a fragile solution.

Can someone with more expertise on this repository take a look and let me know if this is a viable solution to the problem, and if we can get this fix introduced into the official library somewhere?

@eight04 your example works but isn’t quite the same as issue we are experiencing.

In your example, the t.get() call is going to block until the r.stream throws an exception. And that doesn’t occur until the server tries to send some new bytes down the line and then the stream moves on to the NEXT iteration.

If I replace the httpbin example with an SSE server that sends some initial payload but then waits a minute before sending the next payload, your example will block until that minute has lapsed and the new data has been pushed down.

Is there a way we can avoid this blocking behavior?

Here’s something I just came up with that appears to work, but it’s pretty hacky and I’m not sure how portable it is:

import socket

def force_close_http_response(resp: HTTPResponse):
    socket.close(resp._fp.fileno())

When I substitute this for resp.close(), it has the desired effect of immediately closing the socket so that the current blocking read errors out.