requests: timeout issues for chunked responses

When I set a timeout value for my simple GET request, the requests.get call takes a really long time to complete when I don’t iterate over the returned content. If I iterate over the content, the requests.get call comes back very quickly. I’ve delved pretty deep into the requests source code and can’t figure out what the issue is.

Here is a test case to illustrate my issue: https://gist.github.com/zsalzbank/90ebcaaf94f6c34e9559

What I would expect to happen is that the non-streamed and non-iterated responses would return just as quickly as all the other tests, but instead, they take 25x longer. When the timeout is removed, the responses come back quickly as well.

From my understanding of the timeout parameter, it is only timing out for the initial data response from the server. I know that some data comes back quickly because I modified the requests library and printed out the data as soon as it comes back (added a print after this line: https://github.com/kennethreitz/requests/blob/master/requests/models.py#L655).

What is going on here? Am I doing something wrong on the server side? I know that no Content-Length comes back from the server, but Connection: close does come back. I’m not sure if this is related to #1041 (it sounds similar).

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Comments: 41 (31 by maintainers)

Most upvoted comments

@zsalzbank To be clear, the fundamental bug is absolutely with the server: it just turns out that mostly Python handles it pretty well. The server should do one of three things:

  1. Use Transfer-Encoding: chunked, and send the empty chunk to terminate the data stream. We can then see that the server sent Connection: close, and close the connection.
  2. Use Content-Length, and we can then close once we’ve read that content.
  3. Send neither, and the server has to close the connection.

None of the above is happening.