httpx: Overall response timeout.
- The bug is reproducible against the latest release and/or
master. - There are no similar issues or pull requests to fix it yet.
Currently both async and sync client requests are succeptible to low-and-slow “attacks” unless explicitly handled. In other words when request is being streamed as long as 1 byte every 2 seconds (or aprox.) is being generated the connection will not close or timeout with none of the current available settings.
It’s possible for the server to serve single html page almost idefinitely and hang the client thread.
Maybe httpx should introduce some optional or even default handlers for this? Maybe it’s already possible to hook something like this in easily?
To reproduce
resp = httpx.get(
"http://httpbin.org/drip?duration=30&numbytes=30&code=200&delay=2",
timeout=Timeout(connect=3, read=3, write=3, pool=3, timeout=3),
)
The above code snippet will take 30+ seconds to complete (though this could be practically infinity) disregarding any timeout settings. The only way to avoid this is to explitictly wrap everything in either asyncio.wait_for() for async code and for sync code seems to be much more complicated (?).
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 6
- Comments: 23 (10 by maintainers)
It’s just that slowlorris makes sense as a client-side attack on the server, allowing attackers to DDOS a server. Whereas in the converse case, where the server has been owned by the attacker, it’s an odd scenario to envisage an “attack” that is “let’s make the clients using this service really slow”. Maybe that’s a thing that’s happened(?), but I’ve never heard of it.
You’d more typically see simply this kind of issue on the client side framed as “resource limits to ensure graceful degradation vs. overloaded servers”.
Anyways, I think it’s valid either ways around, and I think a useful guideline for considering resource limits is to think about it in the same way as services like Heroku treat resource limiting.
The cases that seem like likely candidates to me might be:
Heya, thanks for raising this.
I’d potentially be a bit cautious about labelling this under “attacks”/“security”, since controlling the server and attacking the client isn’t a very typical scenario, but yes I do think there’s scope for a few more resource limits, which I’ve so far only noted in passing.
pause - I’m going to follow up on this more in a moment, but first lunch is up…
@gsakkis You’re right, the other URL “http://httpbin.org/drip” is even better for testing the total timeout. But even with “https://httpbin.org/delay/3” the normal httpx timeout takes around 1.4s instead of 1.0s, while
wait_forterminates after 1.0s.@falkoschindler you don’t need
asyncio.wait_forfor this url, the httpx timeout (or read timeout) is sufficient. An example where it’s not sufficient is the one in the original post, where the response is returned slowly with a short interval between the chunks.Hello everyone! I stumbled over this issue while looking for an option to timeout our requests to ensure we are not waiting longer for a response than absolutely necessary. For us something like
timeout = httpx.Timeout(total=1.0)would also work to achieve this. Will something like this or like @rthalley described be implemented in an upcoming release?I like this proposal, it’s good to be able to limit the resources you can expend on a task (no so much for security as for coping with the amazing variety of brokenness you encounter on the Internet). My sense is that the response timeout should probably be off by default as it’s really hard to guess what a good timeout would be, especially for HTTP. Someone might be getting some massive document over a slow connection, and any “default to on” behavior might break existing code. I think giving people a mechanism they can activate and specify values appropriate to their situation is enough.
One default-related thing I do think is worth discussing is what is the meaning of
client = httpx.Client(timeout = 10)You could argue that for backwards compatibility that this means “10 second idle timeout for all operations, no overall response timeout”, or you could argue “least surprise” and say it means a 10 second value for all timeouts, and thus a max block time of 10 seconds. I’m not sure which way to go on this one, as I don’t know how much code out there is expecting the timeout to be only an idle timeout because they read the documentation carefully, and how much code out there made the same mistake we did in dnspython by assuming it was an overall request timeout 😃.