scrapy: Adjust throttling for 429 response codes

HTTP 429 response code is returned when we reach a rate limit for an API at a given time. Usually, it is a matter of waiting a bit to start sending new requests. The “problem” is that, if concurrency settings are greater than the allowed number of requests from the API, we’ll always get 429s.

A solution would be to tune throttling so it delays requests based on 429s. It can be a extension/middleware, as AutoThrottle seems quite specific for throttle control over latency.

Also it could be worth considering that some APIs return the waiting time https://github.com/scrapy/scrapy/issues/3849

Here is a previous PR for this https://github.com/scrapy/scrapy/pull/3061

About this issue

  • Original URL
  • State: open
  • Created 7 years ago
  • Comments: 15 (7 by maintainers)

Most upvoted comments

@Gallaecio I think a per-domain basis would be more fitting for my use-case. However, with #5015 close to being approved (I’ve been eagerly following that discussion), I do believe that functionality will make it easier to account for status codes for throttling by subclassing the existing AutoThrottle middleware. Though, I feel this functionally would fall under a common dilemma, thus calling for an official middleware to address it.

Hi, I want scrapy to automatically tune itself based on “please slow down” messages from the server. I think this ticket is what I want. It has been idle for some months. I’m too dumb to understand vortexkd’s branch, so I implemented “bare minimum” functionality:

class TooManyRequestsPleaseSlowDownMiddleware:
    """Work around https://github.com/scrapy/scrapy/issues/4424 (no 429 Please Slow Down backoff)

    NOTE: ``raise CloseSpider`` only works in Spider (not DownloaderMiddleware) classes?
          Calling crawler.engine.close_spider seems to work;
          I stole it from scrapy.extensions.closespider.
    """

    def process_response(self, request, response, spider):
        if response.status == {420, 429, 430}:
            spider.crawler.engine.close_spider(
                spider,
                "Server said {response.status} (Please Slow Down)."
                " Too stupid to slow down, so stopping instead.")
        return response

And enable it in settings.py like this (run before RetryMiddleware):

DOWNLOADER_MIDDLEWARES = {
    'argent_labrose.downloadermiddlewares.TooManyRequestsMiddleware': 549,
}

I think this is working for me. Is this useful to anyone?

PS: see also: