Flexget: Many HTTP 520 responses from rarbg

Expected behaviour:

rarbg plugin reliably returns search results.

Actual behaviour:

There seem to be two separate issues:

Occasionally the timeout between requests is too short (<2s), I believe their request rate is/was 1 req/2s.
Even when timeout is longer than 2s, rarbg returns 520 which is their way of saying there are too many requests.

Initially I thought that my VPN exit node might be too “popular” but one of the google searches lead me to https://github.com/Sonarr/Sonarr/issues/5206 which outlines much longer timeouts between requests for rarbg.

Log:

(click to expand)

2023-03-02 18:30:16 DEBUG    utils.requests download-tv-seasons-4k GETing URL https://torrentapi.org/pubapi_v2.php with args () and kwargs {'params': {'mode': 'search', 'ranked': 1, 'min_seeders': 0, 'min_leechers': 0, 'sort': 'seeders', 'category': '49', 'format': 'json_extended', 'app_id': 'flexget', 'search_string': b'Series 1 (2019) S04', 'token': 'removed'}, 'allow_redirects': True, 'timeout': 30}
2023-03-02 18:30:17 DEBUG    rarbg         download-tv-seasons-4k requesting: https://torrentapi.org/pubapi_v2.php?mode=search&ranked=1&min_seeders=0&min_leechers=0&sort=seeders&category=49&format=json_extended&app_id=flexget&search_string=removed&token=removed
2023-03-02 18:30:17 DEBUG    rarbg         download-tv-seasons-4k No results found for b'Series 1 (2019) S04'. Message from rarbg: No results found
2023-03-02 18:30:17 DEBUG    utils.requests download-tv-seasons-4k Waiting 1.74 seconds until next request to torrentapi.org
2023-03-02 18:30:19 DEBUG    utils.requests download-tv-seasons-4k GETing URL https://torrentapi.org/pubapi_v2.php with args () and kwargs {'params': {'mode': 'search', 'ranked': 1, 'min_seeders': 0, 'min_leechers': 0, 'sort': 'seeders', 'category': '49', 'format': 'json_extended', 'app_id': 'flexget', 'token': 'removed', 'search_string': b'removed'}, 'allow_redirects': True, 'timeout': 30}
2023-03-02 18:30:19 ERROR    rarbg         download-tv-seasons-4k Rarbg request failed: 520 Server Error:  for url: https://torrentapi.org/pubapi_v2.php?mode=search&ranked=1&min_seeders=0&min_leechers=0&sort=seeders&category=49&format=json_extended&app_id=flexget&token=removed&search_string=removed
2023-03-02 18:30:19 DEBUG    discover      download-tv-seasons-4k No results from rarbg
2023-03-02 18:30:19 VERBOSE  discover      download-tv-seasons-4k No search results for `Series 1 (2019) S04`
2023-03-02 18:30:19 DEBUG    series.db     download-tv-seasons-4k latest season pack for series <Series(id=114,name=Series 1 (2019))>, with downloaded set to True and season set to None
2023-03-02 18:30:19 DEBUG    series.db     download-tv-seasons-4k latest episode for series <Series(id=114,name=Series 1 (2019))>, with downloaded set to True and season set to 4
2023-03-02 18:30:19 VERBOSE  discover      download-tv-seasons-4k Searching for `Series 2 (2021) S02` with plugin `rarbg` (2 of 21)
2023-03-02 18:30:19 DEBUG    utils.requests download-tv-seasons-4k Waiting 2.76 seconds until next request to torrentapi.org
2023-03-02 18:30:22 DEBUG    utils.requests download-tv-seasons-4k GETing URL https://torrentapi.org/pubapi_v2.php with args () and kwargs {'params': {'mode': 'search', 'ranked': 1, 'min_seeders': 0, 'min_leechers': 0, 'sort': 'seeders', 'category': '49', 'format': 'json_extended', 'app_id': 'flexget', 'search_string': b'Series 2 (2021) S02', 'token': 'removed'}, 'allow_redirects': True, 'timeout': 30}
2023-03-02 18:30:22 ERROR    rarbg         download-tv-seasons-4k Rarbg request failed: 520 Server Error:  for url: https://torrentapi.org/pubapi_v2.php?mode=search&ranked=1&min_seeders=0&min_leechers=0&sort=seeders&category=49&format=json_extended&app_id=flexget&search_string=removed&token=removed

Additional information:

FlexGet version: 3.5.27

About this issue

Original URL
State: closed
Created a year ago
Reactions: 2
Comments: 17 (14 by maintainers)

Commits related to this issue

Radarr sonarr api v3 - Fix merge conflicts with develop (#3775) * Update docker/build-push-action action to v4 * Update irc to download tracker config correctly (#3678) [fix] irc: Update irc to... — committed to Flexget/Flexget by patrickegli a year ago
Update Sonarr/Radarr to use v3 api (#3051) * [fix] imdb_lookup: updated the scraper for the newish html structure * fix some tests * cassettes * ... * move to https as imdb now redirects ... — committed to Flexget/Flexget by stevezau a year ago
[change] rarbg: allow using domain_delay and increase default delay. fix #3705 Increase rarbg delay from 3s to 6s. Allow using domain_delay to override the built in delay for rarbg. Add an option to n... — committed to luizoti/Flexget by gazpachoking a year ago

Most upvoted comments

Enabling domain_delay to work was a bit easier than I thought. I’ve now increased the default to 6s, as well as allowed a custom delay to be set by domain_delay.

gazpachoking on Apr 10, 2023

That might fix the problem for some people but making the plugin obey the value set with domain_delay would probably be better as then people can adjust the delay as needed. Does flexget run requests concurrently? For VPN users this is probably never going to work as waiting a few seconds won’t be enough and waiting minutes between requests means tasks running for hours. Impractical to say the least.

girtsl on Apr 10, 2023