twarc: 503 Error with search/all
I noticed when searching with --archive that I occasionally am getting this in the log:
2021-04-29 10:14:07,737 WARNING 503 from Twitter API, sleeping 60
I think this is related?
https://twittercommunity.com/t/perpetual-503-v2-full-archive-search-default-vs-all-fields/151441
Hopefully it’s a bug that will be fixed on Twitter’s end. But if not we may need a way to dial down the max_results.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 18 (10 by maintainers)
@edsu in this particular search it was 5273637 tweets in a two year time period.
@edsu it worked. Thank you for your interest! I had to wait quite a while, but in the end I got what I wanted. So probably --max-results 100 increases the probability that one gets what he wants, but owing to Twitter this probability is only close to 1 and depends on pure luck. One should not get discouraged after initial failure, only try for a couple of days.
it was just a single word.
I haven’t been seeing this recently, so I’m gonna close it and hopefully it won’t pop up again.
I’m going to leave this open to track what’s going on Twitter. But there is a
--max-resultsoption for search in v2.0.10. It does seem that dialing that back to 100 when using--archiveworks.