Checklist
Verbose log
[debug] System config: []
[debug] User config: ['-o', '%(title)s.%(ext)s']
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.reddit.com/r/memes/comments/ppbsg1/skyrim_be_like/']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.06.06
[debug] Python version 3.9.7 (CPython) - macOS-10.15.7-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.4, ffprobe 4.4, rtmpdump 2.4
[debug] Proxy map: {}
[RedditR] ppbsg1: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 429: Too Many Requests (caused by <HTTPError 429: 'Too Many Requests'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "/usr/local/Cellar/youtube-dl/2021.6.6/libexec/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 634, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/usr/local/Cellar/youtube-dl/2021.6.6/libexec/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 2288, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 523, in open
response = meth(req, response)
File "/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 632, in http_response
response = self.parent.error(
File "/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 561, in error
return self._call_chain(*args)
File "/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 494, in _call_chain
result = func(*args)
File "/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/urllib/request.py", line 641, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
Description
- Reddit now returns
429 Too Many Requests for any video downloading over youtube-dl. Not affected by rate-limiting, as even the first video downloaded over youtube-dl will fail immediately.
- The video plays completely fine in browser both while logged in and on fresh private tabs, so this is likely related to Reddit detecting
youtube-dl.
- Using various headers (user agent etc.) to imitate a browser has had no effect.
- Other generic downloading programs like
wget return 200 OK.
@89z your choice of words is inappropriate and hurtful. You’re free to disagree with other, but please consider your language when making comments like that.
@89z I’m not sure why you’re talking about a filter here, i’m just following this thread. You’re right that the original message could have been worded a bit better, but i do agree with the gist of the post. I’m sure your tool is useful as well, but “use another tool” is not a constructive solution to this particular issue.
What? Nobody wants to go visit another Github page to decipher your post.
Another easy workaround is to use the
old.reddit.comURI; whilefails, the conversion to
https://old.reddit.com/r/Justrolledintotheshop/comments/k1ov8f/customer_installed_the_wrong_battery_and_slammed/succeeds:block 89z right now don’t hear to that
@89z i’m not a maintainer of this repo, but i’m free to comment when i believe something is against good taste and respectful manners. Seeing that the comment in question has been removed (either by you or somebody else) i suppose i’m not the only one.
Working on a PR that’ll just generate random text and set it as the cookie. Seems to work.
Also, sites may be fronted through Cloudflare-like services that may apply different opaque policies to police traffic, depending on location, etc.
Regarding the Reddit extractor, as I pointed out before, it’s easy to identify since it gets a URL not used in a browser session (at least, not now), and that’s the one that gets 429-ed.
wow i am stupid, here I thought I was at the github page for yt-dlp lol. thanks for the heads up.
#30839 :
😜
The extractor tries to load JSON that the browser doesn’t appear to fetch, and yet it still works:
Presumably the failures seen by reporters above would be bypassed if the extractor instead used the JSON sent in the main page in this element:
<script id="data">window.___r = {"pages": ...};</script>This is the
.pages.modelsobject in that JSON:Only working workaround that I have right now is to extract the
.m3u8playlist URL directly from the browser. Inspect Element on the video player, find thedata-hls-urlproperty of thevideo-*div and pass it toyoutube-dlafter stripping HTTP parameters. Reddit does not seem to be filtering requests to direct resources.By removing lines from cookies.txt until the command failed, I have determined that the only necessary cookie is
reddit_session. Have that cookie and the download works–no matter what you set it to, as far as I can tell.Temporary workaround: