youtube-dl: Unable to download JSON metadata error on yt

Checklist

  • I’m reporting a broken site support issue
  • I’ve verified that I’m running youtube-dl version 2021.02.22 – youtube-dl is up-to-date (2021.02.22)
  • I’ve checked that all provided URLs are alive and playable in a browser
  • I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
  • I’ve searched the bugtracker for similar bug reports including closed ones
  • I’ve read bugs section in FAQ

Verbose log

[debug] System config: []
[debug] User config: ['--download-archive', 'archive.txt', '-i', '-w', '-c', '--no-mtime', '--restrict-filenames', '--prefer-ffmpeg', '--min-sleep-interval', '2', '--max-sleep-interval', '20', '-o', '%(title)s.%(ext)s']
[debug] Custom config: []
[debug] Command-line args: ['--download-archive', 'foodwishes.txt', '--verbose', '-i', '-q', '--no-warnings', '--restrict-filenames', '--add-metadata', '--min-sleep-interval', '1.5', '--max-sleep-interval', '7', '-o', '%(title)s.%(ext)s', 'https://www.youtube.com/user/foodwishes']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.02.22
[debug] Python version 3.8.3 (CPython) - Linux-4.18.0-240.10.1.el8_3.x86_64-x86_64-with-glibc2.2.5
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4
[debug] Proxy map: {}
ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 632, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2275, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib64/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib64/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib64/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib64/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 632, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2275, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib64/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib64/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib64/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib64/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

Description

Simple download of a fav playlist from yt. It has been working for years and after updating to 02.22, it seems to have broken this morning.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 3
  • Comments: 17 (4 by maintainers)

Commits related to this issue

Most upvoted comments

Hi, sorry to comment on a closed issue, but I’m seeing a similar issue with recent updates (2021.03.02) as well:

verbose output
❯ mpv https://www.youtube.com/playlist\?list\=PLAbMhAYRuCUhawCEV2oXZGrienoKTN16X --ytdl-raw-options=verbose=true
[ytdl_hook] [debug] System config: []
[ytdl_hook] [debug] User config: []
[ytdl_hook] [debug] Custom config: []
[ytdl_hook] [debug] Command-line args: ['--no-warnings', '-J', '--flat-playlist', '--sub-format', 'ass/srt/best', '--format', 'bestvideo[height<=?1080][fps<=?60][vcodec!=?vp9]+bestaudio/best', '--verbose', 'true', '--all-subs', '--no-playlist', '--', 'https://www.youtube.com/playlist?list=PLAbMhAYRuCUhawCEV2oXZGrienoKTN16X']
[ytdl_hook] [debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[ytdl_hook] [debug] youtube-dl version 2021.03.02
[ytdl_hook] [debug] Python version 3.9.2 (CPython) - macOS-10.14.6-x86_64-i386-64bit
[ytdl_hook] [debug] exe versions: ffmpeg 4.3.2, ffprobe 4.3.2, rtmpdump 2.4
[ytdl_hook] [debug] Proxy map: {}
[ytdl_hook] ERROR: 'true' is not a valid URL. Set --default-search "ytsearch" (or run  youtube-dl "ytsearch:true" ) to search YouTube
[ytdl_hook] Traceback (most recent call last):
[ytdl_hook]   File "/usr/local/Cellar/youtube-dl/2021.3.2/libexec/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 806, in wrapper
[ytdl_hook]     return func(self, *args, **kwargs)
[ytdl_hook]   File "/usr/local/Cellar/youtube-dl/2021.3.2/libexec/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 827, in __extract_info
[ytdl_hook]     ie_result = ie.extract(url)
[ytdl_hook]   File "/usr/local/Cellar/youtube-dl/2021.3.2/libexec/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 532, in extract
[ytdl_hook]     ie_result = self._real_extract(url)
[ytdl_hook]   File "/usr/local/Cellar/youtube-dl/2021.3.2/libexec/lib/python3.9/site-packages/youtube_dl/extractor/generic.py", line 2381, in _real_extract
[ytdl_hook]     raise ExtractorError(
[ytdl_hook] youtube_dl.utils.ExtractorError: 'true' is not a valid URL. Set --default-search "ytsearch" (or run  youtube-dl "ytsearch:true" ) to search YouTube
[ytdl_hook] youtube-dl failed: unexpected error occurred
Failed to recognize file format.

Exiting... (Errors when loading file)

Then I tried setting --default-search=ytsearch as suggested and got a little further but still seeing the same error:

❯ mpv https://www.youtube.com/playlist\?list\=PLAbMhAYRuCUhawCEV2oXZGrienoKTN16X --ytdl-raw-options=verbose=true,default-search=ytsearch

Playing: https://youtu.be/Al_Rm2jEOLU
[ytdl_hook] failed to parse JSON data: error
Failed to recognize file format.

If you add retries for the 404 in most causes it seems to eventually sort itself out, so I’m thinking this is probably server side. Wonder if this has any similarity to #28194… Edit: though right now there seems to be an influx in 404s, where I’m trying hundreds of retries while in the browser there’s no problem.

I’m using the latest EXE version 2021.03.02, same error with version 2021.02.22.

$ youtube-dl.exe https://www.youtube.com/c/EHPMusicChannelII/videos --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.youtube.com/c/EHPMusicChannelII/videos', '--verbose']
[debug] Encodings: locale cp1252, fs mbcs, out cp65001, pref cp1252
[debug] youtube-dl version 2021.03.02
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: ffmpeg git-2020-01-02-81172b5, ffprobe git-2020-01-02-81172b5, phantomjs 2.5.0
[debug] Proxy map: {}
[youtube:tab] EHPMusicChannelII: Downloading webpage
[download] Downloading playlist: EHPMusicChannel - Videos
[youtube:tab] Downloading page 1
ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmp_qzp9er5\build\youtube_dl\extractor\common.py", line 632, in _request_webpage
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmp_qzp9er5\build\youtube_dl\YoutubeDL.py", line 2275, in urlopen
  File "C:\Python\Python34\lib\urllib\request.py", line 470, in open
  File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response
  File "C:\Python\Python34\lib\urllib\request.py", line 508, in error
  File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
  File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default

I can confirm, everything goes through this new front-end API endpoint now (JSON-based). Subscriptions, channel feed etc.

https://www.youtube.com/youtubei/v1/browse?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8 (note the key is exactly the same as above)

This might not be server side, rather an API change (or YouTube doing A/B testing to remove the old API?)

I notice that youtube-dl is accessing https://www.youtube.com/browse_ajax?ctoken=XYZ for getting the next page, while YouTube in the browser is sending a POST request to https://www.youtube.com/youtubei/v1/browse?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8 with the continuation token in the payload.

Testing with the latter in using the _call_api function in the YouTube extractor seems to work without 404s.

Unfortunately it has been hard to reproduce to debug this up until now.

Confirmed that my copy was working fine a few hours ago, and now is 404 every time. Let me know if there’s any debugging I can assist with.