yt-dlp: HTTPError 426: 'Upgrade Required' when downloading from CBC

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

I’m reporting a bug unrelated to a specific site
I’ve verified that I’m running yt-dlp version 2023.03.04 (update instructions) or later (specify commit)
I’ve checked that all provided URLs are playable in a browser with the same IP and same login details
I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
I’ve searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
I’ve read the guidelines for opening an issue

Provide a description that is worded well enough to be understood

Hello! I was downloading from CBC just fine for a few days and everything was fine. Suddenly all my CBC downloads started failing with the following error:

yt-dlp -o "C:\Users\...\yt-dlp\%%(channel)s - %%(title)s.%%(ext)s" https://gem.cbc.ca/media/the-fifth-estate/s41e09
[gem.cbc.ca] Extracting URL: https://gem.cbc.ca/media/the-fifth-estate/s41e09
[gem.cbc.ca] the-fifth-estate/s41e09: Downloading JSON metadata
ERROR: [gem.cbc.ca] the-fifth-estate/s41e09: Unable to download JSON metadata: HTTP Error 426: Upgrade Required (caused by <HTTPError 426: 'Upgrade Required'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U

yt-dlp -U

Available version: stable@2023.03.04, Current version: stable@2023.03.04
yt-dlp is up to date (stable@2023.03.04)

Provide verbose output that clearly demonstrates the problem

Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
If using API, add 'verbose': True to YoutubeDL params instead
Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

yt_dlp -vU -o "C:\<redacted>\yt-dlp\%%(channel)s - %%(title)s.%%(ext)s" https://gem.cbc.ca/media/the-fifth-estate/s41e09
[debug] Command-line config: ['-vU', '-o', 'C:\\<redacted>\\yt-dlp\\%%(channel)s - %%(title)s.%%(ext)s', 'https://gem.cbc.ca/media/the-fifth-estate/s41e09']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (source)
[debug] Lazy loading extractors is disabled
[debug] Git HEAD: 68be95bd0
[debug] Python 3.10.10 (CPython AMD64 64bit) - Windows-10-10.0.22621-SP0 (OpenSSL 1.1.1s  1 Nov 2022)
[debug] exe versions: none
[debug] Optional libraries: sqlite3-2.6.0
[debug] Proxy map: {}
[debug] Loaded 1791 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.03.04, Current version: stable@2023.03.04
yt-dlp is up to date (stable@2023.03.04)
[debug] Using fake IP 99.230.144.152 (CA) as X-Forwarded-For
[gem.cbc.ca] Extracting URL: https://gem.cbc.ca/media/the-fifth-estate/s41e09
[gem.cbc.ca] the-fifth-estate/s41e09: Downloading JSON metadata
ERROR: [gem.cbc.ca] the-fifth-estate/s41e09: Unable to download JSON metadata: HTTP Error 426: Upgrade Required (caused by <HTTPError 426: 'Upgrade Required'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "C:\<redacted>\yt_dlp\extractor\common.py", line 694, in extract
    ie_result = self._real_extract(url)
  File "C:\<redacted>\yt_dlp\extractor\cbc.py", line 352, in _real_extract
    video_info = self._download_json('https://services.radio-canada.ca/ott/cbc-api/v2/assets/' + video_id, video_id)
  File "C:\<redacted>\yt_dlp\extractor\common.py", line 1048, in download_content
    res = getattr(self, download_handle.__name__)(url_or_request, video_id, **kwargs)
  File "C:\<redacted>\yt_dlp\extractor\common.py", line 1012, in download_handle
    res = self._download_webpage_handle(
  File "C:\<redacted>\yt_dlp\extractor\common.py", line 882, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "C:\<redacted>\yt_dlp\extractor\common.py", line 839, in _request_webpage
    raise ExtractorError(errmsg, cause=err)

  File "C:\<redacted>\yt_dlp\extractor\common.py", line 821, in _request_webpage
    return self._downloader.urlopen(self._create_request(url_or_request, data, headers, query))
  File "C:\<redacted>\YoutubeDL.py", line 3743, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 525, in open
    response = meth(req, response)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 634, in http_response
    response = self.parent.error(
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 563, in error
    return self._call_chain(*args)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 496, in _call_chain
    result = func(*args)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 426: Upgrade Required

About this issue

Original URL
State: closed
Created a year ago
Comments: 17 (10 by maintainers)

Commits related to this issue

[extractor/cbc] Ignore 426 from API (#6781) Closes #6716 Authored by: jo-nike — committed to yt-dlp/yt-dlp by jo-nike a year ago

Most upvoted comments

I will investigate what the browser client does, as that is what is best to copy. But when I actually visit the URL, while it does return HTTP code 426 and have that non-standard Upgrade header, all the expected JSON metadata is there. I suspect if youtube-dl just ignored the HTTP code things would continue working fine.

Will look more into this later as mentioned above.

makew0rld on Apr 7, 2023

@makew0rld thank you, I for one will appreciate it.

cagenuts on Apr 7, 2023

I can confirm this issue, I will look into it. Worst case I will be free to explore this in a couple weeks.

makew0rld on Apr 7, 2023

@makew0rld I wanted to test your theory and was able to confirm that you are correct.

As a test I added expected_status = 426 here: https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/extractor/common.py#L881, recompiled my executable and confirmed it worked.

Before:

redacted@redacted:/mnt/c/Users/redacted/Downloads$ ./yt-dlp https://gem.cbc.ca/media/plan-b/s01e03 -v
[debug] Command-line config: ['https://gem.cbc.ca/media/plan-b/s01e03', '-v']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (zip)
[debug] Python 3.8.10 (CPython x86_64 64bit) - Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.29 (OpenSSL 1.1.1f  31 Mar 2020, glibc 2.31)
[debug] exe versions: ffmpeg 4.2.7, ffprobe 4.2.7
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, secretstorage-2.3.1, sqlite3-2.6.0, websockets-11.0.1
[debug] Proxy map: {}
[debug] Loaded 1786 extractors
[debug] Using fake IP 99.229.117.88 (CA) as X-Forwarded-For
[gem.cbc.ca] Extracting URL: https://gem.cbc.ca/media/plan-b/s01e03
[gem.cbc.ca] plan-b/s01e03: Downloading JSON metadata
ERROR: [gem.cbc.ca] plan-b/s01e03: Unable to download JSON metadata: HTTP Error 426: Upgrade Required (caused by <HTTPError 426: 'Upgrade Required'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "./yt-dlp/yt_dlp/extractor/common.py", line 694, in extract
    ie_result = self._real_extract(url)
  File "./yt-dlp/yt_dlp/extractor/cbc.py", line 349, in _real_extract
    video_info = self._download_json('https://services.radio-canada.ca/ott/cbc-api/v2/assets/' + video_id, video_id)
  File "./yt-dlp/yt_dlp/extractor/common.py", line 1048, in download_content
    res = getattr(self, download_handle.__name__)(url_or_request, video_id, **kwargs)
  File "./yt-dlp/yt_dlp/extractor/common.py", line 1012, in download_handle
    res = self._download_webpage_handle(
  File "./yt-dlp/yt_dlp/extractor/common.py", line 882, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "./yt-dlp/yt_dlp/extractor/common.py", line 839, in _request_webpage
    raise ExtractorError(errmsg, cause=err)

  File "./yt-dlp/yt_dlp/extractor/common.py", line 821, in _request_webpage
    return self._downloader.urlopen(self._create_request(url_or_request, data, headers, query))
  File "./yt-dlp/yt_dlp/YoutubeDL.py", line 3742, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 426: Upgrade Required

After:

redacted@redacted:/mnt/d/Code/yt-dlp$ ./yt-dlp https://gem.cbc.ca/media/plan-b/s01e03 -v
[debug] Command-line config: ['https://gem.cbc.ca/media/plan-b/s01e03', '-v']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (zip)
[debug] Python 3.8.10 (CPython x86_64 64bit) - Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.29 (OpenSSL 1.1.1f  31 Mar 2020, glibc 2.31)
[debug] exe versions: ffmpeg 4.2.7, ffprobe 4.2.7
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, secretstorage-2.3.1, sqlite3-2.6.0, websockets-11.0.1
[debug] Proxy map: {}
[debug] Loaded 1791 extractors
[debug] Using fake IP 99.235.152.163 (CA) as X-Forwarded-For
[gem.cbc.ca] Extracting URL: https://gem.cbc.ca/media/plan-b/s01e03
[gem.cbc.ca] plan-b/s01e03: Downloading JSON metadata
[gem.cbc.ca] plan-b/s01e03: Downloading JSON metadata
[gem.cbc.ca] plan-b/s01e03: Downloading m3u8 information
[gem.cbc.ca] plan-b/s01e03: Downloading secret XML
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] plan-b/s01e03: Downloading 1 format(s): sec-1080+hls-audio-English
[debug] Invoking hlsnative downloader on "https://cbcrcott-gem.akamaized.net/9a7e7072-ad0a-4e25-b860-2a9f40c8761a/CBC_PLANB_CBC_S01E03.ism/QualityLevels(5999851)/Manifest(video,format=m3u8-aapl)"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 438
[download] Destination: Episode 3 [plan-b⧸s01e03].fsec-1080.mp4
[download] 100% of    1.88GiB in 00:01:58 at 16.24MiB/s
[debug] Invoking hlsnative downloader on "https://cbcrcott-gem.akamaized.net/9a7e7072-ad0a-4e25-b860-2a9f40c8761a/CBC_PLANB_CBC_S01E03.ism/QualityLevels(192000)/Manifest(audio_eng_aacl,format=m3u8-aapl,filter=desktop)"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 435
[download] Destination: Episode 3 [plan-b⧸s01e03].fhls-audio-English.m4a
[download] 100% of   62.86MiB in 00:00:39 at 1.57MiB/s
[debug] ffmpeg command line: ffprobe -show_streams 'file:Episode 3 [plan-b⧸s01e03].fhls-audio-English.m4a'
[Merger] Merging formats into "Episode 3 [plan-b⧸s01e03].mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:Episode 3 [plan-b⧸s01e03].fsec-1080.mp4' -i 'file:Episode 3 [plan-b⧸s01e03].fhls-audio-English.m4a' -c copy -map 0:v:0 -map 1:a:0 -bsf:a:0 aac_adtstoasc -movflags +faststart 'file:Episode 3 [plan-b⧸s01e03].temp.mp4'
Deleting original file Episode 3 [plan-b⧸s01e03].fhls-audio-English.m4a (pass -k to keep)
Deleting original file Episode 3 [plan-b⧸s01e03].fsec-1080.mp4 (pass -k to keep)

jo-nike on Apr 11, 2023

~/.config/yt-dlp/plugins/extractor/yt_dlp_plugins/cbc.py should be ~/.config/yt-dlp/plugins/cbc_plugin/yt_dlp_plugins/extractor/cbc.py

bashonly on Apr 30, 2023

@jo-nike Brilliant! Thanks man.

cagenuts on Apr 11, 2023

My above message wasn’t necessarily a suggestion for a patch, was just a dirty confirmation of the theory.

This would be more gracious:

This line can be modified: https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/extractor/cbc.py#L352

from: video_info = self._download_json('https://services.radio-canada.ca/ott/cbc-api/v2/assets/' + video_id, video_id)

to video_info = self._download_json('https://services.radio-canada.ca/ott/cbc-api/v2/assets/' + video_id, video_id, expected_status=426)

jo-nike on Apr 11, 2023

So we need to upgrade the protocol sent from yt-dlp.

cagenuts on Apr 7, 2023