yt-dlp: HiDive Extractor broken after site update
DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE
- I understand that I will be blocked if I intentionally remove or skip any mandatory* field
Checklist
- I’m reporting that yt-dlp is broken on a supported site
- I’ve verified that I have updated yt-dlp to nightly or master (update instructions)
- I’ve checked that all provided URLs are playable in a browser with the same IP and same login details
- I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
- I’ve searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
- I’ve read the guidelines for opening an issue
- I’ve read about sharing account credentials and I’m willing to share it if required
Region
No response
Provide a description that is worded well enough to be understood
HiDive was updated earlier today which appears to be a complete rework of the site (or at least their URLs). Below I’ve included two logs, one using the old format and one using the new format. I’m not expecting the new one to actually work but included it anyway in case there’s anything interesting.
For example these two URLs are the same video, the first being the old format and the second being how it appears after the update:
https://www.hidive.com/stream/chained-soldier/s01e010
https://www.hidive.com/video/590403?seasonId=20685
Provide verbose output that clearly demonstrates the problem
- Run your yt-dlp command with -vU flag added (
yt-dlp -vU <your command line>
) - If using API, add
'verbose': True
toYoutubeDL
params instead - Copy the WHOLE output (starting with
[debug] Command-line config
) and insert it below
Complete Verbose Output
yt-dlp -f "(b[language=ja][height<=1080])" "https://www.hidive.com/stream/chained-soldier/s01e010" --cookies-from-browser firefox -o "%(series)s_S%(season_number)s_E%(episode_number)s_%(episode)s.%(ext)s" --embed-subs --sub-lang english-subs --sub-format vtt --write-subs -N 64 --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0" -vU
[debug] Command-line config: ['-f', '(b[language=ja][height<=1080])', 'https://www.hidive.com/stream/chained-soldier/s01e010', '--cookies-from-browser', 'firefox', '-o', '%(series)s_S%(season_number)s_E%(episode_number)s_%(episode)s.%(ext)s', '--embed-subs', '--sub-lang', 'english-subs', '--sub-format', 'vtt', '--write-subs', '-N', '64', '--user-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0', '-vU']
Extracting cookies from firefox
[debug] Extracting cookies from: "/home/user/.mozilla/firefox/ci8h4s4e.default-release/cookies.sqlite"
Extracted 477 cookies from firefox
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.12.30 from yt-dlp/yt-dlp [f10589e34] (pip)
[debug] Python 3.11.7 (CPython x86_64 64bit) - Linux-6.7.6-arch1-1-x86_64-with-glibc2.39 (OpenSSL 3.2.1 30 Jan 2024, glibc 2.39)
[debug] exe versions: ffmpeg 6.1.1 (setts), ffprobe 6.1.1
[debug] Optional libraries: Cryptodome-3.18.0, brotli-1.0.9, certifi-2023.07.22, mutagen-1.46.0, requests-2.31.0, sqlite3-3.45.1, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1798 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: stable@2023.12.30 from yt-dlp/yt-dlp
yt-dlp is up to date (stable@2023.12.30 from yt-dlp/yt-dlp)
[HiDive] Extracting URL: https://www.hidive.com/stream/chained-soldier/s01e010
[HiDive] chained-soldier/s01e010: Downloading JSON metadata
ERROR: [HiDive] chained-soldier/s01e010: chained-soldier/s01e010: Failed to parse JSON (caused by JSONDecodeError("Expecting value in '': line 1 column 1 (char 0)")); please report this issue on https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using yt-dlp -U
File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 718, in extract
ie_result = self._real_extract(url)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/extractor/hidive.py", line 76, in _real_extract
settings = self._call_api(video_id, title, key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/extractor/hidive.py", line 70, in _call_api
return self._download_json(
^^^^^^^^^^^^^^^^^^^^
File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 1072, in download_content
res = getattr(self, download_handle.__name__)(url_or_request, video_id, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 1042, in download_handle
return parse(self, content, video_id, transform_source=transform_source, fatal=fatal, errnote=errnote), urlh
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 1032, in parse
return getattr(ie, parser)(content, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 1019, in _parse_json
self.__print_error('Failed to parse JSON' if errnote is None else errnote, fatal, video_id, ve)
File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 1002, in __print_error
raise ExtractorError(f'{video_id}: {errnote}', cause=err)
File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/utils/_utils.py", line 553, in decode
return super().decode(s)
^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 1016, in _parse_json
return json.loads(
^^^^^^^^^^^
File "/usr/lib/python3.11/json/__init__.py", line 359, in loads
return cls(**kw).decode(s)
^^^^^^^^^^^^^^^^^^^
File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/utils/_utils.py", line 561, in decode
raise type(e)(f'{e.msg} in {s[e.pos - 10:e.pos + 10]!r}', s, e.pos)
json.decoder.JSONDecodeError: Expecting value in '': line 1 column 1 (char 0)
New URL:
yt-dlp -f "(b[language=ja][height<=1080])" "https://www.hidive.com/video/590403?seasonId=20685" --cookies-from-browser firefox -o "%(series)s_S%(season_number)s_E%(episode_number)s_%(episode)s.%(ext)s" --embed-subs --sub-lang english-subs --sub-format vtt --write-subs -N 64 --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0" -vU
[debug] Command-line config: ['-f', '(b[language=ja][height<=1080])', 'https://www.hidive.com/video/590403?seasonId=20685', '--cookies-from-browser', 'firefox', '-o', '%(series)s_S%(season_number)s_E%(episode_number)s_%(episode)s.%(ext)s', '--embed-subs', '--sub-lang', 'english-subs', '--sub-format', 'vtt', '--write-subs', '-N', '64', '--user-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0', '-vU']
Extracting cookies from firefox
[debug] Extracting cookies from: "/home/user/.mozilla/firefox/ci8h4s4e.default-release/cookies.sqlite"
Extracted 477 cookies from firefox
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.12.30 from yt-dlp/yt-dlp [f10589e34] (pip)
[debug] Python 3.11.7 (CPython x86_64 64bit) - Linux-6.7.6-arch1-1-x86_64-with-glibc2.39 (OpenSSL 3.2.1 30 Jan 2024, glibc 2.39)
[debug] exe versions: ffmpeg 6.1.1 (setts), ffprobe 6.1.1
[debug] Optional libraries: Cryptodome-3.18.0, brotli-1.0.9, certifi-2023.07.22, mutagen-1.46.0, requests-2.31.0, sqlite3-3.45.1, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1798 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: stable@2023.12.30 from yt-dlp/yt-dlp
yt-dlp is up to date (stable@2023.12.30 from yt-dlp/yt-dlp)
[generic] Extracting URL: https://www.hidive.com/video/590403?seasonId=20685
[generic] 590403?seasonId=20685: Downloading webpage
WARNING: [generic] Falling back on generic information extractor
[generic] 590403?seasonId=20685: Extracting information
[debug] Looking for embeds
WARNING: [MediaStream] None: Failed to parse JSON: Expecting property name enclosed in double quotes in '"1422",\n\t\t}': line 11 column 3 (char 418)
WARNING: [generic] 590403?seasonId=20685: Failed to parse JSON: Expecting property name enclosed in double quotes in '"1422",\n\t\t}': line 11 column 3 (char 418)
ERROR: Unsupported URL: https://www.hidive.com/video/590403?seasonId=20685
Traceback (most recent call last):
File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 1587, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 1722, in __extract_info
ie_result = ie.extract(url)
^^^^^^^^^^^^^^^
File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 718, in extract
ie_result = self._real_extract(url)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/extractor/generic.py", line 2531, in _real_extract
raise UnsupportedError(url)
yt_dlp.utils.UnsupportedError: Unsupported URL: https://www.hidive.com/video/590403?seasonId=20685
About this issue
- Original URL
- State: open
- Created 4 months ago
- Comments: 20 (7 by maintainers)
I can take a crack at fixing it later tonight and/or this weekend.
If anyone has ideas for getting the “old api” working, feel free to discuss.
We have already established that the “new API” has only DRM formats. Further discussion on it will force me to lock the issue
@hazy-kun Writing “feel free to delete” is not an excuse to break rules.
Unfortunately I’m not sure if this is going to be fixable, looks like they added DRM.
Here’s roughly how their new backend works:
A request is made to the init endpoint
https://dce-frontoffice.imggaming.com/api/v1/init/
using a refresh token stored in browser storage to get a bearer token.HiDive makes a request to
https://dce-frontoffice.imggaming.com/api/v4/vod/589879?includePlaybackDetails=URL
to get a JSON representation of the episode which includes description, rating, episode information, etc.dve-api.imggaming.com
which provides a customerId, auth token, and timestamp in the URL which is required to fetch the HLS file.The JSON file at the URL specified by playerUrlCallback is loaded which contains a dash and HLS section. Both dash and HLS have an associated DRM object. Both dash and HLS contain the subtitle files (three formats now–VTT, SRT, and SCC) and a URL.
The URL is loaded which contains the
.mpd
or.m3u8
file. Firefox is loading the MPD file so that’s what I can see right now. This is an XML that contains the different AdaptationSets for audio and different video resolutions. The URL isdve-streams.akamaized.net
for these and for all of the MP4 files. Everything is wrapped in<ContentProtection>
headers.Using the bearer token I’ve been able to get the JSON file from step one and the file from step two that has the link to the MPD or HLS file in it. I was able to retrieve the
.m3u8
file one time using Postman but haven’t been able to manage it since. The URL appears to be burned after one request as well, so if you load the page in your browser and it requests the MPD or m3u8 file that URL can no longer be used to request the file again.I converted the HiDive extractor to use MPD instead of m3u8 and downloaded the response locally then mocked it up to give it that file as a URL for the MPD extractor and got a DRM exception.
So unless I’ve done something wrong looks like it might be over. I’d very much like to be wrong but… 😦
If anyone wants or needs any more information let me know and I can provide snippets and code samples.
@levaculik2 Whether this is fixable or not isn’t something yt-dlp contributors are in control of, as it depends on how HiDive have changed things. That said, unless there’s any luck with being able to continue working with the old API, this is not going to be fixable.
Since the issue on my project was referenced, I’ll throw my hat in the ring, from what I can tell, both the new Mobile API and the new Web API are DRM only.
The old API still works with the exception of login from what I can tell. Login I believe only works for those that had an account and subscription before they updated the site from what I can tell.
I haven’t looked at the extractor, but implementing the old API should work for legacy users, from what I can tell though new ones might be SOL
master-7531546c-f684-493c-a73d-bf6b28ad9911.m3u8 is “FAIRPLAY” master-36196614-0f46-4dab-9f08-581e0d42d2f1.mpd is “WIDEVINE”, “PLAYREADY”, “COMMON”
@Tama47 crunchyroll provides non-DRM formats. Scott’s findings suggest that HiDive does not.
@ScottSemian Before we write this off as completely DRM, could you share the API response JSON for a video? You can redact any PII. Also, a sample of the variant m3u8 would clear things right up