yt-dlp: HiDive Extractor broken after site update

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

I’m reporting that yt-dlp is broken on a supported site
I’ve verified that I have updated yt-dlp to nightly or master (update instructions)
I’ve checked that all provided URLs are playable in a browser with the same IP and same login details
I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
I’ve searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
I’ve read the guidelines for opening an issue
I’ve read about sharing account credentials and I’m willing to share it if required

Region

No response

Provide a description that is worded well enough to be understood

HiDive was updated earlier today which appears to be a complete rework of the site (or at least their URLs). Below I’ve included two logs, one using the old format and one using the new format. I’m not expecting the new one to actually work but included it anyway in case there’s anything interesting.

For example these two URLs are the same video, the first being the old format and the second being how it appears after the update:

https://www.hidive.com/stream/chained-soldier/s01e010 https://www.hidive.com/video/590403?seasonId=20685

Provide verbose output that clearly demonstrates the problem

Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
If using API, add 'verbose': True to YoutubeDL params instead
Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

yt-dlp -f "(b[language=ja][height<=1080])" "https://www.hidive.com/stream/chained-soldier/s01e010" --cookies-from-browser firefox -o "%(series)s_S%(season_number)s_E%(episode_number)s_%(episode)s.%(ext)s" --embed-subs --sub-lang english-subs --sub-format vtt --write-subs  -N 64 --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0" -vU
[debug] Command-line config: ['-f', '(b[language=ja][height<=1080])', 'https://www.hidive.com/stream/chained-soldier/s01e010', '--cookies-from-browser', 'firefox', '-o', '%(series)s_S%(season_number)s_E%(episode_number)s_%(episode)s.%(ext)s', '--embed-subs', '--sub-lang', 'english-subs', '--sub-format', 'vtt', '--write-subs', '-N', '64', '--user-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0', '-vU']
Extracting cookies from firefox
[debug] Extracting cookies from: "/home/user/.mozilla/firefox/ci8h4s4e.default-release/cookies.sqlite"
Extracted 477 cookies from firefox
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.12.30 from yt-dlp/yt-dlp [f10589e34] (pip)
[debug] Python 3.11.7 (CPython x86_64 64bit) - Linux-6.7.6-arch1-1-x86_64-with-glibc2.39 (OpenSSL 3.2.1 30 Jan 2024, glibc 2.39)
[debug] exe versions: ffmpeg 6.1.1 (setts), ffprobe 6.1.1
[debug] Optional libraries: Cryptodome-3.18.0, brotli-1.0.9, certifi-2023.07.22, mutagen-1.46.0, requests-2.31.0, sqlite3-3.45.1, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1798 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: stable@2023.12.30 from yt-dlp/yt-dlp
yt-dlp is up to date (stable@2023.12.30 from yt-dlp/yt-dlp)
[HiDive] Extracting URL: https://www.hidive.com/stream/chained-soldier/s01e010
[HiDive] chained-soldier/s01e010: Downloading JSON metadata
ERROR: [HiDive] chained-soldier/s01e010: chained-soldier/s01e010: Failed to parse JSON (caused by JSONDecodeError("Expecting value in '': line 1 column 1 (char 0)")); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 718, in extract
    ie_result = self._real_extract(url)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/extractor/hidive.py", line 76, in _real_extract
    settings = self._call_api(video_id, title, key)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/extractor/hidive.py", line 70, in _call_api
    return self._download_json(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 1072, in download_content
    res = getattr(self, download_handle.__name__)(url_or_request, video_id, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 1042, in download_handle
    return parse(self, content, video_id, transform_source=transform_source, fatal=fatal, errnote=errnote), urlh
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 1032, in parse
    return getattr(ie, parser)(content, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 1019, in _parse_json
    self.__print_error('Failed to parse JSON' if errnote is None else errnote, fatal, video_id, ve)
  File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 1002, in __print_error
    raise ExtractorError(f'{video_id}: {errnote}', cause=err)

  File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/utils/_utils.py", line 553, in decode
    return super().decode(s)
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 1016, in _parse_json
    return json.loads(
           ^^^^^^^^^^^
  File "/usr/lib/python3.11/json/__init__.py", line 359, in loads
    return cls(**kw).decode(s)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/utils/_utils.py", line 561, in decode
    raise type(e)(f'{e.msg} in {s[e.pos - 10:e.pos + 10]!r}', s, e.pos)
json.decoder.JSONDecodeError: Expecting value in '': line 1 column 1 (char 0)


New URL:

yt-dlp -f "(b[language=ja][height<=1080])" "https://www.hidive.com/video/590403?seasonId=20685" --cookies-from-browser firefox -o "%(series)s_S%(season_number)s_E%(episode_number)s_%(episode)s.%(ext)s" --embed-subs --sub-lang english-subs --sub-format vtt --write-subs  -N 64 --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0" -vU
[debug] Command-line config: ['-f', '(b[language=ja][height<=1080])', 'https://www.hidive.com/video/590403?seasonId=20685', '--cookies-from-browser', 'firefox', '-o', '%(series)s_S%(season_number)s_E%(episode_number)s_%(episode)s.%(ext)s', '--embed-subs', '--sub-lang', 'english-subs', '--sub-format', 'vtt', '--write-subs', '-N', '64', '--user-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0', '-vU']
Extracting cookies from firefox
[debug] Extracting cookies from: "/home/user/.mozilla/firefox/ci8h4s4e.default-release/cookies.sqlite"
Extracted 477 cookies from firefox
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.12.30 from yt-dlp/yt-dlp [f10589e34] (pip)
[debug] Python 3.11.7 (CPython x86_64 64bit) - Linux-6.7.6-arch1-1-x86_64-with-glibc2.39 (OpenSSL 3.2.1 30 Jan 2024, glibc 2.39)
[debug] exe versions: ffmpeg 6.1.1 (setts), ffprobe 6.1.1
[debug] Optional libraries: Cryptodome-3.18.0, brotli-1.0.9, certifi-2023.07.22, mutagen-1.46.0, requests-2.31.0, sqlite3-3.45.1, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1798 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: stable@2023.12.30 from yt-dlp/yt-dlp
yt-dlp is up to date (stable@2023.12.30 from yt-dlp/yt-dlp)
[generic] Extracting URL: https://www.hidive.com/video/590403?seasonId=20685
[generic] 590403?seasonId=20685: Downloading webpage
WARNING: [generic] Falling back on generic information extractor
[generic] 590403?seasonId=20685: Extracting information
[debug] Looking for embeds
WARNING: [MediaStream] None: Failed to parse JSON: Expecting property name enclosed in double quotes in '"1422",\n\t\t}': line 11 column 3 (char 418)
WARNING: [generic] 590403?seasonId=20685: Failed to parse JSON: Expecting property name enclosed in double quotes in '"1422",\n\t\t}': line 11 column 3 (char 418)
ERROR: Unsupported URL: https://www.hidive.com/video/590403?seasonId=20685
Traceback (most recent call last):
  File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 1587, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 1722, in __extract_info
    ie_result = ie.extract(url)
                ^^^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/extractor/common.py", line 718, in extract
    ie_result = self._real_extract(url)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/pipx/venvs/yt-dlp/lib/python3.11/site-packages/yt_dlp/extractor/generic.py", line 2531, in _real_extract
    raise UnsupportedError(url)
yt_dlp.utils.UnsupportedError: Unsupported URL: https://www.hidive.com/video/590403?seasonId=20685

About this issue

Original URL
State: open
Created 4 months ago
Comments: 20 (7 by maintainers)

Most upvoted comments

I can take a crack at fixing it later tonight and/or this weekend.

ScottSemian on Mar 7, 2024

If anyone has ideas for getting the “old api” working, feel free to discuss.

We have already established that the “new API” has only DRM formats. Further discussion on it will force me to lock the issue

@hazy-kun Writing “feel free to delete” is not an excuse to break rules.

pukkandan on Mar 10, 2024

Unfortunately I’m not sure if this is going to be fixable, looks like they added DRM.

Here’s roughly how their new backend works:

A request is made to the init endpoint https://dce-frontoffice.imggaming.com/api/v1/init/ using a refresh token stored in browser storage to get a bearer token.
HiDive makes a request to https://dce-frontoffice.imggaming.com/api/v4/vod/589879?includePlaybackDetails=URL to get a JSON representation of the episode which includes description, rating, episode information, etc.
- This also includes a playerUrlCallback to dve-api.imggaming.com which provides a customerId, auth token, and timestamp in the URL which is required to fetch the HLS file.
The JSON file at the URL specified by playerUrlCallback is loaded which contains a dash and HLS section. Both dash and HLS have an associated DRM object. Both dash and HLS contain the subtitle files (three formats now–VTT, SRT, and SCC) and a URL.
The URL is loaded which contains the .mpd or .m3u8 file. Firefox is loading the MPD file so that’s what I can see right now. This is an XML that contains the different AdaptationSets for audio and different video resolutions. The URL is dve-streams.akamaized.net for these and for all of the MP4 files. Everything is wrapped in <ContentProtection> headers.

Using the bearer token I’ve been able to get the JSON file from step one and the file from step two that has the link to the MPD or HLS file in it. I was able to retrieve the .m3u8 file one time using Postman but haven’t been able to manage it since. The URL appears to be burned after one request as well, so if you load the page in your browser and it requests the MPD or m3u8 file that URL can no longer be used to request the file again.

I converted the HiDive extractor to use MPD instead of m3u8 and downloaded the response locally then mocked it up to give it that file as a URL for the MPD extractor and got a DRM exception.

  File "yt-dlp/yt_dlp/YoutubeDL.py", line 1091, in raise_no_formats
    raise ExtractorError(msg, video_id=info['id'], ie=info['extractor'],
yt_dlp.utils.ExtractorError: [HiDive] 590403: This video is DRM protected

So unless I’ve done something wrong looks like it might be over. I’d very much like to be wrong but… 😦

If anyone wants or needs any more information let me know and I can provide snippets and code samples.

ScottSemian on Mar 9, 2024

@levaculik2 Whether this is fixable or not isn’t something yt-dlp contributors are in control of, as it depends on how HiDive have changed things. That said, unless there’s any luck with being able to continue working with the old API, this is not going to be fixable.

Sophira on Mar 11, 2024

Since the issue on my project was referenced, I’ll throw my hat in the ring, from what I can tell, both the new Mobile API and the new Web API are DRM only.

The old API still works with the exception of login from what I can tell. Login I believe only works for those that had an account and subscription before they updated the site from what I can tell.

I haven’t looked at the extractor, but implementing the old API should work for legacy users, from what I can tell though new ones might be SOL

Jaynator495 on Mar 10, 2024

master-7531546c-f684-493c-a73d-bf6b28ad9911.m3u8 is “FAIRPLAY” master-36196614-0f46-4dab-9f08-581e0d42d2f1.mpd is “WIDEVINE”, “PLAYREADY”, “COMMON”

@hazy-kun How were you able to request the MPD URL? The vast majority of the time I do it I get a 403 response. I think I have gotten it to work like twice.

In theory if you can request the MPD URL you should be able to get the HLS (m3u8) one instead.

hazy-kun on Mar 9, 2024

@Tama47 crunchyroll provides non-DRM formats. Scott’s findings suggest that HiDive does not.

@ScottSemian Before we write this off as completely DRM, could you share the API response JSON for a video? You can redact any PII. Also, a sample of the variant m3u8 would clear things right up

bashonly on Mar 9, 2024