yt-dlp: [Broken] Niconico

Just to note: For SMILE formats, it’s still “present” in older embed pages. The method is indicated in nicozon.net and the code below is doing the same. (that’s why _VALID_URL points to nicozon.net) I was able to download so-called “economy” format, but others are 403. Looks like we have to dig into SWF to download other formats, if it’s still available internally.

https://github.com/nao20010128nao/ytdl-patched/blob/master/youtube_dl/extractor/niconico_smile.py

I’ve pushed a partial fix based on animelover1984/youtube-dl@b5eff52

• SMILE formats still dont work and many of the improvements from #49 is therefore broken
• I haven’t tested if all metadata is extracted correctly.
• Code also needs refactoring. Maybe @tsukumijima or @kurumigi may be interested in fixing it?

I’ll also be taking a better look at my extractor issues when I have a chance. I made some quick fixes in the interest of restoring basic functionality, but I don’t think it’s ideal.

Could you make a PR with your changes? that would be easier

I created a PR. I’ve changed the implementation around the API headers to my personal taste, but it’s not required. It depends on how clean the code you want. https://github.com/pukkandan/yt-dlp-dev/pull/2

I found out why yt-dlp-dev didn’t work. The cause was a simple mistake, the dictionary key was wrong.

Simply replace x['video']['delivery']['trackingId'] with x['media']['delivery']['trackingId'] and you can now download. … It also works without the Origin and Referer headers. I also didn’t need the X-Niconico-Language header. In the first place, I don’t think the specification of this header will affect the response of the API.

Also, for unencrypted videos, just because a request to https://nvapi.nicovideo.jp/v1/2ab0cbaa/watch fails does not mean that subsequent downloads will fail. Therefore, it is wrong to raise ExtractorError if the request to https://nvapi.nicovideo.jp/v1/2ab0cbaa/watch fails.

The modified code is as follows.

        def ping():
            status = try_get(
                self._download_json(
                    'https://nvapi.nicovideo.jp/v1/2ab0cbaa/watch', video_id,
                    query={'t': try_get(api_data, lambda x: x['media']['delivery']['trackingId'])},
                    note='Acquiring permission for downloading video',
                    headers={
                        'X-Frontend-Id': '6',
                        'X-Frontend-Version': '0',
                    }),
                lambda x: x['meta']['status'])
            if status != 200:
                self.report_warning('Failed to acquire permission for playing video. The video may not download.')

I think this will be helpful (Japanese): https://qiita.com/tor4kichi/items/91550a71119f3878bfba#comment-346de200d84c79876831 If it is not different from the old specification, you need to request “https://nvapi.nicovideo.jp/v1/2ab0cbaa/watch?t={tracking_id}” before accessing the encryption key. If you do not make a request or make a request without the required headers, a fake encryption key will be returned.

It seems that the title information of the form <meta property="og:title" content="..."> is not available when (and only when) logged in. Maybe we have to get it separately without logging in.

If you do not need the title info, you can use the following patch as a workaround (which sets the title field to NA):

--- a/yt_dlp/extractor/niconico.py
+++ b/yt_dlp/extractor/niconico.py
@@ -496,9 +496,7 @@
         if not title:
             title = self._og_search_title(webpage, default=None)
         if not title:
-            title = self._html_search_regex(
-                r'<span[^>]+class="videoHeaderTitle"[^>]*>([^<]+)</span>',
-                webpage, 'video title')
+            title = "NA"
 
         watch_api_data_string = self._html_search_regex(
             r'<div[^>]+id="watchAPIDataContainer"[^>]+>([^<]+)</div>',

As my opinion, we should remove support for FLV formats, since Niconico will no longer provide Flash version.

This is why user experience error message saying to set credential.

P.S. A fix from @animelover1984 worked well. Thanks a lot. https://github.com/nao20010128nao/ytdl-patched/commit/171a9c33e96216f0fd6d51afea4a5f1c257bea50

@tsukumijima I think yes, but idk what client still uses it, since SWF player seems to use DMC for playback.

It was merged in https://github.com/yt-dlp/yt-dlp/commit/2291dbce2a1aa9c5e9e060eb7ac306f037234eb1

Could you make a PR with your changes? that would be easier

Since niconico’s API JSON has changed, the api_data['media']['delivery']['encryption'] key is always present, even if the video is unencrypted. If the video is unencrypted, the value of the api_data['media']['delivery']['encryption'] key is null (None in Python).

Therefore, you need to change the code as follows. Otherwise, the download will fail with the unencrypted DMC video.

        # m3u8 (encryption)
        if try_get(api_data, lambda x: x['media']['delivery']['encryption']) is not None:  # ← modified
            protocol = 'm3u8'
            encryption = self._parse_json(session_api_data['token'], video_id)['hls_encryption']

I am so sorry I wasn’t able to catch that dictionary key mistake at first. I should have do a full scan at first 😓

You have tested this modified code, correct? I’m asking since I wont be able to test it myself

I tried. With ytdl-patched, the encrypted HLS video download was successful. yt-dlp-dev failed because it couldn’t decrypt the HLS video.

First, you’ll need to test whether either https://github.com/pukkandan/yt-dlp-dev/tree/nico or https://github.com/nao20010128nao/ytdl-patched work to get the encrypted videos.

If @nao20010128nao’s patch work, but mine doesn’t, we need to figure out why (my version is basically just nao’s patch applied on top of #49). The prime candidates that could be causing the issue are either some of the headers, or location of the ping

What is the output when using --hls-prefer-native @CXwudi ?

I am sorry, I am also failing to decode files in m3u8 playlist…

Here is my code fix and my run log

Maybe contributors from #49 can help?

@CXwudi Yes. my version does not use dmcInfo, so it’s possible.

@CXwudi I’m now trying to reverse it, and HLS downloading is now successful with non-encrypted video.

@xtkoba very thanks for this hint, I tried title = get_video_info_web("title") and it works now

title = get_video_info_web('originalTitle') or get_video_info_web('title')
if not title:
    title = self._og_search_title(webpage, default=None)
if not title:
    title = self._html_search_regex(
        r'<span[^>]+class="videoHeaderTitle"[^>]*>([^<]+)</span>',
        webpage, 'video title')

@pukkandan would you mind confirming if this fix?

The fix does not seem to work for videos which require credentials:

[debug] Command-line config: ['-u', 'PRIVATE', '-p', 'PRIVATE', '--verbose', '--ignore-config', 'https://www.nicovideo.jp/watch/so37994933']
[debug] Loading archive file None
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] yt-dlp version 2021.03.15 
[debug] Python version 3.7.3 (CPython 64bit) - Linux-4.19.0-14-amd64-x86_64-with-debian-10.8
[debug] exe versions: ffmpeg 4.1.6-1, ffprobe 4.1.6-1, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[niconico] Logging in
[niconico] so37994933: Downloading webpage
[niconico] so37994933: Downloading video info page
WARNING: [niconico] Unable to obtain smile video information
[debug] Sort order given by user: 
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
ERROR: Unable to extract video title; please report this issue on https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/home/username/.local/lib/python3.7/site-packages/yt_dlp/YoutubeDL.py", line 1024, in wrapper
    return func(self, *args, **kwargs)
  File "/home/username/.local/lib/python3.7/site-packages/yt_dlp/YoutubeDL.py", line 1045, in __extract_info
    ie_result = ie.extract(url)
  File "/home/username/.local/lib/python3.7/site-packages/yt_dlp/extractor/common.py", line 554, in extract
    ie_result = self._real_extract(url)
  File "/home/username/.local/lib/python3.7/site-packages/yt_dlp/extractor/niconico.py", line 501, in _real_extract
    webpage, 'video title')
  File "/home/username/.local/lib/python3.7/site-packages/yt_dlp/extractor/common.py", line 1050, in _html_search_regex
    res = self._search_regex(pattern, string, name, default, fatal, flags, group)
  File "/home/username/.local/lib/python3.7/site-packages/yt_dlp/extractor/common.py", line 1041, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
yt_dlp.utils.RegexNotFoundError: Unable to extract video title; please report this issue on https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.

Using the same command (edit: without logging in) with a normal video works.

EDIT: using https://github.com/animelover1984/youtube-dl works, but yt-dlp does not for paid videos. @CXwudi Oh I see. I never bother logging in for videos which don’t require it so I didn’t think of that.

I’ve pushed a partial fix based on https://github.com/animelover1984/youtube-dl/commit/b5eff52dd9ed5565672ea1694b38c9296db3fade

• SMILE formats still dont work and many of the improvements from #49 is therefore broken
• I haven’t tested if all metadata is extracted correctly.
• Code also needs refactoring. Maybe @tsukumijima or @kurumigi may be interested in fixing it?

Btw I forgot to mention that it also now asks for login credentials. Just last week I didn’t need any login credentials to download and check formats.