yt-dlp: [Broken] Niconico
Checklist
- I’m reporting a broken site support
- I’ve verified that I’m running yt-dlp version 2021.03.15
- I’ve checked that all provided URLs are alive and playable in a browser
- I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
- I’ve searched the bugtracker for similar issues including closed ones
Verbose log
[debug] Command-line config: ['--list-formats', '--verbose', 'https://www.nicovideo.jp/watch/sm31589985']
[debug] Loading archive file None
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] yt-dlp version 2021.03.15
[debug] Python version 3.8.7 (CPython 64bit) - Windows-10-10.0.19041-SP0
[debug] exe versions: ffmpeg n4.3.1-29-g89daac5fe2, ffprobe n4.3.1-29-g89daac5fe2
[debug] Proxy map: {}
[niconico] sm31589985: Downloading webpage
[niconico] sm31589985: Downloading video info page
ERROR: The video can't downloaded.
Traceback (most recent call last):
File "c:\users\cleo\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\extractor\niconico.py", line 398, in _real_extract
dmc_info = api_data['video']['dmcInfo']
KeyError: 'dmcInfo'
Traceback (most recent call last):
File "c:\users\cleo\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\extractor\niconico.py", line 398, in _real_extract
dmc_info = api_data['video']['dmcInfo']
KeyError: 'dmcInfo'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\users\cleo\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 1024, in wrapper
return func(self, *args, **kwargs)
File "c:\users\cleo\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\YoutubeDL.py", line 1045, in __extract_info
ie_result = ie.extract(url)
File "c:\users\cleo\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\extractor\common.py", line 554, in extract
ie_result = self._real_extract(url)
File "c:\users\cleo\appdata\local\programs\python\python38\lib\site-packages\yt_dlp\extractor\niconico.py", line 400, in _real_extract
raise ExtractorError('The video can\'t downloaded.',
yt_dlp.utils.ExtractorError: The video can't downloaded.
Description
This just happened recently. I tried doing the clear cache method but it still doesn’t work. I also did use login credentials but same error happens (even on other youtube-dl forks)
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 56 (50 by maintainers)
Commits related to this issue
- [niconico] Partial fix TODO: Smile formats don't work Extraction code that no longer works needs to be removed Adapted from: https://github.com/animelover1984/youtube-dl/commit/b5eff52dd9ed5565672ea... — committed to yt-dlp/yt-dlp by pukkandan 3 years ago
- [niconico] Fix for when logged in Related: https://github.com/yt-dlp/yt-dlp/issues/171#issuecomment-803692716 Co-authored by: CXwudi, xtkoba — committed to yt-dlp/yt-dlp by pukkandan 3 years ago
- [niconico] Partial fix TODO: Smile formats don't work Extraction code that no longer works needs to be removed Adapted from: https://github.com/animelover1984/youtube-dl/commit/b5eff52dd9ed5565672ea... — committed to nixxo/yt-dlp by pukkandan 3 years ago
- [niconico] Fix for when logged in Related: https://github.com/yt-dlp/yt-dlp/issues/171#issuecomment-803692716 Co-authored by: CXwudi, xtkoba — committed to nixxo/yt-dlp by pukkandan 3 years ago
- [niconico] Fix HLS formats Closes #171 * The structure of the API JSON was changed * Smile Video seems to be no longer available. So remove the warning * Move ping to downloader * Change heartbeat in... — committed to nixxo/yt-dlp by pukkandan 3 years ago
Just to note: For SMILE formats, it’s still “present” in older embed pages. The method is indicated in nicozon.net and the code below is doing the same. (that’s why _VALID_URL points to nicozon.net) I was able to download so-called “economy” format, but others are 403. Looks like we have to dig into SWF to download other formats, if it’s still available internally.
https://github.com/nao20010128nao/ytdl-patched/blob/master/youtube_dl/extractor/niconico_smile.py
I’ll also be taking a better look at my extractor issues when I have a chance. I made some quick fixes in the interest of restoring basic functionality, but I don’t think it’s ideal.
I created a PR. I’ve changed the implementation around the API headers to my personal taste, but it’s not required. It depends on how clean the code you want. https://github.com/pukkandan/yt-dlp-dev/pull/2
I found out why yt-dlp-dev didn’t work. The cause was a simple mistake, the dictionary key was wrong.
Simply replace
x['video']['delivery']['trackingId']
withx['media']['delivery']['trackingId']
and you can now download. … It also works without theOrigin
andReferer
headers. I also didn’t need theX-Niconico-Language
header. In the first place, I don’t think the specification of this header will affect the response of the API.Also, for unencrypted videos, just because a request to
https://nvapi.nicovideo.jp/v1/2ab0cbaa/watch
fails does not mean that subsequent downloads will fail. Therefore, it is wrong to raise ExtractorError if the request tohttps://nvapi.nicovideo.jp/v1/2ab0cbaa/watch
fails.The modified code is as follows.
I think this will be helpful (Japanese): https://qiita.com/tor4kichi/items/91550a71119f3878bfba#comment-346de200d84c79876831 If it is not different from the old specification, you need to request “https://nvapi.nicovideo.jp/v1/2ab0cbaa/watch?t={tracking_id}” before accessing the encryption key. If you do not make a request or make a request without the required headers, a fake encryption key will be returned.
It seems that the title information of the form
<meta property="og:title" content="...">
is not available when (and only when) logged in. Maybe we have to get it separately without logging in.If you do not need the title info, you can use the following patch as a workaround (which sets the title field to
NA
):As my opinion, we should remove support for FLV formats, since Niconico will no longer provide Flash version.
This is why user experience error message saying to set credential.
P.S. A fix from @animelover1984 worked well. Thanks a lot. https://github.com/nao20010128nao/ytdl-patched/commit/171a9c33e96216f0fd6d51afea4a5f1c257bea50
@tsukumijima I think yes, but idk what client still uses it, since SWF player seems to use DMC for playback.
It was merged in https://github.com/yt-dlp/yt-dlp/commit/2291dbce2a1aa9c5e9e060eb7ac306f037234eb1
Could you make a PR with your changes? that would be easier
Since niconico’s API JSON has changed, the
api_data['media']['delivery']['encryption']
key is always present, even if the video is unencrypted. If the video is unencrypted, the value of theapi_data['media']['delivery']['encryption']
key is null (None in Python).Therefore, you need to change the code as follows. Otherwise, the download will fail with the unencrypted DMC video.
I am so sorry I wasn’t able to catch that dictionary key mistake at first. I should have do a full scan at first 😓
You have tested this modified code, correct? I’m asking since I wont be able to test it myself
I tried. With ytdl-patched, the encrypted HLS video download was successful. yt-dlp-dev failed because it couldn’t decrypt the HLS video.
First, you’ll need to test whether either https://github.com/pukkandan/yt-dlp-dev/tree/nico or https://github.com/nao20010128nao/ytdl-patched work to get the encrypted videos.
If @nao20010128nao’s patch work, but mine doesn’t, we need to figure out why (my version is basically just nao’s patch applied on top of #49). The prime candidates that could be causing the issue are either some of the headers, or location of the ping
What is the output when using
--hls-prefer-native
@CXwudi ?I am sorry, I am also failing to decode files in m3u8 playlist…
Here is my code fix and my run log
Maybe contributors from #49 can help?
@CXwudi Yes. my version does not use
dmcInfo
, so it’s possible.@CXwudi I’m now trying to reverse it, and HLS downloading is now successful with non-encrypted video.
@xtkoba very thanks for this hint, I tried
title = get_video_info_web("title")
and it works now@pukkandan would you mind confirming if this fix?
The fix does not seem to work for videos which require credentials:
Using the same command (edit: without logging in) with a normal video works.
EDIT: using https://github.com/animelover1984/youtube-dl works, but yt-dlp does not for paid videos. @CXwudi Oh I see. I never bother logging in for videos which don’t require it so I didn’t think of that.
I’ve pushed a partial fix based on https://github.com/animelover1984/youtube-dl/commit/b5eff52dd9ed5565672ea1694b38c9296db3fade
Btw I forgot to mention that it also now asks for login credentials. Just last week I didn’t need any login credentials to download and check formats.