youtube-dl: youtube-dl unable to download private vk video do to allegedly invalid credentials

There are two issues:

1. The site’s login workflow has changed, as above, but someone with a VK account will have to fix it.
2. The request that the extractor uses to fetch video data, except for embedded videos, no longer works, and has to be re-worked as in https://github.com/yt-dlp/yt-dlp/pull/4128.

Consequently:

• only embedded videos like https://vk.com/video_ext.php?oid=-77521&id=162222515&hash=87b046504ccd8bfa can be downloaded with the extractor as-is
• if there were an embedded video that needed a login then it would presumably be accessible using --cookies ..., but perhaps embedded and private are mutually exclusive.

This patch fixes the extractor for public videos and should, I hope, enable private videos with --cookies ... as well (it also includes _VALID_URL improvements from yt-dlp).

--- old/youtube_dl/extractor/vk.py
+++ new/youtube_dl/extractor/vk.py
@@ -16,6 +16,7 @@
     orderedSet,
     str_or_none,
     str_to_int,
+    try_get,
     unescapeHTML,
     unified_timestamp,
     url_or_none,
@@ -64,10 +65,14 @@
 
     def _download_payload(self, path, video_id, data, fatal=True):
         data['al'] = 1
+        url = 'https://vk.com/%s.php' % path
         code, payload = self._download_json(
-            'https://vk.com/%s.php' % path, video_id,
+            url, video_id,
             data=urlencode_postdata(data), fatal=fatal,
-            headers={'X-Requested-With': 'XMLHttpRequest'})['payload']
+            headers={
+                'X-Requested-With': 'XMLHttpRequest',
+                'Referer': url,
+            })['payload']
         if code == '3':
             self.raise_login_required()
         elif code == '8':
@@ -87,22 +92,22 @@
                             )
                             ext\.php\?(?P<embed_query>.*?\boid=(?P<oid>-?\d+).*?\bid=(?P<id>\d+).*)|
                             (?:
-                                (?:(?:m|new)\.)?vk\.com/(?:.+?\?.*?z=)?video|
+                                (?:(?:m|new)\.)?vk\.com/(?:.+?\?.*?z=)?video|clip|
                                 (?:www\.)?daxab.com/embed/
                             )
-                            (?P<videoid>-?\d+_\d+)(?:.*\blist=(?P<list_id>[\da-f]+))?
+                            (?P<videoid>-?\d+_\d+)(?:.*\blist=(?P<list_id>[\da-f]+|ln-[\da-zA-Z]+))?
                         )
                     '''
     _TESTS = [
         {
             'url': 'http://vk.com/videos-77521?z=video-77521_162222515%2Fclub77521',
-            'md5': '7babad3b85ea2e91948005b1b8b0cb84',
+            'md5': '56df4e32658e1ffb77fdf2548327f73a',
             'info_dict': {
                 'id': '-77521_162222515',
                 'ext': 'mp4',
                 'title': 'ProtivoGunz - Хуёвая песня',
                 'uploader': 're:(?:Noize MC|Alexander Ilyashenko).*',
-                'uploader_id': '-77521',
+                'uploader_id': '39545378',
                 'duration': 195,
                 'timestamp': 1329049880,
                 'upload_date': '20120212',
@@ -124,7 +129,7 @@
         {
             'note': 'Embedded video',
             'url': 'https://vk.com/video_ext.php?oid=-77521&id=162222515&hash=87b046504ccd8bfa',
-            'md5': '7babad3b85ea2e91948005b1b8b0cb84',
+            'md5': '56df4e32658e1ffb77fdf2548327f73a',
             'info_dict': {
                 'id': '-77521_162222515',
                 'ext': 'mp4',
@@ -133,7 +138,7 @@
                 'duration': 195,
                 'upload_date': '20120212',
                 'timestamp': 1329049880,
-                'uploader_id': '-77521',
+                'uploader_id': '39545378',
             },
         },
         {
@@ -180,6 +185,19 @@
                 'view_count': int,
             },
             'skip': 'Removed',
+        },
+        {
+            'url': 'https://vk.com/video-93049196_456239755?list=ln-cBjJ7S4jYYx3ADnmDT',
+            'info_dict': {
+                'id': '-93049196_456239755',
+                'ext': 'mp4',
+                'title': '8 серия (озвучка)',
+                'duration': 8383,
+                'timestamp': 1640162189,
+                'upload_date': '20211222',
+                'uploader': 'Dizi2021',
+                'uploader_id': '-93049196',
+            },
         },
         {
             # video (removed?) only available with list id
@@ -227,6 +245,7 @@
             'params': {
                 'skip_download': True,
             },
+            'skip': 'This video has been deleted',
         },
         {
             # video key is extra_data not url\d+
@@ -254,8 +273,6 @@
                 'title': 'ИгроМир 2016 День 1 — Игромания Утром',
                 'uploader': 'Игромания',
                 'duration': 5239,
-                # TODO: use act=show to extract view_count
-                # 'view_count': int,
                 'upload_date': '20160929',
                 'uploader_id': '-387766',
                 'timestamp': 1475137527,
@@ -297,6 +314,10 @@
         {
             # The video is not available in your region.
             'url': 'https://vk.com/video-51812607_171445436',
+            'only_matching': True,
+        },
+        {
+            'url': 'https://vk.com/clip30014565_456240946',
             'only_matching': True,
         }]
 
@@ -314,7 +335,7 @@
         mv_data = {}
         if video_id:
             data = {
-                'act': 'show_inline',
+                'act': 'show',
                 'video': video_id,
             }
             # Some videos (removed?) can only be downloaded with list id specified
@@ -323,8 +344,10 @@
                 data['list'] = list_id
 
             payload = self._download_payload('al_video', video_id, data)
+            opts = try_get(payload, lambda x: x[-1], dict)
+            if not opts:
+                raise ExtractorError('Invalid payload data')
             info_page = payload[1]
-            opts = payload[-1]
             mv_data = opts.get('mvData') or {}
             player = opts.get('player') or {}
         else:

-g/--get-url just prints the download link(s).

There will be a PR. If someone manages to fix the login soon, I’ll add that before merging. Then, real soon now, a 2022.0n.mm release.

For me, nor youtube-dl -u … -p … , youtube-dl -n or cookies works.

Please tell us the details how you use cookies for downloading from vk.com? I tried to save cookies by the chrome extension (get cookies.txt), but I got an error:

youtube-dl -v --cookies vk.com_cookies.txt https://vk.com/video-31480508_456243099
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'--cookies', u'vk.com_cookies.txt', u'https://vk.com/video-31480508_456243099']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.06.06
[debug] Python version 2.7.18 (CPython) - Darwin-21.1.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 5.0.1, ffprobe 5.0.1, rtmpdump 2.4
[debug] Proxy map: {}
[vk] -31480508_456243099: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

What am I doing wrong?

Спасибо, а есть какие-то гайды как использовать файлы cookie я не совсем понимаю как их подключить

Back-port of the newly updated yt-dlp extractor handles https://vk.com/video-133441491_456261066.

doesn’t work for me in Europe

yt-dlp -F “https://vk.com/video-133441491_456261066” [vk] -133441491_456261066: Downloading JSON metadata ERROR: This video is only available for registered users. Use --cookies, --username and --password or --netrc to provide account credentials

can be “handled” by latest yt-dlp, at least from my location (SE-Europe), without any cookies or credentials:

yt-dlp -F "https://vk.com/video-133441491_456261066"

Спасибо, а есть какие-то гайды как использовать файлы cookie я не совсем понимаю как их подключить

Нужно узнать, какой у браузера User-Agent, и указать параметр --user-agent такой же. Потом если один раз запустить с параметром --cookie vk.txt, то пока yt-dl потыкается в сайт, он создаст vk.txt с кукисами в правильном формате. Открываем в браузере список куки, в Firefox при открытом залогиненном ВКонтакте Ctrl+Shift+K, вкладка Хранилище, раздел Куки, и по одной чик-чик-чик Ctrl+C, Ctrl+V имя и значение в vk.txt, через табуляцию, как требует формат файла. С таким содержимым vk.txt можно будет качать видео. У меня получилось скачать generic движком из мобильной (m.vk.com) версии сайта, но для этого ещё в эту мобильную версию зайти надо, у мобильной свои куки, и только потом снимать куки в vk.txt

youtube-dl --cookies vk.txt --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0" --force-generic-extractor https://m.vk.com/video-12345_67890

I had success with cookies method, generic extractor and a MOBILE (m.vk.com/video) version. You need to be sure saved cookies match the mobile version, so login to mobile version first in real browser

def _download_payload(self, path, video_id, data, fatal=True):

It works! Thank you!

а что это? надо вставить куда-то?

From https://github.com/ytdl-org/youtube-dl/issues/31041#issue-1277401990:

The authentication algorithm on the site vk.com changed. Now on the start page in the input form you need to specify only the login, after clicking on the button in the next form you need to enter only the password. Please correct the authentication for vk.com in your module vk.py

Please discuss this in the yt-dlp tracker to avoid confusion.

вы можете просто файл загрузить куда-то целиком, а не кусками. может я не туда вставил что-то

я всё сделал по гайду обновил но. пишет опять тоже самое [vk] 77496727_456240355: Downloading JSON metadata ERROR: This video is only available for registered users. Use --username and --password or --netrc to provide account credentials.

So not only me getting error about downloading vk video? I created _netrc, created config file, tried to use -u -p/–username --password but yet it getting me this error