youtube-dl: [Youtube] youtube-dl fails to retrieve the hls manifest for livestreams

A quick and dirty patch for HLS:

diff --git a/youtube_dl/extractor/youtube.py b/youtube_dl/extractor/youtube.py
index 0919bef0e..9ad662777 100644
--- a/youtube_dl/extractor/youtube.py
+++ b/youtube_dl/extractor/youtube.py
@@ -1537,7 +1537,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             ytplayer_config = self._get_ytplayer_config(video_id, video_webpage)
             if ytplayer_config:
                 args = ytplayer_config['args']
-                if args.get('url_encoded_fmt_stream_map'):
+                if args.get('url_encoded_fmt_stream_map') or args.get('hlsvp'):
                     # Convert to the same format returned by compat_parse_qs
                     video_info = dict((k, [v]) for k, v in args.items())
                     add_dash_mpd(video_info)
@@ -1802,15 +1802,16 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
 
         chapters = self._extract_chapters(description_original, video_duration)
 
+        formats = []
         if 'conn' in video_info and video_info['conn'][0].startswith('rtmp'):
             self.report_rtmp_download()
-            formats = [{
+            formats.append({
                 'format_id': '_rtmp',
                 'protocol': 'rtmp',
                 'url': video_info['conn'][0],
                 'player_url': player_url,
-            }]
-        elif len(video_info.get('url_encoded_fmt_stream_map', [''])[0]) >= 1 or len(video_info.get('adaptive_fmts', [''])[0]) >= 1:
+            })
+        if len(video_info.get('url_encoded_fmt_stream_map', [''])[0]) >= 1 or len(video_info.get('adaptive_fmts', [''])[0]) >= 1:
             encoded_url_map = video_info.get('url_encoded_fmt_stream_map', [''])[0] + ',' + video_info.get('adaptive_fmts', [''])[0]
             if 'rtmpe%3Dyes' in encoded_url_map:
                 raise ExtractorError('rtmpe downloads are not supported, see https://github.com/rg3/youtube-dl/issues/343 for more information.', expected=True)
@@ -1827,7 +1828,6 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                                 'width': int_or_none(width_height[0]),
                                 'height': int_or_none(width_height[1]),
                             }
-            formats = []
             for url_data_str in encoded_url_map.split(','):
                 url_data = compat_parse_qs(url_data_str)
                 if 'itag' not in url_data or 'url' not in url_data:
@@ -1934,9 +1934,8 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                             if codecs:
                                 dct.update(parse_codecs(codecs))
                 formats.append(dct)
-        elif video_info.get('hlsvp'):
+        if video_info.get('hlsvp'):
             manifest_url = video_info['hlsvp'][0]
-            formats = []
             m3u8_formats = self._extract_m3u8_formats(
                 manifest_url, video_id, 'mp4', fatal=False)
             for a_format in m3u8_formats:
@@ -1952,7 +1951,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 # Accept-Encoding header causes failures in live streams on Youtube and Youtube Gaming
                 a_format.setdefault('http_headers', {})['Youtubedl-no-compression'] = 'True'
                 formats.append(a_format)
-        else:
+        if not formats:
             unavailable_message = self._html_search_regex(
                 r'(?s)<h1[^>]+id="unavailable-message"[^>]*>(.+?)</h1>',
                 video_webpage, 'unavailable message', default=None)

Apparently Youtube still has HLS for live streams: For Euronews Live:

youtube-dl -j https://www.youtube.com/watch?v=xAE4xX8Izvo Only produces MPD formats

youtube-dl --print-traffic https://www.youtube.com/watch?v=xAE4xX8Izvo Has no HLS entries

BUT curl -sL -k “https://www.youtube.com/watch?v=xAE4xX8Izvo” has a very valid HLS link:

“hlsvp”:"https://manifest.googlevideo.com/api/manifest/hls_variant/gcr.…m3u8

This stream (https://youtu.be/9h6Vz6ylIdE) and the NASA one mentioned above have been active since before this change by YouTube.

If you youtube-dl -F any new streams, you’ll see (at least this is what I’m seeing anyway) that they don’t have hls streams anymore.

Further - it appears that on the browser, when clicking the nerd stats, there is no longer the line that indicates if it’s a DASH stream or not, seemingly confirming that it’s just all DASH now. It’s not overly surprising given that HLS streams were 30fps irrespective of what went in - every stream had to be converted for archive and this had long been just a compatibility feature.

Perhaps what is needed is to change this to a feature request; simply for youtube-dl to support dynamic MPD manifests (it already fully supports static ones)

I know from manual debugging, and lots of grep and jq (and more recently using youtube-dl with the --youtube-include-dash-manifest option), that the MPD playlist is available from the earliest segment available on the stream (about 4 hours worth if the stream is 24/7 from memory), so it could be cleanly implemented.

As a sidenote - same deal for streamlink - this never supported DASH at all for live streams with youtube, but it seems now that a combination of youtubes new defaults and some minor changes mentioned on this thread (and fixed [here])(https://github.com/streamlink/streamlink/pull/1423), that there is some playability.

The live dash manifest URL is in the player config JSON.

youtube-dl --print-traffic https://youtu.be/<id> \
| grep -oP "/api/manifest/dash/.*? " \
| echo "https://manifest.googlevideo.com/$(cat -)"

https://manifest.googlevideo.com//api/manifest/dash/etc....

Or extract it yourself:

curl -sL "https://youtu.be/<id>" \
| grep -oP "ytplayer.config =\K.*?}(?=;)" \
| jq -r .args.dashmpd

https://manifest.googlevideo.com/api/manifest/dash/etc....