yt-dlp: [MSN] Download videos from msn failed with error "Unable to extract error".

Checklist

Region

Singapore

Description

Could anyone help me with this? Thanks.

PS C:\Users\v-owendeng> yt-dlp https://www.msn.com/en-us/news/local/jeffco-public-schools-says-no-limit-on-guests-for-end-of-year-events/vi-BB1gIARh [MSN] jeffco-public-schools-says-no-limit-on-guests-for-end-of-year-events: Downloading webpage ERROR: [MSN] BB1gIARh: Unable to extract error; please report this issue on https://github.com/yt-dlp/yt-dlp , filling out the “Broken site” issue template properly. Confirm you are on the latest version using yt-dlp -U

Here are 3 test URLs: https://www.msn.com/en-us/video/cardio/crash-shuts-down-portion-of-brownsville-road/vi-AAL8z9q https://www.msn.com/en-us/news/local/jeffco-public-schools-says-no-limit-on-guests-for-end-of-year-events/vi-BB1gIARh https://www.msn.com/en-us/video/peopleandplaces/boston-mayor-janey-to-sign-measure-limiting-police-use-of-tear-gas/vp-BB1gchKd

Verbose log

PS C:\Users\v-owendeng> yt-dlp -vU "https://www.msn.com/en-us/video/cardio/crash-shuts-down-portion-of-brownsville-road/vi-AAL8z9q"
[debug] Command-line config: ['-vU', 'https://www.msn.com/en-us/video/cardio/crash-shuts-down-portion-of-brownsville-road/vi-AAL8z9q']
[debug] Encodings: locale cp936, fs utf-8, out utf-8, err utf-8, pref cp936
[debug] yt-dlp version 2022.03.08.1 [c0c2c57] (win_exe)
[debug] Python version 3.8.10 (CPython 64bit) - Windows-10-10.0.22000-SP0
[debug] exe versions: ffmpeg 2022-03-17-git-242c07982a-full_build-www.gyan.dev (setts), ffprobe 2022-03-17-git-242c07982a-full_build-www.gyan.dev
[debug] Optional libraries: brotli, Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
Latest version: 2022.03.08.1, Current version: 2022.03.08.1
yt-dlp is up to date (2022.03.08.1)
[debug] [MSN] Extracting URL: https://www.msn.com/en-us/video/cardio/crash-shuts-down-portion-of-brownsville-road/vi-AAL8z9q
[MSN] crash-shuts-down-portion-of-brownsville-road: Downloading webpage
ERROR: [MSN] AAL8z9q: Unable to extract error; please report this issue on  https://github.com/yt-dlp/yt-dlp , filling out the "Broken site" issue template properly. Confirm you are on the latest version using  yt-dlp -U
  File "yt_dlp\extractor\common.py", line 617, in extract
  File "yt_dlp\extractor\msn.py", line 166, in _real_extract
  File "yt_dlp\extractor\common.py", line 1192, in _search_regex

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 20 (10 by maintainers)

Commits related to this issue

Most upvoted comments

That seems to apply here exactly.

In this case it seems like the URL pattern should be tweaked to extract locale as the xx-xx initial path component and then, instead of fetching the webpage, get the JSON from f'https://assets.msn.com/content/view/v2/Detail/{locale}/{page_id)'.

The page from https://github.com/yt-dlp/yt-dlp/issues/3225#issuecomment-1506285806 gets 410 Gone for this; MSN still serves a page full of clickbait and ads without telling you that it’s not the page you expected.

A current URL in the same area (https://www.msn.com/en-us/weather/topstories/midwest-northeast-bracing-for-potentially-dangerous-long-duration-heat-wave-next-week/vi-BB1oaoek?ocid=windirect&cvid=981228a2079b4d07908de15c5820a4e2&ei=98) with locale en-us and page_id BB1oaoek yields valid JSON. In the JSON, there are these potentially useful values:

{
  ...,
  'videoMetadata': {
    'playTime': 209,
    'closedCaptions': [
      {
        'locale': 'en-us',
        'href': 'https://prod-video-cms-amp-microsoft-com.akamaized.net/tenant/amp/entityid/BB1oaoek?blobrefkey=closedcaptionen-us&$blob=1'
      }
    ],
    'externalVideoFiles': [
      {
        'url': 'https://prod-streaming-video-msn-com.akamaized.net/aec52b0b-227e-47ac-a315-5ae3db64dcda/386123b1-6d12-4474-bc0a-455ae9bdc511.mp4',
        'contentType': 'video/mp4',
        'fileSize': 136785082,
        'format': '1001'
      },
      {
        'url': 'https://prod-streaming-video-msn-com.akamaized.net/8abb4c3d-a2d6-436b-8b67-f7e93c623f88/386123b1-6d12-4474-bc0a-455ae9bd.ism/manifest',
        'width': 1280,
        'height': 720,
        'format': '1004'
      },
      {
        'url': 'https://prod-streaming-video-msn-com.akamaized.net/8abb4c3d-a2d6-436b-8b67-f7e93c623f88/386123b1-6d12-4474-bc0a-455ae9bd.ism/manifest(format=m3u8-aapl)',
        'width': 1280,
        'height': 720,
        'format': '1006'
      },
      /* 5 more video files */
      ...,
    ]
  },
  ...,
  'id': 'BB1oaoek',
  'name': '',
  'source': 'msn',
  'type': 'video',
  ...,
  'createdDateTime': '2024-06-13T14:17:54Z',
  'updatedDateTime': '2024-06-13T14:27:11Z',
  'publishedDateTime': '2024-06-13T14:06:49Z',
  ...,
}

Then (using upstream code):

$ python3 -m youtube_dl -vF 'https://www.msn.com/en-us/weather/topstories/midwest-northeast-bracing-for-potentially-dangerous-long-duration-heat-wave-next-week/vi-BB1oaoek?ocid=windirect&cvid=981228a2079b4d07908de15c5820a4e2&ei=98'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vF', 'https://www.msn.com/en-us/weather/topstories/midwest-northeast-bracing-for-potentially-dangerous-long-duration-heat-wave-next-week/vi-BB1oaoek?ocid=windirect&cvid=981228a2079b4d07908de15c5820a4e2&ei=98']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: fc03c3bb3
[debug] Python 3.11.2 (CPython i686 32bit) - Linux-6.1.0-20-686-pae-i686-with-glibc2.36 - OpenSSL 3.0.11 19 Sep 2023 - glibc 2.36
[debug] exe versions: ffmpeg 5.1.4-0, ffprobe 5.1.4-0
[debug] Proxy map: {}
[MSN] midwest-northeast-bracing-for-potentially-dangerous-long-duration-heat-wave-next-week: Downloading page JSON
[MSN] midwest-northeast-bracing-for-potentially-dangerous-long-duration-heat-wave-next-week: Downloading ISM manifest
[MSN] midwest-northeast-bracing-for-potentially-dangerous-long-duration-heat-wave-next-week: Downloading m3u8 information
[MSN] midwest-northeast-bracing-for-potentially-dangerous-long-duration-heat-wave-next-week: Downloading MPD manifest
[download] Downloading playlist: BB1oaoek
[MSN] playlist BB1oaoek: Collected 1 video ids (downloading 1 of them)
[download] Downloading video 1 of 1
[info] Available formats for BB1oaoek:
format code                     extension  resolution note
hls-audio-aac_und_2_96021_2_1   mp4        audio only [und] 
mss-aac_und_2_96021_2_1-96      isma       audio only   96k , AACL  (48000Hz)
dash-5_A_aac_und_2_96021_2_1_1  m4a        audio only DASH audio   96k , m4a_dash container, mp4a.40.2 (48000Hz)
mss-656                         ismv       640x360     656k , H264, video only
dash-1_V_video_5                mp4        640x360    DASH video  656k , mp4_dash container, avc1.64001E, video only
hls-784                         mp4        640x360     784k , avc1.64001e, video only
mss-1008                        ismv       640x360    1008k , H264, video only
dash-1_V_video_4                mp4        640x360    DASH video 1008k , mp4_dash container, avc1.64001E, video only
hls-1145                        mp4        640x360    1145k , avc1.64001e, video only
mss-1510                        ismv       960x540    1510k , H264, video only
dash-1_V_video_3                mp4        960x540    DASH video 1510k , mp4_dash container, avc1.64001F, video only
hls-1658                        mp4        960x540    1658k , avc1.64001f, video only
mss-2261                        ismv       960x540    2261k , H264, video only
dash-1_V_video_2                mp4        960x540    DASH video 2261k , mp4_dash container, avc1.64001F, video only
hls-2425                        mp4        960x540    2425k , avc1.64001f, video only
mss-3397                        ismv       1280x720   3397k , H264, video only
dash-1_V_video_1                mp4        1280x720   DASH video 3397k , mp4_dash container, avc1.64001F, video only
hls-3586                        mp4        1280x720   3586k , avc1.64001f, video only
mp4-101                         mp4        640x360     650k
mp4-102                         mp4        960x540    1500k
mp4-103                         mp4        960x540    2250k
mp4-104                         mp4        1280x720   3400k
mp4-1001                        mp4        unknown    (best)
[download] Finished downloading playlist: BB1oaoek
$

@nealmcb PhantomJS is only used by 4 extractors, and it’s only used as a last resort when there is no other path to extraction except by actually executing the site’s javascript. (And for the most widely used of those 4 extractors, there is an alternative available). PhantomJS is likely not relevant to this issue.

If you’re writing an extractor and the information you need isn’t found in the page source/html, then what you should try to do is to find out what the javascript is doing to get the required info and recreate that in the extractor code; this usually amounts to making an API request (or multiple). In your browser’s dev tools you can monitor all network requests being made as the page/video loads. Start there; more often than not the XHR calls will tell you all you need to know and you won’t even need to look at the javascript code itself.

Congratulations and thanks to you all for helping improve yt-dlp. SInce there are so many PRs active, for the record, here is the MSN PR: https://github.com/yt-dlp/yt-dlp/pull/12513

On Sun, Mar 2, 2025 at 7:21 AM thedenv @.***> wrote:

You have still not created a Pull Request. Go here master…thedenv:yt-dlp:master https://github.com/yt-dlp/yt-dlp/compare/master...thedenv:yt-dlp:master and select the branch you want to PR. Me or another maintainer will then review your code.

Its my first time and thankfully, now I know what to do and I have just did what you asked. What a rollercoaster! Thank you for your communication and patience. Im still a noob at all this.

— Reply to this email directly, view it on GitHub https://github.com/yt-dlp/yt-dlp/issues/3225#issuecomment-2692755473, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA5FMFS7UPSXJ5IOFGIA7T2SMHYDAVCNFSM6AAAAABYDPRIUGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJSG42TKNBXGM . You are receiving this because you were mentioned.Message ID: <yt-dlp/yt-dlp/issues/3225/2692755473 @.***> [image: thedenv]thedenv left a comment (yt-dlp/yt-dlp#3225) https://github.com/yt-dlp/yt-dlp/issues/3225#issuecomment-2692755473

You have still not created a Pull Request. Go here master…thedenv:yt-dlp:master https://github.com/yt-dlp/yt-dlp/compare/master...thedenv:yt-dlp:master and select the branch you want to PR. Me or another maintainer will then review your code.

Its my first time and thankfully, now I know what to do and I have just did what you asked. What a rollercoaster! Thank you for your communication and patience. Im still a noob at all this.

— Reply to this email directly, view it on GitHub https://github.com/yt-dlp/yt-dlp/issues/3225#issuecomment-2692755473, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA5FMFS7UPSXJ5IOFGIA7T2SMHYDAVCNFSM6AAAAABYDPRIUGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJSG42TKNBXGM . You are receiving this because you were mentioned.Message ID: <yt-dlp/yt-dlp/issues/3225/2692755473 @.***>

– Neal McBurnett https http://neal.mcburnett.org/😕/ bcn.boulder.co.us/~neal/

You have still not created a Pull Request. Go here https://github.com/yt-dlp/yt-dlp/compare/master...thedenv:yt-dlp:master and select the branch you want to PR. Me or another maintainer will then review your code.

Yes you would fork yt-dlp, create a new branch, then modify yt-dlp/extractor/msn.py, and then create a pull request. Do take a look at the yt-dlp coding conventions: https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#yt-dlp-coding-conventions You will need to adjust the logic of your script a bit. For example in yt-dlp you can not import requests directly, but would use helper functions such as self._download_json

@thedenv Nice work. Instead of having this be an external script, consider adjusting the MSN extractor in yt-dlp and opening a PR. That way people can just use yt-dlp directly instead of having to download and setup your script.

The problem MSN URLs mentioned in existing issues no longer lead to the expected content as far as I can tell.

Someone could open an issue upstream (or post here) suggesting various currently valid MSN URLs with playable media that should be supported, describing for each what is expected to be found (one MSN video, as above, a video from YT or DM or some other external host, a playlist of any of these). Suggestions as to what values should be expected for the standard JSON extraction parameters would also be useful.

What the video above gets
$ python3 -m youtube_dl -j 'https://www.msn.com/en-us/weather/topstories/midwest-northeast-bracing-for-potentially-dangerous-long-duration-heat-wave-next-week/vi-BB1oaoek?ocid=windirect&cvid=981228a2079b4d07908de15c5820a4e2&ei=98' | jq 'del(.["formats","requested_formats"])' 
{
  "id": "BB1oaoek",
  "title": "Midwest, Northeast bracing for potentially dangerous, long-duration heat wave next week",
  "subtitles": {
    "en-us": [
      {
        "url": "https://prod-video-cms-amp-microsoft-com.akamaized.net/tenant/amp/entityid/BB1oaoek?blobrefkey=closedcaptionen-us&$blob=1",
        "ext": "ttml"
      }
    ]
  },
  "description": "The first long-duration heat wave of the summer is expected to blast millions of people from the Midwest to the Northeast starting this weekend and lasting into next week.",
  "thumbnail": "https://img-s-msn-com.akamaized.net/tenant/amp/entityid/BB1oaoed.img",
  "uploader": "Fox Weather",
  "duration": 209,
  "timestamp": 1718287609,
  "n_entries": 1,
  "playlist": "BB1oaoek",
  "playlist_id": "BB1oaoek",
  "playlist_title": null,
  "playlist_uploader": null,
  "playlist_uploader_id": null,
  "playlist_index": 1,
  "extractor": "MSN",
  "webpage_url": "https://www.msn.com/en-us/weather/topstories/midwest-northeast-bracing-for-potentially-dangerous-long-duration-heat-wave-next-week/vi-BB1oaoek?ocid=windirect&cvid=981228a2079b4d07908de15c5820a4e2&ei=98",
  "webpage_url_basename": "vi-BB1oaoek",
  "extractor_key": "MSN",
  "thumbnails": [
    {
      "url": "https://img-s-msn-com.akamaized.net/tenant/amp/entityid/BB1oaoed.img",
      "id": "0"
    }
  ],
  "display_id": "BB1oaoek",
  "upload_date": "20240613",
  "requested_subtitles": null,
  "http_headers": null,
  "format": "hls-3586 - 1280x720+dash-5_A_aac_und_2_96021_2_1_1 - audio only (DASH audio)",
  "format_id": "hls-3586+dash-5_A_aac_und_2_96021_2_1_1",
  "width": 1280,
  "height": 720,
  "resolution": null,
  "fps": null,
  "vcodec": "avc1.64001f",
  "vbr": null,
  "stretched_ratio": null,
  "acodec": "mp4a.40.2",
  "abr": null,
  "ext": "mp4",
  "fulltitle": "Midwest, Northeast bracing for potentially dangerous, long-duration heat wave next week",
  "_filename": "Midwest, Northeast bracing for potentially dangerous, long-duration heat wave next week-BB1oaoek.mp4"
}

Then extractor patches can be validated against those and via a PR upstream and the process of pulling upstream changes to yt-dlp (or a PR here if someone makes one) your wishes could be realised.