yt-dlp: [youtube] nsig extraction failed: You may experience throttling for some formats
DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE
- I understand that I will be blocked if I remove or skip any mandatory* field
Checklist
- I’m reporting a bug unrelated to a specific site
- I’ve verified that I’m running yt-dlp version 2022.08.08 (update instructions) or later (specify commit)
- I’ve checked that all provided URLs are playable in a browser with the same IP and same login details
- I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
- I’ve searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
- I’ve read the guidelines for opening an issue
Provide a description that is worded well enough to be understood
I assume that this is a new player as I could not find a reference to it in the repo. Receive WARNING: [youtube] nsig extraction failed: You may experience throttling for some formats for all actions against the affected video.
Provide verbose output that clearly demonstrates the problem
- Run your yt-dlp command with -vU flag added (
yt-dlp -vU <your command line>) - Copy the WHOLE output (starting with
[debug] Command-line config) and insert it below
Complete Verbose Output
I meant to include this properly but the entire js player is output to the log, so Github tells me "There was an error creating your issue: body is too long (maximum is 65536 characters)."
Please forgive me, here is a pastebin link instead: https://pastebin.com/62j1x2yd
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 10
- Comments: 145 (46 by maintainers)
Links to this issue
Commits related to this issue
- [jsinterp] Truncate error messages Related: #4635 — committed to yt-dlp/yt-dlp by pukkandan 2 years ago
- [extractor/youtube] Add fallback to phantomjs Related #4635 — committed to yt-dlp/yt-dlp by pukkandan 2 years ago
- [jsinterp] Fix bug in operator precedence Fixes https://github.com/yt-dlp/yt-dlp/issues/4635#issuecomment-1226659543 — committed to yt-dlp/yt-dlp by pukkandan 2 years ago
- [jsinterp] Fix `_separate` Ref: https://github.com/yt-dlp/yt-dlp/issues/4635#issuecomment-1231126941 — committed to yt-dlp/yt-dlp by pukkandan 2 years ago
- [jsinterp] Workaround operator associativity issue https://github.com/yt-dlp/yt-dlp/issues/4635#issuecomment-1235384480 — committed to yt-dlp/yt-dlp by pukkandan 2 years ago
- [jsinterp] Workaround operator associativity issue * temporary fix for player 5a3b6271 [1] 1. https://github.com/yt-dlp/yt-dlp/issues/4635#issuecomment-1235384480 — committed to dirkf/youtube-dl by pukkandan 2 years ago
- [jsinterp] Workaround operator associativity issue * temporary fix for player 5a3b6271 [1] 1. https://github.com/yt-dlp/yt-dlp/issues/4635#issuecomment-1235384480 — committed to belamenso/youtube-dl by pukkandan 2 years ago
I’ll make a proper release when I can. Meanwhile, here’s a build: https://github.com/pukkandan/yt-dlp-dev/releases/tag/2022.08.13.235132
Do test it out and let me know (with log) if there are any signatures that still fail
There seems to be a new player
5a3b6271and it looks like PhantomJS did it’s job (?):Tested with video isC8DbsvS8A.
TL;DR There is no impact on metadata extraction, and high quality unthrottled formats are still available in most cases.
YouTube has changed the n-param js function to be more complex than our js interpreter can handle (the infamous throttling parameter).
This issue only applies to web client formats, where the formats will be throttled and de-prioritized. There isn’t much to worry about since yt-dlp prefers android client formats over web client by default. Android formats don’t suffer from this and has most if not all the high quality formats web has.
I believe @pukkandan is attempting to write a fix for this. We’ll also need to fix the printing the entire js file to console too… 😉
Related: https://github.com/LuanRT/YouTube.js/issues/134, https://github.com/TeamNewPipe/NewPipe/issues/8760, https://github.com/ytdl-org/youtube-dl/issues/31164, https://github.com/TeamNewPipe/NewPipeExtractor/issues/902
Yes.
You could, but it is<strike>n’t yet</strike> superseded by https://github.com/ytdl-org/youtube-dl/commit/55c823634db890a328ffc23588fcd6f35d9b3ddf.
@pukkandan YouTube added its new player 113ca41c that throttled some or all of the video’s formats:
FWIW, at least the example given by @gamer191 downloads at full speed now without warning. 🙂 (yt-dlp built with the latest commit) Yesterday it was ~600kb/sec wit aria2…
This signature function is way more complicated than anything we have previously dealt with.
WIP branch if anyone wants to help out: https://github.com/yt-dlp/yt-dlp/compare/master...pukkandan:yt-dlp-dev:features/jsinterp
Ive trimmed it in the above branch
A similar fix works for yt-dl. But looking forward to the version with something like
let a=100; a/=/[0-9]+/.exec('divide by 20 today')[0];Aaand… A new one again:
At least it’s decrypted fine now. 🙂
The reason for PhantomJS doesn’t finish is simple: the passed code is bogus
Bonus: here’s the code https://gist.github.com/7cd88010f68419df69bd0778a0fc52e1
With PhantomJS throttled as well:
Player 1f7d5369 contains this element of a JS list:
So regex. Quite unpleasant.
yt-dlp’s breakage is not (just) due to quoting issue. Newpipe uses a full JS interpretter but we use an inhouse approximation of one. So fixing the full issue is more tedious for us and will take time. You can see my WIP branch above (and contributions to it are welcome)
@TheBlode What’s the urgency? Despite the warning, everything currently works as expected due to our fallbacks(No longer the case)don’t give them ideas 😅
… Google have started serving today new ytplayer
a7eb1f5d; seems compatible with latestjsinterp.py👍 :The issue is operator associativity. This expression
should evaluate as
((205000/1E3)%37)+(64+28) == 112but instead we’re calculating(205000/(1E3%37))+(64+28) == 205092. Operators of the same precedence need to be split at the same time rather than just taking the leftmost split. Then we could have a list of expressions that can be reduced pairwise from left to right, or right-to-left, as with eg**:4**3**2 == 4**9.In that case, you could probably update to:
https://github.com/ytdl-patched/yt-dlp/releases/tag/2022.08.31.810but Google 😡 may have already spoiled it for you by releasing ytplayer
c57c113c<strike>(which hasn’t yet arrived in my part of the woods)</strike> …I have added fallback to phantomjs. I still intend to keep fixing new players, so report if you find any that’s broken. But it will no longer be treated as high priority
PS: I’ll also make another release asap
God it feels like YouTube is watching this repo as we speak.
Some (but not all) age-gated content is being completely throttled (tested on master, to avoid an extremely long verbose log):
Sure, here’s a couple from my logs:
Not sure if this is expected, but the n value is different every time I rerun yt-dlp on the same video.
Can confirm on my side too, seems to be happening intermittently with most videos not being affected yet.
324f67b9 is not the only new player, I’ve noticed 4c3f79c5 too.
Completely unrelated to this issue. Use
-N/aria2cto speed up fragmented downloadsWorks again after commit https://github.com/yt-dlp/yt-dlp/commit/0468a3b3253957bfbeb98b4a7c71542ff80e9e06 😀️
If they keep using a combination of 8 letters and numbers, there are (26+10)⁸= 2.821109907×10¹² nsig’s, if I’m correct …
So, they can update them every 10 minutes if they want. And when they’ve used them all, they just add one more to it.
But jokes aside, is it really necessary to keep
_NSIG_TESTSthat are no longer used intest_youtube_signature.py?The phantomjs is working nicely for me, is it possible to suppress the WARNING in the stdout if the native nsig extraction fails but I have phantomjs?
We’ll have to use some full-fledged JS engine. PhantomJS support is already added. Other alternatives can be discussed in https://github.com/yt-dlp/yt-dlp/issues/1354.
Another option is to write one from scratch (https://github.com/yt-dlp/yt-dlp/issues/923), but I doubt this is feasible
It’s a mistake in the error message. PhantomJS will not actually help for this particular error
Due to different version numbers, the cache invalidation maynot always work for the unofficial daily builds (till actual release)
Thanks! That did the trick. 👍️
I have the same issue on the same player too, here are my logs if it’s at all helpful (removed some repetition from the download archive argument):
that’s a problem, because https://github.com/yt-dlp/yt-dlp/commit/992dc6b4863d0e60f2a1ce3933f67814d8a17f8d implements a timeout for PhantomJS (I am not sure how long the timeout is set for though, because I am not sure what unit the
5000is)use master or wait for the next release
for sure, but it certainly is interesting that they are throttling youtube-dl and yt-dlp, instead of outright blocking them
I believe that was fixed by https://github.com/yt-dlp/yt-dlp/commit/5e01315aa1ad0c56be33cb5b6a4d079068ee7145, unless youtube’s newest player also requires clearing your cache (and killing three birds with one stone would be quite an impressive achievement, I must say)
if you would like to self compile (personally I’d just use the link that @Vangelis66 sent), you just have to fork this repo and then run the
buildGithub action, and then Github will compile the master branch on their servers and add it to your repo (in the releases section)https://github.com/yt-dlp/yt-dlp/issues/1354
Well, player
c57c113cdid arrive over here… 😡 I’m usingyt-dlpdaily build2022.08.31.810, which was compiled from yt-dlp@https://github.com/yt-dlp/yt-dlp/commit/bfbecd1174a9e2ee08117352c26e664d36f1cc17 ; as already reported, the native JSInterp will barf on that new player 😭 … But I havePhantomJSalready installed and, in fact,yt-dlpcorrectly picks it up, yet it doesn’t fallback to it for n-sig decryption (at the same time it advises to install PhantomJS - it’s already installed):NB the contradiction:
What is actually going on here?
To complement this report:
I’ve gotten this new player too when downloading https://www.youtube.com/watch?v=rPxa8Q0bIOE on current release version. I have
phantomjsinstalled for nsig extraction, but that method also fails with this new player, hanging somewhere during the execution, similarly to what happens in https://github.com/yt-dlp/yt-dlp/issues/4716. Clearing cache doesn’t help either.Got a new player:
1f7d5369I downloaded master version (11-11-2022). No more “sig error” but slow / throttled download remained. Yes these videos are quite old i think. Try to download these videos it takes almost forever to download 100mb in a fast internet line (50mbs)!
Linux with bullseye 11.
… Yes, in the first hours (UTC) of Oct 11th, Google shipped YT-player
7a062b77that broke the native yt-signature decryptor; posting a more detailed log:Fortunately, the fallback to
PhantomJSdid work this time 👍 ; as it stands,youtube-dlis currently also broken, with no similar fallback 😢 …https://github.com/yt-dlp/yt-dlp/issues/4635#issuecomment-1235595110
YT’s player has been giving me a lot of problems (lagging, freezing) lately so hopefully they are testing fixes to those problems. Ironically enough just yesterday I had a video I could not play in Firefox and I downloaded it with yt-dlp instead, worked just fine, no throttling issue. So if YT is trying to break downloaders it’s having the opposite effect they intended.
… That sadist is reading this issue thread 😜 … Google have already moved to ytplayer
6a14c793:… Thus, in the course of ca. 24h we’ve had 3 different ytplayers:
a7eb1f5d=>f96f6702=>6a14c793What next? Hourly updates?
Actually, I can reproduce it on the latest version of yt-dlp. Ignore the PhantomJS timeout, it’s because my computer is slow as…
However, I can’t reproduce it on master
And when I ran a git bisect, with https://github.com/yt-dlp/yt-dlp/commit/b76e9cedb33d23f21060281596f7443750f67758 and https://github.com/yt-dlp/yt-dlp/commit/ae1035646a6be09c2aed3e22eb8910f341ddacfe as the starting commits, I got https://github.com/yt-dlp/yt-dlp/commit/1a7c9fad9f89b8994911c7d83f012da5f1aef445 as the first working commit. Obviously I’m human, and I could have made a mistake while doing the bisect (such as if I forgot to clear my cache one time), but I’d say that all evidence points to that commit having fixed this player
Haha. Perhaps some sadist is making these as their 20% time project🤣
There is some future-proofing in yt-dl’s version that may not be in yt-dlp, but https://github.com/yt-dlp/yt-dlp/issues/4635#issuecomment-1240203772 seems to be a false alarm.
The WARNING may be justified because running PhantomJS can be a lot slower than JSInterp.
Anyhow, https://github.com/ytdl-org/youtube-dl/issues/31173#issuecomment-1218088872 applies.
@jshir and if the version you are running is2022.09.01, can you please re-attempt to download one of the videos that causes the warning, with--verboseat the end of your command (eg:yt-dlp URL --verbose), and send a log of the output?So does yt-dlp’s master (see @nicolaasjan’s log). So either @jshir is using an outdated version of yt-dlp, or https://github.com/yt-dlp/yt-dlp/commit/1a7c9fad9f89b8994911c7d83f012da5f1aef445 and https://github.com/ytdl-org/youtube-dl/commit/7009bb9f3182449ae8cc05cc28b768b63030a485 fixed it (in which case Google really hasn’t been paying attention🤣)
@jshir can you please runyt-dlp --versionand send the outputFor whatever reason, yt-dl 7009bb9f3182449ae8cc05cc28b768b63030a485 (latest master) seems to handle f96f6702 fine.
I’m not the one you asked this question, but here is one (PhantomJS is not used):
You are the first person to report that player.
As such, can you please post a verbose log of it, so that the native js interpreter can be fixed?Sorry, I have autism, so I sometimes take things too literally. Anyway, I’m not a maintainer, so it’s not my decision, but I strongly doubt that the warning will be removed. What would be the point of the native js interpreter if no one cared whether it worked or not?
@jshir Out of interest, which player is falling back to PhantomJS? Is it
5a3b6271? If not, or if you don’t know, can you please post a verbose log?you could use
--no-warnings(although I wouldn’t recommend it). I’m not sure if there’s a way to only ignore specific warningsSometimes you may also get a 403 if nsig fails
*throttling-you can still download videos if the nsig extraction fails, but they will download at about 50kb/s
It seems that YouTube is blocking downloads, so they are changing every few days a new player.
good point, I forgot about that
The yt-dlp’s fallback method depends on the jsinterp’s lexer/parser. Therefore, it may not function as a fallback. This PR uses methods that do not(JS injection).
I would suggest doing that at yt-dlp as well. Also, phantomjs should not be used… However, current injecting methods are easily counteracted, so it may be necessary to consider better methods.
The challenge finishes in line 25, but that parsing issue is fixed, as noted above.
As to the latest player, the challenge is in lines 1279-1304, but neither the yt-dlp master nor my yt-dl WIP that handles 113ca41c extract the challenge code properly. Probably it’s this regex
/,][}",],()}(\[)/.The previous line in my log also indicates that I use the bundled version (linux_exe a.k.a. yt-dlp_linux), and I can’t compile it either because of the python version discrepancy.
New player: c57c113c https://www.youtube.com/s/player/c57c113c/player_ias.vflset/en_US/base.js `
`
It happens to me, even on a 4 year old Ryzen CPU so you are not alone.
@Vangelis66
I didn’t mean to post a duplicate, I scrolled through the comments but I didn’t see anyone else experience phantomjs to hang completely.
In that case, I’ll do my part in trying to convince more and more people to mirror their stuff on alt tech video sites.
Sadly I can’t do that, since Ubuntu 18.04’s
aptdoesn’t work with python versions above 3.6, and yt-dlp doesn’t work with 3.6 anymore. I’m not gonna go into specifics because I’ve already said it in the appropriate ticket and it’d be completely off topic here.@Vangelis66 No need to point to my supposed duplicates. I’ve been punished already by the hidden status. 😀️
PS https://github.com/yt-dlp/yt-dlp/issues/4635#issuecomment-1231126941 was not marked as duplicate.
@csehszlovakze
With respect, there’s no need in posting yet another duplicate 😉 : https://github.com/yt-dlp/yt-dlp/issues/4635#issuecomment-1231126941 https://github.com/yt-dlp/yt-dlp/issues/4635#issuecomment-1231135121
i.e. your case is not unique!
If I’m allowed a personal comment here, with the introduction of ytplayer
113ca41cGoogle managed (albeit briefly) to kill “two birds with one stone” 😠 ; invalidatingPhantomJStoo (fallback mechanism tojsinterp.py) doesn’t look like a coincidence here; if it was indeed intentional, they directly targeted https://github.com/yt-dlp/yt-dlp/commit/25836db6bea78501c514bfbe5840f305b33afdcd 😡 … All the more consolidates the suspicions of another dev:The ytplayer
113ca41crelated breakage has already been dealt with inyt-dlp’s master branch: https://github.com/yt-dlp/yt-dlp/commit/c4b2df872d0ab49da939bf8bda001fa4e2d2ea06 https://github.com/yt-dlp/yt-dlp/commit/d81ba7d491bf2c89246d8817438db48a5a4e4ae9PLease, update to that (master) and make sure you also perform
--rm-cache-dirto regain unthrottled YT downloads…here’s another one, but in my case, phantomjs just hangs.
after about 10 minutes I gave up:
Two issues (tested in yt-dl, though):
NaNhas to be added to the global namespace with valuefloat('nan').Then:
At least, I hope those values give unthrottled links.
Well, I think that’d be not necessary… Your posted log excerpts are ALL about youtube’s new player,
dc0c6770, and @kynndev was the first to post about it… The devs are already aware and have pushed a fix in theyt-dlpmaster branch 👍 … The fallback to usingPhantomJSworked nicely in my case (before the fix arrived…).WARNING: [youtube] 7XL7yNuIsVs: nsig extraction failed: You may experience throttling for some formats Install PhantomJS to workaround the issue n = 5EHDMgYLV6HPGk_Mu-kk ; player = https://www.youtube.com/s/player/dc0c6770/player_ias.vflset/en_US/base.js
Player 009f1d77 doesn’t exercise the regex and exception stuff just added. It has this interesting string
'][;;],'as well asDate()andMath.pow()expressions.Google have now already rolled out 😠 new player
009f1d77, but the latest release2022.08.19has no trouble decrypting the nsig (I’ve used the same video below for testing as @nicolaasjan previously in this thread):… and:
All in all, an exquisite job! Well done, pukkandan and company! 🥇
“extractors where javascript needs to be run” includes youtube
It’s only taken from PATH. Same for the other executables (except ffmpeg ofc). You can open a new issue if you want the option to set their location
Many thanks! 👍 FWIW, the linked Windows binary is 32-bit and has Vista SP2 (32-bit) as minimum OS requirement, so the same as
yt-dlp_x86.exe👍 (I’m just stating this, because I’ve seen some WinXP-compatible “adaptations” 😉 ofyt-dlpfloating elsewhere 😄 …).BTW, is there some way to declare a “custom”
phantomjs.exepath toyt-dlp, or is it exclusively picked up byyt-dlp_x86.exewhen placed adjacent to it?PS: I would also extend that same question to include the rest of the Windows binaries used as dependencies (i.e.
AtomicParsley.exe,RTMPdump.exe) - baringFFmpeg.exeandFFprobe.exe, for which there is--ffmpeg-locationalready), but I sense this is material for aNew Issue…When encountering the error, yt-dlp could have saved wrong function in cache. So clear the cache when trying new version
Did you delete yt-dlp’s cache first?
I tried with
w1C0fObjruo, and it worked.Thanks! 🙂 That fixed things for now.
Had to delete the cache (
~/.cache/yt-dlp/youtube-nsig) first though.The function also throws and catches errors now
Got something similar with using latest
masterbuild (3ce2933) on videofEaMDCbFi5Ifrom that playlist (download at ~48KiB/s…):Or use https://github.com/ytdl-patched/yt-dlp
@DavidH-2022 Wait for the next update
Turns out the satisfaction only applies in Py2.7, though I haven’t managed to work out why the operator fixes from 6d3e742 aren’t needed there. Anyhow, the version mentioned in #yt-dl/31173, which is the same problem resulting from player c81bbb4a as reported above, incorporates the same functionality.
So yes.
Ideally, there would be an extra line in the debug output, like this that is shown when running with
python -m yt_dlp:New player c81bbb4a, but (satisfyingly) doesn’t break yt-dl master, which understands
===.