yt-dlp: [Youtube] throttling and inconsistent speed in Russia
šØClick here for workaroundsšØ
DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE
- I understand that I will be blocked if I intentionally remove or skip any mandatory* field
Checklist
- Iām reporting that yt-dlp is broken on a supported site
- Iāve verified that I have updated yt-dlp to nightly or master (update instructions)
- Iāve checked that all provided URLs are playable in a browser with the same IP and same login details
- Iāve checked that all URLs and arguments with special characters are properly quoted or escaped
- Iāve searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
- Iāve read the guidelines for opening an issue
- Iāve read about sharing account credentials and Iām willing to share it if required
Region
No response
Provide a description that is worded well enough to be understood
For the last couple of days Iām having constant throttling issues with Youtube downloads. Itās the same on the latest public (2024.07.09) and nightly (2024.07.11) releases. I use yt-dlp by itself, with ytdlp-interface and with mpc-hc (I also use PotPlayer which has itās own way of delivering Youtube streams and itās also having trouble, but I donāt know if itās relevant here). In all cases the speed I get is inconsistent, but consistently bad. Itās usually hovering somewhere around 200KiB or lower, sometimes going up to 400-500 or having a rare spike of even faster speed. For some videos the base speed is better, for some itās worse, but I didnāt had one downloading at a normal speed for a couple of days.
Iāve tried adding PhantomJS and Iāve stopped getting a warning about it, but the speed and itās inconsistency didnāt really improved.
Also, when watching on the actual Youtube (in Firefox) I donāt have any noticeable playback issues. It doesnāt feel any slower to load the video or skip around.
Provide verbose output that clearly demonstrates the problem
- Run your yt-dlp command with -vU flag added (
yt-dlp -vU <your command line>) - If using API, add
'verbose': TruetoYoutubeDLparams instead - Copy the WHOLE output (starting with
[debug] Command-line config) and insert it below
Complete Verbose Output
[debug] Command-line config: ['-vU', 'https://www.youtube.com/watch?v=e8ehi-1FwUU']
[debug] User config "C:\Users\User\AppData\Roaming\yt-dlp\config": ['-o', 'c:/users/user/desktop/%(title)s.%(ext)s', '-r', '999K', '-f', '136+140', '--cookies-from-browser', 'firefox:ko0oafee.default-release']
[debug] Encodings: locale cp1251, fs utf-8, pref cp1251, out utf-8 (No VT), error utf-8 (No VT), screen utf-8 (No VT)
[debug] yt-dlp version nightly@2024.07.11.232833 from yt-dlp/yt-dlp-nightly-builds [cc1a3098c] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-7-6.1.7601-SP1 (OpenSSL 1.1.1k 25 Mar 2021)
[debug] exe versions: ffmpeg 4.2, phantomjs 2.1.1
[debug] Optional libraries: Cryptodome-3.20.0, brotli-1.1.0, certifi-2024.07.04, curl_cffi-0.7.0, mutagen-1.47.0, requests-2.32.3, sqlite3-3.35.5, urllib3-2.2.2, websockets-12.0
[debug] Proxy map: {}
Extracting cookies from firefox
[debug] Extracting cookies from: "C:\Users\User\AppData\Roaming\Mozilla\Firefox\Profiles\ko0oafee.default-release\cookies.sqlite"
Extracted 1195 cookies from firefox
[debug] Request Handlers: urllib, requests, websockets, curl_cffi
[debug] Loaded 1834 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Latest version: nightly@2024.07.11.232833 from yt-dlp/yt-dlp-nightly-builds
yt-dlp is up to date (nightly@2024.07.11.232833 from yt-dlp/yt-dlp-nightly-builds)
[youtube] Extracting URL: https://www.youtube.com/watch?v=e8ehi-1FwUU
[youtube] e8ehi-1FwUU: Downloading webpage
[debug] [youtube] Extracted SAPISID cookie
[youtube] e8ehi-1FwUU: Downloading ios player API JSON
[debug] [youtube] Extracting signature function js_8d9f6215_109
[debug] Loading youtube-sigfuncs.js_8d9f6215_109 from cache
[debug] Loading youtube-nsig.8d9f6215 from cache
[debug] [youtube] Decrypted nsig mLbhvbt9lcHfXg2xqNA => LuTqjoDb154j6w
[debug] [youtube] Extracting signature function js_8d9f6215_105
[debug] Loading youtube-sigfuncs.js_8d9f6215_105 from cache
[debug] Loading youtube-nsig.8d9f6215 from cache
[debug] [youtube] Decrypted nsig VwS3V6ro-hTWB4onsBQ => TiCGZvfd9Iu7sA
[youtube] e8ehi-1FwUU: Downloading m3u8 information
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, size, br, asr, vext, aext, hasaud, id
[info] e8ehi-1FwUU: Downloading 1 format(s): 136+140
[debug] Invoking http downloader on "https://rr4---sn-gvnuxaxjvh-n8vd.googlevideo.com/videoplayback?expire=1720824020&ei=dFyRZunLDKmqv_IP-b6yoAg&ip=109.237.236.140&id=o-AM6fE97szfyB0QHyf3y6DN1YA4URRH64SjJC64rZhhwb&itag=136&aitags=133%2C134%2C135%2C136%2C160%2C242%2C243%2C244%2C247%2C278%2C298%2C299%2C302%2C303&source=youtube&requiressl=yes&xpc=EgVo2aDSNQ%3D%3D&mh=kV&mm=31%2C29&mn=sn-gvnuxaxjvh-n8vd%2Csn-n8v7kn7y&ms=au%2Crdu&mv=m&mvi=4&pl=20&initcwndbps=762500&siu=1&bui=AXc671L83m4SFUlcVxtqgsp7NZrA6yPagMQZy6l7PiP0ih9BK1qnTI7BUYKV1c9hxg3JlZ-AhA&spc=NO7bAdTtcqmysz7c7NCgnLcxTzi2njcVfrLBjm2NE1z9FS2FaIfJnuWYFlCHAxBAr9gg_AaD8w&vprv=1&svpuc=1&mime=video%2Fmp4&ns=JfigxZQxmdTvHRE4Vq2DblMQ&rqh=1&gir=yes&clen=1747681569&dur=7075.066&lmt=1720670775794204&mt=1720802002&fvip=12&keepalive=yes&c=WEB&sefc=1&txp=7309224&n=TiCGZvfd9Iu7sA&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cxpc%2Csiu%2Cbui%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cns%2Crqh%2Cgir%2Cclen%2Cdur%2Clmt&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AHlkHjAwRAIgeOIdXTJjcTvAk9PwIS3A5aGcbhVVnZsYTlKWG2RWVHECIC8Oyalj5h856_nJR7m-2fNmKCewLUzWMFru6icAclOX&sig=AJfQdSswRAIgchPdmffm6X4IHBuPxVtCelmjO7C2FXe5_spuNvdBB9ECIHC2o8wPQXi56ujmLp5VMwdDdInJmD_BjmD6uhZj6ZT7"
[debug] File locking is not supported. Proceeding without locking
[download] Destination: c:\users\user\desktop\?#3rdAnniversaRyS?HOLOLIVE SONG KARAOKE ?? to CELEBRATE 3 YEARS HERE!!?3 Year Anniversary Stream?.f136.m
p4
[download] 0.3% of 1.63GiB at 85.55KiB/s ETA 05:31:27
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 9
- Comments: 81 (13 by maintainers)
This is not yt-dlp issue. The throttling is done using Deep Packet Inspection appliances by Russian government.
(updated 26.07.2024 20:25 MSK)
Workaround only for yt-dlp
ByeDPI (Windows/Linux)
./ciadpi-x86_64 -i 127.0.0.1 -p 10801 -d 1yt-dlp --proxy 'socks5://127.0.0.1:10801' ā¦Workaround system-wide
On Windows: GoodbyeDPI
youtube-domain.txttext file withgooglevideo.comtextgoodbyedpi.exe --blacklist youtube-domain.txt -6goodbyedpi.exe --blacklist youtube-domain.txt -e 1 --reverse-fragif the above didnāt workOn Linux: zapret
youtube-domain.txttext file withgooglevideo.comtextsudo ./nfqws --qnum=200 --dpi-desync=disorder2 --dpi-desync-split-pos=1 --hostlist=youtube-domain.txtsudo iptables -I OUTPUT -o <network_interface> -p tcp --dport 443 -m connbytes --connbytes-dir=original --connbytes-mode=packets --connbytes 1:6 -m mark ! --mark 0x40000000/0x40000000 -j NFQUEUE --queue-num 200 --queue-bypasssudo ip6tables -I OUTPUT -o <network_interface> -p tcp --dport 443 -m connbytes --connbytes-dir=original --connbytes-mode=packets --connbytes 1:6 -m mark ! --mark 0x40000000/0x40000000 -j NFQUEUE --queue-num 200 --queue-bypassOn Android: ByeDPIAndroid
To make it work in browsers, you must disable support for the QUIC protocol and the Kyber cipher suite. Chrome-based (Yandex, Edge, Opera, Vivaldi): disable in
chrome://flags(search by āQUICā, āKyberā), in Firefox:about:config(network.http.http3.enable false, Kyber is disabled by default).This is needed only for system-wide workarounds, proxy should work without additional tuning.
Just relax and be prepared Youtube to be blocked in Mother Russia! š
@krasnh
TL;DR: the fact youāre getting full speed in Firefox is a filtering deficiency. More information: https://ntc.party/t/замеГление-youtube-в-ŃŠ¾ŃŃŠøŠø/8055/ and https://ntc.party/t/обŃŃŠ¶Š“ение-замеГление-youtube-в-ŃŠ¾ŃŃŠøŠø/8074/
Does passing
--extractor-args "youtube:formats=dashy"help? You can try additionally passing-N 4(or adjust the value as desired)@WeAreLiving1984 an infinite loop retrying something that will never succeed, and the incessant hammering of the youtube API and videoplayback endpoints (or whatever site youāre downloading from) could lead to an IP and/or account ban
--retries/-Ris applied to more than just--throttled-rate. You could run into an unresolvable error that yt-dlp retries on, leading to an infinite loop, leading to your playlist/batch download stalling at that point, leading to an IP banNote: this thread will be locked soon to prevent the useful workarounds comment from disappearing into the hidden comments. Please only comment if you have something useful to add. No more chit chat please.
I found this utility (mentioned on the forum that Waujito linked): https://github.com/ValdikSS/GoodbyeDPI
Running it with these parameters
goodbyedpi.exe -6 --frag-by-snicompletely recovers the speed. Both in yt-dlp and players. But you need to grab it fromActionsnot fromReleases.goodbyedpi c -6 Š²Š¾Š¾Š±ŃŠµ ни на ŃŃŠ¾ не Š²Š»ŠøŃеŃ. РоŃŃŠµŠ»ŠµŠŗŠ¾Š¼.
Extra note about
goodbyedpi.exe. InReleasesthereās a script to turn it into a service for convenience. You just need to change arguments togoodbyedpi.exe -6 --frag-by-sniin the script and make sure that it will use theActionsversion of the exe.Yes, VladikSS found that RKN blocks it based on SNI. So even curl request with SNI replacement works:
(the request is for reference and may not work on your IPs, grab URI from --verbose option for qzmtchās yt-dlp command)
The fact that it works also breaks the theory about googleās internal tests as they wonāt relay on so low-level thing as TLS SNI. Also the thing that proves that this is an RKN jokes is throttling of the SNI-replaced packages to selectelās speedtest (curl requests provided by VladikSS in top of the thread, duplicated for fullness):
Adding
--extractor-args "youtube:formats=dashy"or-N 4separately didnāt change anything, but adding them both together did the trick. Speed is in 1MiB range, sometimes dipping to 700-800KiB or rising to 1.2-1.5MiB.What does video data being re-extracted even mean in this context?
I would caution against using
-R infinite, or usinginfiniteas an argument for any retry option, unless you will never be leaving any downloads unattended. Also, re-extracting the video so many times (as--throttled-ratedoes) could easily lead to an IP/account banAs a workaround, I add āāthrottled-rate 500K -R infinite --socket-timeout 1ā to yt-dlp. Throttling switches on after several seconds of downloading. It works for me.
The use of http3 does not help for me.
I just want to take a note about GoodbyeDPI, you donāt need SNI to fix that, it will work with just
-6argument. But if you used blacklist txt file as well, then you would need to add domaingooglevideo.comto that txt file and restart GDPI, this should fix throttling.Hello! I have written an userspace application for linux that bypasses these restrictions. This is a simple application aims only to googlevideo. After a bunch of research it turned out that the best decision is to split the packet in SNI position, just like the GoodbyeDPI does. I made it work with Firefox, Chrome and yt-dlp.
For some, this is a decision. What about Linux? )
You can find out more about this on the ntc party forum (Note: blocked in Russia, use VPN)
Russian IP here too, experiencing the same throttling issue. And, likewise, opening YT in a browser works as it should without any throttling. However, once I started using yt-dlp with an HTTPS proxy (by passing the --proxy option) the throttling in yt-dlp is gone and I can finally download videos normally, with download speed going back up to ~10MiB/s, exactly how it was before.
Something strange is definitely going on here, and itās hard to say who is to blame. This doesnāt seem like RKNās MO, at least not intentionally.
I also watch Youtube in MPC-HC (with yt-dlp) and PotPlayer (that uses itās own way of getting a stream), and both of them are struggling. 200KiB is not a stream speed you can watch 720p/1080p video at. 480p is watchable, but, you knowā¦
Roscomnadzor doesnāt throttle sites. It blocks them outright (or fails like with Telegram). And if there were attempting to block Youtube, runet wouldāve been in an uproar. Itās unlikely Iāve just missed the memo on that.
I donāt see any errors in the network logs, other then uBlock and Privacy Badger blocking things.
I donāt have ipv6, so canāt check how
-6would behave for me.Hello, as I can see by Yokosvenās IP he is Russian. It is sad to say but Russian censorship seems to have started dropping random youtube packets (they stay it is due to googleās GGC ineffectiveness in Russia but it anyway uses Ireland/Sweden servers so thatās a lie. Also Itās hard to believe that google itself will random drop packets). I have 184 Kib/s with yt-dlp now when downloading 4k60 fps. Also I have noticible throttling in youtube itself(but I think youtube behaves better with that than yt-dlp). By the way everything turns out to be ok for ipv6. (9 Mib/s). But not too much ISPs have support for ipv6 now. I think here may be a workaround for this problem. May be multiple parallel connections or something like this.
yt-dlp and https proxy (I recommend a proxy on port 9002)
yt-dlp --no-check-certificates --proxy='https://xxx.xxx.xxx.xxx:xxxx' 'https://www.youtube.com/watch?v=xxxxxxxx'Avpanov, this is the best advice for increasing the download speed. I would like to notice that the owner of the Chrome browser and YouTube is the same company. YouTube can also help slow down traffic by manipulating local CDN and harming yt-dlp along the way.
yt-dlp URL -f 137 --downloader=curl --downloader-args="curl:--http3-only"Add -v to the arguments for detailed logging. Add -k to ignore certificates, useful in curl for Windows.
As it seems that this problem is not going anywhere any time soon and is likely to appear in other countries (as dpi technologies progresses), could an option to automatically launch byedpi from PATH be added to yt-dlp?
https://github.com/bol-van/zapret
maybe they want more authors to duplicate their content on Russian sites, like VK, Rutube, OK
Yea, but they were talking about youtube slowdown a lot in the past. Also here is an about 7 days-old announcement from Rostelecom that google canāt update GGC in Russia due to the US sanctions and here may be outperforms in youtube video load speed. BUT I havenāt seen any Russian IP in googlevideo after that announcement. And I think it wasnāt here for a lot of time so we all have been watching youtube from foreign cache servers which are unrelated to Russia. So I dont think that google updated software on their servers to invalidate random packets and send random rsts for Russian ips. And even more that I got that thing only on ipv4 and http/1.1 (=> over tcp proto). When I completly disabled quic on firefox (disabled http3 and http2, network.http.http(2,3).enable(d) in about:config, you can try it and check load speed) it has started to load on 200 KiB/s with tons of random tcp errors in wireshark. When I enabled ipv6 and keeped firefox to use http/1.1 everything started to load normally. So I dont think that google ingeneers are too dumb to detect Russian ipv6 if they wish to do so. And I donāt think that they will support quic on both ipv4 and ipv6, tcp on ipv6 and slow down only tcp on ipv4. So logic statement from this is a middleware which loves Russian children and want to protect them from Western propaganda, Grand Roscomnadzor. They may be doing tests for youtube slowdown and checking how their servers behaves with this. And if this is true the most we can do is to pray that they will not touch QUIC.
But also it may be google that wish to move completly on quic and just tests http/1.1 slowdown for youtube for better performance on their servers.
So in my opinion for this time it seems like the solution is http/3 support. yt-dlp uses HTTP/1.1 with TCP when firefox/chrome uses HTTP/3 with QUIC and itās the reason why you may be ok in browser and have bad speed with yt-dlp. Interesting that I didnāt got quic earlier in firefox by default. And got scared that this is the end š. May be firefox needed restart when I disabled ipv6 or anything else which may prevent it from establishing http/3 and enforce fallback to http/1.1.
I think the behavior is expected because one thing is to load chunks of video as it is playing and another is to download the video as fast as possible. Unluckily for us that might be done for performance/speed reasons but it turns out to break under that censor of Roscomnadzor. It seems like the most that contributors can do is to fine-tune module to behave as youtube itself. About your browser you can check network logs of youtube tab (in developer options) and see NS_BINDING_ABORTEDās. Or open wireshark and see how tcp packets are invalidated. Also you can try to pass -6 option for yt-dlp to force ipv6 (if it is available for you) and check the speed.
Yes, agreed. But it seems like some ISPs (Rostelecom for me) provide ipv6 and donāt put censor for it. Even other services which are completly blocked on ipv4 are available on ipv6.
Another interesting option is that youtube drops to http/1.1 on firefox on ipv4 (unstable, bad speed), http/3 on firefox on ipv6 and uses quic on chrome which has no drops, clear logs, good speed and works for ipv4. May be the solution is to implement quic?
Tried 2024.07.14.232743. Everything is the same, so far. Iāll stop reporting on each nightly build, until thereās one that brings noticeable changes.
Have tried 2024.07.13.232701. No noticeable difference in speed with and without
--extractor-args "youtube:formats=dashy" -N 4. Without still ~200KiB. With still starts fast and slows down. I get read time outs more frequently, but itās the same on nightly or stable.Things are getting worse for me. Even with these parameters, what Iām seeing more and more is that the download will start fast, but then slow down to a crawl of 200~400KiB, sometimes getting faster, but then slowing back down. Downloads of audio files are the worst part. Their speed is all over the place, including 0, leading to timeouts.
yes
Does
--extractor-args "youtube:formats=dashy"download the same thing but using a different protocol? I canāt understand that from the doc.