yt-dlp: YouTube videos with lots of comments seem impossible to download, keep failing while downloading comments

Checklist

I’m asking a question
I’ve looked through the README and FAQ for similar questions
I’ve searched the bugtracker for similar questions including closed ones

Question

Trying to download videos with comments is very failure prone: why is that? I thought adding --cookies cookies.txt and especially --sleep-requests 1 would prevent them from erroring out but it still happens. Thus, videos with 20k+ comments seem impossible to download. Especially annoying with --sleep-requests 1 because it will take over 10 minutes for it to download the first 5k or so comments, only to fail anyway.

It doesn’t even matter how many comments a vid has, even some vids with 800 comments will fail at like 200 or 600. What can I do to prevent these from erroring out?

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 24 (10 by maintainers)

Most upvoted comments

The reason I ask is because I’m currently redownloading my entire youtube-dl archive using yt-dlp in order to get comments. I just took the urls from the archive.log and find+replace’d them to make standard video urls which I then put in the batch.txt. It’s just annoying having to delete the videos that I don’t want again

If it was possible to just get the comments without redownloading the videos, I could have done that and not have to use 2x the storage to fit everything before finally comparing the old with the new and deleting the old.

I’m actually downloading videos using the worst format quality and deleting the vids so that I can transfer the comment info.json’s to my pc, where I then download the vid without comments in the best format quality.

Use --skip-download

is it possible to make the archiving of comments a separate thing, using a separate archive.log (archive-comments.log), so that we could first download videos without comments, and then go back and download the comments to those videos at a later date?

Just run these seperately:

yt-dlp --write-comments --skip-download --download-archive archive-comments.log --force-write-archive -- URL
yt-dlp --download-archive archive.log -- URL

(for some reason, some channel urls also downloaded those channels’ personal playlists such as their favourites, which I don’t want).

youtube-dl nowadays downloads whatever youtube page the link points to. So if you point it to the channels home page, it will download all the videos in the home page and not the channel’s uploaded videos. The workaround is to give link to the videos page like this: https://www.youtube.com/c/3blue1brown/videos. yt-dlp on the other hand, automatically redirects the home page to the videos page since that is the expected behaviour by most users.

As a side, over the past 3 days I’ve been having random videos throttle at 50KB/s which I could either wait them out, or restart the script, but restarting takes about half an hour to retry the videos with thousands of comments that it consistently errors on before it finally gets to the throttled video, only to then throttle maybe 3 videos later.

This is a recent thing. There is already an issue open in youtube-dl. Currently the only known workaround is to use a multi-connection downloader like aria2c

He’s running inferior specs compared to my main pc, but superior specs compared to my mini pc: i5-7300HQ, 8GB RAM, on wi-fi.

Specs don’t matter. The only thing that matters is the internet’s stability

I am closing this since I believe all the questions have been answered

pukkandan on Jun 21, 2021