yt-dlp: Memory leaks

Checklist

  • I’m reporting a bug unrelated to a specific site
  • I’ve verified that I’m running yt-dlp version 2021.12.01. (update instructions)
  • I’ve checked that all provided URLs are alive and playable in a browser
  • I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
  • I’ve searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
  • I’ve read the guidelines for opening an issue

Description

I embed yt-dlp in my app and it cause continuous memory consumption increasing. Tried debug a little bit and i found that python gc can’t free some yt-dlp objects. Calling gc.garbage will return huge list of leaked objects

Verbose log

import gc
import yt_dlp as y

gc.set_debug(gc.DEBUG_UNCOLLECTABLE | gc.DEBUG_LEAK)

y.YoutubeDL().extract_info("https://www.youtube.com/watch?v=mIv2puMWFSc",download=False)

gc.collect()
print(gc.garbage)

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 1
  • Comments: 23 (19 by maintainers)

Most upvoted comments

Hello, I don’t know if I should’ve open a new issue but I also experience memory leaks with a more minimal example & they are bigger than 20MB over 1000 iterations.

To reproduce:

import tracemalloc
import yt_dlp

tracemalloc.start()
old = tracemalloc.take_snapshot()

count = 0
for _ in range(100):
    with yt_dlp.YoutubeDL() as yt:
        pass

    count += 1
    if count % 10 == 0:
        new = tracemalloc.take_snapshot()
        
        print(f'total count: {count}')
        for stat in new.compare_to(old, 'lineno')[:10]:
            if 'yt_dlp' in str(stat) and stat.size > 100 * 1024:
                print(stat)

        old = new

These are the results (all on yt-dlp==2023.11.16):

Python 3.9 (with lazy extractors): 182KiB per iteration 
total count: 10
.../python3.9/site-packages/yt_dlp/extractor/lazy_extractors.py:54: size=1101 KiB (+1101 KiB), count=19010 (+19010), average=59 B
.../python3.9/site-packages/yt_dlp/YoutubeDL.py:805: size=720 KiB (+720 KiB), count=10 (+10), average=72.0 KiB
total count: 20
.../python3.9/site-packages/yt_dlp/extractor/lazy_extractors.py:54: size=2201 KiB (+1101 KiB), count=38020 (+19010), average=59 B
.../python3.9/site-packages/yt_dlp/YoutubeDL.py:805: size=1440 KiB (+720 KiB), count=20 (+10), average=72.0 KiB
total count: 30
.../python3.9/site-packages/yt_dlp/extractor/lazy_extractors.py:54: size=3302 KiB (+1101 KiB), count=57030 (+19010), average=59 B
.../python3.9/site-packages/yt_dlp/YoutubeDL.py:805: size=2161 KiB (+720 KiB), count=30 (+10), average=72.0 KiB
total count: 40
.../python3.9/site-packages/yt_dlp/extractor/lazy_extractors.py:54: size=4402 KiB (+1101 KiB), count=76040 (+19010), average=59 B
.../python3.9/site-packages/yt_dlp/YoutubeDL.py:805: size=2881 KiB (+720 KiB), count=40 (+10), average=72.0 KiB
total count: 50
.../python3.9/site-packages/yt_dlp/extractor/lazy_extractors.py:54: size=5503 KiB (+1101 KiB), count=95050 (+19010), average=59 B
.../python3.9/site-packages/yt_dlp/YoutubeDL.py:805: size=3601 KiB (+720 KiB), count=50 (+10), average=72.0 KiB
total count: 60
.../python3.9/site-packages/yt_dlp/extractor/lazy_extractors.py:54: size=6603 KiB (+1101 KiB), count=114060 (+19010), average=59 B
.../python3.9/site-packages/yt_dlp/YoutubeDL.py:805: size=4321 KiB (+720 KiB), count=60 (+10), average=72.0 KiB
total count: 70
.../python3.9/site-packages/yt_dlp/extractor/lazy_extractors.py:54: size=7704 KiB (+1101 KiB), count=133070 (+19010), average=59 B
.../python3.9/site-packages/yt_dlp/YoutubeDL.py:805: size=5042 KiB (+720 KiB), count=70 (+10), average=72.0 KiB
total count: 80
.../python3.9/site-packages/yt_dlp/extractor/lazy_extractors.py:54: size=8804 KiB (+1101 KiB), count=152080 (+19010), average=59 B
.../python3.9/site-packages/yt_dlp/YoutubeDL.py:805: size=5762 KiB (+720 KiB), count=80 (+10), average=72.0 KiB
total count: 90
.../python3.9/site-packages/yt_dlp/extractor/lazy_extractors.py:54: size=9905 KiB (+1101 KiB), count=171090 (+19010), average=59 B
.../python3.9/site-packages/yt_dlp/YoutubeDL.py:805: size=6482 KiB (+720 KiB), count=90 (+10), average=72.0 KiB
total count: 100
.../python3.9/site-packages/yt_dlp/extractor/lazy_extractors.py:54: size=10.7 MiB (+1101 KiB), count=190100 (+19010), average=59 B
.../python3.9/site-packages/yt_dlp/YoutubeDL.py:805: size=7202 KiB (+720 KiB), count=100 (+10), average=72.0 KiB
Python 3.9 (without lazy extractors): 182KiB per iteration 
total count: 10
.../python3.9/site-packages/yt_dlp/extractor/common.py:787: size=1101 KiB (+1101 KiB), count=19026 (+19026), average=59 B
.../python3.9/site-packages/yt_dlp/YoutubeDL.py:805: size=720 KiB (+720 KiB), count=10 (+10), average=72.0 KiB
total count: 20
.../python3.9/site-packages/yt_dlp/extractor/common.py:787: size=2202 KiB (+1101 KiB), count=38036 (+19010), average=59 B
.../python3.9/site-packages/yt_dlp/YoutubeDL.py:805: size=1440 KiB (+720 KiB), count=20 (+10), average=72.0 KiB
total count: 30
.../python3.9/site-packages/yt_dlp/extractor/common.py:787: size=3302 KiB (+1101 KiB), count=57046 (+19010), average=59 B
.../python3.9/site-packages/yt_dlp/YoutubeDL.py:805: size=2161 KiB (+720 KiB), count=30 (+10), average=72.0 KiB
total count: 40
.../python3.9/site-packages/yt_dlp/extractor/common.py:787: size=4403 KiB (+1101 KiB), count=76056 (+19010), average=59 B
.../python3.9/site-packages/yt_dlp/YoutubeDL.py:805: size=2881 KiB (+720 KiB), count=40 (+10), average=72.0 KiB
total count: 50
.../python3.9/site-packages/yt_dlp/extractor/common.py:787: size=5503 KiB (+1101 KiB), count=95066 (+19010), average=59 B
.../python3.9/site-packages/yt_dlp/YoutubeDL.py:805: size=3601 KiB (+720 KiB), count=50 (+10), average=72.0 KiB
total count: 60
.../python3.9/site-packages/yt_dlp/extractor/common.py:787: size=6604 KiB (+1101 KiB), count=114076 (+19010), average=59 B
.../python3.9/site-packages/yt_dlp/YoutubeDL.py:805: size=4321 KiB (+720 KiB), count=60 (+10), average=72.0 KiB
total count: 70
.../python3.9/site-packages/yt_dlp/extractor/common.py:787: size=7704 KiB (+1101 KiB), count=133086 (+19010), average=59 B
.../python3.9/site-packages/yt_dlp/YoutubeDL.py:805: size=5042 KiB (+720 KiB), count=70 (+10), average=72.0 KiB
total count: 80
.../python3.9/site-packages/yt_dlp/extractor/common.py:787: size=8805 KiB (+1101 KiB), count=152096 (+19010), average=59 B
.../python3.9/site-packages/yt_dlp/YoutubeDL.py:805: size=5762 KiB (+720 KiB), count=80 (+10), average=72.0 KiB
total count: 90
.../python3.9/site-packages/yt_dlp/extractor/common.py:787: size=9905 KiB (+1101 KiB), count=171106 (+19010), average=59 B
.../python3.9/site-packages/yt_dlp/YoutubeDL.py:805: size=6482 KiB (+720 KiB), count=90 (+10), average=72.0 KiB
total count: 100
.../python3.9/site-packages/yt_dlp/extractor/common.py:787: size=10.7 MiB (+1101 KiB), count=190116 (+19010), average=59 B
.../python3.9/site-packages/yt_dlp/YoutubeDL.py:805: size=7202 KiB (+720 KiB), count=100 (+10), average=72.0 KiB
Python 3.12 (with lazy extractors): 146KiB per iteration 
total count: 10
.../python3.12/site-packages/yt_dlp/extractor/lazy_extractors.py:54: size=952 KiB (+952 KiB), count=19010 (+19010), average=51 B
.../python3.12/site-packages/yt_dlp/YoutubeDL.py:805: size=507 KiB (+507 KiB), count=10 (+10), average=50.7 KiB
total count: 20
.../python3.12/site-packages/yt_dlp/extractor/lazy_extractors.py:54: size=1904 KiB (+952 KiB), count=38020 (+19010), average=51 B
.../python3.12/site-packages/yt_dlp/YoutubeDL.py:805: size=1014 KiB (+507 KiB), count=20 (+10), average=50.7 KiB
total count: 30
.../python3.12/site-packages/yt_dlp/extractor/lazy_extractors.py:54: size=2856 KiB (+952 KiB), count=57030 (+19010), average=51 B
.../python3.12/site-packages/yt_dlp/YoutubeDL.py:805: size=1521 KiB (+507 KiB), count=30 (+10), average=50.7 KiB
total count: 40
.../python3.12/site-packages/yt_dlp/extractor/lazy_extractors.py:54: size=3808 KiB (+952 KiB), count=76040 (+19010), average=51 B
.../python3.12/site-packages/yt_dlp/YoutubeDL.py:805: size=2028 KiB (+507 KiB), count=40 (+10), average=50.7 KiB
total count: 50
.../python3.12/site-packages/yt_dlp/extractor/lazy_extractors.py:54: size=4760 KiB (+952 KiB), count=95050 (+19010), average=51 B
.../python3.12/site-packages/yt_dlp/YoutubeDL.py:805: size=2534 KiB (+507 KiB), count=50 (+10), average=50.7 KiB
total count: 60
.../python3.12/site-packages/yt_dlp/extractor/lazy_extractors.py:54: size=5712 KiB (+952 KiB), count=114060 (+19010), average=51 B
.../python3.12/site-packages/yt_dlp/YoutubeDL.py:805: size=3041 KiB (+507 KiB), count=60 (+10), average=50.7 KiB
total count: 70
.../python3.12/site-packages/yt_dlp/extractor/lazy_extractors.py:54: size=6664 KiB (+952 KiB), count=133070 (+19010), average=51 B
.../python3.12/site-packages/yt_dlp/YoutubeDL.py:805: size=3548 KiB (+507 KiB), count=70 (+10), average=50.7 KiB
total count: 80
.../python3.12/site-packages/yt_dlp/extractor/lazy_extractors.py:54: size=7616 KiB (+952 KiB), count=152080 (+19010), average=51 B
.../python3.12/site-packages/yt_dlp/YoutubeDL.py:805: size=4055 KiB (+507 KiB), count=80 (+10), average=50.7 KiB
total count: 90
.../python3.12/site-packages/yt_dlp/extractor/lazy_extractors.py:54: size=8568 KiB (+952 KiB), count=171090 (+19010), average=51 B
.../python3.12/site-packages/yt_dlp/YoutubeDL.py:805: size=4562 KiB (+507 KiB), count=90 (+10), average=50.7 KiB
total count: 100
.../python3.12/site-packages/yt_dlp/extractor/lazy_extractors.py:54: size=9520 KiB (+952 KiB), count=190100 (+19010), average=51 B
.../python3.12/site-packages/yt_dlp/YoutubeDL.py:805: size=5069 KiB (+507 KiB), count=100 (+10), average=50.7 KiB
Python 3.12 (without lazy extractors): 146KiB per iteration 
total count: 10
.../python3.12/site-packages/yt_dlp/extractor/common.py:787: size=953 KiB (+953 KiB), count=19026 (+19026), average=51 B
.../python3.12/site-packages/yt_dlp/YoutubeDL.py:805: size=507 KiB (+507 KiB), count=10 (+10), average=50.7 KiB
total count: 20
.../python3.12/site-packages/yt_dlp/extractor/common.py:787: size=1905 KiB (+952 KiB), count=38036 (+19010), average=51 B
.../python3.12/site-packages/yt_dlp/YoutubeDL.py:805: size=1014 KiB (+507 KiB), count=20 (+10), average=50.7 KiB
total count: 30
.../python3.12/site-packages/yt_dlp/extractor/common.py:787: size=2857 KiB (+952 KiB), count=57046 (+19010), average=51 B
.../python3.12/site-packages/yt_dlp/YoutubeDL.py:805: size=1521 KiB (+507 KiB), count=30 (+10), average=50.7 KiB
total count: 40
.../python3.12/site-packages/yt_dlp/extractor/common.py:787: size=3809 KiB (+952 KiB), count=76056 (+19010), average=51 B
.../python3.12/site-packages/yt_dlp/YoutubeDL.py:805: size=2028 KiB (+507 KiB), count=40 (+10), average=50.7 KiB
total count: 50
.../python3.12/site-packages/yt_dlp/extractor/common.py:787: size=4761 KiB (+952 KiB), count=95066 (+19010), average=51 B
.../python3.12/site-packages/yt_dlp/YoutubeDL.py:805: size=2534 KiB (+507 KiB), count=50 (+10), average=50.7 KiB
total count: 60
.../python3.12/site-packages/yt_dlp/extractor/common.py:787: size=5713 KiB (+952 KiB), count=114076 (+19010), average=51 B
.../python3.12/site-packages/yt_dlp/YoutubeDL.py:805: size=3041 KiB (+507 KiB), count=60 (+10), average=50.7 KiB
total count: 70
.../python3.12/site-packages/yt_dlp/extractor/common.py:787: size=6665 KiB (+952 KiB), count=133086 (+19010), average=51 B
.../python3.12/site-packages/yt_dlp/YoutubeDL.py:805: size=3548 KiB (+507 KiB), count=70 (+10), average=50.7 KiB
total count: 80
.../python3.12/site-packages/yt_dlp/extractor/common.py:787: size=7617 KiB (+952 KiB), count=152096 (+19010), average=51 B
.../python3.12/site-packages/yt_dlp/YoutubeDL.py:805: size=4055 KiB (+507 KiB), count=80 (+10), average=50.7 KiB
total count: 90
.../python3.12/site-packages/yt_dlp/extractor/common.py:787: size=8569 KiB (+952 KiB), count=171106 (+19010), average=51 B
.../python3.12/site-packages/yt_dlp/YoutubeDL.py:805: size=4562 KiB (+507 KiB), count=90 (+10), average=50.7 KiB
total count: 100
.../python3.12/site-packages/yt_dlp/extractor/common.py:787: size=9521 KiB (+952 KiB), count=190116 (+19010), average=51 B
.../python3.12/site-packages/yt_dlp/YoutubeDL.py:805: size=5069 KiB (+507 KiB), count=100 (+10), average=50.7 KiB

+19010s in outputs seem to be 10 times len(yt_dlp.YoutubeDL()._ies), but I couldn’t find why they leak.

+4
berkanteber on Dec 27, 2023

Yes, same.

+1
berkanteber on Mar 28, 2024

The latest version (2024.03.10) fixed the memory leak for me, I no longer see memory increase.

+1
vinismarques on Mar 28, 2024

@vinismarques @zackees see #8922 and check if #9032 resolves your memory leak issue

+1
bashonly on Feb 16, 2024

You are right about DEBUG_SAVEALL , but read docs again: gc.DEBUG_SAVEALL When set, ALL UNREACHABLE OBJECTS FOUND WILL BE APPENDED TO GARBAGE rather than being freed. This can be useful for debugging a leaking program. ~It will not be freed until program exit~(edit: some objects maybe but some objects may not) event if you will not set set_debug

+1
k3ns1n on Dec 11, 2021
Read more comments on GitHub
← yt-dlp: kick.com videos - HTTP Error 403: Forbidden
yt-dlp: "module yt_dlp.dependencies.Cryptodome has no attribute Cipher" error after upgrading from 2023.01.06 to 2023.02.17 via pip →