YoutubeDL-Material: [BUG] The same video is displayed in several.

Describe the bug The webpage is not responding. The same video is displayed in several.

To Reproduce Steps to reproduce the behavior:

git clone https://github.com/Tzahi12345/YoutubeDL-Material.git
docker build -t ytdl:0.1 .
edit docker-compose.yml file (container name, mount path)
goto YTDL setting. select yt-dlp, use cookies
then Subscribe to about 15 channels. (It happened when I subscribed to a new channel or new video)
After a few hours the webpage becomes unresponsive.
So I restarted the container.

Expected behavior The same video appears.

Screenshots

Environment

After downloading git clone, I made an image with docker build.

Additional context I don’t know if it’s a virtual machine problem or a Docker problem. (I’m running an Ubuntu virtual machine on xcp-ng, and I’ve set up 6 cpu and 8 GB of ram. Also, I am using a Synology NAS, and it is mounted through smb.) The webpage goes unresponsive within a few hours of running the container. So I had no choice but to restart the container. I don’t know if that’s the reason, but the same video appears multiple times.

version: "2"
services:
    ytdl_material:
        environment:
            ALLOW_CONFIG_MUTATIONS: 'true'
            ytdl_mongodb_connection_string: 'mongodb://ytdl-mongo-db:27017'
            ytdl_use_local_db: 'false'
            write_ytdl_config: 'true'
        restart: always
        volumes:
            - ./appdata:/app/appdata
            - /mnt/z/audio:/app/audio
            - /mnt/z/video:/app/video
            - /mnt/z/subscriptions:/app/subscriptions
            - ./users:/app/users
        ports:
            - "8998:17442"
        container_name: ytdl
        image: ytdl:0.1
    ytdl-mongo-db:
        image: mongo:latest
        restart: always
        ports:
            - "27017:27017"
        logging:
            driver: "none"
        container_name: mongo-db
        volumes:
            - ./db/:/data/db

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 33 (10 by maintainers)

Most upvoted comments

I pushed another change that will hopefully fix this. Beforehand, the logic to avoid duplicates used the path to the file which was inconsistent as mentioned above due to different formatting. Now this is done just using the video URL (see commit here).

Note that #420 should fix this problem regardless once that gets merged.

Tzahi12345 on Aug 24, 2021

You might have pulled a slightly outdated nightly build there before, because there have been two builds in rapid succession and sometimes DockerHub takes a little while to pick up on a new build.

Glad to hear it seems solved for you. As for UI responsiveness, that as well is something that yours truly is constantly trialing with a database of… oh… you know… over 30k videos 😄 (and atm 48 subscriptions)

It’s something that is currently being looked into a lot more, so maybe that helps you rest a little easier. 😉

#420 will definitely help a lot in this regard as well. Overall YoutubeDL-Material has undergone already one MASSIVE increase in performance on large databases both thanks to migrating to MongoDB which is a fair step up from running off of json files as well as the recent change to load the UI right away on container start rather than waiting for a bunch of background processes to finish which for heavy users like me meant something between 10-20 minutes of start up time or even more until I could get into the UI. (and more it was towards the end for sure…)

We’ve come a long way and I’m very optimistic about more optimizations happening that’ll make using the application even more fun and reliable. 😃

GlassedSilver on Aug 25, 2021

I pushed another change that will hopefully fix this. Beforehand, the logic to avoid duplicates used the path to the file which was inconsistent as mentioned above due to different formatting. Now this is done just using the video URL (see commit here).

Note that #420 should fix this problem regardless once that gets merged.

I think my problem has been solved. However, the problem recurred when the YTDL frontend fell into unresponsiveness. (It seems to be a disk I/O problem or a network I/O problem, as the problem occurs when receiving a large amount of video. I’m not sure). In this case, I was able to avoid the problem by re-subscribing to that channel. Other than the above problem, it worked very correctly. Thank you.

arithxp on Aug 25, 2021

I’ve been very busy lately, so I’m late to check. sorry. I checked today and the problem seems to have been resolved. thank you.

arithxp on Aug 20, 2021

While we’re talking about this, I’ve had my json database corrupted due to bad memory in my server, after recovering from backup of the db I have entries that don’t have a matching file since it was already deleted. I think it would be worthwhile to also check for any entries for which the files don’t exist anymore. Maybe on container startup, or just a button settings.

QrchackOfficial on Aug 14, 2021