mmaction2: Training process stuck at the beginning

I am going to training on my 4 action classes dataset, this is the dataset for reproducing this error. At first, i trained it on colab, but it stuck at 70th iterration in 1st epoch, then i change cfg.data.videos_per_gpu from 8 to 4 , then it stuck at 140th iterration in 1st epoch. I am wonderring why it stuck at the same place ? Is the problem of my video dataset or the problem of code ?

Then, i trained it on my computer whose GPU is GTX 1080 Ti, this time it stucked exactly at the beginning, it has been stuck for several hours, it is similar to this issue, this issue still occurred even i set workers_per_gpu=0

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 16 (13 by maintainers)

Most upvoted comments

Hi @rlleshi , afterwards i used subclip instead of ffmpeg_extract_subclip, the script is :

from moviepy.editor import VideoFileClip
import os
video_folder = "/media/weidawang/DATA/FFOutput/test_folder"
outputs_folder = video_folder + "/output_videos2/"

if not os.path.exists(outputs_folder):
    os.makedirs(outputs_folder)

def SplitVideoWithFixedNumberOfClips():
    for video_name in os.listdir(video_folder):
        video_path = os.path.join(video_folder, video_name)
        if os.path.isdir(video_path):  # exclude outputs_folder
            continue
        print(f"Processing {video_path}")

        with VideoFileClip(video_path) as video:
            duration = video.duration
            print(duration)
            interval = duration // 6
            print(interval)
            for i in range(5):
                starttime = int(interval * i)
                endtime = int(interval * i + 5)
                new = video.subclip(starttime, endtime)
                new.write_videofile(outputs_folder + video_name.split('.')[0] + "_" + str(i) + ".mp4", audio_codec='aac')

SplitVideoWithFixedNumberOfClips()

Although the videos outputed by this script won’t make training freeze anymore, the width and height of some videos are changed…

wwdok on May 1, 2021

Yes, the videos could be the possible cause of training freeze, because when people are preparing their own video dataset, in this process it may happen some unpredictable faults, in my case, that is the video meta info is unmatched with its real length, that is to say, i expect to cut video to 5 seconds, but the output video show 13 seconds in whick the last 8 seconds are null, this issue may only happen to me, but other kinds of video issue may happend to other peoples.

wwdok on Feb 1, 2021

Maybe i have found the reason, it is very likely a problem with the video. During i run the build_rawframe.py, i found some video happened following warning ‘Early stop with {i + 1} out of {len(vr)} frames.’ :

these problematic videos were cut out from some long videos with moviepy, so i modified the python script and re-generate videos with moviepy , this time all videos are normal and won’t happen above warning again. Now i have succefully trained 4 epochs without stuck.

wwdok on Jan 29, 2021

I have tried decoder == 0.4.1, it didn’t work. I will try RawFrames these days.

wwdok on Jan 27, 2021

Could you try to train on RawFrames? This can rule out the error caused by decoders. BTW if you are using decord, please use the version == 0.4.1. They introduced bugs on 0.4.2

innerlee on Jan 27, 2021