vision: VideoClips Assertion Error

Hello,

I’m trying to load a big video. Following https://github.com/pytorch/vision/issues/1446 I used a VideoClips object, but it’s crashing when trying to get clips with certain ids with this error:

AssertionError Traceback (most recent call last) <ipython-input-9-6e97949ad7f5> in <module>() ----> 1 x = video_clips.get_clip(1)

/usr/local/lib/python3.6/dist-packages/torchvision/datasets/video_utils.py in get_clip(self, idx) 324 video = video[resampling_idx] 325 info[“video_fps”] = self.frame_rate –> 326 assert len(video) == self.num_frames, “{} x {}”.format(video.shape, self.num_frames) 327 return video, audio, info, video_idx

AssertionError: torch.Size([0, 1, 1, 3]) x 32

The code I use is just this:

from torchvision.datasets.video_utils import VideoClips
video_clips = VideoClips(["test_video.mp4"], clip_length_in_frames=32, frames_between_clips=32)
for i in range(video_clips.num_clips()):
    x = video_clips.get_clip(i)

video_clips.num_clips() is much bigger than the ids that are failing. Changing the clipt_length or frames_between doesn’t help.

Checking the code I see [0,1,1,3] is returned by read_video when no vframes are read: https://github.com/pytorch/vision/blob/85b8fbfd31e9324e64e24ca25410284ef238bcb3/torchvision/io/video.py#L251-L254 But, for some clip ids and clip_lengths it’s just that the sizes don’t match, as the assertion error is something like this AssertionError: torch.Size([19, 360, 640, 3]) x 128

I followed the issue to _read_from_stream and checked no AV exceptions where raised. And running this part of the function: https://github.com/pytorch/vision/blob/85b8fbfd31e9324e64e24ca25410284ef238bcb3/torchvision/io/video.py#L144-L150 I saw that for an start_pts=32032, end_pts=63063 it returned just one frame on frames with pts=237237. Which is later discarted as it’s a lot bigger than end_pts.

Also, the stream.time_base is Fraction(1, 24000) which doesn’t match the start and end pts provided by VideoClips.

So it seems there is a problem with the seeking on my video. But it has a standard h264 encoding and I have no problem reading it sequentially with pyav. I’m wondering if I’m doing something wrong or there might be an issue with the read_video seeking (as the warning says it should be using seconds?).

This is the video info according to ffmpeg:

Metadata: major_brand : mp42 minor_version : 0 compatible_brands: mp42isom creation_time : 2016-10-10T15:36:46.000000Z Duration: 00:21:24.37, start: 0.000000, bitrate: 1002 kb/s Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 640x360 [SAR 1:1 DAR 16:9], 900 kb/s, 23.98 fps, 23.98 tbr, 24k tbn, 47.95 tbc (default) Metadata: handler_name : Telestream Inc. Telestream Media Framework - Release TXGP 2016.42.192059 encoder : AVC Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 93 kb/s (default) Metadata: handler_name : Telestream Inc. Telestream Media Framework - Release TXGP 2016.42.192059

Thanks!