vidgear: OpenCV video decoding too slow for vidgear - ffmpeg might be better suited for decoding

Detailed Description

I’ve ran some encode and decode benchmarks, and it’s becoming fairly obvious, that VidGear (CamGear API) is currently unable to decode 1080p60 video on computers where, given certain settings, ffmpeg can encode 1080p60 in real time or slightly faster without hardware acceleration.

Context

Currently, it’s impossible to pass 1080p60 H264 (only tested codec) video through VidGear on computers that should be able to decode 1080p60 video just fine. The idea would be to replace the OpenCV VideoStream API with something more performant, like ffmpeg, since ffmpeg is capable of outputting raw video into stdout.

The CamGear API should not need to change from the developer’s standpoint.

Your Environment

VidGear version: 0.1.8
Branch: PyPI
Python version: 3.8.2
pip version: 19.2.3
Operating System and version: Win10Pro 1909

Any Other Important Information

Encode/Decode benchmarks for VidGear (encode is compressed, so ffmpeg):

> poetry run sp-benchmark
Results:
        Encode:
                1080p: 71.34738617215714
                900p: 94.22808977047293
                720p: 137.51681644444432
                480p: 388.3952044196786
                360p: 506.7212349134308
                240p: 1020.0560010744543
                144p: 1860.4607896260777
        Decode:
                1080p: 36.054442749368185
                900p: 44.78923780306475
                720p: 55.349642074620796
                480p: 76.08067848749076
                360p: 81.93545752827764
                240p: 90.02970867849261
                144p: 99.95882945711747

ffmpeg decode benchmark (4k60):

> ffmpeg -i .\streampipe\benchmark\resources\bbb_sunflower_2160p_60fps_normal.mp4 -f null -
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '.\streampipe\benchmark\resources\bbb_sunflower_2160p_60fps_normal.mp4':
  Duration: 00:10:34.53, start: 0.000000, bitrate: 8487 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 3840x2160 [SAR 1:1 DAR 16:9], 8002 kb/s, 60 fps, 60 tbr, 60k tbn, 120 tbc (default)
    Stream #0:1(und): Audio: mp3 (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 160 kb/s (default)
    Stream #0:2(und): Audio: ac3 (ac-3 / 0x332D6361), 48000 Hz, 5.1(side), fltp, 320 kb/s (default)
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
  Stream #0:2 -> #0:1 (ac3 (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
    Stream #0:0(und): Video: wrapped_avframe, yuv420p(progressive), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 60 fps, 60 tbn, 60 tbc (default)
    Stream #0:1(und): Audio: pcm_s16le, 48000 Hz, 5.1(side), s16, 4608 kb/s (default)
frame=38072 fps=152 q=-0.0 Lsize=N/A time=00:10:34.56 bitrate=N/A speed=2.53x
video:19928kB audio:356706kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

About this issue

Original URL
State: open
Created 4 years ago
Comments: 31 (12 by maintainers)

Most upvoted comments

The idea would be to replace the OpenCV VideoStream API with something more performant, like ffmpeg, since ffmpeg is capable of outputting raw video into stdout.

@golyalpha I Don’t think that replacing OpenCV with FFmpeg is a good idea. Since the only leverage of FFmpeg over OpenCV is performance, While OpenCV is more advantageous as:

OpenCV seamlessly support multiple backends, which include powerful GStreamer, libav and FFmpeg itself too.
OpenCV is available on pretty much every Linux distribution, while FFmpeg might not be (for legal reasons).
OpenCV itself is available under flexible 3-clause BSD license while FFmpeg you have to make sure that no GPL components are enabled (some notable examples are x264 (H264 encoder) and libac3 (Dolby AC3 audio codec)). See https://www.ffmpeg.org/legal.html for details.
Plus FFmpeg is still buggy to be completely adopted reliably in my experience.

This is some of the reason why it’s not a good idea. Being said that, we can still implement another Videocapture gear which works purely on FFmpeg. Thanks for this idea anyways.

abhiTronix on Jul 19, 2020

🎉 A new library is here: https://github.com/abhiTronix/deffcode

👍🏽 Any suggestions are most welcomed.

abhiTronix on Mar 7, 2022

@AbdulrahmanSoliman1 It will happen eventually but I need to finish releasing Vidgear v0.2.6 and then I can work on anything else. Actually I’m doing this in my free time which is very limited right now, therefore once I get the appropriate amount of time, I’ll work on it.

abhiTronix on Jul 5, 2022

webcam code example

Unfortunately, Webcams are yet to be supported by deffcode. You can try CUDA backend for OpenCV Videocapture Decoding to speed up things while decoding. FYI Videogear or vidgear in general cannot accelerate beyond the speed of OpenCV itself. It can offload process to a different threads by multi-threading, but if the producer thread is already slow, it cannot do anything.

abhiTronix on May 9, 2022

@golyalpha That’s good (reasonably close i guess?), also thanks for the tip on ffmpeg backend for opencv. Also depending on CPU you might be limiting your decoding with encoding taking up the CPU.

@abhiTronix I wouldn’t mind a PyAV backend, but the reason i like VidGears WriteGear is that i can basically just supply my own arguments i use with ffmpeg for other applications and i’m good to go. No need to find out what i need to call to make sure it works as i want. So i hope it that won’t get completely replaced. Not to mention this brings hardware support given the user has a ffmpeg build with support.

Thomasedv on Jul 20, 2020

Alright, it seems that OpenCV is the part slowing down the decode process, because I just ran the same benchmark with ffmpeg as OpenCV backend, and the results are pretty much within the margin of error compared to the first one. Implementing a new gear specifically for ffmpeg decoding does seem to be a direction we could both agree on.

golyalpha on Jul 19, 2020