opencv: Unable to enable Cudacodec VideoReader

For the record, I got the cuda video_reader demo working on Windows 10 using the native NVidia video codec SDK (no FFMPEG trick as suggested by dapicard). There is no extra trip to transfer data from CPU to GPU for cuda decoding. The original video_reader.cpp also works without modification. Lol

To me the biggest problem was to find the correct versions of a bunch of softwares. I list my settings here. I hope it can help others too:

OS: Window 10 CUDA: 10.0 NVidia-driver: 411.31 (come with the CUDA installation) OpenCV: 4.0.1 Video Codec SDK: Video_Codec_SDK_8.2.16 Visual studio: 2017 community CMake: 3.15.3

What didn’t work: OpenCV 3.4.1 with Video_Codec_SDK 8.0, which used dynlink_nvcuvid.h and dynlink_cuviddec.h. That was a big mess.

A few notes:

• Copy nvcuvid.h and cuviddec.h from Video_Codec_SDK_8.2.16/Samples/NvCodec/NvDecoder/ to NVIDIA GPU Computing Toolkit/CUDA/v10.0/include

• Set WITH_NVCUVID=ON when you build OpenCV from source. Importantly, make sure NVCUVID is included in CMake’s NVIDIA CUDA output. Otherwise the “opencv_cudacodec” module can still compile but will miss some implementation. This output works for me: NVIDIA CUDA: YES (ver 10.0, CUFFT CUBLAS NVCUVID FAST_MATH)

• In order to have NVCUVID included, I need to set this: CUDA_nvcuvid_LIBRARY=PATH_TO_Video_Codec_SDK_8.2.16/Samples/NvCodec/Lib/x64/nvcuvid.lib

• In my case, the original video_camera.cpp (OpenCV 4.0.1) works. GPU decode is significantly faster (a 2070 mobile card is 4x faster than a i7-8750H mobile CPU). You do need to set up the necessary paths in Visual Studio to make the code compile. The most important one is to include the opencv_cudacodec401d.lib as additional library dependency. It has the implementation of the CUDA decoder.

Happy decoding!

I faced the same situation, and this is why I patched the ffmpeg backend to allow to force the codec to use. See https://github.com/opencv/opencv/issues/11480 Compile ffmpeg from sources with CUDA support and compile OpenCV’s master using this ffmpeg backend. Then, before starting your OpenCV application, set the environment variable :

OPENCV_FFMPEG_CAPTURE_OPTIONS=“video_codec;h264_cuvid”

This way, the video will be decoded using the Nvidia hardware decoder. But it doesn’t provide GpuMat directly, so there is always an additional GPU/CPU memory trip…

cuda video decoder (nvcuvid), will be deprecated. no longer exists in cuda (latest 10.1)

but if you want to use it with opencv, install NVIDIA Video Decoder SDK https://developer.nvidia.com/nvidia-video-codec-sdk

after install it, you will have nvcuvid.lib in SDK/lib/(platform) directory. just pointing the opencv setting cuda_nvcuvid_library to that file. and make sure you check WITH NVCUVID

@khurramHazen I’m not suprise that the performance decreases this way : you are doing, at least, one more memory trip between the GPU memory and the CPU one. And because you’re not splitting reading and processing in, at least, two threads, this memory trip could not be done in parellel of other stuff.

When I see some performance increases using h264_cuvid, I was doing a lot of processing operations, some of them on CPU, others on GPU and I split these operations on up to 6 threads, pipelined by blocking and non-blocking queues. And I did it in C++, allowing me to do exactly what I want, when I want. Then, better performance mainly comes from the fact that using h264_cuvid allow to have more CPU cores available for processing stuff as Nvidia cards provides specific “cores” for video read/write. But if you don’t use all your CPU cores, you can use them to decode video ; it will be faster.

For a simple video read, I’m pretty sure that you better have to use CPU reader.

@dapicard how I can run your project to test video reader with GPU ??

You have to use at least OpenCV 3.4.2, then :

• Create a videoreader using the FFMPEG backend : cap->open(video_src, CAP_FFMPEG);
• set the envvar OPENCV_FFMPEG_CAPTURE_OPTIONS=“video_codec;h264_cuvid” This way, the videoreader will use the h264_cuvid codec.

If your source is an rtsp one, add the entry rtsp_transport;tcp ; so the envvar becomes : OPENCV_FFMPEG_CAPTURE_OPTIONS=“video_codec;h264_cuvid|rtsp_transport;tcp”

Too bad, I’m working a GPU version of a video processing algorithm and the last optimization I could do improve FPS is to read the images directly in the GPU to avoid CPU => GPU upload time (no I cannot use streams and CUDA data / process overlap to reduce this upload time). I’m using CUDA 9.1 and opencv master git branch (v3.4.1).

Maybe dropping OpenCV to use Nvidia video reader to optimize perfs

@keivanmoazami , can you tell me what libs & headers into cuda path and where is cuda path, is it only copy libs & headers? thanks!

copy nvcuvid.lib and nvencodeapi.lib from Video Codec SDK to lib folder of installed cuda path (default: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\lib\x64). also copy header files from video codec sdk (cuviddec.h, nvcuvid.h, nvEncodeAPI.h) into cuda headers path (default: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\include).

because cmake file search for libs only in cuda path. if you don’t find cuda path, you need to install cuda toolkit before build opencv with cuda.

Has there been an update to this? I don’t see why we still have the videoreader class in the codebase if not. Its been 3 years since NVIDIA made this change but we still don’t have a solution to reading to GpuMat directly?

It would be nice, indeed but I’m not sure that it would improve significantly the performance : in a project I lead currently, we are using the h264_cuvid in one thread to read the video, and we compute the Mat in some other threads, so the GPU/CPU trips are done alongside of the computing. Furthermore, as well as some algorithms are not implemented on CUDA (typically findContours), we always need to work on a Mat and a GpuMat. So such a functionnality will just save one CPU->GPU trip. On the other hand, rewrite and/or update of the cuda::videoreader to provide a configurable reader could be a bit complicated, so I’m not sure that the benefit justifies the cost. Using ffmpeg backend is a cheap and efficient way to use the Nvidia decoders.

(And if you want to use the nvenc, I did a patch that can’t be pulled on the opencv’s master for many reasons, but it does the job : https://github.com/dapicard/opencv/tree/3.4.2-nvenc ; you just have to define the FFMPEG encoder to use using the OPENCV_FFMPEG_ENCODER envvar ; once more, cheap and efficient)

@grzegorzk I did a patch some years ago to manually set the ffmpeg encoder. But it’s a fork and, I don’t maintain it and it only works with H264, as I remember. So, I can’t advise you to use it.

https://github.com/dapicard/opencv/commit/06de2fbe3a4f27cc61be71383f361b967c63cb78

Checking in to say it works very well. Thanks @dapicard!

@khurramHazen : as promised, the sample project allowing to process multiple videos in many threads, in C++ : https://github.com/dapicard/opencv-parallelize-example

@dapicard thanks a lot

@stevenyeun I’ve never tested it on Windows, but :

• install FFMpeg with CUDA hardware acceleration on Windows (I can’t help you for that)
• given yout input file/stream : “my_h264_file.mkv”, try to do some (useless) stuff using ffmpeg :
  • without cuvid : # ffmpeg -i my_h264_file.mkv -c:v libx264 -preset veryslow output.mp4
  • with cuvid : # ffmpeg -hwaccel cuvid -c:v h264_cuvid -i my_h264_file.mkv -c:v h264_nvenc output.mp4

If you’re not able to use ffmpeg with cuvid in the cli, it will not work with OpenCV.

More information on FFMPEG with cuvid : https://developer.nvidia.com/ffmpeg

@khurramHazen I think I will push a sample of my processing stream on github in the next days, but I can’t promise.

@stevenyeun What happens if you try to open your stream with ffmpeg directly ?

This way, the video will be decoded using the Nvidia hardware decoder. But it doesn’t provide GpuMat directly, so there is always an additional GPU/CPU memory trip…

Has there been an update to this? I don’t see why we still have the videoreader class in the codebase if not. Its been 3 years since NVIDIA made this change but we still don’t have a solution to reading to GpuMat directly?

I found a way to define the codec used by the FFMpeg backend :

export OPENCV_FFMPEG_CAPTURE_OPTIONS="video_codec|h264_cuvid"

More generally, it is possible to define theses parameters, using the syntax parameter_name|value;parameter_name2|value2

I didn’t find an equivalent for the VideoWriter FFMpeg backend.