vidgear: Small number of frames causes no output.
Description
I an trying to make a video from a low number of frames. However, the ffmpeg process is terminated before the writing is finished. I also add audio from a clip that has the same duration as the video made. It’s also possible this causes the last few frames of longer clips to go missing as well, but isn’t that noticable unless you pay attention at the end, expect for situation where chapters won’t show due to incomplete file closes. (Which happens with ffmpeg, copying such files with ffmpeg again will reintroduce the chapters in a video. Pardon the sidestep, but i have had that problem with WriteGear as well, so i strongly suspect it the same cause.)
(Note: In the example below, the clip doesn’t have audio but the result is still the same.)
Acknowledgment
- I have searched the issues for my issue and found nothing related or helpful.
- I have read the Documentation.
- I have read the Issue Guidelines. (Outdate link i think)
Environment
- VidGear version: 0.1.9-dev
- Branch: Development
- Python version: 3.6
- pip version: Not relevant
- Operating System and version: Windows 10
Expected Behavior
Expected video
Actual Behavior
0 byte file is made.
Possible Fix
The ffmpeg process is terminated when calling close() with the -i flag is supplied as a parameter, which i assume kills ffmpeg before it has finished processing the frames. There are many situations where slower encoders are picked where there closing might happen before a frame write happens.
Relevant line of code: https://github.com/abhiTronix/vidgear/blob/b9d11098045e864afcceaced18973a0080c756ce/vidgear/gears/writegear.py#L534-L535
Steps to reproduce
Debug setup In folder: vid.webm # Video where audio is copied from (14 Frames) frames # Folder with frames for vidgear (both video and frames can be uploaded)
Code:
from vidgear.gears import WriteGear
from PIL import Image
import os
import numpy as np
import vidgear
print(vidgear.version.__version__)
source = 'vid.webm'
output_params = {"-input_framerate": 29.97*2, # Double the framerate of the video, double frames in the frames folder.
'-i': source,
'-clones': ['-map', '0:v:0', '-map', '1:a?', '-map', '1:s?']}
writer = WriteGear(output_filename='test_out.mkv', logging=True, **output_params)
for image in os.listdir('frames'):
img = Image.open(os.path.join('frames', image))
writer.write(np.asarray(img))
writer.close()
Default ffmpeg is used, and let Writegear handle output format and everything
Logging:
0.1.9-dev
WriteGear :: DEBUG :: Compression Mode is enabled therefore checking for valid FFmpeg executables.
WriteGear :: DEBUG :: Output_params Dict: {'-input_framerate': '59.94', '-i': 'vid.webm', '-clones': ['-map', '0:v:0', '-map', '1:a?', '-map', '1:s?']}
Helper :: DEBUG :: FFmpeg Windows Download Path: C:\Users\thoma\AppData\Local\Temp
Helper :: DEBUG :: Final FFmpeg Path: C:\Users\thoma\AppData\Local\Temp\ffmpeg-latest-win64-static/bin/ffmpeg.exe
Helper :: DEBUG :: FFmpeg validity Test Passed!
Helper :: DEBUG :: Found valid FFmpeg Version: `b'git-2020-07-16-d11cc74'` installed on this system
WriteGear :: DEBUG :: Found valid FFmpeg executables: `C:\Users\thoma\AppData\Local\Temp\ffmpeg-latest-win64-static/bin/ffmpeg.exe`.
WriteGear :: DEBUG :: Compression Mode is configured properly!
WriteGear :: DEBUG :: InputFrame => Height:372 Width:390 Channels:3
WriteGear :: DEBUG :: Setting Input FrameRate = 59.94
WriteGear :: DEBUG :: Executing FFmpeg command: `C:\Users\thoma\AppData\Local\Temp\ffmpeg-latest-win64-static/bin/ffmpeg.exe -y -f rawvideo -vcodec rawvideo -s 390x372 -pix_fmt bgr24 -framerate 59.94 -i - -i vid.webm -map 0:v:0 -map 1:a? -map 1:s? -vcodec libx264 -crf 18 -preset fast D:\AI_RRIN\RRIN\test_softmax\test_out.mkv`
ffmpeg version git-2020-07-16-d11cc74 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 9.3.1 (GCC) 20200621
configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libsrt --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libgsm --disable-w32threads --enable-libmfx --enable-ffnvcodec --enable-cuda-llvm --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt --enable-amf
libavutil 56. 55.100 / 56. 55.100
libavcodec 58. 96.100 / 58. 96.100
libavformat 58. 48.100 / 58. 48.100
libavdevice 58. 11.101 / 58. 11.101
libavfilter 7. 87.100 / 7. 87.100
libswscale 5. 8.100 / 5. 8.100
libswresample 3. 8.100 / 3. 8.100
libpostproc 55. 8.100 / 55. 8.100
Input #0, rawvideo, from 'pipe:':
Duration: N/A, start: 0.000000, bitrate: 208706 kb/s
Stream #0:0: Video: rawvideo (BGR[24] / 0x18524742), bgr24, 390x372, 208706 kb/s, 59.94 tbr, 59.94 tbn, 59.94 tbc
Input #1, matroska,webm, from 'vid.webm':
Metadata:
title : xbox_loop_test3.1
PURL : https://www.youtube.com/watch?v=FrL8YFNvEWM
DATE : 20171024
COMPATIBLE_BRANDS: iso6avc1mp41
MAJOR_BRAND : dash
MINOR_VERSION : 0
ARTIST : Real Game Media
DESCRIPTION : Nice to see the original boot animation again after all these years. http://realgamemedia.com/original-xbox-backwards-compatibility-finally/ Link to US XBOX ...
COMMENT : Nice to see the original boot animation again after all these years. http://realgamemedia.com/original-xbox-backwards-compatibility-finally/ Link to US XBOX ...
ENCODER : Lavf58.45.100
Duration: 00:00:00.50, start: 0.033000, bitrate: 358 kb/s
Stream #1:0: Video: vp9 (Profile 0), yuv420p(tv), 410x412, SAR 1:1 DAR 205:206, 29.97 fps, 29.97 tbr, 1k tbn, 1k tbc (default)
Metadata:
HANDLER_NAME : VideoHandler
ENCODER : Lavc58.91.100 libvpx-vp9
DURATION : 00:00:00.500000000
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
[swscaler @ 00000168e82a5600] Warning: data is not aligned! This can lead to a speed loss
[libx264 @ 00000168e8244ac0] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 00000168e8244ac0] profile High 4:4:4 Predictive, level 3.0, 4:4:4, 8-bit
[libx264 @ 00000168e8244ac0] 264 - core 160 - H.264/MPEG-4 AVC codec - Copyleft 2003-2020 - http://www.videolan.org/x264.html - options: cabac=1 ref=2 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=6 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=4 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=1 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=30 rc=crf mbtree=1 crf=18.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, matroska, to 'D:\AI_RRIN\RRIN\test_softmax\test_out.mkv':
Metadata:
encoder : Lavf58.48.100
Stream #0:0: Video: h264 (libx264) (H264 / 0x34363248), yuv444p, 390x372, q=-1--1, 59.94 fps, 1k tbn, 59.94 tbc
Metadata:
encoder : Lavc58.96.100 libx264
Side data:
cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
WriteGear :: DEBUG :: Terminating WriteGear Processes.
Logging with the mentioned lines commented out: (which has working result)
0.1.9-dev
WriteGear :: DEBUG :: Compression Mode is enabled therefore checking for valid FFmpeg executables.
WriteGear :: DEBUG :: Output_params Dict: {'-input_framerate': '59.94', '-i': 'vid.webm', '-clones': ['-map', '0:v:0', '-map', '1:a?', '-map', '1:s?']}
Helper :: DEBUG :: FFmpeg Windows Download Path: C:\Users\thoma\AppData\Local\Temp
Helper :: DEBUG :: Final FFmpeg Path: C:\Users\thoma\AppData\Local\Temp\ffmpeg-latest-win64-static/bin/ffmpeg.exe
Helper :: DEBUG :: FFmpeg validity Test Passed!
Helper :: DEBUG :: Found valid FFmpeg Version: `b'git-2020-07-16-d11cc74'` installed on this system
WriteGear :: DEBUG :: Found valid FFmpeg executables: `C:\Users\thoma\AppData\Local\Temp\ffmpeg-latest-win64-static/bin/ffmpeg.exe`.
WriteGear :: DEBUG :: Compression Mode is configured properly!
WriteGear :: DEBUG :: InputFrame => Height:372 Width:390 Channels:3
WriteGear :: DEBUG :: Setting Input FrameRate = 59.94
WriteGear :: DEBUG :: Executing FFmpeg command: `C:\Users\thoma\AppData\Local\Temp\ffmpeg-latest-win64-static/bin/ffmpeg.exe -y -f rawvideo -vcodec rawvideo -s 390x372 -pix_fmt bgr24 -framerate 59.94 -i - -i vid.webm -map 0:v:0 -map 1:a? -map 1:s? -vcodec libx264 -crf 18 -preset fast D:\AI_RRIN\RRIN\test_softmax\test_out.mkv`
ffmpeg version git-2020-07-16-d11cc74 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 9.3.1 (GCC) 20200621
configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libsrt --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libgsm --disable-w32threads --enable-libmfx --enable-ffnvcodec --enable-cuda-llvm --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt --enable-amf
libavutil 56. 55.100 / 56. 55.100
libavcodec 58. 96.100 / 58. 96.100
libavformat 58. 48.100 / 58. 48.100
libavdevice 58. 11.101 / 58. 11.101
libavfilter 7. 87.100 / 7. 87.100
libswscale 5. 8.100 / 5. 8.100
libswresample 3. 8.100 / 3. 8.100
libpostproc 55. 8.100 / 55. 8.100
Input #0, rawvideo, from 'pipe:':
Duration: N/A, start: 0.000000, bitrate: 208706 kb/s
Stream #0:0: Video: rawvideo (BGR[24] / 0x18524742), bgr24, 390x372, 208706 kb/s, 59.94 tbr, 59.94 tbn, 59.94 tbc
Input #1, matroska,webm, from 'vid.webm':
Metadata:
title : xbox_loop_test3.1
PURL : https://www.youtube.com/watch?v=FrL8YFNvEWM
DATE : 20171024
COMPATIBLE_BRANDS: iso6avc1mp41
MAJOR_BRAND : dash
MINOR_VERSION : 0
ARTIST : Real Game Media
DESCRIPTION : Nice to see the original boot animation again after all these years. http://realgamemedia.com/original-xbox-backwards-compatibility-finally/ Link to US XBOX ...
COMMENT : Nice to see the original boot animation again after all these years. http://realgamemedia.com/original-xbox-backwards-compatibility-finally/ Link to US XBOX ...
ENCODER : Lavf58.45.100
Duration: 00:00:00.50, start: 0.033000, bitrate: 358 kb/s
Stream #1:0: Video: vp9 (Profile 0), yuv420p(tv), 410x412, SAR 1:1 DAR 205:206, 29.97 fps, 29.97 tbr, 1k tbn, 1k tbc (default)
Metadata:
HANDLER_NAME : VideoHandler
ENCODER : Lavc58.91.100 libvpx-vp9
DURATION : 00:00:00.500000000
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
[swscaler @ 0000019397bc5600] Warning: data is not aligned! This can lead to a speed loss
[libx264 @ 0000019397b64ac0] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0000019397b64ac0] profile High 4:4:4 Predictive, level 3.0, 4:4:4, 8-bit
[libx264 @ 0000019397b64ac0] 264 - core 160 - H.264/MPEG-4 AVC codec - Copyleft 2003-2020 - http://www.videolan.org/x264.html - options: cabac=1 ref=2 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=6 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=4 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=1 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=30 rc=crf mbtree=1 crf=18.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, matroska, to 'D:\AI_RRIN\RRIN\test_softmax\test_out.mkv':
Metadata:
encoder : Lavf58.48.100
Stream #0:0: Video: h264 (libx264) (H264 / 0x34363248), yuv444p, 390x372, q=-1--1, 59.94 fps, 1k tbn, 59.94 tbc
Metadata:
encoder : Lavc58.96.100 libx264
Side data:
cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
WriteGear :: DEBUG :: Terminating WriteGear Processes.
frame= 27 fps=0.0 q=-1.0 Lsize= 49kB time=00:00:00.40 bitrate=1000.2kbits/s speed=2.17x
video:48kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.693746%
[libx264 @ 0000019397b64ac0] frame I:2 Avg QP:21.34 size: 2958
[libx264 @ 0000019397b64ac0] frame P:9 Avg QP:21.42 size: 2278
[libx264 @ 0000019397b64ac0] frame B:16 Avg QP:20.98 size: 1388
[libx264 @ 0000019397b64ac0] consecutive B-frames: 11.1% 22.2% 22.2% 44.4%
[libx264 @ 0000019397b64ac0] mb I I16..4: 46.4% 52.8% 0.8%
[libx264 @ 0000019397b64ac0] mb P I16..4: 31.6% 33.1% 0.5% P16..4: 17.3% 4.9% 1.0% 0.0% 0.0% skip:11.7%
[libx264 @ 0000019397b64ac0] mb B I16..4: 10.1% 8.4% 0.3% B16..8: 28.5% 6.3% 0.2% direct:11.0% skip:35.1% L0:49.8% L1:45.6% BI: 4.6%
[libx264 @ 0000019397b64ac0] 8x8 transform intra:49.4% inter:71.8%
[libx264 @ 0000019397b64ac0] coded y,u,v intra: 22.7% 10.8% 14.3% inter: 7.9% 6.6% 7.4%
[libx264 @ 0000019397b64ac0] i16 v,h,dc,p: 21% 16% 6% 58%
[libx264 @ 0000019397b64ac0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 27% 20% 17% 5% 8% 7% 7% 6% 3%
[libx264 @ 0000019397b64ac0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 30% 18% 18% 3% 8% 8% 8% 5% 2%
[libx264 @ 0000019397b64ac0] Weighted P-Frames: Y:44.4% UV:22.2%
[libx264 @ 0000019397b64ac0] ref P L0: 76.8% 23.2%
[libx264 @ 0000019397b64ac0] ref B L0: 79.4% 20.6%
[libx264 @ 0000019397b64ac0] ref B L1: 92.1% 7.9%
[libx264 @ 0000019397b64ac0] kb/s:863.56
Take note how there is stuff printed after calling close in this last log, but not in the first one. While i do know printing doesn’t follow proper ordering, i do still believe this to be accurate here. After close is called, the ffmpeg process is terminated and won’t continue so nothing is printed.
When run, there is created a 0 byte file, the typical ones created by ffmpeg before frames have been written to it. Commenting out the code mentioned above, makes a proper video file.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 21 (13 by maintainers)
Tested it quickly, and seems to be working.
Thanks again for the good work!
boolean needs to be changed to bool. Crashes otherwise since it’s not defined. There’s also an edge case if the user provides the termination flag, but not “-i” where that key is not removed. It’s not pretty, but the following code is a suggested fix:
Edit: Flipped a boolean.
I’d could have made pull requests for these things, it just that i have no experience when it comes to creating pull requests and such.
While i can say i have a pretty decent understanding of writegear.py, i can’t say as much for the rest of VidGear. Streaming isn’t my strong side either, but I’m happy to have a look at it, just not sure how much i can actually do.
Used testing branch as development is removed at time of writing. Pardon the delayed response.
The option probably works, but ffmpeg crashes due to the option not being deleted from the output parameters before being passed to it.
Also, as per the docs in the referenced commit, the default usage is letting the user provide a boolean, but this is not checked in the actual code, only the existence of the termination key. This may lead a user to incorrectly assume that passing the flag with the a False value will cause the default behavior, when it does not. To solve both issues, i propose to change the following lines
https://github.com/abhiTronix/vidgear/blob/d76c8be8df075b3d0feef2c6de1b40f7706c5ffc/vidgear/gears/writegear.py#L183-L186
to the following:
Edit: Removed some comment, reading old Stack Overflow for too old python versions is not always good.
@Thomasedv Give me few days, as I’m stacked up with work right now. I’ll notify you here when I’ll be pushing related commits.
That seems proper. I’d hold on implementing until one of us has tested the ffmpeg flag i mentioned , it might just do the job for us, and can be either manually or automatically added to the outout_params as well. And optionally be disabled just like termination. And I feel that if you can avoid termination altogether you’ll be handling stopping much better overall. I can do a test of that once I’m off work.
On the topic of time.sleep, I think it would be too arbitrary, in edge cases where encoding takes a long time, eg. AV1 encoding with vmaf tune, I think one frame can take upwards to 30 sec for example going by a post i saw earlier. Which may leave the user in the same situation.
It’s what I assumed. I honestly don’t know exactly if that is possible, the best suggestion I have is to add a keyword argument to the close method that let’s users turn of the termination. Eg. Have a
is_livestream=Truewhich would support backwards compatibility and let a user set to false to let it finish at its own pace.I would think ffmpeg has some potential options, I’m not at liberty to test for a while, but the
-shortestflag might address that bug, assuming ffmpeg can understand its over once the stdin closes. I only Google from mobile so we’ll see, I can test with a random audio clip later.vidgear_test.zip
Images and video is part of original xbox startup(minor flash warning in case of epilepsy), and the frames are the same video but interpolatated which is why the input has to be double the framerate of the video. This works well for anything else where frames are written. Frames extracted are processed by my original python script, but in this case I just wrote them as png directly instead of using WriteGear.
No audio in the original video, I didn’t add that when i made the short video. Shouldn’t matter for the case at hand, as the lack of audio shouldn’t prevent anything from being made. It’s my mistake however, but if you insist I can make a new example after work.
os.listdir() is sorted by filename on windows, but i added the sorted for good measure, same with the -r tag, i noticed ffmpeg could drop frames without it with my own settings. (because it assumed it was working realtime) But i just went barebones for the sake of the simple code.
And again, that i consistently have no output with .terminate() there, and correct video without it, seems pretty likely that the subprocess is killing it before it finished. On Windows it calls the win32 api which has the following documentation:
It’s a bit vague for me, but it sounds like it may prevent any IO from happening, not to mention stop ffmpeg from processing the last frames given to it, with any slow encoder it would take longer than the call to close after the last write call.