frigate: [Support]: ffmpeg crashing

Describe the problem you are having

I’ve just upgraded one of my cameras to a 4k Hikvison model however it appears that ffmpeg is crashing out after a short period as can be seen in the logs below. This never recovers and can only be solved by restarting Frigate.

Version

0.9.4-26AE608

Frigate config file

Drive:
    mqtt:
      timestamp: False
      bounding_box: False
      crop: True
      quality: 100
      height: 600
    ffmpeg:
      hwaccel_args:
        - -hwaccel
        - qsv
        - -qsv_device
        - /dev/dri/renderD128
        - -c:v
        - h264_qsv
      input_args: -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1
      output_args:
        detect: -vf hwdownload,format=nv12 -f rawvideo -pix_fmt yuvj420p
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy
      inputs:
        - path: rtsp://***:****@10.0.0.110:554/Streaming/Channels/101/
          roles:
            - detect
            - clips
    objects:
      track:
        - person
        - car
        - cat
      filters:
        car:
          min_score: 0.2
          threshold: 0.5
          mask:
            - 2926,621,3016,867,3840,2160,3840,0,0,0,0,2160,3840,2160,2991,839,2012,649,2311,490,2308,265,2944,343
    motion:
      mask:
        - 2095,173,2938,323,3071,378,3840,858,3840,0,1701,0,0,0,0,441,1114,339,1118,217
    rtmp:
      enabled: true
    snapshots:
      required_zones:
        - entrance
        - everywhere
      enabled: true
      timestamp: true
      bounding_box: true
      crop: true
      quality: 100
      height: 600
      retain:
        default: 10
    detect:
      width: 3840
      height: 2160
    record:
      enabled: True
      retain_days: 0
      events:
        required_zones:
          - entrance
          - everywhere
        retain:
          default: 10
    zones:
      entrance:
        coordinates: 1992,642,2997,835,2941,602,2953,284,2355,209,2323,437
        objects:
          - car
      everywhere:
        coordinates: 2359,331,2953,441,2949,0,3840,543,3840,2160,0,2160,0,0,1040,0,1055,331,1158,299,1142,173,2339,209
        objects:
          - person
          - cat

Relevant log output

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[2022-01-27 11:33:33] frigate.app INFO : Starting Frigate (0.9.4-26ae608)
Starting migrations
[2022-01-27 11:33:33] peewee_migrate INFO : Starting migrations
There is nothing to migrate
[2022-01-27 11:33:33] peewee_migrate INFO : There is nothing to migrate
[2022-01-27 11:33:33] frigate.mqtt INFO : MQTT connected
[2022-01-27 11:33:33] detector.coral_pci INFO : Starting detection process: 233
[2022-01-27 11:33:33] frigate.app INFO : Output process started: 235
[2022-01-27 11:33:33] ws4py INFO : Using epoll
[2022-01-27 11:33:33] frigate.edgetpu INFO : Attempting to load TPU as pci
[2022-01-27 11:33:33] frigate.app INFO : Camera processor started for Drive: 239
[2022-01-27 11:33:33] frigate.edgetpu INFO : TPU found
[2022-01-27 11:33:33] frigate.app INFO : Camera processor started for Garden: 245
[2022-01-27 11:33:33] frigate.app INFO : Capture process started for Drive: 253
[2022-01-27 11:33:33] frigate.app INFO : Capture process started for Garden: 254
[2022-01-27 11:33:33] ws4py INFO : Using epoll
[2022-01-27 11:33:43] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:55456]
[2022-01-27 11:34:43] ws4py INFO : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:55456]
[2022-01-27 11:40:56] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:40:56] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:40:56] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:40:56] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:40:56] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:40:56] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:40:56] frigate.video INFO : Drive: ffmpeg process is not running. exiting capture thread...
[2022-01-27 11:41:03] watchdog.Drive ERROR : FFMPEG process crashed unexpectedly for Drive.

[2022-01-27 11:41:03] watchdog.Drive ERROR : The following ffmpeg logs include the last 100 lines prior to exit.

[2022-01-27 11:41:03] watchdog.Drive ERROR : You may have invalid args defined for this camera.

[2022-01-27 11:41:03] ffmpeg.Drive.detect ERROR : [swscaler @ 0x55f3bc43f400] deprecated pixel format used, make sure you did set range correctly

[2022-01-27 11:41:03] ffmpeg.Drive.detect ERROR : av_interleaved_write_frame(): Connection reset by peer

[2022-01-27 11:41:03] ffmpeg.Drive.detect ERROR : [flv @ 0x55f3bc088a80] Failed to update header with correct duration.

[2022-01-27 11:41:03] ffmpeg.Drive.detect ERROR : [flv @ 0x55f3bc088a80] Failed to update header with correct filesize.

[2022-01-27 11:41:03] ffmpeg.Drive.detect ERROR : Error writing trailer of rtmp://127.0.0.1/live/Drive: Connection reset by peer

[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg process is not running. exiting capture thread...
[2022-01-27 11:41:23] watchdog.Drive ERROR : FFMPEG process crashed unexpectedly for Drive.

[2022-01-27 11:41:23] watchdog.Drive ERROR : The following ffmpeg logs include the last 100 lines prior to exit.

[2022-01-27 11:41:23] watchdog.Drive ERROR : You may have invalid args defined for this camera.

[2022-01-27 11:41:23] ffmpeg.Drive.detect ERROR : [rtsp @ 0x55e04e9fef00] DTS discontinuity in stream 1: packet 3 with DTS 72468809565588, packet 4 with DTS 72468809572656

[2022-01-27 11:41:23] ffmpeg.Drive.detect ERROR : [segment @ 0x55e04ea25200] Non-monotonous DTS in output stream 0:1; previous: 14822, current: 14820; changing to 14823. This may result in incorrect timestamps in the output file.

[2022-01-27 11:41:23] ffmpeg.Drive.detect ERROR : [swscaler @ 0x55e04f295440] deprecated pixel format used, make sure you did set range correctly

[2022-01-27 11:41:23] ffmpeg.Drive.detect ERROR : av_interleaved_write_frame(): Connection reset by peer

[2022-01-27 11:41:23] ffmpeg.Drive.detect ERROR : [flv @ 0x55e04ea21cc0] Failed to update header with correct duration.

[2022-01-27 11:41:23] ffmpeg.Drive.detect ERROR : [flv @ 0x55e04ea21cc0] Failed to update header with correct filesize.

[2022-01-27 11:41:23] ffmpeg.Drive.detect ERROR : Error writing trailer of rtmp://127.0.0.1/live/Drive: Connection reset by peer

[2022-01-27 12:11:18] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:34024]
[2022-01-27 12:12:04] ws4py INFO : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:34024]
[2022-01-27 12:12:16] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:34984]
[2022-01-27 12:13:43] ws4py INFO : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:34984]
[2022-01-27 12:16:40] ws4py INFO : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:58516]
[2022-01-27 12:16:42] ws4py INFO : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:58516]
[2022-01-27 13:15:28] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:46596]
[2022-01-27 13:16:43] ws4py INFO : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:46596]
Fatal Python error: Bus error


Current thread 0x0000148991f31700 (most recent call first):
File "/opt/frigate/frigate/video.py", line 150 in capture_frames
File "/opt/frigate/frigate/video.py", line 290 in run
File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x0000148994534700 (most recent call first):
File "/usr/lib/python3.8/threading.py", line 302 in wait
File "/usr/lib/python3.8/multiprocessing/queues.py", line 227 in _feed
File "/usr/lib/python3.8/threading.py", line 870 in run
File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x0000148991d30700 (most recent call first):
File "/usr/lib/python3.8/threading.py", line 302 in wait
File "/usr/lib/python3.8/multiprocessing/queues.py", line 227 in _feed
File "/usr/lib/python3.8/threading.py", line 870 in run
File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x0000148994132700 (most recent call first):
File "/usr/lib/python3.8/multiprocessing/synchronize.py", line 261 in wait
File "/usr/lib/python3.8/multiprocessing/synchronize.py", line 349 in wait
File "/opt/frigate/frigate/video.py", line 215 in run
File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x0000148994333700 (most recent call first):
File "/opt/frigate/frigate/log.py", line 65 in run
File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x00001489a5550740 (most recent call first):
File "/usr/lib/python3.8/threading.py", line 1027 in _wait_for_tstate_lock
File "/usr/lib/python3.8/threading.py", line 1011 in join
File "/opt/frigate/frigate/video.py", line 321 in capture_camera
File "/usr/lib/python3.8/multiprocessing/process.py", line 108 in run
File "/usr/lib/python3.8/multiprocessing/process.py", line 315 in _bootstrap
File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 75 in _launch
File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 19 in __init__
File "/usr/lib/python3.8/multiprocessing/context.py", line 277 in _Popen
File "/usr/lib/python3.8/multiprocessing/context.py", line 224 in _Popen
File "/usr/lib/python3.8/multiprocessing/process.py", line 121 in start
File "/opt/frigate/frigate/app.py", line 255 in start_camera_capture_processes
File "/opt/frigate/frigate/app.py", line 330 in start
File "/opt/frigate/frigate/__main__.py", line 17 in <module>
File "/usr/lib/python3.8/runpy.py", line 87 in _run_code
File "/usr/lib/python3.8/runpy.py", line 194 in _run_module_as_main
[2022-01-27 17:24:58] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:58926]

FFprobe output from your camera

ffprobe version 4.3.1 Copyright (c) 2007-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libfreetype --enable-libvidstab --enable-libmfx --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-vaapi --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Input #0, rtsp, from 'rtsp://***:****@10.0.0.110:554/Streaming/Channels/101/':
  Metadata:
    title           : Media Presentation
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: h264, yuvj420p(pc, progressive), 3840x2160 [SAR 1:1 DAR 16:9], 20 fps, 25 tbr, 90k tbn, 40 tbc
    Stream #0:1: Audio: aac, 44100 Hz, mono, fltp

Frigate stats

{"Drive":{"camera_fps":5.1,"capture_pid":253,"detection_fps":0.5,"pid":239,"process_fps":5.0,"skipped_fps":0.0},"Garden":{"camera_fps":5.1,"capture_pid":254,"detection_fps":0.0,"pid":245,"process_fps":5.1,"skipped_fps":0.0},"detection_fps":0.5,"detectors":{"coral_pci":{"detection_start":0.0,"inference_speed":9.79,"pid":233}},"service":{"storage":{"/dev/shm":{"free":40.5,"mount_type":"tmpfs","total":67.1,"used":26.6},"/media/frigate/clips":{"free":396711.5,"mount_type":"fuse.shfs","total":480112.7,"used":79918.4},"/media/frigate/recordings":{"free":396711.5,"mount_type":"fuse.shfs","total":480112.7,"used":79918.4},"/tmp/cache":{"free":7256.0,"mount_type":"rootfs","total":8243.1,"used":987.1}},"uptime":21495,"version":"0.9.4-26ae608"}}

Operating system

UNRAID

Install method

Docker CLI

Coral version

M.2

Network connection

Wired

Camera make and model

Hikvision DS-2CD2387G2-LU

Any other information that may be helpful

No response