frigate: [Support]: Record stream doesn't recover after camera restart

Describe the problem you are having

I have 10 Amcrest/Dahua turret cameras running on Frigate (see config below).

If a camera is restarted, Frigate will recover the event and live stream, however recordings stop and never recover until I restart Frigate.

I can simulate this by just manually restarting the camera.

Version

0.10.1-83481af

Frigate config file

database:
  path: /db/frigate.db

mqtt:
  host: *************
  user: *************
  password: *************

detectors:
  coral1:
    type: edgetpu
    device: pci:0

record:
  enabled: True
  retain:
    days: 14
    mode: all
  events:
    retain:
      default: 14

detect:
  width: 1280
  height: 720

model:
  labelmap:
    15: animal
    16: animal
    17: animal
    18: animal
    19: animal
    20: animal
    21: animal
    22: animal
    23: animal
    24: animal

objects:
  track:
    - person
    - bicycle
    - car
    - motorcycle
    - animal

ffmpeg:
  hwaccel_args:
    - -hwaccel
    - vaapi
    - -hwaccel_device
    - /dev/dri/renderD128
    - -hwaccel_output_format
    - yuv420p
  input_args: -avoid_negative_ts make_zero -rtsp_transport tcp
  output_args:
    record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy

cameras:

  front_yard_ne:
    ffmpeg:
      inputs:
        - path: rtsp://admin::***************@@front_yard_ne.localcamera:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
        - path: rtsp://admin::***************@@front_yard_ne.localcamera:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - rtmp
        - path: rtsp://admin::***************@@front_yard_ne.localcamera:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect
    zones:
      front_yard_ne_driveway:
        coordinates: ..........

  front_door:
    ffmpeg:
      inputs:
        - path: rtsp://admin::***************@@front_door.localcamera:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
        - path: rtsp://admin::***************@@front_door.localcamera:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - rtmp
        - path: rtsp://admin::***************@@front_door.localcamera:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect
    objects:
      track:
        - person
        - animal
    zones:
      front_door_at_door:
        coordinates: ..........

  front_yard_drive:
    ffmpeg:
      inputs:
        - path: rtsp://admin::***************@@front_yard_drive.localcamera:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
        - path: rtsp://admin::***************@@front_yard_drive.localcamera:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - rtmp
        - path: rtsp://admin::***************@@front_yard_drive.localcamera:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect
    zones:
      front_yard_drive_driveway:
        coordinates: ..........

  garage:
    ffmpeg:
      inputs:
        - path: rtsp://admin::***************@@garage.localcamera:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
        - path: rtsp://admin::***************@@garage.localcamera:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - rtmp
        - path: rtsp://admin::***************@@garage.localcamera:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect
    objects:
      track:
        - person
        - animal

  front_yard_se:
    ffmpeg:
      inputs:
        - path: rtsp://admin::***************@@front_yard_se.localcamera:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
        - path: rtsp://admin::***************@@front_yard_se.localcamera:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - rtmp
        - path: rtsp://admin::***************@@front_yard_se.localcamera:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect

  side_yard_se:
    ffmpeg:
      inputs:
        - path: rtsp://admin::***************@@side_yard_se.localcamera:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
        - path: rtsp://admin::***************@@side_yard_se.localcamera:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - rtmp
        - path: rtsp://admin::***************@@side_yard_se.localcamera:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect
    objects:
      track:
        - person
        - animal

  back_yard_sw:
    ffmpeg:
      inputs:
        - path: rtsp://admin::***************@@back_yard_sw.localcamera:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
        - path: rtsp://admin::***************@@back_yard_sw.localcamera:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - rtmp
        - path: rtsp://admin::***************@@back_yard_sw.localcamera:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect
    objects:
      track:
        - person
        - animal
        - boat

  back_yard_nw:
    ffmpeg:
      inputs:
        - path: rtsp://admin::***************@@back_yard_nw.localcamera:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
        - path: rtsp://admin::***************@@back_yard_nw.localcamera:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - rtmp
        - path: rtsp://admin::***************@@back_yard_nw.localcamera:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect
    objects:
      track:
        - person
        - animal
        - boat

  side_yard_nw:
    ffmpeg:
      inputs:
        - path: rtsp://admin::***************@@side_yard_nw.localcamera:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
        - path: rtsp://admin:***************@side_yard_nw.localcamera:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - rtmp
        - path: rtsp://admin::***************@@side_yard_nw.localcamera:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect
    objects:
      track:
        - person
        - animal

  boathouse:
    ffmpeg:
      output_args:
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an
      inputs:
        - path: rtsp://admin::***************@@boathouse.localcamera:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
        - path: rtsp://admin::***************@@boathouse.localcamera:554/cam/realmonitor?channel=1&subtype=1
          roles:
            - rtmp
            - detect
    detect:
      width: 640
      height: 480
    live:
      height: 480
    objects:
      track:
        - person
        - animal
        - boat
    zones:
      boathouse_main_deck:
        coordinates: ..........

Relevant log output

[2022-04-18 21:22:18] frigate.app                    INFO    : Starting Frigate (0.10.1-83481af)
Starting migrations
[2022-04-18 21:22:18] peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
[2022-04-18 21:22:18] peewee_migrate                 INFO    : There is nothing to migrate
[2022-04-18 21:22:18] frigate.app                    INFO    : Output process started: 213
[2022-04-18 21:22:18] frigate.app                    INFO    : Camera processor started for front_yard_ne: 215
[2022-04-18 21:22:18] ws4py                          INFO    : Using epoll
[2022-04-18 21:22:18] frigate.app                    INFO    : Camera processor started for front_door: 218
[2022-04-18 21:22:18] frigate.app                    INFO    : Camera processor started for front_yard_drive: 221
[2022-04-18 21:22:18] frigate.app                    INFO    : Camera processor started for garage: 224
[2022-04-18 21:22:18] frigate.app                    INFO    : Camera processor started for front_yard_se: 225
[2022-04-18 21:22:18] frigate.app                    INFO    : Camera processor started for side_yard_se: 227
[2022-04-18 21:22:18] frigate.app                    INFO    : Camera processor started for back_yard_sw: 229
[2022-04-18 21:22:18] frigate.app                    INFO    : Camera processor started for back_yard_nw: 231
[2022-04-18 21:22:18] frigate.app                    INFO    : Camera processor started for side_yard_nw: 232
[2022-04-18 21:22:18] frigate.app                    INFO    : Camera processor started for boathouse: 234
[2022-04-18 21:22:18] frigate.app                    INFO    : Capture process started for front_yard_ne: 236
[2022-04-18 21:22:18] frigate.app                    INFO    : Capture process started for front_door: 238
[2022-04-18 21:22:19] frigate.app                    INFO    : Capture process started for front_yard_drive: 241
[2022-04-18 21:22:19] frigate.app                    INFO    : Capture process started for garage: 250
[2022-04-18 21:22:19] frigate.app                    INFO    : Capture process started for front_yard_se: 255
[2022-04-18 21:22:19] frigate.app                    INFO    : Capture process started for side_yard_se: 259
[2022-04-18 21:22:19] frigate.app                    INFO    : Capture process started for back_yard_sw: 268
[2022-04-18 21:22:19] frigate.app                    INFO    : Capture process started for back_yard_nw: 274
[2022-04-18 21:22:19] frigate.app                    INFO    : Capture process started for side_yard_nw: 285
[2022-04-18 21:22:19] frigate.app                    INFO    : Capture process started for boathouse: 297
[2022-04-18 21:22:19] ws4py                          INFO    : Using epoll
W driver/beagle/beagle_kernel_top_level_handler.cc:131] Could not set performance expectation : 92 (Inappropriate ioctl for device)
[2022-04-18 21:22:18] detector.coral1                INFO    : Starting detection process: 212
[2022-04-18 21:22:18] frigate.edgetpu                INFO    : Attempting to load TPU as pci:0
[2022-04-18 21:22:19] frigate.edgetpu                INFO    : TPU found
[2022-04-18 21:22:37] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:58174]



(I restarted camera front_yard_ne at this point)



[2022-04-18 21:23:59] watchdog.front_yard_ne         INFO    : No frames received from front_yard_ne in 20 seconds. Exiting ffmpeg...
[2022-04-18 21:23:59] watchdog.front_yard_ne         INFO    : Waiting for ffmpeg to exit gracefully...
[2022-04-18 21:24:29] watchdog.front_yard_ne         INFO    : FFmpeg didnt exit. Force killing...
[2022-04-18 21:24:29] frigate.video                  ERROR   : front_yard_ne: Unable to read frames from ffmpeg process.
[2022-04-18 21:24:29] frigate.video                  ERROR   : front_yard_ne: Unable to read frames from ffmpeg process.
[2022-04-18 21:24:29] frigate.video                  ERROR   : front_yard_ne: ffmpeg process is not running. exiting capture thread...
[2022-04-18 21:24:29] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:58174]
[2022-04-18 21:24:39] watchdog.front_yard_ne         ERROR   : Ffmpeg process crashed unexpectedly for front_yard_ne.
[2022-04-18 21:24:39] watchdog.front_yard_ne         ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.

FFprobe output from your camera

$ docker exec frigate ffprobe 'rtsp://admin:******************@front_yard_ne.localcamera:554/cam/realmonitor?channel=1&subtype=0'
ffprobe version 4.3.1 Copyright (c) 2007-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libfreetype --enable-libvidstab --enable-libmfx --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-vaapi --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
[rtsp @ 0x55ab6e563b80] max delay reached. need to consume packet
[rtsp @ 0x55ab6e563b80] RTP: missed 33 packets
[rtsp @ 0x55ab6e563b80] max delay reached. need to consume packet
[rtsp @ 0x55ab6e563b80] RTP: missed 33 packets
[rtsp @ 0x55ab6e563b80] max delay reached. need to consume packet
[rtsp @ 0x55ab6e563b80] RTP: missed 22 packets
[h264 @ 0x55ab6e569780] concealing 28713 DC, 28713 AC, 28713 MV errors in I frame
[rtsp @ 0x55ab6e563b80] max delay reached. need to consume packet
[rtsp @ 0x55ab6e563b80] RTP: missed 25 packets
[rtsp @ 0x55ab6e563b80] max delay reached. need to consume packet
[rtsp @ 0x55ab6e563b80] RTP: missed 17 packets
[h264 @ 0x55ab6e569780] error while decoding MB 63 54, bytestream -5
[h264 @ 0x55ab6e569780] concealing 19426 DC, 19426 AC, 19426 MV errors in P frame
[rtsp @ 0x55ab6e563b80] max delay reached. need to consume packet
[rtsp @ 0x55ab6e563b80] RTP: missed 14 packets
[rtsp @ 0x55ab6e563b80] max delay reached. need to consume packet
[rtsp @ 0x55ab6e563b80] RTP: missed 23 packets
[rtsp @ 0x55ab6e563b80] max delay reached. need to consume packet
[rtsp @ 0x55ab6e563b80] RTP: missed 23 packets
Input #0, rtsp, from 'rtsp://admin:******************@front_yard_ne.localcamera:554/cam/realmonitor?channel=1&subtype=0':
  Metadata:
    title           : Media Server
  Duration: N/A, start: 0.001000, bitrate: N/A
    Stream #0:0: Video: h264, yuvj420p(pc, bt709, progressive), 3840x2160 [SAR 7713:7712 DAR 857:482], 15 fps, 100 tbr, 90k tbn, 30 tbc
    Stream #0:1: Audio: aac, 16000 Hz, mono, fltp

Frigate stats

No response

Operating system

Debian

Install method

Docker Compose

Coral version

M.2

Network connection

Wired

Camera make and model

Amcrest IP8M-T2499EB-28MM

Any other information that may be helpful

Same issue seems to happen with all 10 of my cameras.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 22 (1 by maintainers)

Most upvoted comments

Frigate does restart streams when they fail. Pretty sure this is happening because the default input args were modified to remove the -timeout parameter.

blakeblackshear on Nov 1, 2022