frigate: Crashing every few hours, seemingly no error in logs

Describe the problem you are having

Frigate crashes every few hours, sometimes a few times in quick succession. Was experiencing similar on a pi 3 but have moved to a Synology NAS since 0.12.

Have tried playing with the shm value as well as increasing cache from 0.5 to 1 GB of memory but no luck.

I have a suspicion it might be the NAS running out of memory or the CPU being 100% utilised as occasionally it does also cause home assistant and everything else on the NAS to slow but not consistently. However when I check the NAS system monitor it’s within reasonable bounds (~75% memory usage and cpu usage).

I have a discord notification when it crashes and I check as soon as I see it but can’t ever see any errors in there, even with debug logs.

Version

0.12

Frigate config file

mqtt:
  host: 192.168.1.1
  user: mqtt
  password: pw

logger:
  default: debug

# Optional: birdseye configuration
birdseye:
  # Optional: Enable birdseye view (default: shown below)
  enabled: True
  # Optional: Mode of the view. Available options are: objects, motion, and continuous
  #   objects - cameras are included if they have had a tracked object within the last 30 seconds
  #   motion - cameras are included if motion was detected in the last 30 seconds
  #   continuous - all cameras are included always
  mode: continuous

database:
  path: /config/frigate.db

detect:
  fps: 5

detectors:
  coral:
    type: edgetpu
    device: usb

ffmpeg:
  hwaccel_args: preset-vaapi
  output_args:
    # Optional: output args for record streams (modified default to include audio)
    record: preset-record-generic-audio-aac

# NOTE: Can be overridden at the camera level
objects:
  track:
    - cat
    - dog
    - person

record:
  enabled: True
  retain:
    days: 7
    # Optional: Mode for retention. Available options are: all, motion, and active_objects
    #   all - save all recording segments regardless of activity
    #   motion - save all recordings segments with any detected motion
    #   active_objects - save all recording segments with active/moving objects
    # NOTE: this mode only applies when the days setting above is greater than 0
    mode: all
  events:
    retain:
      default: 7

snapshots:
  enabled: True
  timestamp: True
  retain:
    default: 7

timestamp_style:
  format: "%Y/%m/%d %H:%M:%S"

go2rtc:
  streams:
    doorbell:
      - "rtsp://admin:pw@192.168.3.10:554/cam/realmonitor?channel=1&subtype=0#backchannel=0"
      - "ffmpeg:doorbell#audio=opus" # <- copy of the stream which transcodes audio to the missing codec (usually will be opus)
    doorbell_sub:
      - "rtsp://admin:pw@192.168.3.10:554/cam/realmonitor?channel=1&subtype=1#backchannel=0"
      - "ffmpeg:doorbell_sub#audio=opus" # <- copy of the stream which transcodes audio to the missing codec (usually will be opus)
    front:
      - rtsp://admin:pw@192.168.3.11:554/cam/realmonitor?channel=1&subtype=0
    front_sub:
      - rtsp://admin:pw@192.168.3.11:554/cam/realmonitor?channel=1&subtype=1
    side:
      - rtsp://admin:pw@192.168.3.12:554/cam/realmonitor?channel=1&subtype=0
    side_sub:
      - rtsp://admin:pw@192.168.3.12:554/cam/realmonitor?channel=1&subtype=1
    indoor_hall_sub:
      - rtsp://admin:pw@192.168.3.15:554/12
    indoor_utility_sub:
      - rtsp://admin:pw@192.168.3.14:554/12
  webrtc:
    candidates:
      - 192.168.1.1:8555
      - stun:8555
  
cameras:
  doorbell:
    # SHM Size 5.6MB
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        # main stream
        - path: rtsp://127.0.0.1:8554/doorbell
          input_args: preset-rtsp-restream
          roles:
            - record
        # sub stream
        - path: rtsp://127.0.0.1:8554/doorbell_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    detect:
      width: 720
      height: 576
    live:
      stream_name: doorbell_sub
  front:
    # SHM Size 4.61MB
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        # main stream
        - path: rtsp://127.0.0.1:8554/front
          input_args: preset-rtsp-restream
          roles:
            - record
        # sub stream
        - path: rtsp://127.0.0.1:8554/front_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    detect:
      width: 480
      height: 704
    live:
      stream_name: front_sub
    zones:
      driveway:
        coordinates: 480,704,480,398,391,364,323,356,254,432,144,579,126,704
  side:
    # SHM Size 4.61MB
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        # main stream
        - path: rtsp://127.0.0.1:8554/side
          input_args: preset-rtsp-restream
          roles:
            - record
        # sub stream
        - path: rtsp://127.0.0.1:8554/side_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    detect:
      width: 704
      height: 480
    live:
      stream_name: side_sub
  indoor_hall:
    # SHM Size 3.16MB
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        # sub stream
        - path: rtsp://127.0.0.1:8554/indoor_hall_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
            - record
    detect:
      width: 640
      height: 352
    live:
      stream_name: indoor_hall_sub
    record:
      retain:
        mode: motion
  indoor_utility:
    # SHM Size 3.16MB
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        # sub stream
        - path: rtsp://127.0.0.1:8554/indoor_utility_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
            - record
    detect:
      width: 640
      height: 352
    live:
      stream_name: indoor_utility_sub
    record:
      retain:
        mode: motion

Relevant log output

Will add as separate comment. GitHub is complaining body length is too long.

FFprobe output from your camera

doorbell:

  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
[rtsp @ 0x5596fd64b140] method DESCRIBE failed: 404 Not Found
rtsp://admin:pw@192.168.3.10:554/cam/realmonitor?channel=1: Server returned 404 Not Found

front:

libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
[rtsp @ 0x55f848de1140] method DESCRIBE failed: 404 Not Found
rtsp://admin:2cQvMbgZLDHibGGs266u@192.168.3.11:554/cam/realmonitor?channel=1: Server returned 404 Not Found

side:

libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
[rtsp @ 0x55b347828140] method DESCRIBE failed: 404 Not Found
rtsp://admin:pw@192.168.3.12:554/cam/realmonitor?channel=1: Server returned 404 Not Found

indoor_hall

libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100

indoor_utility:

libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100

Frigate stats

{"cpu_usages":{"%Cpu(s):":{"cpu":"id,","mem":"13.5"},"1":{"cpu":"0.0","mem":"0.0"},"109":{"cpu":"0.0","mem":"0.0"},"112":{"cpu":"11.0","mem":"2.5"},"121":{"cpu":"0.0","mem":"0.0"},"136":{"cpu":"0.0","mem":"0.0"},"137":{"cpu":"0.0","mem":"0.0"},"138":{"cpu":"0.0","mem":"0.1"},"139":{"cpu":"0.0","mem":"0.0"},"17":{"cpu":"0.0","mem":"0.0"},"19":{"cpu":"0.0","mem":"0.0"},"26":{"cpu":"0.0","mem":"0.0"},"27":{"cpu":"0.0","mem":"0.0"},"2796":{"cpu":"0.0","mem":"0.0"},"28":{"cpu":"0.0","mem":"0.0"},"2834":{"cpu":"0.0","mem":"0.1"},"2838":{"cpu":"0.3","mem":"0.1"},"2840":{"cpu":"0.0","mem":"0.0"},"286":{"cpu":"1.0","mem":"0.2"},"29":{"cpu":"0.0","mem":"0.0"},"292":{"cpu":"0.7","mem":"0.0"},"293":{"cpu":"3.3","mem":"0.3"},"295":{"cpu":"1.7","mem":"0.2"},"297":{"cpu":"0.0","mem":"0.0"},"299":{"cpu":"0.0","mem":"0.0"},"30":{"cpu":"0.0","mem":"0.0"},"300":{"cpu":"0.7","mem":"0.4"},"301":{"cpu":"0.0","mem":"0.0"},"302":{"cpu":"5.3","mem":"0.4"},"303":{"cpu":"0.3","mem":"0.3"},"304":{"cpu":"0.0","mem":"0.0"},"305":{"cpu":"1.0","mem":"0.3"},"306":{"cpu":"0.0","mem":"0.0"},"307":{"cpu":"0.0","mem":"0.0"},"31":{"cpu":"0.0","mem":"0.0"},"313":{"cpu":"1.3","mem":"0.4"},"319":{"cpu":"1.0","mem":"0.3"},"32":{"cpu":"0.0","mem":"0.0"},"320":{"cpu":"1.0","mem":"0.3"},"322":{"cpu":"0.7","mem":"0.2"},"325":{"cpu":"1.0","mem":"0.2"},"326":{"cpu":"13.6","mem":"0.4"},"328":{"cpu":"1.0","mem":"0.2"},"33":{"cpu":"0.0","mem":"0.0"},"338":{"cpu":"11.6","mem":"0.4"},"339":{"cpu":"1.3","mem":"0.1"},"34":{"cpu":"0.0","mem":"0.0"},"340":{"cpu":"14.0","mem":"0.4"},"347":{"cpu":"1.7","mem":"0.1"},"348":{"cpu":"1.3","mem":"0.2"},"349":{"cpu":"8.3","mem":"0.4"},"43":{"cpu":"0.0","mem":"0.0"},"44":{"cpu":"0.0","mem":"0.0"},"696":{"cpu":"9.3","mem":"0.4"},"80":{"cpu":"0.0","mem":"0.0"},"82":{"cpu":"0.3","mem":"0.0"},"86":{"cpu":"0.0","mem":"0.0"},"97":{"cpu":"14.3","mem":"0.3"},"MiB":{"cpu":"977.4","mem":"avail"},"PID":{"cpu":"%CPU","mem":"%MEM"},"Tasks:":{"cpu":"stopped,","mem":"0"},"top":{"cpu":"users,","mem":"load"}},"detection_fps":5.7,"detectors":{"coral":{"detection_start":0.0,"inference_speed":11.24,"pid":293}},"doorbell":{"camera_fps":5.2,"capture_pid":319,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":326,"pid":300,"process_fps":5.2,"skipped_fps":0.0},"front":{"camera_fps":5.1,"capture_pid":320,"detection_enabled":1,"detection_fps":5.7,"ffmpeg_pid":340,"pid":302,"process_fps":5.1,"skipped_fps":0.0},"gpu_usages":{"error-gpu":{"gpu":-1,"mem":-1}},"indoor_hall":{"camera_fps":5.1,"capture_pid":325,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":696,"pid":305,"process_fps":5.1,"skipped_fps":0.0},"indoor_utility":{"camera_fps":5.0,"capture_pid":328,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":349,"pid":313,"process_fps":5.0,"skipped_fps":0.0},"service":{"last_updated":1681761265,"latest_version":"0.12.0","storage":{"/dev/shm":{"free":308.8,"mount_type":"tmpfs","total":314.6,"used":5.8},"/media/frigate/clips":{"free":1618984.8,"mount_type":"btrfs","total":2875829.0,"used":1256844.2},"/media/frigate/recordings":{"free":1618984.8,"mount_type":"btrfs","total":2875829.0,"used":1256844.2},"/tmp/cache":{"free":486.5,"mount_type":"tmpfs","total":500.0,"used":13.5}},"temperatures":{},"uptime":1836,"version":"0.12.0-da3e197"},"side":{"camera_fps":5.1,"capture_pid":322,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":338,"pid":303,"process_fps":5.1,"skipped_fps":0.0}}

Operating system

Other Linux (Synology NAS)

Install method

Docker Compose

Coral version

USB

Network connection

Wired

Camera make and model

Amcrest AD410, 2x IP5M-T1179EW-28MM, ieGeek 2x1080p

Any other information that may be helpful

Docker compose:

version: "3"
services:
  frigate:
    image: ghcr.io/blakeblackshear/frigate:stable
    container_name: frigate
    devices:
      - /dev/dri:/dev/dri # hwaccel
    environment:
      FRIGATE_RTSP_PASSWORD: "pw"
      TZ: Europe/London
    network_mode: bridge
    ports:
      - 5000:5000
      - 1984:1984 # go2rtc
      - 8554:8554 # go2rtc
      - 8555:8555 # go2rtc
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    shm_size: "300MB" # update for your cameras based on calculation above
    volumes:
      - /dev/bus/usb:/dev/bus/usb # usb coral
      - /etc/localtime:/etc/localtime:ro
      - /volume1/docker/frigate/config.yml:/config/config.yml
      - /volume1/CCTV:/media/frigate
      - type: tmpfs # Optional: 0.5GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 500000000