frigate: [Support]: Unable to keep up with recording segments

Describe the problem you are having

I get the log message Unable to keep up with recording segments in cache for <camera>. Keeping the 5 most recent segments out of <x> and discarding the rest... in my log a LOT. I’ve enabled debug logging for recordings, but I don’t see any noticeable slowdown in file copies (generally 0.2s or less per file) out of the cache. Watching iotop the DISK write is mostly in the range of 50-100 K/s, with occasional spurts of 20-50 M/s. iftop indicates I’m doing ~205 Mbps of incoming bandwidth. My recording volume is a RAID-0 of 6x Western Digital Purple drives (though in an unusual configuration, so it’s highly likely only two spindles will be used at any given time). Even if it was a single drive, the sustained throughput is rated at 145MB/s, so I shouldn’t be coming anywhere near the actual storage throughput. Snapshots and database all go to the main OS volume, which is a 250 GB SSD. All hard drives are connected via SATA (4 of them via an external eSATA enclosure connected to a PCI Express eSATA card).

Analyzing my log file, of the last 14,502 segments that were copied, the slowest one took 0.84s, and on average they take 0.17s.

CPU usage normally sits at 50-60%.

I have 64GB of RAM. Usually about ~7GB is in use, and the rest is in buffers/cache.

It might be that my storage is too slow, but it sure seems like either Frigate is being too impatient with a large number of cameras, or something else is happening outside of the actual copy process that is being too slow.

Version

0.12.0-27A31E7

Frigate config file

detectors:
  coral:
    type: edgetpu
    device: usb

ffmpeg:
  hwaccel_args: preset-vaapi
  input_args: preset-rtsp-restream-low-latency
  output_args:
    record: preset-record-generic-audio-aac

ui:
  live_mode: webrtc
  use_experimental: false

birdseye:
  mode: objects
  width: 1920
  height: 1080
  restream: true

logger:
  logs:
    frigate.record: debug

go2rtc:
  <elided>

detect:
  width: 1280
  height: 720
  fps: 6
  max_disappeared: 18
  stationary:
    interval: 12

objects:
  track:
    - person
    - dog
    - cat

record:
  enabled: true
  retain:
    days: 14
    mode: all
  events:
    pre_capture: 2
    post_capture: 2

snapshots:
  enabled: true
  retain:
    default: 14

cameras:
  <33 cameras>

Relevant log output

See https://gist.github.com/ccutrer/224fc652c70661d91f56811a1cd88f3f (log is too long to include inline)

FFprobe output from your camera

N/A

Frigate stats

{"back_yard_south":{"camera_fps":6.0,"capture_pid":477,"detection_enabled":1,"detection_fps":0.2,"ffmpeg_pid":482,"pid":427,"process_fps":6.0,"skipped_fps":0.0},"basketball":{"camera_fps":6.1,"capture_pid":481,"detection_enabled":1,"detection_fps":2.6,"ffmpeg_pid":489,"pid":429,"process_fps":5.7,"skipped_fps":0.0},"bunk":{"camera_fps":6.1,"capture_pid":483,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":495,"pid":430,"process_fps":6.1,"skipped_fps":0.0},"cpu_usages":{"%Cpu(s):":{"cpu":"id,","mem":"0.6"},"1":{"cpu":"0.0","mem":"0.0"},"104":{"cpu":"0.0","mem":"0.0"},"128":{"cpu":"0.0","mem":"0.0"},"129":{"cpu":"0.0","mem":"0.0"},"130":{"cpu":"0.0","mem":"0.0"},"131":{"cpu":"0.0","mem":"0.0"},"134":{"cpu":"0.0","mem":"0.0"},"145":{"cpu":"0.0","mem":"0.0"},"15":{"cpu":"0.0","mem":"0.0"},"16":{"cpu":"0.0","mem":"0.0"},"166":{"cpu":"0.0","mem":"0.0"},"211":{"cpu":"0.0","mem":"0.0"},"2393":{"cpu":"3.0","mem":"0.1"},"2395":{"cpu":"2.6","mem":"0.0"},"24":{"cpu":"0.0","mem":"0.0"},"2403":{"cpu":"2.0","mem":"0.0"},"25":{"cpu":"0.0","mem":"0.0"},"26":{"cpu":"0.0","mem":"0.0"},"27":{"cpu":"0.0","mem":"0.0"},"28":{"cpu":"0.0","mem":"0.0"},"29":{"cpu":"0.0","mem":"0.0"},"30":{"cpu":"0.0","mem":"0.0"},"31":{"cpu":"0.0","mem":"0.0"},"40":{"cpu":"0.0","mem":"0.0"},"41":{"cpu":"0.0","mem":"0.0"},"412":{"cpu":"0.0","mem":"0.1"},"418":{"cpu":"2.0","mem":"0.0"},"419":{"cpu":"28.1","mem":"0.2"},"420":{"cpu":"54.3","mem":"0.2"},"427":{"cpu":"2.3","mem":"0.2"},"428":{"cpu":"0.0","mem":"0.0"},"429":{"cpu":"3.0","mem":"0.2"},"430":{"cpu":"0.7","mem":"0.2"},"431":{"cpu":"0.0","mem":"0.0"},"432":{"cpu":"1.0","mem":"0.2"},"433":{"cpu":"2.6","mem":"0.2"},"434":{"cpu":"0.0","mem":"0.0"},"435":{"cpu":"14.2","mem":"0.2"},"436":{"cpu":"16.9","mem":"0.2"},"437":{"cpu":"0.0","mem":"0.0"},"438":{"cpu":"1.7","mem":"0.2"},"439":{"cpu":"0.7","mem":"0.2"},"440":{"cpu":"0.0","mem":"0.0"},"441":{"cpu":"1.7","mem":"0.2"},"442":{"cpu":"0.0","mem":"0.0"},"443":{"cpu":"0.3","mem":"0.2"},"444":{"cpu":"0.0","mem":"0.0"},"445":{"cpu":"0.0","mem":"0.0"},"446":{"cpu":"1.0","mem":"0.2"},"447":{"cpu":"1.0","mem":"0.2"},"448":{"cpu":"0.7","mem":"0.2"},"449":{"cpu":"1.0","mem":"0.2"},"450":{"cpu":"1.0","mem":"0.2"},"451":{"cpu":"0.7","mem":"0.2"},"452":{"cpu":"0.7","mem":"0.2"},"453":{"cpu":"1.0","mem":"0.2"},"454":{"cpu":"13.6","mem":"0.2"},"455":{"cpu":"0.0","mem":"0.0"},"456":{"cpu":"0.3","mem":"0.2"},"457":{"cpu":"0.7","mem":"0.2"},"458":{"cpu":"0.0","mem":"0.0"},"459":{"cpu":"0.0","mem":"0.0"},"460":{"cpu":"2.6","mem":"0.2"},"461":{"cpu":"16.2","mem":"0.2"},"462":{"cpu":"0.0","mem":"0.0"},"463":{"cpu":"1.0","mem":"0.2"},"464":{"cpu":"0.0","mem":"0.0"},"465":{"cpu":"0.3","mem":"0.2"},"466":{"cpu":"14.9","mem":"0.2"},"467":{"cpu":"0.0","mem":"0.0"},"468":{"cpu":"16.9","mem":"0.2"},"469":{"cpu":"0.7","mem":"0.2"},"470":{"cpu":"0.0","mem":"0.0"},"471":{"cpu":"0.7","mem":"0.2"},"472":{"cpu":"0.7","mem":"0.2"},"473":{"cpu":"0.0","mem":"0.0"},"47312":{"cpu":"5.0","mem":"0.0"},"474":{"cpu":"14.6","mem":"0.2"},"475":{"cpu":"2.3","mem":"0.2"},"476":{"cpu":"0.0","mem":"0.0"},"477":{"cpu":"2.0","mem":"0.2"},"478":{"cpu":"0.0","mem":"0.0"},"481":{"cpu":"2.0","mem":"0.2"},"482":{"cpu":"2.3","mem":"0.1"},"483":{"cpu":"1.7","mem":"0.2"},"484":{"cpu":"0.0","mem":"0.0"},"489":{"cpu":"2.6","mem":"0.1"},"490":{"cpu":"2.3","mem":"0.2"},"493":{"cpu":"2.6","mem":"0.0"},"494":{"cpu":"0.0","mem":"0.0"},"495":{"cpu":"3.0","mem":"0.1"},"497":{"cpu":"2.3","mem":"0.2"},"502":{"cpu":"0.0","mem":"0.0"},"503":{"cpu":"2.6","mem":"0.1"},"503573":{"cpu":"3.3","mem":"0.1"},"503575":{"cpu":"1.3","mem":"0.0"},"504":{"cpu":"1.7","mem":"0.2"},"508":{"cpu":"1.0","mem":"0.0"},"509":{"cpu":"3.0","mem":"0.1"},"510":{"cpu":"2.3","mem":"0.2"},"511":{"cpu":"2.3","mem":"0.0"},"513":{"cpu":"0.0","mem":"0.0"},"518":{"cpu":"2.3","mem":"0.0"},"520":{"cpu":"3.6","mem":"0.1"},"521":{"cpu":"3.0","mem":"0.0"},"525":{"cpu":"2.3","mem":"0.0"},"526":{"cpu":"2.6","mem":"0.1"},"527":{"cpu":"1.7","mem":"0.2"},"528":{"cpu":"0.0","mem":"0.0"},"531":{"cpu":"2.0","mem":"0.2"},"536":{"cpu":"3.0","mem":"0.0"},"538":{"cpu":"2.0","mem":"0.2"},"541":{"cpu":"2.6","mem":"0.1"},"542":{"cpu":"0.0","mem":"0.0"},"544":{"cpu":"2.0","mem":"0.0"},"545":{"cpu":"1.7","mem":"0.2"},"546":{"cpu":"2.6","mem":"0.1"},"547528":{"cpu":"0.3","mem":"0.0"},"547543":{"cpu":"1.3","mem":"0.1"},"549":{"cpu":"0.0","mem":"0.0"},"550":{"cpu":"1.7","mem":"0.0"},"555":{"cpu":"2.6","mem":"0.1"},"558":{"cpu":"2.6","mem":"0.1"},"560":{"cpu":"3.0","mem":"0.0"},"561":{"cpu":"2.0","mem":"0.0"},"563":{"cpu":"2.3","mem":"0.0"},"564":{"cpu":"0.0","mem":"0.0"},"565":{"cpu":"2.3","mem":"0.2"},"576":{"cpu":"1.3","mem":"0.0"},"577":{"cpu":"2.3","mem":"0.2"},"578":{"cpu":"3.0","mem":"0.0"},"581":{"cpu":"0.0","mem":"0.0"},"582":{"cpu":"2.3","mem":"0.1"},"583":{"cpu":"2.0","mem":"0.0"},"584":{"cpu":"2.0","mem":"0.2"},"589":{"cpu":"2.3","mem":"0.2"},"590":{"cpu":"3.3","mem":"0.1"},"591":{"cpu":"0.0","mem":"0.0"},"592":{"cpu":"3.0","mem":"0.0"},"594":{"cpu":"2.0","mem":"0.0"},"598":{"cpu":"2.0","mem":"0.2"},"599":{"cpu":"2.6","mem":"0.1"},"602":{"cpu":"0.0","mem":"0.0"},"603":{"cpu":"2.0","mem":"0.2"},"605":{"cpu":"2.0","mem":"0.0"},"607":{"cpu":"3.0","mem":"0.0"},"615":{"cpu":"2.3","mem":"0.2"},"616":{"cpu":"3.0","mem":"0.1"},"618":{"cpu":"2.6","mem":"0.0"},"619":{"cpu":"2.3","mem":"0.0"},"620":{"cpu":"2.0","mem":"0.2"},"623":{"cpu":"0.0","mem":"0.0"},"624":{"cpu":"2.6","mem":"0.1"},"626":{"cpu":"3.0","mem":"0.1"},"627":{"cpu":"2.3","mem":"0.0"},"629":{"cpu":"2.0","mem":"0.2"},"630":{"cpu":"3.0","mem":"0.0"},"635":{"cpu":"0.0","mem":"0.0"},"636":{"cpu":"1.7","mem":"0.2"},"641":{"cpu":"2.6","mem":"0.0"},"642":{"cpu":"3.3","mem":"0.0"},"643":{"cpu":"1.7","mem":"0.2"},"644":{"cpu":"2.6","mem":"0.0"},"647":{"cpu":"2.6","mem":"0.1"},"650":{"cpu":"0.0","mem":"0.0"},"651":{"cpu":"2.3","mem":"0.2"},"653":{"cpu":"3.3","mem":"0.1"},"654":{"cpu":"1.7","mem":"0.2"},"658":{"cpu":"2.0","mem":"0.0"},"659":{"cpu":"3.0","mem":"0.1"},"662":{"cpu":"0.0","mem":"0.0"},"664":{"cpu":"1.7","mem":"0.2"},"668":{"cpu":"2.6","mem":"0.1"},"669":{"cpu":"2.6","mem":"0.1"},"671":{"cpu":"1.7","mem":"0.2"},"673":{"cpu":"1.7","mem":"0.0"},"675":{"cpu":"2.0","mem":"0.0"},"676":{"cpu":"13.2","mem":"0.1"},"677":{"cpu":"0.7","mem":"0.2"},"681":{"cpu":"3.0","mem":"0.0"},"687":{"cpu":"2.6","mem":"0.0"},"689":{"cpu":"1.0","mem":"0.0"},"690":{"cpu":"3.0","mem":"0.1"},"691":{"cpu":"2.0","mem":"0.0"},"698":{"cpu":"2.3","mem":"0.0"},"700":{"cpu":"4.0","mem":"0.2"},"706":{"cpu":"1.7","mem":"0.2"},"707":{"cpu":"3.0","mem":"0.1"},"717":{"cpu":"2.0","mem":"0.1"},"718":{"cpu":"1.7","mem":"0.0"},"729":{"cpu":"1.7","mem":"0.2"},"735":{"cpu":"5.6","mem":"0.1"},"749":{"cpu":"1.7","mem":"0.2"},"755":{"cpu":"2.6","mem":"0.1"},"756":{"cpu":"3.0","mem":"0.0"},"761":{"cpu":"1.7","mem":"0.2"},"763":{"cpu":"2.3","mem":"0.0"},"765":{"cpu":"2.6","mem":"0.0"},"766":{"cpu":"3.0","mem":"0.1"},"768":{"cpu":"2.0","mem":"0.0"},"769":{"cpu":"2.0","mem":"0.2"},"770":{"cpu":"3.3","mem":"0.0"},"772":{"cpu":"1.0","mem":"0.0"},"775":{"cpu":"2.3","mem":"0.0"},"776":{"cpu":"2.6","mem":"0.1"},"78":{"cpu":"0.0","mem":"0.0"},"783":{"cpu":"3.0","mem":"0.1"},"790":{"cpu":"1.0","mem":"0.0"},"793":{"cpu":"3.0","mem":"0.0"},"794":{"cpu":"2.0","mem":"0.0"},"795":{"cpu":"3.0","mem":"0.1"},"798":{"cpu":"3.0","mem":"0.0"},"799":{"cpu":"1.3","mem":"0.0"},"80":{"cpu":"0.0","mem":"0.0"},"800":{"cpu":"2.3","mem":"0.0"},"803":{"cpu":"1.3","mem":"0.0"},"804":{"cpu":"2.0","mem":"0.0"},"805":{"cpu":"1.7","mem":"0.0"},"81":{"cpu":"0.0","mem":"0.0"},"876":{"cpu":"1.7","mem":"0.0"},"88":{"cpu":"66.2","mem":"0.0"},"93":{"cpu":"33.8","mem":"0.8"},"993":{"cpu":"1.7","mem":"0.0"},"MiB":{"cpu":"53267.2","mem":"avail"},"PID":{"cpu":"%CPU","mem":"%MEM"},"Tasks:":{"cpu":"stopped,","mem":"0"},"top":{"cpu":"users,","mem":"load"}},"deck":{"camera_fps":6.0,"capture_pid":490,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":503,"pid":432,"process_fps":6.0,"skipped_fps":0.0},"detection_fps":44.7,"detectors":{"coral":{"detection_start":1679595253.171513,"inference_speed":18.52,"pid":419}},"dining":{"camera_fps":6.0,"capture_pid":497,"detection_enabled":1,"detection_fps":0.5,"ffmpeg_pid":509,"pid":433,"process_fps":6.0,"skipped_fps":0.0},"doorbell":{"camera_fps":6.0,"capture_pid":504,"detection_enabled":1,"detection_fps":4.8,"ffmpeg_pid":520,"pid":435,"process_fps":5.6,"skipped_fps":0.0},"driveway":{"camera_fps":6.0,"capture_pid":510,"detection_enabled":1,"detection_fps":5.7,"ffmpeg_pid":526,"pid":436,"process_fps":3.6,"skipped_fps":0.0},"entry":{"camera_fps":6.1,"capture_pid":527,"detection_enabled":1,"detection_fps":0.5,"ffmpeg_pid":541,"pid":438,"process_fps":6.1,"skipped_fps":0.0},"family":{"camera_fps":6.1,"capture_pid":531,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":546,"pid":439,"process_fps":6.1,"skipped_fps":0.0},"fire_pit":{"camera_fps":6.1,"capture_pid":538,"detection_enabled":1,"detection_fps":0.9,"ffmpeg_pid":555,"pid":441,"process_fps":6.0,"skipped_fps":0.0},"gaming":{"camera_fps":6.0,"capture_pid":545,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":558,"pid":443,"process_fps":6.0,"skipped_fps":0.0},"garage":{"camera_fps":6.1,"capture_pid":565,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":582,"pid":446,"process_fps":6.1,"skipped_fps":0.0},"gpu_usages":{"intel-vaapi":{"gpu":"1.0 %","mem":"- %"}},"great":{"camera_fps":6.1,"capture_pid":577,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":590,"pid":447,"process_fps":6.1,"skipped_fps":0.0},"hot_tub":{"camera_fps":6.1,"capture_pid":584,"detection_enabled":1,"detection_fps":0.1,"ffmpeg_pid":599,"pid":448,"process_fps":6.1,"skipped_fps":0.0},"john_deere":{"camera_fps":6.1,"capture_pid":589,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":2393,"pid":449,"process_fps":6.1,"skipped_fps":0.0},"laundry":{"camera_fps":6.0,"capture_pid":598,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":616,"pid":450,"process_fps":6.0,"skipped_fps":0.0},"loft":{"camera_fps":6.0,"capture_pid":603,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":624,"pid":451,"process_fps":6.0,"skipped_fps":0.0},"loft_door":{"camera_fps":6.1,"capture_pid":615,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":626,"pid":452,"process_fps":6.1,"skipped_fps":0.0},"man_door":{"camera_fps":6.0,"capture_pid":620,"detection_enabled":1,"detection_fps":0.3,"ffmpeg_pid":653,"pid":453,"process_fps":6.2,"skipped_fps":0.0},"mud":{"camera_fps":6.1,"capture_pid":629,"detection_enabled":1,"detection_fps":2.8,"ffmpeg_pid":647,"pid":454,"process_fps":5.8,"skipped_fps":0.0},"pantry":{"camera_fps":6.1,"capture_pid":636,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":659,"pid":456,"process_fps":6.1,"skipped_fps":0.0},"patio":{"camera_fps":6.1,"capture_pid":643,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":668,"pid":457,"process_fps":6.1,"skipped_fps":0.0},"playset":{"camera_fps":6.1,"capture_pid":651,"detection_enabled":1,"detection_fps":0.1,"ffmpeg_pid":669,"pid":460,"process_fps":6.1,"skipped_fps":0.0},"porch":{"camera_fps":6.0,"capture_pid":654,"detection_enabled":1,"detection_fps":5.1,"ffmpeg_pid":690,"pid":461,"process_fps":4.1,"skipped_fps":0.0},"service":{"last_updated":1679595257,"latest_version":"0.11.1","storage":{"/dev/shm":{"free":455.7,"mount_type":"tmpfs","total":536.9,"used":81.1},"/media/frigate/clips":{"free":119362.6,"mount_type":"ext4","total":241770.0,"used":110051.7},"/media/frigate/recordings":{"free":8300459.7,"mount_type":"ext4","total":27892949.5,"used":18192199.2},"/tmp/cache":{"free":1052.3,"mount_type":"tmpfs","total":2147.5,"used":1095.2}},"temperatures":{},"uptime":85107,"version":"0.12.0-27a31e7"},"sewing":{"camera_fps":6.0,"capture_pid":664,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":503573,"pid":463,"process_fps":6.0,"skipped_fps":0.0},"solar_panels":{"camera_fps":6.1,"capture_pid":671,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":707,"pid":465,"process_fps":6.1,"skipped_fps":0.0},"south_side":{"camera_fps":6.1,"capture_pid":677,"detection_enabled":1,"detection_fps":5.6,"ffmpeg_pid":717,"pid":466,"process_fps":5.6,"skipped_fps":0.0},"street":{"camera_fps":12.1,"capture_pid":700,"detection_enabled":1,"detection_fps":5.6,"ffmpeg_pid":735,"pid":468,"process_fps":3.7,"skipped_fps":0.0},"theater_back":{"camera_fps":6.1,"capture_pid":706,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":755,"pid":469,"process_fps":6.1,"skipped_fps":0.0},"theater_front":{"camera_fps":6.0,"capture_pid":729,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":766,"pid":471,"process_fps":6.0,"skipped_fps":0.0},"toy":{"camera_fps":6.0,"capture_pid":749,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":776,"pid":472,"process_fps":6.0,"skipped_fps":0.0},"trampoline":{"camera_fps":6.2,"capture_pid":761,"detection_enabled":1,"detection_fps":5.7,"ffmpeg_pid":783,"pid":474,"process_fps":2.4,"skipped_fps":0.0},"volleyball":{"camera_fps":6.0,"capture_pid":769,"detection_enabled":1,"detection_fps":4.2,"ffmpeg_pid":795,"pid":475,"process_fps":3.9,"skipped_fps":0.0}}

Operating system

HassOS

Install method

HassOS Addon

Coral version

USB

Network connection

Wired

Camera make and model

Mostly Hikvisions of various models. One UniFi doorbell cam. All cameras are >= 4MP

Any other information that may be helpful

No response