frigate: [Support]: High CPU usage for ffmpeg process

Describe the problem you are having

CPU usage of the ffmpeg process is high, and slighlty higher (!) with hwaccel turned on. I’d like to optimise it as I can’t scale down the resolution further on some cameras (G4s) and CPU usage is still high.

I’m running on a Raspberry Pi CM4 with 8GB RAM, and I’ve set GPU memory to 256MB. I’m using the cameras named “Tapo2” and “G42” just to watch the stream in Home Assistant. No detection, no recording. And most of the time, little change there. The camera named “G41” is in the middle of the house, with lots of movements, and detection enabled (processed by the Coral USB).

Tests done (G41 and G42 have very similar CPU usage despite different configs) - see pictures below

• (1) Tapo2 on 640x360 stream (lowest quality), no hwaccel --> ffpmeg CPU process around 10% (HAPPY)
• (2) Tapo2 on 640x360 stream (lowest quality), with hwaccel --> ffpmeg CPU process around 14% (still HAPPY, but surprising)
• (1) G41 on 1280x720 stream (lowest quality!), no hwaccel --> ffpmeg CPU process around 68% (SAD)
• (2) G41 on 1280x720 stream (lowest quality!), with hwaccel --> ffpmeg CPU process around 74% (SAD)
• (3) Tapo2 on 1920x1080 stream (high quality), no hwaccel --> ffpmeg CPU process around 77% (SAD)

What I don’t understand:

• Between Tapo2 low resolution and a G4 there are 4x more pixels (and twice more fps?) --> CPU usage is way higher (10% vs 68%)
• Between a G4 and Tapo2 high resolution there are 2x more pixels --> CPU usage is slightly higher (68% vs 77%)
• Why is hwaccel CPU usage for ffmpeg worse?
• Is there any unwanted operations happening in ffmpeg (resizing?) I could deactivate?

Is my configuration incorrect in some ways? Should I add other arguments to ffmpeg input/output config? Any other suggestion? (some links from https://github.com/blakeblackshear/frigate/issues/3860 are now dead, and couldn’t find another idea to try from other “high CPU usage” issues)

Thanks a lot for the guidance

Version

0.12.0-DA3E197

Frigate config file (Test number 2: low res, with hwaccel)

# yaml-language-server: $schema=http://localhost:5555/api/config/schema
mqtt:
  host: mosquitto

## Hardware acceleration for raspberry 4 (64bit OS)
ffmpeg:
  hwaccel_args: preset-rpi-64-h264
# ffmpeg:
#   hwaccel_args:
#     - -c:v
#     - h264_v4l2m2m

cameras:

  camera_tapo2: # <------ Name the camera
    ffmpeg:
      inputs:
        - path: rtsp://xxx:xxx@xxx:554/stream2 # Low 640x360 stream
          # roles:
          #   - rtmp
    rtmp:
      enabled: False # <-- RTMP should be disabled if your stream is not H264
    detect:
      enabled: False # No detection in kids room
      width: 640 # <---- update for your camera's resolution
      height: 360 # <---- update for your camera's resolution
    mqtt:
      enabled: false # Deactivated to save CPU time
      quality: 100
      timestamp: false
      bounding_box: false
      crop: True
      height: 360

  camera_g41: # <------ Name the camera
    ffmpeg:
      inputs:
          # From the RTSPS link in Unifi protect, remove the s, change port to 7447
          # and remove the final enableSTPS
        - path: rtsp://xxx:7447/xxx # Medium 1280x720 stream
          roles:
            - detect
    rtmp:
      enabled: False # <-- RTMP should be disabled if your stream is not H264
    detect:
      width: 1280 # <---- update for your camera's resolution
      height: 720 # <---- update for your camera's resolution
    mqtt:
      enabled: true
      quality: 100
      timestamp: false
      bounding_box: false
      crop: True
      height: 720

  camera_g42: # <------ Name the camera
    ffmpeg:
      inputs:
          # From the RTSPS link in Unifi protect, remove the s, change port to 7447
          # and remove the final enableSTPS
        - path: rtsp://xxx:7447/xxx # Medium 1280x720 stream
          # roles:
          #   - rtmp
    rtmp:
      enabled: False # <-- RTMP should be disabled if your stream is not H264
    detect:
      enabled: False # No detection for FFA
      width: 1280 # <---- update for your camera's resolution
      height: 720 # <---- update for your camera's resolution
    mqtt:
      enabled: false # Deactivated to save CPU time
      quality: 100
      timestamp: false
      bounding_box: false
      crop: True
      height: 720

  camera_iegeek1: # <------ Name the camera
    ffmpeg:
      inputs:
        - path: rtsp://xxx:554/2 # Low 640x352 stream
          roles:
            - detect
    motion:
      mask:
        #- 0,0,0,981,523,142,1212,105,1920,326,1920,0 # Mask for High quality stream
        - 0,0,0,352,139,32,344,31,640,260,640,0 # Mask for Low quality stream
    rtmp:
      enabled: False # <-- RTMP should be disabled if your stream is not H264
    detect:
      enabled: True
      width: 640 # <---- update for your camera's resolution
      height: 352 # <---- update for your camera's resolution
    objects:
      filters:
        person:
          min_score: 0.7
          threshold: 0.8
    record:
      enabled: true
      retain:
        days: 0
        mode: motion
      events:
        retain:
          default: 2
          mode: active_objects
        pre_capture: 5
        post_capture: 5
        objects:
          - person
    mqtt:
      enabled: false # Deactivated to save CPU time
      quality: 100
      timestamp: false
      bounding_box: false
      crop: True
      height: 352

snapshots:
  # Optional: draw bounding box on the snapshots (default: shown below)
  bounding_box: True


detectors:
  coral:
    type: edgetpu
    device: usb

Relevant log output

Nothing very relevant to me here…

2023-05-10 20:13:39.176710078  [INFO] Starting Frigate...
2023-05-10 20:13:43.706241317  [2023-05-10 20:13:43] frigate.app                    INFO    : Starting Frigate (0.12.0-da3e197)
2023-05-10 20:13:43.847165359  [2023-05-10 20:13:43] peewee_migrate                 INFO    : Starting migrations
2023-05-10 20:13:43.917734907  [2023-05-10 20:13:43] peewee_migrate                 INFO    : There is nothing to migrate
2023-05-10 20:13:44.019192115  [2023-05-10 20:13:44] detector.coral                 INFO    : Starting detection process: 292
2023-05-10 20:13:44.029202701  [2023-05-10 20:13:44] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as usb
2023-05-10 20:13:44.036203392  [2023-05-10 20:13:44] frigate.app                    INFO    : Output process started: 294
2023-05-10 20:13:44.080914675  [2023-05-10 20:13:44] frigate.app                    INFO    : Camera processor started for camera_tapo2: 300
2023-05-10 20:13:44.117291725  [2023-05-10 20:13:44] frigate.app                    INFO    : Camera processor started for camera_g41: 302
2023-05-10 20:13:44.138320408  [2023-05-10 20:13:44] frigate.app                    INFO    : Camera processor started for camera_g42: 303
2023-05-10 20:13:44.182987895  [2023-05-10 20:13:44] frigate.app                    INFO    : Camera processor started for camera_iegeek1: 305
2023-05-10 20:13:44.215882201  [2023-05-10 20:13:44] frigate.app                    INFO    : Capture process started for camera_tapo2: 307
2023-05-10 20:13:44.237082641  [2023-05-10 20:13:44] frigate.app                    INFO    : Capture process started for camera_g41: 308
2023-05-10 20:13:44.276377793  [2023-05-10 20:13:44] frigate.app                    INFO    : Capture process started for camera_g42: 312
2023-05-10 20:13:44.329073553  [2023-05-10 20:13:44] frigate.app                    INFO    : Capture process started for camera_iegeek1: 317
2023-05-10 20:13:46.818909676  [2023-05-10 20:13:46] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found

FFprobe output from your cameras

Tapo2:
  Metadata:
    title           : Session streamed by "TP-LINK RTSP Server"
    comment         : stream2
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 640x360, 15 fps, 28.58 tbr, 90k tbn, 30 tbc
    Stream #0:1: Audio: pcm_alaw, 8000 Hz, 1 channels, s16, 64 kb/s

G41:
  Metadata:
    title           : D021F99547EB_1
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Audio: aac (LC), 16000 Hz, mono, fltp
    Stream #0:1: Audio: opus, 48000 Hz, stereo, fltp
    Stream #0:2: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 1280x720, 30 fps, 30 tbr, 90k tbn, 60 tbc

G42:
  Metadata:
    title           : D021F995415F_1
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Audio: aac (LC), 16000 Hz, mono, fltp
    Stream #0:1: Audio: opus, 48000 Hz, stereo, fltp
    Stream #0:2: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 1280x720, 30 fps, 30 tbr, 90k tbn, 60 tbc

Frigate stats

See screen shots below

Operating system

Other Linux

Install method

Docker Compose

Coral version

USB

Network connection

Wired

Camera make and model

1x Tapo c200, 2x Ubiquiti G4, 1x ieGeek whatever

Any other information that may be helpful

• Test (1) - low res, no hwaccel
• Test (2) - low res, with hwaccel
• Test (3) - tapo high res, no hwaccel