frigate: [Support]: Just can't figure this out: False Negatives at Night Only

Describe the problem you are having

I noticed that in the dark after my camera goes to night mode, Frigate struggles to detect humans in the frame. This appears to work just fine during the day when the image is in color mode. But once the image goes black and white with the IR spotlight running, object detection really struggles. The camera has plenty of illumination and the frames look well-lit to me, with someone obviously walking in the recordings. If I move slowly, Frigate will pick me up and create an event with recordings. But when walking at my normal speed, 90% of the time, I can pass by without a registered event.

I have tried multiple things, including reducing the min_area and min_score parameters, and increasing camera detection framerate. When monitoring the MQTT output, I don’t see anything published regarding detection in that camera. When viewing the live debug window, I do see the motion boxes, and correctly identified person with the orange box. I believe that I have the min_area set low enough with good margin. I am assuming that the detection score just doesn’t meet the threshold, which is left at the default 0.7.

Although I am running this on an RPi4 with a Coral TPU, I don’t believe the host performance is an issue here. Frigate does not report any skipped frames in detection. Although I am on the 12.x beta build, this issue has existed on the 11.x release builds as well. I don’t think this has anything to do with a regression in the new builds.

Here is a screen recording of the live debug window showing the bounding boxes, regions etc. as I walk up my driveway. About 25% of the time, it will catch me in the “upper driveway” zone, and almost never catch me in the “lower driveway” zone.

https://photos.app.goo.gl/ZpeQCLCSmrZ8PYcV6

I just can’t figure out how to debug this better. Any tips appreciated!

Version

0.12.0-0DBF909

Frigate config file

mqtt:
  host: 192.168.0.192
  user: frigate
  password: frigate1234


rtmp:
  # Optional: Enable the RTMP stream (default: False)
  enabled: False

# Optional: Restream configuration
# NOTE: Can be overridden at the camera level
restream:
  # Optional: Enable the restream (default: True)
  enabled: True
  # Optional: Force audio compatibility with browsers (default: shown below)
  # force_audio: True
  # Optional: Video encoding to be used. By default the codec will be copied but
  # it can be switched to another or an MJPEG stream can be encoded and restreamed
  # as h264 (default: shown below)
  video_encoding: "copy"
  # Optional: Restream birdseye via RTSP (default: shown below)
  # NOTE: Enabling this will set birdseye to run 24/7 which may increase CPU usage somewhat.
  birdseye: False
  # Optional: jsmpeg stream configuration for WebUI
  # jsmpeg:
  #   # Optional: Set the height of the jsmpeg stream. (default: 720)
  #   # This must be less than or equal to the height of the detect stream. Lower resolutions
  #   # reduce bandwidth required for viewing the jsmpeg stream. Width is computed to match known aspect ratio.
  #   height: 720
  #   # Optional: Set the encode quality of the jsmpeg stream (default: shown below)
  #   # 1 is the highest quality, and 31 is the lowest. Lower quality feeds utilize less CPU resources.
  #   quality: 8

cameras:
  puppy_cam1:
    ffmpeg:
      hwaccel_args:
      #   #output_args:
      #   #  detect: -f rawvideo -vcodec h264_v4l2m2m -pix_fmt yuv420p
        - -c:v
      #     # - -vcodec
      #     #- h264_mmal
        - h264_v4l2m2m
      input_args:
        # - -avoid_negative_ts
        # - make_zero
        # - -fflags
        # - nobuffer
        # - -flags
        # - low_delay
        # - -strict
        # - experimental
        # - -fflags
        # - +genpts+discardcorrupt
        - -rtsp_transport 
        - 'tcp'
        - -use_wallclock_as_timestamps
        - "1"
      inputs:
        - path: rtsp://USER:PASSWORD@192.168.0.219:554/h264Preview_01_sub
          roles:
            - record
            # - rtmp
    restream:
      # Optional: Enable the restream (default: True)
      enabled: False
    detect:
      enabled: False
      width: 640
      height: 352
      fps: 1
    record:
      # Optional: Enable recording (default: global setting)
      enabled: True
      # Optional: Number of days to retain (default: global setting)
      retain:
        days: 3
      # events:
      #   retain:
      #     default: 14

  front_door:
    ffmpeg:
      hwaccel_args:
        #output_args:
        #  detect: -f rawvideo -vcodec h264_v4l2m2m -pix_fmt yuv420p
          - -c:v
          # - -vcodec
          #- h264_mmal
          - h264_v4l2m2m
      
      input_args:
        #- -avoid_negative_ts
        #- make_zero
        # - -fflags
        # - nobuffer
        # - -flags
        # - low_delay
        # - -strict
        # - experimental
        # - -fflags
        # - +genpts+discardcorrupt
        - -rtsp_transport 
        - 'tcp'
#        - -stimeout 
#        - '5000000'
        #- -reorder_queue_size
        #- '1000'
      #output_args:
        #record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy
      inputs:
        - path: rtsp://USER:PASSWORD@192.168.0.191:554/cam/realmonitor?channel=1&subtype=0
        # - path: rtsp://192.168.0.192:8554/front_door
          roles:
            - record
            - restream
            - detect
            #- record

    zones:
      porch:
        coordinates: 608,930,889,933,897,810,1580,846,1920,1080,0,1080,0,0,649,0
        filters:
          person:
            min_area: 25000
    
    detect:
      width: 1920
      height: 1080
      fps: 5
    motion:
      mask:
        #- 598,455,688,519,882,496,1055,483,1173,594,1682,595,1920,395,1920,0,545,0
        #- 288,358,688,519,882,496,1055,483,1173,594,1920,606,1920,395,1920,0,0,0
        #- 0,1080,613,1080,606,539,897,561,1038,555,1157,620,1920,606,1920,395,1920,0,0,0
        - 0,1080,613,1080,598,587,926,613,1157,620,1312,883,1920,1080,1920,395,1920,0,0,0
    objects:
      track:
        - person
      filters:
        person:
          min_area: 10000
    #clips:
    #  enabled: True
      # retain:
      #   default: 5
    record:
      # Optional: Enable recording (default: global setting)
      enabled: True
      # Optional: Number of days to retain (default: global setting)
      retain:
        days: 7
      events:
        retain:
          default: 14
      
  front_driveway:
    objects:
      track:
        - person
        # - car
      filters:
        person:
          # min_area: 10000
          mask: 183,142,398,36,896,38,896,36,896,0,0,0,0,198
          min_score: 0.4          
        # car:
        #   min_area: 2000
    zones:
      # driveway_entrance:
      #   # coordinates: 235,118,334,131,332,194,0,308,0,197
      #   coordinates: 186,156,358,178,0,308,0,223
      #   objects:
      #     - car
      #   filters:
      #     car:
      #       min_area: 5000
      lower_driveway:
        # coordinates: 90,418,615,230,620,104,234,135,0,223,0,377
        coordinates: 82,395,776,227,778,105,619,80,282,139,0,242,0,377
        objects:
          - person
          # - car
        filters:
          person:
            min_area: 1200
          # car:
          #   min_area: 5000
      upper_driveway:
        coordinates: 896,512,896,133,782,109,776,227,83,400,0,372,0,512
        objects:
          - person
          # - car
        filters:
          person:
            min_area: 4000
          # car:
          #   min_area: 5000

#     motion:
#       mask:
# #        - 92,227,211,122,336,58,626,62,640,0,0,0,0,257
#         - 242,189,336,58,896,72,896,0,0,0,0,268
    # motion:
    #   mask:
    #     - 174,150,336,58,896,72,896,0,0,0,0,209
    motion:
      mask:
        - 183,142,398,36,896,38,896,36,896,0,0,0,0,198


    ffmpeg:
      inputs:
        - path: http://192.168.0.241/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=USER&password=PASSWORD
          roles:
            - detect
            - restream
            # - record
            #- clips
          hwaccel_args:
              - -c:v
              - h264_v4l2m2m
            
          input_args:
            - -avoid_negative_ts
            - make_zero
            - -fflags
            - +genpts+discardcorrupt
            - -flags
            - low_delay
            - -strict
            - experimental
            - -analyzeduration
            - 1000M
            - -probesize
            - 1000M
            - -rw_timeout
            - "5000000"
        - path: rtmp://192.168.0.241/bcs/channel0_main.bcs?channel=0&stream=0&user=USER&password=PASSWORD
          input_args:
            - -avoid_negative_ts
            - make_zero
            # - -flags
            # - low_delay
            - -strict
            - experimental
            - -fflags
            - +genpts+discardcorrupt
            - -rw_timeout
            - "15000000"
          roles:
            - record
    detect:
      width: 896
      height: 512
      fps: 8

    snapshots:
      # Optional: Enable writing jpg snapshot to /media/frigate/clips (default: shown below)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: True
      # Optional: save a clean PNG copy of the snapshot image (default: shown below)
      clean_copy: True
      # Optional: print a timestamp on the snapshots (default: shown below)
      timestamp: False
      # Optional: draw bounding box on the snapshots (default: shown below)
      bounding_box: True
      # Optional: crop the snapshot (default: shown below)
      crop: False
      # Optional: height to resize the snapshot to (default: original size)
      # height: 175
      # Optional: Restrict snapshots to objects that entered any of the listed zones (default: no required zones)
      required_zones:
        - upper_driveway
        - lower_driveway
        # - driveway_entrance
      # Optional: Camera override for retention settings (default: global values)
      retain:
        # Required: Default retention days (default: shown below)
        default: 14
        # objects:
        #   car: 7

    record:
      # Optional: Enable recording (default: global setting)
      enabled: True
      # Optional: Number of days to retain (default: global setting)
      retain:
        days: 7
      events:
        objects:
          - person
          # - car
        required_zones:
          - upper_driveway
          - lower_driveway
          # - driveway_entrance
        retain:
          default: 14

  front_driveway_sd:
    ffmpeg:
      inputs:
        - path: rtmp://192.168.0.241/bcs/channel0_sub.bcs?channel=0&stream=0&user=USER&password=PASSWORD
          input_args:
            - -avoid_negative_ts
            - make_zero
            - -strict
            - experimental
            - -fflags
            - +genpts+discardcorrupt
            - -rw_timeout
            - "15000000"
          roles:
            - record
          hwaccel_args:
            - -c:v
            - h264_v4l2m2m
    record:
      # Optional: Enable recording (default: global setting)
      enabled: True
      # Optional: Number of days to retain (default: global setting)
      retain:
        days: 1
    snapshots:
      # Optional: Enable writing jpg snapshot to /media/frigate/clips (default: shown below)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: False
    detect:
      enabled: False
      width: 640
      height: 360
      fps: 1
    restream:
      enabled: False


detectors:
  coral:
    type: edgetpu
    device: usb

# ffmpeg:
  # hwaccel_args:
  # #output_args:
  # #  detect: -f rawvideo -vcodec h264_v4l2m2m -pix_fmt yuv420p
  #   - -c:v
  #   # - -vcodec
  #   #- h264_mmal
  #   - h264_v4l2m2m
    #- -level
    #- '41'
    #- -num_output_buffers
    #- '32'
    #- -num_capture_buffers
    #- '16'
    #- -c:a
    #- copy
#  global_args: -hide_banner -loglevel debug
  # output_args:
    # record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac <=== used to be enabled.
    
detect:
  stationary:
    interval: 20
    threshold: 50
    max_frames:
      # Optional: Default for all object types (default: not set, track forever)
      default: 300
      # Optional: Object specific values
      # objects:
      #   car: 300

Relevant log output

NA

FFprobe output from your camera

Input #0, flv, from 'http://192.168.0.241/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=USER&password=PASSWORD':
  Metadata:
    displayWidth    : 896
    displayHeight   : 512
  Duration: 00:00:00.00, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: h264 (High), yuv420p(progressive), 896x512, 11 fps, 11.99 tbr, 1k tbn
    Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp

Frigate stats

No response

Operating system

Debian

Install method

HassOS Addon

Coral version

USB

Network connection

Wired

Camera make and model

Reolink RLC-810A

Any other information that may be helpful

No response