frigate: [Support]: OpenVINO Does not work with YOLOX model

Describe the problem you are having

Awesome job getting OpenVINO into Frigate @NateMeyer! I just got a thin client with a J5005, and OpenVINO support was perfectly timed!

I tried using the yolox-tiny model with OpenVINO, and got the following error:

2023-01-21 20:32:53.946348978  [2023-01-21 12:32:53] detector.ov                    INFO    : Starting detection process: 376
2023-01-21 20:32:55.017048607  [2023-01-21 12:32:55] frigate.detectors.plugins.openvino INFO    : Model Input Shape: {1, 3, 416, 416}
2023-01-21 20:32:55.017056288  [2023-01-21 12:32:55] frigate.detectors.plugins.openvino INFO    : Model Output-0 Shape: {1, 3549, 85}
2023-01-21 20:32:55.017058803  [2023-01-21 12:32:55] frigate.detectors.plugins.openvino INFO    : Model has 1 Output Tensors
2023-01-21 20:32:55.019078340  Process detector:ov:
2023-01-21 20:32:55.042640498  Traceback (most recent call last):
2023-01-21 20:32:55.042646939    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-01-21 20:32:55.042648687      self.run()
2023-01-21 20:32:55.042650805    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-01-21 20:32:55.042652584      self._target(*self._args, **self._kwargs)
2023-01-21 20:32:55.042654630    File "/opt/frigate/frigate/object_detection.py", line 120, in run_detector
2023-01-21 20:32:55.042656489      detections = object_detector.detect_raw(input_frame)
2023-01-21 20:32:55.042658500    File "/opt/frigate/frigate/object_detection.py", line 71, in detect_raw
2023-01-21 20:32:55.042660363      return self.detect_api.detect_raw(tensor_input=tensor_input)
2023-01-21 20:32:55.042662445    File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 45, in detect_raw
2023-01-21 20:32:55.042664053      infer_request.infer([tensor_input])
2023-01-21 20:32:55.042666257    File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 151, in infer
2023-01-21 20:32:55.042667807      return super().infer(
2023-01-21 20:32:55.042670418  RuntimeError: Can't set input blob with name: images, because model input (shape={1,3,416,416}) and blob (shape=(1.416.416.3)) are incompatible

Looks like there is an incompatibility between how Frigate sends frames to the yolo model, and what the model expects as its input. When I use the included ssdlite_mobilenet_v2 model, object detection is working as expected.

P.S.: I also tried Intel’s own person-detection-0201 model. Frigate ran without errors, but no detections were showing up. I don’t think that’s related, but just a data point.

Version

0.12.0-12D51D3

Frigate config file

mqtt:
  enabled: False

detectors:
  ov:
    type: openvino
    device: GPU
    model:
      path: /media/frigate/openvino-model/yolox-tiny.xml   

model:
  width: 416
  height: 416
  input_tensor: nhwc
  input_pixel_format: bgr
  labelmap_path: /media/frigate/openvino-model/coco_80cl.txt 

cameras:
  front_driveway:
    objects:
      track:
        - person
    ffmpeg:
      inputs:
        - path: http://192.168.0.241/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=admin&password=password
          roles:
            - detect
            - restream
          hwaccel_args: preset-intel-qsv-h264
          input_args:
            - -strict
            - experimental
            - -analyzeduration
            - 1000M
            - -probesize
            - 1000M
            - -rw_timeout
            - "5000000"

    detect:
      enabled: True
      width: 896
      height: 512
      fps: 10

    zones:

      lower_driveway:
        coordinates: 480,191,486,81,320,92,191,136,185,178,0,242,0,377
        objects:
          - person
        filters:
          person:
            min_area: 1000
            max_area: 10000
      middle_driveway:
        coordinates: 83,400,704,326,784,228,798,92,478,69,474,182,0,372
        objects:
          - person
        filters:
          person:
            min_area: 2000
            max_area: 20000          
      upper_driveway:
        coordinates: 896,512,896,123,783,89,776,227,703,326,83,400,0,372,0,512
        objects:
          - person
        filters:
          person:
            min_area: 4500
            max_area: 100000   

    motion:
      mask:
        - 201,113,398,36,896,38,896,36,896,0,0,0,0,165
      improve_contrast: True
      contour_area: 25

    snapshots:
      # Optional: Enable writing jpg snapshot to /media/frigate/clips (default: shown below)
      # This value can be set via MQTT and will be updated in startup based on retained value
      enabled: True
      # Optional: save a clean PNG copy of the snapshot image (default: shown below)
      clean_copy: True
      # Optional: print a timestamp on the snapshots (default: shown below)
      timestamp: False
      # Optional: draw bounding box on the snapshots (default: shown below)
      bounding_box: True
      # Optional: crop the snapshot (default: shown below)
      crop: False
      required_zones:
        - upper_driveway
        - middle_driveway
        - lower_driveway
      # Optional: height to resize the snapshot to (default: original size)
      # height: 175
      # Optional: Restrict snapshots to objects that entered any of the listed zones (default: no required zones)
        # - driveway_entrance
      # Optional: Camera override for retention settings (default: global values)
      retain:
        # Required: Default retention days (default: shown below)
        default: 2
        # objects:
        #   car: 7
    record:
      # Optional: Enable recording (default: global setting)
      enabled: True
      # Optional: Number of days to retain (default: global setting)
      retain:
        days: 1
      events:
        objects:
          - person
        required_zones:
          - upper_driveway
          - middle_driveway
          - lower_driveway
        retain:
          default: 2

Relevant log output

2023-01-21 20:32:53.946348978  [2023-01-21 12:32:53] detector.ov                    INFO    : Starting detection process: 376
2023-01-21 20:32:55.017048607  [2023-01-21 12:32:55] frigate.detectors.plugins.openvino INFO    : Model Input Shape: {1, 3, 416, 416}
2023-01-21 20:32:55.017056288  [2023-01-21 12:32:55] frigate.detectors.plugins.openvino INFO    : Model Output-0 Shape: {1, 3549, 85}
2023-01-21 20:32:55.017058803  [2023-01-21 12:32:55] frigate.detectors.plugins.openvino INFO    : Model has 1 Output Tensors
2023-01-21 20:32:55.019078340  Process detector:ov:
2023-01-21 20:32:55.042640498  Traceback (most recent call last):
2023-01-21 20:32:55.042646939    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-01-21 20:32:55.042648687      self.run()
2023-01-21 20:32:55.042650805    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-01-21 20:32:55.042652584      self._target(*self._args, **self._kwargs)
2023-01-21 20:32:55.042654630    File "/opt/frigate/frigate/object_detection.py", line 120, in run_detector
2023-01-21 20:32:55.042656489      detections = object_detector.detect_raw(input_frame)
2023-01-21 20:32:55.042658500    File "/opt/frigate/frigate/object_detection.py", line 71, in detect_raw
2023-01-21 20:32:55.042660363      return self.detect_api.detect_raw(tensor_input=tensor_input)
2023-01-21 20:32:55.042662445    File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 45, in detect_raw
2023-01-21 20:32:55.042664053      infer_request.infer([tensor_input])
2023-01-21 20:32:55.042666257    File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 151, in infer
2023-01-21 20:32:55.042667807      return super().infer(
2023-01-21 20:32:55.042670418  RuntimeError: Can't set input blob with name: images, because model input (shape={1,3,416,416}) and blob (shape=(1.416.416.3)) are incompatible

FFprobe output from your camera

"[\n  {\n    \"return_code\": 0,\n    \"stderr\": {},\n    \"stdout\": {\n      \"programs\": [],\n      \"streams\": [\n        {\n          \"avg_frame_rate\": \"19/1\",\n          \"codec_long_name\": \"H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10\",\n          \"height\": 512,\n          \"width\": 896\n        },\n        {\n          \"avg_frame_rate\": \"0/0\",\n          \"codec_long_name\": \"AAC (Advanced Audio Coding)\"\n        }\n      ]\n    }\n  }\n]"

Frigate stats

No response

Operating system

Other Linux

Install method

HassOS Addon

Coral version

Other

Network connection

Wired

Camera make and model

Reolink RLC-810A

Any other information that may be helpful

No response

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 80 (42 by maintainers)

Most upvoted comments

Hmm that’s interesting. I also wonder if that version already has the input normalized, and the color order reversed.

Let’s take the guessing out of the game. I have made this Colab for you guys. Give this a try and let me know. Convert YOLOv8 to OpenVino for Frigate.ipynb

+5
aeozyalcin on Mar 1, 2023

Hmm that’s interesting. I also wonder if that version already has the input normalized, and the color order reversed.

Let’s take the guessing out of the game. I have made this Colab for you guys. Give this a try and let me know. Convert YOLOv8 to OpenVino for Frigate.ipynb

Once again amazing help. Seems to be working perfectly. Ill continue with my journey to try and build a pruned coco trained model that only has people, dogs and cats in it, but thats for another day.

+3
nickp27 on Mar 1, 2023

https://github.com/blakeblackshear/frigate/pull/5285

+2
aeozyalcin on Jan 29, 2023

Beta is coming soon™

+1
NickM-27 on Jul 28, 2023

Leave that true. It’s for Nvidia GPUs. FP16 with optimize_cpu true is most likely the most optimal for your internal Intel GPU.

+1
aeozyalcin on Mar 1, 2023

Very interesting. Nothing stands out to me as something that could be causing the poor performance. I am running the detector on a 720p/1080p stream, so it’s in the similar resolution range as what you are running yours at. The only thing I can think of is maybe the subjects are too far, but I wouldn’t be able to compare to mine without seeing a picture from your camera.

thanks for checking 👍

Few unrelated remarks, I see you are using the CPU detector with FP32 precision. Is there a reason why you aren’t using FP16 with the GPU detector instead? I assume you have an Intel CPU, so you probably also have an Intel integrated GPU?

It’s unrelated to frigate, I got a problem with kernel 5.19 sometimes GPU is hanging and until I have time to revert to 5.15, I use CPU. Regarding 32 vs 16 precision, I’m not sure why I chose 32 tbh, probably because my lack of knowledge around what to use in what circumstances and laziness to google it. You think I should make it 16 ?

PS: I just got yolov5 working in Frigate with OpenVino, and it’s working really promising so far. I will create a PR when I feel it’s ready. I love how flexible Yolov5 is.

fantastic work 🥳 looking forward to trying it out 🦾

+1
alez007 on Feb 13, 2023

Very nice work @aeozyalcin. Looking forward to the new beta to try this.

How are your experiences so far between yolox_tiny and ssdlite_mobilenet_v2 ?

+1
Caros2017 on Feb 4, 2023

@aeozyalcin whenever I got an error like that, it was because I already had a frigate container that wasn’t removed.

The docker image is fairly painless to build. Fork/clone the dev branch of the repo and run make local to generate a frigate:latest image you can use on your machine.

+1
NateMeyer on Jan 27, 2023

The model input can be fixed by seeing input_tensor: nchw, but the OpenVino detector expects the output tensor to be an SSD format and will error out with the YoloX output shape.

+1
NateMeyer on Jan 22, 2023
Read more comments on GitHub
← frigate: [Support]: OpenVINO detector crashing, hanging machine
frigate: [Support]: orangepi 5B run ghcr.io/blakeblackshear/frigate:0.13.0-beta6-rk fail →