frigate: [Detector Support]: OpenVINO not running yolo model

Describe the problem you are having

I am currently running in OpenVINO Detector mode with the default ssd model, which runs fine with hardware acceleration enabled. But when I try switching to yolox_tiny or yolov8, frigate is able to start, but throws some error with the detetor process, and the camera stream is not showing anything on the webui.

Frigate is running in a Ubuntu VM within proxmox6.2

Version

0.12.0-DA3E197

Frigate config file

mqtt:
 host: xxxxxxxxx
 port: xxxxxxxxxxx
 user: xxxxxxx
 password: xxxxxxxxx
 client_id: xxxxxxxxx

detectors:
  ov:
    type: openvino
    device: AUTO
    model:
      path: /media/frigate/yolox_tiny.xml

model:
  width: 416
  height: 416
  input_tensor: nchw
  input_pixel_format: bgr
  model_type: yolox
  labelmap_path: /media/frigate/coco_80cl.txt

objects:
  track:
    - person
  filters:
    person:
      min_score: 0.5

cameras: 
  camera_1: 
    record:
      enabled: false 
    ffmpeg:
      hwaccel_args: preset-vaapi
      inputs:
        - path: rtsp://xxxxxxxxxxxxxxxxxxx
          roles:
            - detect
    detect:
      width: 1280
      height: 720


rtmp:
  # Optional: Enable the RTMP stream (default: True)
  enabled: False

docker-compose file or Docker CLI command

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "64mb" # update for your cameras based on calculation above
    devices:
      - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /home/madafaka/frigate/config/config.yml:/config/config.yml
      - /home/madafaka/frigate:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "1935:1935" # RTMP feeds

    environment:
      FRIGATE_RTSP_PASSWORD: "password"

Relevant log output

2023-04-27 13:58:05.003881913  [INFO] Starting Frigate...
2023-04-27 13:58:05.963998016  [2023-04-27 13:58:05] frigate.app                    INFO    : Starting Frigate (0.12.0-da3e197)
2023-04-27 13:58:05.980415277  [2023-04-27 13:58:05] frigate.config                 WARNING : Customizing more than a detector model path is unsupported.
2023-04-27 13:58:05.985063592  [2023-04-27 13:58:05] peewee_migrate                 INFO    : Starting migrations
2023-04-27 13:58:05.989579990  [2023-04-27 13:58:05] peewee_migrate                 INFO    : There is nothing to migrate
2023-04-27 13:58:06.012421253  [2023-04-27 13:58:06] frigate.app                    INFO    : Output process started: 216
2023-04-27 13:58:06.022377762  [2023-04-27 13:58:06] frigate.app                    INFO    : Camera processor started for camera_1: 218
2023-04-27 13:58:06.028934129  [2023-04-27 13:58:06] frigate.app                    INFO    : Capture process started for camera_1: 220
2023-04-27 13:58:06.033817977  [2023-04-27 13:58:06] detector.ov                    INFO    : Starting detection process: 215
2023-04-27 13:58:06.064916777  Process detector:ov:
2023-04-27 13:58:06.066389060  Traceback (most recent call last):
2023-04-27 13:58:06.066470549    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-04-27 13:58:06.066472536      self.run()
2023-04-27 13:58:06.066474190    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-04-27 13:58:06.066478658      self._target(*self._args, **self._kwargs)
2023-04-27 13:58:06.066480967    File "/opt/frigate/frigate/object_detection.py", line 98, in run_detector
2023-04-27 13:58:06.066494475      object_detector = LocalObjectDetector(detector_config=detector_config)
2023-04-27 13:58:06.066496284    File "/opt/frigate/frigate/object_detection.py", line 52, in __init__
2023-04-27 13:58:06.066498293      self.detect_api = create_detector(detector_config)
2023-04-27 13:58:06.066499820    File "/opt/frigate/frigate/detectors/__init__.py", line 24, in create_detector
2023-04-27 13:58:06.066501234      return api(detector_config)
2023-04-27 13:58:06.066502905    File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 26, in __init__
2023-04-27 13:58:06.066541312      self.ov_model = self.ov_core.read_model(detector_config.model.path)
2023-04-27 13:58:06.066543034  RuntimeError: Check 'false' failed at ../src/frontends/common/src/frontend.cpp:53:
2023-04-27 13:58:06.066555050  Converting input model
2023-04-27 13:58:06.066556330  
2023-04-27 13:58:07.797525702  [2023-04-27 13:58:07] frigate.video                  ERROR   : camera_1: Unable to read frames from ffmpeg process.
2023-04-27 13:58:07.799072158  [2023-04-27 13:58:07] frigate.video                  ERROR   : camera_1: ffmpeg process is not running. exiting capture thread...

Operating system

Proxmox

Install method

Docker Compose

Coral version

Other

Any other information that may be helpful

No response

About this issue

Original URL
State: closed
Created a year ago
Comments: 24 (7 by maintainers)

Most upvoted comments

I am having the same issue trying to setup yolox_tiny.

Model conversion to FP16 has been successful, using

omz_converter --name yolox-tiny

xml and label file have been copied in an accessible folder.

========== Converting yolox-tiny to ONNX
Conversion to ONNX command: /usr/bin/python3 -- /usr/local/lib/python3.9/dist-packages/openvino/model_zoo/internal_scripts/pytorch_to_onnx.py --model-path=/usr/local/lib/python3.9/dist-packages/openvino/model_zoo/models/public/yolox-tiny --model-path=/root/public/yolox-tiny --model-name=create_model --import-module=model '--model-param=weights=r"/root/public/yolox-tiny/yolox_tiny.pth"' --input-shape=1,3,416,416 --input-names=images --output-names=output --output-file=/root/public/yolox-tiny/yolox-tiny.onnx

============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

ONNX check passed successfully.

========== Converting yolox-tiny to IR (FP16)
Conversion command: /usr/bin/python3 -- /usr/local/bin/mo --framework=onnx --data_type=FP16 --output_dir=/root/public/yolox-tiny/FP16 --model_name=yolox-tiny --input=images '--mean_values=images[123.675,116.28,103.53]' '--scale_values=images[58.395,57.12,57.375]' --reverse_input_channels --output=output --input_model=/root/public/yolox-tiny/yolox-tiny.onnx '--layout=images(NCHW)' '--input_shape=[1, 3, 416, 416]'

[ WARNING ]  Use of deprecated cli option --data_type detected. Option use in the following releases will be fatal. 
Check for a new version of Intel(R) Distribution of OpenVINO(TM) toolkit here https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit/download.html?cid=other&source=prod&campid=ww_2023_bu_IOTG_OpenVINO-2022-3&content=upg_all&medium=organic or on https://github.com/openvinotoolkit/openvino
[ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11.
Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/latest/openvino_2_0_transition_guide.html
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /root/public/yolox-tiny/FP16/yolox-tiny.xml
[ SUCCESS ] BIN file: /root/public/yolox-tiny/FP16/yolox-tiny.bin

The errors I have in my log are exactly the same as OP.

EDIT: I just created a yolov8 model using @aeozyalcin colab and also got the same error:

2023-05-11 08:27:08.160692654  [2023-05-11 10:27:08] frigate.app                    INFO    : Starting Frigate (0.12.0-bc16ad1)
2023-05-11 08:27:08.192834786  [2023-05-11 10:27:08] frigate.config                 WARNING : Customizing more than a detector model path is unsupported.
2023-05-11 08:27:08.198703630  [2023-05-11 10:27:08] frigate.app                    INFO    : Creating directory: /tmp/cache
2023-05-11 08:27:08.200963801  [2023-05-11 10:27:08] peewee_migrate                 INFO    : Starting migrations
2023-05-11 08:27:08.205537574  [2023-05-11 10:27:08] peewee_migrate                 INFO    : There is nothing to migrate
2023-05-11 08:27:08.224000695  [2023-05-11 10:27:08] frigate.app                    INFO    : Output process started: 300
2023-05-11 08:27:08.231060140  [2023-05-11 10:27:08] detector.ov                    INFO    : Starting detection process: 299
2023-05-11 08:27:08.241083810  [2023-05-11 10:27:08] frigate.app                    INFO    : Camera processor started for Parking_cam: 304
2023-05-11 08:27:08.244106245  Process detector:ov:
2023-05-11 08:27:08.245734199  Traceback (most recent call last):
2023-05-11 08:27:08.246335791    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-05-11 08:27:08.246338433      self.run()
2023-05-11 08:27:08.246704498  [2023-05-11 10:27:08] frigate.app                    INFO    : Camera processor started for Shed_cam: 305
2023-05-11 08:27:08.246854245    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-05-11 08:27:08.246856581      self._target(*self._args, **self._kwargs)
2023-05-11 08:27:08.246980952    File "/opt/frigate/frigate/object_detection.py", line 98, in run_detector
2023-05-11 08:27:08.246983287      object_detector = LocalObjectDetector(detector_config=detector_config)
2023-05-11 08:27:08.247105717    File "/opt/frigate/frigate/object_detection.py", line 52, in __init__
2023-05-11 08:27:08.247107884      self.detect_api = create_detector(detector_config)
2023-05-11 08:27:08.247214681    File "/opt/frigate/frigate/detectors/__init__.py", line 24, in create_detector
2023-05-11 08:27:08.247216679      return api(detector_config)
2023-05-11 08:27:08.247326151    File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 26, in __init__
2023-05-11 08:27:08.247328352      self.ov_model = self.ov_core.read_model(detector_config.model.path)
2023-05-11 08:27:08.247449968  RuntimeError: Check 'false' failed at ../src/frontends/common/src/frontend.cpp:53:
2023-05-11 08:27:08.247451909  Converting input model

EDIT EDIT: It was the stupidest thing ever. I copied the xml and the model description, but was missing the other .mapping and .bin files. With all the files there, I have been able to use yolox-tiny and yolov8 without any issue

yannpub on Jun 13, 2023

@aeozyalcin you’re right! The problem is the version of openvino-dev. I installed a version before the latest one (i.e., 2022.3.1), and the three files are created in both directories (FP16 and FP32). After putting it on Frigate, it works like a charm! Thank you!

Log file:

2023-06-22 19:18:36.162158769  [2023-06-22 19:18:36] frigate.detectors.plugins.openvino INFO    : Model Input Shape: {1, 3, 416, 416}
2023-06-22 19:18:36.162284517  [2023-06-22 19:18:36] frigate.detectors.plugins.openvino INFO    : Model Output-0 Shape: {1, 3549, 85}
2023-06-22 19:18:36.162381106  [2023-06-22 19:18:36] frigate.detectors.plugins.openvino INFO    : Model has 1 Output Tensors
2023-06-22 19:18:36.162468181  [2023-06-22 19:18:36] frigate.detectors.plugins.openvino INFO    : YOLOX model has 80 classes

Config file:

detectors:
  ov:
    type: openvino
    device: GPU

model:
  path: /media/frigate/models/yolox-tiny/FP16/yolox-tiny.xml
  width: 416
  height: 416
  input_tensor: nchw
  input_pixel_format: bgr
  model_type: yolox
  labelmap_path: /media/frigate/models/yolox-tiny/FP16/coco_80cl.txt

Inference Speed ~25 ms in a UHD Graphics 770 / 12th Gen Intel® Core™ i5-12500T.

ArlindoFNeto on Jun 22, 2023

Thanks aeozyalcin. This is what I have done. I have left my holiday house, were the camera setup sits, last night. I will resume testing when I get back there, about two weeks from now, and report back. Philippe

philippedegeer on Jun 14, 2023

It is finally working now, thanks for updating the collab notebook. I have done a few things in yesterday:

1, regenerating the yolov8n model using your updated collab notebook.

2, updated the Proxmox version from 6.2 to 7.4, and the process also updated the Proxmox kernel from 5.4 to 5.15. After this update, in lspci command, it displays the detailed iGPU name (UHD610) instead of general intel vga device before.

3, I also updated the kernel of the Ubuntu VM from 5.4 to 5.15 also.

DysinLarshee on Apr 30, 2023