frigate: [HW Accel Support]: Host machine/container cuda version mismatch.

Describe the problem you are having

My host machine has cuda12 installed and the container has 11.7 listed in requirements-tensorrt.txt

Pretty sure cuda is backwards compatible so a bump shouldn’t break anything.

Produces the log below:

2023-04-15 09:35:28.398163981  [2023-04-15 09:35:28] frigate.detectors.plugins.tensorrt ERROR   : ERROR: failed to load libraries. libcudart.so.12: cannot open shared object file: No such file or directory

this is the file present inside the container:

root@160c9873dfa9:/opt/frigate# find / | grep libcudart
/usr/local/lib/python3.9/dist-packages/nvidia/cuda_runtime/lib/libcudart.so.11.0

my system info

[creynolds@masterblaster frigate]$ uname -a
Linux masterblaster 6.2.9-200.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 30 22:31:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
[creynolds@masterblaster frigate]$ cat /etc/redhat-release
Fedora release 37 (Thirty Seven)
[creynolds@masterblaster frigate]$ nvidia-smi
Sat Apr 15 09:48:37 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.41.03              Driver Version: 530.41.03    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Quadro P2000                    Off| 00000000:01:00.0 Off |                  N/A |
| 45%   31C    P8                4W /  75W|      2MiB /  5120MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+



### Version

0.12.0-da3e197

### Frigate config file

```yaml
mqtt:
  host: mqtt
database:
  path: /db/frigate.db

detectors:
  tensorrt:
    type: tensorrt
    device: 0 #This is the default, select the first GPU

model:
  path: /trt-models/yolov7x-640.trt
  input_tensor: nchw
  input_pixel_format: rgb
  width: 640
  height: 480

birdseye:
  enabled: True
  restream: False
  width: 1280
  height: 720
  quality: 8
  mode: continuous

detect:
  width: 1280
  height: 720
  fps: 15

record:
  enabled: True
  retain:
    days: 0
  events:
    pre_capture: 8
    post_capture: 8
    retain:
      default: 30

snapshots:
  enabled: True

go2rtc:
  streams:
    camera1:
      - rtsp://viewonly:viewonly12@192.168.7.181:554/cam/realmonitor?channel=1&subtype=0
      - "ffmpeg:camera1#video=h264"
    camera1_sub:
      - rtsp://viewonly:viewonly12@192.168.7.181:554/cam/realmonitor?channel=1&subtype=1
    camera2:
      - rtsp://viewonly:viewonly12@192.168.7.182:554/cam/realmonitor?channel=1&subtype=0
      - "ffmpeg:camera2#video=h264"
    camera2_sub:
      - rtsp://viewonly:viewonly12@192.168.7.182:554/cam/realmonitor?channel=1&subtype=1
    camera3:
      - rtsp://viewonly:viewonly12@192.168.7.183:554/cam/realmonitor?channel=1&subtype=0
      - "ffmpeg:camera3#video=h264"
    camera3_sub:
      - rtsp://viewonly:viewonly12@192.168.7.183:554/cam/realmonitor?channel=1&subtype=1
    camera4:
      - rtsp://viewonly:viewonly12@192.168.7.184:554/cam/realmonitor?channel=1&subtype=0
      - "ffmpeg:camera4#video=h264"
    camera4_sub:
      - rtsp://viewonly:viewonly12@192.168.7.184:554/cam/realmonitor?channel=1&subtype=1
  webrtc:
    candidates:
      - 192.168.7.201:8555
      - stun:8555

cameras:
  camera1:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/camera1
          input_args: preset-rtsp-restream
          roles:
            - record
            #- detect
        - path: rtsp://127.0.0.1:8554/camera1_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
  camera2:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/camera2
          input_args: preset-rtsp-restream
          roles:
            - record
            #- detect
        - path: rtsp://127.0.0.1:8554/camera2_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
  camera3:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/camera3
          input_args: preset-rtsp-restream
          roles:
            - record
            #- detect
        - path: rtsp://127.0.0.1:8554/camera3_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
  camera4:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/camera4
          input_args: preset-rtsp-restream
          roles:
            - record
            #- detect
        - path: rtsp://127.0.0.1:8554/camera4_sub
          input_args: preset-rtsp-restream
          roles:
            - detect

docker-compose file or Docker CLI command

version: "3"
services:
  frigate:
    container_name: frigate
    # add groups from host for render, plugdev, video
    group_add:
      - "998" # render
      - "39"  # video
    shm_size: "1024mb"
    image: ghcr.io/blakeblackshear/frigate:0.12.0-tensorrt
    environment:
      - NVIDIA_DRIVER_CAPABILITIES=all
      - NVIDIA_VISIBLE_DEVICES=all
    devices:
      #      - /dev/dri:/dev/dri
      - /dev/nvidia0:/dev/nvidia0
      - /dev/nvidiactl:/dev/nvidiactl
      - /dev/nvidia-modeset:/dev/nvidia-modeset
      - /dev/nvidia-uvm:/dev/nvidia-uvm
      - /dev/nvidia-uvm-tools:/dev/nvidia-uvm-tools
      - /dev/bus/usb:/dev/bus/usb
      - /dev/dri:/dev/dri
    deploy:
          resources:
              reservations:
                  devices:
                      - driver: nvidia
                        count: 1
                        capabilities: [gpu]
    privileged: true
    ports:
       - '5000:5000/tcp'
       - '5001:5001/tcp'
    volumes:
      - .:/workspace/frigate:cached
      - ./web/dist:/opt/frigate/web:cached
      - /etc/localtime:/etc/localtime:ro
      - ./config/:/config/:ro
      - ./debug:/media/frigate
      - ./db/:/db/
      - ./trt-models:/trt-models
      - /dev/bus/usb:/dev/bus/usb
  mqtt:
    container_name: mqtt
    image: eclipse-mosquitto:1.6
    ports:
      - "1883:1883"

Relevant log output

s6-rc: info: service s6rc-fdholder: starting
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service s6rc-fdholder successfully started
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service log-prepare: starting
s6-rc: info: service log-prepare successfully started
s6-rc: info: service nginx-log: starting
s6-rc: info: service go2rtc-log: starting
s6-rc: info: service frigate-log: starting
s6-rc: info: service nginx-log successfully started
s6-rc: info: service go2rtc-log successfully started
s6-rc: info: service go2rtc: starting
s6-rc: info: service frigate-log successfully started
s6-rc: info: service go2rtc successfully started
s6-rc: info: service go2rtc-healthcheck: starting
s6-rc: info: service frigate: starting
s6-rc: info: service go2rtc-healthcheck successfully started
s6-rc: info: service frigate successfully started
s6-rc: info: service nginx: starting
s6-rc: info: service nginx successfully started
s6-rc: info: service legacy-services: starting
2023-04-15 09:35:24.474028731  [INFO] Preparing go2rtc config...
2023-04-15 09:35:24.474069947  [INFO] Starting Frigate...
2023-04-15 09:35:24.474677048  [INFO] Starting NGINX...
s6-rc: info: service legacy-services successfully started
2023-04-15 09:35:25.421057805  [INFO] Not injecting WebRTC candidates into go2rtc config as it has been set manually
2023-04-15 09:35:25.466178842  [INFO] Starting go2rtc...
2023-04-15 09:35:25.585350853  09:35:25.584 INF go2rtc version 1.2.0 linux/amd64
2023-04-15 09:35:25.585573528  09:35:25.585 INF [api] listen addr=:1984
2023-04-15 09:35:25.586033460  09:35:25.585 INF [rtsp] listen addr=:8554
2023-04-15 09:35:25.586615313  09:35:25.586 INF [srtp] listen addr=:8443
2023-04-15 09:35:25.586987129  09:35:25.586 INF [webrtc] listen addr=:8555
2023-04-15 09:35:27.948783263  [2023-04-15 09:35:27] frigate.app                    INFO    : Starting Frigate (0.12.0-da3e197)
2023-04-15 09:35:28.016844500  [2023-04-15 09:35:28] peewee_migrate                 INFO    : Starting migrations
2023-04-15 09:35:28.069571516  [2023-04-15 09:35:28] peewee_migrate                 INFO    : There is nothing to migrate
2023-04-15 09:35:28.120357245  [2023-04-15 09:35:28] frigate.app                    INFO    : Output process started: 441
2023-04-15 09:35:28.130203777  [2023-04-15 09:35:28] frigate.app                    INFO    : Camera processor started for camera1: 444
2023-04-15 09:35:28.140065643  [2023-04-15 09:35:28] frigate.app                    INFO    : Camera processor started for camera2: 445
2023-04-15 09:35:28.151050113  [2023-04-15 09:35:28] frigate.app                    INFO    : Camera processor started for camera3: 447
2023-04-15 09:35:28.152124504  [2023-04-15 09:35:28] detector.tensorrt              INFO    : Starting detection process: 439
2023-04-15 09:35:28.161122272  [2023-04-15 09:35:28] frigate.app                    INFO    : Camera processor started for camera4: 449
2023-04-15 09:35:28.176210619  [2023-04-15 09:35:28] frigate.app                    INFO    : Capture process started for camera1: 450
2023-04-15 09:35:28.187715454  [2023-04-15 09:35:28] frigate.app                    INFO    : Capture process started for camera2: 454
2023-04-15 09:35:28.224273371  [2023-04-15 09:35:28] frigate.app                    INFO    : Capture process started for camera3: 459
2023-04-15 09:35:28.257211216  [2023-04-15 09:35:28] frigate.app                    INFO    : Capture process started for camera4: 467
2023-04-15 09:35:28.398163981  [2023-04-15 09:35:28] frigate.detectors.plugins.tensorrt ERROR   : ERROR: failed to load libraries. libcudart.so.12: cannot open shared object file: No such file or directory
2023-04-15 09:35:29.565415398  [2023-04-15 09:35:29] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init CUDA: CPU +189, GPU +0, now: CPU 245, GPU 132 (MiB)
2023-04-15 09:35:30.175258220  Process camera_processor:camera2:
2023-04-15 09:35:30.183258956  Traceback (most recent call last):
2023-04-15 09:35:30.183265741    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-04-15 09:35:30.183267707      self.run()
2023-04-15 09:35:30.183269993    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-04-15 09:35:30.183275489      self._target(*self._args, **self._kwargs)
2023-04-15 09:35:30.183277555    File "/opt/frigate/frigate/video.py", line 480, in track_camera
2023-04-15 09:35:30.183304188      process_frames(
2023-04-15 09:35:30.183306224    File "/opt/frigate/frigate/video.py", line 725, in process_frames
2023-04-15 09:35:30.183308433      detect(
2023-04-15 09:35:30.183310221    File "/opt/frigate/frigate/video.py", line 546, in detect
2023-04-15 09:35:30.183312136      region_detections = object_detector.detect(tensor_input)
2023-04-15 09:35:30.183313963    File "/opt/frigate/frigate/object_detection.py", line 208, in detect
2023-04-15 09:35:30.183343166      self.np_shm[:] = tensor_input[:]
2023-04-15 09:35:30.183345564  ValueError: could not broadcast input array from shape (1,640,480,3) into shape (1,480,640,3)
2023-04-15 09:35:30.200231636  Process camera_processor:camera4:
2023-04-15 09:35:30.201536917  Traceback (most recent call last):
2023-04-15 09:35:30.201779862    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-04-15 09:35:30.201782797      self.run()
2023-04-15 09:35:30.201923154    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-04-15 09:35:30.201925922      self._target(*self._args, **self._kwargs)
2023-04-15 09:35:30.202043936    File "/opt/frigate/frigate/video.py", line 480, in track_camera
2023-04-15 09:35:30.202046225      process_frames(
2023-04-15 09:35:30.202174690    File "/opt/frigate/frigate/video.py", line 725, in process_frames
2023-04-15 09:35:30.202176791      detect(
2023-04-15 09:35:30.202306790    File "/opt/frigate/frigate/video.py", line 546, in detect
2023-04-15 09:35:30.202309232      region_detections = object_detector.detect(tensor_input)
2023-04-15 09:35:30.202449917    File "/opt/frigate/frigate/object_detection.py", line 208, in detect
2023-04-15 09:35:30.202452628      self.np_shm[:] = tensor_input[:]
2023-04-15 09:35:30.202609751  ValueError: could not broadcast input array from shape (1,640,480,3) into shape (1,480,640,3)
2023-04-15 09:35:30.287129708  Process camera_processor:camera1:
2023-04-15 09:35:30.289133575  Traceback (most recent call last):
2023-04-15 09:35:30.289495419    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-04-15 09:35:30.289498810      self.run()
2023-04-15 09:35:30.289677821    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-04-15 09:35:30.289680734      self._target(*self._args, **self._kwargs)
2023-04-15 09:35:30.289816025    File "/opt/frigate/frigate/video.py", line 480, in track_camera
2023-04-15 09:35:30.289818605      process_frames(
2023-04-15 09:35:30.289955394    File "/opt/frigate/frigate/video.py", line 725, in process_frames
2023-04-15 09:35:30.289957788      detect(
2023-04-15 09:35:30.290086416    File "/opt/frigate/frigate/video.py", line 546, in detect
2023-04-15 09:35:30.290089061      region_detections = object_detector.detect(tensor_input)
2023-04-15 09:35:30.290232236    File "/opt/frigate/frigate/object_detection.py", line 208, in detect
2023-04-15 09:35:30.290234801      self.np_shm[:] = tensor_input[:]
2023-04-15 09:35:30.290401080  ValueError: could not broadcast input array from shape (1,640,480,3) into shape (1,480,640,3)
2023-04-15 09:35:30.369906527  Process camera_processor:camera3:
2023-04-15 09:35:30.371179709  Traceback (most recent call last):
2023-04-15 09:35:30.371332794    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-04-15 09:35:30.371335671      self.run()
2023-04-15 09:35:30.371394953    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-04-15 09:35:30.371397665      self._target(*self._args, **self._kwargs)
2023-04-15 09:35:30.371449085    File "/opt/frigate/frigate/video.py", line 480, in track_camera
2023-04-15 09:35:30.371451429      process_frames(
2023-04-15 09:35:30.371504476    File "/opt/frigate/frigate/video.py", line 725, in process_frames
2023-04-15 09:35:30.371506496      detect(
2023-04-15 09:35:30.371549675    File "/opt/frigate/frigate/video.py", line 546, in detect
2023-04-15 09:35:30.371552363      region_detections = object_detector.detect(tensor_input)
2023-04-15 09:35:30.371602921    File "/opt/frigate/frigate/object_detection.py", line 208, in detect
2023-04-15 09:35:30.371605359      self.np_shm[:] = tensor_input[:]
2023-04-15 09:35:30.371678946  ValueError: could not broadcast input array from shape (1,640,480,3) into shape (1,480,640,3)
2023-04-15 09:35:32.459394714  [2023-04-15 09:35:32] frigate.detectors.plugins.tensorrt INFO    : Loaded engine size: 328 MiB

FFprobe output from your camera

root@160c9873dfa9:/opt/frigate# ffprobe "rtsp://viewonly:viewonly12@192.168.7.181:554/cam/realmonitor?channel=1&subtype=0"
ffprobe version n5.1-2-g915ef932a3-20220731 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 12.1.0 (crosstool-NG 1.25.0.55_3defb7b)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-ffbuild-linux-gnu- --arch=x86_64 --target-os=linux --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-mbedtls --enable-librist --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --enable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags=-pie --extra-libs='-ldl -lgomp' --extra-version=20220731
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, rtsp, from 'rtsp://viewonly:viewonly12@192.168.7.181:554/cam/realmonitor?channel=1&subtype=0':
  Metadata:
    title           : Media Server
  Duration: N/A, start: 0.138000, bitrate: N/A
  Stream #0:0: Video: hevc (Main), yuvj420p(pc), 3840x2160, 15 fps, 100 tbr, 90k tbn
  Stream #0:1: Audio: aac (LC), 8000 Hz, mono, fltp
root@160c9873dfa9:/opt/frigate# ffprobe "rtsp://viewonly:viewonly12@192.168.7.181:554/cam/realmonitor?channel=1&subtype=1"
ffprobe version n5.1-2-g915ef932a3-20220731 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 12.1.0 (crosstool-NG 1.25.0.55_3defb7b)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=x86_64-ffbuild-linux-gnu- --arch=x86_64 --target-os=linux --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-libdav1d --enable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-mbedtls --enable-librist --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --enable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --enable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags=-pie --extra-libs='-ldl -lgomp' --extra-version=20220731
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, rtsp, from 'rtsp://viewonly:viewonly12@192.168.7.181:554/cam/realmonitor?channel=1&subtype=1':
  Metadata:
    title           : Media Server
  Duration: N/A, start: 0.128000, bitrate: N/A
  Stream #0:0: Video: hevc (Main), yuvj420p(pc), 704x480, 15 fps, 100 tbr, 90k tbn
  Stream #0:1: Audio: aac (LC), 8000 Hz, mono, fltp
root@160c9873dfa9:/opt/frigate#

Operating system

Other Linux

Install method

Docker Compose

Network connection

Wired

Camera make and model

Amcrest https://www.amazon.com/dp/B07RHQGS8V/

Any other information that may be helpful

No response