frigate: [Support]: bug with Proxmox (takes down Proxmox entirely with a green screen)

Describe the problem you are having

I have been trying to get to the bottom of an issue where Proxmox randomly crashes. Following several days of troubleshooting I have come to the realisation if you run the Frigate HA add-on + Frigate HACS Integration + Frigate HA Integration on the same HAOS instance after a while the entire Proxmox server becomes unresponsive and only way out is physical power cycle the machine Proxmox is installed. I provide version numbers below, but to me it seems some kind of kernel panic type of error.

It seems I have overcome this issue by splitting Frigate on two HA instances. One instance runs MQTT and Frigate add-on, the other HA instance runs Frigate Proxy, Frigate HACS integration + card and Frigate HA integration. I have yet to try on a pure Debian installation, however, I am confident this will also work.

To me it seems a fatal conflict between MQTT, Frigate HA add-on + Frigate HACS Integration + Frigate HA integration running on the same HA instance, bringing the Proxmox server down (no error messages are logged in Proxmox or HAOS),

I am testing on a simple Frigate installation using just one camera stream.

Everything below is on latest production (non-beta) versions as of 15 Aug 2022.

Frigate System: HW: AMD HX5900 - allocated to HAOS/Frigate VM: 8-cores/8GB RAM/64GB SSD DSK

Latest HAOS with following add-ons (all latest production versions):

Proxmox 7.2-7 no-subscription, up to date with latest packages (15 Aug 2022). I have yet to test on Proxmox with subscription. HAOS 8.4

  • MariaDB (recorder)
  • MQTT
  • File Editor
  • SSh & Web Terminal
  • Samba
  • Google Backup
  • ESPHome

  • DLNA Renderer
  • DLNA Server
  • Radio Browser
  • Shopping List
  • Sun
  • UPnP/IGD

  • Cloudflare
  • Meteorologisk Institutt (Met.no)
  • Pi-Hole

  • Broadlink
  • Shelly

HACS

  • Adaptive Lighting
  • Frigate
  • Music Assistant

  • Sun Card
  • Frigate Card
  • Mushroom Cards

I have tried to troubleshoot and it seems MQTT interacting with Frigate HA add-on and Frigate HACS integration is the culprit. I am still testing as we speak, and will update as soon as I have valid input to provide. Having split the add-on from the integration seems to have resolved the issue (no problem for several hours), but the acid test will be tomorrow; if no crash then the issue is in the interaction between the three components mentioned earlier (MQTT, add-on, HACS).

Version

DEBUG 0.10.1-83481AF

Frigate config file

mqtt:
  host: haos-frigate.local
  port: 1883
  user: admin
  password: admin
  client_id: haos-frigate
  topic_prefix: haos-frigate
  stats_interval: 60

birdseye:
  enabled: True
  width: 1280
  height: 720
  quality: 8
  mode: continuous

cameras:
  living-room:
    ffmpeg:
      inputs:
        - path: rtsp://rtsp:xxxxxxx@192.168.0.72:554/av_stream/ch0
          roles:
            - detect
            - record
            - rtmp
    detect:
      width: 1920
      height: 1080

record:
  enabled: True
  expire_interval: 60
  retain:
    days: 0
    mode: all

  events:
    max_seconds: 240
    pre_capture: 4
    post_capture: 4
    objects:
      - person
    required_zones: []
    retain:
      default: 8
      mode: motion
      objects:
        person: 8

snapshots:
  enabled: True
  timestamp: True
  bounding_box: False
  crop: False
  height: 175
  required_zones: []
  retain:
    default: 8
    objects:
      person: 8

Relevant log output

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[2022-08-15 14:57:43] frigate.app                    INFO    : Starting Frigate (0.10.1-83481af)
[2022-08-15 14:57:43] frigate.app                    INFO    : Creating directory: /tmp/cache
Starting migrations
[2022-08-15 14:57:43] peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
[2022-08-15 14:57:43] peewee_migrate                 INFO    : There is nothing to migrate
[2022-08-15 14:57:43] frigate.app                    INFO    : Output process started: 217
[2022-08-15 14:57:43] frigate.app                    INFO    : Camera processor started for living-room: 220
[2022-08-15 14:57:43] frigate.app                    INFO    : Capture process started for living-room: 221
[2022-08-15 14:57:43] ws4py                          INFO    : Using epoll
[2022-08-15 14:57:43] ws4py                          INFO    : Using epoll
[2022-08-15 14:57:43] detector.cpu                   INFO    : Starting detection process: 216
[2022-08-15 14:57:43] frigate.edgetpu                WARNING : CPU detectors are not recommended and should only be used for testing or for trial purposes.
[2022-08-15 15:13:35] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:51658]
[2022-08-15 15:14:53] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:51658]
[2022-08-15 15:15:35] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:58810]
[2022-08-15 15:15:46] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:58810]
[2022-08-15 15:17:39] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:55776]
[2022-08-15 15:18:33] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:55776]

FFprobe output from your camera

N/A

Frigate stats

{
  "birdseye": {
    "enabled": true,
    "height": 720,
    "mode": "continuous",
    "quality": 8,
    "width": 1280
  },
  "cameras": {
    "living-room": {
      "best_image_timeout": 60,
      "detect": {
        "enabled": true,
        "fps": 5,
        "height": 1080,
        "max_disappeared": 25,
        "stationary": {
          "interval": 0,
          "max_frames": {
            "default": null,
            "objects": {}
          },
          "threshold": 50
        },
        "width": 1920
      },
      "ffmpeg": {
        "global_args": [
          "-hide_banner",
          "-loglevel",
          "warning"
        ],
        "hwaccel_args": [],
        "input_args": [
          "-avoid_negative_ts",
          "make_zero",
          "-fflags",
          "+genpts+discardcorrupt",
          "-rtsp_transport",
          "tcp",
          "-stimeout",
          "5000000",
          "-use_wallclock_as_timestamps",
          "1"
        ],
        "inputs": [
          {
            "global_args": [],
            "hwaccel_args": [],
            "input_args": [],
            "path": "rtsp://rtsp:xxxxxx@192.168.0.72:554/av_stream/ch0",
            "roles": [
              "record",
              "rtmp",
              "detect"
            ]
          }
        ],
        "output_args": {
          "detect": [
            "-f",
            "rawvideo",
            "-pix_fmt",
            "yuv420p"
          ],
          "record": [
            "-f",
            "segment",
            "-segment_time",
            "10",
            "-segment_format",
            "mp4",
            "-reset_timestamps",
            "1",
            "-strftime",
            "1",
            "-c",
            "copy",
            "-an"
          ],
          "rtmp": [
            "-c",
            "copy",
            "-f",
            "flv"
          ]
        }
      },
      "ffmpeg_cmds": [
        {
          "cmd": "ffmpeg -hide_banner -loglevel warning -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://rtsp:xxxxxxx@192.168.0.72:554/av_stream/ch0 -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/living-room-%Y%m%d%H%M%S.mp4 -c copy -f flv rtmp://127.0.0.1/live/living-room -r 5 -s 1920x1080 -f rawvideo -pix_fmt yuv420p pipe:",
          "roles": [
            "record",
            "rtmp",
            "detect"
          ]
        }
      ],
      "live": {
        "height": 720,
        "quality": 8
      },
      "motion": {
        "contour_area": 30,
        "delta_alpha": 0.2,
        "frame_alpha": 0.2,
        "frame_height": 50,
        "improve_contrast": false,
        "mask": "",
        "threshold": 25
      },
      "mqtt": {
        "bounding_box": true,
        "crop": true,
        "enabled": true,
        "height": 270,
        "quality": 70,
        "required_zones": [],
        "timestamp": true
      },
      "name": "living-room",
      "objects": {
        "filters": {
          "person": {
            "mask": null,
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          }
        },
        "mask": "",
        "track": [
          "person"
        ]
      },
      "record": {
        "enabled": true,
        "events": {
          "max_seconds": 240,
          "objects": [
            "person"
          ],
          "post_capture": 4,
          "pre_capture": 4,
          "required_zones": [],
          "retain": {
            "default": 8,
            "mode": "motion",
            "objects": {
              "person": 8
            }
          }
        },
        "expire_interval": 60,
        "retain": {
          "days": 0,
          "mode": "all"
        },
        "retain_days": null
      },
      "rtmp": {
        "enabled": true
      },
      "snapshots": {
        "bounding_box": false,
        "clean_copy": true,
        "crop": false,
        "enabled": true,
        "height": 175,
        "quality": 70,
        "required_zones": [],
        "retain": {
          "default": 8,
          "mode": "motion",
          "objects": {
            "person": 8
          }
        },
        "timestamp": true
      },
      "timestamp_style": {
        "color": {
          "blue": 255,
          "green": 255,
          "red": 255
        },
        "effect": null,
        "format": "%m/%d/%Y %H:%M:%S",
        "position": "tl",
        "thickness": 2
      },
      "zones": {}
    }
  },
  "database": {
    "path": "/media/frigate/frigate.db"
  },
  "detect": {
    "enabled": true,
    "fps": 5,
    "height": 720,
    "max_disappeared": null,
    "stationary": {
      "interval": 0,
      "max_frames": {
        "default": null,
        "objects": {}
      },
      "threshold": null
    },
    "width": 1280
  },
  "detectors": {
    "cpu": {
      "device": "usb",
      "num_threads": 3,
      "type": "cpu"
    }
  },
  "environment_vars": {},
  "ffmpeg": {
    "global_args": [
      "-hide_banner",
      "-loglevel",
      "warning"
    ],
    "hwaccel_args": [],
    "input_args": [
      "-avoid_negative_ts",
      "make_zero",
      "-fflags",
      "+genpts+discardcorrupt",
      "-rtsp_transport",
      "tcp",
      "-stimeout",
      "5000000",
      "-use_wallclock_as_timestamps",
      "1"
    ],
    "output_args": {
      "detect": [
        "-f",
        "rawvideo",
        "-pix_fmt",
        "yuv420p"
      ],
      "record": [
        "-f",
        "segment",
        "-segment_time",
        "10",
        "-segment_format",
        "mp4",
        "-reset_timestamps",
        "1",
        "-strftime",
        "1",
        "-c",
        "copy",
        "-an"
      ],
      "rtmp": [
        "-c",
        "copy",
        "-f",
        "flv"
      ]
    }
  },
  "live": {
    "height": 720,
    "quality": 8
  },
  "logger": {
    "default": "info",
    "logs": {}
  },
  "model": {
    "height": 320,
    "labelmap": {},
    "labelmap_path": null,
    "path": null,
    "width": 320
  },
  "motion": null,
  "mqtt": {
    "client_id": "haos-frigate",
    "host": "haos-frigate.local",
    "password": "xxxxxxxx",
    "port": 1883,
    "stats_interval": 60,
    "tls_ca_certs": null,
    "tls_client_cert": null,
    "tls_client_key": null,
    "tls_insecure": null,
    "topic_prefix": "haos-frigate",
    "user": "admin"
  },
  "objects": {
    "filters": null,
    "mask": "",
    "track": [
      "person"
    ]
  },
  "record": {
    "enabled": true,
    "events": {
      "max_seconds": 240,
      "objects": [
        "person"
      ],
      "post_capture": 4,
      "pre_capture": 4,
      "required_zones": [],
      "retain": {
        "default": 8,
        "mode": "motion",
        "objects": {
          "person": 8
        }
      }
    },
    "expire_interval": 60,
    "retain": {
      "days": 0,
      "mode": "all"
    },
    "retain_days": null
  },
  "rtmp": {
    "enabled": true
  },
  "snapshots": {
    "bounding_box": false,
    "clean_copy": true,
    "crop": false,
    "enabled": true,
    "height": 175,
    "quality": 70,
    "required_zones": [],
    "retain": {
      "default": 8,
      "mode": "motion",
      "objects": {
        "person": 8
      }
    },
    "timestamp": true
  },
  "timestamp_style": {
    "color": {
      "blue": 255,
      "green": 255,
      "red": 255
    },
    "effect": null,
    "format": "%m/%d/%Y %H:%M:%S",
    "position": "tl",
    "thickness": 2
  }
}

Operating system

Proxmox

Install method

HassOS Addon

Coral version

CPU (no coral)

Network connection

Wired

Camera make and model

Sonoff

Any other information that may be helpful

All logs Proxmox, HA, Frigate are clean. No errors.

image

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 23

Most upvoted comments

Issue persists (Sep 2022) but unfortunately no solution found so far. The same freezing issue is experienced with any NVR running under Proxmox either as KVM or LXC. Tested NVRs are: AgentDVR Frigate ZoneMinder