frigate: [Support]: hwaccel stopped working on raspberry pi after 5.15.61 kernel update

Describe the problem you are having

Everything was working fine with my frigate setup. I’ve been using the 0.11 betas/rc’s since the beginning. All of the sudden one of my cameras stopped working in frigate completely. Blue Iris was still working fine with it. VLC opened the configured URL no problem. I’ve issues once in a while in the past where I’d see events stop happening so I’d just bounce frigate and all would be well. This time, bouncing frigate and the rebooting the camera hasn’t helped.

The really strange thing is that I can’t think of anything that has changed. IIRC, the camera was working fine with 0.11RC2 for at least a few days before this randomly started happening. No matter what I tried I just get repeating No frames received from driveway in 20 seconds errors. I do have another camera that is the same brand but a little bit older model that continued to work fine. I removed it from the config since the issue with the one camera still occurs.

Version

0.11.0-c461c9e

Frigate config file

mqtt:
  host: mqtt.home
  user: 'XXXX'
  password: 'XXXXXXX'

database:
  path: /db/frigate.db

detectors:
  coral:
    type: edgetpu
    device: usb

ffmpeg:
  hwaccel_args:
    - -c:v
    - h264_v4l2m2m

record:
  enabled: True
  events:
    retain:
      default: 30

snapshots:
  enabled: True
  retain:
    default: 30

rtmp:
  enabled: False

cameras:
  driveway:
    ffmpeg:
      inputs:
        - path: rtsp://XXXXX:XXXXXX@driveway.cam.home:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - rtmp
            - record
        - path: rtsp://XXXXXX:XXXXXXX@driveway.cam.home:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect
    detect:
      width: 1280
      height: 720
      fps: 5
    motion:
      mask:
        - 1230,441,1252,310,1259,120,1257,0,1280,0,1280,720,1130,720
        - 396,116,649,67,824,57,1002,70,1134,94,1259,122,1257,0,949,0,819,0,626,0,0,0,0,260
    zones:
      driveway_entry:
        coordinates: 819,248,639,209,1152,99,1202,124
      driveway_main:
        coordinates: 1095,720,332,720,244,560,174,308,425,217,476,233,528,225,630,214,821,266,867,305,1207,433
    objects:
      track:
        - person
        - car
        - dog

Relevant log output

frigate  | [2022-09-07 02:01:17] frigate.video                  ERROR   : driveway: Unable to read frames from ffmpeg process.                                                       
frigate  | [2022-09-07 02:01:17] frigate.video                  ERROR   : driveway: ffmpeg process is not running. exiting capture thread...                                         
frigate  | [2022-09-07 02:01:27] watchdog.driveway              ERROR   : Ffmpeg process crashed unexpectedly for driveway.                                                          
frigate  | [2022-09-07 02:01:27] watchdog.driveway              ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.                                        
frigate  | [2022-09-07 02:01:45] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:53484]                                
frigate  | [2022-09-07 02:01:47] watchdog.driveway              INFO    : No frames received from driveway in 20 seconds. Exiting ffmpeg...                                          
frigate  | [2022-09-07 02:01:47] watchdog.driveway              INFO    : Waiting for ffmpeg to exit gracefully...                                                                   
frigate  | [2022-09-07 02:01:50] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:51066]                                   
frigate  | [2022-09-07 02:02:17] watchdog.driveway              INFO    : FFmpeg didnt exit. Force killing...
frigate  | [2022-09-07 02:02:17] frigate.video                  ERROR   : driveway: Unable to read frames from ffmpeg process.
frigate  | [2022-09-07 02:02:17] frigate.video                  ERROR   : driveway: ffmpeg process is not running. exiting capture thread...
frigate  | [2022-09-07 02:02:27] watchdog.driveway              ERROR   : Ffmpeg process crashed unexpectedly for driveway.
frigate  | [2022-09-07 02:02:27] watchdog.driveway              ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
frigate  | [2022-09-07 02:02:37] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:56424]
frigate  | [2022-09-07 02:02:47] watchdog.driveway              INFO    : No frames received from driveway in 20 seconds. Exiting ffmpeg...
frigate  | [2022-09-07 02:02:47] watchdog.driveway              INFO    : Waiting for ffmpeg to exit gracefully...
frigate  | [2022-09-07 02:02:50] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:51066]
frigate  | [2022-09-07 02:02:55] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:45176]
frigate  | [2022-09-07 02:03:17] watchdog.driveway              INFO    : FFmpeg didnt exit. Force killing...
frigate  | [2022-09-07 02:03:17] frigate.video                  ERROR   : driveway: Unable to read frames from ffmpeg process.
frigate  | [2022-09-07 02:03:17] frigate.video                  ERROR   : driveway: Unable to read frames from ffmpeg process.
frigate  | [2022-09-07 02:03:17] frigate.video                  ERROR   : driveway: Unable to read frames from ffmpeg process.
frigate  | [2022-09-07 02:03:17] frigate.video                  ERROR   : driveway: Unable to read frames from ffmpeg process.
frigate  | [2022-09-07 02:03:17] frigate.video                  ERROR   : driveway: ffmpeg process is not running. exiting capture thread...
frigate  | [2022-09-07 02:03:27] watchdog.driveway              ERROR   : Ffmpeg process crashed unexpectedly for driveway.
frigate  | [2022-09-07 02:03:27] watchdog.driveway              ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
frigate  | [2022-09-07 02:03:47] watchdog.driveway              INFO    : No frames received from driveway in 20 seconds. Exiting ffmpeg...
frigate  | [2022-09-07 02:03:47] watchdog.driveway              INFO    : Waiting for ffmpeg to exit gracefully...
frigate  | [2022-09-07 02:03:55] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:45176]
frigate  | [2022-09-07 02:04:00] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:51724]
frigate  | [2022-09-07 02:04:17] watchdog.driveway              INFO    : FFmpeg didnt exit. Force killing...
frigate  | [2022-09-07 02:04:17] frigate.video                  ERROR   : driveway: Unable to read frames from ffmpeg process.
frigate  | [2022-09-07 02:04:17] frigate.video                  ERROR   : driveway: Unable to read frames from ffmpeg process.
frigate  | [2022-09-07 02:04:17] frigate.video                  ERROR   : driveway: ffmpeg process is not running. exiting capture thread...
frigate  | [2022-09-07 02:04:27] watchdog.driveway              ERROR   : Ffmpeg process crashed unexpectedly for driveway.
frigate  | [2022-09-07 02:04:27] watchdog.driveway              ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
frigate  | [2022-09-07 02:04:47] watchdog.driveway              INFO    : No frames received from driveway in 20 seconds. Exiting ffmpeg...
frigate  | [2022-09-07 02:04:47] watchdog.driveway              INFO    : Waiting for ffmpeg to exit gracefully...
frigate  | [2022-09-07 02:05:00] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:51724]
frigate  | [2022-09-07 02:05:05] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:58226]
frigate  | [2022-09-07 02:05:17] watchdog.driveway              INFO    : FFmpeg didnt exit. Force killing...
frigate  | [2022-09-07 02:05:17] frigate.video                  ERROR   : driveway: Unable to read frames from ffmpeg process.
frigate  | [2022-09-07 02:05:17] frigate.video                  ERROR   : driveway: ffmpeg process is not running. exiting capture thread...
frigate  | [2022-09-07 02:05:27] watchdog.driveway              ERROR   : Ffmpeg process crashed unexpectedly for driveway.
frigate  | [2022-09-07 02:05:27] watchdog.driveway              ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
frigate  | [2022-09-07 02:05:47] watchdog.driveway              INFO    : No frames received from driveway in 20 seconds. Exiting ffmpeg...
frigate  | [2022-09-07 02:05:47] watchdog.driveway              INFO    : Waiting for ffmpeg to exit gracefully...

FFprobe output from your camera

root@edcaa9ad0fcf:/opt/frigate# ffprobe "rtsp://xxxxx:xxxxx@driveway.cam.home:554/cam/realmonitor?channel=1&subtype=0"
ffprobe version n5.1-2-g915ef932a3-20220731 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 12.1.0 (crosstool-NG 1.25.0.55_3defb7b)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=aarch64-ffbuild-linux-gnu- --arch=aarch64 --target-os=linux --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-libdav1d --disable-libdavs2 --disable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-mbedtls --enable-librist --enable-libtheora --enable-libvpx --enable-libwebp --enable-lv2 --disable-libmfx --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --disable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --disable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags=-pie --extra-libs='-ldl -lgomp' --extra-version=20220731
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, rtsp, from 'rtsp://xxxxx:xxxxx@driveway.cam.home:554/cam/realmonitor?channel=1&subtype=0':
  Metadata:
    title           : Media Server
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: h264 (Main), yuvj420p(pc, bt470bg/bt470bg/bt709, progressive), 2688x1520, 10 fps, 10 tbr, 90k tbn
  Stream #0:1: Audio: pcm_alaw, 8000 Hz, mono, s16, 64 kb/s

Frigate stats

{
  "detection_fps": 0,
  "detectors": {
    "coral": {
      "detection_start": 0,
      "inference_speed": 10,
      "pid": 215
    }
  },
  "driveway": {
    "camera_fps": 0,
    "capture_pid": 227,
    "detection_fps": 0,
    "pid": 221,
    "process_fps": 0,
    "skipped_fps": 0
  },
  "service": {
    "latest_version": "0.10.1",
    "storage": {
      "/dev/shm": {
        "free": 133.9,
        "mount_type": "tmpfs",
        "total": 134.2,
        "used": 0.3
      },
      "/media/frigate/clips": {
        "free": 9131015.5,
        "mount_type": "cifs",
        "total": 19859668.5,
        "used": 10728652.9
      },
      "/media/frigate/recordings": {
        "free": 9131015.5,
        "mount_type": "cifs",
        "total": 19859668.5,
        "used": 10728652.9
      },
      "/tmp/cache": {
        "free": 954,
        "mount_type": "tmpfs",
        "total": 1000,
        "used": 46
      }
    },
    "temperatures": {},
    "uptime": 1041,
    "version": "0.11.0-c461c9e"
  }
}

Operating system

Debian

Install method

Docker Compose

Coral version

USB

Network connection

Wired

Camera make and model

Dahua IPC-T5442T-ZE

Any other information that may be helpful

No response

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 9
Comments: 173 (1 by maintainers)

Links to this issue

Frigate - a few questions, if noone objects

Most upvoted comments

A fix is already in progress https://github.com/blakeblackshear/frigate/pull/4955

+15

NickM-27 on Jan 7, 2023

OK, submitted via https://trac.ffmpeg.org/ticket/10060.

jherby2k on Nov 21, 2022

https://github.com/blakeblackshear/frigate/pull/4726

NickM-27 on Dec 17, 2022

I plan to test more tonight to confirm all is good then will share whatever I remember of my instructions. I’m learning as I go so things got pretty messy, but I do at least have my config file.

I haven’t noticed a dramatic reduction in CPU, but then again I’m using the second stream at D1 resolution.

idontcare99999 on Oct 24, 2022

I’m experiencing this issue as well since updating the PI kernel and trying to figure out a solution. My understanding is that:

The kernel update broke things, but the issue is with FFmpeg
The fix is in for the 4.3.4 version of FFmpeg packaged with the PI, but hasn’t been released yet, though this won’t solve the issue because Frigate uses a different version of FFmpeg inside the Docker container
Frigate is using a BtbN build of FFmpeg 5.1 which appears to be auto-built from the official FFmpeg repo

After looking at the fix that @idontcare99999 mentioned, I checked the official FFmpeg repo and it doesn’t appear to contain the fix. Has anyone else managed to get hwaccel working again other than by downgrading to the previous kernel (which I’m not opposed to but have never done before)?

joe248 on Sep 24, 2022

We’ve seen other users complain that the latest RPi kernel broke hwaccel as well

NickM-27 on Sep 7, 2022

If you’re happy to do this, drop the HAOS version back to 8.5 and this will restore hardware decoding:

ha os update --version 8.5

As it happens, I’ve just done this myself as I suspect the real fix might take quite a while to propagate into ffmpeg. No problems at all - one command, and it took around 10 mins to install and reboot, with no obvious negative side effects. Note: this does not affect the version of HA you are running, just the underlying host OS.

daernsinstantfortress on Dec 13, 2022

Recommend moving addon build to another discussion so as to not crowd the discussion on the RPI issue itself 👍

NickM-27 on Nov 7, 2022

Here’s my latest compile.

It works for both 264 and 265 but h264_v4l2m2m fails for me after running for a bit.

https://www.dropbox.com/s/chxy90fpgif6rlq/ffmpeg.1104.gz?dl=0

Have fun!

idontcare99999 on Nov 5, 2022

I used the following process to downgrade the kernel from 5.15.61-v8+ to 5.15.32-v8+ and hwaccel is working again:

Check /var/log/apt to find last kernel date version before upgrade. For me it was 1.20220331-1.
Check https://github.com/raspberrypi/rpi-firmware/commits/master to find commit hash for that date (a54fe46c85fd4a2155f2282454bee3c2a3d5b5eb)
Power down the PI, put the SD card into a computer, and create a backup image. For Mac OS I used: sudo dd bs=4m if=/dev/disk2 of=raspbian.img
Put the SD card back into the PI and boot. Run sudo rpi-update a54fe46c85fd4a2155f2282454bee3c2a3d5b5eb
Reboot

joe248 on Oct 3, 2022

I don’t know much about ffmpeg and hardware acceleration but I searched the raspberry pi kernel issues for the kernel version I just updated to (5.15.61-v8+) and found a couple recent issues that may be related: https://github.com/raspberrypi/linux/issues/5150 https://github.com/raspberrypi/linux/issues/5166

kylehendricks on Sep 7, 2022

rpi-update 109147b5dd3cdb736173be079efc03950c1d20b3 just a word of warning for the next person. This was destructive for me, it made my RPi 4B un-bootable from SSD. I would live without hardware acceleration if I had known the massive pain this would cause.

rjpearce on Apr 8, 2023

Does this issue still exist in Frigate 11.1? My RPI4 kernel is 6.19-v8+, default for the Raspberry PI OS Install currently

Yes, 0.11 does not contain a fix

NickM-27 on Mar 24, 2023

Feel free to upvote the ticket here for attention and to hopefully get some resources assigned to it: https://trac.ffmpeg.org/ticket/10060

I have created an account a month ago just to do that but the upvote button does nothing … maybe the account has to have a certain type or privileges. Was able to comment though.

Right click it and copy link address and whack it in a new tab. Done the trick for me.

LaurenceGough on Jan 3, 2023

You’re right it will be a very useful information for all the new users of Frigate, I just don’t know how to do that but I can’t agree more.

Clooos on Dec 17, 2022

Hi, I can say that I had the same issue on HAOS and even with the previous builds of Frigate, I found somewhere that downgrading the OS to the 6.2 build is the best so far (it has the previous kernel version that is working with Frigate), “hwaccel_args: h264_v4l2m2m” is now working and my Pi 4 CPU feel way better.

I still can’t wait for a fix for security reasons and because the new build of Home Assistant (2022.11) is still working with it but I have now this error “Unsupported system - Systemd Journal issues” and they say that the fix is to update the OS.

Thank you @NickM-27 for your work on that issue!

Clooos on Nov 24, 2022

Currently frigate installs ffmpeg for 64bit pi from btbn https://github.com/BtbN/FFmpeg-Builds

NickM-27 on Nov 21, 2022

Yes, if you check the original ffmpeg directory it includes ffmpeg, ffprobe, and ffplay. The directory is being overwritten with just ffmpeg which causes the issue

Ok i have copy missing from “normal” container and add files to my custom ffmpeg next destroy and create container with custom ffmpeg and no more error ^^

Thanks all for you patience, your work, and time spend on my issue !!

todely on Nov 10, 2022

Yes, if you check the original ffmpeg directory it includes ffmpeg, ffprobe, and ffplay. The directory is being overwritten with just ffmpeg which causes the issue

NickM-27 on Nov 10, 2022

Here’s the 4.3.5 version for your testing pleasure.

idontcare99999 on Nov 8, 2022

Here’s my latest compile. It works for both 264 and 265 but h264_v4l2m2m fails for me after running for a bit. https://www.dropbox.com/s/chxy90fpgif6rlq/ffmpeg.1104.gz?dl=0 Have fun!

Thanks so much @idontcare99999 , I now appear to have cameras working with hwaccel enabled in the config although CPU usage is still just as high which is odd as it was about half as much before when hwaccel was working from what I remember. In portainer I made the following map: /mnt/pi_userdata/docker-data/volumes/Frigate/custom-ffmpeg/bin/ffmpeg : /usr/lib/btbn-ffmpeg/bin/ffmpeg

Yes, I opened this issue offering to help troubleshoot the CPU usage https://github.com/jc-kynesim/rpi-ffmpeg/issues/54

Thank you, I thought I was going crazy. So it seems no benefit for h264 users but it brings back hardware accel for h265 users?

Thanks for your hard work still. Shame but I don’t think my cameras support h265 for the substreams and one of them not at all.

How come there is such little activity on this, I would have thought with the millions of Raspberry Pi’s in the world there would be a hive of activity on this. Is it due to the particular way Frigate is accessing streams? Home assistant has over 100,000 installs I guess quite a few of them are on Pi’s.

I’m confused, but that doesn’t take much to be fair.

I’m still thinking of rolling back the kernel upgrade that caused all this but I want a bullet proof backup first.

I’m trying to plow forward.

Here’s my latest compile of the 4.3.4 build that provides better h264_v4l2m2m performance but you must revert back to “stimeout” like this:

ffmpeg: input_args: -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 hwaccel_args: -c:v h264_v4l2m2m

Note: I did not patch flv for h265 on this version - just use the latest 5.1.2.

https://www.dropbox.com/s/as0d4zbqi2b4vtz/ffmpeg.1105.gz?dl=0

idontcare99999 on Nov 5, 2022

Here’s my latest compile. It works for both 264 and 265 but h264_v4l2m2m fails for me after running for a bit. https://www.dropbox.com/s/chxy90fpgif6rlq/ffmpeg.1104.gz?dl=0 Have fun!

Thanks so much @idontcare99999 , I now appear to have cameras working with hwaccel enabled in the config although CPU usage is still just as high which is odd as it was about half as much before when hwaccel was working from what I remember.

In portainer I made the following map:

/mnt/pi_userdata/docker-data/volumes/Frigate/custom-ffmpeg/bin/ffmpeg : /usr/lib/btbn-ffmpeg/bin/ffmpeg

Yes, I opened this issue offering to help troubleshoot the CPU usage

https://github.com/jc-kynesim/rpi-ffmpeg/issues/54

idontcare99999 on Nov 5, 2022

Thanks, i’m running it now! So far stable at h264 (my cameras don’t do h265). However I’m not sure there is any advantage to the “acceleration”. Either its not actually running an optimized path, or its just completely underwhelming. I’m at 50% CPU running 3 cameras at 960x576 resolution. Looks like HD isn’t in the cards for me.

It’s important to note that lower resolutions don’t always benefit from hwaccel as there is more cost in moving the data between cpu and GPU memory. Meaning if you bump your resolution it may be only marginal increase in CPU usage but without hwaccel it would be much higher. Also if you’re detect -> width / height doesn’t match the camera streams native resolution then that will be a hit on the CPU

I’d love to not have to run through 5 steps every time i restart… anyone know how to persist changes to the container?

If you have the static build on your host then you can use https://docs.frigate.video/configuration/advanced#custom-ffmpeg-build to overwrite the default ffmpeg

NickM-27 on Nov 5, 2022

I was able to compile the 5.1.2/rpi_import_1 branch according to the build instructions. Unfortunately it didn’t fix the issue for me, but posting the (arm64) binary in case someone else wants to give it a shot: ffmpeg.zip

Is it ok for hassio? Where I have to put this file?

I used Docker to substitute my binary for the included one (instructions are in the documentation).

roger- on Oct 31, 2022

That’s awesome! I’ve very eager to test this… can you make your binary available and/or provide instructions?

jherby2k on Oct 24, 2022

and the fix is in… https://github.com/raspberrypi/linux/issues/5150#issuecomment-1247922065

idontcare99999 on Sep 16, 2022

Yeah same issue here sadly. 😦 Turning off hw Accel works for now but the poor thing is deffo working harder even with just two cameras setup.

It happens with rc1 and rc2 for me. I’m using two different hikvision cameras sadly neither work. H264.

If I find a better fix I’ll be sure to share

LaurenceGough on Sep 8, 2022