frigate: [Support]: Frigate fails to respond to /vod/event/xxx/master.m3u8 requests (503 Service Temporarily Unavailable)

Describe the problem you are having

When trying to view recordings (e.g. /vod/event/xxx/master.m3u8) the player is black with the text overlayed: The media could not be loaded, either because the server or network failed or because the format is not supported.

frigate

This worked fined until recently.

Watching the network tab in Chrome devtools shows the request being responded with

<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->

Everything else is working without problem; /live/driveway, /api/driveway/latest.jpg, /api/events/xxx/thumbnail.jpg

The only error I can see related to frigate is:

This error originated from a custom integration.

Logger: custom_components.frigate.api
Source: custom_components/frigate/api.py:51
Integration: Frigate (documentation, issues)
First occurred: October 31, 2022 at 9:36:08 PM (514 occurrences)
Last logged: 10:13:56 AM

Error fetching information from http://ccab4aaf-frigate-fa:5000/api/stats: Cannot connect to host ccab4aaf-frigate-fa:5000 ssl:default [Name does not resolve]
Error fetching information from http://ccab4aaf-frigate-fa:5000/api/stats: 502, message='Bad Gateway', url=URL('http://ccab4aaf-frigate-fa:5000/api/stats')
Timeout error fetching information from http://ccab4aaf-frigate-fa:5000/api/stats:
Error fetching information from http://ccab4aaf-frigate-fa:5000/api/stats: Cannot connect to host ccab4aaf-frigate-fa:5000 ssl:default [Connect call failed ('172.30.33.3', 5000)]
Error fetching information from http://ccab4aaf-frigate-fa:5000/api/stats: Server disconnected

ccab4aaf-frigate-fa is the correct hostname being shown in the addons view at /hassio/addon/ccab4aaf_frigate-fa/info

I can watch the individual recordings fine from Home Assistants “Local Media” folder. Using the “Frigate” media folder does not work. Snapshots shows up fine but clips and recording crashes with a “503 Service Temporarily Unavailable, nginx/1.18.0”-response

Where can I troubleshoot this next?

Version

0.11.1-2EADA21

Frigate config file

mqtt:
  host: core-mosquitto
  user: ***
  password: ***
  
detectors:
  coral:
    type: edgetpu
    device: usb

logger:
  default: info
  logs:
    frigate.app: info
    frigate.video: info

cameras:
  driveway:
    ffmpeg:
      inputs:
        - path: rtsp://***@192.168.1.104:554/stream1
          roles:
            - detect
            - rtmp
            - record

    detect:
      width: 2304
      height: 1296
      fps: 5

    objects:
      track:
        - person
      filters:
        person:
          min_area: 8000
          min_score: 0.65

    motion:
      mask:
        - 0,0,0,256,531,202,1083,190,1665,207,1722,0
        - 1742,714,1846,544,1685,363,1424,339,1292,511,1371,702
        - 1186,231,1158,347,1355,430,1485,357,1485,243,1318,164
        - 1101,138,1093,221,1166,256,1253,237,1326,189,1294,122,1182,105
        - 2304,0,1646,0,1628,58,2105,150,2304,197
      threshold: 50

    zones:
      driveway1:
        coordinates: 1568,191,1736,282,1387,300,1274,187
        objects:
          - car
          - person
      driveway2:
        coordinates: 1367,1296,2304,1296,2304,507,1801,276,1438,333
        objects:
          - person
          - car
      lawn:
        coordinates: 1917,1296,0,1296,0,296,1253,160
        objects: 
          - person

    record:
        enabled: True
        retain:
          days: 3
          mode: motion

    snapshots:
        enabled: True

Relevant log output

FRIGATE (FULL ACCESS) Log:
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[2022-11-02 10:13:57] frigate.app                    INFO    : Starting Frigate (0.11.1-2eada21)
[2022-11-02 10:13:57] frigate.app                    INFO    : Creating directory: /tmp/cache
Starting migrations
[2022-11-02 10:13:57] peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
[2022-11-02 10:13:57] peewee_migrate                 INFO    : There is nothing to migrate
[2022-11-02 10:13:57] frigate.app                    INFO    : Output process started: 217
[2022-11-02 10:13:57] ws4py                          INFO    : Using epoll
[2022-11-02 10:13:57] frigate.app                    INFO    : Camera processor started for driveway: 223
[2022-11-02 10:13:57] frigate.app                    INFO    : Capture process started for driveway: 226
[2022-11-02 10:13:57] ws4py                          INFO    : Using epoll
[2022-11-02 10:13:57] detector.coral                 INFO    : Starting detection process: 216
[2022-11-02 10:13:57] frigate.edgetpu                INFO    : Attempting to load TPU as usb
[2022-11-02 10:14:00] frigate.edgetpu                INFO    : TPU found
[2022-11-02 10:14:19] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:39540]
[2022-11-02 10:16:07] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:39540]
[2022-11-02 10:19:13] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:46062]
[2022-11-02 10:19:15] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:46062]
[2022-11-02 10:22:32] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:45834]
[2022-11-02 10:22:47] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:45834]
[2022-11-02 10:22:49] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:58608]
[2022-11-02 10:22:52] ws4py                          INFO    : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:46904]
[2022-11-02 10:23:22] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:46904]
[2022-11-02 10:24:07] ws4py                          INFO    : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:58608]

FFprobe output from your camera

ffprobe version 4.2.7-0ubuntu0.1 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
  configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[rtsp @ 0x5637f0da1a00] UDP timeout, retrying with TCP
[rtsp @ 0x5637f0da1a00] CSeq 7 expected, 6 received.
[rtsp @ 0x5637f0da1a00] Could not find codec parameters for stream 0 (Video: h264, none): unspecified size
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, rtsp, from 'rtsp://****@192.168.1.104:554/stream1':
  Metadata:
    title           : Session streamed by "TP-LINK RTSP Server"
    comment         : stream2
  Duration: N/A, start: 0.000000, bitrate: 64 kb/s
    Stream #0:0: Video: h264, none, 90k tbr, 90k tbn, 180k tbc
    Stream #0:1: Audio: pcm_alaw, 8000 Hz, 1 channels, s16, 64 kb/s

Frigate stats

No response

Operating system

HassOS

Install method

HassOS Addon

Coral version

USB

Network connection

Wired

Camera make and model

Tapo C310

Any other information that may be helpful

No response

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 17 (3 by maintainers)

Most upvoted comments

I was able to remove these records with the following command in the same directory as the frigate db after frigate is stopped:

sqlite3 frigate.db 'delete from recordings where (end_time - start_time) > 1000;'

If you want to see how many records will be deleted first you can run:

sqlite3 frigate.db 'select COUNT(*) from recordings where (end_time - start_time) > 1000;'

ScottRoach on Mar 6, 2023

Makes sense, good to hear! Thanks for your help again, really appreciate it!

joeShuff on Dec 21, 2022

So I’ve taken a dive into the DB and this was what I did which has resulted in successfully saving the majority of my events/recordings!

First I stopped Frigate, waited for it to gracefully close and then copy the DB to my PC and then also make a backup on there.

Opened the db file in dbbrowser and ran the following query in the SQL query tab.

SELECT * FROM recordings where camera="CAMERA_NAME" and end_time > 1671650631 the long number at the end was the epoch time I ran it at, so any segments larger than that technically can’t exist MAKE SURE YOU UPDATE THIS NUMBER TO BE SLIGHTLY IN THE FUTURE SO YOU DON’T DELETE VALID RECORDINGS https://www.epochconverter.com/

I got the following Illegal entries that are probably the culprits! (Take a look at the “end_time” column, they apparently ended at 3333606724, which is in 53 years)

Let’s delete them by slightly modifying that query. (this means those segments won’t exist anymore, so this is some lost data, but according to Nick, the files would be corrupt anyways so no loss really) DELETE FROM recordings where camera="CAMERA_NAME" and end_time> 1671650631

Then click “Write Database” at the top then “Close Database”, now we can try to launch frigate with that DB.

Deleted the old frigate.db and accompanying .shm files. Replace with our new one and run!

It worked! All my existing events now have clips and the recordings page works too!

Thanks a lot @NickM-27!

joeShuff on Dec 22, 2022

No, it will only work going forward. An alternative to deleting the database is just deleting the records for the recording segments which have an incorrect end time.

NickM-27 on Nov 4, 2022

This is an issue that can be caused by bad segment durations, will be fixed in 0.12.0 (when caused by bad segment durations)

NickM-27 on Nov 4, 2022

I don’t know if my database got corrupted. But changing the path for the database got everything working again

database:
  path: /media/frigate/frigate2.db

Perhaps it got corrupted when Home Assistant updated and restarted or something

KevinJCandlert on Nov 4, 2022