viseron: High CPU utilization in postgres
After switching to 3.0.0, the CPU utilization was around 20%, only a little bit higher than before; however, when I switched from using the old recorder retention config, to the new storage component, the CPU utilization went up considerably, and wants to use all the cores in the machine (4 cores). Continuous recording is disabled in all cameras. In reality, I can’t be sure if it happened right after the config switch or simply was already getting worse and by the time I looked again after the config switch, it was already pretty high. I only have metrics for the last seven days.
I looked into what exactly wants to use that much CPU, and it is postgresql. I did some profiling with perf, and it appears that the CPU is being utilized for table scanning and date/time parsing.
I’m still digging into it, but perhaps there is a SQL query that runs continually that it is missing an index?
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 2
- Comments: 31 (14 by maintainers)
Short update, i have a PoC working atm that is using numpy instead of the SQL query. Seems to perform better and gives me a bit more control over the resource usage than using SQL.
Not quite ready to merge yet tho, need a few more weeks to do proper testing
Made some more changes, they are in the current
devimage.File handling is now delegated to the subprocess to spread the load, and you can tune the performance by playing around with the new config options
tier_check_cpu_limit,tier_check_batch_size,tier_check_sleep_between_batchesandtier_check_workersIf you see the Viseron proces spiking, you should increase
tier_check_sleep_between_batchesand decreasetier_check_batch_sizeI have some more ideas how to make it even better but i will leave it like it is for now to make room for some other development, unless something comes up.
Closing this issue since the main issue has been addressed
When you restart Viseron, a check will be triggered, and since you had a high check interval there were probably a lot of files to move. The handling of the files is still done in the main process.
Another step could be to try to move the handling as well to a separate throttled process.
But I am glad that it seems to make some difference at least!
I just realized now that the builds are failing for some reason so you are probably still on the old version (unless you pull the
roflcoopter/amd64-viseron:devorroflcoopter/amd64-cuda-viseron:devimages which built fine)The
devtag now contains a change which i hope addresses this.CPU usage is decreased, but memory usage is slightly increased since everything has to be read into memory. You can also to some extent control how much CPU is allocated to this process by using the new
tier_check_cpu_limitconfig option.There could potentially be issues if you store a very very high number of files.
Please try it out when you get the chance and let me know your findings!