salt: GitFS cause extra high CPU usage on the master

Description of Issue/Question

Over the last 6-12 months we’ve found state.apply to be very slow and causing very high CPU usage on our salt master. We have about 20 formulas, one external state tree with the master top file and one external pillar, each in separate Git repos.

While debugging I decided to disable the git fileserver in the master configuration and go back to using the basic roots file system, creating a CRON job to simulate gitfs by running git pull every 30 seconds.

This small change has made state.apply over 10x faster than it was before and has lowered CPU usage on the salt master so much that we’ve been able to reduce it from a 6vCPU 8GB instance to a 2vCPU 4GB instance and still achieve a lower average CPU usage than before.

Here is the before and after CPU usage on the salt master: