alloy: loki.source.windowsevent - not able to handle more than 5000 events per minute "in time" and starts to lag behind
What’s wrong?
Wer are running a windows server 2022 and on this server alloy 1.5.1. This windows Server receives Windows Eventlos via Eventlog Forwarding. These logs are stored in “ForwardedEvents” Event Log.
We receive between 4.000 - 8.000 events per minute. Grafana alloy 1.5.1 loki.source.windowsevent is not able to collect these logs in time. If the amount of logs is lower than I can see the logs in loki only a few seconds later. If the amount of logs per minute increases then alloy falls behind. could be 5 minutes and could be 20 minutes.
I checked the Windows Server ressources but they are all fine. I checked the alloy process CPU usage vi prometheus.exporter.windows and the CPU usage is always around 75%. so it is not saturation 1 CPU completely and not using more CPUs which would be available.
I have another tool on this windows server which is reading these ForwardedEvents Logs and this tool is using around 20% and has all the logs within seconds.
So there is maybe something in the loki.source.windowsevent code which should be optimized. maybe newer versions of windows allow other/faster APIs than what is used in alloy.
I also tried to reduce the “ForwardedEvents” file size from 8GB to 1GB and 256MB and 512MB. It looks like the bigger the file the harder it is for alloy to keep up.
In addition if I scrape these forwardedEvents the alloy /-/reload endpoint is not working anymore. The browser can not reload the config. I have to do a “restart” via Windows Services.
Steps to reproduce
- Try to collect ForwardedEvents with more than 5k events per minute or better more.
- Try to reload config via /-/reload endpoint
- try to get logs in loki in “real time”
- check alloy CPU usage
System information
Windows Server 2022
Software version
Grafana Alloy 1.5.1
Configuration
Alloy config:
loki.source.windowsevent "forwardedevents" {
eventlog_name = "ForwardedEvents"
use_incoming_timestamp = true
locale = 1033
poll_interval = "250ms"
labels = {
"service_name" = "windows_forwardedevents",
}
forward_to = [loki.relabel.windows_event_level.receiver]
}
Logs
N/A
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 39 (11 by maintainers)
Should we move the conversation there and close this issue?
I created a new ticket to further improve the performance of this component: https://github.com/grafana/alloy/issues/2615. I will try to work on it soon and contact you if I find some ways to improve it
Hello,
4634 in my local Security log works. However 4634 events from forwarded events do not. I don’t think it is an issue with this type of event itself because I had the crash with other events, too (did not find the ID) but I think there is something in the message, which is maybe specific for our environment and this causes the crash.
maybe it is with a further additional processing step to format the logs. maybe I will try to check this.
Ok that’s good to know, at least we can treat the improvement and this bug separately. Next step is to find the event that causes the crash so that I can try to reproduce it on my machine. What if you have only one event with the task category set to a number and you try to ingest it, does it crash?
@Nachtfalkeaw did the crashes occure with an older version of alloy, too?
At the beginning I forwarded nearly all logs (application, system, security) from all servers to the eventlog collectors. I started to narrow down the forwarded events because alloy was not able to keep up with eventlog. The only crashes I experienced occured with errors in alloy’s configuration.
I have set up several subscriptions with event id filters for account, security, etc. on the collectors (source initiated). And a group policy to configure servers and clients to push their logs to the collectors.
Hey!
Alloy kept up with both our log collectors the whole day yesterday. This is a big improvment! Grafana alarms are finally usable 😃 I now changed the eventlogsize of both servers to 10 GB (10485760kb).
One server had 40GB before. I stopped alloy, deleted the log, reduced its size and started alloy again. It is running fine since. The other server had 10GB before. I just increased the log size without a restart of the service.
I only collect logs with alloy, no metrics since we already had another tool set up to do this.
Hey @Nachtfalkeaw, sorry I thought I replied to the comment already but I guess I somehow forgot to press enter. Thanks for the testing, I agree that the evidence that you shared indicates a good CPU performance improvement.
But the panic is worrying. I checked the logs that you sent me. They are very helpful but the root cause of the panic is still unclear to me, it happens deep in Windows system calls.
I have a few questions to further debug it:
Regarding the profiling not working:
Hey! I am OK with testing the fix but to see if it improves performance I have to run it in our productive environment from where I cannot share dumps with a clear conscience. Our testing environment does not have enough servers and users to generate the required amount of logs. Windows installer would be best for me.