alloy: loki.source.kubernetes_events fails to start with large number of events in a namespace ("event watcher exited with error")
What’s wrong?
We are using grafana-agent to ship Events off cluster and we are using a system that has a crazy pod churn.
The component worked fine initially, but as our churn increased it just stopped working after startup with:
“event watcher exited with error”
It is timing out trying to start the informer.
In the code you have this comment:
I think for our use case it is not generous enough, becasue the initial sync is too heavy.
I would be cool if we could just start with new events or something and ignore the backlog, if that is at all possible.
Steps to reproduce
Use the loki.source.kubernetes_events in a cluster with a very large amount of events.
Observe timeout error.
System information
GKE 1.27 and RKE2 1.26
Software version
docker.io/grafana/agent:v0.39.1
Configuration
logging {
level = "debug"
}
//
// Ship Kubernetes Events
//
loki.source.kubernetes_events "all" {
forward_to = [loki.process.events.receiver]
job_name = "integrations/kubernetes/eventhandler"
// log_format = "logfmt"
// namespaces = ["xyz"]
}
loki.process "events" {
forward_to = [loki.write.central.receiver]
stage.match {
selector = "{namespace=~\"xzy\"}"
stage.tenant {
label = "namespace"
}
}
stage.metrics {
metric.counter {
name = "events_total"
description = "total number of events"
prefix = "agent_"
match_all = true
action = "inc"
max_idle_duration = "5m"
}
metric.counter {
name = "event_bytes_total"
description = "total bytes of events"
prefix = "agent_"
match_all = true
count_entry_bytes = true
action = "add"
max_idle_duration = "5m"
}
}
}
loki.write "central" {
endpoint {
url = "https://xyz/loki/api/v1/push"
tenant_id = "main"
basic_auth {
username = "loki"
password = "xyz"
}
}
external_labels = {
cluster = "xyz",
}
}
Logs
ts=2024-02-25T20:35:47.698339436Z level=error msg="event watcher exited with error" component=loki.source.kubernetes_events.all err="failed to configure informers: timeout exceeded while configuring informers. Check the connection to the Kubernetes API is stable and that the Agent has appropriate RBAC permissions for &Event{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:,Namespace:,Name:,UID:,APIVersion:,ResourceVersion:,FieldPath:,},Reason:,Message:,Source:EventSource{Component:,Host:,},FirstTimestamp:0001-01-01 00:00:00 +0000 UTC,LastTimestamp:0001-01-01 00:00:00 +0000 UTC,Count:0,Type:,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:,ReportingInstance:,}"
ts=2024-02-25T20:35:47.698440294Z level=info msg="stopping watcher for events"
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 4
- Comments: 18 (13 by maintainers)
I think the window for v1.2.0 closed a while ago. This is either a candidate for v1.2.1 after we have a slightly bigger pile of bug fixes, or v1.3.0 at the worst case.