alloy: loki.source.kubernetes_events fails to start with large number of events in a namespace ("event watcher exited with error")

What’s wrong?

We are using grafana-agent to ship Events off cluster and we are using a system that has a crazy pod churn.

The component worked fine initially, but as our churn increased it just stopped working after startup with:

“event watcher exited with error”

It is timing out trying to start the informer.

In the code you have this comment:

https://github.com/grafana/agent/blob/main/component/loki/source/kubernetes_events/kubernetes_events.go#L25-L26

I think for our use case it is not generous enough, becasue the initial sync is too heavy.

I would be cool if we could just start with new events or something and ignore the backlog, if that is at all possible.

Steps to reproduce

Use the loki.source.kubernetes_events in a cluster with a very large amount of events.

Observe timeout error.

System information

GKE 1.27 and RKE2 1.26

Software version

docker.io/grafana/agent:v0.39.1

Configuration

logging {
      level = "debug"
    }

    //
    // Ship Kubernetes Events
    //
    loki.source.kubernetes_events "all" {
      forward_to = [loki.process.events.receiver]
      job_name = "integrations/kubernetes/eventhandler"
      // log_format = "logfmt"
      // namespaces = ["xyz"]
    }

    loki.process "events" {
      forward_to = [loki.write.central.receiver]

      stage.match {
        selector = "{namespace=~\"xzy\"}"

        stage.tenant {
          label = "namespace"
        }
      }

      stage.metrics {
        metric.counter {
          name        = "events_total"
          description = "total number of events"
          prefix      = "agent_"

          match_all         = true
          action            = "inc"
          max_idle_duration = "5m"
        }

        metric.counter {
          name        = "event_bytes_total"
          description = "total bytes of events"
          prefix      = "agent_"

          match_all         = true
          count_entry_bytes = true
          action            = "add"
          max_idle_duration = "5m"
        }
      }
    }

    loki.write "central" {
      endpoint {
        url = "https://xyz/loki/api/v1/push"
        tenant_id = "main"
        basic_auth {
          username = "loki"
          password = "xyz"
        }
      }
      external_labels = {
        cluster = "xyz",
      }
    }

Logs

ts=2024-02-25T20:35:47.698339436Z level=error msg="event watcher exited with error" component=loki.source.kubernetes_events.all err="failed to configure informers: timeout exceeded while configuring informers. Check the connection to the Kubernetes API is stable and that the Agent has appropriate RBAC permissions for &Event{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:,Namespace:,Name:,UID:,APIVersion:,ResourceVersion:,FieldPath:,},Reason:,Message:,Source:EventSource{Component:,Host:,},FirstTimestamp:0001-01-01 00:00:00 +0000 UTC,LastTimestamp:0001-01-01 00:00:00 +0000 UTC,Count:0,Type:,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:,ReportingInstance:,}" ts=2024-02-25T20:35:47.698440294Z level=info msg="stopping watcher for events"

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 4
  • Comments: 18 (13 by maintainers)

Most upvoted comments

@rfratto: in terms of release expectations, do you see this as being a candidate for v1.2.0?

I think the window for v1.2.0 closed a while ago. This is either a candidate for v1.2.1 after we have a slightly bigger pile of bug fixes, or v1.3.0 at the worst case.