prometheus-engine: rule-evaluator doesn't get updated alertmanager pod ipv4s

hello - we’re currently using Managed Prometheus and a self-hosted Alertmanager deployment. This has been functioning properly for over a year. We’re currently on this version of rule-evaluator

gke.gcr.io/prometheus-engine/rule-evaluator:v0.8.1-gke.9

our rule-evaluator sends events to a self-managed Alertmanager statefulset, which lives in a separate namespace. We configure this via the OperatorConfig CRD:

---
apiVersion: monitoring.googleapis.com/v1
kind: OperatorConfig
metadata:
  namespace: gmp-public
  name: config

# https://github.com/GoogleCloudPlatform/prometheus-engine/blob/main/doc/api.md#ruleevaluatorspec
rules:
  alerting:
    alertmanagers:
      # configures where the rule-evaluator will send alerting events to
      - name: alertmanager
        namespace: monitoring
        port: 9093

# https://github.com/GoogleCloudPlatform/prometheus-engine/blob/main/doc/api.md#managedalertmanagerspec
# NOTE: this section is unused, as it points to an empty default Alertmanager configuration file
# since we are using a self-deployed Alertmanager instead of the one provided by GMP
managedAlertmanager:
  configSecret:
    name: alertmanager
    key: alertmanager.yaml

in the last month or so, we’ve noticed that the rule-evaluator will be unable to resolve the downstream Alertmanager address after the Alertmanager pod is rescheduled.

From there, we’ll see the rule-evaluator log this out:

{
alertmanager: "http://10.34.25.18:9093/api/v2/alerts"
caller: "notifier.go:532"
component: "notifier"
count: 1
err: "Post "http://10.34.25.18:9093/api/v2/alerts": context deadline exceeded"
level: "error"
msg: "Error sending alert"
ts: "2024-02-12T18:17:33.124842335Z"
}

this can go on for an hour - we have pages set up to notify us when the rule-evaluator stops pinging Alertmanager through a custom heartbeat rule. The only way to resolve this is by restarting the rule-evaluator deployment

this suggests that the rule-evaluator is not reconciling downstream ip addresses after startup, since we provide the k8s DNS components in the OperatorConfig for the Alertmanager receiver

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 2
  • Comments: 17

Most upvoted comments

Hi @bwplotka , I’m going to configure VPA for the rule evaluator and test it.

Oh wait - I actually think your debug logs from your second comment hold the clue. Specifically:

evaluator {"caller":"manager.go:359", "component":"discovery manager notify", "level":"debug", "msg":"Discovery receiver's channel was full so will retry the next cycle", "ts":"2024-05-10T09:04:02.572195445Z"}

The discovery manager is not able to keep up with service discovery events (changes) in this case. Is your rule-evaluator resource-starved in any way?

We would actually be able to track the delay through discovery metrics, but we don’t enable those in our rule-evaluator (we should!) - filed https://github.com/GoogleCloudPlatform/prometheus-engine/issues/973 to do that.