prometheus-engine: rule-evaluator doesn't get updated alertmanager pod ipv4s
hello - we’re currently using Managed Prometheus and a self-hosted Alertmanager deployment. This has been functioning properly for over a year. We’re currently on this version of rule-evaluator
gke.gcr.io/prometheus-engine/rule-evaluator:v0.8.1-gke.9
our rule-evaluator sends events to a self-managed Alertmanager statefulset, which lives in a separate namespace. We configure this via the OperatorConfig CRD:
---
apiVersion: monitoring.googleapis.com/v1
kind: OperatorConfig
metadata:
namespace: gmp-public
name: config
# https://github.com/GoogleCloudPlatform/prometheus-engine/blob/main/doc/api.md#ruleevaluatorspec
rules:
alerting:
alertmanagers:
# configures where the rule-evaluator will send alerting events to
- name: alertmanager
namespace: monitoring
port: 9093
# https://github.com/GoogleCloudPlatform/prometheus-engine/blob/main/doc/api.md#managedalertmanagerspec
# NOTE: this section is unused, as it points to an empty default Alertmanager configuration file
# since we are using a self-deployed Alertmanager instead of the one provided by GMP
managedAlertmanager:
configSecret:
name: alertmanager
key: alertmanager.yaml
in the last month or so, we’ve noticed that the rule-evaluator will be unable to resolve the downstream Alertmanager address after the Alertmanager pod is rescheduled.
From there, we’ll see the rule-evaluator log this out:
{
alertmanager: "http://10.34.25.18:9093/api/v2/alerts"
caller: "notifier.go:532"
component: "notifier"
count: 1
err: "Post "http://10.34.25.18:9093/api/v2/alerts": context deadline exceeded"
level: "error"
msg: "Error sending alert"
ts: "2024-02-12T18:17:33.124842335Z"
}
this can go on for an hour - we have pages set up to notify us when the rule-evaluator stops pinging Alertmanager through a custom heartbeat rule. The only way to resolve this is by restarting the rule-evaluator deployment
this suggests that the rule-evaluator is not reconciling downstream ip addresses after startup, since we provide the k8s DNS components in the OperatorConfig for the Alertmanager receiver
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 2
- Comments: 17
Hi @bwplotka , I’m going to configure VPA for the rule evaluator and test it.
Oh wait - I actually think your debug logs from your second comment hold the clue. Specifically:
The discovery manager is not able to keep up with service discovery events (changes) in this case. Is your rule-evaluator resource-starved in any way?
We would actually be able to track the delay through discovery metrics, but we don’t enable those in our rule-evaluator (we should!) - filed https://github.com/GoogleCloudPlatform/prometheus-engine/issues/973 to do that.