grafana-operator: Alert rules are constantly updated - causing alert state reset
Describe the bug I am using grafana-operator to deploy Grafana alert rules.
The alert rule groups are synced to the Grafana instances every 10 minutes (default). Sometimes, the alert rules are updated in a manner, that their fingerprint changes. This causes the alert rule state to reset, which again causes a lot of noise by triggering new notifications after the pending period for already firing alerts.
I tried to debug the issue by checking if there is constant drift between the alert rules in the CRs of the Grafana operator and the Grafana instances itself. But i can not find anything wrong. I did also check the alert_rule_version table in the Grafana DB. The only columns that are different between the alert rule versions are:
- id (makes sense)
- parent_version (makes sense)
- version (makes sense)
- created (makes sense)
- rule_group_idx (not sure what that is, some ID for the whole rule group that changes?)
How can i debug this further? Any ideas to why the alert rules are updated constantly?
Version Grafana operator: v5.17.0 Grafana: v11.5.2
To Reproduce Steps to reproduce the behavior:
- Create alert rule groups with multiple alert rules with grafana operator
- Check alert rule history in Grafana to see them being updated constantly
Expected behavior Alert rules only update, when they actually change.
Additional context
Output of the Grafana alert_rule_version table for one specific alert rule, that has this problem:
id | rule_org_id | rule_uid | rule_namespace_uid | rule_group | parent_version | restored_from | version | created | title | condition | data | interval_seconds | no_data_state | exec_err_state | for | annotations | labels | rule_group_idx | is_paused | notification_settings | record | metadata
--------+-------------+------------------------------+--------------------------------------+--------------+----------------+---------------+---------+---------------------+------------------------------+-----------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+---------------+----------------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------+----------------+-----------+---------------------------+--------+-----------------------------------------------------------------------------------------------------------------
664168 | 1 | example-dev-k8s-mem-pressure | 477b7cd3-2fee-4393-a40f-be43fc80ef4c | example-dev | 21 | 0 | 22 | 2025-03-14 12:54:15 | KubernetesNodeMemoryPressure | B | [{"refId":"A","queryType":"","relativeTimeRange":{"from":600,"to":0},"datasourceUid":"prometheus-example-dev","model":{"datasource":{"type":"prometheus","uid":"prometheus-example-dev"},"editorMode":"code","expr":"kube_node_status_condition{condition=\"MemoryPressure\",status=\"true\"}","instant":true,"intervalMs":1000,"legendFormat":"__auto","maxDataPoints":43200,"range":false,"refId":"A"}},{"refId":"B","queryType":"","relativeTimeRange":{"from":600,"to":0},"datasourceUid":"__expr__","model":{"conditions":[{"evaluator":{"params":[0],"type":"gt"},"operator":{"type":"and"},"query":{"params":["C"]},"reducer":{"params":[],"type":"last"},"type":"query"}],"datasource":{"type":"__expr__","uid":"__expr__"},"expression":"A","intervalMs":1000,"maxDataPoints":43200,"refId":"B","type":"threshold"}}] | 600 | NoData | Error | 600000000000 | {"description":"Node {{ $labels.node }} has MemoryPressure condition\nVALUE = {{ $value }}\nLABELS = {{ $labels }}","summary":"Kubernetes memory pressure (node {{ $labels.node }})"} | {"project":"example","severity":"critical","stage":"dev"} | 1 | f | [{"receiver":"opsgenie"}] | | {"editor_settings":{"simplified_query_and_expressions_section":false,"simplified_notifications_section":false}}
664230 | 1 | example-dev-k8s-mem-pressure | 477b7cd3-2fee-4393-a40f-be43fc80ef4c | example-dev | 22 | 0 | 23 | 2025-03-14 13:04:15 | KubernetesNodeMemoryPressure | B | [{"refId":"A","queryType":"","relativeTimeRange":{"from":600,"to":0},"datasourceUid":"prometheus-example-dev","model":{"datasource":{"type":"prometheus","uid":"prometheus-example-dev"},"editorMode":"code","expr":"kube_node_status_condition{condition=\"MemoryPressure\",status=\"true\"}","instant":true,"intervalMs":1000,"legendFormat":"__auto","maxDataPoints":43200,"range":false,"refId":"A"}},{"refId":"B","queryType":"","relativeTimeRange":{"from":600,"to":0},"datasourceUid":"__expr__","model":{"conditions":[{"evaluator":{"params":[0],"type":"gt"},"operator":{"type":"and"},"query":{"params":["C"]},"reducer":{"params":[],"type":"last"},"type":"query"}],"datasource":{"type":"__expr__","uid":"__expr__"},"expression":"A","intervalMs":1000,"maxDataPoints":43200,"refId":"B","type":"threshold"}}] | 600 | NoData | Error | 600000000000 | {"description":"Node {{ $labels.node }} has MemoryPressure condition\nVALUE = {{ $value }}\nLABELS = {{ $labels }}","summary":"Kubernetes memory pressure (node {{ $labels.node }})"} | {"project":"example","severity":"critical","stage":"dev"} | 0 | f | [{"receiver":"opsgenie"}] | | {"editor_settings":{"simplified_query_and_expressions_section":false,"simplified_notifications_section":false}}
664238 | 1 | example-dev-k8s-mem-pressure | 477b7cd3-2fee-4393-a40f-be43fc80ef4c | example-dev | 23 | 0 | 24 | 2025-03-14 13:04:16 | KubernetesNodeMemoryPressure | B | [{"refId":"A","queryType":"","relativeTimeRange":{"from":600,"to":0},"datasourceUid":"prometheus-example-dev","model":{"datasource":{"type":"prometheus","uid":"prometheus-example-dev"},"editorMode":"code","expr":"kube_node_status_condition{condition=\"MemoryPressure\",status=\"true\"}","instant":true,"intervalMs":1000,"legendFormat":"__auto","maxDataPoints":43200,"range":false,"refId":"A"}},{"refId":"B","queryType":"","relativeTimeRange":{"from":600,"to":0},"datasourceUid":"__expr__","model":{"conditions":[{"evaluator":{"params":[0],"type":"gt"},"operator":{"type":"and"},"query":{"params":["C"]},"reducer":{"params":[],"type":"last"},"type":"query"}],"datasource":{"type":"__expr__","uid":"__expr__"},"expression":"A","intervalMs":1000,"maxDataPoints":43200,"refId":"B","type":"threshold"}}] | 600 | NoData | Error | 600000000000 | {"description":"Node {{ $labels.node }} has MemoryPressure condition\nVALUE = {{ $value }}\nLABELS = {{ $labels }}","summary":"Kubernetes memory pressure (node {{ $labels.node }})"} | {"project":"example","severity":"critical","stage":"dev"} | 1 | f | [{"receiver":"opsgenie"}] | | {"editor_settings":{"simplified_query_and_expressions_section":false,"simplified_notifications_section":false}}
664300 | 1 | example-dev-k8s-mem-pressure | 477b7cd3-2fee-4393-a40f-be43fc80ef4c | example-dev | 24 | 0 | 25 | 2025-03-14 13:14:16 | KubernetesNodeMemoryPressure | B | [{"refId":"A","queryType":"","relativeTimeRange":{"from":600,"to":0},"datasourceUid":"prometheus-example-dev","model":{"datasource":{"type":"prometheus","uid":"prometheus-example-dev"},"editorMode":"code","expr":"kube_node_status_condition{condition=\"MemoryPressure\",status=\"true\"}","instant":true,"intervalMs":1000,"legendFormat":"__auto","maxDataPoints":43200,"range":false,"refId":"A"}},{"refId":"B","queryType":"","relativeTimeRange":{"from":600,"to":0},"datasourceUid":"__expr__","model":{"conditions":[{"evaluator":{"params":[0],"type":"gt"},"operator":{"type":"and"},"query":{"params":["C"]},"reducer":{"params":[],"type":"last"},"type":"query"}],"datasource":{"type":"__expr__","uid":"__expr__"},"expression":"A","intervalMs":1000,"maxDataPoints":43200,"refId":"B","type":"threshold"}}] | 600 | NoData | Error | 600000000000 | {"description":"Node {{ $labels.node }} has MemoryPressure condition\nVALUE = {{ $value }}\nLABELS = {{ $labels }}","summary":"Kubernetes memory pressure (node {{ $labels.node }})"} | {"project":"example","severity":"critical","stage":"dev"} | 0 | f | [{"receiver":"opsgenie"}] | | {"editor_settings":{"simplified_query_and_expressions_section":false,"simplified_notifications_section":false}}
664308 | 1 | example-dev-k8s-mem-pressure | 477b7cd3-2fee-4393-a40f-be43fc80ef4c | example-dev | 25 | 0 | 26 | 2025-03-14 13:14:17 | KubernetesNodeMemoryPressure | B | [{"refId":"A","queryType":"","relativeTimeRange":{"from":600,"to":0},"datasourceUid":"prometheus-example-dev","model":{"datasource":{"type":"prometheus","uid":"prometheus-example-dev"},"editorMode":"code","expr":"kube_node_status_condition{condition=\"MemoryPressure\",status=\"true\"}","instant":true,"intervalMs":1000,"legendFormat":"__auto","maxDataPoints":43200,"range":false,"refId":"A"}},{"refId":"B","queryType":"","relativeTimeRange":{"from":600,"to":0},"datasourceUid":"__expr__","model":{"conditions":[{"evaluator":{"params":[0],"type":"gt"},"operator":{"type":"and"},"query":{"params":["C"]},"reducer":{"params":[],"type":"last"},"type":"query"}],"datasource":{"type":"__expr__","uid":"__expr__"},"expression":"A","intervalMs":1000,"maxDataPoints":43200,"refId":"B","type":"threshold"}}] | 600 | NoData | Error | 600000000000 | {"description":"Node {{ $labels.node }} has MemoryPressure condition\nVALUE = {{ $value }}\nLABELS = {{ $labels }}","summary":"Kubernetes memory pressure (node {{ $labels.node }})"} | {"project":"example","severity":"critical","stage":"dev"} | 1 | f | [{"receiver":"opsgenie"}] | | {"editor_settings":{"simplified_query_and_expressions_section":false,"simplified_notifications_section":false}}
(5 rows)
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 15
A quick update on this. The fix has been rolled out for all Grafana Cloud users. If anyone here is using Cloud and could confirm it fixed the issue it would be great 🙏
Hey there 👋
Thanks for raising this. This is a known issue from our side. I’ve raised with the team, and we’re considering whether to deprecate
rule_group_idxentirely.In the meantime, I’ve merged a PR now to remove this field from the fingerprint calculation in the scheduler (which should stop it from reseting the rule state in those cases). From my tests, it seems to have fixed the issue but I haven’t used the grafana-operator so there might be more to it.
I’ll leave the issue open for now since the PR was just merged, but would be nice if someone could verify the fix works once it’s available 😄
@titolins thanks for the update! I am happy to test as soon as a version is released with this fix
@titolins thanks for the update! We’ll wait for user feedback on that and will probably bump the default Grafanna version used by the operator once a new release is out.
I ran into the same issue. I agree, that the issue does not happen with 1 alert per yaml file. But when more than 1 alert is added, then the first one does not have this issue, however all the subsequent ones have this.
@weisdd sorry for the delay, i was out of office last week. I have split the alert rules up and will post the results tomorrow. The updates only occour every few hours, so it takes a while to test it.
I have also seen that Grafana 11.6 released. There are a bunch of updates related to the alerting. I will also try the new version, maybe there is a fix for my issue in there.
It could be that Grafana updates the rule_group_idx internally and then the Operator applying the old value back. But we need to look further into this.