VictoriaMetrics: vmalert: doesn't restore the state of alert properly
Describe the bug It seems like vmalert doesn’t restore the state of alert between restarts.
To Reproduce I have remote write and read specified in helm chart:
remote:
write:
url: "https://victoria-metrics.example.com/insert/0/prometheus"
read:
url: "https://victoria-metrics.example.com/select/0/prometheus"
When it starts up, I see that it was able to restore the state of alert:
"VictoriaMetrics/app/vmalert/alerting.go:439","msg":"alert \"TLAAdminDown\"(11235504160798675190) restored to state at 2020-10-29 00:37:20 +0000 UTC"}
But, in api/v1/alerts for vmalert I am seeing this:
"id": "17739412673847745131",
"name": "TLAAdminDown",
"group_id": "17747172397439028248",
"expression": "up{osfamily=\"debian\", snmp_module=\"ibm-imm\"} == 0",
"state": "pending",
"value": "0e+00",
Could it be that id of alert is not preserved for some reason and vmalert thinks it is a new alert?
Version 1.44
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 17
Commits related to this issue
- vmalert: skip automatically added labels on alerts restore Label `alertgroup` was introduced in #611 and automatically added to generated time series. By mistake, this new label wasn't correctly purg... — committed to VictoriaMetrics/VictoriaMetrics by hagen1778 4 years ago
- vmalert: skip automatically added labels on alerts restore (#871) Label `alertgroup` was introduced in #611 and automatically added to generated time series. By mistake, this new label wasn't correc... — committed to VictoriaMetrics/VictoriaMetrics by hagen1778 4 years ago
- vmalert: skip automatically added labels on alerts restore (#871) Label `alertgroup` was introduced in #611 and automatically added to generated time series. By mistake, this new label wasn't correc... — committed to VictoriaMetrics/VictoriaMetrics by hagen1778 4 years ago
- vmalert: explicitly set extra labels to alert entities The previous implementation treated extra labels (global and rule labels) as separate label set to returned time series labels. Hence, time seri... — committed to VictoriaMetrics/VictoriaMetrics by hagen1778 4 years ago
- vmalert: explicitly set extra labels to alert entities (#886) The previous implementation treated extra labels (global and rule labels) as separate label set to returned time series labels. Hence, t... — committed to VictoriaMetrics/VictoriaMetrics by hagen1778 4 years ago
- vmalert: explicitly set extra labels to alert entities (#886) The previous implementation treated extra labels (global and rule labels) as separate label set to returned time series labels. Hence, t... — committed to VictoriaMetrics/VictoriaMetrics by hagen1778 4 years ago
- docs/CHANGELOG.md: mention abot explicit setting of extra labels in alert entities (see 3adf8c5a6f4a8eea5f2025dd53ba975d42f8b8d1) Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/870 — committed to VictoriaMetrics/VictoriaMetrics by valyala 4 years ago
- docs/CHANGELOG.md: mention abot explicit setting of extra labels in alert entities (see 3adf8c5a6f4a8eea5f2025dd53ba975d42f8b8d1) Updates https://github.com/VictoriaMetrics/VictoriaMetrics/issues/870 — committed to VictoriaMetrics/VictoriaMetrics by valyala 4 years ago
@valyala Confirmed, works correctly now! Thank you!
Hi @sagor999! The fix was merged in commit https://github.com/VictoriaMetrics/VictoriaMetrics/pull/886/commits/08a89cf794e1c99c823ba7033edad634615d0087. You can verify it by building vmalert from sources. The change will be also included in the next release.
Hi @sagor999! Confirming the issue.