rook: CephPGImbalance Prometheus Alert doesn't account for device differences
The CephPGImbalance alert fires in our cluster for every nvme OSD because they have fewer PGs than the “average” OSD. Aren’t there a variety of reasons an OSD would have fewer or more PGs than average? Device Class… weights/capacity, crush rules for various Pools, etc?
Here’s the same query but modified to include the device_class, and that seems to work better for our environment. However, this still doesn’t account for differences in ceph_osd_stat_bytes (disk sizes). Maybe it should be ceph_osd_numpg / ceph_osd_stat_bytes? I can work on that query if that sounds reasonable, and submit a PR.
abs(
((ceph_osd_numpg * on (ceph_daemon) group_left(device_class) ceph_osd_metadata > 0) - on (job, device_class) group_left avg(ceph_osd_numpg * on (ceph_daemon) group_left(device_class) ceph_osd_metadata > 0) by (job, device_class)) /
on (job, device_class) group_left avg(ceph_osd_numpg * on (ceph_daemon) group_left(device_class) ceph_osd_metadata > 0) by (job, device_class)
) * on (ceph_daemon) group_left(hostname, device_class) ceph_osd_metadata > 0.30
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 5
- Comments: 15 (8 by maintainers)
Commits related to this issue
- feat(silence): add new Silence configuration for Ceph PG imbalance alert ref: https://github.com/rook/rook/issues/13257 Signed-off-by: billimek <jeff@billimek.com> — committed to billimek/k8s-gitops by billimek a year ago
- fix: CephPGImbalance alert to consider OSD size The CephPGImbalance alert has been updated to consider the relative size of OSDs when calculating the imbalance. This change ensures that the alert is ... — committed to patsoffice/rook by patsoffice a year ago
- fix: ceph: CephPGImbalance alert to consider OSD size The CephPGImbalance alert has been updated to consider the relative size of OSDs when calculating the imbalance. This change ensures that the ale... — committed to patsoffice/rook by patsoffice a year ago
- ceph: CephPGImbalance alert to consider OSD size The CephPGImbalance alert has been updated to consider the relative size of OSDs when calculating the imbalance. This change ensures that the alert is... — committed to patsoffice/rook by patsoffice a year ago
- monitoring: imbalance alert to consider osd size The CephPGImbalance alert has been updated to consider the relative size of OSDs when calculating the imbalance. This change ensures that the alert is... — committed to patsoffice/rook by patsoffice a year ago
- monitoring: imbalance alert to consider osd size The CephPGImbalance alert has been updated to consider the relative size of OSDs when calculating the imbalance. This change ensures that the alert is... — committed to patsoffice/rook by patsoffice a year ago
Grok provided this optimization which I think is cleaner and easier to understand: