rook: CephPGImbalance Prometheus Alert doesn't account for device differences

The CephPGImbalance alert fires in our cluster for every nvme OSD because they have fewer PGs than the “average” OSD. Aren’t there a variety of reasons an OSD would have fewer or more PGs than average? Device Class… weights/capacity, crush rules for various Pools, etc?

https://github.com/rook/rook/blob/510d2721b4ca350f26c998d528926eb1e3b66e4f/deploy/charts/rook-ceph-cluster/prometheus/localrules.yaml#L240C17-L253

Here’s the same query but modified to include the device_class, and that seems to work better for our environment. However, this still doesn’t account for differences in ceph_osd_stat_bytes (disk sizes). Maybe it should be ceph_osd_numpg / ceph_osd_stat_bytes? I can work on that query if that sounds reasonable, and submit a PR.

abs(
 ((ceph_osd_numpg * on (ceph_daemon) group_left(device_class) ceph_osd_metadata > 0) - on (job, device_class) group_left avg(ceph_osd_numpg * on (ceph_daemon) group_left(device_class) ceph_osd_metadata > 0) by (job, device_class)) /
 on (job, device_class) group_left avg(ceph_osd_numpg * on (ceph_daemon) group_left(device_class) ceph_osd_metadata > 0) by (job, device_class)
) * on (ceph_daemon) group_left(hostname, device_class) ceph_osd_metadata > 0.30

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 5
  • Comments: 15 (8 by maintainers)

Commits related to this issue

Most upvoted comments

Grok provided this optimization which I think is cleaner and easier to understand:

abs(
  (
    (
      (ceph_osd_numpg > 0) / clamp_min(ceph_osd_stat_bytes, 1)
    ) * on(job) group_left max(ceph_osd_stat_bytes) by(job)
  ) - 
  on(job) group_left avg(
    (
      (ceph_osd_numpg > 0) / clamp_min(ceph_osd_stat_bytes, 1)
    ) * on(job) group_left max(ceph_osd_stat_bytes) by(job)
  ) by(job)
) / 
on(job) group_left clamp_min(
  avg(
    (
      (ceph_osd_numpg > 0) / clamp_min(ceph_osd_stat_bytes, 1)
    ) * on(job) group_left max(ceph_osd_stat_bytes) by(job)
  ) by(job),
  1
) * on(ceph_daemon) group_left(device_class, hostname) ceph_osd_metadata