rook: Operator default of rook/ceph:master is bad idea

Similar to #12944

Helm: v1.13.5 Version: 18.2.1-0

If you k rollout restart ceph-mon... the rook-ceph-operator never reconciles itself, even if you set skipUpgradeChecks: true

This is worrisome because what if these nodes lost power suddenly?

I noticed that mon-a was not visible then I tried to manually start rook-ceph-mon-a with k rollout .... I decided to go one step further and restart the rest of the mons and it looks like I’m at the mercy of the operator working correctly?

2024-03-01 00:02:56.473146 I | ceph-spec: ceph-file-controller: CephCluster "rook-ceph" found but skipping reconcile since ceph health is &{Health:HEALTH_ERR Details:map[error:{Severity:Urgent Message:failed to get status. . timed out: exit status 1}] LastChecked:2024-03-01T00:01:42Z LastChanged:2024-02-29T23:06:12Z PreviousHealth:HEALTH_WARN Capacity:{TotalBytes:48009345761280 UsedBytes:1680641310720 AvailableBytes:46328704450560 LastUpdated:2024-02-29T23:31:48Z} Versions:<nil> FSID:}
2024-03-01 00:02:56.473192 I | ceph-spec: ceph-fs-subvolumegroup-controller: CephCluster "rook-ceph" found but skipping reconcile since ceph health is &{Health:HEALTH_ERR Details:map[error:{Severity:Urgent Message:failed to get status. . timed out: exit status 1}] LastChecked:2024-03-01T00:01:42Z LastChanged:2024-02-29T23:06:12Z PreviousHealth:HEALTH_WARN Capacity:{TotalBytes:48009345761280 UsedBytes:1680641310720 AvailableBytes:46328704450560 LastUpdated:2024-02-29T23:31:48Z} Versions:<nil> FSID:}
2024-03-01 00:03:06.473699 I | ceph-spec: ceph-file-controller: CephCluster "rook-ceph" found but skipping reconcile since ceph health is &{Health:HEALTH_ERR Details:map[error:{Severity:Urgent Message:failed to get status. . timed out: exit status 1}] LastChecked:2024-03-01T00:01:42Z LastChanged:2024-02-29T23:06:12Z PreviousHealth:HEALTH_WARN Capacity:{TotalBytes:48009345761280 UsedBytes:1680641310720 AvailableBytes:46328704450560 LastUpdated:2024-02-29T23:31:48Z} Versions:<nil> FSID:}
2024-03-01 00:03:06.473746 I | ceph-spec: ceph-fs-subvolumegroup-controller: CephCluster "rook-ceph" found but skipping reconcile since ceph health is &{Health:HEALTH_ERR Details:map[error:{Severity:Urgent Message:failed to get status. . timed out: exit status 1}] LastChecked:2024-03-01T00:01:42Z LastChanged:2024-02-29T23:06:12Z PreviousHealth:HEALTH_WARN Capacity:{TotalBytes:48009345761280 UsedBytes:1680641310720 AvailableBytes:46328704450560 LastUpdated:2024-02-29T23:31:48Z} Versions:<nil> FSID:}
2024-03-01 00:03:12.761208 E | ceph-cluster-controller: failed to get ceph status. failed to get status. . timed out: exit status 1
2024-03-01 00:03:13.300621 I | op-mon: mon a is not yet running
2024-03-01 00:03:13.321545 I | op-mon: mon b is not yet running
2024-03-01 00:03:13.418760 I | op-mon: mon c is not yet running
2024-03-01 00:03:13.418770 I | op-mon: mons running: []

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 19 (8 by maintainers)

Most upvoted comments

Your local rook-ceph-values.yaml must have the tag:master. You’ll need to remove that so the chart will use the release version. Your values.yaml is only expected to contain the values you want to modify from the released chart.

So I believe the issue is that my rook operator was using rook/ceph:master. I switched it to rook/ceph:v1.12.11-18.g582c940e8.

I experienced this issue yet again in this helm chart and ceph version.