rook: Operator default of rook/ceph:master is bad idea
Similar to #12944
Helm: v1.13.5 Version: 18.2.1-0
If you k rollout restart ceph-mon... the rook-ceph-operator never reconciles itself, even if you set skipUpgradeChecks: true
This is worrisome because what if these nodes lost power suddenly?
I noticed that mon-a was not visible then I tried to manually start rook-ceph-mon-a with k rollout .... I decided to go one step further and restart the rest of the mons and it looks like I’m at the mercy of the operator working correctly?
2024-03-01 00:02:56.473146 I | ceph-spec: ceph-file-controller: CephCluster "rook-ceph" found but skipping reconcile since ceph health is &{Health:HEALTH_ERR Details:map[error:{Severity:Urgent Message:failed to get status. . timed out: exit status 1}] LastChecked:2024-03-01T00:01:42Z LastChanged:2024-02-29T23:06:12Z PreviousHealth:HEALTH_WARN Capacity:{TotalBytes:48009345761280 UsedBytes:1680641310720 AvailableBytes:46328704450560 LastUpdated:2024-02-29T23:31:48Z} Versions:<nil> FSID:}
2024-03-01 00:02:56.473192 I | ceph-spec: ceph-fs-subvolumegroup-controller: CephCluster "rook-ceph" found but skipping reconcile since ceph health is &{Health:HEALTH_ERR Details:map[error:{Severity:Urgent Message:failed to get status. . timed out: exit status 1}] LastChecked:2024-03-01T00:01:42Z LastChanged:2024-02-29T23:06:12Z PreviousHealth:HEALTH_WARN Capacity:{TotalBytes:48009345761280 UsedBytes:1680641310720 AvailableBytes:46328704450560 LastUpdated:2024-02-29T23:31:48Z} Versions:<nil> FSID:}
2024-03-01 00:03:06.473699 I | ceph-spec: ceph-file-controller: CephCluster "rook-ceph" found but skipping reconcile since ceph health is &{Health:HEALTH_ERR Details:map[error:{Severity:Urgent Message:failed to get status. . timed out: exit status 1}] LastChecked:2024-03-01T00:01:42Z LastChanged:2024-02-29T23:06:12Z PreviousHealth:HEALTH_WARN Capacity:{TotalBytes:48009345761280 UsedBytes:1680641310720 AvailableBytes:46328704450560 LastUpdated:2024-02-29T23:31:48Z} Versions:<nil> FSID:}
2024-03-01 00:03:06.473746 I | ceph-spec: ceph-fs-subvolumegroup-controller: CephCluster "rook-ceph" found but skipping reconcile since ceph health is &{Health:HEALTH_ERR Details:map[error:{Severity:Urgent Message:failed to get status. . timed out: exit status 1}] LastChecked:2024-03-01T00:01:42Z LastChanged:2024-02-29T23:06:12Z PreviousHealth:HEALTH_WARN Capacity:{TotalBytes:48009345761280 UsedBytes:1680641310720 AvailableBytes:46328704450560 LastUpdated:2024-02-29T23:31:48Z} Versions:<nil> FSID:}
2024-03-01 00:03:12.761208 E | ceph-cluster-controller: failed to get ceph status. failed to get status. . timed out: exit status 1
2024-03-01 00:03:13.300621 I | op-mon: mon a is not yet running
2024-03-01 00:03:13.321545 I | op-mon: mon b is not yet running
2024-03-01 00:03:13.418760 I | op-mon: mon c is not yet running
2024-03-01 00:03:13.418770 I | op-mon: mons running: []
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 19 (8 by maintainers)
Your local rook-ceph-values.yaml must have the
tag:master. You’ll need to remove that so the chart will use the release version. Your values.yaml is only expected to contain the values you want to modify from the released chart.So I believe the issue is that my rook operator was using
rook/ceph:master. I switched it torook/ceph:v1.12.11-18.g582c940e8.I experienced this issue yet again in this helm chart and ceph version.