rook: Module 'devicehealth' has failed: osd pool create failed: No suitable CRUSH rule exists, check 'osd pool default crush *' config options retval: -2 (MGR_MODULE_ERROR)
Is this a bug report or feature request?
- Bug Report
The message I get in this bug is the same i got on #12356, but now using the 1.12.0 release and the .mgr pool has been created just fine, but i still get the same error.
I am building a small test cluster in stretched mode. I have 3 nodes: stg1 -> zone dc7 stg2 -> zone dcc stg3 -> zone dca arbiter
stg1 and stg2 has 3 extra disks, which will be used for the 4 osds required and I am using subFailureDomain: osd.
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
name: rook-ceph
namespace: rook-ceph # namespace:cluster
spec:
crashCollector:
disable: true
dataDirHostPath: /var/lib/rook
monitoring:
enabled: false
mon:
# Five mons must be created for stretch mode
count: 5
allowMultiplePerNode: false
stretchCluster:
failureDomainLabel: topology.kubernetes.io/zone
subFailureDomain: osd
zones:
- name: dca
arbiter: true
- name: dcc
- name: dc7
mgr:
count: 2
cephVersion:
image: quay.io/ceph/ceph:v17.2.6
allowUnsupported: true
skipUpgradeChecks: false
continueUpgradeAfterChecksEvenIfNotHealthy: false
dashboard:
enabled: true
ssl: false
#network:
# provider: host
storage:
useAllNodes: true
useAllDevices: true
deviceFilter: ""
# OSD placement is expected to include the non-arbiter zones
placement:
all:
tolerations:
- key: taint/node-role
operator: Exists
osd:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
- dcc
- dc7
- key: storage-node
operator: Exists
priorityClassNames:
mon: system-node-critical
osd: system-node-critical
mgr: system-cluster-critical
disruptionManagement:
managePodBudgets: true
---
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
name: builtin-mgr
namespace: rook-ceph
spec:
name: .mgr
failureDomain: zone
replicated:
size: 4
requireSafeReplicaSize: true
replicasPerFailureDomain: 2
subFailureDomain: osd
Logs to submit:
- Operator’s logs, if necessary operator.log
Cluster Status to submit:
ceph status
cluster:
id: 6136021e-be33-4ec5-8f44-2e7a85d270d2
health: HEALTH_ERR
Module 'devicehealth' has failed: osd pool create failed: No suitable CRUSH rule exists, check 'osd pool default crush *' config options retval: -2
services:
mon: 5 daemons, quorum a,b,c,d,e (age 39m)
mgr: b(active, since 39m), standbys: a
osd: 6 osds: 6 up (since 39m), 6 in (since 39m)
data:
pools: 1 pools, 32 pgs
objects: 0 objects, 0 B
usage: 120 MiB used, 120 GiB / 120 GiB avail
pgs: 32 active+clean
# ceph health
HEALTH_ERR Module 'devicehealth' has failed: osd pool create failed: No suitable CRUSH rule exists, check 'osd pool default crush *' config options retval: -2
# ceph osd crush rule ls
replicated_rule
default_stretch_cluster_rule
# ceph osd pool ls
.mgr
# ceph osd pool get .mgr crush_rule
crush_rule: default_stretch_cluster_rule
# ceph osd getcrushmap -o /tmp/cm.bin
# crushtool -d /tmp/cm.bin -o /tmp/cm.bin.ascii
# cat /tmp/cm.bin.ascii
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable chooseleaf_stable 1
tunable straw_calc_version 1
tunable allowed_bucket_algs 54
# devices
device 0 osd.0 class hdd
device 1 osd.1 class hdd
device 2 osd.2 class hdd
device 3 osd.3 class hdd
device 4 osd.4 class hdd
device 5 osd.5 class hdd
# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 zone
type 10 region
type 11 root
# buckets
host stg1 {
id -3 # do not change unnecessarily
id -5 class hdd # do not change unnecessarily
# weight 0.05846
alg straw2
hash 0 # rjenkins1
item osd.4 weight 0.01949
item osd.0 weight 0.01949
item osd.2 weight 0.01949
}
zone dcc {
id -4 # do not change unnecessarily
id -6 class hdd # do not change unnecessarily
# weight 0.05846
alg straw2
hash 0 # rjenkins1
item stg1 weight 0.05846
}
host stg2 {
id -7 # do not change unnecessarily
id -9 class hdd # do not change unnecessarily
# weight 0.05846
alg straw2
hash 0 # rjenkins1
item osd.5 weight 0.01949
item osd.1 weight 0.01949
item osd.3 weight 0.01949
}
zone dc7 {
id -8 # do not change unnecessarily
id -10 class hdd # do not change unnecessarily
# weight 0.05846
alg straw2
hash 0 # rjenkins1
item stg2 weight 0.05846
}
root default {
id -1 # do not change unnecessarily
id -2 class hdd # do not change unnecessarily
# weight 0.11691
alg straw2
hash 0 # rjenkins1
item dcc weight 0.05846
item dc7 weight 0.05846
}
# rules
rule replicated_rule {
id 0
type replicated
step take default
step chooseleaf firstn 0 type host
step emit
}
rule default_stretch_cluster_rule {
id 1
type replicated
step take default
step choose firstn 0 type zone
step chooseleaf firstn 2 type osd
step emit
}
# end crush map
# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.11691 root default
-8 0.05846 zone dc7
-7 0.05846 host stg2
1 hdd 0.01949 osd.1 up 1.00000 1.00000
3 hdd 0.01949 osd.3 up 1.00000 1.00000
5 hdd 0.01949 osd.5 up 1.00000 1.00000
-4 0.05846 zone dcc
-3 0.05846 host stg1
0 hdd 0.01949 osd.0 up 1.00000 1.00000
2 hdd 0.01949 osd.2 up 1.00000 1.00000
4 hdd 0.01949 osd.4 up 1.00000 1.00000
To get the status of the cluster, use `kubectl rook-ceph ceph status`
For more details, see the [Rook Krew Plugin](https://rook.io/docs/rook/latest-release/Troubleshooting/krew-plugin)
**Environment**:
* OS (e.g. from /etc/os-release): AlmaLinux release 9.2 (Turquoise Kodkod)
* Kernel (e.g. `uname -a`): 5.14.0-284.18.1.el9_2.x86_64
* Cloud provider or hardware configuration: onprem
* Rook version (use `rook version` inside of a Rook Pod):
rook: v1.12.0 go: go1.20.5
* Storage backend version (e.g. for ceph do `ceph -v`):
* Kubernetes version (use `kubectl version`):
Client Version: version.Info{Major:“1”, Minor:“26”, GitVersion:“v1.26.5”, GitCommit:“890a139214b4de1f01543d15003b5bda71aae9c7”, GitTreeState:“clean”, BuildDate:“2023-05-17T14:14:46Z”, GoVersion:“go1.19.9”, Compiler:“gc”, Platform:“linux/amd64”} Kustomize Version: v4.5.7 Server Version: version.Info{Major:“1”, Minor:“26”, GitVersion:“v1.26.5”, GitCommit:“890a139214b4de1f01543d15003b5bda71aae9c7”, GitTreeState:“clean”, BuildDate:“2023-05-17T14:08:49Z”, GoVersion:“go1.19.9”, Compiler:“gc”, Platform:“linux/amd64”}
* Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): onprem/kubespray
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 26 (11 by maintainers)
I restarted all nodes ( including masters ) and apparently it cleared the problem.
@travisn yes I tried with 1000m and sometimes it worked sometimes no. With 1500m works much better. The interesting thing is that the issue is not only about the CephBlockPool but it can happen also during the cephobjectstore creation or during the cephobjectzone… I tried to increase the limit just because I read an issue related to a missing timeout on an async call here https://github.com/rook/rook/issues/10712 so I supposed that a similar issue can be present also in other places… so I tried to make the operator faster.