rook: Module 'devicehealth' has failed: osd pool create failed: No suitable CRUSH rule exists, check 'osd pool default crush *' config options retval: -2 (MGR_MODULE_ERROR)

Is this a bug report or feature request?

  • Bug Report

The message I get in this bug is the same i got on #12356, but now using the 1.12.0 release and the .mgr pool has been created just fine, but i still get the same error.

I am building a small test cluster in stretched mode. I have 3 nodes: stg1 -> zone dc7 stg2 -> zone dcc stg3 -> zone dca arbiter

stg1 and stg2 has 3 extra disks, which will be used for the 4 osds required and I am using subFailureDomain: osd.

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph # namespace:cluster
spec:
  crashCollector:
    disable: true
  dataDirHostPath: /var/lib/rook
  monitoring:
    enabled: false
  mon:
    # Five mons must be created for stretch mode
    count: 5
    allowMultiplePerNode: false
    stretchCluster:
      failureDomainLabel: topology.kubernetes.io/zone
      subFailureDomain: osd
      zones:
        - name: dca
          arbiter: true
        - name: dcc
        - name: dc7
  mgr:
    count: 2
  cephVersion:
    image: quay.io/ceph/ceph:v17.2.6
    allowUnsupported: true
  skipUpgradeChecks: false
  continueUpgradeAfterChecksEvenIfNotHealthy: false
  dashboard:
    enabled: true
    ssl: false
  #network:
  #  provider: host
  storage:
    useAllNodes: true
    useAllDevices: true
    deviceFilter: ""
  # OSD placement is expected to include the non-arbiter zones
  placement:
    all:
      tolerations:
        - key: taint/node-role
          operator: Exists
    osd:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: topology.kubernetes.io/zone
                  operator: In
                  values:
                    - dcc
                    - dc7
                - key: storage-node
                  operator: Exists
  priorityClassNames:
    mon: system-node-critical
    osd: system-node-critical
    mgr: system-cluster-critical
  disruptionManagement:
    managePodBudgets: true

---
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: builtin-mgr
  namespace: rook-ceph
spec:
  name: .mgr
  failureDomain: zone
  replicated:
    size: 4
    requireSafeReplicaSize: true
    replicasPerFailureDomain: 2
    subFailureDomain: osd


Logs to submit:

Cluster Status to submit:

ceph status
  cluster:
    id:     6136021e-be33-4ec5-8f44-2e7a85d270d2
    health: HEALTH_ERR
            Module 'devicehealth' has failed: osd pool create failed: No suitable CRUSH rule exists, check 'osd pool default crush *' config options retval: -2
 
  services:
    mon: 5 daemons, quorum a,b,c,d,e (age 39m)
    mgr: b(active, since 39m), standbys: a
    osd: 6 osds: 6 up (since 39m), 6 in (since 39m)
 
  data:
    pools:   1 pools, 32 pgs
    objects: 0 objects, 0 B
    usage:   120 MiB used, 120 GiB / 120 GiB avail
    pgs:     32 active+clean

# ceph  health
HEALTH_ERR Module 'devicehealth' has failed: osd pool create failed: No suitable CRUSH rule exists, check 'osd pool default crush *' config options retval: -2
# ceph osd crush rule ls
replicated_rule
default_stretch_cluster_rule
# ceph osd pool ls
.mgr
# ceph osd pool get .mgr crush_rule
crush_rule: default_stretch_cluster_rule
# ceph osd getcrushmap -o /tmp/cm.bin
# crushtool -d /tmp/cm.bin -o /tmp/cm.bin.ascii
# cat /tmp/cm.bin.ascii
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable chooseleaf_stable 1
tunable straw_calc_version 1
tunable allowed_bucket_algs 54

# devices
device 0 osd.0 class hdd
device 1 osd.1 class hdd
device 2 osd.2 class hdd
device 3 osd.3 class hdd
device 4 osd.4 class hdd
device 5 osd.5 class hdd

# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 zone
type 10 region
type 11 root

# buckets
host stg1 {
	id -3		# do not change unnecessarily
	id -5 class hdd		# do not change unnecessarily
	# weight 0.05846
	alg straw2
	hash 0	# rjenkins1
	item osd.4 weight 0.01949
	item osd.0 weight 0.01949
	item osd.2 weight 0.01949
}
zone dcc {
	id -4		# do not change unnecessarily
	id -6 class hdd		# do not change unnecessarily
	# weight 0.05846
	alg straw2
	hash 0	# rjenkins1
	item stg1 weight 0.05846
}
host stg2 {
	id -7		# do not change unnecessarily
	id -9 class hdd		# do not change unnecessarily
	# weight 0.05846
	alg straw2
	hash 0	# rjenkins1
	item osd.5 weight 0.01949
	item osd.1 weight 0.01949
	item osd.3 weight 0.01949
}
zone dc7 {
	id -8		# do not change unnecessarily
	id -10 class hdd		# do not change unnecessarily
	# weight 0.05846
	alg straw2
	hash 0	# rjenkins1
	item stg2 weight 0.05846
}
root default {
	id -1		# do not change unnecessarily
	id -2 class hdd		# do not change unnecessarily
	# weight 0.11691
	alg straw2
	hash 0	# rjenkins1
	item dcc weight 0.05846
	item dc7 weight 0.05846
}

# rules
rule replicated_rule {
	id 0
	type replicated
	step take default
	step chooseleaf firstn 0 type host
	step emit
}
rule default_stretch_cluster_rule {
	id 1
	type replicated
	step take default
	step choose firstn 0 type zone
	step chooseleaf firstn 2 type osd
	step emit
}

# end crush map

# ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME          STATUS  REWEIGHT  PRI-AFF
-1         0.11691  root default                                
-8         0.05846      zone dc7                                
-7         0.05846          host stg2                           
 1    hdd  0.01949              osd.1      up   1.00000  1.00000
 3    hdd  0.01949              osd.3      up   1.00000  1.00000
 5    hdd  0.01949              osd.5      up   1.00000  1.00000
-4         0.05846      zone dcc                                
-3         0.05846          host stg1                           
 0    hdd  0.01949              osd.0      up   1.00000  1.00000
 2    hdd  0.01949              osd.2      up   1.00000  1.00000
 4    hdd  0.01949              osd.4      up   1.00000  1.00000

  To get the status of the cluster, use `kubectl rook-ceph ceph status`
  For more details, see the [Rook Krew Plugin](https://rook.io/docs/rook/latest-release/Troubleshooting/krew-plugin)

**Environment**:
* OS (e.g. from /etc/os-release): AlmaLinux release 9.2 (Turquoise Kodkod)
* Kernel (e.g. `uname -a`): 5.14.0-284.18.1.el9_2.x86_64
* Cloud provider or hardware configuration: onprem
* Rook version (use `rook version` inside of a Rook Pod):

rook: v1.12.0 go: go1.20.5

* Storage backend version (e.g. for ceph do `ceph -v`):
* Kubernetes version (use `kubectl version`):

Client Version: version.Info{Major:“1”, Minor:“26”, GitVersion:“v1.26.5”, GitCommit:“890a139214b4de1f01543d15003b5bda71aae9c7”, GitTreeState:“clean”, BuildDate:“2023-05-17T14:14:46Z”, GoVersion:“go1.19.9”, Compiler:“gc”, Platform:“linux/amd64”} Kustomize Version: v4.5.7 Server Version: version.Info{Major:“1”, Minor:“26”, GitVersion:“v1.26.5”, GitCommit:“890a139214b4de1f01543d15003b5bda71aae9c7”, GitTreeState:“clean”, BuildDate:“2023-05-17T14:08:49Z”, GoVersion:“go1.19.9”, Compiler:“gc”, Platform:“linux/amd64”}

* Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): onprem/kubespray

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 26 (11 by maintainers)

Most upvoted comments

I restarted all nodes ( including masters ) and apparently it cleared the problem.

@travisn yes I tried with 1000m and sometimes it worked sometimes no. With 1500m works much better. The interesting thing is that the issue is not only about the CephBlockPool but it can happen also during the cephobjectstore creation or during the cephobjectzone… I tried to increase the limit just because I read an issue related to a missing timeout on an async call here https://github.com/rook/rook/issues/10712 so I supposed that a similar issue can be present also in other places… so I tried to make the operator faster.