rook: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:unknown Count:60}]"

Hi, I created a rook-ceph cluster via helm chart on new k8s cluster v1.26.3, new disk. helm install -f ./values.yaml rook-ceph rook-release/rook-ceph --version 1.10.7 --namespace rook-ceph image: quay.io/ceph/ceph:v17.2.3

I am seeing that no osd are created. host" failure domains: []. osd is down in failure domain. Pls. advise.

Disk /dev/sdb: 1 TiB, 1099511627776 bytes, 2147483648 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes 
lsblk -l
NAME          FSTYPE      LABEL UUID                                   MOUNTPOINT
sda
|-sda1        vfat              5F7D-C125                              /boot/efi
|-sda2        xfs               d2cc7d78-e0ec-486f-bebb-5bb8dcf62139   /boot
`-sda3        LVM2_member       clzjyk-SRnA-LnZf-f2GV-id1P-zL5t-ApwKmz
  |-rhel-root xfs               14005b17-abdd-46d7-838f-ab551433787c   /
  `-rhel-swap swap              9d0ee4d4-679d-4ac2-9e34-35223aee8f8d
sdb
lsblk -o KNAME,TYPE,SIZE,MODEL
KNAME TYPE  SIZE MODEL
sda   disk   32G Virtual disk
sda1  part  600M
sda2  part    1G
sda3  part 30.4G
sdb   disk    1T Virtual disk
dm-0  lvm  27.2G
dm-1  lvm   3.2G

This is the storage

  storage: # cluster level storage configuration and selection
    useAllNodes: false
    useAllDevices: false
    deviceFilter: sd[b-z]
    config:
    nodes:
      - name: "node3"
        devices: # specific devices to use for storage can be specified for each node
          - name: "sdb"
      - name: "node4"
        devices: # specific devices to use for storage can be specified for each node
          - name: "sdb"
      - name: "node5"
        devices: # specific devices to use for storage can be specified for each node
          - name: "sdb"
      - name: "node6"
        devices: # specific devices to use for storage can be specified for each node
          - name: "sdb"
      - name: "node7"
        devices: # specific devices to use for storage can be specified for each node
          - name: "sdb"
      - name: "node8"
        devices: # specific devices to use for storage can be specified for each node
          - name: "sdb"

I also zapped the disk sgdisk --zap-all /dev/sdb dd if=/dev/zero of=/dev/sdb bs=1M count=100 oflag=direct,dsync partprobe /dev/sdb

Portion of operator log

c000e15600 b:0xc000e15640 c:0xc000e15680]}
2023-03-30 11:35:57.597821 I | ceph-spec: detecting the ceph image version for image quay.io/ceph/ceph:v17.2.3...
2023-03-30 11:35:57.602429 D | op-k8sutil: ConfigMap ceph-object-controller-detect-version is already deleted
2023-03-30 11:35:58.511812 D | cephclient: checking any images/snapshots present in pool "ceph-blockpool"
2023-03-30 11:35:58.511844 D | exec: Running command: rbd pool stats ceph-blockpool --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --name=client.admin --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json
2023-03-30 11:35:59.192981 D | CmdReporter: job ceph-object-controller-detect-version has returned results
2023-03-30 11:35:59.210053 I | ceph-spec: detected ceph image version: "17.2.3-0 quincy"
2023-03-30 11:35:59.210093 D | exec: Running command: ceph versions --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --name=client.admin --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json
2023-03-30 11:35:59.238135 D | ceph-spec: object "ceph-object-controller-detect-version" did not match on delete
2023-03-30 11:35:59.238185 D | ceph-spec: object "ceph-object-controller-detect-version" did not match on delete
2023-03-30 11:35:59.238191 D | ceph-spec: object "ceph-object-controller-detect-version" did not match on delete
2023-03-30 11:35:59.238211 D | ceph-spec: object "ceph-object-controller-detect-version" did not match on delete
2023-03-30 11:35:59.497785 D | clusterdisruption-controller: pg health check timeout is not set in the cluster. waiting for PGs to get active+clean
2023-03-30 11:35:59.497820 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:unknown Count:60}]"
2023-03-30 11:35:59.499512 D | op-k8sutil: kubernetes version fetched 1.26.3
2023-03-30 11:35:59.505492 D | op-k8sutil: kubernetes version fetched 1.26.3
2023-03-30 11:35:59.505537 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2023-03-30 11:35:59.505585 D | clusterdisruption-controller: reconciling "rook-ceph/"
2023-03-30 11:35:59.506589 D | op-k8sutil: kubernetes version fetched 1.26.3
2023-03-30 11:35:59.507522 D | op-k8sutil: kubernetes version fetched 1.26.3
2023-03-30 11:35:59.507617 D | exec: Running command: ceph status --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --name=client.admin --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json
2023-03-30 11:35:59.710589 D | cephclient: {"mon":{"ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)":3},"mgr":{"ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)":2},"osd":{},"mds":{"ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)":2},"overall":{"ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)":7}}
2023-03-30 11:35:59.710620 D | cephclient: {"mon":{"ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)":3},"mgr":{"ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)":2},"osd":{},"mds":{"ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)":2},"overall":{"ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)":7}}
2023-03-30 11:35:59.710738 D | ceph-cluster-controller: updating ceph cluster "rook-ceph" status and condition to &{Health:{Status:HEALTH_WARN Checks:map[MDS_SLOW_METADATA_IO:{Severity:HEALTH_WARN Summary:{Message:1 MDSs report slow metadata IOs}} PG_AVAILABILITY:{Severity:HEALTH_WARN Summary:{Message:Reduced data availability: 60 pgs inactive}} TOO_FEW_OSDS:{Severity:HEALTH_WARN Summary:{Message:OSD count 0 < osd_pool_default_size 3}}]} FSID:3b2f2eb4-0a8f-49af-8a70-a712c4672952 ElectionEpoch:12 Quorum:[0 1 2] QuorumNames:[a b c] MonMap:{Epoch:3 FSID: CreatedTime: ModifiedTime: Mons:[]} OsdMap:{OsdMap:{Epoch:0 NumOsd:0 NumUpOsd:0 NumInOsd:0 Full:false NearFull:false NumRemappedPgs:0}} PgMap:{PgsByState:[{StateName:unknown Count:60}] Version:0 NumPgs:60 DataBytes:0 UsedBytes:0 AvailableBytes:0 TotalBytes:0 ReadBps:0 WriteBps:0 ReadOps:0 WriteOps:0 RecoveryBps:0 RecoveryObjectsPerSec:0 RecoveryKeysPerSec:0 CacheFlushBps:0 CacheEvictBps:0 CachePromoteBps:0} MgrMap:{Epoch:0 ActiveGID:0 ActiveName: ActiveAddr: Available:true Standbys:[]} Fsmap:{Epoch:63 ID:1 Up:1 In:1 Max:1 ByRank:[{FilesystemID:1 Rank:0 Name:ceph-filesystem-b Status:up:creating Gid:14494}] UpStandby:1}}, True, ClusterCreated, Cluster created successfully
2023-03-30 11:35:59.710753 D | ceph-spec: CephCluster "rook-ceph" status: "Ready". "Cluster created successfully"
2023-03-30 11:35:59.816366 D | ceph-cluster-controller: checking for stuck pods on not ready nodes
2023-03-30 11:35:59.816493 D | ceph-spec: found 1 ceph clusters in namespace "rook-ceph"
2023-03-30 11:35:59.816503 D | ceph-cluster-controller: update event on CephCluster CR

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 18 (8 by maintainers)

Most upvoted comments

@ZYWNB666

means none of the OSD daemons are running. So need to check why the OSDs are not running.

Can you share:

  • kubectl get pods -n rook-ceph
  • rook-ceph-operator pod logs
  • Logs of the rook-ceph-osd-prepare pods.
  • rook-ceph cluster yaml

Thanks for your reply, I have found the cause of the problem, because the disk was not cleaned after using ceph last time, I cleaned it again, and now it is working normally