rook: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:unknown Count:60}]"
Hi, I created a rook-ceph cluster via helm chart on new k8s cluster v1.26.3, new disk. helm install -f ./values.yaml rook-ceph rook-release/rook-ceph --version 1.10.7 --namespace rook-ceph image: quay.io/ceph/ceph:v17.2.3
I am seeing that no osd are created. host" failure domains: []. osd is down in failure domain. Pls. advise.
Disk /dev/sdb: 1 TiB, 1099511627776 bytes, 2147483648 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
lsblk -l
NAME FSTYPE LABEL UUID MOUNTPOINT
sda
|-sda1 vfat 5F7D-C125 /boot/efi
|-sda2 xfs d2cc7d78-e0ec-486f-bebb-5bb8dcf62139 /boot
`-sda3 LVM2_member clzjyk-SRnA-LnZf-f2GV-id1P-zL5t-ApwKmz
|-rhel-root xfs 14005b17-abdd-46d7-838f-ab551433787c /
`-rhel-swap swap 9d0ee4d4-679d-4ac2-9e34-35223aee8f8d
sdb
lsblk -o KNAME,TYPE,SIZE,MODEL
KNAME TYPE SIZE MODEL
sda disk 32G Virtual disk
sda1 part 600M
sda2 part 1G
sda3 part 30.4G
sdb disk 1T Virtual disk
dm-0 lvm 27.2G
dm-1 lvm 3.2G
This is the storage
storage: # cluster level storage configuration and selection
useAllNodes: false
useAllDevices: false
deviceFilter: sd[b-z]
config:
nodes:
- name: "node3"
devices: # specific devices to use for storage can be specified for each node
- name: "sdb"
- name: "node4"
devices: # specific devices to use for storage can be specified for each node
- name: "sdb"
- name: "node5"
devices: # specific devices to use for storage can be specified for each node
- name: "sdb"
- name: "node6"
devices: # specific devices to use for storage can be specified for each node
- name: "sdb"
- name: "node7"
devices: # specific devices to use for storage can be specified for each node
- name: "sdb"
- name: "node8"
devices: # specific devices to use for storage can be specified for each node
- name: "sdb"
I also zapped the disk sgdisk --zap-all /dev/sdb dd if=/dev/zero of=/dev/sdb bs=1M count=100 oflag=direct,dsync partprobe /dev/sdb
Portion of operator log
c000e15600 b:0xc000e15640 c:0xc000e15680]}
2023-03-30 11:35:57.597821 I | ceph-spec: detecting the ceph image version for image quay.io/ceph/ceph:v17.2.3...
2023-03-30 11:35:57.602429 D | op-k8sutil: ConfigMap ceph-object-controller-detect-version is already deleted
2023-03-30 11:35:58.511812 D | cephclient: checking any images/snapshots present in pool "ceph-blockpool"
2023-03-30 11:35:58.511844 D | exec: Running command: rbd pool stats ceph-blockpool --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --name=client.admin --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json
2023-03-30 11:35:59.192981 D | CmdReporter: job ceph-object-controller-detect-version has returned results
2023-03-30 11:35:59.210053 I | ceph-spec: detected ceph image version: "17.2.3-0 quincy"
2023-03-30 11:35:59.210093 D | exec: Running command: ceph versions --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --name=client.admin --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json
2023-03-30 11:35:59.238135 D | ceph-spec: object "ceph-object-controller-detect-version" did not match on delete
2023-03-30 11:35:59.238185 D | ceph-spec: object "ceph-object-controller-detect-version" did not match on delete
2023-03-30 11:35:59.238191 D | ceph-spec: object "ceph-object-controller-detect-version" did not match on delete
2023-03-30 11:35:59.238211 D | ceph-spec: object "ceph-object-controller-detect-version" did not match on delete
2023-03-30 11:35:59.497785 D | clusterdisruption-controller: pg health check timeout is not set in the cluster. waiting for PGs to get active+clean
2023-03-30 11:35:59.497820 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:unknown Count:60}]"
2023-03-30 11:35:59.499512 D | op-k8sutil: kubernetes version fetched 1.26.3
2023-03-30 11:35:59.505492 D | op-k8sutil: kubernetes version fetched 1.26.3
2023-03-30 11:35:59.505537 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2023-03-30 11:35:59.505585 D | clusterdisruption-controller: reconciling "rook-ceph/"
2023-03-30 11:35:59.506589 D | op-k8sutil: kubernetes version fetched 1.26.3
2023-03-30 11:35:59.507522 D | op-k8sutil: kubernetes version fetched 1.26.3
2023-03-30 11:35:59.507617 D | exec: Running command: ceph status --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --name=client.admin --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json
2023-03-30 11:35:59.710589 D | cephclient: {"mon":{"ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)":3},"mgr":{"ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)":2},"osd":{},"mds":{"ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)":2},"overall":{"ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)":7}}
2023-03-30 11:35:59.710620 D | cephclient: {"mon":{"ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)":3},"mgr":{"ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)":2},"osd":{},"mds":{"ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)":2},"overall":{"ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)":7}}
2023-03-30 11:35:59.710738 D | ceph-cluster-controller: updating ceph cluster "rook-ceph" status and condition to &{Health:{Status:HEALTH_WARN Checks:map[MDS_SLOW_METADATA_IO:{Severity:HEALTH_WARN Summary:{Message:1 MDSs report slow metadata IOs}} PG_AVAILABILITY:{Severity:HEALTH_WARN Summary:{Message:Reduced data availability: 60 pgs inactive}} TOO_FEW_OSDS:{Severity:HEALTH_WARN Summary:{Message:OSD count 0 < osd_pool_default_size 3}}]} FSID:3b2f2eb4-0a8f-49af-8a70-a712c4672952 ElectionEpoch:12 Quorum:[0 1 2] QuorumNames:[a b c] MonMap:{Epoch:3 FSID: CreatedTime: ModifiedTime: Mons:[]} OsdMap:{OsdMap:{Epoch:0 NumOsd:0 NumUpOsd:0 NumInOsd:0 Full:false NearFull:false NumRemappedPgs:0}} PgMap:{PgsByState:[{StateName:unknown Count:60}] Version:0 NumPgs:60 DataBytes:0 UsedBytes:0 AvailableBytes:0 TotalBytes:0 ReadBps:0 WriteBps:0 ReadOps:0 WriteOps:0 RecoveryBps:0 RecoveryObjectsPerSec:0 RecoveryKeysPerSec:0 CacheFlushBps:0 CacheEvictBps:0 CachePromoteBps:0} MgrMap:{Epoch:0 ActiveGID:0 ActiveName: ActiveAddr: Available:true Standbys:[]} Fsmap:{Epoch:63 ID:1 Up:1 In:1 Max:1 ByRank:[{FilesystemID:1 Rank:0 Name:ceph-filesystem-b Status:up:creating Gid:14494}] UpStandby:1}}, True, ClusterCreated, Cluster created successfully
2023-03-30 11:35:59.710753 D | ceph-spec: CephCluster "rook-ceph" status: "Ready". "Cluster created successfully"
2023-03-30 11:35:59.816366 D | ceph-cluster-controller: checking for stuck pods on not ready nodes
2023-03-30 11:35:59.816493 D | ceph-spec: found 1 ceph clusters in namespace "rook-ceph"
2023-03-30 11:35:59.816503 D | ceph-cluster-controller: update event on CephCluster CR
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 18 (8 by maintainers)
Thanks for your reply, I have found the cause of the problem, because the disk was not cleaned after using ceph last time, I cleaned it again, and now it is working normally