rook: OSD in CrashLoopBackOff - start interval does not contain the required bound
Is this a bug report or feature request?
- Bug Report
Deviation from expected behavior: mons, osd, mds pods don’t start after moving the K8S cluster to another network Expected behavior:
How to reproduce it (minimal and precise):
I changed all the IP addresses in my cluster nodes and then I started all the nodes.
After that, 3 mon’s, 3 osd, mgr, mds didn’t work for me.
Then I repair mon’s using kubectl rook-ceph mons restore-quorum с, but osd’s didn’t work.
Logs to submit:
- Crashing pod(s) logs:
kubectl logs rook-ceph-osd-1-7499d8df4f-bnm5k -n rook-ceph
debug -74> 2022-11-15T11:05:45.503+0000 7f110888f700 -1 log_channel(cluster) log [ERR] : 11.15s1 past_intervals [1174,10993) start interval does not contain the required bound [887,10993) start
debug -73> 2022-11-15T11:05:45.503+0000 7f110888f700 -1 osd.1 pg_epoch: 10994 pg[11.15s1( empty local-lis/les=0/0 n=0 ec=1174/1174 lis/c=984/886 les/c/f=985/887/0 sis=10993) [1,NONE,NONE]p1(0) r=-1 lpr=10993 pi=[1174,10993)/4 crt=0'0 mlcod 0'0 unknown mbc={}] 11.15s1 past_intervals [1174,10993) start interval does not contain the required bound [887,10993) start
debug -72> 2022-11-15T11:05:45.507+0000 7f112bb40080 0 osd.1 11116 done with init, starting boot process
debug -71> 2022-11-15T11:05:45.507+0000 7f112bb40080 10 monclient: _renew_subs
debug -70> 2022-11-15T11:05:45.507+0000 7f112bb40080 10 monclient: _send_mon_message to mon.k at v2:10.233.16.22:3300/0
debug -69> 2022-11-15T11:05:45.507+0000 7f112bb40080 1 osd.1 11116 start_boot
Cluster Status to submit:
-
Output of krew commands, if necessary
To get the health of the cluster, use
kubectl rook-ceph health
Info: Checking if at least three mon pods are running on different nodes
rook-ceph-mon-h-657654bfff-49dls 1/1 Running 0 36m
rook-ceph-mon-j-6bc66dbb6f-2lhs9 1/1 Running 0 26h
rook-ceph-mon-k-7664dd456c-7bmvv 1/1 Running 0 41m
Info: Checking mon quorum and ceph health details
Warning: HEALTH_WARN 1 filesystem is degraded; insufficient standby MDS daemons available; 1 MDSs report slow metadata IOs; 2 osds down; 3 hosts (3 osds) down; 1 root (3 osds) down; Reduced data availability: 177 pgs inactive
[WRN] FS_DEGRADED: 1 filesystem is degraded
fs ceph-filesystem is degraded
[WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available
have 0; want 1 more
[WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs
mds.ceph-filesystem-a(mds.0): 4 slow metadata IOs are blocked > 30 secs, oldest blocked for 3375 secs
[WRN] OSD_DOWN: 2 osds down
osd.0 (root=default,host=k8s-worker-01) is down
osd.1 (root=default,host=k8s-worker-02) is down
[WRN] OSD_HOST_DOWN: 3 hosts (3 osds) down
host k8s-worker-02 (root=default) (1 osds) is down
host k8s-worker-01 (root=default) (1 osds) is down
host k8s-worker-03 (root=default) (1 osds) is down
[WRN] OSD_ROOT_DOWN: 1 root (3 osds) down
root default (3 osds) is down
[WRN] PG_AVAILABILITY: Reduced data availability: 177 pgs inactive
pg 2.8 is stuck inactive for 56m, current state unknown, last acting []
pg 2.9 is stuck inactive for 56m, current state unknown, last acting []
pg 2.a is stuck inactive for 56m, current state unknown, last acting []
pg 2.b is stuck inactive for 56m, current state unknown, last acting []
pg 2.c is stuck inactive for 56m, current state unknown, last acting []
pg 2.d is stuck inactive for 56m, current state unknown, last acting []
pg 2.e is stuck inactive for 56m, current state unknown, last acting []
pg 3.8 is stuck inactive for 56m, current state unknown, last acting []
pg 3.9 is stuck inactive for 56m, current state unknown, last acting []
pg 3.a is stuck inactive for 56m, current state unknown, last acting []
pg 3.b is stuck inactive for 56m, current state unknown, last acting []
pg 3.c is stuck inactive for 56m, current state unknown, last acting []
pg 3.d is stuck inactive for 56m, current state unknown, last acting []
pg 3.f is stuck inactive for 56m, current state unknown, last acting []
pg 4.1 is stuck inactive for 56m, current state unknown, last acting []
pg 5.0 is stuck inactive for 56m, current state unknown, last acting []
pg 5.9 is stuck inactive for 56m, current state unknown, last acting []
pg 5.a is stuck inactive for 56m, current state unknown, last acting []
pg 5.b is stuck inactive for 56m, current state unknown, last acting []
pg 5.c is stuck inactive for 56m, current state unknown, last acting []
pg 5.d is stuck inactive for 56m, current state unknown, last acting []
pg 5.e is stuck inactive for 56m, current state unknown, last acting []
pg 5.f is stuck inactive for 56m, current state unknown, last acting []
pg 8.0 is stuck inactive for 56m, current state unknown, last acting []
pg 8.1 is stuck inactive for 56m, current state unknown, last acting []
pg 8.2 is stuck inactive for 56m, current state unknown, last acting []
pg 8.3 is stuck inactive for 56m, current state unknown, last acting []
pg 8.4 is stuck inactive for 56m, current state unknown, last acting []
pg 8.6 is stuck inactive for 56m, current state unknown, last acting []
pg 8.7 is stuck inactive for 56m, current state unknown, last acting []
pg 9.0 is stuck inactive for 56m, current state unknown, last acting []
pg 9.1 is stuck inactive for 56m, current state unknown, last acting []
pg 9.2 is stuck inactive for 56m, current state unknown, last acting []
pg 9.3 is stuck inactive for 56m, current state unknown, last acting []
pg 9.5 is stuck inactive for 56m, current state unknown, last acting []
pg 9.6 is stuck inactive for 56m, current state unknown, last acting []
pg 9.7 is stuck inactive for 56m, current state unknown, last acting []
pg 10.0 is stuck inactive for 56m, current state unknown, last acting []
pg 10.1 is stuck inactive for 56m, current state unknown, last acting []
pg 10.2 is stuck inactive for 56m, current state unknown, last acting []
pg 10.3 is stuck inactive for 56m, current state unknown, last acting []
pg 10.4 is stuck inactive for 56m, current state unknown, last acting []
pg 10.5 is stuck inactive for 56m, current state unknown, last acting []
pg 10.6 is stuck inactive for 56m, current state unknown, last acting []
pg 11.0 is stuck inactive for 56m, current state unknown, last acting []
pg 11.1 is stuck inactive for 56m, current state unknown, last acting []
pg 11.2 is stuck inactive for 56m, current state unknown, last acting []
pg 11.3 is stuck inactive for 56m, current state unknown, last acting []
pg 11.4 is stuck inactive for 56m, current state unknown, last acting []
pg 11.5 is stuck inactive for 56m, current state unknown, last acting []
pg 11.7 is stuck inactive for 56m, current state unknown, last acting []
Info: Checking if at least three osd pods are running on different nodes
rook-ceph-osd-0-6694cb4845-ghtzt 0/1 Init:0/2 0 30m
rook-ceph-osd-1-7499d8df4f-bnm5k 0/1 CrashLoopBackOff 15 25d
rook-ceph-osd-2-7c98f8c676-cqf2t 0/1 CrashLoopBackOff 15 267d
Info: Pods that are in 'Running' status
NAME READY STATUS RESTARTS AGE
csi-cephfsplugin-7w6gl 3/3 Running 0 25d
csi-cephfsplugin-959qj 3/3 Running 0 410d
csi-cephfsplugin-bkt5b 3/3 Running 0 430d
csi-cephfsplugin-dw25v 3/3 Running 0 437d
csi-cephfsplugin-fkbws 3/3 Running 0 430d
csi-cephfsplugin-hc9ft 3/3 Running 0 456d
csi-cephfsplugin-ncth7 3/3 Running 0 437d
csi-cephfsplugin-provisioner-7dcc8ff54d-bkcpb 6/6 Running 0 19d
csi-cephfsplugin-provisioner-7dcc8ff54d-td9mt 6/6 Running 0 267d
csi-cephfsplugin-q5r4h 3/3 Running 0 410d
csi-cephfsplugin-qbcwc 3/3 Running 0 456d
csi-cephfsplugin-qrzkj 3/3 Running 0 456d
csi-cephfsplugin-zw4jj 3/3 Running 0 19d
csi-rbdplugin-6x5w4 3/3 Running 0 437d
csi-rbdplugin-72xjq 3/3 Running 0 19d
csi-rbdplugin-c2bvt 3/3 Running 0 430d
csi-rbdplugin-cpvpc 3/3 Running 0 456d
csi-rbdplugin-d75nv 3/3 Running 0 410d
csi-rbdplugin-h4mvx 3/3 Running 0 456d
csi-rbdplugin-htxfr 3/3 Running 0 437d
csi-rbdplugin-mn7x5 3/3 Running 0 410d
csi-rbdplugin-provisioner-6799bd4cb7-4dczc 6/6 Running 0 267d
csi-rbdplugin-provisioner-6799bd4cb7-v7s8c 6/6 Running 0 20d
csi-rbdplugin-qcsgp 3/3 Running 0 25d
csi-rbdplugin-r66rn 3/3 Running 0 430d
csi-rbdplugin-wlnqc 3/3 Running 0 456d
rook-ceph-crashcollector-k8s-master-01-78587cd96-5cfhj 1/1 Running 0 19d
rook-ceph-crashcollector-k8s-master-02-5967cc9d47-2k9fz 1/1 Running 0 26h
rook-ceph-crashcollector-k8s-worker-01-658f976cbb-6nln8 1/1 Running 0 15d
rook-ceph-crashcollector-k8s-worker-02-58c4fddfb7-jhz9l 1/1 Running 0 16d
rook-ceph-crashcollector-k8s-worker-03-84bfd94bd4-2sfhg 1/1 Running 0 19d
rook-ceph-crashcollector-k8s-worker-04-7fc567979f-5mdg9 1/1 Running 0 14d
rook-ceph-mds-ceph-filesystem-a-99bb5c94d-t9hxf 1/1 Running 1 20d
rook-ceph-mgr-a-ffb69db56-wpf7f 1/1 Running 0 20d
rook-ceph-mon-h-657654bfff-49dls 1/1 Running 0 36m
rook-ceph-mon-j-6bc66dbb6f-2lhs9 1/1 Running 0 26h
rook-ceph-mon-k-7664dd456c-7bmvv 1/1 Running 0 41m
rook-ceph-operator-6d564f8965-tpnrc 1/1 Running 1 26h
rook-ceph-osd-1-7499d8df4f-bnm5k 0/1 CrashLoopBackOff 15 25d
rook-ceph-osd-2-7c98f8c676-cqf2t 0/1 CrashLoopBackOff 15 267d
rook-ceph-rgw-ceph-objectstore-a-6866bf9596-xzgsw 0/1 CrashLoopBackOff 19 14d
rook-ceph-tools-65c64bbb8b-6g77j 1/1 Running 0 267d
Warning: Pods that are 'Not' in 'Running' status
NAME READY STATUS RESTARTS AGE
rook-ceph-osd-0-6694cb4845-ghtzt 0/1 Init:0/2 0 30m
Info: checking placement group status
Warning: 177 pgs: 177 unknown; 0 B data, 0 B used, 0 B / 0 B avail
Info: checking if at least one mgr pod is running
rook-ceph-mgr-a-ffb69db56-wpf7f Running k8s-master-01
To get the status of the cluster, use kubectl rook-ceph ceph status
cluster:
id: 420185c0-7245-41b0-4d9c-902f5e50c460
health: HEALTH_WARN
1 filesystem is degraded
insufficient standby MDS daemons available
1 MDSs report slow metadata IOs
2 osds down
3 hosts (3 osds) down
1 root (3 osds) down
Reduced data availability: 177 pgs inactive
services:
mon: 3 daemons, quorum h,j,k (age 37m)
mgr: a(active, since 57m)
mds: 1/1 daemons up
osd: 3 osds: 0 up (since 26h), 2 in (since 12d)
data:
volumes: 0/1 healthy, 1 recovering
pools: 11 pools, 177 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs: 100.000% pgs unknown
177 unknown
Environment:
- OS: Debian GNU/Linux 10 (buster)
- Kernel: Linux k8s-worker-01 4.19.0-17-amd64 #1 SMP Debian 4.19.194-3 (2021-07-18) x86_64 GNU/Linux
- Cloud provider or hardware configuration: hardware Proxmox
- Rook version (use
rook versioninside of a Rook Pod): rook: v1.7.1 go: go1.16.3 - Hardware: Proxmox (3 master, 3 etcd, 3 worker)
- Storage backend version (e.g. for ceph do
ceph -v): ceph version 16.2.5 pacific (stable) - Kubernetes version (use
kubectl version): 1.20.7 - Storage backend status (e.g. for Ceph use
ceph healthin the Rook Ceph toolbox): HEALTH_WARN 1 filesystem is degraded; insufficient standby MDS daemons available; 1 MDSs report slow metadata IOs; 2 osds down; 3 hosts (3 osds) down; 1 root (3 osds) down; Reduced data availability: 177 pgs inactive
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 28 (12 by maintainers)
After following your guide and executing it once, the problem still persists, and more OSDs are down. This operation is high high-risk, so proceed with caution.