rook: OSD in CrashLoopBackOff - start interval does not contain the required bound

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior: mons, osd, mds pods don’t start after moving the K8S cluster to another network Expected behavior:

How to reproduce it (minimal and precise): I changed all the IP addresses in my cluster nodes and then I started all the nodes. After that, 3 mon’s, 3 osd, mgr, mds didn’t work for me. Then I repair mon’s using kubectl rook-ceph mons restore-quorum с, but osd’s didn’t work.

Logs to submit:

  • Crashing pod(s) logs: kubectl logs rook-ceph-osd-1-7499d8df4f-bnm5k -n rook-ceph
debug    -74> 2022-11-15T11:05:45.503+0000 7f110888f700 -1 log_channel(cluster) log [ERR] : 11.15s1 past_intervals [1174,10993) start interval does not contain the required bound [887,10993) start
debug    -73> 2022-11-15T11:05:45.503+0000 7f110888f700 -1 osd.1 pg_epoch: 10994 pg[11.15s1( empty local-lis/les=0/0 n=0 ec=1174/1174 lis/c=984/886 les/c/f=985/887/0 sis=10993) [1,NONE,NONE]p1(0) r=-1 lpr=10993 pi=[1174,10993)/4 crt=0'0 mlcod 0'0 unknown mbc={}] 11.15s1 past_intervals [1174,10993) start interval does not contain the required bound [887,10993) start
debug    -72> 2022-11-15T11:05:45.507+0000 7f112bb40080  0 osd.1 11116 done with init, starting boot process
debug    -71> 2022-11-15T11:05:45.507+0000 7f112bb40080 10 monclient: _renew_subs
debug    -70> 2022-11-15T11:05:45.507+0000 7f112bb40080 10 monclient: _send_mon_message to mon.k at v2:10.233.16.22:3300/0
debug    -69> 2022-11-15T11:05:45.507+0000 7f112bb40080  1 osd.1 11116 start_boot

Cluster Status to submit:

  • Output of krew commands, if necessary

    To get the health of the cluster, use kubectl rook-ceph health

Info:  Checking if at least three mon pods are running on different nodes
rook-ceph-mon-h-657654bfff-49dls                          1/1     Running            0          36m
rook-ceph-mon-j-6bc66dbb6f-2lhs9                          1/1     Running            0          26h
rook-ceph-mon-k-7664dd456c-7bmvv                          1/1     Running            0          41m

Info:  Checking mon quorum and ceph health details
Warning:  HEALTH_WARN 1 filesystem is degraded; insufficient standby MDS daemons available; 1 MDSs report slow metadata IOs; 2 osds down; 3 hosts (3 osds) down; 1 root (3 osds) down; Reduced data availability: 177 pgs inactive
[WRN] FS_DEGRADED: 1 filesystem is degraded
    fs ceph-filesystem is degraded
[WRN] MDS_INSUFFICIENT_STANDBY: insufficient standby MDS daemons available
    have 0; want 1 more
[WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs
    mds.ceph-filesystem-a(mds.0): 4 slow metadata IOs are blocked > 30 secs, oldest blocked for 3375 secs
[WRN] OSD_DOWN: 2 osds down
    osd.0 (root=default,host=k8s-worker-01) is down
    osd.1 (root=default,host=k8s-worker-02) is down
[WRN] OSD_HOST_DOWN: 3 hosts (3 osds) down
    host k8s-worker-02 (root=default) (1 osds) is down
    host k8s-worker-01 (root=default) (1 osds) is down
    host k8s-worker-03 (root=default) (1 osds) is down
[WRN] OSD_ROOT_DOWN: 1 root (3 osds) down
    root default (3 osds) is down
[WRN] PG_AVAILABILITY: Reduced data availability: 177 pgs inactive
    pg 2.8 is stuck inactive for 56m, current state unknown, last acting []
    pg 2.9 is stuck inactive for 56m, current state unknown, last acting []
    pg 2.a is stuck inactive for 56m, current state unknown, last acting []
    pg 2.b is stuck inactive for 56m, current state unknown, last acting []
    pg 2.c is stuck inactive for 56m, current state unknown, last acting []
    pg 2.d is stuck inactive for 56m, current state unknown, last acting []
    pg 2.e is stuck inactive for 56m, current state unknown, last acting []
    pg 3.8 is stuck inactive for 56m, current state unknown, last acting []
    pg 3.9 is stuck inactive for 56m, current state unknown, last acting []
    pg 3.a is stuck inactive for 56m, current state unknown, last acting []
    pg 3.b is stuck inactive for 56m, current state unknown, last acting []
    pg 3.c is stuck inactive for 56m, current state unknown, last acting []
    pg 3.d is stuck inactive for 56m, current state unknown, last acting []
    pg 3.f is stuck inactive for 56m, current state unknown, last acting []
    pg 4.1 is stuck inactive for 56m, current state unknown, last acting []
    pg 5.0 is stuck inactive for 56m, current state unknown, last acting []
    pg 5.9 is stuck inactive for 56m, current state unknown, last acting []
    pg 5.a is stuck inactive for 56m, current state unknown, last acting []
    pg 5.b is stuck inactive for 56m, current state unknown, last acting []
    pg 5.c is stuck inactive for 56m, current state unknown, last acting []
    pg 5.d is stuck inactive for 56m, current state unknown, last acting []
    pg 5.e is stuck inactive for 56m, current state unknown, last acting []
    pg 5.f is stuck inactive for 56m, current state unknown, last acting []
    pg 8.0 is stuck inactive for 56m, current state unknown, last acting []
    pg 8.1 is stuck inactive for 56m, current state unknown, last acting []
    pg 8.2 is stuck inactive for 56m, current state unknown, last acting []
    pg 8.3 is stuck inactive for 56m, current state unknown, last acting []
    pg 8.4 is stuck inactive for 56m, current state unknown, last acting []
    pg 8.6 is stuck inactive for 56m, current state unknown, last acting []
    pg 8.7 is stuck inactive for 56m, current state unknown, last acting []
    pg 9.0 is stuck inactive for 56m, current state unknown, last acting []
    pg 9.1 is stuck inactive for 56m, current state unknown, last acting []
    pg 9.2 is stuck inactive for 56m, current state unknown, last acting []
    pg 9.3 is stuck inactive for 56m, current state unknown, last acting []
    pg 9.5 is stuck inactive for 56m, current state unknown, last acting []
    pg 9.6 is stuck inactive for 56m, current state unknown, last acting []
    pg 9.7 is stuck inactive for 56m, current state unknown, last acting []
    pg 10.0 is stuck inactive for 56m, current state unknown, last acting []
    pg 10.1 is stuck inactive for 56m, current state unknown, last acting []
    pg 10.2 is stuck inactive for 56m, current state unknown, last acting []
    pg 10.3 is stuck inactive for 56m, current state unknown, last acting []
    pg 10.4 is stuck inactive for 56m, current state unknown, last acting []
    pg 10.5 is stuck inactive for 56m, current state unknown, last acting []
    pg 10.6 is stuck inactive for 56m, current state unknown, last acting []
    pg 11.0 is stuck inactive for 56m, current state unknown, last acting []
    pg 11.1 is stuck inactive for 56m, current state unknown, last acting []
    pg 11.2 is stuck inactive for 56m, current state unknown, last acting []
    pg 11.3 is stuck inactive for 56m, current state unknown, last acting []
    pg 11.4 is stuck inactive for 56m, current state unknown, last acting []
    pg 11.5 is stuck inactive for 56m, current state unknown, last acting []
    pg 11.7 is stuck inactive for 56m, current state unknown, last acting []

Info:  Checking if at least three osd pods are running on different nodes
rook-ceph-osd-0-6694cb4845-ghtzt                          0/1     Init:0/2           0          30m
rook-ceph-osd-1-7499d8df4f-bnm5k                          0/1     CrashLoopBackOff   15         25d
rook-ceph-osd-2-7c98f8c676-cqf2t                          0/1     CrashLoopBackOff   15         267d

Info:  Pods that are in 'Running' status
NAME                                                      READY   STATUS             RESTARTS   AGE
csi-cephfsplugin-7w6gl                                    3/3     Running            0          25d
csi-cephfsplugin-959qj                                    3/3     Running            0          410d
csi-cephfsplugin-bkt5b                                    3/3     Running            0          430d
csi-cephfsplugin-dw25v                                    3/3     Running            0          437d
csi-cephfsplugin-fkbws                                    3/3     Running            0          430d
csi-cephfsplugin-hc9ft                                    3/3     Running            0          456d
csi-cephfsplugin-ncth7                                    3/3     Running            0          437d
csi-cephfsplugin-provisioner-7dcc8ff54d-bkcpb             6/6     Running            0          19d
csi-cephfsplugin-provisioner-7dcc8ff54d-td9mt             6/6     Running            0          267d
csi-cephfsplugin-q5r4h                                    3/3     Running            0          410d
csi-cephfsplugin-qbcwc                                    3/3     Running            0          456d
csi-cephfsplugin-qrzkj                                    3/3     Running            0          456d
csi-cephfsplugin-zw4jj                                    3/3     Running            0          19d
csi-rbdplugin-6x5w4                                       3/3     Running            0          437d
csi-rbdplugin-72xjq                                       3/3     Running            0          19d
csi-rbdplugin-c2bvt                                       3/3     Running            0          430d
csi-rbdplugin-cpvpc                                       3/3     Running            0          456d
csi-rbdplugin-d75nv                                       3/3     Running            0          410d
csi-rbdplugin-h4mvx                                       3/3     Running            0          456d
csi-rbdplugin-htxfr                                       3/3     Running            0          437d
csi-rbdplugin-mn7x5                                       3/3     Running            0          410d
csi-rbdplugin-provisioner-6799bd4cb7-4dczc                6/6     Running            0          267d
csi-rbdplugin-provisioner-6799bd4cb7-v7s8c                6/6     Running            0          20d
csi-rbdplugin-qcsgp                                       3/3     Running            0          25d
csi-rbdplugin-r66rn                                       3/3     Running            0          430d
csi-rbdplugin-wlnqc                                       3/3     Running            0          456d
rook-ceph-crashcollector-k8s-master-01-78587cd96-5cfhj    1/1     Running            0          19d
rook-ceph-crashcollector-k8s-master-02-5967cc9d47-2k9fz   1/1     Running            0          26h
rook-ceph-crashcollector-k8s-worker-01-658f976cbb-6nln8   1/1     Running            0          15d
rook-ceph-crashcollector-k8s-worker-02-58c4fddfb7-jhz9l   1/1     Running            0          16d
rook-ceph-crashcollector-k8s-worker-03-84bfd94bd4-2sfhg   1/1     Running            0          19d
rook-ceph-crashcollector-k8s-worker-04-7fc567979f-5mdg9   1/1     Running            0          14d
rook-ceph-mds-ceph-filesystem-a-99bb5c94d-t9hxf           1/1     Running            1          20d
rook-ceph-mgr-a-ffb69db56-wpf7f                           1/1     Running            0          20d
rook-ceph-mon-h-657654bfff-49dls                          1/1     Running            0          36m
rook-ceph-mon-j-6bc66dbb6f-2lhs9                          1/1     Running            0          26h
rook-ceph-mon-k-7664dd456c-7bmvv                          1/1     Running            0          41m
rook-ceph-operator-6d564f8965-tpnrc                       1/1     Running            1          26h
rook-ceph-osd-1-7499d8df4f-bnm5k                          0/1     CrashLoopBackOff   15         25d
rook-ceph-osd-2-7c98f8c676-cqf2t                          0/1     CrashLoopBackOff   15         267d
rook-ceph-rgw-ceph-objectstore-a-6866bf9596-xzgsw         0/1     CrashLoopBackOff   19         14d
rook-ceph-tools-65c64bbb8b-6g77j                          1/1     Running            0          267d

Warning:  Pods that are 'Not' in 'Running' status
NAME                                        READY   STATUS      RESTARTS   AGE
rook-ceph-osd-0-6694cb4845-ghtzt            0/1     Init:0/2    0          30m

Info:  checking placement group status
Warning:  177 pgs: 177 unknown; 0 B data, 0 B used, 0 B / 0 B avail

Info:  checking if at least one mgr pod is running
rook-ceph-mgr-a-ffb69db56-wpf7f                           Running     k8s-master-01

To get the status of the cluster, use kubectl rook-ceph ceph status

  cluster:
    id:     420185c0-7245-41b0-4d9c-902f5e50c460
    health: HEALTH_WARN
            1 filesystem is degraded
            insufficient standby MDS daemons available
            1 MDSs report slow metadata IOs
            2 osds down
            3 hosts (3 osds) down
            1 root (3 osds) down
            Reduced data availability: 177 pgs inactive
 
  services:
    mon: 3 daemons, quorum h,j,k (age 37m)
    mgr: a(active, since 57m)
    mds: 1/1 daemons up
    osd: 3 osds: 0 up (since 26h), 2 in (since 12d)
 
  data:
    volumes: 0/1 healthy, 1 recovering
    pools:   11 pools, 177 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:     100.000% pgs unknown
             177 unknown

Environment:

  • OS: Debian GNU/Linux 10 (buster)
  • Kernel: Linux k8s-worker-01 4.19.0-17-amd64 #1 SMP Debian 4.19.194-3 (2021-07-18) x86_64 GNU/Linux
  • Cloud provider or hardware configuration: hardware Proxmox
  • Rook version (use rook version inside of a Rook Pod): rook: v1.7.1 go: go1.16.3
  • Hardware: Proxmox (3 master, 3 etcd, 3 worker)
  • Storage backend version (e.g. for ceph do ceph -v): ceph version 16.2.5 pacific (stable)
  • Kubernetes version (use kubectl version): 1.20.7
  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox): HEALTH_WARN 1 filesystem is degraded; insufficient standby MDS daemons available; 1 MDSs report slow metadata IOs; 2 osds down; 3 hosts (3 osds) down; 1 root (3 osds) down; Reduced data availability: 177 pgs inactive

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 28 (12 by maintainers)

Most upvoted comments

That’s my mistake in OSD log

osd.0 pg_epoch: 2966 pg[11.15s2( empty local-lis/les=0/0 n=0 ec=1174/1174 lis/c=984/886 les/c/f=985/887/0 sis=2966) [1,NONE,0]p1(0) r=2 lpr=2966 pi=[1174,2966)/3 crt=0'0 mlcod 0'0 unknown NOTIFY mbc={}] 11.15s2 past_intervals [1174,2966) start interval does not contain the required bound [887,2966) start
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.5/rpm/el8/BUILD/ceph-16.2.5/src/osd/PeeringState.cc: In function 'void PeeringState::check_past_interval_bounds() const' thread 7f83978c4700 time 2022-11-16T14:22:43.252371+0000

To solve my main problem start interval does not contain the required bound needs:

  1. Connect to node using ssh user@worker-1
  2. Install ceph-objectstore-tool
  3. Run the following command: sudo ceph-objectstore-tool --data-path /var/lib/rook/rook-ceph/420621c0-7245-41b0-8a8c-90264d50c460_07cd101f-244e-412c-b9aa-c521b4716a00 --pgid 11.15s2 --op export-remove > osd.1-11.15s2.bin

After following your guide and executing it once, the problem still persists, and more OSDs are down. This operation is high high-risk, so proceed with caution.