rook: ECBackend.cc/ECUtil.cc assert failure after EC spec change

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior:

Since adjusting the EC configuration, my osd pods have been consistently having assert failures, and causing the OSD container within the pods to restart. This has caused chaos within the cluster.

This behavior only seems to be effecting the OSDs of the device class where the EC were stored (IE hdd but not ssd). I only had my EC filesystem on hdd. SSD is behaving normally.

Expected behavior:

After adjusting the erasure coded file system definition (codingChucks/dataChunks), the system will adjust the existing data on the fly without causing a system outage. However, some data unavailability while things are reconfigured would be expected.

How to reproduce it (minimal and precise):

I’m not sure about the minimal reproduction steps, but my steps were to adjust the configuration of the erasure encoded spec: codingChunks: 2 (no change) dataChunks: 7 --> 4

File(s) to submit:

  • Cluster CR (custom resource), typically called cluster.yaml, if necessary

Logs to submit: OSD Errors seen:

type 1

debug     -2> 2024-07-08T13:10:35.864+0000 7f5f02139700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/osd/ECUtil.cc: In function 'int ECUtil::decode(const ECUtil::stripe_info_t&, ceph::ErasureCodeInterfaceRef&, std::map<int, ceph::buffer::v15_2_0::list>&, std::map<int, ceph::buffer::v15_2_0::list*>&)' thread 7f5f02139700 time 2024-07-08T13:10:35.863236+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/osd/ECUtil.cc: 110: FAILED ceph_assert(r == 0)

type 2

debug     -7> 2024-07-08T13:28:12.232+0000 7f0ad0a42700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/osd/ECBackend.cc: In function 'void ECBackend::continue_recovery_op(ECBackend::RecoveryOp&, RecoveryMessages*)' thread 7f0ad0a42700 time 2024-07-08T13:28:12.215978+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/osd/ECBackend.cc: 676: FAILED ceph_assert(pop.data.length() == sinfo.aligned_logical_offset_to_chunk_offset( after_progress.data_recovered_to - op.recovery_progress.data_recovered_to))

Cluster Status to submit:

  • Output of kubectl commands, if necessary

    To get the health of the cluster, use kubectl rook-ceph health:

A majority of this is normal, minus what is below.

Error: HEALTH_ERR
PG ERRORS: 
Info: Checking placement group status
Warning:        PgState: undersized+degraded+peered, PgCount: 231
Warning:        PgState: active+undersized+degraded, PgCount: 123
Info:   PgState: active+clean, PgCount: 111
Error:  PgState: stale+down, PgCount: 97
Error:  PgState: down, PgCount: 77
Warning:        PgState: stale+undersized+degraded+peered, PgCount: 76
Error:  PgState: down+remapped, PgCount: 39
Warning:        PgState: peering, PgCount: 20
Warning:        PgState: stale+activating+undersized+degraded, PgCount: 19
Warning:        PgState: stale+active+undersized+degraded, PgCount: 17
Warning:        PgState: stale+peering, PgCount: 13
Warning:        PgState: stale+active+clean, PgCount: 6
Warning:        PgState: activating+undersized, PgCount: 4
Warning:        PgState: active+clean+scrubbing+deep, PgCount: 2
Error:  PgState: down+remapped+inconsistent, PgCount: 2
Warning:        PgState: activating, PgCount: 2
Warning:        PgState: activating+undersized+degraded, PgCount: 1
Error:  PgState: down+inconsistent, PgCount: 1
Warning:        PgState: stale+undersized+degraded+inconsistent+peered, PgCount: 1

To get the status of the cluster, use kubectl rook-ceph ceph status:

Info: running 'ceph' command with args: [status]
  cluster:
    id:     6ce19b86-31ec-43e9-9f22-ce5db4d1200a
    health: HEALTH_ERR
            1 MDSs report slow metadata IOs
            mon q is low on available space
            13 osds down
            1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set
            4 hosts (4 osds) down
            436 scrub errors
            Reduced data availability: 434 pgs inactive, 231 pgs down, 7 pgs peering, 40 pgs stale
            Possible data damage: 2 pgs inconsistent
            Degraded data redundancy: 2813036/66360450 objects degraded (4.239%), 498 pgs degraded, 116 pgs undersized
            624 pgs not deep-scrubbed in time
            607 pgs not scrubbed in time
            1867 daemons have recently crashed
            2 mgr modules have recently crashed
 
  services:
    mon: 3 daemons, quorum p,q,r (age 3d)
    mgr: a(active, since 4d), standbys: b
    mds: 1/1 daemons up, 1 hot standby
    osd: 43 osds: 26 up (since 9s), 39 in (since 2h); 41 remapped pgs
 
  data:
    volumes: 1/1 healthy
    pools:   6 pools, 842 pgs
    objects: 8.83M objects, 33 TiB
    usage:   75 TiB used, 115 TiB / 190 TiB avail
    pgs:     70.190% pgs not active
             2813036/66360450 objects degraded (4.239%)
             263 undersized+degraded+peered
             127 down
             123 active+undersized+degraded
             99  active+clean
             88  stale+undersized+degraded+peered
             62  stale+down
             40  down+remapped
             24  stale+active+undersized+degraded
             9   stale+peering
             3   active+clean+scrubbing+deep
             2   stale+active+clean
             1   down+inconsistent
             1   down+remapped+inconsistent
 
  io:
    client:   1.1 KiB/s rd, 6.5 KiB/s wr, 2 op/s rd, 0 op/s wr
    recovery: 49 MiB/s, 12 objects/s

For more details, see the Rook kubectl Plugin

Environment:

  • OS (e.g. from /etc/os-release): Ubuntu 22.04
  • Kernel (e.g. uname -a): 5.15.0-1061-kvm #66-Ubuntu SMP Mon Jun 17 20:01:21 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
  • Cloud provider or hardware configuration: Proxmox VM
  • Rook version (use rook version inside of a Rook Pod): v1.14.8
  • Storage backend version (e.g. for ceph do ceph -v): v18.2.2
  • Kubernetes version (use kubectl version): v1.28.9+rke2r1
  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): RKE2
  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox):
HEALTH_ERR 1 MDSs report slow metadata IOs; mon q is low on available space; 11 osds down; 1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set; 4 hosts (4 osds) down; 658 scrub errors; Reduced data availability: 476 pgs inactive, 176 pgs down, 104 pgs stale; Possible data damage: 3 pgs inconsistent; Degraded data redundancy: 2827423/78724353 objects degraded (3.592%), 528 pgs degraded, 411 pgs undersized; 624 pgs not deep-scrubbed in time; 607 pgs not scrubbed in time; 1867 daemons have recently crashed; 2 mgr modules have recently crashed

Extra Info: The four osd/nodes down, are expected. I was working on purging these when this issues popped up.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 22 (11 by maintainers)

Most upvoted comments

@rzarzynski Thanks for the clarification. Looking again, I don’t see a requirement where we were trying to support updating the EC params, it was just a side effect of some other issue where we wanted to force update the EC profile if a pool was deleted and re-created (see #7820). So we will restrict these parameters from being updated in the future.