rook: ECBackend.cc/ECUtil.cc assert failure after EC spec change
Is this a bug report or feature request?
- Bug Report
Deviation from expected behavior:
Since adjusting the EC configuration, my osd pods have been consistently having assert failures, and causing the OSD container within the pods to restart. This has caused chaos within the cluster.
This behavior only seems to be effecting the OSDs of the device class where the EC were stored (IE hdd but not ssd). I only had my EC filesystem on hdd. SSD is behaving normally.
Expected behavior:
After adjusting the erasure coded file system definition (codingChucks/dataChunks), the system will adjust the existing data on the fly without causing a system outage. However, some data unavailability while things are reconfigured would be expected.
How to reproduce it (minimal and precise):
I’m not sure about the minimal reproduction steps, but my steps were to adjust the configuration of the erasure encoded spec: codingChunks: 2 (no change) dataChunks: 7 --> 4
File(s) to submit:
- Cluster CR (custom resource), typically called
cluster.yaml, if necessary
Logs to submit: OSD Errors seen:
type 1
debug -2> 2024-07-08T13:10:35.864+0000 7f5f02139700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/osd/ECUtil.cc: In function 'int ECUtil::decode(const ECUtil::stripe_info_t&, ceph::ErasureCodeInterfaceRef&, std::map<int, ceph::buffer::v15_2_0::list>&, std::map<int, ceph::buffer::v15_2_0::list*>&)' thread 7f5f02139700 time 2024-07-08T13:10:35.863236+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/osd/ECUtil.cc: 110: FAILED ceph_assert(r == 0)
type 2
debug -7> 2024-07-08T13:28:12.232+0000 7f0ad0a42700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/osd/ECBackend.cc: In function 'void ECBackend::continue_recovery_op(ECBackend::RecoveryOp&, RecoveryMessages*)' thread 7f0ad0a42700 time 2024-07-08T13:28:12.215978+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/osd/ECBackend.cc: 676: FAILED ceph_assert(pop.data.length() == sinfo.aligned_logical_offset_to_chunk_offset( after_progress.data_recovered_to - op.recovery_progress.data_recovered_to))
Cluster Status to submit:
-
Output of kubectl commands, if necessary
To get the health of the cluster, use
kubectl rook-ceph health:
A majority of this is normal, minus what is below.
Error: HEALTH_ERR
PG ERRORS:
Info: Checking placement group status
Warning: PgState: undersized+degraded+peered, PgCount: 231
Warning: PgState: active+undersized+degraded, PgCount: 123
Info: PgState: active+clean, PgCount: 111
Error: PgState: stale+down, PgCount: 97
Error: PgState: down, PgCount: 77
Warning: PgState: stale+undersized+degraded+peered, PgCount: 76
Error: PgState: down+remapped, PgCount: 39
Warning: PgState: peering, PgCount: 20
Warning: PgState: stale+activating+undersized+degraded, PgCount: 19
Warning: PgState: stale+active+undersized+degraded, PgCount: 17
Warning: PgState: stale+peering, PgCount: 13
Warning: PgState: stale+active+clean, PgCount: 6
Warning: PgState: activating+undersized, PgCount: 4
Warning: PgState: active+clean+scrubbing+deep, PgCount: 2
Error: PgState: down+remapped+inconsistent, PgCount: 2
Warning: PgState: activating, PgCount: 2
Warning: PgState: activating+undersized+degraded, PgCount: 1
Error: PgState: down+inconsistent, PgCount: 1
Warning: PgState: stale+undersized+degraded+inconsistent+peered, PgCount: 1
To get the status of the cluster, use kubectl rook-ceph ceph status:
Info: running 'ceph' command with args: [status]
cluster:
id: 6ce19b86-31ec-43e9-9f22-ce5db4d1200a
health: HEALTH_ERR
1 MDSs report slow metadata IOs
mon q is low on available space
13 osds down
1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set
4 hosts (4 osds) down
436 scrub errors
Reduced data availability: 434 pgs inactive, 231 pgs down, 7 pgs peering, 40 pgs stale
Possible data damage: 2 pgs inconsistent
Degraded data redundancy: 2813036/66360450 objects degraded (4.239%), 498 pgs degraded, 116 pgs undersized
624 pgs not deep-scrubbed in time
607 pgs not scrubbed in time
1867 daemons have recently crashed
2 mgr modules have recently crashed
services:
mon: 3 daemons, quorum p,q,r (age 3d)
mgr: a(active, since 4d), standbys: b
mds: 1/1 daemons up, 1 hot standby
osd: 43 osds: 26 up (since 9s), 39 in (since 2h); 41 remapped pgs
data:
volumes: 1/1 healthy
pools: 6 pools, 842 pgs
objects: 8.83M objects, 33 TiB
usage: 75 TiB used, 115 TiB / 190 TiB avail
pgs: 70.190% pgs not active
2813036/66360450 objects degraded (4.239%)
263 undersized+degraded+peered
127 down
123 active+undersized+degraded
99 active+clean
88 stale+undersized+degraded+peered
62 stale+down
40 down+remapped
24 stale+active+undersized+degraded
9 stale+peering
3 active+clean+scrubbing+deep
2 stale+active+clean
1 down+inconsistent
1 down+remapped+inconsistent
io:
client: 1.1 KiB/s rd, 6.5 KiB/s wr, 2 op/s rd, 0 op/s wr
recovery: 49 MiB/s, 12 objects/s
For more details, see the Rook kubectl Plugin
Environment:
- OS (e.g. from /etc/os-release): Ubuntu 22.04
- Kernel (e.g.
uname -a): 5.15.0-1061-kvm #66-Ubuntu SMP Mon Jun 17 20:01:21 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux - Cloud provider or hardware configuration: Proxmox VM
- Rook version (use
rook versioninside of a Rook Pod): v1.14.8 - Storage backend version (e.g. for ceph do
ceph -v): v18.2.2 - Kubernetes version (use
kubectl version): v1.28.9+rke2r1 - Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): RKE2
- Storage backend status (e.g. for Ceph use
ceph healthin the Rook Ceph toolbox):
HEALTH_ERR 1 MDSs report slow metadata IOs; mon q is low on available space; 11 osds down; 1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set; 4 hosts (4 osds) down; 658 scrub errors; Reduced data availability: 476 pgs inactive, 176 pgs down, 104 pgs stale; Possible data damage: 3 pgs inconsistent; Degraded data redundancy: 2827423/78724353 objects degraded (3.592%), 528 pgs degraded, 411 pgs undersized; 624 pgs not deep-scrubbed in time; 607 pgs not scrubbed in time; 1867 daemons have recently crashed; 2 mgr modules have recently crashed
Extra Info: The four osd/nodes down, are expected. I was working on purging these when this issues popped up.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 22 (11 by maintainers)
@rzarzynski Thanks for the clarification. Looking again, I don’t see a requirement where we were trying to support updating the EC params, it was just a side effect of some other issue where we wanted to force update the EC profile if a pool was deleted and re-created (see #7820). So we will restrict these parameters from being updated in the future.