longhorn: [BUG] Pods crash with disk error, EXT4 error, high cpu load on node
Describe the bug Sporadicly pods - with a persistent volume running with longhorn - crashes. Sometimes the restarted pod is not able to attach the pvc again at the first few times. The node running the affected pod runs several other pods wich do not crash. I was not able to find useful logs in the longhorn logs, but I found filesystem errors of longhorn’s virtual volumes in the kernel log of the node. Additionally, the node running the pod and also other nodes of the k3s cluster have high cpu load at the time.
Maybe related to #2261
To Reproduce Error occurrs infrequently on several different pods and nodes.
Expected behavior The volume should not get unavailable and the pod should not crash.
Log longhorn-support-bundle_7e49028f-f78c-4669-a1ed-8a9443daf8b5_2021-07-07T08-43-19Z.zip
Kernel Log
Jul 6 14:45:33 adjucator kernel: [23767.717584] sd 3:0:0:1: [sdb] tag#50 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 6 14:45:33 adjucator kernel: [23767.717587] sd 3:0:0:1: [sdb] tag#50 Sense Key : Medium Error [current]
Jul 6 14:45:33 adjucator kernel: [23767.717623] sd 3:0:0:1: [sdb] tag#50 Add. Sense: Unrecovered read error
Jul 6 14:45:33 adjucator kernel: [23767.717625] sd 3:0:0:1: [sdb] tag#50 CDB: Write(10) 2a 00 00 8b d8 38 00 07 58 00
Jul 6 14:45:33 adjucator kernel: [23767.717627] blk_update_request: critical medium error, dev sdb, sector 9164856 op 0x1:(WRITE) flags 0x0 phys_seg 235 prio class 0
Jul 6 14:45:33 adjucator kernel: [23767.718029] EXT4-fs warning (device sdb): ext4_end_bio:309: I/O error 7 writing to inode 39059461 (offset 0 size 2039808 starting block 1145842)
Jul 6 14:45:33 adjucator kernel: [23767.718032] Buffer I/O error on device sdb, logical block 1145607
Jul 6 14:45:33 adjucator kernel: [23767.718136] sd 3:0:0:1: [sdb] tag#49 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 6 14:45:33 adjucator kernel: [23767.718300] sd 3:0:0:1: [sdb] tag#49 Sense Key : Medium Error [current]
Jul 6 14:45:33 adjucator kernel: [23767.718301] sd 3:0:0:1: [sdb] tag#49 Add. Sense: Unrecovered read error
Jul 6 14:45:33 adjucator kernel: [23767.718304] Buffer I/O error on device sdb, logical block 1145608
Jul 6 14:45:33 adjucator kernel: [23767.718305] sd 3:0:0:1: [sdb] tag#49 CDB: Write(10) 2a 00 00 8b d0 00 00 08 38 00
Jul 6 14:45:33 adjucator kernel: [23767.718308] blk_update_request: critical medium error, dev sdb, sector 9162752 op 0x1:(WRITE) flags 0x4000 phys_seg 263 prio class 0
Jul 6 14:45:33 adjucator kernel: [23767.718468] Buffer I/O error on device sdb, logical block 1145609
Jul 6 14:45:33 adjucator kernel: [23767.718838] EXT4-fs warning (device sdb): ext4_end_bio:309: I/O error 7 writing to inode 39059461 (offset 0 size 2039808 starting block 1145607)
Jul 6 14:45:33 adjucator kernel: [23767.718979] Buffer I/O error on device sdb, logical block 1145344
Jul 6 14:45:33 adjucator kernel: [23767.718982] Buffer I/O error on device sdb, logical block 1145610
Jul 6 14:45:33 adjucator kernel: [23767.719205] Buffer I/O error on device sdb, logical block 1145345
Jul 6 14:45:33 adjucator kernel: [23767.719349] Buffer I/O error on device sdb, logical block 1145611
Jul 6 14:45:33 adjucator kernel: [23767.719490] Buffer I/O error on device sdb, logical block 1145346
Jul 6 14:45:33 adjucator kernel: [23767.719634] Buffer I/O error on device sdb, logical block 1145612
Jul 6 14:45:33 adjucator kernel: [23767.719803] Buffer I/O error on device sdb, logical block 1145347
Jul 6 14:45:34 adjucator kernel: [23768.755711] blk_update_request: I/O error, dev sdf, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0
Jul 6 14:45:34 adjucator kernel: [23768.762217] sd 7:0:0:1: [sdf] Synchronizing SCSI cache
Jul 6 14:45:34 adjucator kernel: [23768.848044] blk_update_request: I/O error, dev sdd, sector 193304880 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Jul 6 14:45:34 adjucator kernel: [23768.848342] EXT4-fs warning (device sdd): ext4_end_bio:309: I/O error 10 writing to inode 6030350 (offset 0 size 0 starting block 24163111)
Jul 6 14:45:34 adjucator kernel: [23768.848362] blk_update_request: I/O error, dev sdd, sector 193215376 op 0x1:(WRITE) flags 0x0 phys_seg 2 prio class 0
Jul 6 14:45:34 adjucator kernel: [23768.848531] EXT4-fs warning (device sdd): ext4_end_bio:309: I/O error 10 writing to inode 6033980 (offset 0 size 0 starting block 24151924)
Jul 6 14:45:34 adjucator kernel: [23768.848535] blk_update_request: I/O error, dev sdd, sector 193301208 op 0x1:(WRITE) flags 0x0 phys_seg 6 prio class 0
Jul 6 14:45:34 adjucator kernel: [23768.848707] EXT4-fs warning (device sdd): ext4_end_bio:309: I/O error 10 writing to inode 6030575 (offset 0 size 24576 starting block 24162657)
Jul 6 14:45:34 adjucator kernel: [23768.848712] blk_update_request: I/O error, dev sdd, sector 193303328 op 0x1:(WRITE) flags 0x0 phys_seg 59 prio class 0
Jul 6 14:45:34 adjucator kernel: [23768.848888] EXT4-fs warning (device sdd): ext4_end_bio:309: I/O error 10 writing to inode 6034257 (offset 0 size 86016 starting block 24162937)
Jul 6 14:45:34 adjucator kernel: [23768.848894] EXT4-fs warning (device sdd): ext4_end_bio:309: I/O error 10 writing to inode 6031134 (offset 0 size 155648 starting block 24162975)
Jul 6 14:45:34 adjucator kernel: [23768.848904] blk_update_request: I/O error, dev sdd, sector 193205112 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Jul 6 14:45:34 adjucator kernel: [23768.849109] EXT4-fs warning (device sdd): ext4_end_bio:309: I/O error 10 writing to inode 6034267 (offset 0 size 4096 starting block 24150640)
Jul 6 14:45:34 adjucator kernel: [23768.849113] blk_update_request: I/O error, dev sdd, sector 193211232 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Jul 6 14:45:34 adjucator kernel: [23768.849293] EXT4-fs warning (device sdd): ext4_end_bio:309: I/O error 10 writing to inode 6031717 (offset 0 size 4096 starting block 24151405)
Jul 6 14:45:34 adjucator kernel: [23768.855333] blk_update_request: I/O error, dev sdd, sector 0 op 0x1:(WRITE) flags 0x0 phys_seg 2 prio class 0
Jul 6 14:45:34 adjucator kernel: [23768.855799] Buffer I/O error on dev sdd, logical block 0, lost async page write
Jul 6 14:45:34 adjucator kernel: [23768.856153] Buffer I/O error on dev sdd, logical block 1, lost async page write
Jul 6 14:45:34 adjucator kernel: [23768.856517] Buffer I/O error on dev sdd, logical block 1039, lost async page write
Jul 6 14:45:34 adjucator kernel: [23768.856760] Buffer I/O error on dev sdd, logical block 24117264, lost async page write
Jul 6 14:45:34 adjucator kernel: [23768.857028] Buffer I/O error on dev sdd, logical block 24117280, lost async page write
Jul 6 14:45:34 adjucator kernel: [23768.857303] Buffer I/O error on dev sdd, logical block 24117281, lost async page write
Jul 6 14:45:34 adjucator kernel: [23768.857532] Buffer I/O error on dev sdd, logical block 24117285, lost async page write
Jul 6 14:45:34 adjucator kernel: [23768.857763] Buffer I/O error on dev sdd, logical block 24117291, lost async page write
Jul 6 14:45:34 adjucator kernel: [23768.858054] Buffer I/O error on dev sdd, logical block 24117327, lost async page write
Jul 6 14:45:34 adjucator kernel: [23768.858373] Buffer I/O error on dev sdd, logical block 24117345, lost async page write
Jul 6 14:45:34 adjucator kernel: [23768.858713] JBD2: Detected IO errors while flushing file data on sdd-8
Jul 6 14:45:34 adjucator kernel: [23768.858718] Aborting journal on device sdd-8.
Jul 6 14:45:34 adjucator kernel: [23768.859014] JBD2: Error -5 detected when updating journal superblock for sdd-8.
Jul 6 14:45:34 adjucator kernel: [23768.889107] JBD2: Error while async write back metadata bh 0.
Jul 6 14:45:34 adjucator kernel: [23768.889110] JBD2: Error while async write back metadata bh 1.
Jul 6 14:45:34 adjucator kernel: [23768.889453] JBD2: Error while async write back metadata bh 1039.
Jul 6 14:45:34 adjucator kernel: [23768.895789] JBD2: Error while async write back metadata bh 24117264.
Jul 6 14:45:34 adjucator kernel: [23768.896107] JBD2: Error while async write back metadata bh 24117280.
Jul 6 14:45:34 adjucator kernel: [23768.896513] JBD2: Error while async write back metadata bh 24117281.
Jul 6 14:45:34 adjucator kernel: [23768.896844] JBD2: Error while async write back metadata bh 24117285.
Jul 6 14:45:34 adjucator kernel: [23768.897140] JBD2: Error while async write back metadata bh 24117291.
Jul 6 14:45:34 adjucator kernel: [23768.897799] JBD2: Error while async write back metadata bh 24117327.
Jul 6 14:45:34 adjucator kernel: [23768.898199] JBD2: Error while async write back metadata bh 24117345.
Jul 6 14:45:34 adjucator kernel: [23768.898559] JBD2: Error while async write back metadata bh 24117359.
Jul 6 14:45:34 adjucator kernel: [23768.898899] JBD2: Error while async write back metadata bh 24117412.
Jul 6 14:45:34 adjucator kernel: [23768.900758] JBD2: Error while async write back metadata bh 24117432.
Jul 6 14:45:34 adjucator kernel: [23768.901265] JBD2: Error while async write back metadata bh 24117466.
Jul 6 14:45:34 adjucator kernel: [23768.901628] JBD2: Error while async write back metadata bh 24117468.
Jul 6 14:45:34 adjucator kernel: [23768.901958] JBD2: Error while async write back metadata bh 24117487.
Jul 6 14:45:34 adjucator kernel: [23768.902308] JBD2: Error while async write back metadata bh 24117505.
Jul 6 14:45:34 adjucator kernel: [23768.902628] JBD2: Error while async write back metadata bh 24117548.
Jul 6 14:45:34 adjucator kernel: [23768.902912] JBD2: Error while async write back metadata bh 24117551.
Jul 6 14:45:34 adjucator kernel: [23768.904598] JBD2: Error while async write back metadata bh 24117553.
Jul 6 14:45:34 adjucator kernel: [23768.904880] JBD2: Error while async write back metadata bh 24117554.
Jul 6 14:45:34 adjucator kernel: [23768.905159] JBD2: Error while async write back metadata bh 24117564.
Jul 6 14:45:34 adjucator kernel: [23768.906058] JBD2: Error while async write back metadata bh 24117571.
Jul 6 14:45:34 adjucator kernel: [23768.906307] JBD2: Error while async write back metadata bh 24117573.
Jul 6 14:45:34 adjucator kernel: [23768.906541] JBD2: Error while async write back metadata bh 24117577.
Jul 6 14:45:34 adjucator kernel: [23768.906768] JBD2: Error while async write back metadata bh 24117580.
Jul 6 14:45:34 adjucator kernel: [23768.907256] JBD2: Error while async write back metadata bh 24117581.
Jul 6 14:45:34 adjucator kernel: [23768.907471] JBD2: Error while async write back metadata bh 24117582.
Jul 6 14:45:34 adjucator kernel: [23768.907673] JBD2: Error while async write back metadata bh 24117583.
Jul 6 14:45:34 adjucator kernel: [23768.907867] JBD2: Error while async write back metadata bh 24117584.
Jul 6 14:45:34 adjucator kernel: [23768.908087] JBD2: Error while async write back metadata bh 24117585.
Jul 6 14:45:34 adjucator kernel: [23768.908270] JBD2: Error while async write back metadata bh 24117586.
Jul 6 14:45:34 adjucator kernel: [23768.908440] JBD2: Error while async write back metadata bh 24117593.
Jul 6 14:45:34 adjucator kernel: [23768.908603] JBD2: Error while async write back metadata bh 24117595.
Jul 6 14:45:34 adjucator kernel: [23768.908765] JBD2: Error while async write back metadata bh 24117596.
Jul 6 14:45:34 adjucator kernel: [23768.908922] JBD2: Error while async write back metadata bh 24117598.
Jul 6 14:45:34 adjucator kernel: [23768.909069] JBD2: Error while async write back metadata bh 24117599.
Jul 6 14:45:34 adjucator kernel: [23768.909205] JBD2: Error while async write back metadata bh 24117603.
Jul 6 14:45:34 adjucator kernel: [23768.909347] JBD2: Error while async write back metadata bh 24117605.
Jul 6 14:45:34 adjucator kernel: [23768.909485] JBD2: Error while async write back metadata bh 24117607.
Jul 6 14:45:34 adjucator kernel: [23768.909607] JBD2: Error while async write back metadata bh 24117608.
Jul 6 14:45:34 adjucator kernel: [23768.909728] JBD2: Error while async write back metadata bh 24117610.
Jul 6 14:45:34 adjucator kernel: [23768.909854] JBD2: Error while async write back metadata bh 24117612.
Jul 6 14:45:34 adjucator kernel: [23768.909975] JBD2: Error while async write back metadata bh 24117613.
Jul 6 14:45:34 adjucator kernel: [23768.910096] JBD2: Error while async write back metadata bh 24117618.
Jul 6 14:45:34 adjucator kernel: [23768.910222] JBD2: Error while async write back metadata bh 24117619.
Jul 6 14:45:34 adjucator kernel: [23768.910345] JBD2: Error while async write back metadata bh 24125472.
Jul 6 14:45:34 adjucator kernel: [23768.910461] JBD2: Error while async write back metadata bh 24125474.
Jul 6 14:45:34 adjucator kernel: [23768.910562] JBD2: Error while async write back metadata bh 24125476.
Jul 6 14:45:34 adjucator kernel: [23768.910660] JBD2: Error while async write back metadata bh 24125478.
Jul 6 14:45:34 adjucator kernel: [23768.910752] JBD2: Error while async write back metadata bh 24125482.
Jul 6 14:45:34 adjucator kernel: [23768.910845] JBD2: Error while async write back metadata bh 24125492.
Jul 6 14:45:34 adjucator kernel: [23768.910937] JBD2: Error while async write back metadata bh 24125493.
Jul 6 14:45:34 adjucator kernel: [23768.912003] JBD2: Error while async write back metadata bh 24125509.
Jul 6 14:45:34 adjucator kernel: [23768.912134] JBD2: Error while async write back metadata bh 24125511.
Jul 6 14:45:34 adjucator kernel: [23768.912228] JBD2: Error while async write back metadata bh 24125513.
Jul 6 14:45:34 adjucator kernel: [23768.912321] JBD2: Error while async write back metadata bh 24125515.
Jul 6 14:45:34 adjucator kernel: [23768.912411] JBD2: Error while async write back metadata bh 24125517.
Jul 6 14:45:34 adjucator kernel: [23768.912502] JBD2: Error while async write back metadata bh 24125519.
Jul 6 14:45:34 adjucator kernel: [23768.912591] JBD2: Error while async write back metadata bh 24125521.
Jul 6 14:45:34 adjucator kernel: [23768.912685] JBD2: Error while async write back metadata bh 24125523.
Jul 6 14:45:34 adjucator kernel: [23768.912772] JBD2: Error while async write back metadata bh 24125525.
Jul 6 14:45:34 adjucator kernel: [23768.912860] JBD2: Error while async write back metadata bh 24125527.
Jul 6 14:45:34 adjucator kernel: [23768.912947] JBD2: Error while async write back metadata bh 24150293.
Jul 6 14:45:34 adjucator kernel: [23768.954914] sd 5:0:0:1: [sdd] Synchronizing SCSI cache
Jul 6 14:45:34 adjucator kernel: [23769.187642] Aborting journal on device sdc-8.
Jul 6 14:45:34 adjucator kernel: [23769.191009] JBD2: Error -5 detected when updating journal superblock for sdc-8.
Jul 6 14:45:35 adjucator kernel: [23769.369411] JBD2: Error while async write back metadata bh 0.
Jul 6 14:45:35 adjucator kernel: [23769.369414] JBD2: Error while async write back metadata bh 1.
Jul 6 14:45:35 adjucator kernel: [23769.369667] JBD2: Error while async write back metadata bh 1042.
Jul 6 14:45:35 adjucator kernel: [23769.369806] JBD2: Error while async write back metadata bh 1048577.
Jul 6 14:45:35 adjucator kernel: [23769.369933] JBD2: Error while async write back metadata bh 1048592.
Jul 6 14:45:35 adjucator kernel: [23769.370063] JBD2: Error while async write back metadata bh 1048609.
Jul 6 14:45:35 adjucator kernel: [23769.370277] JBD2: Error while async write back metadata bh 1048610.
Jul 6 14:45:35 adjucator kernel: [23769.370414] JBD2: Error while async write back metadata bh 1048611.
Jul 6 14:45:35 adjucator kernel: [23769.370540] JBD2: Error while async write back metadata bh 1056800.
Jul 6 14:45:35 adjucator kernel: [23769.370664] JBD2: Error while async write back metadata bh 1056801.
Jul 6 14:45:35 adjucator kernel: [23769.468085] sd 4:0:0:1: [sdc] Synchronizing SCSI cache
Jul 6 14:45:35 adjucator kernel: [23769.623156] sd 8:0:0:1: [sdg] Synchronizing SCSI cache
Jul 6 14:45:35 adjucator kernel: [23769.892361] cni0: port 29(veth1168f6fa) entered disabled state
Jul 6 14:45:35 adjucator kernel: [23769.901079] device veth1168f6fa left promiscuous mode
Jul 6 14:45:35 adjucator kernel: [23769.901086] cni0: port 29(veth1168f6fa) entered disabled state
Jul 6 14:45:37 adjucator kernel: [23771.833988] EXT4-fs error (device sdd): __ext4_find_entry:1531: inode #6029319: comm postgres: reading directory lblock 0
Jul 6 14:45:37 adjucator kernel: [23771.835547] EXT4-fs error (device sdd): ext4_journal_check_start:61: Detected aborted journal
Jul 6 14:45:37 adjucator kernel: [23771.835845] EXT4-fs (sdd): Remounting filesystem read-only
Jul 6 14:45:37 adjucator kernel: [23771.836214] ------------[ cut here ]------------
Jul 6 14:45:37 adjucator kernel: [23771.836214] bdi-block not registered
Jul 6 14:45:37 adjucator kernel: [23771.836261] WARNING: CPU: 1 PID: 2601316 at fs/fs-writeback.c:2330 __mark_inode_dirty+0x19b/0x3a0
Jul 6 14:45:37 adjucator kernel: [23771.836262] Modules linked in: nf_conntrack_netlink iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi xt_multiport veth ip6table_nat ip6_tables xt_recent xt_nat xt_statistic xt_MASQUERADE xt_mark xt_tcpudp vxlan ip6_udp_tunnel udp_tunnel ipt_REJECT nf_reject_ipv4 xt_physdev xt_addrtype xt_set xt_conntrack xt_comment ip_set_hash_ip iptable_filter iptable_mangle bpfilter ip_set nfnetlink ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 overlay br_netfilter bridge stp llc dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua input_leds joydev serio_raw mac_hid qemu_fw_cfg sch_fq_codel ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid bochs_drm drm_vram_helper crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ttm drm_kms_helper aesni_intel crypto_simd syscopyarea sysfillrect cryptd glue_helper sysimgblt
Jul 6 14:45:37 adjucator kernel: [23771.836302] virtio_net fb_sys_fops psmouse net_failover virtio_scsi failover drm i2c_piix4 pata_acpi floppy
Jul 6 14:45:37 adjucator kernel: [23771.836309] CPU: 1 PID: 2601316 Comm: postgres Not tainted 5.4.0-66-generic #74-Ubuntu
Jul 6 14:45:37 adjucator kernel: [23771.836310] Hardware name: netcup KVM Server, BIOS RS 2000 G9 10/06/2020
Jul 6 14:45:37 adjucator kernel: [23771.836312] RIP: 0010:__mark_inode_dirty+0x19b/0x3a0
Jul 6 14:45:37 adjucator kernel: [23771.836313] Code: ff ff 8b 4d d4 49 89 c5 48 8b 00 f6 40 5c 02 75 1e 49 8b 55 08 83 e2 01 75 15 48 8b 70 50 48 c7 c7 cb 91 38 8a e8 30 d0 d8 ff <0f> 0b 8b 4d d4 48 8b 05 c9 21 2f 01 49 89 84 24 c8 00 00 00 85 c9
Jul 6 14:45:37 adjucator kernel: [23771.836314] RSP: 0018:ffffac40c1e8bcc8 EFLAGS: 00010282
Jul 6 14:45:37 adjucator kernel: [23771.836315] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000006
Jul 6 14:45:37 adjucator kernel: [23771.836316] RDX: 0000000000000007 RSI: 0000000000000086 RDI: ffff8fccafa978c0
Jul 6 14:45:37 adjucator kernel: [23771.836316] RBP: ffffac40c1e8bcf8 R08: 0000000000000475 R09: 0000000000000004
Jul 6 14:45:37 adjucator kernel: [23771.836317] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8fcabace00e8
Jul 6 14:45:37 adjucator kernel: [23771.836317] R13: ffff8fcaccd01878 R14: ffff8fcabace0170 R15: 0000000000000000
Jul 6 14:45:37 adjucator kernel: [23771.836320] FS: 00007f34a55dc380(0000) GS:ffff8fccafa80000(0000) knlGS:0000000000000000
Jul 6 14:45:37 adjucator kernel: [23771.836320] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 6 14:45:37 adjucator kernel: [23771.836321] CR2: 00007fd30237a720 CR3: 00000003355aa000 CR4: 0000000000340ee0
Jul 6 14:45:37 adjucator kernel: [23771.836323] Call Trace:
Jul 6 14:45:37 adjucator kernel: [23771.836328] generic_update_time+0x9d/0xc0
Jul 6 14:45:37 adjucator kernel: [23771.836329] file_update_time+0xeb/0x140
Jul 6 14:45:37 adjucator kernel: [23771.836330] ? current_time+0x43/0x80
Jul 6 14:45:37 adjucator kernel: [23771.836340] __generic_file_write_iter+0x9a/0x1d0
Jul 6 14:45:37 adjucator kernel: [23771.836349] ext4_file_write_iter+0xb9/0x360
Jul 6 14:45:37 adjucator kernel: [23771.836351] new_sync_write+0x125/0x1c0
Jul 6 14:45:37 adjucator kernel: [23771.836352] __vfs_write+0x29/0x40
Jul 6 14:45:37 adjucator kernel: [23771.836353] vfs_write+0xb9/0x1a0
Jul 6 14:45:37 adjucator kernel: [23771.836354] ksys_write+0x67/0xe0
Jul 6 14:45:37 adjucator kernel: [23771.836356] __x64_sys_write+0x1a/0x20
Jul 6 14:45:37 adjucator kernel: [23771.836367] do_syscall_64+0x57/0x190
Jul 6 14:45:37 adjucator kernel: [23771.836386] entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 6 14:45:37 adjucator kernel: [23771.836388] RIP: 0033:0x7f34a27db8a8
Jul 6 14:45:37 adjucator kernel: [23771.836389] Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 b5 4c 2d 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4 55
Jul 6 14:45:37 adjucator kernel: [23771.836390] RSP: 002b:00007ffd8bed5a78 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
Jul 6 14:45:37 adjucator kernel: [23771.836419] RAX: ffffffffffffffda RBX: 000000000000006e RCX: 00007f34a27db8a8
Jul 6 14:45:37 adjucator kernel: [23771.836420] RDX: 000000000000006e RSI: 0000000002311920 RDI: 0000000000000007
Jul 6 14:45:37 adjucator kernel: [23771.836420] RBP: 0000000002311920 R08: 00000000022d53b0 R09: 0000000000000001
Jul 6 14:45:37 adjucator kernel: [23771.836421] R10: 0000000000000000 R11: 0000000000000246 R12: 00000000022d52d0
Jul 6 14:45:37 adjucator kernel: [23771.836421] R13: 000000000000006e R14: 00007f34a2aa7880 R15: 000000000000006e
Jul 6 14:45:37 adjucator kernel: [23771.836423] ---[ end trace 58eb8bd68b9b54f5 ]---
Jul 6 14:45:42 adjucator kernel: [23777.102840] EXT4-fs error (device sdd): __ext4_find_entry:1531: inode #6029319: comm patroni: reading directory lblock 0
Jul 6 14:45:42 adjucator kernel: [23777.105956] EXT4-fs warning (device sdd): htree_dirblock_to_tree:997: inode #6029313: lblock 0: comm patroni: error -5 reading directory block
Jul 6 14:45:44 adjucator kernel: [23779.281374] EXT4-fs error (device sdd): __ext4_find_entry:1531: inode #6029313: comm postgres: reading directory lblock 0
Jul 6 14:45:45 adjucator kernel: [23779.476688] EXT4-fs error (device sdd): __ext4_find_entry:1531: inode #6029313: comm postgres: reading directory lblock 0
Jul 6 14:45:45 adjucator kernel: [23779.501374] EXT4-fs error (device sdd): __ext4_find_entry:1531: inode #6029319: comm postgres: reading directory lblock 0
Jul 6 14:45:49 adjucator kernel: [23784.276356] sd 3:0:0:1: [sdb] tag#51 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 6 14:45:49 adjucator kernel: [23784.276359] sd 3:0:0:1: [sdb] tag#51 Sense Key : Medium Error [current]
Jul 6 14:45:49 adjucator kernel: [23784.276361] sd 3:0:0:1: [sdb] tag#51 Add. Sense: Unrecovered read error
Jul 6 14:45:49 adjucator kernel: [23784.276364] sd 3:0:0:1: [sdb] tag#51 CDB: Write(10) 2a 00 4b 46 8a b8 00 00 10 00
Jul 6 14:45:49 adjucator kernel: [23784.276365] print_req_error: 60 callbacks suppressed
Jul 6 14:45:49 adjucator kernel: [23784.276367] blk_update_request: critical medium error, dev sdb, sector 1262914232 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0
Jul 6 14:45:49 adjucator kernel: [23784.277215] EXT4-fs warning (device sdb): ext4_end_bio:309: I/O error 7 writing to inode 39454295 (offset 0 size 8192 starting block 157864281)
Jul 6 14:45:49 adjucator kernel: [23784.277218] buffer_io_error: 558 callbacks suppressed
Jul 6 14:45:49 adjucator kernel: [23784.277219] Buffer I/O error on device sdb, logical block 157864279
Jul 6 14:45:49 adjucator kernel: [23784.277519] Buffer I/O error on device sdb, logical block 157864280
Jul 6 14:45:49 adjucator kernel: [23784.282910] blk_update_request: I/O error, dev sdb, sector 1074907792 op 0x1:(WRITE) flags 0x800 phys_seg 16 prio class 0
Jul 6 14:45:49 adjucator kernel: [23784.283240] blk_update_request: I/O error, dev sdb, sector 1262917160 op 0x1:(WRITE) flags 0x800 phys_seg 4 prio class 0
Jul 6 14:45:49 adjucator kernel: [23784.284087] EXT4-fs warning (device sdb): ext4_end_bio:309: I/O error 10 writing to inode 39454299 (offset 0 size 16384 starting block 157864649)
Jul 6 14:45:49 adjucator kernel: [23784.284091] Buffer I/O error on device sdb, logical block 157864645
Jul 6 14:45:49 adjucator kernel: [23784.284814] Buffer I/O error on device sdb, logical block 157864646
Jul 6 14:45:49 adjucator kernel: [23784.285512] Buffer I/O error on device sdb, logical block 157864647
Jul 6 14:45:49 adjucator kernel: [23784.286216] Buffer I/O error on device sdb, logical block 157864648
Jul 6 14:45:49 adjucator kernel: [23784.286974] JBD2: Detected IO errors while flushing file data on sdb-8
Jul 6 14:45:49 adjucator kernel: [23784.286990] Aborting journal on device sdb-8.
Jul 6 14:45:49 adjucator kernel: [23784.287696] blk_update_request: I/O error, dev sdb, sector 1074003968 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
Jul 6 14:45:49 adjucator kernel: [23784.288373] buffer_io_error: 66 callbacks suppressed
Jul 6 14:45:49 adjucator kernel: [23784.288375] Buffer I/O error on dev sdb, logical block 134250496, lost sync page write
Jul 6 14:45:49 adjucator kernel: [23784.289090] JBD2: Error -5 detected when updating journal superblock for sdb-8.
Jul 6 14:45:49 adjucator kernel: [23784.307037] blk_update_request: I/O error, dev sdb, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
Jul 6 14:45:49 adjucator kernel: [23784.307754] Buffer I/O error on dev sdb, logical block 0, lost sync page write
Jul 6 14:45:49 adjucator kernel: [23784.308287] EXT4-fs (sdb): I/O error while writing superblock
Jul 6 14:45:49 adjucator kernel: [23784.308824] EXT4-fs error (device sdb): ext4_journal_check_start:61: Detected aborted journal
Jul 6 14:45:49 adjucator kernel: [23784.309331] EXT4-fs (sdb): Remounting filesystem read-only
Jul 6 14:45:49 adjucator kernel: [23784.309813] blk_update_request: I/O error, dev sdb, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
Jul 6 14:45:49 adjucator kernel: [23784.310269] Buffer I/O error on dev sdb, logical block 0, lost sync page write
Jul 6 14:45:49 adjucator kernel: [23784.314549] EXT4-fs (sdb): I/O error while writing superblock
Jul 6 14:45:50 adjucator kernel: [23784.563681] sd 3:0:0:1: [sdb] Synchronizing SCSI cache
Jul 6 14:45:50 adjucator kernel: [23785.171739] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:45:50 adjucator kernel: [23785.248842] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #5374256: comm minio: reading directory lblock 0
Jul 6 14:45:51 adjucator kernel: [23785.675177] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:45:51 adjucator kernel: [23785.893766] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:45:51 adjucator kernel: [23786.130230] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:45:52 adjucator kernel: [23787.104087] EXT4-fs error (device sdd): __ext4_find_entry:1531: inode #6029319: comm patroni: reading directory lblock 0
Jul 6 14:45:52 adjucator kernel: [23787.113961] EXT4-fs warning (device sdd): htree_dirblock_to_tree:997: inode #6029313: lblock 0: comm patroni: error -5 reading directory block
Jul 6 14:45:52 adjucator kernel: [23787.136587] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:45:52 adjucator kernel: [23787.138081] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:45:52 adjucator kernel: [23787.195594] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:45:52 adjucator kernel: [23787.196867] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:45:52 adjucator kernel: [23787.198194] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:45:56 adjucator kernel: [23790.888800] EXT4-fs error: 24 callbacks suppressed
Jul 6 14:45:56 adjucator kernel: [23790.888803] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:45:56 adjucator kernel: [23790.889578] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:45:56 adjucator kernel: [23791.084012] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:45:56 adjucator kernel: [23791.084729] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:45:57 adjucator kernel: [23791.465850] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:45:57 adjucator kernel: [23791.513738] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:45:57 adjucator kernel: [23791.559849] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:45:57 adjucator kernel: [23791.744802] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:45:57 adjucator kernel: [23791.746148] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:45:57 adjucator kernel: [23792.335427] IPv6: ADDRCONF(NETDEV_CHANGE): veth8d0fe479: link becomes ready
Jul 6 14:45:57 adjucator kernel: [23792.335524] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jul 6 14:45:57 adjucator kernel: [23792.338444] cni0: port 14(veth8d0fe479) entered blocking state
Jul 6 14:45:57 adjucator kernel: [23792.338447] cni0: port 14(veth8d0fe479) entered disabled state
Jul 6 14:45:57 adjucator kernel: [23792.338632] device veth8d0fe479 entered promiscuous mode
Jul 6 14:45:57 adjucator kernel: [23792.338812] cni0: port 14(veth8d0fe479) entered blocking state
Jul 6 14:45:57 adjucator kernel: [23792.338813] cni0: port 14(veth8d0fe479) entered forwarding state
Jul 6 14:46:00 adjucator kernel: [23794.508491] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:46:00 adjucator kernel: [23795.065542] EXT4-fs error (device sdf): __ext4_get_inode_loc:4712: inode #12: block 1059: comm 10_dirty_io_sch: unable to read itable block
Jul 6 14:46:00 adjucator kernel: [23795.066751] EXT4-fs error (device sdf) in ext4_reserve_inode_write:5956: IO failure
Jul 6 14:46:00 adjucator kernel: [23795.067207] ------------[ cut here ]------------
Jul 6 14:46:00 adjucator kernel: [23795.067208] bdi-block not registered
Jul 6 14:46:00 adjucator kernel: [23795.067225] WARNING: CPU: 0 PID: 27697 at fs/fs-writeback.c:2330 __mark_inode_dirty+0x19b/0x3a0
Jul 6 14:46:00 adjucator kernel: [23795.067226] Modules linked in: nf_conntrack_netlink iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi xt_multiport veth ip6table_nat ip6_tables xt_recent xt_nat xt_statistic xt_MASQUERADE xt_mark xt_tcpudp vxlan ip6_udp_tunnel udp_tunnel ipt_REJECT nf_reject_ipv4 xt_physdev xt_addrtype xt_set xt_conntrack xt_comment ip_set_hash_ip iptable_filter iptable_mangle bpfilter ip_set nfnetlink ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 overlay br_netfilter bridge stp llc dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua input_leds joydev serio_raw mac_hid qemu_fw_cfg sch_fq_codel ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid bochs_drm drm_vram_helper crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ttm drm_kms_helper aesni_intel crypto_simd syscopyarea sysfillrect cryptd glue_helper sysimgblt
Jul 6 14:46:00 adjucator kernel: [23795.067314] virtio_net fb_sys_fops psmouse net_failover virtio_scsi failover drm i2c_piix4 pata_acpi floppy
Jul 6 14:46:00 adjucator kernel: [23795.067321] CPU: 0 PID: 27697 Comm: 10_dirty_io_sch Tainted: G W 5.4.0-66-generic #74-Ubuntu
Jul 6 14:46:00 adjucator kernel: [23795.067322] Hardware name: netcup KVM Server, BIOS RS 2000 G9 10/06/2020
Jul 6 14:46:00 adjucator kernel: [23795.067324] RIP: 0010:__mark_inode_dirty+0x19b/0x3a0
Jul 6 14:46:00 adjucator kernel: [23795.067325] Code: ff ff 8b 4d d4 49 89 c5 48 8b 00 f6 40 5c 02 75 1e 49 8b 55 08 83 e2 01 75 15 48 8b 70 50 48 c7 c7 cb 91 38 8a e8 30 d0 d8 ff <0f> 0b 8b 4d d4 48 8b 05 c9 21 2f 01 49 89 84 24 c8 00 00 00 85 c9
Jul 6 14:46:00 adjucator kernel: [23795.067326] RSP: 0018:ffffac40c40e3c20 EFLAGS: 00010282
Jul 6 14:46:00 adjucator kernel: [23795.067327] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000006
Jul 6 14:46:00 adjucator kernel: [23795.067327] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff8fccafa178c0
Jul 6 14:46:00 adjucator kernel: [23795.067328] RBP: ffffac40c40e3c50 R08: 00000000000004e2 R09: 0000000000000004
Jul 6 14:46:00 adjucator kernel: [23795.067328] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8fcb27483890
Jul 6 14:46:00 adjucator kernel: [23795.067329] R13: ffff8fcba8381878 R14: ffff8fcb27483918 R15: 0000000000000000
Jul 6 14:46:00 adjucator kernel: [23795.067331] FS: 00007fb08cf20700(0000) GS:ffff8fccafa00000(0000) knlGS:0000000000000000
Jul 6 14:46:00 adjucator kernel: [23795.067332] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 6 14:46:00 adjucator kernel: [23795.067332] CR2: 00007fb02964c000 CR3: 00000002bfb30000 CR4: 0000000000340ef0
Jul 6 14:46:00 adjucator kernel: [23795.067334] Call Trace:
Jul 6 14:46:00 adjucator kernel: [23795.067340] generic_update_time+0x9d/0xc0
Jul 6 14:46:00 adjucator kernel: [23795.067341] file_update_time+0xeb/0x140
Jul 6 14:46:00 adjucator kernel: [23795.067344] __generic_file_write_iter+0x9a/0x1d0
Jul 6 14:46:00 adjucator kernel: [23795.067347] ext4_file_write_iter+0xb9/0x360
Jul 6 14:46:00 adjucator kernel: [23795.067363] ? common_file_perm+0x5e/0x110
Jul 6 14:46:00 adjucator kernel: [23795.067366] do_iter_readv_writev+0x14f/0x1d0
Jul 6 14:46:00 adjucator kernel: [23795.067367] do_iter_write+0x84/0x1a0
Jul 6 14:46:00 adjucator kernel: [23795.067368] vfs_writev+0xa6/0xf0
Jul 6 14:46:00 adjucator kernel: [23795.067371] ? _cond_resched+0x19/0x30
Jul 6 14:46:00 adjucator kernel: [23795.067372] do_writev+0x71/0x120
Jul 6 14:46:00 adjucator kernel: [23795.067373] __x64_sys_writev+0x1c/0x20
Jul 6 14:46:00 adjucator kernel: [23795.067376] do_syscall_64+0x57/0x190
Jul 6 14:46:00 adjucator kernel: [23795.067378] entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 6 14:46:00 adjucator kernel: [23795.067380] RIP: 0033:0x7fb0d0daa61d
Jul 6 14:46:00 adjucator kernel: [23795.067380] Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 5a 01 f8 ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 14 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2f 44 89 c7 48 89 44 24 08 e8 8e 01 f8 ff 48
Jul 6 14:46:00 adjucator kernel: [23795.067381] RSP: 002b:00007fb08cf1fa80 EFLAGS: 00000293 ORIG_RAX: 0000000000000014
Jul 6 14:46:00 adjucator kernel: [23795.067382] RAX: ffffffffffffffda RBX: 0000000000000400 RCX: 00007fb0d0daa61d
Jul 6 14:46:00 adjucator kernel: [23795.067382] RDX: 0000000000000001 RSI: 00007fb08cf1fb28 RDI: 0000000000000056
Jul 6 14:46:00 adjucator kernel: [23795.067383] RBP: 00007fb08cf20680 R08: 0000000000000000 R09: 000000000000003b
Jul 6 14:46:00 adjucator kernel: [23795.067383] R10: 0000000000000020 R11: 0000000000000293 R12: 0000000000000000
Jul 6 14:46:00 adjucator kernel: [23795.067383] R13: 0000000000000001 R14: 00007fb08cf1fb28 R15: 000055bc7ba4b410
Jul 6 14:46:00 adjucator kernel: [23795.067385] ---[ end trace 58eb8bd68b9b54f6 ]---
Jul 6 14:46:00 adjucator kernel: [23795.078484] EXT4-fs warning (device sdf): ext4_end_bio:309: I/O error 10 writing to inode 12 (offset 0 size 0 starting block 9489)
Jul 6 14:46:00 adjucator kernel: [23795.078486] Buffer I/O error on device sdf, logical block 9489
Jul 6 14:46:00 adjucator kernel: [23795.080377] EXT4-fs warning (device sdf): htree_dirblock_to_tree:997: inode #2: lblock 0: comm 6_dirty_io_sche: error -5 reading directory block
Jul 6 14:46:00 adjucator kernel: [23795.080385] EXT4-fs error (device sdf): __ext4_get_inode_loc:4712: inode #2: block 1059: comm 6_dirty_io_sche: unable to read itable block
Jul 6 14:46:00 adjucator kernel: [23795.081742] EXT4-fs error (device sdf) in ext4_reserve_inode_write:5956: IO failure
Jul 6 14:46:00 adjucator kernel: [23795.082424] ------------[ cut here ]------------
Jul 6 14:46:00 adjucator kernel: [23795.082425] bdi-block not registered
Jul 6 14:46:00 adjucator kernel: [23795.082495] WARNING: CPU: 3 PID: 27693 at fs/fs-writeback.c:2330 __mark_inode_dirty+0x19b/0x3a0
Jul 6 14:46:00 adjucator kernel: [23795.082497] Modules linked in: nf_conntrack_netlink iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi xt_multiport veth ip6table_nat ip6_tables xt_recent xt_nat xt_statistic xt_MASQUERADE xt_mark xt_tcpudp vxlan ip6_udp_tunnel udp_tunnel ipt_REJECT nf_reject_ipv4 xt_physdev xt_addrtype xt_set xt_conntrack xt_comment ip_set_hash_ip iptable_filter iptable_mangle bpfilter ip_set nfnetlink ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 overlay br_netfilter bridge stp llc dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua input_leds joydev serio_raw mac_hid qemu_fw_cfg sch_fq_codel ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid bochs_drm drm_vram_helper crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ttm drm_kms_helper aesni_intel crypto_simd syscopyarea sysfillrect cryptd glue_helper sysimgblt
Jul 6 14:46:00 adjucator kernel: [23795.082543] virtio_net fb_sys_fops psmouse net_failover virtio_scsi failover drm i2c_piix4 pata_acpi floppy
Jul 6 14:46:00 adjucator kernel: [23795.082551] CPU: 3 PID: 27693 Comm: 6_dirty_io_sche Tainted: G W 5.4.0-66-generic #74-Ubuntu
Jul 6 14:46:00 adjucator kernel: [23795.082552] Hardware name: netcup KVM Server, BIOS RS 2000 G9 10/06/2020
Jul 6 14:46:00 adjucator kernel: [23795.082554] RIP: 0010:__mark_inode_dirty+0x19b/0x3a0
Jul 6 14:46:00 adjucator kernel: [23795.082556] Code: ff ff 8b 4d d4 49 89 c5 48 8b 00 f6 40 5c 02 75 1e 49 8b 55 08 83 e2 01 75 15 48 8b 70 50 48 c7 c7 cb 91 38 8a e8 30 d0 d8 ff <0f> 0b 8b 4d d4 48 8b 05 c9 21 2f 01 49 89 84 24 c8 00 00 00 85 c9
Jul 6 14:46:00 adjucator kernel: [23795.082557] RSP: 0018:ffffac40c40b3df8 EFLAGS: 00010282
Jul 6 14:46:00 adjucator kernel: [23795.082559] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000006
Jul 6 14:46:00 adjucator kernel: [23795.082559] RDX: 0000000000000007 RSI: 0000000000000082 RDI: ffff8fccafb978c0
Jul 6 14:46:00 adjucator kernel: [23795.082560] RBP: ffffac40c40b3e28 R08: 0000000000000510 R09: 0000000000000004
Jul 6 14:46:00 adjucator kernel: [23795.082561] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8fcc9e0d4df8
Jul 6 14:46:00 adjucator kernel: [23795.082562] R13: ffff8fcba8381878 R14: ffff8fcc9e0d4e80 R15: 0000000000000000
Jul 6 14:46:00 adjucator kernel: [23795.082567] FS: 00007fb08d06c700(0000) GS:ffff8fccafb80000(0000) knlGS:0000000000000000
Jul 6 14:46:00 adjucator kernel: [23795.082569] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 6 14:46:00 adjucator kernel: [23795.082570] CR2: 00007fe71b861290 CR3: 00000002bfb30000 CR4: 0000000000340ee0
Jul 6 14:46:00 adjucator kernel: [23795.082574] Call Trace:
Jul 6 14:46:00 adjucator kernel: [23795.082581] generic_update_time+0x9d/0xc0
Jul 6 14:46:00 adjucator kernel: [23795.082583] touch_atime+0xbe/0xe0
Jul 6 14:46:00 adjucator kernel: [23795.082585] iterate_dir+0xe6/0x1b0
Jul 6 14:46:00 adjucator kernel: [23795.082588] ksys_getdents64+0x9c/0x140
Jul 6 14:46:00 adjucator kernel: [23795.082591] ? filldir+0x190/0x190
Jul 6 14:46:00 adjucator kernel: [23795.082593] __x64_sys_getdents64+0x1a/0x20
Jul 6 14:46:00 adjucator kernel: [23795.082598] do_syscall_64+0x57/0x190
Jul 6 14:46:00 adjucator kernel: [23795.082603] entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 6 14:46:00 adjucator kernel: [23795.082605] RIP: 0033:0x7fb0d0d7443b
Jul 6 14:46:00 adjucator kernel: [23795.082608] Code: 0f 1e fa 48 8b 47 20 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 81 fa ff ff ff 7f b8 ff ff ff 7f 48 0f 47 d0 b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 21 9a 10 00 f7 d8
Jul 6 14:46:00 adjucator kernel: [23795.082612] RSP: 002b:00007fb08d06bc38 EFLAGS: 00000293 ORIG_RAX: 00000000000000d9
Jul 6 14:46:00 adjucator kernel: [23795.082614] RAX: ffffffffffffffda RBX: 00007fb054000bb0 RCX: 00007fb0d0d7443b
Jul 6 14:46:00 adjucator kernel: [23795.082614] RDX: 0000000000008000 RSI: 00007fb054000bb0 RDI: 0000000000000056
Jul 6 14:46:00 adjucator kernel: [23795.082615] RBP: ffffffffffffff80 R08: 0000000000000030 R09: 000000000000007c
Jul 6 14:46:00 adjucator kernel: [23795.082615] R10: 0000000000000000 R11: 0000000000000293 R12: 00007fb054000b84
Jul 6 14:46:00 adjucator kernel: [23795.082616] R13: 0000000000000011 R14: 00007fb054000b80 R15: 00007fb08efe06c0
Jul 6 14:46:00 adjucator kernel: [23795.082617] ---[ end trace 58eb8bd68b9b54f7 ]---
Jul 6 14:46:00 adjucator kernel: [23795.082853] EXT4-fs error (device sdf): __ext4_find_entry:1531: inode #2: comm 2_dirty_io_sche: reading directory lblock 0
Jul 6 14:46:00 adjucator kernel: [23795.084725] EXT4-fs error (device sdg): __ext4_get_inode_loc:4712: inode #393218: block 1572896: comm 6_dirty_io_sche: unable to read itable block
Jul 6 14:46:00 adjucator kernel: [23795.085944] EXT4-fs error (device sdg) in ext4_reserve_inode_write:5956: IO failure
Jul 6 14:46:00 adjucator kernel: [23795.086544] ------------[ cut here ]------------
Jul 6 14:46:00 adjucator kernel: [23795.086545] bdi-block not registered
Jul 6 14:46:00 adjucator kernel: [23795.086594] WARNING: CPU: 3 PID: 27693 at fs/fs-writeback.c:2330 __mark_inode_dirty+0x19b/0x3a0
Jul 6 14:46:00 adjucator kernel: [23795.086595] Modules linked in: nf_conntrack_netlink iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi xt_multiport veth ip6table_nat ip6_tables xt_recent xt_nat xt_statistic xt_MASQUERADE xt_mark xt_tcpudp vxlan ip6_udp_tunnel udp_tunnel ipt_REJECT nf_reject_ipv4 xt_physdev xt_addrtype xt_set xt_conntrack xt_comment ip_set_hash_ip iptable_filter iptable_mangle bpfilter ip_set nfnetlink ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 overlay br_netfilter bridge stp llc dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua input_leds joydev serio_raw mac_hid qemu_fw_cfg sch_fq_codel ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid bochs_drm drm_vram_helper crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ttm drm_kms_helper aesni_intel crypto_simd syscopyarea sysfillrect cryptd glue_helper sysimgblt
Jul 6 14:46:00 adjucator kernel: [23795.086633] virtio_net fb_sys_fops psmouse net_failover virtio_scsi failover drm i2c_piix4 pata_acpi floppy
Jul 6 14:46:00 adjucator kernel: [23795.086639] CPU: 3 PID: 27693 Comm: 6_dirty_io_sche Tainted: G W 5.4.0-66-generic #74-Ubuntu
Jul 6 14:46:00 adjucator kernel: [23795.086640] Hardware name: netcup KVM Server, BIOS RS 2000 G9 10/06/2020
Jul 6 14:46:00 adjucator kernel: [23795.086641] RIP: 0010:__mark_inode_dirty+0x19b/0x3a0
Jul 6 14:46:00 adjucator kernel: [23795.086643] Code: ff ff 8b 4d d4 49 89 c5 48 8b 00 f6 40 5c 02 75 1e 49 8b 55 08 83 e2 01 75 15 48 8b 70 50 48 c7 c7 cb 91 38 8a e8 30 d0 d8 ff <0f> 0b 8b 4d d4 48 8b 05 c9 21 2f 01 49 89 84 24 c8 00 00 00 85 c9
Jul 6 14:46:00 adjucator kernel: [23795.086643] RSP: 0018:ffffac40c40b3bf8 EFLAGS: 00010282
Jul 6 14:46:00 adjucator kernel: [23795.086644] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000006
Jul 6 14:46:00 adjucator kernel: [23795.086645] RDX: 0000000000000007 RSI: 0000000000000082 RDI: ffff8fccafb978c0
Jul 6 14:46:00 adjucator kernel: [23795.086645] RBP: ffffac40c40b3c28 R08: 0000000000000537 R09: 0000000000000004
Jul 6 14:46:00 adjucator kernel: [23795.086646] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8fcb3f3f1650
Jul 6 14:46:00 adjucator kernel: [23795.086646] R13: ffff8fcbf80a3078 R14: ffff8fcb3f3f16d8 R15: 0000000000000000
Jul 6 14:46:00 adjucator kernel: [23795.086648] FS: 00007fb08d06c700(0000) GS:ffff8fccafb80000(0000) knlGS:0000000000000000
Jul 6 14:46:00 adjucator kernel: [23795.086649] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 6 14:46:00 adjucator kernel: [23795.086649] CR2: 00007fe71b861290 CR3: 00000002bfb30000 CR4: 0000000000340ee0
Jul 6 14:46:00 adjucator kernel: [23795.086651] Call Trace:
Jul 6 14:46:00 adjucator kernel: [23795.086655] generic_update_time+0x9d/0xc0
Jul 6 14:46:00 adjucator kernel: [23795.086656] touch_atime+0xbe/0xe0
Jul 6 14:46:00 adjucator kernel: [23795.086659] generic_file_buffered_read+0x9ac/0xc10
Jul 6 14:46:00 adjucator kernel: [23795.086662] ? schedule+0x42/0xb0
Jul 6 14:46:00 adjucator kernel: [23795.086663] ? _cond_resched+0x19/0x30
Jul 6 14:46:00 adjucator kernel: [23795.086665] ? futex_wait_queue_me+0xcf/0x120
Jul 6 14:46:00 adjucator kernel: [23795.086667] generic_file_read_iter+0xdc/0x140
Jul 6 14:46:00 adjucator kernel: [23795.086669] ext4_file_read_iter+0x4a/0x100
Jul 6 14:46:00 adjucator kernel: [23795.086671] new_sync_read+0x122/0x1b0
Jul 6 14:46:00 adjucator kernel: [23795.086672] __vfs_read+0x29/0x40
Jul 6 14:46:00 adjucator kernel: [23795.086673] vfs_read+0xab/0x160
Jul 6 14:46:00 adjucator kernel: [23795.086675] ksys_pread64+0x6d/0xa0
Jul 6 14:46:00 adjucator kernel: [23795.086676] __x64_sys_pread64+0x1e/0x20
Jul 6 14:46:00 adjucator kernel: [23795.086679] do_syscall_64+0x57/0x190
Jul 6 14:46:00 adjucator kernel: [23795.086682] entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 6 14:46:00 adjucator kernel: [23795.086683] RIP: 0033:0x7fb0d0e99c0f
Jul 6 14:46:00 adjucator kernel: [23795.086685] Code: 08 89 3c 24 48 89 4c 24 18 e8 ed f3 ff ff 4c 8b 54 24 18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 1d f4 ff ff 48 8b
Jul 6 14:46:00 adjucator kernel: [23795.086686] RSP: 002b:00007fb08d06bbe0 EFLAGS: 00000293 ORIG_RAX: 0000000000000011
Jul 6 14:46:00 adjucator kernel: [23795.086687] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb0d0e99c0f
Jul 6 14:46:00 adjucator kernel: [23795.086687] RDX: 0000000000002000 RSI: 00007fb08d779430 RDI: 0000000000000053
Jul 6 14:46:00 adjucator kernel: [23795.086688] RBP: 0000000000000000 R08: 0000000000000000 R09: 000055bc7a309554
Jul 6 14:46:00 adjucator kernel: [23795.086688] R10: 0000000000000538 R11: 0000000000000293 R12: 00007fb08d06c680
Jul 6 14:46:00 adjucator kernel: [23795.086689] R13: 00007fb08d06bc60 R14: 0000000000000538 R15: 000055bc7ba49988
Jul 6 14:46:00 adjucator kernel: [23795.086690] ---[ end trace 58eb8bd68b9b54f8 ]---
Jul 6 14:46:00 adjucator kernel: [23795.095540] EXT4-fs error (device sdg): __ext4_get_inode_loc:4712: inode #393219: block 1572896: comm 2_dirty_io_sche: unable to read itable block
Jul 6 14:46:00 adjucator kernel: [23795.097074] EXT4-fs error (device sdg) in ext4_reserve_inode_write:5956: IO failure
Jul 6 14:46:00 adjucator kernel: [23795.097754] ------------[ cut here ]------------
Jul 6 14:46:00 adjucator kernel: [23795.097755] bdi-block not registered
Jul 6 14:46:00 adjucator kernel: [23795.097773] WARNING: CPU: 2 PID: 27689 at fs/fs-writeback.c:2330 __mark_inode_dirty+0x19b/0x3a0
Jul 6 14:46:00 adjucator kernel: [23795.097773] Modules linked in: nf_conntrack_netlink iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi xt_multiport veth ip6table_nat ip6_tables xt_recent xt_nat xt_statistic xt_MASQUERADE xt_mark xt_tcpudp vxlan ip6_udp_tunnel udp_tunnel ipt_REJECT nf_reject_ipv4 xt_physdev xt_addrtype xt_set xt_conntrack xt_comment ip_set_hash_ip iptable_filter iptable_mangle bpfilter ip_set nfnetlink ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 overlay br_netfilter bridge stp llc dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua input_leds joydev serio_raw mac_hid qemu_fw_cfg sch_fq_codel ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid bochs_drm drm_vram_helper crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ttm drm_kms_helper aesni_intel crypto_simd syscopyarea sysfillrect cryptd glue_helper sysimgblt
Jul 6 14:46:00 adjucator kernel: [23795.097813] virtio_net fb_sys_fops psmouse net_failover virtio_scsi failover drm i2c_piix4 pata_acpi floppy
Jul 6 14:46:00 adjucator kernel: [23795.097820] CPU: 2 PID: 27689 Comm: 2_dirty_io_sche Tainted: G W 5.4.0-66-generic #74-Ubuntu
Jul 6 14:46:00 adjucator kernel: [23795.097821] Hardware name: netcup KVM Server, BIOS RS 2000 G9 10/06/2020
Jul 6 14:46:00 adjucator kernel: [23795.097823] RIP: 0010:__mark_inode_dirty+0x19b/0x3a0
Jul 6 14:46:00 adjucator kernel: [23795.097824] Code: ff ff 8b 4d d4 49 89 c5 48 8b 00 f6 40 5c 02 75 1e 49 8b 55 08 83 e2 01 75 15 48 8b 70 50 48 c7 c7 cb 91 38 8a e8 30 d0 d8 ff <0f> 0b 8b 4d d4 48 8b 05 c9 21 2f 01 49 89 84 24 c8 00 00 00 85 c9
Jul 6 14:46:00 adjucator kernel: [23795.097825] RSP: 0018:ffffac40c3fffbf8 EFLAGS: 00010282
Jul 6 14:46:00 adjucator kernel: [23795.097826] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000006
Jul 6 14:46:00 adjucator kernel: [23795.097826] RDX: 0000000000000007 RSI: 0000000000000082 RDI: ffff8fccafb178c0
Jul 6 14:46:00 adjucator kernel: [23795.097827] RBP: ffffac40c3fffc28 R08: 0000000000000564 R09: 0000000000000004
Jul 6 14:46:00 adjucator kernel: [23795.097827] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8fcb3f3f49b0
Jul 6 14:46:00 adjucator kernel: [23795.097828] R13: ffff8fcbf80a3078 R14: ffff8fcb3f3f4a38 R15: 0000000000000000
Jul 6 14:46:00 adjucator kernel: [23795.097830] FS: 00007fb08d1b8700(0000) GS:ffff8fccafb00000(0000) knlGS:0000000000000000
Jul 6 14:46:00 adjucator kernel: [23795.097831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 6 14:46:00 adjucator kernel: [23795.097831] CR2: 000055b306eed3f8 CR3: 00000002bfb30000 CR4: 0000000000340ee0
Jul 6 14:46:00 adjucator kernel: [23795.097833] Call Trace:
Jul 6 14:46:00 adjucator kernel: [23795.097839] generic_update_time+0x9d/0xc0
Jul 6 14:46:00 adjucator kernel: [23795.097840] touch_atime+0xbe/0xe0
Jul 6 14:46:00 adjucator kernel: [23795.097843] generic_file_buffered_read+0x9ac/0xc10
Jul 6 14:46:00 adjucator kernel: [23795.097846] ? schedule+0x42/0xb0
Jul 6 14:46:00 adjucator kernel: [23795.097847] ? _cond_resched+0x19/0x30
Jul 6 14:46:00 adjucator kernel: [23795.097850] ? futex_wait_queue_me+0xcf/0x120
Jul 6 14:46:00 adjucator kernel: [23795.097851] generic_file_read_iter+0xdc/0x140
Jul 6 14:46:00 adjucator kernel: [23795.097854] ext4_file_read_iter+0x4a/0x100
Jul 6 14:46:00 adjucator kernel: [23795.097855] new_sync_read+0x122/0x1b0
Jul 6 14:46:00 adjucator kernel: [23795.097857] __vfs_read+0x29/0x40
Jul 6 14:46:00 adjucator kernel: [23795.097857] vfs_read+0xab/0x160
Jul 6 14:46:00 adjucator kernel: [23795.097859] ksys_pread64+0x6d/0xa0
Jul 6 14:46:00 adjucator kernel: [23795.097860] __x64_sys_pread64+0x1e/0x20
Jul 6 14:46:00 adjucator kernel: [23795.097863] do_syscall_64+0x57/0x190
Jul 6 14:46:00 adjucator kernel: [23795.097864] entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 6 14:46:00 adjucator kernel: [23795.097866] RIP: 0033:0x7fb0d0e99c0f
Jul 6 14:46:00 adjucator kernel: [23795.097867] Code: 08 89 3c 24 48 89 4c 24 18 e8 ed f3 ff ff 4c 8b 54 24 18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 04 24 e8 1d f4 ff ff 48 8b
Jul 6 14:46:00 adjucator kernel: [23795.097867] RSP: 002b:00007fb08d1b7be0 EFLAGS: 00000293 ORIG_RAX: 0000000000000011
Jul 6 14:46:00 adjucator kernel: [23795.097868] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb0d0e99c0f
Jul 6 14:46:00 adjucator kernel: [23795.097869] RDX: 0000000000000004 RSI: 000055bc7ba45268 RDI: 0000000000000054
Jul 6 14:46:00 adjucator kernel: [23795.097869] RBP: 0000000000000000 R08: 0000000000000000 R09: 000055bc7a309554
Jul 6 14:46:00 adjucator kernel: [23795.097869] R10: 0000000000000010 R11: 0000000000000293 R12: 00007fb08d1b8680
Jul 6 14:46:00 adjucator kernel: [23795.097870] R13: 00007fb08d1b7c60 R14: 0000000000000010 R15: 00007fb08d76a2f8
Jul 6 14:46:00 adjucator kernel: [23795.097871] ---[ end trace 58eb8bd68b9b54f9 ]---
Jul 6 14:46:00 adjucator kernel: [23795.100059] EXT4-fs warning (device sdf): htree_dirblock_to_tree:997: inode #2: lblock 0: comm 2_dirty_io_sche: error -5 reading directory block
Jul 6 14:46:00 adjucator kernel: [23795.100161] EXT4-fs error (device sdf): __ext4_find_entry:1531: inode #2: comm 2_dirty_io_sche: reading directory lblock 0
Jul 6 14:46:00 adjucator kernel: [23795.113339] EXT4-fs warning (device sdf): htree_dirblock_to_tree:997: inode #2: lblock 0: comm 2_dirty_io_sche: error -5 reading directory block
Jul 6 14:46:00 adjucator kernel: [23795.113447] EXT4-fs error (device sdf): __ext4_find_entry:1531: inode #2: comm 2_dirty_io_sche: reading directory lblock 0
Jul 6 14:46:00 adjucator kernel: [23795.123992] EXT4-fs warning (device sdf): htree_dirblock_to_tree:997: inode #2: lblock 0: comm 6_dirty_io_sche: error -5 reading directory block
Jul 6 14:46:00 adjucator kernel: [23795.124105] EXT4-fs error (device sdf): __ext4_find_entry:1531: inode #2: comm 6_dirty_io_sche: reading directory lblock 0
Jul 6 14:46:00 adjucator kernel: [23795.135300] EXT4-fs warning (device sdf): htree_dirblock_to_tree:997: inode #2: lblock 0: comm 6_dirty_io_sche: error -5 reading directory block
Jul 6 14:46:00 adjucator kernel: [23795.135390] EXT4-fs error (device sdf): __ext4_find_entry:1531: inode #2: comm 6_dirty_io_sche: reading directory lblock 0
Jul 6 14:46:00 adjucator kernel: [23795.151009] EXT4-fs warning (device sdf): htree_dirblock_to_tree:997: inode #2: lblock 0: comm 2_dirty_io_sche: error -5 reading directory block
Jul 6 14:46:00 adjucator kernel: [23795.151131] EXT4-fs error (device sdf): __ext4_find_entry:1531: inode #2: comm 2_dirty_io_sche: reading directory lblock 0
Jul 6 14:46:00 adjucator kernel: [23795.272232] EXT4-fs error (device sdg): __ext4_find_entry:1531: inode #393217: comm 2_dirty_io_sche: reading directory lblock 0
Jul 6 14:46:00 adjucator kernel: [23795.282784] EXT4-fs error (device sdg): __ext4_find_entry:1531: inode #393730: comm 2_dirty_io_sche: reading directory lblock 0
Jul 6 14:46:00 adjucator kernel: [23795.284520] EXT4-fs error (device sdg): __ext4_find_entry:1531: inode #393217: comm 2_dirty_io_sche: reading directory lblock 0
Jul 6 14:46:00 adjucator kernel: [23795.287687] EXT4-fs error (device sdg): __ext4_find_entry:1531: inode #393286: comm 2_dirty_io_sche: reading directory lblock 0
Jul 6 14:46:00 adjucator kernel: [23795.291855] EXT4-fs error (device sdg): __ext4_find_entry:1531: inode #393217: comm 2_dirty_io_sche: reading directory lblock 0
Jul 6 14:46:00 adjucator kernel: [23795.293340] EXT4-fs error (device sdg): __ext4_find_entry:1531: inode #393217: comm 2_dirty_io_sche: reading directory lblock 0
Jul 6 14:46:00 adjucator kernel: [23795.330842] EXT4-fs warning (device sdf): htree_dirblock_to_tree:997: inode #2: lblock 0: comm 5_dirty_io_sche: error -5 reading directory block
Jul 6 14:46:01 adjucator kernel: [23796.130743] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:46:02 adjucator kernel: [23796.948112] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:46:02 adjucator kernel: [23797.078870] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:46:02 adjucator kernel: [23797.084849] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:46:02 adjucator kernel: [23797.103771] EXT4-fs error (device sdd): __ext4_find_entry:1531: inode #6029319: comm patroni: reading directory lblock 0
Jul 6 14:46:02 adjucator kernel: [23797.105104] EXT4-fs warning (device sdd): htree_dirblock_to_tree:997: inode #6029313: lblock 0: comm patroni: error -5 reading directory block
Jul 6 14:46:02 adjucator kernel: [23797.171285] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:46:02 adjucator kernel: [23797.175741] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:46:02 adjucator kernel: [23797.182208] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:46:02 adjucator kernel: [23797.185022] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:46:02 adjucator kernel: [23797.189478] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:46:02 adjucator kernel: [23797.192651] EXT4-fs error (device sdb): __ext4_find_entry:1531: inode #39059457: comm minio: reading directory lblock 0
Jul 6 14:46:03 adjucator kernel: [23798.013511] EXT4-fs error (device sdd): __ext4_find_entry:1531: inode #6029313: comm patroni: reading directory lblock 0
Jul 6 14:46:03 adjucator kernel: [23798.014421] EXT4-fs error (device sdd): __ext4_find_entry:1531: inode #6029313: comm patroni: reading directory lblock 0
Jul 6 14:46:04 adjucator kernel: [23798.507442] cni0: port 32(vethc1886ba5) entered disabled state
Jul 6 14:46:04 adjucator kernel: [23798.516261] device vethc1886ba5 left promiscuous mode
Jul 6 14:46:04 adjucator kernel: [23798.516272] cni0: port 32(vethc1886ba5) entered disabled state
Jul 6 14:46:04 adjucator kernel: [23798.727300] connection4:0: detected conn error (1020)
Jul 6 14:46:04 adjucator kernel: [23798.727323] connection7:0: detected conn error (1020)
Jul 6 14:46:04 adjucator kernel: [23798.727544] scsi 3:0:0:1: [sdb] tag#52 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK
Jul 6 14:46:04 adjucator kernel: [23798.727548] scsi 3:0:0:1: [sdb] tag#52 CDB: Read(10) 28 00 4b 04 2f 48 00 00 20 00
Jul 6 14:46:04 adjucator kernel: [23798.727550] blk_update_request: I/O error, dev sdb, sector 1258565448 op 0x0:(READ) flags 0x80700 phys_seg 4 prio class 0
Jul 6 14:46:05 adjucator kernel: [23799.471085] cni0: port 28(veth4ca72b9e) entered disabled state
Jul 6 14:46:05 adjucator kernel: [23799.478139] device veth4ca72b9e left promiscuous mode
Jul 6 14:46:05 adjucator kernel: [23799.478149] cni0: port 28(veth4ca72b9e) entered disabled state
Jul 6 14:46:05 adjucator kernel: [23799.690469] cni0: port 31(vethe5517482) entered disabled state
Jul 6 14:46:05 adjucator kernel: [23799.696806] device vethe5517482 left promiscuous mode
Jul 6 14:46:05 adjucator kernel: [23799.696831] cni0: port 31(vethe5517482) entered disabled state
Jul 6 14:46:05 adjucator kernel: [23800.334152] Aborting journal on device sdg-8.
Jul 6 14:46:05 adjucator kernel: [23800.335373] Buffer I/O error on dev sdg, logical block 1081344, lost sync page write
Jul 6 14:46:05 adjucator kernel: [23800.337176] JBD2: Error -5 detected when updating journal superblock for sdg-8.
Jul 6 14:46:06 adjucator kernel: [23800.590108] Aborting journal on device sdf-8.
Jul 6 14:46:06 adjucator kernel: [23800.591392] Buffer I/O error on dev sdf, logical block 1081344, lost sync page write
Jul 6 14:46:06 adjucator kernel: [23800.592265] JBD2: Error -5 detected when updating journal superblock for sdf-8.
Jul 6 14:46:11 adjucator kernel: [23805.792801] IPv6: ADDRCONF(NETDEV_CHANGE): veth9812c9f9: link becomes ready
Jul 6 14:46:11 adjucator kernel: [23805.792967] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jul 6 14:46:11 adjucator kernel: [23805.793263] cni0: port 16(veth9812c9f9) entered blocking state
Jul 6 14:46:11 adjucator kernel: [23805.793265] cni0: port 16(veth9812c9f9) entered disabled state
Jul 6 14:46:11 adjucator kernel: [23805.793362] device veth9812c9f9 entered promiscuous mode
Jul 6 14:46:11 adjucator kernel: [23805.793537] cni0: port 16(veth9812c9f9) entered blocking state
Jul 6 14:46:11 adjucator kernel: [23805.793538] cni0: port 16(veth9812c9f9) entered forwarding state
Jul 6 14:46:14 adjucator kernel: [23809.062488] sd 6:0:0:1: rejecting I/O to offline device
Jul 6 14:46:14 adjucator kernel: [23809.063441] blk_update_request: I/O error, dev sde, sector 8344 op 0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
Jul 6 14:46:14 adjucator kernel: [23809.064160] sd 6:0:0:1: rejecting I/O to offline device
Jul 6 14:46:14 adjucator kernel: [23809.064165] EXT4-fs error (device sde): ext4_read_inode_bitmap:197: comm 5_dirty_io_sche: Cannot read inode bitmap - block_group = 0, inode_bitmap = 1043
Jul 6 14:46:14 adjucator kernel: [23809.065147] blk_update_request: I/O error, dev sde, sector 282888 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Jul 6 14:46:14 adjucator kernel: [23809.065151] EXT4-fs warning (device sde): ext4_end_bio:309: I/O error 10 writing to inode 393225 (offset 0 size 4096 starting block 35362)
Jul 6 14:46:14 adjucator kernel: [23809.065155] Buffer I/O error on device sde, logical block 35361
Jul 6 14:46:14 adjucator kernel: [23809.069178] sd 6:0:0:1: rejecting I/O to offline device
Jul 6 14:46:14 adjucator kernel: [23809.070201] blk_update_request: I/O error, dev sde, sector 12890264 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Jul 6 14:46:14 adjucator kernel: [23809.071321] EXT4-fs warning (device sde): ext4_end_bio:309: I/O error 10 writing to inode 393228 (offset 4096 size 4096 starting block 1611284)
Jul 6 14:46:14 adjucator kernel: [23809.071325] Buffer I/O error on device sde, logical block 1611283
Jul 6 14:46:14 adjucator kernel: [23809.072365] sd 6:0:0:1: rejecting I/O to offline device
Jul 6 14:46:14 adjucator kernel: [23809.073330] blk_update_request: I/O error, dev sde, sector 12894264 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Jul 6 14:46:14 adjucator kernel: [23809.074218] EXT4-fs warning (device sde): ext4_end_bio:309: I/O error 10 writing to inode 393228 (offset 0 size 0 starting block 1611784)
Jul 6 14:46:14 adjucator kernel: [23809.074221] Buffer I/O error on device sde, logical block 1611783
Jul 6 14:46:14 adjucator kernel: [23809.075178] sd 6:0:0:1: rejecting I/O to offline device
Jul 6 14:46:14 adjucator kernel: [23809.075908] blk_update_request: I/O error, dev sde, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
Jul 6 14:46:14 adjucator kernel: [23809.076649] Buffer I/O error on dev sde, logical block 0, lost sync page write
Jul 6 14:46:14 adjucator kernel: [23809.077400] EXT4-fs (sde): I/O error while writing superblock
Jul 6 14:46:14 adjucator kernel: [23809.078331] EXT4-fs error (device sde) in ext4_free_inode:352: IO failure
Jul 6 14:46:14 adjucator kernel: [23809.079182] sd 6:0:0:1: rejecting I/O to offline device
Jul 6 14:46:14 adjucator kernel: [23809.079901] blk_update_request: I/O error, dev sde, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
Jul 6 14:46:14 adjucator kernel: [23809.080598] Buffer I/O error on dev sde, logical block 0, lost sync page write
Jul 6 14:46:14 adjucator kernel: [23809.081389] EXT4-fs (sde): I/O error while writing superblock
Jul 6 14:46:14 adjucator kernel: [23809.082265] JBD2: Detected IO errors while flushing file data on sde-8
Jul 6 14:46:14 adjucator kernel: [23809.082272] sd 6:0:0:1: rejecting I/O to offline device
Jul 6 14:46:14 adjucator kernel: [23809.083409] EXT4-fs error: 244 callbacks suppressed
Jul 6 14:46:14 adjucator kernel: [23809.083413] EXT4-fs error (device sdg): __ext4_find_entry:1531: inode #393745: comm 5_dirty_io_sche: reading directory lblock 0
Jul 6 14:46:14 adjucator kernel: [23809.083579] blk_update_request: I/O error, dev sde, sector 8665752 op 0x1:(WRITE) flags 0x800 phys_seg 10 prio class 0
Jul 6 14:46:14 adjucator kernel: [23809.083586] Aborting journal on device sde-8.
Jul 6 14:46:14 adjucator kernel: [23809.086770] EXT4-fs error (device sdg): __ext4_find_entry:1531: inode #393217: comm 5_dirty_io_sche: reading directory lblock 0
Jul 6 14:46:14 adjucator kernel: [23809.087284] sd 6:0:0:1: rejecting I/O to offline device
Jul 6 14:46:14 adjucator kernel: [23809.089695] blk_update_request: I/O error, dev sde, sector 8650752 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
Jul 6 14:46:14 adjucator kernel: [23809.090648] Buffer I/O error on dev sde, logical block 1081344, lost sync page write
Jul 6 14:46:14 adjucator kernel: [23809.092301] EXT4-fs warning (device sdf): htree_dirblock_to_tree:997: inode #2: lblock 0: comm 6_dirty_io_sche: error -5 reading directory block
Jul 6 14:46:14 adjucator kernel: [23809.092409] EXT4-fs error: 1 callbacks suppressed
Jul 6 14:46:14 adjucator kernel: [23809.092411] EXT4-fs error (device sdf): __ext4_find_entry:1531: inode #2: comm 6_dirty_io_sche: reading directory lblock 0
Jul 6 14:46:14 adjucator kernel: [23809.094172] JBD2: Error -5 detected when updating journal superblock for sde-8.
Jul 6 14:46:14 adjucator kernel: [23809.104492] EXT4-fs error (device sdg): __ext4_find_entry:1531: inode #393217: comm 10_dirty_io_sch: reading directory lblock 0
Jul 6 14:46:14 adjucator kernel: [23809.108348] EXT4-fs error (device sdg): __ext4_find_entry:1531: inode #393730: comm 10_dirty_io_sch: reading directory lblock 0
Jul 6 14:46:14 adjucator kernel: [23809.111265] EXT4-fs error (device sdg): __ext4_find_entry:1531: inode #393217: comm 6_dirty_io_sche: reading directory lblock 0
Jul 6 14:46:14 adjucator kernel: [23809.113977] EXT4-fs error (device sdg): __ext4_find_entry:1531: inode #393286: comm 6_dirty_io_sche: reading directory lblock 0
Jul 6 14:46:14 adjucator kernel: [23809.116380] EXT4-fs error (device sdg): __ext4_find_entry:1531: inode #393217: comm 6_dirty_io_sche: reading directory lblock 0
Jul 6 14:46:14 adjucator kernel: [23809.118711] EXT4-fs error (device sdg): __ext4_find_entry:1531: inode #393217: comm 10_dirty_io_sch: reading directory lblock 0
Jul 6 14:46:14 adjucator kernel: [23809.121290] EXT4-fs error (device sdg): __ext4_find_entry:1531: inode #393580: comm 10_dirty_io_sch: reading directory lblock 0
Jul 6 14:46:14 adjucator kernel: [23809.123657] EXT4-fs error (device sdg): __ext4_find_entry:1531: inode #393217: comm 10_dirty_io_sch: reading directory lblock 0
Jul 6 14:46:14 adjucator kernel: [23809.150312] EXT4-fs warning (device sdf): htree_dirblock_to_tree:997: inode #2: lblock 0: comm 2_dirty_io_sche: error -5 reading directory block
Jul 6 14:46:14 adjucator kernel: [23809.150432] EXT4-fs error (device sdf): __ext4_find_entry:1531: inode #2: comm 2_dirty_io_sche: reading directory lblock 0
Jul 6 14:46:14 adjucator kernel: [23809.265998] sd 6:0:0:1: rejecting I/O to offline device
Jul 6 14:46:14 adjucator kernel: [23809.267320] blk_update_request: I/O error, dev sde, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
Jul 6 14:46:14 adjucator kernel: [23809.268374] Buffer I/O error on dev sde, logical block 0, lost sync page write
Jul 6 14:46:14 adjucator kernel: [23809.271087] EXT4-fs (sde): previous I/O error to superblock detected
Jul 6 14:46:14 adjucator kernel: [23809.272347] sd 6:0:0:1: rejecting I/O to offline device
Jul 6 14:46:14 adjucator kernel: [23809.273314] blk_update_request: I/O error, dev sde, sector 0 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
Jul 6 14:46:14 adjucator kernel: [23809.274311] Buffer I/O error on dev sde, logical block 0, lost sync page write
Jul 6 14:46:14 adjucator kernel: [23809.275278] EXT4-fs (sde): I/O error while writing superblock
Jul 6 14:46:14 adjucator kernel: [23809.276200] EXT4-fs error (device sde): ext4_journal_check_start:61: Detected aborted journal
Jul 6 14:46:14 adjucator kernel: [23809.277139] EXT4-fs (sde): Remounting filesystem read-only
Jul 6 14:46:14 adjucator kernel: [23809.278113] sd 6:0:0:1: rejecting I/O to offline device
Jul 6 14:46:14 adjucator kernel: [23809.279042] Buffer I/O error on dev sde, logical block 0, lost sync page write
Jul 6 14:46:14 adjucator kernel: [23809.280373] EXT4-fs (sde): I/O error while writing superblock
Jul 6 14:46:14 adjucator kernel: [23809.280468] EXT4-fs (sde): I/O error while writing superblock
Jul 6 14:46:14 adjucator kernel: [23809.281845] EXT4-fs error (device sde): ext4_journal_check_start:61: Detected aborted journal
Jul 6 14:46:15 adjucator kernel: [23809.446117] sd 6:0:0:1: [sde] Synchronizing SCSI cache
Jul 6 14:46:15 adjucator kernel: [23809.446320] sd 6:0:0:1: [sde] Synchronize Cache(10) failed: Result: hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
Jul 6 14:46:15 adjucator kernel: [23810.096935] scsi host3: iSCSI Initiator over TCP/IP
Jul 6 14:46:15 adjucator kernel: [23810.106176] scsi 3:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5
Jul 6 14:46:15 adjucator kernel: [23810.106582] scsi 3:0:0:0: Attached scsi generic sg2 type 12
Jul 6 14:46:15 adjucator kernel: [23810.106979] scsi 3:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
Jul 6 14:46:15 adjucator kernel: [23810.107298] sd 3:0:0:1: Attached scsi generic sg3 type 0
Jul 6 14:46:15 adjucator kernel: [23810.107365] sd 3:0:0:1: Power-on or device reset occurred
Jul 6 14:46:15 adjucator kernel: [23810.108471] sd 3:0:0:1: [sdb] 2147483648 512-byte logical blocks: (1.10 TB/1.00 TiB)
Jul 6 14:46:15 adjucator kernel: [23810.108600] sd 3:0:0:1: [sdb] Write Protect is off
Jul 6 14:46:15 adjucator kernel: [23810.108602] sd 3:0:0:1: [sdb] Mode Sense: 69 00 10 08
Jul 6 14:46:15 adjucator kernel: [23810.108862] sd 3:0:0:1: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
Jul 6 14:46:16 adjucator kernel: [23811.142002] scsi host4: iSCSI Initiator over TCP/IP
Jul 6 14:46:16 adjucator kernel: [23811.150319] scsi 4:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5
Jul 6 14:46:16 adjucator kernel: [23811.151079] scsi 4:0:0:0: Attached scsi generic sg4 type 12
Jul 6 14:46:16 adjucator kernel: [23811.152169] scsi 4:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
Jul 6 14:46:16 adjucator kernel: [23811.152564] sd 4:0:0:1: Attached scsi generic sg5 type 0
Jul 6 14:46:16 adjucator kernel: [23811.152628] sd 4:0:0:1: Power-on or device reset occurred
Jul 6 14:46:16 adjucator kernel: [23811.156764] sd 4:0:0:1: [sdc] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB)
Jul 6 14:46:16 adjucator kernel: [23811.156857] sd 4:0:0:1: [sdc] Write Protect is off
Jul 6 14:46:16 adjucator kernel: [23811.156862] sd 4:0:0:1: [sdc] Mode Sense: 69 00 10 08
Jul 6 14:46:16 adjucator kernel: [23811.157015] sd 4:0:0:1: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA
Jul 6 14:46:16 adjucator kernel: [23811.183742] sd 3:0:0:1: [sdb] Attached SCSI disk
Jul 6 14:46:17 adjucator kernel: [23812.181004] sd 4:0:0:1: [sdc] Attached SCSI disk
Jul 6 14:46:17 adjucator kernel: [23812.219085] scsi host5: iSCSI Initiator over TCP/IP
Jul 6 14:46:17 adjucator kernel: [23812.229544] scsi 5:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5
Jul 6 14:46:17 adjucator kernel: [23812.230262] scsi 5:0:0:0: Attached scsi generic sg6 type 12
Jul 6 14:46:17 adjucator kernel: [23812.230784] scsi 5:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
Jul 6 14:46:17 adjucator kernel: [23812.231993] sd 5:0:0:1: Power-on or device reset occurred
Jul 6 14:46:17 adjucator kernel: [23812.232198] sd 5:0:0:1: [sdd] 209715200 512-byte logical blocks: (107 GB/100 GiB)
Jul 6 14:46:17 adjucator kernel: [23812.232231] sd 5:0:0:1: [sdd] Write Protect is off
Jul 6 14:46:17 adjucator kernel: [23812.232232] sd 5:0:0:1: [sdd] Mode Sense: 69 00 10 08
Jul 6 14:46:17 adjucator kernel: [23812.232299] sd 5:0:0:1: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
Jul 6 14:46:17 adjucator kernel: [23812.232688] sd 5:0:0:1: Attached scsi generic sg7 type 0
Jul 6 14:46:18 adjucator kernel: [23813.259412] sd 5:0:0:1: [sdd] Attached SCSI disk
Jul 6 14:46:20 adjucator kernel: [23814.824364] scsi host9: iSCSI Initiator over TCP/IP
Jul 6 14:46:20 adjucator kernel: [23814.829396] scsi 9:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5
Jul 6 14:46:20 adjucator kernel: [23814.829798] scsi 9:0:0:0: Attached scsi generic sg8 type 12
Jul 6 14:46:20 adjucator kernel: [23814.830560] scsi 9:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
Jul 6 14:46:20 adjucator kernel: [23814.831018] sd 9:0:0:1: Attached scsi generic sg9 type 0
Jul 6 14:46:20 adjucator kernel: [23814.831590] sd 9:0:0:1: Power-on or device reset occurred
Jul 6 14:46:20 adjucator kernel: [23814.831995] sd 9:0:0:1: [sdh] 209715200 512-byte logical blocks: (107 GB/100 GiB)
Jul 6 14:46:20 adjucator kernel: [23814.832051] sd 9:0:0:1: [sdh] Write Protect is off
Jul 6 14:46:20 adjucator kernel: [23814.832052] sd 9:0:0:1: [sdh] Mode Sense: 69 00 10 08
Jul 6 14:46:20 adjucator kernel: [23814.832177] sd 9:0:0:1: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA
Jul 6 14:46:21 adjucator kernel: [23815.854896] sd 9:0:0:1: [sdh] Attached SCSI disk
Jul 6 14:46:22 adjucator kernel: [23816.897957] scsi host10: iSCSI Initiator over TCP/IP
Jul 6 14:46:22 adjucator kernel: [23816.903450] scsi 10:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5
Jul 6 14:46:22 adjucator kernel: [23816.903774] scsi 10:0:0:0: Attached scsi generic sg10 type 12
Jul 6 14:46:22 adjucator kernel: [23816.904025] scsi 10:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
Jul 6 14:46:22 adjucator kernel: [23816.904256] sd 10:0:0:1: Attached scsi generic sg11 type 0
Jul 6 14:46:22 adjucator kernel: [23816.904921] sd 10:0:0:1: Power-on or device reset occurred
Jul 6 14:46:22 adjucator kernel: [23816.905870] sd 10:0:0:1: [sdi] 209715200 512-byte logical blocks: (107 GB/100 GiB)
Jul 6 14:46:22 adjucator kernel: [23816.905911] sd 10:0:0:1: [sdi] Write Protect is off
Jul 6 14:46:22 adjucator kernel: [23816.905912] sd 10:0:0:1: [sdi] Mode Sense: 69 00 10 08
Jul 6 14:46:22 adjucator kernel: [23816.906026] sd 10:0:0:1: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA
Jul 6 14:46:23 adjucator kernel: [23817.927068] sd 10:0:0:1: [sdi] tag#58 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 6 14:46:23 adjucator kernel: [23817.927071] sd 10:0:0:1: [sdi] tag#58 Sense Key : Medium Error [current]
Jul 6 14:46:23 adjucator kernel: [23817.927073] sd 10:0:0:1: [sdi] tag#58 Add. Sense: Unrecovered read error
Jul 6 14:46:23 adjucator kernel: [23817.927075] sd 10:0:0:1: [sdi] tag#58 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
Jul 6 14:46:23 adjucator kernel: [23817.927076] print_req_error: 1 callbacks suppressed
Jul 6 14:46:23 adjucator kernel: [23817.927077] blk_update_request: critical medium error, dev sdi, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jul 6 14:46:23 adjucator kernel: [23817.928488] Buffer I/O error on dev sdi, logical block 0, async page read
Jul 6 14:46:23 adjucator kernel: [23817.938105] sd 10:0:0:1: [sdi] tag#59 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 6 14:46:23 adjucator kernel: [23817.938109] sd 10:0:0:1: [sdi] tag#59 Sense Key : Medium Error [current]
Jul 6 14:46:23 adjucator kernel: [23817.938111] sd 10:0:0:1: [sdi] tag#59 Add. Sense: Unrecovered read error
Jul 6 14:46:23 adjucator kernel: [23817.938113] sd 10:0:0:1: [sdi] tag#59 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
Jul 6 14:46:23 adjucator kernel: [23817.938116] blk_update_request: critical medium error, dev sdi, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jul 6 14:46:23 adjucator kernel: [23817.939678] Buffer I/O error on dev sdi, logical block 0, async page read
Jul 6 14:46:23 adjucator kernel: [23817.941786] sd 10:0:0:1: [sdi] tag#60 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 6 14:46:23 adjucator kernel: [23817.941789] sd 10:0:0:1: [sdi] tag#60 Sense Key : Medium Error [current]
Jul 6 14:46:23 adjucator kernel: [23817.941791] sd 10:0:0:1: [sdi] tag#60 Add. Sense: Unrecovered read error
Jul 6 14:46:23 adjucator kernel: [23817.941835] sd 10:0:0:1: [sdi] tag#60 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
Jul 6 14:46:23 adjucator kernel: [23817.941838] blk_update_request: critical medium error, dev sdi, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jul 6 14:46:23 adjucator kernel: [23817.943228] Buffer I/O error on dev sdi, logical block 0, async page read
Jul 6 14:46:23 adjucator kernel: [23817.944479] ldm_validate_partition_table(): Disk read failed.
Jul 6 14:46:23 adjucator kernel: [23817.945589] sd 10:0:0:1: [sdi] tag#61 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 6 14:46:23 adjucator kernel: [23817.945592] sd 10:0:0:1: [sdi] tag#61 Sense Key : Medium Error [current]
Jul 6 14:46:23 adjucator kernel: [23817.945594] sd 10:0:0:1: [sdi] tag#61 Add. Sense: Unrecovered read error
Jul 6 14:46:23 adjucator kernel: [23817.945597] sd 10:0:0:1: [sdi] tag#61 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
Jul 6 14:46:23 adjucator kernel: [23817.945600] blk_update_request: critical medium error, dev sdi, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jul 6 14:46:23 adjucator kernel: [23817.947130] Buffer I/O error on dev sdi, logical block 0, async page read
Jul 6 14:46:23 adjucator kernel: [23817.949966] sd 10:0:0:1: [sdi] tag#68 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 6 14:46:23 adjucator kernel: [23817.949969] sd 10:0:0:1: [sdi] tag#68 Sense Key : Medium Error [current]
Jul 6 14:46:23 adjucator kernel: [23817.949971] sd 10:0:0:1: [sdi] tag#68 Add. Sense: Unrecovered read error
Jul 6 14:46:23 adjucator kernel: [23817.949973] sd 10:0:0:1: [sdi] tag#68 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
Jul 6 14:46:23 adjucator kernel: [23817.949976] blk_update_request: critical medium error, dev sdi, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jul 6 14:46:23 adjucator kernel: [23817.951422] Buffer I/O error on dev sdi, logical block 0, async page read
Jul 6 14:46:23 adjucator kernel: [23817.954207] sd 10:0:0:1: [sdi] tag#62 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 6 14:46:23 adjucator kernel: [23817.954210] sd 10:0:0:1: [sdi] tag#62 Sense Key : Medium Error [current]
Jul 6 14:46:23 adjucator kernel: [23817.954212] sd 10:0:0:1: [sdi] tag#62 Add. Sense: Unrecovered read error
Jul 6 14:46:23 adjucator kernel: [23817.954215] sd 10:0:0:1: [sdi] tag#62 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
Jul 6 14:46:23 adjucator kernel: [23817.954217] blk_update_request: critical medium error, dev sdi, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jul 6 14:46:23 adjucator kernel: [23817.955423] Buffer I/O error on dev sdi, logical block 0, async page read
Jul 6 14:46:23 adjucator kernel: [23817.958983] sd 10:0:0:1: [sdi] tag#63 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 6 14:46:23 adjucator kernel: [23817.958986] sd 10:0:0:1: [sdi] tag#63 Sense Key : Medium Error [current]
Jul 6 14:46:23 adjucator kernel: [23817.958988] sd 10:0:0:1: [sdi] tag#63 Add. Sense: Unrecovered read error
Jul 6 14:46:23 adjucator kernel: [23817.958990] sd 10:0:0:1: [sdi] tag#63 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
Jul 6 14:46:23 adjucator kernel: [23817.958993] blk_update_request: critical medium error, dev sdi, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jul 6 14:46:23 adjucator kernel: [23817.960348] Buffer I/O error on dev sdi, logical block 0, async page read
Jul 6 14:46:23 adjucator kernel: [23817.961300] Dev sdi: unable to read RDB block 0
Jul 6 14:46:23 adjucator kernel: [23817.963114] sd 10:0:0:1: [sdi] tag#48 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 6 14:46:23 adjucator kernel: [23817.963123] sd 10:0:0:1: [sdi] tag#48 Sense Key : Medium Error [current]
Jul 6 14:46:23 adjucator kernel: [23817.963125] sd 10:0:0:1: [sdi] tag#48 Add. Sense: Unrecovered read error
Jul 6 14:46:23 adjucator kernel: [23817.963126] sd 10:0:0:1: [sdi] tag#48 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
Jul 6 14:46:23 adjucator kernel: [23817.963128] blk_update_request: critical medium error, dev sdi, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jul 6 14:46:23 adjucator kernel: [23817.964388] Buffer I/O error on dev sdi, logical block 0, async page read
Jul 6 14:46:23 adjucator kernel: [23817.966883] sd 10:0:0:1: [sdi] tag#49 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 6 14:46:23 adjucator kernel: [23817.966886] sd 10:0:0:1: [sdi] tag#49 Sense Key : Medium Error [current]
Jul 6 14:46:23 adjucator kernel: [23817.966935] sd 10:0:0:1: [sdi] tag#49 Add. Sense: Unrecovered read error
Jul 6 14:46:23 adjucator kernel: [23817.966939] sd 10:0:0:1: [sdi] tag#49 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
Jul 6 14:46:23 adjucator kernel: [23817.966942] blk_update_request: critical medium error, dev sdi, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jul 6 14:46:23 adjucator kernel: [23817.968309] Buffer I/O error on dev sdi, logical block 0, async page read
Jul 6 14:46:23 adjucator kernel: [23817.971884] sd 10:0:0:1: [sdi] tag#50 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 6 14:46:23 adjucator kernel: [23817.971888] sd 10:0:0:1: [sdi] tag#50 Sense Key : Medium Error [current]
Jul 6 14:46:23 adjucator kernel: [23817.971891] sd 10:0:0:1: [sdi] tag#50 Add. Sense: Unrecovered read error
Jul 6 14:46:23 adjucator kernel: [23817.971893] sd 10:0:0:1: [sdi] tag#50 CDB: Read(10) 28 00 00 00 00 18 00 00 08 00
Jul 6 14:46:23 adjucator kernel: [23817.971897] blk_update_request: critical medium error, dev sdi, sector 24 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Jul 6 14:46:23 adjucator kernel: [23817.973521] Buffer I/O error on dev sdi, logical block 3, async page read
Jul 6 14:46:23 adjucator kernel: [23817.977573] sdi: unable to read partition table
Jul 6 14:46:23 adjucator kernel: [23817.979533] sd 10:0:0:1: [sdi] Attached SCSI disk
Jul 6 14:46:23 adjucator kernel: [23817.986031] scsi host11: iSCSI Initiator over TCP/IP
Jul 6 14:46:23 adjucator kernel: [23817.991339] scsi 11:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5
Jul 6 14:46:23 adjucator kernel: [23817.991773] scsi 11:0:0:0: Attached scsi generic sg12 type 12
Jul 6 14:46:23 adjucator kernel: [23817.992567] scsi 11:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
Jul 6 14:46:23 adjucator kernel: [23817.992931] sd 11:0:0:1: Power-on or device reset occurred
Jul 6 14:46:23 adjucator kernel: [23817.993571] sd 11:0:0:1: [sdj] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB)
Jul 6 14:46:23 adjucator kernel: [23817.993671] sd 11:0:0:1: [sdj] Write Protect is off
Jul 6 14:46:23 adjucator kernel: [23817.993673] sd 11:0:0:1: [sdj] Mode Sense: 69 00 10 08
Jul 6 14:46:23 adjucator kernel: [23817.993898] sd 11:0:0:1: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA
Jul 6 14:46:23 adjucator kernel: [23817.996197] sd 11:0:0:1: Attached scsi generic sg13 type 0
Jul 6 14:46:24 adjucator kernel: [23819.023306] sd 11:0:0:1: [sdj] Attached SCSI disk
Jul 6 14:46:30 adjucator kernel: [23824.590348] sd 9:0:0:1: [sdh] Synchronizing SCSI cache
Jul 6 14:46:30 adjucator kernel: [23824.664615] cni0: port 30(veth72cea02a) entered disabled state
Jul 6 14:46:30 adjucator kernel: [23824.673977] device veth72cea02a left promiscuous mode
Jul 6 14:46:30 adjucator kernel: [23824.673982] cni0: port 30(veth72cea02a) entered disabled state
Jul 6 14:46:31 adjucator kernel: [23825.794096] sd 10:0:0:1: [sdi] Synchronizing SCSI cache
Jul 6 14:46:35 adjucator kernel: [23829.793915] sd 11:0:0:1: [sdj] Synchronizing SCSI cache
Jul 6 14:46:36 adjucator kernel: [23830.985808] sd 5:0:0:1: [sdd] Synchronizing SCSI cache
Jul 6 14:46:41 adjucator kernel: [23835.618369] sd 4:0:0:1: [sdc] Synchronizing SCSI cache
Jul 6 14:46:42 adjucator kernel: [23836.861395] EXT4-fs (sdb): mounted filesystem with ordered data mode. Opts: (null)
Jul 6 14:46:51 adjucator kernel: [23846.348081] scsi host4: iSCSI Initiator over TCP/IP
Jul 6 14:46:52 adjucator kernel: [23846.360529] scsi 4:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5
Jul 6 14:46:52 adjucator kernel: [23846.361337] scsi 4:0:0:0: Attached scsi generic sg4 type 12
Jul 6 14:46:52 adjucator kernel: [23846.361711] scsi 4:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
Jul 6 14:46:52 adjucator kernel: [23846.362256] sd 4:0:0:1: Attached scsi generic sg5 type 0
Jul 6 14:46:52 adjucator kernel: [23846.362341] sd 4:0:0:1: Power-on or device reset occurred
Jul 6 14:46:52 adjucator kernel: [23846.363476] sd 4:0:0:1: [sdc] 209715200 512-byte logical blocks: (107 GB/100 GiB)
Jul 6 14:46:52 adjucator kernel: [23846.363555] sd 4:0:0:1: [sdc] Write Protect is off
Jul 6 14:46:52 adjucator kernel: [23846.363595] sd 4:0:0:1: [sdc] Mode Sense: 69 00 10 08
Jul 6 14:46:52 adjucator kernel: [23846.366261] sd 4:0:0:1: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA
Jul 6 14:46:53 adjucator kernel: [23847.406154] sd 4:0:0:1: [sdc] Attached SCSI disk
Jul 6 14:46:54 adjucator kernel: [23848.424820] scsi host5: iSCSI Initiator over TCP/IP
Jul 6 14:46:54 adjucator kernel: [23848.429891] scsi 5:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5
Jul 6 14:46:54 adjucator kernel: [23848.430308] scsi 5:0:0:0: Attached scsi generic sg6 type 12
Jul 6 14:46:54 adjucator kernel: [23848.431937] scsi 5:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
Jul 6 14:46:54 adjucator kernel: [23848.432658] sd 5:0:0:1: Attached scsi generic sg7 type 0
Jul 6 14:46:54 adjucator kernel: [23848.433342] sd 5:0:0:1: Power-on or device reset occurred
Jul 6 14:46:54 adjucator kernel: [23848.433848] sd 5:0:0:1: [sdd] 209715200 512-byte logical blocks: (107 GB/100 GiB)
Jul 6 14:46:54 adjucator kernel: [23848.433918] sd 5:0:0:1: [sdd] Write Protect is off
Jul 6 14:46:54 adjucator kernel: [23848.433920] sd 5:0:0:1: [sdd] Mode Sense: 69 00 10 08
Jul 6 14:46:54 adjucator kernel: [23848.434085] sd 5:0:0:1: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
Jul 6 14:46:55 adjucator kernel: [23849.451866] sd 5:0:0:1: [sdd] Attached SCSI disk
Jul 6 14:46:57 adjucator kernel: [23851.638239] scsi host6: iSCSI Initiator over TCP/IP
Jul 6 14:46:57 adjucator kernel: [23851.642986] scsi 6:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5
Jul 6 14:46:57 adjucator kernel: [23851.643895] scsi 6:0:0:0: Attached scsi generic sg8 type 12
Jul 6 14:46:57 adjucator kernel: [23851.646184] scsi 6:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
Jul 6 14:46:57 adjucator kernel: [23851.646652] sd 6:0:0:1: Attached scsi generic sg9 type 0
Jul 6 14:46:57 adjucator kernel: [23851.646904] sd 6:0:0:1: Power-on or device reset occurred
Jul 6 14:46:57 adjucator kernel: [23851.647279] sd 6:0:0:1: [sde] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB)
Jul 6 14:46:57 adjucator kernel: [23851.647313] sd 6:0:0:1: [sde] Write Protect is off
Jul 6 14:46:57 adjucator kernel: [23851.647314] sd 6:0:0:1: [sde] Mode Sense: 69 00 10 08
Jul 6 14:46:57 adjucator kernel: [23851.647399] sd 6:0:0:1: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA
Jul 6 14:46:58 adjucator kernel: [23852.695448] sd 6:0:0:1: [sde] Attached SCSI disk
Jul 6 14:46:59 adjucator kernel: [23853.783059] scsi host7: iSCSI Initiator over TCP/IP
Jul 6 14:46:59 adjucator kernel: [23853.789773] scsi 7:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5
Jul 6 14:46:59 adjucator kernel: [23853.790546] scsi 7:0:0:0: Attached scsi generic sg10 type 12
Jul 6 14:46:59 adjucator kernel: [23853.792228] scsi 7:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
Jul 6 14:46:59 adjucator kernel: [23853.793425] sd 7:0:0:1: Power-on or device reset occurred
Jul 6 14:46:59 adjucator kernel: [23853.793528] sd 7:0:0:1: Attached scsi generic sg11 type 0
Jul 6 14:46:59 adjucator kernel: [23853.794325] sd 7:0:0:1: [sdf] 209715200 512-byte logical blocks: (107 GB/100 GiB)
Jul 6 14:46:59 adjucator kernel: [23853.794475] sd 7:0:0:1: [sdf] Write Protect is off
Jul 6 14:46:59 adjucator kernel: [23853.794477] sd 7:0:0:1: [sdf] Mode Sense: 69 00 10 08
Jul 6 14:46:59 adjucator kernel: [23853.795010] sd 7:0:0:1: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA
Jul 6 14:47:00 adjucator kernel: [23854.820291] sd 7:0:0:1: [sdf] Attached SCSI disk
Jul 6 14:47:01 adjucator kernel: [23855.876776] scsi host8: iSCSI Initiator over TCP/IP
Jul 6 14:47:01 adjucator kernel: [23855.881771] scsi 8:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5
Jul 6 14:47:01 adjucator kernel: [23855.882454] scsi 8:0:0:0: Attached scsi generic sg12 type 12
Jul 6 14:47:01 adjucator kernel: [23855.883668] scsi 8:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
Jul 6 14:47:01 adjucator kernel: [23855.885862] scsi 8:0:0:1: Attached scsi generic sg13 type 0
Jul 6 14:47:01 adjucator kernel: [23855.887093] sd 8:0:0:1: Power-on or device reset occurred
Jul 6 14:47:01 adjucator kernel: [23855.887747] sd 8:0:0:1: [sdg] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB)
Jul 6 14:47:01 adjucator kernel: [23855.887815] sd 8:0:0:1: [sdg] Write Protect is off
Jul 6 14:47:01 adjucator kernel: [23855.887819] sd 8:0:0:1: [sdg] Mode Sense: 69 00 10 08
Jul 6 14:47:01 adjucator kernel: [23855.887964] sd 8:0:0:1: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA
Jul 6 14:47:02 adjucator kernel: [23856.914607] sd 8:0:0:1: [sdg] Attached SCSI disk
Jul 6 14:47:14 adjucator kernel: [23869.057172] EXT4-fs (sdd): mounted filesystem with ordered data mode. Opts: (null)
Jul 6 14:47:15 adjucator kernel: [23869.372424] EXT4-fs (sdc): mounted filesystem with ordered data mode. Opts: (null)
Jul 6 14:47:15 adjucator kernel: [23869.691179] IPv6: ADDRCONF(NETDEV_CHANGE): veth5e0f6084: link becomes ready
Jul 6 14:47:15 adjucator kernel: [23869.691282] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jul 6 14:47:15 adjucator kernel: [23869.708894] cni0: port 17(veth5e0f6084) entered blocking state
Jul 6 14:47:15 adjucator kernel: [23869.708896] cni0: port 17(veth5e0f6084) entered disabled state
Jul 6 14:47:15 adjucator kernel: [23869.708991] device veth5e0f6084 entered promiscuous mode
Jul 6 14:47:15 adjucator kernel: [23869.709020] cni0: port 17(veth5e0f6084) entered blocking state
Jul 6 14:47:15 adjucator kernel: [23869.709021] cni0: port 17(veth5e0f6084) entered forwarding state
Jul 6 14:47:37 adjucator kernel: [23891.672969] EXT4-fs (sde): mounted filesystem with ordered data mode. Opts: (null)
Jul 6 14:47:37 adjucator kernel: [23891.781706] EXT4-fs (sdf): mounted filesystem with ordered data mode. Opts: (null)
Jul 6 14:47:37 adjucator kernel: [23892.106547] EXT4-fs (sdg): mounted filesystem with ordered data mode. Opts: (null)
Jul 6 14:47:38 adjucator kernel: [23892.487969] cni0: port 18(veth5cb33dcf) entered blocking state
Jul 6 14:47:38 adjucator kernel: [23892.487971] cni0: port 18(veth5cb33dcf) entered disabled state
Jul 6 14:47:38 adjucator kernel: [23892.488035] device veth5cb33dcf entered promiscuous mode
Jul 6 14:47:38 adjucator kernel: [23892.488155] IPv6: ADDRCONF(NETDEV_CHANGE): veth5cb33dcf: link becomes ready
Jul 6 14:47:38 adjucator kernel: [23892.488211] cni0: port 18(veth5cb33dcf) entered blocking state
Jul 6 14:47:38 adjucator kernel: [23892.488212] cni0: port 18(veth5cb33dcf) entered forwarding state
Jul 6 14:47:38 adjucator kernel: [23892.488255] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jul 6 14:47:47 adjucator kernel: [23901.519828] IPv6: ADDRCONF(NETDEV_CHANGE): vethe8959545: link becomes ready
Jul 6 14:47:47 adjucator kernel: [23901.519947] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jul 6 14:47:47 adjucator kernel: [23901.522770] cni0: port 21(vethe8959545) entered blocking state
Jul 6 14:47:47 adjucator kernel: [23901.522771] cni0: port 21(vethe8959545) entered disabled state
Jul 6 14:47:47 adjucator kernel: [23901.523112] device vethe8959545 entered promiscuous mode
Jul 6 14:47:47 adjucator kernel: [23901.523288] cni0: port 21(vethe8959545) entered blocking state
Jul 6 14:47:47 adjucator kernel: [23901.523289] cni0: port 21(vethe8959545) entered forwarding state
Monitoring screenshots
Environment:
- Longhorn version: 1.1.1
- Installation method (e.g. Rancher Catalog App/Helm/Kubectl): k3sup
- Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: k3s v1.19.8+k3s1
- Number of management node in the cluster: 3
- Number of worker node in the cluster: 1 (only worker) + 3 (management + worker)
- Node config
- OS type and version: Ubuntu 20.04
- CPU per node: 4
- Memory per node: 16G
- Disk type(e.g. SSD/NVMe): SSD
- Network bandwidth between the nodes: 1Gbps
- Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): KVM
- Number of Longhorn volumes in the cluster: 20
Additional context Add any other context about the problem here.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 1
- Comments: 23 (10 by maintainers)
This could be caused by a lack of replica redundancy.
numberofreplicasis set to 1.numberofreplicasis set to 2, but because thereplica-soft-anti-affinity is enabled, this could cause replicas to be scheduled onto the same node.In both cases when node pressure kicks in, it might take a while to rebuild replicas.
@mstingl can you try with these settings to see if it makes difference?
numberofreplicasto at least 2.replica-soft-anti-affinityso it doesn’t duplicate the replica scheduling to the same node.