mayastor: Kernel panic on 6.1.14 in nvme_tcp

Describe the bug When io-engine dies for whatever reason, there is nothing logged in before crash. All app nodes using mounted volume crash (not only one where io-engine was running)

To Reproduce I do not know exact trigger, but was able to reproduce it by deleting pod with io-engine

Expected behavior This could be kernel bug but, I don’t know implementation details of mayastor for reporting it there.

Screenshots Reproduced by killing pod:

[ 2437.197218] nvme nvme4: starting error recovery
[ 2437.197835] nvme nvme4: Reconnecting in 10 seconds...
[ 2437.203659] nvme nvme3: starting error recovery
[ 2437.204157] nvme nvme3: Reconnecting in 10 seconds...
[ 2437.207250] nvme nvme0: starting error recovery
[ 2437.207703] nvme nvme0: Reconnecting in 10 seconds...
[ 2437.210338] nvme nvme1: starting error recovery
[ 2437.210768] nvme nvme1: Reconnecting in 10 seconds...
[ 2437.213314] nvme nvme2: starting error recovery
[ 2437.214038] nvme nvme2: Reconnecting in 10 seconds...
[ 2438.055584] block nvme1n1: no usable path - requeuing I/O
[ 2438.781645] BUG: kernel NULL pointer dereference, address: 0000000000000038
[ 2438.781848] #PF: supervisor read access in kernel mode
[ 2438.781996] #PF: error_code(0x0000) - not-present page
[ 2438.782144] PGD 0 P4D 0
[ 2438.782273] Oops: 0000 [#1] PREEMPT SMP NOPTI
[ 2438.782422] CPU: 4 PID: 9790 Comm: agent-ha-node Not tainted 6.1.14-200.fc37.x86_64 #1
[ 2438.782588] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 0.0.0 02/06/2015
[ 2438.782746] RIP: 0010:kernel_getsockname+0xb/0x20
[ 2438.782920] Code: 1f 44 00 00 48 8b 47 20 48 8b 40 20 ff e0 cc 66 90 cc 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f 44 00 00 48 8b 47 20 31 d2 <48> 8b 40 38 ff e0 cc 66 90 cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f
[ 2438.783230] RSP: 0018:ffffa54fc8a3bcf0 EFLAGS: 00010246
[ 2438.783377] RAX: 0000000000000000 RBX: 000000000000001f RCX: 0000000000000001
[ 2438.783533] RDX: 0000000000000000 RSI: ffffa54fc8a3bcf8 RDI: ffff914dbd047b80
[ 2438.783695] RBP: 0000000000001000 R08: 0000000000000004 R09: ffff914b4bc3b01e
[ 2438.783842] R10: ffffffffffffffff R11: 0000000000000000 R12: ffffa54fc8a3bcf8
[ 2438.783990] R13: ffff914b4bc3b000 R14: ffff914d61ccb400 R15: 0000000000000001
[ 2438.784143] FS:  00007fb9e69ca540(0000) GS:ffff915a3f900000(0000) knlGS:0000000000000000
[ 2438.784292] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2438.784438] CR2: 0000000000000038 CR3: 0000000f5247a001 CR4: 0000000000170ee0
[ 2438.784592] Call Trace:
[ 2438.784721]  <TASK>
[ 2438.784840]  nvme_tcp_get_address+0x59/0xd0 [nvme_tcp]
[ 2438.785010]  nvme_sysfs_show_address+0x1b/0x30 [nvme_core]
[ 2438.785217]  dev_attr_show+0x15/0x40
[ 2438.785396]  sysfs_kf_seq_show+0xa0/0xe0
[ 2438.785561]  seq_read_iter+0x11f/0x450
[ 2438.785739]  vfs_read+0x217/0x2f0
[ 2438.785899]  ksys_read+0x5b/0xd0
[ 2438.786044]  do_syscall_64+0x58/0x80
[ 2438.786208]  ? do_syscall_64+0x67/0x80
[ 2438.786355]  ? do_syscall_64+0x67/0x80
[ 2438.786504]  ? do_syscall_64+0x67/0x80
[ 2438.786654]  ? do_syscall_64+0x67/0x80
[ 2438.786799]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 2438.786951] RIP: 0033:0x7fb9e6acd7c4
[ 2438.787169] Code: 84 00 00 00 00 00 41 54 49 89 d4 55 48 89 f5 53 89 fb 48 83 ec 10 e8 0b 9e f8 ff 4c 89 e2 48 89 ee 89 df 41 89 c0 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 3c 44 89 c7 48 89 44 24 08 e8 67 9e f8 ff 48
[ 2438.787477] RSP: 002b:00007ffe1a90d910 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 2438.787632] RAX: ffffffffffffffda RBX: 000000000000000c RCX: 00007fb9e6acd7c4
[ 2438.787783] RDX: 0000000000001000 RSI: 000055bc105e8780 RDI: 000000000000000c
[ 2438.787938] RBP: 000055bc105e8780 R08: 0000000000000000 R09: 0000000000000000
[ 2438.788087] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000
[ 2438.788238] R13: 0000000000000000 R14: 00007ffe1a90da50 R15: 000000000000000c
[ 2438.788388]  </TASK>                                [ 2438.788507] Modules linked in: tls nfsd nfs_acl xt_multiport xt_set ipt_rpfilter ip_set_hash_ip ip_set_hash_net ipip tunnel4 ip_tunnel bpf_preload wireguard curve25519_x86_64 libcurve25519_generic ip6_udp_tunnel udp_tunnel rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver
nfs lockd grace fscache netfs veth nf_conntrack_netlink xt_addrtype xt_nat xt_statistic ipt_REJECT ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs xt_MASQUERADE xt_mark xt_conntrack xt_comment nft_compat overlay nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ip
v4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables nfnetlink sunrpc vfat intel_rapl_msr fat intel_rapl_common kvm_intel xfs kvm irqbypass rapl i2c_piix4 joydev virtio_balloon tcp_bbr sch_fq nvme_tcp n
vme_fabrics nvme_core nvme_common loop zram crct10dif_pclmul crc32_pclmul virtio_net crc32c_intel polyval_clmulni bochs polyval_generic drm_vram_helper ghash_clmulni_intel net_failover
[ 2438.788698]  drm_ttm_helper ttm sha512_ssse3 failover virtio_console virtio_scsi serio_raw ata_generic pata_acpi fuse qemu_fw_cfg
[ 2438.792182] CR2: 0000000000000038
[ 2438.792605] ---[ end trace 0000000000000000 ]---
[ 2438.793066] RIP: 0010:kernel_getsockname+0xb/0x20
[ 2438.793463] Code: 1f 44 00 00 48 8b 47 20 48 8b 40 20 ff e0 cc 66 90 cc 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f 44 00 00 48 8b 47 20 31 d2 <48> 8b 40 38 ff e0 cc 66 90 cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f
[ 2438.794286] RSP: 0018:ffffa54fc8a3bcf0 EFLAGS: 00010246
[ 2438.794767] RAX: 0000000000000000 RBX: 000000000000001f RCX: 0000000000000001
[ 2438.795176] RDX: 0000000000000000 RSI: ffffa54fc8a3bcf8 RDI: ffff914dbd047b80
[ 2438.795616] RBP: 0000000000001000 R08: 0000000000000004 R09: ffff914b4bc3b01e
[ 2438.796023] R10: ffffffffffffffff R11: 0000000000000000 R12: ffffa54fc8a3bcf8
[ 2438.796401] R13: ffff914b4bc3b000 R14: ffff914d61ccb400 R15: 0000000000000001
[ 2438.796806] FS:  00007fb9e69ca540(0000) GS:ffff915a3f900000(0000) knlGS:0000000000000000
[ 2438.797233] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2438.797867] CR2: 0000000000000038 CR3: 0000000f5247a001 CR4: 0000000000170ee0

caught happen organically:

[  503.657524] nvme nvme0: queue 0: timeout request 0x0 type 4
[  503.657532] nvme nvme0: starting error recovery
[  503.657881] nvme nvme0: failed nvme_keep_alive_end_io error=10
[  503.664549] nvme nvme0: Reconnecting in 10 seconds...
[  504.425613] nvme nvme1: queue 0: timeout request 0x0 type 4
[  504.425621] nvme nvme1: starting error recovery
[  504.425935] nvme nvme1: failed nvme_keep_alive_end_io error=10
[  504.432515] nvme nvme1: Reconnecting in 10 seconds...
[  505.129536] nvme nvme2: queue 0: timeout request 0x0 type 4
[  505.129544] nvme nvme2: starting error recovery
[  505.129806] nvme nvme2: failed nvme_keep_alive_end_io error=10
[  505.136528] nvme nvme2: Reconnecting in 10 seconds...
[  506.455595] BUG: kernel NULL pointer dereference, address: 0000000000000038
[  506.455687] #PF: supervisor read access in kernel mode
[  506.455741] #PF: error_code(0x0000) - not-present page
[  506.455793] PGD 0 P4D 0
[  506.455838] Oops: 0000 [#1] PREEMPT SMP NOPTI
[  506.455890] CPU: 6 PID: 7291 Comm: agent-ha-node Not tainted 6.1.14-200.fc37.x86_64 #1
[  506.455949] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 0.0.0 02/06/2015
[  506.456005] RIP: 0010:kernel_getsockname+0xb/0x20
[  506.456072] Code: 1f 44 00 00 48 8b 47 20 48 8b 40 20 ff e0 cc 66 90 cc 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f 44 00 00 48 8b 47 20 31 d2 <48> 8b 40 38 ff e0 cc 66 90 cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f
[  506.456182] RSP: 0018:ffffade047e37d00 EFLAGS: 00010246
[  506.456234] RAX: 0000000000000000 RBX: 000000000000001f RCX: 0000000000000001
[  506.456287] RDX: 0000000000000000 RSI: ffffade047e37d08 RDI: ffff8fb64c7afb80
[  506.456339] RBP: 0000000000001000 R08: 0000000000000004 R09: ffff8fb64390501e
[  506.456392] R10: ffffffffffffffff R11: 0000000000000000 R12: ffffade047e37d08
[  506.456445] R13: ffff8fb643905000 R14: ffff8fb686cbb000 R15: 0000000000000001
[  506.456502] FS:  00007f8080e99540(0000) GS:ffff8fc53f980000(0000) knlGS:0000000000000000
[  506.456561] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  506.456613] CR2: 0000000000000038 CR3: 0000000190e48001 CR4: 0000000000170ee0
[  506.456669] Call Trace:
[  506.456710]  <TASK>
[  506.456752]  nvme_tcp_get_address+0x59/0xd0 [nvme_tcp]
[  506.456828]  nvme_sysfs_show_address+0x1b/0x30 [nvme_core]
[  506.456924]  dev_attr_show+0x15/0x40
[  506.456984]  sysfs_kf_seq_show+0xa0/0xe0
[  506.457046]  seq_read_iter+0x11f/0x450
[  506.457100]  vfs_read+0x217/0x2f0
[  506.457154]  ksys_read+0x5b/0xd0
[  506.457204]  do_syscall_64+0x58/0x80
[  506.457267]  ? do_syscall_64+0x67/0x80
[  506.457329]  ? do_syscall_64+0x67/0x80
[  506.457379]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[  506.457436] RIP: 0033:0x7f8080f9c7c4
[  506.457572] Code: 84 00 00 00 00 00 41 54 49 89 d4 55 48 89 f5 53 89 fb 48 83 ec 10 e8 0b 9e f8 ff 4c 89 e2 48 89 ee 89 df 41 89 c0 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 3c 44 89 c7 48 89 44 24 08 e8 67 9e f8 ff 48
[  506.457684] RSP: 002b:00007ffe7fade290 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[  506.457740] RAX: ffffffffffffffda RBX: 000000000000000b RCX: 00007f8080f9c7c4
[  506.457792] RDX: 0000000000001000 RSI: 000055566374d190 RDI: 000000000000000b
[  506.457844] RBP: 000055566374d190 R08: 0000000000000000 R09: 0000000000000000
[  506.457897] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000001000
[  506.457949] R13: 0000000000000000 R14: 00007ffe7fade3d0 R15: 000000000000000b
[  506.458003]  </TASK>
[  506.458045] Modules linked in: iptable_nat iptable_filter br_netfilter bridge stp llc ip_tables xt_set xt_multiport ipt_rpfilter ip_set_hash_ip ip_set_hash_net ipip tunnel4 ip_tunnel bpf_preload wireguard curve25519_x86_64 libcurve25519_generic ip6_udp_tunnel udp_tunnel
 veth nf_conntrack_netlink xt_addrtype xt_statistic xt_nat ipt_REJECT ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs xt_MASQUERADE xt_mark xt_conntrack xt_comment nft_compat overlay nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject
nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables nfnetlink sunrpc vfat fat intel_rapl_msr intel_rapl_common xfs kvm_intel kvm joydev irqbypass rapl virtio_balloon i2c_piix4 tcp_bbr sch_fq nvme_tcp nvme_fabrics nvme_core nvme_co
mmon loop zram crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni polyval_generic bochs virtio_net drm_vram_helper ghash_clmulni_intel drm_ttm_helper ttm sha512_ssse3
[  506.458211]  net_failover virtio_console virtio_scsi failover serio_raw ata_generic pata_acpi fuse qemu_fw_cfg
[  506.460551] CR2: 0000000000000038
[  506.460979] ---[ end trace 0000000000000000 ]---
[  506.461392] RIP: 0010:kernel_getsockname+0xb/0x20
[  506.461812] Code: 1f 44 00 00 48 8b 47 20 48 8b 40 20 ff e0 cc 66 90 cc 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f 44 00 00 48 8b 47 20 31 d2 <48> 8b 40 38 ff e0 cc 66 90 cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f
[  506.462642] RSP: 0018:ffffade047e37d00 EFLAGS: 00010246
[  506.463047] RAX: 0000000000000000 RBX: 000000000000001f RCX: 0000000000000001
[  506.463450] RDX: 0000000000000000 RSI: ffffade047e37d08 RDI: ffff8fb64c7afb80
[  506.463874] RBP: 0000000000001000 R08: 0000000000000004 R09: ffff8fb64390501e
[  506.464258] R10: ffffffffffffffff R11: 0000000000000000 R12: ffffade047e37d08
[  506.464661] R13: ffff8fb643905000 R14: ffff8fb686cbb000 R15: 0000000000000001
[  506.465043] FS:  00007f8080e99540(0000) GS:ffff8fc53f980000(0000) knlGS:0000000000000000
[  506.465458] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  506.465869] CR2: 0000000000000038 CR3: 0000000190e48001 CR4: 0000000000170ee0

** OS info (please complete the following information)😗*

• Distro: Fedora 37 Server
• Kernel version: 6.1.14-200.fc37.x86_64
• MayaStor revision or container image docker.io/openebs/mayastor-io-engine:v2.0.0 (sha256:d27bbf8e07a3246ccb559fdc655873a6f27c5bf853a4de5eac7e4457fd064cab)

Additional context Not sure if that is any relevant: There are 3 nodes running on proxmox: 1 master and 2 worker nodes each worked node is pinned to one physical cpu only one node is handling storage

Let me know if you need any other info or have any ideas how to debug this further Each worker node is bount