* NVMeoF RDMA IB: I/O timeout and NULL pointer observed during rescan_controller/reset_controller with fio background
[not found] <1823328454.445263.1568796846850.JavaMail.zimbra@redhat.com>
@ 2019-09-18 9:13 ` Yi Zhang
2019-09-18 14:21 ` Max Gurtovoy
0 siblings, 1 reply; 7+ messages in thread
From: Yi Zhang @ 2019-09-18 9:13 UTC (permalink / raw)
To: linux-nvme; +Cc: sagi
Hello
I observed bellow I/O timeout and NULL pointer on 5.3.0, pls help check it, let me know if you need more info or test patch, thanks
reproducer:
1. do fio background testing
2. stress rescan/reset_controller operation
echo 1 > /sys/block/nvme2n1/device/nvme2/rescan_controller
echo 1 > /sys/block/nvme2n1/device/nvme2/reset_controller
kernel log:
[ 384.865550] nvme nvme2: creating 48 I/O queues.
[ 386.784069] nvme nvme2: creating 48 I/O queues.
[ 387.771002] nvme_ns_head_make_request: 159989 callbacks suppressed
[ 387.771004] block nvme2n1: no usable path - requeuing I/O
[ 387.771012] block nvme2n1: no usable path - requeuing I/O
[ 387.771051] block nvme2n1: no usable path - requeuing I/O
[ 387.771061] block nvme2n1: no usable path - requeuing I/O
[ 387.771065] block nvme2n1: no usable path - requeuing I/O
[ 387.771070] block nvme2n1: no usable path - requeuing I/O
[ 387.771077] block nvme2n1: no usable path - requeuing I/O
[ 387.771124] block nvme2n1: no usable path - requeuing I/O
[ 387.771146] block nvme2n1: no usable path - requeuing I/O
[ 387.771155] block nvme2n1: no usable path - requeuing I/O
[ 449.670780] nvme nvme2: I/O 0 QID 0 timeout
[ 449.691674] nvme nvme2: Connect command failed, error wo/DNR bit: 7
[ 449.697974] BUG: kernel NULL pointer dereference, address: 00000000000000c8
[ 449.704945] #PF: supervisor read access in kernel mode
[ 449.710082] #PF: error_code(0x0000) - not-present page
[ 449.715221] PGD 0 P4D 0
[ 449.717761] Oops: 0000 [#1] SMP PTI
[ 449.721254] CPU: 45 PID: 1145 Comm: kworker/u98:2 Not tainted 5.3.0 #12
[ 449.727866] Hardware name: Dell Inc. PowerEdge R740/00WGD1, BIOS 2.2.11 06/13/2019
[ 449.735448] Workqueue: nvme-reset-wq nvme_rdma_reset_ctrl_work [nvme_rdma]
[ 449.742325] RIP: 0010:rdma_disconnect+0x2e/0x90 [rdma_cm]
[ 449.747722] Code: 00 55 53 48 89 fb 48 8b bf a8 02 00 00 48 85 ff 74 65 48 8b 0b 0f b6 83 c0 01 00 00 48 69 c0 b8 00 00 00 48 03 81 80 04 00 00 <8b> 40 10 a8 04 75 0d a8 08 74 42 5b 31 f6 5d e9 fe 72 b2 ff 48 89
[ 449.766466] RSP: 0018:ffffb01f87323de0 EFLAGS: 00010206
[ 449.771693] RAX: 00000000000000b8 RBX: ffff9e4d5a474c00 RCX: ffff9e4d5a475c00
[ 449.778825] RDX: 0000000000000819 RSI: ffff9e4dffb96b88 RDI: ffff9e41af404e00
[ 449.785956] RBP: 0000000000000000 R08: 00000000000008e7 R09: 000000000000002d
[ 449.793098] R10: ffffb01f87323df8 R11: ffffb01f87323ac0 R12: 0000000000000007
[ 449.800229] R13: ffff9e4db1332000 R14: ffff9e42045e2540 R15: ffff9e4db1332000
[ 449.807361] FS: 0000000000000000(0000) GS:ffff9e4dffb80000(0000) knlGS:0000000000000000
[ 449.815456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 449.821195] CR2: 00000000000000c8 CR3: 000000183e60a003 CR4: 00000000007606e0
[ 449.828327] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 449.835457] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 449.842581] PKRU: 55555554
[ 449.845287] Call Trace:
[ 449.847758] nvme_rdma_start_queue+0x8f/0xc0 [nvme_rdma]
[ 449.853070] nvme_rdma_setup_ctrl+0x4ef/0x6a0 [nvme_rdma]
[ 449.858469] nvme_rdma_reset_ctrl_work+0x4e/0x70 [nvme_rdma]
[ 449.864132] process_one_work+0x1a1/0x360
[ 449.868140] worker_thread+0x1c9/0x380
[ 449.871894] ? process_one_work+0x360/0x360
[ 449.876081] kthread+0x10c/0x130
[ 449.879310] ? kthread_create_on_node+0x60/0x60
[ 449.883846] ret_from_fork+0x35/0x40
[ 449.887422] Modules linked in: nvme_rdma nvme_fabrics nvmet_rdma nvmet 8021q garp mrp stp llc ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp iw_cxgb4 libcxgb mlx5_ib vfat fat opa_vnic ib_umad ib_ipoib intel_rapl_msr intel_rapl_common isst_if_common rpcrdma sunrpc skx_edac nfit rdma_ucm libnvdimm ib_iser rdma_cm x86_pkg_temp_thermal intel_powerclamp iw_cm ib_cm coretemp libiscsi kvm_intel scsi_transport_iscsi hfi1 kvm iTCO_wdt iTCO_vendor_support dcdbas ipmi_ssif irqbypass rdmavt bnxt_re crct10dif_pclmul crc32_pclmul ib_uverbs ghash_clmulni_intel intel_cstate intel_uncore ib_core dell_smbios mei_me ipmi_si intel_rapl_perf wmi_bmof dell_wmi_descriptor pcspkr sg mei i2c_i801 lpc_ich ipmi_devintf ipmi_msghandler acpi_power_meter xfs libcrc32c sd_mod mgag200 i2c_algo_bit drm_vram_helper mlx5_core ttm drm_kms_helper syscopyarea sysfillrect sysimgblt ahci fb_sys_fops nvme libahci csiostor drm cxgb4 bnxt_en crc32c_intel nvme_core libata megaraid_sas scsi_trans
port_fc
[ 449.887456] mlxfw tg3 wmi dm_mirror dm_region_hash dm_log dm_mod
[ 449.980847] CR2: 00000000000000c8
[ 449.984182] ---[ end trace aeab63ac2e6510db ]---
[ 450.046884] RIP: 0010:rdma_disconnect+0x2e/0x90 [rdma_cm]
[ 450.052282] Code: 00 55 53 48 89 fb 48 8b bf a8 02 00 00 48 85 ff 74 65 48 8b 0b 0f b6 83 c0 01 00 00 48 69 c0 b8 00 00 00 48 03 81 80 04 00 00 <8b> 40 10 a8 04 75 0d a8 08 74 42 5b 31 f6 5d e9 fe 72 b2 ff 48 89
[ 450.071027] RSP: 0018:ffffb01f87323de0 EFLAGS: 00010206
[ 450.076255] RAX: 00000000000000b8 RBX: ffff9e4d5a474c00 RCX: ffff9e4d5a475c00
[ 450.083387] RDX: 0000000000000819 RSI: ffff9e4dffb96b88 RDI: ffff9e41af404e00
[ 450.090517] RBP: 0000000000000000 R08: 00000000000008e7 R09: 000000000000002d
[ 450.097642] R10: ffffb01f87323df8 R11: ffffb01f87323ac0 R12: 0000000000000007
[ 450.104774] R13: ffff9e4db1332000 R14: ffff9e42045e2540 R15: ffff9e4db1332000
[ 450.111899] FS: 0000000000000000(0000) GS:ffff9e4dffb80000(0000) knlGS:0000000000000000
[ 450.119985] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 450.125729] CR2: 00000000000000c8 CR3: 000000183e60a003 CR4: 00000000007606e0
[ 450.132853] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 450.139987] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 450.147118] PKRU: 55555554
[ 450.149831] Kernel panic - not syncing: Fatal exception
[ 450.155135] Kernel Offset: 0x17200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 450.224883] ---[ end Kernel panic - not syncing: Fatal exception ]---
# gdb /lib/modules/5.3.0/kernel/drivers/nvme/host/nvme-rdma.ko
Reading symbols from /lib/modules/5.3.0/kernel/drivers/nvme/host/nvme-rdma.ko...done.
(gdb) l *(nvme_rdma_start_queue+0x8f)
0x65f is in nvme_rdma_start_queue (drivers/nvme/host/rdma.c:568).
563 }
564
565 static void __nvme_rdma_stop_queue(struct nvme_rdma_queue *queue)
566 {
567 rdma_disconnect(queue->cm_id);
568 ib_drain_qp(queue->qp);
569 }
570
571 static void nvme_rdma_stop_queue(struct nvme_rdma_queue *queue)
572 {
(gdb)
# lspci | grep -i mel
3b:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]
Best Regards,
Yi Zhang
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: NVMeoF RDMA IB: I/O timeout and NULL pointer observed during rescan_controller/reset_controller with fio background
2019-09-18 9:13 ` NVMeoF RDMA IB: I/O timeout and NULL pointer observed during rescan_controller/reset_controller with fio background Yi Zhang
@ 2019-09-18 14:21 ` Max Gurtovoy
2019-09-20 3:37 ` Yi Zhang
0 siblings, 1 reply; 7+ messages in thread
From: Max Gurtovoy @ 2019-09-18 14:21 UTC (permalink / raw)
To: Yi Zhang, linux-nvme; +Cc: sagi
On 9/18/2019 12:13 PM, Yi Zhang wrote:
> Hello
> I observed bellow I/O timeout and NULL pointer on 5.3.0, pls help check it, let me know if you need more info or test patch, thanks
Hi,
Can you try to repro it with older kernel (5.2.0) ?
I want to understand if its a regression from the last few months...
>
> reproducer:
> 1. do fio background testing
> 2. stress rescan/reset_controller operation
> echo 1 > /sys/block/nvme2n1/device/nvme2/rescan_controller
> echo 1 > /sys/block/nvme2n1/device/nvme2/reset_controller
>
> kernel log:
> [ 384.865550] nvme nvme2: creating 48 I/O queues.
> [ 386.784069] nvme nvme2: creating 48 I/O queues.
> [ 387.771002] nvme_ns_head_make_request: 159989 callbacks suppressed
> [ 387.771004] block nvme2n1: no usable path - requeuing I/O
> [ 387.771012] block nvme2n1: no usable path - requeuing I/O
> [ 387.771051] block nvme2n1: no usable path - requeuing I/O
> [ 387.771061] block nvme2n1: no usable path - requeuing I/O
> [ 387.771065] block nvme2n1: no usable path - requeuing I/O
> [ 387.771070] block nvme2n1: no usable path - requeuing I/O
> [ 387.771077] block nvme2n1: no usable path - requeuing I/O
> [ 387.771124] block nvme2n1: no usable path - requeuing I/O
> [ 387.771146] block nvme2n1: no usable path - requeuing I/O
> [ 387.771155] block nvme2n1: no usable path - requeuing I/O
> [ 449.670780] nvme nvme2: I/O 0 QID 0 timeout
> [ 449.691674] nvme nvme2: Connect command failed, error wo/DNR bit: 7
> [ 449.697974] BUG: kernel NULL pointer dereference, address: 00000000000000c8
> [ 449.704945] #PF: supervisor read access in kernel mode
> [ 449.710082] #PF: error_code(0x0000) - not-present page
> [ 449.715221] PGD 0 P4D 0
> [ 449.717761] Oops: 0000 [#1] SMP PTI
> [ 449.721254] CPU: 45 PID: 1145 Comm: kworker/u98:2 Not tainted 5.3.0 #12
> [ 449.727866] Hardware name: Dell Inc. PowerEdge R740/00WGD1, BIOS 2.2.11 06/13/2019
> [ 449.735448] Workqueue: nvme-reset-wq nvme_rdma_reset_ctrl_work [nvme_rdma]
> [ 449.742325] RIP: 0010:rdma_disconnect+0x2e/0x90 [rdma_cm]
> [ 449.747722] Code: 00 55 53 48 89 fb 48 8b bf a8 02 00 00 48 85 ff 74 65 48 8b 0b 0f b6 83 c0 01 00 00 48 69 c0 b8 00 00 00 48 03 81 80 04 00 00 <8b> 40 10 a8 04 75 0d a8 08 74 42 5b 31 f6 5d e9 fe 72 b2 ff 48 89
> [ 449.766466] RSP: 0018:ffffb01f87323de0 EFLAGS: 00010206
> [ 449.771693] RAX: 00000000000000b8 RBX: ffff9e4d5a474c00 RCX: ffff9e4d5a475c00
> [ 449.778825] RDX: 0000000000000819 RSI: ffff9e4dffb96b88 RDI: ffff9e41af404e00
> [ 449.785956] RBP: 0000000000000000 R08: 00000000000008e7 R09: 000000000000002d
> [ 449.793098] R10: ffffb01f87323df8 R11: ffffb01f87323ac0 R12: 0000000000000007
> [ 449.800229] R13: ffff9e4db1332000 R14: ffff9e42045e2540 R15: ffff9e4db1332000
> [ 449.807361] FS: 0000000000000000(0000) GS:ffff9e4dffb80000(0000) knlGS:0000000000000000
> [ 449.815456] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 449.821195] CR2: 00000000000000c8 CR3: 000000183e60a003 CR4: 00000000007606e0
> [ 449.828327] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 449.835457] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [ 449.842581] PKRU: 55555554
> [ 449.845287] Call Trace:
> [ 449.847758] nvme_rdma_start_queue+0x8f/0xc0 [nvme_rdma]
> [ 449.853070] nvme_rdma_setup_ctrl+0x4ef/0x6a0 [nvme_rdma]
> [ 449.858469] nvme_rdma_reset_ctrl_work+0x4e/0x70 [nvme_rdma]
> [ 449.864132] process_one_work+0x1a1/0x360
> [ 449.868140] worker_thread+0x1c9/0x380
> [ 449.871894] ? process_one_work+0x360/0x360
> [ 449.876081] kthread+0x10c/0x130
> [ 449.879310] ? kthread_create_on_node+0x60/0x60
> [ 449.883846] ret_from_fork+0x35/0x40
> [ 449.887422] Modules linked in: nvme_rdma nvme_fabrics nvmet_rdma nvmet 8021q garp mrp stp llc ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp iw_cxgb4 libcxgb mlx5_ib vfat fat opa_vnic ib_umad ib_ipoib intel_rapl_msr intel_rapl_common isst_if_common rpcrdma sunrpc skx_edac nfit rdma_ucm libnvdimm ib_iser rdma_cm x86_pkg_temp_thermal intel_powerclamp iw_cm ib_cm coretemp libiscsi kvm_intel scsi_transport_iscsi hfi1 kvm iTCO_wdt iTCO_vendor_support dcdbas ipmi_ssif irqbypass rdmavt bnxt_re crct10dif_pclmul crc32_pclmul ib_uverbs ghash_clmulni_intel intel_cstate intel_uncore ib_core dell_smbios mei_me ipmi_si intel_rapl_perf wmi_bmof dell_wmi_descriptor pcspkr sg mei i2c_i801 lpc_ich ipmi_devintf ipmi_msghandler acpi_power_meter xfs libcrc32c sd_mod mgag200 i2c_algo_bit drm_vram_helper mlx5_core ttm drm_kms_helper syscopyarea sysfillrect sysimgblt ahci fb_sys_fops nvme libahci csiostor drm cxgb4 bnxt_en crc32c_intel nvme_core libata megaraid_sas scsi_tra
nsport_fc
> [ 449.887456] mlxfw tg3 wmi dm_mirror dm_region_hash dm_log dm_mod
> [ 449.980847] CR2: 00000000000000c8
> [ 449.984182] ---[ end trace aeab63ac2e6510db ]---
> [ 450.046884] RIP: 0010:rdma_disconnect+0x2e/0x90 [rdma_cm]
> [ 450.052282] Code: 00 55 53 48 89 fb 48 8b bf a8 02 00 00 48 85 ff 74 65 48 8b 0b 0f b6 83 c0 01 00 00 48 69 c0 b8 00 00 00 48 03 81 80 04 00 00 <8b> 40 10 a8 04 75 0d a8 08 74 42 5b 31 f6 5d e9 fe 72 b2 ff 48 89
> [ 450.071027] RSP: 0018:ffffb01f87323de0 EFLAGS: 00010206
> [ 450.076255] RAX: 00000000000000b8 RBX: ffff9e4d5a474c00 RCX: ffff9e4d5a475c00
> [ 450.083387] RDX: 0000000000000819 RSI: ffff9e4dffb96b88 RDI: ffff9e41af404e00
> [ 450.090517] RBP: 0000000000000000 R08: 00000000000008e7 R09: 000000000000002d
> [ 450.097642] R10: ffffb01f87323df8 R11: ffffb01f87323ac0 R12: 0000000000000007
> [ 450.104774] R13: ffff9e4db1332000 R14: ffff9e42045e2540 R15: ffff9e4db1332000
> [ 450.111899] FS: 0000000000000000(0000) GS:ffff9e4dffb80000(0000) knlGS:0000000000000000
> [ 450.119985] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 450.125729] CR2: 00000000000000c8 CR3: 000000183e60a003 CR4: 00000000007606e0
> [ 450.132853] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 450.139987] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [ 450.147118] PKRU: 55555554
> [ 450.149831] Kernel panic - not syncing: Fatal exception
> [ 450.155135] Kernel Offset: 0x17200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> [ 450.224883] ---[ end Kernel panic - not syncing: Fatal exception ]---
>
> # gdb /lib/modules/5.3.0/kernel/drivers/nvme/host/nvme-rdma.ko
> Reading symbols from /lib/modules/5.3.0/kernel/drivers/nvme/host/nvme-rdma.ko...done.
>
> (gdb) l *(nvme_rdma_start_queue+0x8f)
> 0x65f is in nvme_rdma_start_queue (drivers/nvme/host/rdma.c:568).
> 563 }
> 564
> 565 static void __nvme_rdma_stop_queue(struct nvme_rdma_queue *queue)
> 566 {
> 567 rdma_disconnect(queue->cm_id);
> 568 ib_drain_qp(queue->qp);
> 569 }
> 570
> 571 static void nvme_rdma_stop_queue(struct nvme_rdma_queue *queue)
> 572 {
> (gdb)
>
> # lspci | grep -i mel
> 3b:00.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]
>
>
>
> Best Regards,
> Yi Zhang
>
>
>
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme@lists.infradead.org
> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.infradead.org%2Fmailman%2Flistinfo%2Flinux-nvme&data=02%7C01%7Cmaxg%40mellanox.com%7C81816cef9009489a9a9608d73c1887fd%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C1%7C637043948365925030&sdata=xDFYdW4FFjVGrqJ74xlr4tTN4hs4Qwwvt7eaGoui%2F%2Fg%3D&reserved=0
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: NVMeoF RDMA IB: I/O timeout and NULL pointer observed during rescan_controller/reset_controller with fio background
2019-09-18 14:21 ` Max Gurtovoy
@ 2019-09-20 3:37 ` Yi Zhang
2019-09-20 16:58 ` Sagi Grimberg
0 siblings, 1 reply; 7+ messages in thread
From: Yi Zhang @ 2019-09-20 3:37 UTC (permalink / raw)
To: Max Gurtovoy, linux-nvme; +Cc: sagi
On 9/18/19 10:21 PM, Max Gurtovoy wrote:
>
> On 9/18/2019 12:13 PM, Yi Zhang wrote:
>> Hello
>> I observed bellow I/O timeout and NULL pointer on 5.3.0, pls help
>> check it, let me know if you need more info or test patch, thanks
>
> Hi,
>
> Can you try to repro it with older kernel (5.2.0) ?
>
> I want to understand if its a regression from the last few months...
>
Hi Max
From my test, seems there are two issue here, one it I/O timeout during
reset_controller,
the other one is reconnecting after I/O timeout lead kernel NULL
pointer, and it was introduced from v5.2.0-rc1
v5.1
#echo 1 > /sys/block/nvme2n1/device/reset_controller
echo: write error: Network dropped connection on reset
[55542.073929] nvme nvme2: Please enable CONFIG_NVME_MULTIPATH for full
support of multi-port devices.
[55542.083098] nvme nvme2: creating 48 I/O queues.
[55607.920015] nvme nvme2: I/O 0 QID 0 timeout
[55607.937863] nvme nvme2: Connect command failed, error wo/DNR bit: 7
[55607.944157] nvme nvme2: failed to connect queue: 0 ret=7
[55607.949478] nvme nvme2: Reconnecting in 10 seconds...
[55618.166395] nvme nvme2: Please enable CONFIG_NVME_MULTIPATH for full
support of multi-port devices.
[55618.175575] nvme nvme2: creating 48 I/O queues.
[55618.960387] nvme nvme2: Successfully reconnected (2 attempts)
v5.2.0-rc1
[ 1463.516023] nvme nvme2: creating 48 I/O queues.
[ 1464.311387] nvme2c2n1: detected capacity change from 0 to 1600321314816
[ 1464.435225] nvme2c2n1: detected capacity change from 0 to 1600321314816
[ 1486.515899] nvme_ns_head_make_request: 167 callbacks suppressed
[ 1486.515903] block nvme2n1: no path available - requeuing I/O
[ 1486.527513] block nvme2n1: no path available - requeuing I/O
[ 1486.533183] block nvme2n1: no path available - requeuing I/O
[ 1486.538852] block nvme2n1: no path available - requeuing I/O
[ 1486.544536] block nvme2n1: no path available - requeuing I/O
[ 1486.550214] block nvme2n1: no path available - requeuing I/O
[ 1486.555880] block nvme2n1: no path available - requeuing I/O
[ 1486.561548] block nvme2n1: no path available - requeuing I/O
[ 1486.567232] block nvme2n1: no path available - requeuing I/O
[ 1486.572910] block nvme2n1: no path available - requeuing I/O
[ 1526.450439] nvme nvme2: I/O 0 QID 0 timeout
[ 1526.469411] nvme nvme2: Connect command failed, error wo/DNR bit: 7
[ 1526.475702] nvme nvme2: failed to connect queue: 0 ret=7
[ 1526.481034] general protection fault: 0000 [#1] SMP PTI
[ 1526.486281] CPU: 21 PID: 16138 Comm: kworker/u98:2 Not tainted
5.2.0-rc1 #15
[ 1526.493336] Hardware name: Dell Inc. PowerEdge R740/00WGD1, BIOS
2.2.11 06/13/2019
[ 1526.500903] Workqueue: nvme-reset-wq nvme_rdma_reset_ctrl_work
[nvme_rdma]
[ 1526.507794] RIP: 0010:__x86_indirect_thunk_rax+0x10/0x20
[ 1526.513120] Code: 03 01 d1 89 ca e9 80 ec ca ff 48 8d 0c c8 e9 81 ea
ca ff 90 90 90 90 90 90 90 e8 07 00 00 00 f3 90 0f ae e8 eb f9 48 89 04
24 <c3> 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 00 00 e8 07 00 00 00 f3
[ 1526.531864] RSP: 0018:ffffc19d2ade7e18 EFLAGS: 00010246
[ 1526.537089] RAX: 730424073f013200 RBX: 0000000000000000 RCX:
0000000000000001
[ 1526.544223] RDX: 0000000000000040 RSI: 0000000be2a19e00 RDI:
ffff9e02ff907d80
[ 1526.551355] RBP: 0000000000000007 R08: 0000000000000000 R09:
0000000000000015
[ 1526.558513] R10: 0000000000000000 R11: ffffc19d2ade7b60 R12:
0000000000000000
[ 1526.565644] R13: ffff9e0e8b9a02f8 R14: ffff9e0e8b9a0000 R15:
ffff9e0e8b9a06e0
[ 1526.572779] FS: 0000000000000000(0000) GS:ffff9e0f06680000(0000)
knlGS:0000000000000000
[ 1526.580863] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1526.586634] CR2: 00007f0179cd1ec8 CR3: 000000183f00a006 CR4:
00000000007606e0
[ 1526.593769] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 1526.600900] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 1526.608033] PKRU: 55555554
[ 1526.610753] Call Trace:
[ 1526.613201] ? nvme_rdma_setup_ctrl+0x1ae/0x6b0 [nvme_rdma]
[ 1526.618782] ? nvme_rdma_reset_ctrl_work+0x4e/0x70 [nvme_rdma]
[ 1526.624642] ? process_one_work+0x1a1/0x3a0
[ 1526.628824] ? worker_thread+0x1c9/0x380
[ 1526.632751] ? process_one_work+0x3a0/0x3a0
[ 1526.636937] ? kthread+0x10c/0x130
[ 1526.640341] ? kthread_create_on_node+0x60/0x60
[ 1526.644876] ? ret_from_fork+0x35/0x40
[ 1526.648635] Modules linked in: nvme_rdma nvme_fabrics nvmet_rdma
nvmet 8021q garp mrp stp llc ib_isert iscsi_target_mod ib_srpt
target_core_mod ib_srp scsi_transport_srp vfat fat mlx5_ib opa_vnic
ib_umad ib_ipoib intel_rapl skx_edac nfit libnvdimm x86_pkg_temp_thermal
intel_powerclamp coretemp rpcrdma sunrpc kvm_intel hfi1 kvm rdma_ucm
ib_iser libiscsi scsi_transport_iscsi iw_cxgb4 ipmi_ssif rdma_cm iw_cm
rdmavt ib_cm bnxt_re libcxgb irqbypass ib_uverbs crct10dif_pclmul
crc32_pclmul ghash_clmulni_intel intel_cstate iTCO_wdt
iTCO_vendor_support intel_uncore ib_core dcdbas mei_me dell_smbios
ipmi_si wmi_bmof dell_wmi_descriptor intel_rapl_perf sg pcspkr mei
ipmi_devintf i2c_i801 lpc_ich ipmi_msghandler acpi_power_meter xfs
libcrc32c sd_mod mgag200 i2c_algo_bit drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops mlx5_core ttm drm cxgb4 csiostor ahci
nvme libahci bnxt_en crc32c_intel nvme_core libata megaraid_sas
scsi_transport_fc mlxfw tg3 wmi dm_mirror dm_region_hash dm_log dm_mod
[ 1526.736119] ---[ end trace 0124b9a2f8dcf21d ]---
[ 1526.757400] RIP: 0010:__x86_indirect_thunk_rax+0x10/0x20
[ 1526.762713] Code: 03 01 d1 89 ca e9 80 ec ca ff 48 8d 0c c8 e9 81 ea
ca ff 90 90 90 90 90 90 90 e8 07 00 00 00 f3 90 0f ae e8 eb f9 48 89 04
24 <c3> 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 00 00 e8 07 00 00 00 f3
[ 1526.781472] RSP: 0018:ffffc19d2ade7e18 EFLAGS: 00010246
[ 1526.786715] RAX: 730424073f013200 RBX: 0000000000000000 RCX:
0000000000000001
[ 1526.793863] RDX: 0000000000000040 RSI: 0000000be2a19e00 RDI:
ffff9e02ff907d80
[ 1526.801014] RBP: 0000000000000007 R08: 0000000000000000 R09:
0000000000000015
[ 1526.808162] R10: 0000000000000000 R11: ffffc19d2ade7b60 R12:
0000000000000000
[ 1526.815304] R13: ffff9e0e8b9a02f8 R14: ffff9e0e8b9a0000 R15:
ffff9e0e8b9a06e0
[ 1526.822457] FS: 0000000000000000(0000) GS:ffff9e0f06680000(0000)
knlGS:0000000000000000
[ 1526.830558] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1526.836337] CR2: 00007f0179cd1ec8 CR3: 000000183f00a006 CR4:
00000000007606e0
[ 1526.843472] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 1526.850628] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 1526.857767] PKRU: 55555554
[ 1526.860501] Kernel panic - not syncing: Fatal exception
[ 1526.865813] Kernel Offset: 0x13000000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 1526.893337] ---[ end Kernel panic - not syncing: Fatal exception ]---
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: NVMeoF RDMA IB: I/O timeout and NULL pointer observed during rescan_controller/reset_controller with fio background
2019-09-20 3:37 ` Yi Zhang
@ 2019-09-20 16:58 ` Sagi Grimberg
2019-09-23 15:25 ` Max Gurtovoy
0 siblings, 1 reply; 7+ messages in thread
From: Sagi Grimberg @ 2019-09-20 16:58 UTC (permalink / raw)
To: Yi Zhang, Max Gurtovoy, linux-nvme
Thanks for reporting Yi,
Does this fix your issue?
--
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index dfa07bb9dfeb..981da9ce3cfc 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -614,7 +614,8 @@ static int nvme_rdma_start_queue(struct
nvme_rdma_ctrl *ctrl, int idx)
if (!ret) {
set_bit(NVME_RDMA_Q_LIVE, &queue->flags);
} else {
- __nvme_rdma_stop_queue(queue);
+ if (test_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags))
+ __nvme_rdma_stop_queue(queue);
dev_info(ctrl->ctrl.device,
"failed to connect queue: %d ret=%d\n", idx, ret);
}
--
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: NVMeoF RDMA IB: I/O timeout and NULL pointer observed during rescan_controller/reset_controller with fio background
2019-09-20 16:58 ` Sagi Grimberg
@ 2019-09-23 15:25 ` Max Gurtovoy
2019-09-24 4:52 ` Yi Zhang
2019-09-27 8:17 ` Yi Zhang
0 siblings, 2 replies; 7+ messages in thread
From: Max Gurtovoy @ 2019-09-23 15:25 UTC (permalink / raw)
To: Sagi Grimberg, Yi Zhang, linux-nvme
Any update Yi ?
we must fix this issue..
On 9/20/2019 7:58 PM, Sagi Grimberg wrote:
> Thanks for reporting Yi,
>
> Does this fix your issue?
>
> --
> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
> index dfa07bb9dfeb..981da9ce3cfc 100644
> --- a/drivers/nvme/host/rdma.c
> +++ b/drivers/nvme/host/rdma.c
> @@ -614,7 +614,8 @@ static int nvme_rdma_start_queue(struct
> nvme_rdma_ctrl *ctrl, int idx)
> if (!ret) {
> set_bit(NVME_RDMA_Q_LIVE, &queue->flags);
> } else {
> - __nvme_rdma_stop_queue(queue);
> + if (test_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags))
> + __nvme_rdma_stop_queue(queue);
> dev_info(ctrl->ctrl.device,
> "failed to connect queue: %d ret=%d\n", idx,
> ret);
> }
> --
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: NVMeoF RDMA IB: I/O timeout and NULL pointer observed during rescan_controller/reset_controller with fio background
2019-09-23 15:25 ` Max Gurtovoy
@ 2019-09-24 4:52 ` Yi Zhang
2019-09-27 8:17 ` Yi Zhang
1 sibling, 0 replies; 7+ messages in thread
From: Yi Zhang @ 2019-09-24 4:52 UTC (permalink / raw)
To: Max Gurtovoy; +Cc: Sagi Grimberg, linux-nvme
Hi Max/Sagi
Sorry for the late response, will update it today.
Best Regards,
Yi Zhang
----- Original Message -----
From: "Max Gurtovoy" <maxg@mellanox.com>
To: "Sagi Grimberg" <sagi@grimberg.me>, "Yi Zhang" <yi.zhang@redhat.com>, linux-nvme@lists.infradead.org
Sent: Monday, September 23, 2019 11:25:31 PM
Subject: Re: NVMeoF RDMA IB: I/O timeout and NULL pointer observed during rescan_controller/reset_controller with fio background
Any update Yi ?
we must fix this issue..
On 9/20/2019 7:58 PM, Sagi Grimberg wrote:
> Thanks for reporting Yi,
>
> Does this fix your issue?
>
> --
> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
> index dfa07bb9dfeb..981da9ce3cfc 100644
> --- a/drivers/nvme/host/rdma.c
> +++ b/drivers/nvme/host/rdma.c
> @@ -614,7 +614,8 @@ static int nvme_rdma_start_queue(struct
> nvme_rdma_ctrl *ctrl, int idx)
> if (!ret) {
> set_bit(NVME_RDMA_Q_LIVE, &queue->flags);
> } else {
> - __nvme_rdma_stop_queue(queue);
> + if (test_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags))
> + __nvme_rdma_stop_queue(queue);
> dev_info(ctrl->ctrl.device,
> "failed to connect queue: %d ret=%d\n", idx,
> ret);
> }
> --
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: NVMeoF RDMA IB: I/O timeout and NULL pointer observed during rescan_controller/reset_controller with fio background
2019-09-23 15:25 ` Max Gurtovoy
2019-09-24 4:52 ` Yi Zhang
@ 2019-09-27 8:17 ` Yi Zhang
1 sibling, 0 replies; 7+ messages in thread
From: Yi Zhang @ 2019-09-27 8:17 UTC (permalink / raw)
To: Sagi Grimberg; +Cc: Max Gurtovoy, linux-nvme
Hi Sagi
Confirmed the NULL pointer issue was fixed by this patch, and "I/O 1 QID 0 timeout" still exists, thanks.
<6>[ 5807.293577] nvme nvme2: creating 48 I/O queues.
<6>[ 5817.210168] nvme nvme2: Removing ctrl: NQN "testnqn"
<6>[ 5821.130348] nvme nvme2: new ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery", addr 172.31.0.186:4420
<6>[ 5821.139830] nvme nvme2: Removing ctrl: NQN "nqn.2014-08.org.nvmexpress.discovery"
<6>[ 5821.185128] nvme nvme2: creating 48 I/O queues.
<6>[ 5821.925736] nvme nvme2: mapped 48/0/0 default/read/poll queues.
<6>[ 5821.950855] nvme nvme2: new ctrl: NQN "testnqn", addr 172.31.0.186:4420
<6>[ 5821.953152] nvme2n1: detected capacity change from 0 to 1600321314816
<4>[ 5826.546586] nvme_ns_head_make_request: 250644 callbacks suppressed
<4>[ 5826.546589] block nvme2n1: no usable path - requeuing I/O
<4>[ 5826.546591] block nvme2n1: no usable path - requeuing I/O
<4>[ 5826.546606] block nvme2n1: no usable path - requeuing I/O
<4>[ 5826.546609] block nvme2n1: no usable path - requeuing I/O
<4>[ 5826.546624] block nvme2n1: no usable path - requeuing I/O
<4>[ 5826.546627] block nvme2n1: no usable path - requeuing I/O
<4>[ 5826.546629] block nvme2n1: no usable path - requeuing I/O
<4>[ 5826.546633] block nvme2n1: no usable path - requeuing I/O
<4>[ 5826.546635] block nvme2n1: no usable path - requeuing I/O
<4>[ 5826.546636] block nvme2n1: no usable path - requeuing I/O
<4>[ 5837.481054] hfi1_opa0.8024: P_Key 0x8024 is not found
<4>[ 5837.486199] hfi1_opa0.8022: P_Key 0x8022 is not found
<6>[ 5837.503278] IPv6: ADDRCONF(NETDEV_CHANGE): hfi1_opa0: link becomes ready
<4>[ 5882.465388] hfi1_opa0.8024: P_Key 0x8024 is not found
<4>[ 5882.470520] hfi1_opa0.8022: P_Key 0x8022 is not found
<6>[ 5882.487647] IPv6: ADDRCONF(NETDEV_CHANGE): hfi1_opa0: link becomes ready
<4>[ 5888.515395] nvme nvme2: I/O 1 QID 0 timeout
<3>[ 5888.533361] nvme nvme2: Connect command failed, error wo/DNR bit: 7
<6>[ 5888.539645] nvme nvme2: failed to connect queue: 0 ret=7
<6>[ 5888.544994] nvme nvme2: Reconnecting in 10 seconds...
<6>[ 5898.774955] nvme nvme2: creating 48 I/O queues.
<6>[ 5899.570053] nvme nvme2: Successfully reconnected (2 attempts)
<4>[ 5927.818466] hfi1_opa0.8024: P_Key 0x8024 is not found
<4>[ 5927.823550] hfi1_opa0.8022: P_Key 0x8022 is not found
<6>[ 5927.843972] IPv6: ADDRCONF(NETDEV_CHANGE): hfi1_opa0: link becomes ready
<6>[ 6004.970479] nvme nvme2: Removing ctrl: NQN "testnqn"
Best Regards,
Yi Zhang
----- Original Message -----
From: "Max Gurtovoy" <maxg@mellanox.com>
To: "Sagi Grimberg" <sagi@grimberg.me>, "Yi Zhang" <yi.zhang@redhat.com>, linux-nvme@lists.infradead.org
Sent: Monday, September 23, 2019 11:25:31 PM
Subject: Re: NVMeoF RDMA IB: I/O timeout and NULL pointer observed during rescan_controller/reset_controller with fio background
Any update Yi ?
we must fix this issue..
On 9/20/2019 7:58 PM, Sagi Grimberg wrote:
> Thanks for reporting Yi,
>
> Does this fix your issue?
>
> --
> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
> index dfa07bb9dfeb..981da9ce3cfc 100644
> --- a/drivers/nvme/host/rdma.c
> +++ b/drivers/nvme/host/rdma.c
> @@ -614,7 +614,8 @@ static int nvme_rdma_start_queue(struct
> nvme_rdma_ctrl *ctrl, int idx)
> if (!ret) {
> set_bit(NVME_RDMA_Q_LIVE, &queue->flags);
> } else {
> - __nvme_rdma_stop_queue(queue);
> + if (test_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags))
> + __nvme_rdma_stop_queue(queue);
> dev_info(ctrl->ctrl.device,
> "failed to connect queue: %d ret=%d\n", idx,
> ret);
> }
> --
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2019-09-27 8:17 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <1823328454.445263.1568796846850.JavaMail.zimbra@redhat.com>
2019-09-18 9:13 ` NVMeoF RDMA IB: I/O timeout and NULL pointer observed during rescan_controller/reset_controller with fio background Yi Zhang
2019-09-18 14:21 ` Max Gurtovoy
2019-09-20 3:37 ` Yi Zhang
2019-09-20 16:58 ` Sagi Grimberg
2019-09-23 15:25 ` Max Gurtovoy
2019-09-24 4:52 ` Yi Zhang
2019-09-27 8:17 ` Yi Zhang
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).