[bug report]null pointer at scsi_mq_exit_request+0x14 with blktests srp/015

[bug report]null pointer at scsi_mq_exit_request+0x14 with blktests srp/015

[Yi Zhang]

Hello

I found this issue with blktests srp/015, could anyone help check it?

[  250.415156] run blktests srp/015 at 2021-02-28 09:23:02
[  250.525538] eno2 speed is unknown, defaulting to 1000
[  250.530599] eno2 speed is unknown, defaulting to 1000
[  250.535665] eno2 speed is unknown, defaulting to 1000
[  250.541768] eno3 speed is unknown, defaulting to 1000
[  250.546821] eno3 speed is unknown, defaulting to 1000
[  250.551881] eno3 speed is unknown, defaulting to 1000
[  250.557844] eno4 speed is unknown, defaulting to 1000
[  250.562912] eno4 speed is unknown, defaulting to 1000
[  250.567978] eno4 speed is unknown, defaulting to 1000
[  250.573945] lo speed is unknown, defaulting to 1000
[  250.578827] lo speed is unknown, defaulting to 1000
[  250.583712] lo speed is unknown, defaulting to 1000
[  250.605512] sd 15:0:0:0: Power-on or device reset occurred
[  250.673453] device-mapper: table: 253:3: multipath: error getting device
[  250.680160] device-mapper: ioctl: error adding target to table
[  250.732648] eno2 speed is unknown, defaulting to 1000
[  250.737724] eno3 speed is unknown, defaulting to 1000
[  250.742784] eno4 speed is unknown, defaulting to 1000
[  250.747843] lo speed is unknown, defaulting to 1000
[  250.752725] ib_srpt MAD registration failed for lo_siw-1.
[  251.106511] scsi host16:   REJ reason 0xffffff98
[  251.111150] scsi host16: ib_srp: Connection 0/4 to 10.16.221.116 failed
[  251.246254] sd 16:0:0:0: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
[  251.260567] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
[  251.260765] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
[  251.280135] sd 16:0:0:1: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
[  251.298968] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
[  252.505389] ext4 filesystem being mounted at /root/blktests/results/tmpdir.srp.015.63f/mnt0 supports timestamps until 2038 (0x7fffffff)
[  253.522565] device-mapper: table: 253:6: multipath: error getting device
[  253.529277] device-mapper: ioctl: error adding target to table
[  257.526935] device-mapper: multipath: 253:4: Failing path 8:32.
[  262.542076] scsi host16:   REJ reason 0xffffff98
[  262.546708] scsi host16: ib_srp: Connection 0/4 to 10.16.221.116 failed
[  262.671752] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
[  262.672079] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
[  262.692394] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
[  267.804429] device-mapper: multipath: 253:4: Failing path 8:32.
[  267.882487] srpt_recv_done: 502 callbacks suppressed
[  269.819904] scsi host16:   REJ reason 0xffffff98
[  269.824533] scsi host16: ib_srp: Connection 0/4 to 10.16.221.116 failed
[  269.951714] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
[  269.952049] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
[  269.972080] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
[  275.085527] device-mapper: multipath: 253:4: Failing path 8:32.
[  275.152684] srpt_recv_done: 502 callbacks suppressed
[  277.100827] scsi host16:   REJ reason 0xffffff98
[  277.105452] scsi host16: ib_srp: Connection 0/4 to 10.16.221.116 failed
[  277.231922] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
[  277.242758] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
[  277.252505] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
[  282.353149] device-mapper: multipath: 253:4: Failing path 8:32.
[  282.419429] srpt_recv_done: 502 callbacks suppressed
[  282.842310] scsi host17:   REJ reason 0xffffff98
[  282.846939] scsi host17: ib_srp: Connection 0/4 to 10.16.221.116 failed
[  282.981432] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
[  282.991501] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
[  283.001358] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
[  284.690990] device-mapper: multipath: 253:4: Failing path 8:96.
[  285.493848] BUG: unable to handle page fault for address: ffffffffc0a89150
[  285.500724] #PF: supervisor read access in kernel mode
[  285.505863] #PF: error_code(0x0000) - not-present page
[  285.511001] PGD 37bc13067 P4D 37bc13067 PUD 37bc15067 PMD 13839d067 PTE 0
[  285.517789] Oops: 0000 [#1] SMP NOPTI
[  285.521454] CPU: 31 PID: 17943 Comm: multipathd Tainted: G S        I       5.11.0 #4
[  285.529279] Hardware name: Dell Inc. PowerEdge R640/06NR82, BIOS 2.10.0 11/12/2020
[  285.536837] RIP: 0010:scsi_mq_exit_request+0x14/0x50
[  285.541804] Code: 00 00 e9 bf eb 58 00 e9 ea bc e0 ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 53 48 8b 7f 60 48 89 f3 48 8b 87 98 00 00 00 <48> 8b 40 30 48 85 c0 74 0c 48 8d b6 10 01 00 00 e8 87 eb 58 00 f6
[  285.560548] RSP: 0018:ffffa66e82e37c98 EFLAGS: 00010286
[  285.565774] RAX: ffffffffc0a89120 RBX: ffff8a0e8a3e0000 RCX: 0000000000000000
[  285.572906] RDX: 0000000000000000 RSI: ffff8a0e8a3e0000 RDI: ffff8a0e692a1000
[  285.580038] RBP: ffff8a0ea8469800 R08: 000000000000020d R09: 000000000002a780
[  285.587170] R10: 00000376b4157308 R11: 0000000000000000 R12: 0000000000000000
[  285.594304] R13: ffff8a0e692a10a8 R14: 0000000000000000 R15: ffff8a0e9244b700
[  285.601435] FS:  00007f616c291700(0000) GS:ffff8a1d801c0000(0000) knlGS:0000000000000000
[  285.609521] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  285.615269] CR2: ffffffffc0a89150 CR3: 000000011e43e003 CR4: 00000000007706e0
[  285.622401] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  285.629532] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  285.636664] PKRU: 55555554
[  285.639369] Call Trace:
[  285.641824]  blk_mq_free_rqs+0x59/0xd0
[  285.645573]  blk_mq_free_map_and_requests+0x31/0x60
[  285.650454]  blk_mq_free_tag_set+0x22/0x80
[  285.654553]  scsi_host_dev_release+0x86/0xe0
[  285.658826]  device_release+0x33/0x90
[  285.662490]  kobject_release+0x46/0x150
[  285.666332]  device_release+0x33/0x90
[  285.669995]  kobject_release+0x46/0x150
[  285.673836]  execute_in_process_context+0x21/0x60
[  285.678542]  device_release+0x33/0x90
[  285.682208]  kobject_release+0x46/0x150
[  285.686046]  scsi_disk_put+0x2b/0x40 [sd_mod]
[  285.690406]  __blkdev_put+0x186/0x1b0
[  285.694072]  blkdev_put+0x4c/0x110
[  285.697476]  blkdev_close+0x21/0x30
[  285.700970]  __fput+0x92/0x230
[  285.704029]  task_work_run+0x70/0xb0
[  285.707608]  exit_to_user_mode_prepare+0x150/0x160
[  285.712402]  syscall_exit_to_user_mode+0x12/0x40
[  285.717021]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  285.722071] RIP: 0033:0x7f616b1ce977
[  285.725652] Code: 12 b8 03 00 00 00 0f 05 48 3d 00 f0 ff ff 77 3b c3 66 90 53 89 fb 48 83 ec 10 e8 e4 fb ff ff 89 df 89 c2 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2b 89 d7 89 44 24 0c e8 26 fc ff ff 8b 44 24
[  285.744397] RSP: 002b:00007f616c28f7b0 EFLAGS: 00000293 ORIG_RAX: 0000000000000003
[  285.751962] RAX: 0000000000000000 RBX: 000000000000000b RCX: 00007f616b1ce977
[  285.759094] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 000000000000000b
[  285.766227] RBP: 00005588a853efb0 R08: 0000000000000001 R09: 0000000000000007
[  285.773359] R10: 0000000000000000 R11: 0000000000000293 R12: 00005588a8215b19
[  285.780492] R13: 00007f615401d630 R14: 0000000000000001 R15: 00007f615401d630
[  285.787624] Modules linked in: target_core_user uio target_core_pscsi target_core_file ib_srpt target_core_iblock target_core_mod scsi_debug siw null_blk ext4 mbcache jbd2 rpcrdma rdma_ucm ib_umad ib_iser libiscsi scsi_transport_iscsi rdma_cm iw_cm ib_cm ib_uverbs ib_core rfkill sunrpc vfat fat dm_service_time dm_multipath intel_rapl_msr intel_rapl_common isst_if_common skx_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel ipmi_ssif kvm mgag200 i2c_algo_bit drm_kms_helper iTCO_wdt iTCO_vendor_support irqbypass dcdbas syscopyarea crct10dif_pclmul sysfillrect crc32_pclmul sysimgblt fb_sys_fops ghash_clmulni_intel drm acpi_ipmi rapl ipmi_si intel_cstate dell_smbios dax_pmem_compat mei_me i2c_i801 ipmi_devintf device_dax intel_uncore mei wmi_bmof dell_wmi_descriptor pcspkr intel_pch_thermal lpc_ich i2c_smbus ipmi_msghandler dax_pmem_core acpi_power_meter ip_tables xfs libcrc32c nd_pmem nd_btt sd_mod t10_pi sg ahci libahci nfit megaraid_sas libata tg3 crc32c_intel libnvdimm wmi
[  285.787672]  dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_transport_srp]
[  285.882703] CR2: ffffffffc0a89150
[  285.886032] ---[ end trace 718c96f79b0576a6 ]---
[  285.896061] RIP: 0010:scsi_mq_exit_request+0x14/0x50
[  285.901024] Code: 00 00 e9 bf eb 58 00 e9 ea bc e0 ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 53 48 8b 7f 60 48 89 f3 48 8b 87 98 00 00 00 <48> 8b 40 30 48 85 c0 74 0c 48 8d b6 10 01 00 00 e8 87 eb 58 00 f6
[  285.919768] RSP: 0018:ffffa66e82e37c98 EFLAGS: 00010286
[  285.924995] RAX: ffffffffc0a89120 RBX: ffff8a0e8a3e0000 RCX: 0000000000000000
[  285.932128] RDX: 0000000000000000 RSI: ffff8a0e8a3e0000 RDI: ffff8a0e692a1000
[  285.939260] RBP: ffff8a0ea8469800 R08: 000000000000020d R09: 000000000002a780
[  285.946394] R10: 00000376b4157308 R11: 0000000000000000 R12: 0000000000000000
[  285.953524] R13: ffff8a0e692a10a8 R14: 0000000000000000 R15: ffff8a0e9244b700
[  285.960658] FS:  00007f616c291700(0000) GS:ffff8a1d801c0000(0000) knlGS:0000000000000000
[  285.968742] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  285.974489] CR2: ffffffffc0a89150 CR3: 000000011e43e003 CR4: 00000000007706e0
[  285.981623] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  285.988753] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  285.995885] PKRU: 55555554
[  285.998590] Kernel panic - not syncing: Fatal exception
[  286.504449] Kernel Offset: 0x2c200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[  286.520574] ---[ end Kernel panic - not syncing: Fatal exception ]---

(gdb) l *(scsi_mq_exit_request+0x14)
0xffffffff81673784 is in scsi_mq_exit_request (drivers/scsi/scsi_lib.c:1785).
1780					 unsigned int hctx_idx)
1781	{
1782		struct Scsi_Host *shost = set->driver_data;
1783		struct scsi_cmnd *cmd = blk_mq_rq_to_pdu(rq);
1784	
1785		if (shost->hostt->exit_cmd_priv)
1786			shost->hostt->exit_cmd_priv(shost, cmd);
1787		scsi_free_sense_buffer(cmd->flags & SCMD_UNCHECKED_ISA_DMA,
1788				       cmd->sense_buffer);
1789	}
(gdb) 

Best Regards,
  Yi Zhang

Re: [bug report]null pointer at scsi_mq_exit_request+0x14 with blktests srp/015

[Chaitanya Kulkarni]

On 2/28/21 01:52, Yi Zhang wrote:
> Hello
>
> I found this issue with blktests srp/015, could anyone help check it?
Until you get some reply you can try and bisect it.

Re: [bug report]null pointer at scsi_mq_exit_request+0x14 with blktests srp/015

[Yi Zhang]

Sure, will do.

On 3/1/21 3:07 AM, Chaitanya Kulkarni wrote:
> On 2/28/21 01:52, Yi Zhang wrote:
>> Hello
>>
>> I found this issue with blktests srp/015, could anyone help check it?
> Until you get some reply you can try and bisect it.
>
>

Re: [bug report]null pointer at scsi_mq_exit_request+0x14 with blktests srp/015

[Yi Zhang]

This issue cannot be reproduced on latest 5.12.0-rc1.

Please ignore this report, sorry for the noise.

On 3/1/21 3:07 AM, Chaitanya Kulkarni wrote:
> On 2/28/21 01:52, Yi Zhang wrote:
>> Hello
>>
>> I found this issue with blktests srp/015, could anyone help check it?
> Until you get some reply you can try and bisect it.
>
>

Re: [bug report]null pointer at scsi_mq_exit_request+0x14 with blktests srp/015

[Bart Van Assche]

On 3/1/21 7:36 AM, Yi Zhang wrote:
> This issue cannot be reproduced on latest 5.12.0-rc1.
> 
> Please ignore this report, sorry for the noise.

How about rerunning the same test against v5.11.2, the latest v5.11
stable kernel? I think your report means that v5.11 can be improved...

Thanks,

Bart.

Re: [bug report]null pointer at scsi_mq_exit_request+0x14 with blktests srp/015

[Yi Zhang]

On 3/2/21 11:59 AM, Bart Van Assche wrote:
> On 3/1/21 7:36 AM, Yi Zhang wrote:
>> This issue cannot be reproduced on latest 5.12.0-rc1.
>>
>> Please ignore this report, sorry for the noise.
> How about rerunning the same test against v5.11.2, the latest v5.11
> stable kernel? I think your report means that v5.11 can be improved...
I tried on stable kernel 5.11.2/5.10.19, and all reproduced, here is the log
5.11.2  https://pastebin.com/hG4iBETG
5.10.19  https://pastebin.com/ExFbQMdg

It' not 100% reproduced and sometimes can be reproduced within 20 times' 
testing.
I tried bisect, but unfortunately my attempts don't land on anything that
looks like the real culprit. :(

> Thanks,
>
> Bart.
>

Re: [bug report]null pointer at scsi_mq_exit_request+0x14 with blktests srp/015

[Bart Van Assche]

On 2/28/21 1:49 AM, Yi Zhang wrote:
> I found this issue with blktests srp/015, could anyone help check it?

Which kernel tree has been used in your tests? One of Linus' trees or a
for-next tree from a kernel maintainer?

Bart.

Re: [bug report]null pointer at scsi_mq_exit_request+0x14 with blktests srp/015

[Yi Zhang]

On 3/1/21 6:04 AM, Bart Van Assche wrote:
> On 2/28/21 1:49 AM, Yi Zhang wrote:
>> I found this issue with blktests srp/015, could anyone help check it?
> Which kernel tree has been used in your tests? One of Linus' trees or a
> for-next tree from a kernel maintainer?
I'm using "tag: v5.11" on Linus' tree.

Re: [bug report]null pointer at scsi_mq_exit_request+0x14 with blktests srp/015

[Ming Lei]

On Sun, Feb 28, 2021 at 04:49:32AM -0500, Yi Zhang wrote:
> Hello
> 
> I found this issue with blktests srp/015, could anyone help check it?
> 
> [  250.415156] run blktests srp/015 at 2021-02-28 09:23:02
> [  250.525538] eno2 speed is unknown, defaulting to 1000
> [  250.530599] eno2 speed is unknown, defaulting to 1000
> [  250.535665] eno2 speed is unknown, defaulting to 1000
> [  250.541768] eno3 speed is unknown, defaulting to 1000
> [  250.546821] eno3 speed is unknown, defaulting to 1000
> [  250.551881] eno3 speed is unknown, defaulting to 1000
> [  250.557844] eno4 speed is unknown, defaulting to 1000
> [  250.562912] eno4 speed is unknown, defaulting to 1000
> [  250.567978] eno4 speed is unknown, defaulting to 1000
> [  250.573945] lo speed is unknown, defaulting to 1000
> [  250.578827] lo speed is unknown, defaulting to 1000
> [  250.583712] lo speed is unknown, defaulting to 1000
> [  250.605512] sd 15:0:0:0: Power-on or device reset occurred
> [  250.673453] device-mapper: table: 253:3: multipath: error getting device
> [  250.680160] device-mapper: ioctl: error adding target to table
> [  250.732648] eno2 speed is unknown, defaulting to 1000
> [  250.737724] eno3 speed is unknown, defaulting to 1000
> [  250.742784] eno4 speed is unknown, defaulting to 1000
> [  250.747843] lo speed is unknown, defaulting to 1000
> [  250.752725] ib_srpt MAD registration failed for lo_siw-1.
> [  251.106511] scsi host16:   REJ reason 0xffffff98
> [  251.111150] scsi host16: ib_srp: Connection 0/4 to 10.16.221.116 failed
> [  251.246254] sd 16:0:0:0: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
> [  251.260567] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
> [  251.260765] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
> [  251.280135] sd 16:0:0:1: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
> [  251.298968] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
> [  252.505389] ext4 filesystem being mounted at /root/blktests/results/tmpdir.srp.015.63f/mnt0 supports timestamps until 2038 (0x7fffffff)
> [  253.522565] device-mapper: table: 253:6: multipath: error getting device
> [  253.529277] device-mapper: ioctl: error adding target to table
> [  257.526935] device-mapper: multipath: 253:4: Failing path 8:32.
> [  262.542076] scsi host16:   REJ reason 0xffffff98
> [  262.546708] scsi host16: ib_srp: Connection 0/4 to 10.16.221.116 failed
> [  262.671752] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
> [  262.672079] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
> [  262.692394] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
> [  267.804429] device-mapper: multipath: 253:4: Failing path 8:32.
> [  267.882487] srpt_recv_done: 502 callbacks suppressed
> [  269.819904] scsi host16:   REJ reason 0xffffff98
> [  269.824533] scsi host16: ib_srp: Connection 0/4 to 10.16.221.116 failed
> [  269.951714] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
> [  269.952049] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
> [  269.972080] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
> [  275.085527] device-mapper: multipath: 253:4: Failing path 8:32.
> [  275.152684] srpt_recv_done: 502 callbacks suppressed
> [  277.100827] scsi host16:   REJ reason 0xffffff98
> [  277.105452] scsi host16: ib_srp: Connection 0/4 to 10.16.221.116 failed
> [  277.231922] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
> [  277.242758] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
> [  277.252505] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
> [  282.353149] device-mapper: multipath: 253:4: Failing path 8:32.
> [  282.419429] srpt_recv_done: 502 callbacks suppressed
> [  282.842310] scsi host17:   REJ reason 0xffffff98
> [  282.846939] scsi host17: ib_srp: Connection 0/4 to 10.16.221.116 failed
> [  282.981432] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
> [  282.991501] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
> [  283.001358] srpt/0xf0d4e2e6e1e000000000000000000000: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
> [  284.690990] device-mapper: multipath: 253:4: Failing path 8:96.
> [  285.493848] BUG: unable to handle page fault for address: ffffffffc0a89150
> [  285.500724] #PF: supervisor read access in kernel mode
> [  285.505863] #PF: error_code(0x0000) - not-present page
> [  285.511001] PGD 37bc13067 P4D 37bc13067 PUD 37bc15067 PMD 13839d067 PTE 0
> [  285.517789] Oops: 0000 [#1] SMP NOPTI
> [  285.521454] CPU: 31 PID: 17943 Comm: multipathd Tainted: G S        I       5.11.0 #4
> [  285.529279] Hardware name: Dell Inc. PowerEdge R640/06NR82, BIOS 2.10.0 11/12/2020
> [  285.536837] RIP: 0010:scsi_mq_exit_request+0x14/0x50
> [  285.541804] Code: 00 00 e9 bf eb 58 00 e9 ea bc e0 ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 53 48 8b 7f 60 48 89 f3 48 8b 87 98 00 00 00 <48> 8b 40 30 48 85 c0 74 0c 48 8d b6 10 01 00 00 e8 87 eb 58 00 f6
> [  285.560548] RSP: 0018:ffffa66e82e37c98 EFLAGS: 00010286
> [  285.565774] RAX: ffffffffc0a89120 RBX: ffff8a0e8a3e0000 RCX: 0000000000000000
> [  285.572906] RDX: 0000000000000000 RSI: ffff8a0e8a3e0000 RDI: ffff8a0e692a1000
> [  285.580038] RBP: ffff8a0ea8469800 R08: 000000000000020d R09: 000000000002a780
> [  285.587170] R10: 00000376b4157308 R11: 0000000000000000 R12: 0000000000000000
> [  285.594304] R13: ffff8a0e692a10a8 R14: 0000000000000000 R15: ffff8a0e9244b700
> [  285.601435] FS:  00007f616c291700(0000) GS:ffff8a1d801c0000(0000) knlGS:0000000000000000
> [  285.609521] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  285.615269] CR2: ffffffffc0a89150 CR3: 000000011e43e003 CR4: 00000000007706e0
> [  285.622401] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [  285.629532] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [  285.636664] PKRU: 55555554
> [  285.639369] Call Trace:
> [  285.641824]  blk_mq_free_rqs+0x59/0xd0
> [  285.645573]  blk_mq_free_map_and_requests+0x31/0x60
> [  285.650454]  blk_mq_free_tag_set+0x22/0x80
> [  285.654553]  scsi_host_dev_release+0x86/0xe0
> [  285.658826]  device_release+0x33/0x90
> [  285.662490]  kobject_release+0x46/0x150
> [  285.666332]  device_release+0x33/0x90
> [  285.669995]  kobject_release+0x46/0x150
> [  285.673836]  execute_in_process_context+0x21/0x60
> [  285.678542]  device_release+0x33/0x90
> [  285.682208]  kobject_release+0x46/0x150
> [  285.686046]  scsi_disk_put+0x2b/0x40 [sd_mod]
> [  285.690406]  __blkdev_put+0x186/0x1b0
> [  285.694072]  blkdev_put+0x4c/0x110
> [  285.697476]  blkdev_close+0x21/0x30
> [  285.700970]  __fput+0x92/0x230
> [  285.704029]  task_work_run+0x70/0xb0
> [  285.707608]  exit_to_user_mode_prepare+0x150/0x160
> [  285.712402]  syscall_exit_to_user_mode+0x12/0x40
> [  285.717021]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [  285.722071] RIP: 0033:0x7f616b1ce977
> [  285.725652] Code: 12 b8 03 00 00 00 0f 05 48 3d 00 f0 ff ff 77 3b c3 66 90 53 89 fb 48 83 ec 10 e8 e4 fb ff ff 89 df 89 c2 b8 03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2b 89 d7 89 44 24 0c e8 26 fc ff ff 8b 44 24
> [  285.744397] RSP: 002b:00007f616c28f7b0 EFLAGS: 00000293 ORIG_RAX: 0000000000000003
> [  285.751962] RAX: 0000000000000000 RBX: 000000000000000b RCX: 00007f616b1ce977
> [  285.759094] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 000000000000000b
> [  285.766227] RBP: 00005588a853efb0 R08: 0000000000000001 R09: 0000000000000007
> [  285.773359] R10: 0000000000000000 R11: 0000000000000293 R12: 00005588a8215b19
> [  285.780492] R13: 00007f615401d630 R14: 0000000000000001 R15: 00007f615401d630
> [  285.787624] Modules linked in: target_core_user uio target_core_pscsi target_core_file ib_srpt target_core_iblock target_core_mod scsi_debug siw null_blk ext4 mbcache jbd2 rpcrdma rdma_ucm ib_umad ib_iser libiscsi scsi_transport_iscsi rdma_cm iw_cm ib_cm ib_uverbs ib_core rfkill sunrpc vfat fat dm_service_time dm_multipath intel_rapl_msr intel_rapl_common isst_if_common skx_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel ipmi_ssif kvm mgag200 i2c_algo_bit drm_kms_helper iTCO_wdt iTCO_vendor_support irqbypass dcdbas syscopyarea crct10dif_pclmul sysfillrect crc32_pclmul sysimgblt fb_sys_fops ghash_clmulni_intel drm acpi_ipmi rapl ipmi_si intel_cstate dell_smbios dax_pmem_compat mei_me i2c_i801 ipmi_devintf device_dax intel_uncore mei wmi_bmof dell_wmi_descriptor pcspkr intel_pch_thermal lpc_ich i2c_smbus ipmi_msghandler dax_pmem_core acpi_power_meter ip_tables xfs libcrc32c nd_pmem nd_btt sd_mod t10_pi sg ahci libahci nfit megaraid_sas libata tg3 crc32c_intel libnvdimm wmi
> [  285.787672]  dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_transport_srp]
> [  285.882703] CR2: ffffffffc0a89150
> [  285.886032] ---[ end trace 718c96f79b0576a6 ]---
> [  285.896061] RIP: 0010:scsi_mq_exit_request+0x14/0x50
> [  285.901024] Code: 00 00 e9 bf eb 58 00 e9 ea bc e0 ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 53 48 8b 7f 60 48 89 f3 48 8b 87 98 00 00 00 <48> 8b 40 30 48 85 c0 74 0c 48 8d b6 10 01 00 00 e8 87 eb 58 00 f6
> [  285.919768] RSP: 0018:ffffa66e82e37c98 EFLAGS: 00010286
> [  285.924995] RAX: ffffffffc0a89120 RBX: ffff8a0e8a3e0000 RCX: 0000000000000000
> [  285.932128] RDX: 0000000000000000 RSI: ffff8a0e8a3e0000 RDI: ffff8a0e692a1000
> [  285.939260] RBP: ffff8a0ea8469800 R08: 000000000000020d R09: 000000000002a780
> [  285.946394] R10: 00000376b4157308 R11: 0000000000000000 R12: 0000000000000000
> [  285.953524] R13: ffff8a0e692a10a8 R14: 0000000000000000 R15: ffff8a0e9244b700
> [  285.960658] FS:  00007f616c291700(0000) GS:ffff8a1d801c0000(0000) knlGS:0000000000000000
> [  285.968742] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  285.974489] CR2: ffffffffc0a89150 CR3: 000000011e43e003 CR4: 00000000007706e0
> [  285.981623] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [  285.988753] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [  285.995885] PKRU: 55555554
> [  285.998590] Kernel panic - not syncing: Fatal exception
> [  286.504449] Kernel Offset: 0x2c200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> [  286.520574] ---[ end Kernel panic - not syncing: Fatal exception ]---
> 
> (gdb) l *(scsi_mq_exit_request+0x14)
> 0xffffffff81673784 is in scsi_mq_exit_request (drivers/scsi/scsi_lib.c:1785).
> 1780					 unsigned int hctx_idx)
> 1781	{
> 1782		struct Scsi_Host *shost = set->driver_data;
> 1783		struct scsi_cmnd *cmd = blk_mq_rq_to_pdu(rq);
> 1784	
> 1785		if (shost->hostt->exit_cmd_priv)
> 1786			shost->hostt->exit_cmd_priv(shost, cmd);

Looks it is weird since not see any drivers implement .exit_cmd_priv in
v5.11.

-- 
Ming