* DIF/DIX issue related to config CONFIG_SCSI_MQ_DEFAULT
@ 2018-11-27 9:55 chenxiang (M)
2018-11-27 13:08 ` Ming Lei
2018-11-27 20:22 ` Ewan D. Milne
0 siblings, 2 replies; 16+ messages in thread
From: chenxiang (M) @ 2018-11-27 9:55 UTC (permalink / raw)
To: James E.J. Bottomley, Martin K. Petersen, Ming Lei
Cc: linux-scsi, linux-block, John Garry, Linuxarm
[-- Attachment #1: Type: text/plain, Size: 13122 bytes --]
Hi all,
There is a issue which may be related to CONFIG_SCSI_MQ_DEFAULT: before
we developed DIF/DIX feature on kernel 4.18 (disable
CONFIG_SCSI_MQ_DEFAULT default), and
it works well. But when we switch to kernel 4.19-rc1 and 4.20-rc1, Call
trace as follow occurs when running fio and if disable config
CONFIG_SCSI_MQ_DEFAULT,
then it works well. Also if switch ioengine=libaio to ioengine=psync, it
seems also work well. Do you have any idea or encounter similar issue?
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
fio 2.0.5
Starting 12 processes
[ 629.210506] Unable to handle kernel paging request at virtual address
0000ffff8027e048
[ 629.210506] Unable to handle kernel paging request at virtual address
0000ffff8027e048
[ 629.226373] Mem abort info:
[ 629.226373] Mem abort info:
[ 629.231952] ESR = 0x96000006
[ 629.231952] ESR = 0x96000006
[ 629.238052] Exception class = DABT (current EL), IL = 32 bits
[ 629.238052] Exception class = DABT (current EL), IL = 32 bits
[ 629.249898] SET = 0, FnV = 0
[ 629.249898] SET = 0, FnV = 0
[ 629.255998] EA = 0, S1PTW = 0
[ 629.255998] EA = 0, S1PTW = 0
[ 629.262272] Data abort info:
[ 629.262272] Data abort info:
[ 629.268023] ISV = 0, ISS = 0x00000006
[ 629.268023] ISV = 0, ISS = 0x00000006
[ 629.275690] CM = 0, WnR = 0
[ 629.275690] CM = 0, WnR = 0
[ 629.281617] user pgtable: 4k pages, 48-bit VAs, pgdp = 0000000085c91728
[ 629.281617] user pgtable: 4k pages, 48-bit VAs, pgdp = 0000000085c91728
[ 629.294857] [0000ffff8027e048] pgd=00000027a8644003,
pud=00000027a85ea003, pmd=0000000000000000
[ 629.294857] [0000ffff8027e048] pgd=00000027a8644003,
pud=00000027a85ea003, pmd=0000000000000000
[ 629.312278] Internal error: Oops: 96000006 [#1] PREEMPT SMP
[ 629.312278] Internal error: Oops: 96000006 [#1] PREEMPT SMP
[ 629.323427] Modules linked in: hisi_sas_v3_hw [last unloaded:
hisi_sas_v3_hw]
[ 629.323427] Modules linked in: hisi_sas_v3_hw [last unloaded:
hisi_sas_v3_hw]
[ 629.337713] CPU: 13 PID: 4465 Comm: fio Not tainted
4.20.0-rc1-15093-ge876dec #1067
[ 629.337713] CPU: 13 PID: 4465 Comm: fio Not tainted
4.20.0-rc1-15093-ge876dec #1067
[ 629.353040] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI
RC0 - B601 (V6.01) 11/08/2018
[ 629.353040] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI
RC0 - B601 (V6.01) 11/08/2018
[ 629.370633] pstate: 80400009 (Nzcv daif +PAN -UAO)
[ 629.370633] pstate: 80400009 (Nzcv daif +PAN -UAO)
[ 629.380218] pc : deadline_remove_request+0x2c/0xd0
[ 629.380218] pc : deadline_remove_request+0x2c/0xd0
[ 629.389799] lr : dd_dispatch_request+0x184/0x1e0
[ 629.389799] lr : dd_dispatch_request+0x184/0x1e0
[ 629.399031] sp : ffff0000275a3aa0
[ 629.399031] sp : ffff0000275a3aa0
[ 629.405651] x29: ffff0000275a3aa0 x28: ffff8027dc230d40
[ 629.405651] x29: ffff0000275a3aa0 x28: ffff8027dc230d40
[ 629.416278] x27: 0000000000000001 x26: ffff8027dc230d40
[ 629.416278] x27: 0000000000000001 x26: ffff8027dc230d40
[ 629.426903] x25: 0000000000000000 x24: ffff0000094f1000
[ 629.426903] x25: 0000000000000000 x24: ffff0000094f1000
[ 629.437529] x23: ffff8027a1e1d140 x22: 0000000000000006
[ 629.437529] x23: ffff8027a1e1d140 x22: 0000000000000006
[ 629.448155] x21: ffff7e009ec69842 x20: ffff8027a1e1d100
[ 629.448155] x21: ffff7e009ec69842 x20: ffff8027a1e1d100
[ 629.458781] x19: ffff8027a1e1d100 x18: 0000000000000000
[ 629.458781] x19: ffff8027a1e1d100 x18: 0000000000000000
[ 629.469406] x17: 000000009e8c9c80 x16: 00000027a3272000
[ 629.469406] x17: 000000009e8c9c80 x16: 00000027a3272000
[ 629.480032] x15: ffff7e009fff9534 x14: ffff7e009ec698c0
[ 629.480032] x15: ffff7e009fff9534 x14: ffff7e009ec698c0
[ 629.490658] x13: ffff7e009ec69880 x12: 0000000000000009
[ 629.490658] x13: ffff7e009ec69880 x12: 0000000000000009
[ 629.501284] x11: 0000000000000000 x10: ffff8027a7c03560
[ 629.501284] x11: 0000000000000000 x10: ffff8027a7c03560
[ 629.511910] x9 : ffff8027a7c007b8 x8 : 0000000000000007
[ 629.511910] x9 : ffff8027a7c007b8 x8 : 0000000000000007
[ 629.522535] x7 : ffff8027a7c011c8 x6 : 0000000000000000
[ 629.522535] x7 : ffff8027a7c011c8 x6 : 0000000000000000
[ 629.533161] x5 : 0000000000000000 x4 : 0000ffff8027e040
[ 629.533161] x5 : 0000000000000000 x4 : 0000ffff8027e040
[ 629.543787] x3 : ffff8027a1e20040 x2 : ffff8027da87f6a0
[ 629.543787] x3 : ffff8027a1e20040 x2 : ffff8027da87f6a0
[ 629.554413] x1 : ffff8027a1e1d140 x0 : ffff8027a1e1d160
[ 629.554413] x1 : ffff8027a1e1d140 x0 : ffff8027a1e1d160
[ 629.565039] Process fio (pid: 4465, stack limit = 0x000000004e5e4c33)
[ 629.565039] Process fio (pid: 4465, stack limit = 0x000000004e5e4c33)
[ 629.577929] Call trace:
[ 629.577929] Call trace:
[ 629.582808] deadline_remove_request+0x2c/0xd0
[ 629.582808] deadline_remove_request+0x2c/0xd0
[ 629.591693] dd_dispatch_request+0x184/0x1e0
[ 629.591693] dd_dispatch_request+0x184/0x1e0
[ 629.600229] blk_mq_do_dispatch_sched+0x4c/0x104
[ 629.600229] blk_mq_do_dispatch_sched+0x4c/0x104
[ 629.609461] blk_mq_sched_dispatch_requests+0x114/0x18c
[ 629.609461] blk_mq_sched_dispatch_requests+0x114/0x18c
[ 629.619913] __blk_mq_run_hw_queue+0xa4/0x11c
[ 629.619913] __blk_mq_run_hw_queue+0xa4/0x11c
[ 629.628624] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 629.628624] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 629.638553] blk_mq_run_hw_queue+0x9c/0xf8
[ 629.638553] blk_mq_run_hw_queue+0x9c/0xf8
[ 629.646741] blk_mq_sched_insert_requests+0x60/0xac
[ 629.646741] blk_mq_sched_insert_requests+0x60/0xac
[ 629.656495] blk_mq_flush_plug_list+0x144/0x16c
[ 629.656495] blk_mq_flush_plug_list+0x144/0x16c
[ 629.665554] blk_flush_plug_list+0xe8/0x244
[ 629.665554] blk_flush_plug_list+0xe8/0x244
[ 629.673915] blk_finish_plug+0x30/0x40
[ 629.673915] blk_finish_plug+0x30/0x40
[ 629.681408] __arm64_sys_io_submit+0x114/0x218
[ 629.681408] __arm64_sys_io_submit+0x114/0x218
[ 629.690294] el0_svc_common+0x94/0xf0
[ 629.690294] el0_svc_common+0x94/0xf0
[ 629.697611] el0_svc_handler+0x24/0x80
[ 629.697611] el0_svc_handler+0x24/0x80
[ 629.705102] el0_svc+0x8/0x7c0
[ 629.705102] el0_svc+0x8/0x7c0
[ 629.711200] Code: 91010021 f9400ea4 f9402263 f9402662 (f9400496)
[ 629.711200] Code: 91010021 f9400ea4 f9402263 f9402662 (f9400496)
[ 629.723394] ---[ end trace 22423ac28b61ac51 ]---
[ 629.723394] ---[ end trace 22423ac28b61ac51 ]---
[ 629.732631] note: fio[4465] exited with preempt_count 2
[ 629.732631] note: fio[4465] exited with preempt_count 2
[ 629.743112] WARNING: CPU: 9 PID: 4465 at kernel/rcu/tree_plugin.h:337
rcu_note_context_switch+0x38/0x2ec
[ 629.743112] WARNING: CPU: 9 PID: 4465 at kernel/rcu/tree_plugin.h:337
rcu_note_context_switch+0x38/0x2ec
[ 629.762100] Modules linked in: hisi_sas_v3_hw [last unloaded:
hisi_sas_v3_hw]
[ 629.762100] Modules linked in: hisi_sas_v3_hw [last unloaded:
hisi_sas_v3_hw]
[ 629.776387] CPU: 9 PID: 4465 Comm: fio Tainted: G D
4.20.0-rc1-15093-ge876dec #1067
[ 629.776387] CPU: 9 PID: 4465 Comm: fio Tainted: G D
4.20.0-rc1-15093-ge876dec #1067
[ 629.794327] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI
RC0 - B601 (V6.01) 11/08/2018
[ 629.794327] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI
RC0 - B601 (V6.01) 11/08/2018
[ 629.811919] pstate: 20400089 (nzCv daIf +PAN -UAO)
[ 629.811919] pstate: 20400089 (nzCv daIf +PAN -UAO)
[ 629.821500] pc : rcu_note_context_switch+0x38/0x2ec
[ 629.821500] pc : rcu_note_context_switch+0x38/0x2ec
[ 629.831258] lr : __schedule+0x88/0x5f8
[ 629.831258] lr : __schedule+0x88/0x5f8
Jo[bs: 12 (f=0)
629.838748] sp : ffff0000275a34d0
[ 629.838748] sp : ffff0000275a34d0
[ 629.846670] x29: ffff0000275a34d0 x28: ffff8027dc230d40
[ 629.846670] x29: ffff0000275a34d0 x28: ffff8027dc230d40
[ 629.857296] x27: 0000000000000000 x26: ffff000008da18e0
[ 629.857296] x27: 0000000000000000 x26: ffff000008da18e0
[ 629.867922] x25: ffff0000094f1000 x24: ffff8027dc230d40
[ 629.867922] x25: ffff0000094f1000 x24: ffff8027dc230d40
[ 629.878548] x23: ffff0000094ca000 x22: ffff0000094d6000
[ 629.878548] x23: ffff0000094ca000 x22: ffff0000094d6000
[ 629.889174] x21: ffff8027e0ab3b00 x20: ffff8027dc230d40
[ 629.889174] x21: ffff8027e0ab3b00 x20: ffff8027dc230d40
[ 629.899800] x19: ffff8027e0ab4700 x18: 0000000000000000
[ 629.899800] x19: ffff8027e0ab4700 x18: 0000000000000000
[ 629.910425] x17: 0000000000000000 x16: 0000000000000000
[ 629.910425] x17: 0000000000000000 x16: 0000000000000000
[ 629.921050] x15: 0000000000000000 x14: 0000000000000000
[ 629.921050] x15: 0000000000000000 x14: 0000000000000000
[ 629.931676] x13: 0000000000000000 x12: 0000000000000000
[ 629.931676] x13: 0000000000000000 x12: 0000000000000000
[ 629.942301] x11: 0000000000000002 x10: 0000000000000970
[ 629.942301] x11: 0000000000000002 x10: 0000000000000970
[ 629.952927] x9 : ffff0000275a3660 x8 : ffff8027dc231710
[ 629.952927] x9 : ffff0000275a3660 x8 : ffff8027dc231710
[ 629.963553] x7 : 000000927fe91094 x6 : ffff8027a0308428
[ 629.963553] x7 : 000000927fe91094 x6 : ffff8027a0308428
[ 629.974178] x5 : 0000000000000000 x4 : 0000000000000001
[ 629.974178] x5 : 0000000000000000 x4 : 0000000000000001
[ 629.984804] x3 : 674238a07d45b500 x2 : 00008027d75dd000
[ 629.984804] x3 : 674238a07d45b500 x2 : 00008027d75dd000
[ 629.995429] x1 : 00008027d75dd000 x0 : 0000000000000001
[ 629.995429] x1 : 00008027d75dd000 x0 : 0000000000000001
[ 630.006055] Call trace:
[ 630.006055] Call trace:
[ 630.010934] rcu_note_context_switch+0x38/0x2ec
[ 630.010934] rcu_note_context_switch+0x38/0x2ec
[ 630.019993] __schedule+0x88/0x5f8
[ 630.019993] __schedule+0x88/0x5f8
[ 630.026788] schedule+0x38/0x9c
[ 630.026788] schedule+0x38/0x9c
[ 630.033060] schedule_timeout+0x1b8/0x320
[ 630.033060] schedule_timeout+0x1b8/0x320
[ 630.041074] wait_for_common+0xb4/0x164
[ 630.041074] wait_for_common+0xb4/0x164
[ 630.048739] wait_for_completion+0x14/0x1c
[ 630.048739] wait_for_completion+0x14/0x1c
[ 630.056928] exit_aio+0x100/0x114
[ 630.056928] exit_aio+0x100/0x114
[ 630.063549] mmput+0x48/0x118
[ 630.063549] mmput+0x48/0x118
[ 630.069473] do_exit+0x280/0x984
[ 630.069473] do_exit+0x280/0x984
[ 630.075920] die+0x1e0/0x20c
[ 630.075920] die+0x1e0/0x20c
[ 630.081670] die_kernel_fault+0x64/0x74
[ 630.081670] die_kernel_fault+0x64/0x74
[ 6Jobs: 12 (f=0)
30.089335] __do_kernel_fault+0x84/0xac
[ 630.089335] __do_kernel_fault+0x84/0xac
[ 630.098477] do_page_fault+0x114/0x3a8
[ 630.098477] do_page_fault+0x114/0x3a8
[ 630.105968] do_translation_fault+0x68/0x74
[ 630.105968] do_translation_fault+0x68/0x74
[ 630.114330] do_mem_abort+0x3c/0x9c
[ 630.114330] do_mem_abort+0x3c/0x9c
[ 630.121298] el1_da+0x20/0x80
[ 630.121298] el1_da+0x20/0x80
[ 630.127223] deadline_remove_request+0x2c/0xd0
[ 630.127223] deadline_remove_request+0x2c/0xd0
[ 630.136108] dd_dispatch_request+0x184/0x1e0
[ 630.136108] dd_dispatch_request+0x184/0x1e0
[ 630.144644] blk_mq_do_dispatch_sched+0x4c/0x104
[ 630.144644] blk_mq_do_dispatch_sched+0x4c/0x104
[ 630.153877] blk_mq_sched_dispatch_requests+0x114/0x18c
[ 630.153877] blk_mq_sched_dispatch_requests+0x114/0x18c
[ 630.164329] __blk_mq_run_hw_queue+0xa4/0x11c
[ 630.164329] __blk_mq_run_hw_queue+0xa4/0x11c
[ 630.173040] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 630.173040] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 630.182969] blk_mq_run_hw_queue+0x9c/0xf8
[ 630.182969] blk_mq_run_hw_queue+0x9c/0xf8
[ 630.191157] blk_mq_sched_insert_requests+0x60/0xac
[ 630.191157] blk_mq_sched_insert_requests+0x60/0xac
[ 630.200911] blk_mq_flush_plug_list+0x144/0x16c
[ 630.200911] blk_mq_flush_plug_list+0x144/0x16c
[ 630.209970] blk_flush_plug_list+0xe8/0x244
[ 630.209970] blk_flush_plug_list+0xe8/0x244
[ 630.218331] blk_finish_plug+0x30/0x40
[ 630.218331] blk_finish_plug+0x30/0x40
[ 630.225823] __arm64_sys_io_submit+0x114/0x218
[ 630.225823] __arm64_sys_io_submit+0x114/0x218
[ 630.234708] el0_svc_common+0x94/0xf0
[ 630.234708] el0_svc_common+0x94/0xf0
[ 630.242025] el0_svc_handler+0x24/0x80
[ 630.242025] el0_svc_handler+0x24/0x80
[ 630.249516] el0_svc+0x8/0x7c0
[ 630.249516] el0_svc+0x8/0x7c0
[ 630.255613] ---[ end trace 22423ac28b61ac52 ]---
[ 630.255613] ---[ end trace 22423ac28b61ac52 ]---
[-- Attachment #2: DIF_enable_MQ.txt --]
[-- Type: text/plain, Size: 56623 bytes --]
Creat 4k_rw_depth128_fiotest file sucessfully
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
fio 2.0.5
Starting 12 processes
Jobs: 12 (f=0)\rJobs: 12 (f=0)\rJobs: 12 (f=0)\rJobs: 12 (f=0)\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:08s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:08s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:07s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:07s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:07s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:07s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:06s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:06s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:06s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:06s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:05s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:05s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:05s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:05s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:04s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:04s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:04s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:04s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:03s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:03s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:03s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:03s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:02s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:02s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:02s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:02s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:01s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:01s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:01s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:01s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:00s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:00s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:00s]\rJobs: 12 (f=0): [////////////] [-.-% done] [0K/0K /s] [0 /0 iops] [eta 03m:00s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [5.8% done] [14099K/11669K /s] [3442 /2848 iops] [eta 02m:59s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [6.3% done] [902.5M/828.5M /s] [220K/202K iops] [eta 02m:58s] \rJobs: 12 (f=12): [MMMMMMMMMMMM] [6.8% done] [762.6M/982.5M /s] [186K/240K iops] [eta 02m:57s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [7.4% done] [843.9M/722.3M /s] [206K/176K iops] [eta 02m:56s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [7.9% done] [632.4M/631.9M /s] [154K/154K iops] [eta 02m:55s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [8.4% done] [621.2M/620.4M /s] [152K/151K iops] [eta 02m:54s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [8.9% done] [638.9M/636.9M /s] [156K/155K iops] [eta 02m:53s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [9.5% done] [615.1M/615.7M /s] [150K/150K iops] [eta 02m:52s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [10.0% done] [609.8M/606.8M /s] [149K/148K iops] [eta 02m:51s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [10.5% done] [631.8M/636.3M /s] [154K/155K iops] [eta 02m:50s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [11.1% done] [637.5M/633.3M /s] [156K/155K iops] [eta 02m:49s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [11.6% done] [636.7M/635.5M /s] [155K/155K iops] [eta 02m:48s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [12.1% done] [634.1M/636.3M /s] [155K/155K iops] [eta 02m:47s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [12.6% done] [633.2M/636.1M /s] [155K/155K iops] [eta 02m:46s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [13.2% done] [636.2M/633.3M /s] [155K/155K iops] [eta 02m:45s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [13.7% done] [615.3M/614.2M /s] [150K/150K iops] [eta 02m:44s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [14.2% done] [613.8M/611.8M /s] [150K/149K iops] [eta 02m:43s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [14.7% done] [635.8M/634.2M /s] [155K/155K iops] [eta 02m:42s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [15.3% done] [613.9M/615.9M /s] [150K/150K iops] [eta 02m:41s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [15.8% done] [620.9M/618.5M /s] [152K/151K iops] [eta 02m:40s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [16.3% done] [632.9M/635.4M /s] [155K/155K iops] [eta 02m:39s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [16.8% done] [634.6M/634.4M /s] [155K/155K iops] [eta 02m:38s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [17.4% done] [634.4M/632.4M /s] [155K/154K iops] [eta 02m:37s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [17.9% done] [634.9M/636.2M /s] [155K/155K iops] [eta 02m:36s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [18.4% done] [642.1M/644.4M /s] [157K/157K iops] [eta 02m:35s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [18.9% done] [648.2M/647.7M /s] [158K/158K iops] [eta 02m:34s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [19.5% done] [647.4M/647.7M /s] [158K/158K iops] [eta 02m:33s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [20.0% done] [649.6M/647.4M /s] [159K/158K iops] [eta 02m:32s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [20.5% done] [649.4M/648.7M /s] [159K/158K iops] [eta 02m:31s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [21.1% done] [648.8M/649.9M /s] [158K/159K iops] [eta 02m:30s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [21.6% done] [649.4M/648.4M /s] [159K/158K iops] [eta 02m:29s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [22.1% done] [647.1M/648.6M /s] [158K/158K iops] [eta 02m:28s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [22.6% done] [646.5M/650.7M /s] [158K/159K iops] [eta 02m:27s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [23.2% done] [657.5M/658.8M /s] [161K/161K iops] [eta 02m:26s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [23.7% done] [661.1M/657.9M /s] [162K/161K iops] [eta 02m:25s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [24.2% done] [658.9M/663.2M /s] [161K/162K iops] [eta 02m:24s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [24.7% done] [660.2M/658.4M /s] [161K/161K iops] [eta 02m:23s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [25.3% done] [658.3M/661.5M /s] [161K/161K iops] [eta 02m:22s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [25.8% done] [659.5M/661.4M /s] [161K/161K iops] [eta 02m:21s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [26.3% done] [654.9M/655.5M /s] [160K/160K iops] [eta 02m:20s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [26.8% done] [659.6M/656.1M /s] [161K/160K iops] [eta 02m:19s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [27.4% done] [661.8M/660.1M /s] [161K/161K iops] [eta 02m:18s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [27.9% done] [659.1M/661.4M /s] [161K/161K iops] [eta 02m:17s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [28.4% done] [644.3M/643.7M /s] [157K/157K iops] [eta 02m:16s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [28.9% done] [661.2M/659.6M /s] [161K/161K iops] [eta 02m:15s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [29.5% done] [659.3M/658.1M /s] [161K/161K iops] [eta 02m:14s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [30.0% done] [659.2M/659.4M /s] [161K/161K iops] [eta 02m:13s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [30.5% done] [660.6M/660.8M /s] [161K/161K iops] [eta 02m:12s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [31.1% done] [660.3M/660.2M /s] [161K/161K iops] [eta 02m:11s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [31.6% done] [658.8M/661.2M /s] [161K/161K iops] [eta 02m:10s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [32.1% done] [660.6M/658.6M /s] [161K/161K iops] [eta 02m:09s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [32.6% done] [661.5M/658.8M /s] [161K/161K iops] [eta 02m:08s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [33.2% done] [660.1M/659.6M /s] [161K/161K iops] [eta 02m:07s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [33.7% done] [659.4M/659.4M /s] [161K/161K iops] [eta 02m:06s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [34.2% done] [659.2M/659.2M /s] [161K/161K iops] [eta 02m:05s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [34.7% done] [659.9M/659.5M /s] [161K/161K iops] [eta 02m:04s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [35.3% done] [636.6M/638.8M /s] [155K/156K iops] [eta 02m:03s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [35.8% done] [626.7M/627.1M /s] [153K/153K iops] [eta 02m:02s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [36.3% done] [623.1M/627.9M /s] [152K/153K iops] [eta 02m:01s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [36.8% done] [630.3M/625.9M /s] [154K/153K iops] [eta 02m:00s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [37.4% done] [625.6M/629.6M /s] [153K/154K iops] [eta 01m:59s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [37.9% done] [623.7M/626.6M /s] [152K/153K iops] [eta 01m:58s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [38.4% done] [625.5M/624.2M /s] [153K/152K iops] [eta 01m:57s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [38.9% done] [627.2M/623.2M /s] [153K/152K iops] [eta 01m:56s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [39.5% done] [626.6M/627.7M /s] [153K/153K iops] [eta 01m:55s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [40.0% done] [617.9M/616.3M /s] [151K/150K iops] [eta 01m:54s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [40.5% done] [608.9M/609.3M /s] [148K/149K iops] [eta 01m:53s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [41.1% done] [620.5M/618.6M /s] [151K/151K iops] [eta 01m:52s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [41.6% done] [624.9M/628.8M /s] [153K/154K iops] [eta 01m:51s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [42.1% done] [640.2M/643.3M /s] [156K/157K iops] [eta 01m:50s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [42.6% done] [644.8M/648.2M /s] [157K/158K iops] [eta 01m:49s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [43.2% done] [646.7M/647.8M /s] [158K/158K iops] [eta 01m:48s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [43.7% done] [647.2M/644.7M /s] [158K/157K iops] [eta 01m:47s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [44.2% done] [648.7M/646.5M /s] [158K/158K iops] [eta 01m:46s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [44.7% done] [647.4M/647.3M /s] [158K/158K iops] [eta 01m:45s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [45.3% done] [646.6M/648.1M /s] [158K/158K iops] [eta 01m:44s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [45.8% done] [644.8M/647.3M /s] [157K/158K iops] [eta 01m:43s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [46.3% done] [647.8M/649.7M /s] [158K/158K iops] [eta 01m:42s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [46.8% done] [645.2M/647.9M /s] [158K/158K iops] [eta 01m:41s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [47.4% done] [648.1M/646.4M /s] [158K/158K iops] [eta 01m:40s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [47.9% done] [658.9M/656.7M /s] [161K/160K iops] [eta 01m:39s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [48.4% done] [661.6M/658.6M /s] [161K/161K iops] [eta 01m:38s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [48.9% done] [659.3M/663.5M /s] [161K/162K iops] [eta 01m:37s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [49.5% done] [656.1M/663.7M /s] [160K/162K iops] [eta 01m:36s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [50.0% done] [660.9M/657.1M /s] [161K/161K iops] [eta 01m:35s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [50.5% done] [660.2M/662.4M /s] [161K/162K iops] [eta 01m:34s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [51.1% done] [643.4M/643.4M /s] [157K/157K iops] [eta 01m:33s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [51.6% done] [652.4M/653.5M /s] [159K/160K iops] [eta 01m:32s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [52.1% done] [629.3M/630.9M /s] [154K/154K iops] [eta 01m:31s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [52.6% done] [624.3M/625.4M /s] [152K/153K iops] [eta 01m:30s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [53.2% done] [625.3M/622.8M /s] [153K/152K iops] [eta 01m:29s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [53.7% done] [626.3M/624.6M /s] [153K/152K iops] [eta 01m:28s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [54.2% done] [639.3M/639.1M /s] [156K/156K iops] [eta 01m:27s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [54.7% done] [645.5M/648.5M /s] [158K/158K iops] [eta 01m:26s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [55.3% done] [649.6M/652.6M /s] [159K/159K iops] [eta 01m:25s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [55.8% done] [659.1M/661.2M /s] [161K/161K iops] [eta 01m:24s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [56.3% done] [659.8M/659.5M /s] [161K/161K iops] [eta 01m:23s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [56.8% done] [661.5M/660.2M /s] [161K/161K iops] [eta 01m:22s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [57.4% done] [664.5M/657.6M /s] [162K/161K iops] [eta 01m:21s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [57.9% done] [659.2M/661.9M /s] [161K/162K iops] [eta 01m:20s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [58.4% done] [630.4M/632.7M /s] [154K/154K iops] [eta 01m:19s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [58.9% done] [623.3M/626.9M /s] [152K/153K iops] [eta 01m:18s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [59.5% done] [627.7M/626.8M /s] [153K/153K iops] [eta 01m:17s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [60.0% done] [627.9M/629.2M /s] [153K/154K iops] [eta 01m:16s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [60.5% done] [626.7M/625.1M /s] [153K/153K iops] [eta 01m:15s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [61.1% done] [627.9M/624.2M /s] [153K/152K iops] [eta 01m:14s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [61.6% done] [627.1M/626.3M /s] [153K/153K iops] [eta 01m:13s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [62.1% done] [627.2M/629.8M /s] [153K/154K iops] [eta 01m:12s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [62.6% done] [624.8M/625.2M /s] [153K/153K iops] [eta 01m:11s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [63.2% done] [624.9M/624.3M /s] [153K/152K iops] [eta 01m:10s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [63.7% done] [625.6M/626.7M /s] [153K/153K iops] [eta 01m:09s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [64.2% done] [626.7M/626.4M /s] [153K/153K iops] [eta 01m:08s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [64.7% done] [625.1M/630.8M /s] [153K/154K iops] [eta 01m:07s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [65.3% done] [626.3M/626.2M /s] [153K/153K iops] [eta 01m:06s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [65.8% done] [628.7M/627.1M /s] [153K/153K iops] [eta 01m:05s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [66.3% done] [624.5M/625.1M /s] [152K/153K iops] [eta 01m:04s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [66.8% done] [626.6M/626.4M /s] [153K/153K iops] [eta 01m:03s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [67.4% done] [636.6M/636.7M /s] [155K/155K iops] [eta 01m:02s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [67.9% done] [652.4M/652.2M /s] [159K/159K iops] [eta 01m:01s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [68.4% done] [657.3M/659.5M /s] [160K/161K iops] [eta 01m:00s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [68.9% done] [658.5M/658.4M /s] [161K/161K iops] [eta 00m:59s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [69.5% done] [658.8M/657.9M /s] [161K/161K iops] [eta 00m:58s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [70.0% done] [658.1M/658.2M /s] [161K/161K iops] [eta 00m:57s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [70.5% done] [657.1M/659.8M /s] [161K/161K iops] [eta 00m:56s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [71.1% done] [658.2M/657.1M /s] [161K/161K iops] [eta 00m:55s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [71.6% done] [658.9M/659.8M /s] [161K/161K iops] [eta 00m:54s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [72.1% done] [659.6M/661.1M /s] [161K/162K iops] [eta 00m:53s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [72.6% done] [658.8M/662.8M /s] [161K/162K iops] [eta 00m:52s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [73.2% done] [658.7M/660.6M /s] [161K/161K iops] [eta 00m:51s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [73.7% done] [660.7M/660.9M /s] [161K/161K iops] [eta 00m:50s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [74.2% done] [658.9M/660.1M /s] [161K/161K iops] [eta 00m:49s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [74.7% done] [661.2M/659.2M /s] [161K/161K iops] [eta 00m:48s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [75.3% done] [661.5M/659.5M /s] [161K/161K iops] [eta 00m:47s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [75.8% done] [664.1M/659.9M /s] [162K/161K iops] [eta 00m:46s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [76.3% done] [675.6M/674.6M /s] [165K/165K iops] [eta 00m:45s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [76.8% done] [675.2M/674.5M /s] [165K/165K iops] [eta 00m:44s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [77.4% done] [674.7M/676.5M /s] [165K/165K iops] [eta 00m:43s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [77.9% done] [676.4M/673.9M /s] [165K/165K iops] [eta 00m:42s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [78.4% done] [675.6M/674.6M /s] [165K/165K iops] [eta 00m:41s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [78.9% done] [673.4M/674.5M /s] [164K/165K iops] [eta 00m:40s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [79.5% done] [663.8M/662.7M /s] [162K/162K iops] [eta 00m:39s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [80.0% done] [675.5M/674.5M /s] [165K/165K iops] [eta 00m:38s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [80.5% done] [673.1M/674.7M /s] [164K/165K iops] [eta 00m:37s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [81.1% done] [672.6M/672.5M /s] [164K/164K iops] [eta 00m:36s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [81.7% done] [674.8M/673.8M /s] [165K/164K iops] [eta 00m:35s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [82.2% done] [674.3M/673.4M /s] [165K/164K iops] [eta 00m:34s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [82.7% done] [671.5M/674.2M /s] [164K/165K iops] [eta 00m:33s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [83.2% done] [674.3M/673.3M /s] [165K/164K iops] [eta 00m:32s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [83.8% done] [675.4M/673.1M /s] [165K/165K iops] [eta 00m:31s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [84.3% done] [676.4M/673.5M /s] [165K/164K iops] [eta 00m:30s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [84.8% done] [676.4M/675.5M /s] [165K/165K iops] [eta 00m:29s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [85.3% done] [676.3M/675.7M /s] [165K/165K iops] [eta 00m:28s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [85.9% done] [676.8M/676.4M /s] [165K/165K iops] [eta 00m:27s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [86.4% done] [677.6M/672.1M /s] [165K/164K iops] [eta 00m:26s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [86.9% done] [675.1M/675.1M /s] [165K/165K iops] [eta 00m:25s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [87.4% done] [676.4M/675.4M /s] [165K/165K iops] [eta 00m:24s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [88.0% done] [677.7M/677.2M /s] [165K/165K iops] [eta 00m:23s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [88.5% done] [677.6M/676.4M /s] [165K/165K iops] [eta 00m:22s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [89.0% done] [676.8M/676.8M /s] [165K/165K iops] [eta 00m:21s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [89.5% done] [672.8M/680.7M /s] [164K/166K iops] [eta 00m:20s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [90.1% done] [676.5M/677.3M /s] [165K/165K iops] [eta 00m:19s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [90.6% done] [678.1M/676.9M /s] [166K/165K iops] [eta 00m:18s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [91.1% done] [675.8M/677.8M /s] [165K/165K iops] [eta 00m:17s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [91.6% done] [674.4M/675.4M /s] [165K/165K iops] [eta 00m:16s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [92.1% done] [664.6M/663.8M /s] [162K/162K iops] [eta 00m:15s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [92.7% done] [675.5M/671.9M /s] [165K/164K iops] [eta 00m:14s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [93.2% done] [672.7M/672.5M /s] [164K/164K iops] [eta 00m:13s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [93.7% done] [671.4M/668.8M /s] [164K/163K iops] [eta 00m:12s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [94.2% done] [670.4M/672.3M /s] [164K/164K iops] [eta 00m:11s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [94.8% done] [672.8M/675.5M /s] [164K/165K iops] [eta 00m:10s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [95.3% done] [676.2M/670.5M /s] [165K/164K iops] [eta 00m:09s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [95.8% done] [673.9M/675.6M /s] [165K/165K iops] [eta 00m:08s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [96.3% done] [676.3M/672.7M /s] [165K/164K iops] [eta 00m:07s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [96.9% done] [676.8M/671.1M /s] [165K/164K iops] [eta 00m:06s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [97.4% done] [672.5M/673.8M /s] [164K/164K iops] [eta 00m:05s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [97.9% done] [671.4M/669.7M /s] [164K/163K iops] [eta 00m:04s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [98.4% done] [670.4M/674.4M /s] [164K/165K iops] [eta 00m:03s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [99.0% done] [673.7M/674.2M /s] [164K/165K iops] [eta 00m:02s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [99.5% done] [673.2M/673.2M /s] [164K/164K iops] [eta 00m:01s]\rJobs: 12 (f=12): [MMMMMMMMMMMM] [100.0% done] [674.7M/672.4M /s] [165K/164K iops] [eta 00m:00s]
job1: (groupid=0, jobs=12): err= 0: pid=3874
read : io=112337MB, bw=639052KB/s, iops=159758 , runt=180005msec
slat (usec): min=1 , max=4406 , avg= 4.34, stdev= 7.19
clat (usec): min=20 , max=16839 , avg=4888.83, stdev=1496.23
lat (usec): min=28 , max=16841 , avg=4893.26, stdev=1494.28
clat percentiles (usec):
| 1.00th=[ 2928], 5.00th=[ 3248], 10.00th=[ 3312], 20.00th=[ 3408],
| 30.00th=[ 3472], 40.00th=[ 3568], 50.00th=[ 3920], 60.00th=[ 6112],
| 70.00th=[ 6304], 80.00th=[ 6432], 90.00th=[ 6624], 95.00th=[ 6688],
| 99.00th=[ 6944], 99.50th=[ 7136], 99.90th=[ 8256]
bw (KB/s) : min= 0, max=168600, per=8.37%, avg=53494.99, stdev=7987.05
write: io=112343MB, bw=639089KB/s, iops=159767 , runt=180005msec
slat (usec): min=1 , max=3341 , avg= 4.52, stdev= 7.08
clat (usec): min=26 , max=12875 , avg=4714.99, stdev=1449.90
lat (usec): min=34 , max=12877 , avg=4719.60, stdev=1447.95
clat percentiles (usec):
| 1.00th=[ 2896], 5.00th=[ 3152], 10.00th=[ 3216], 20.00th=[ 3280],
| 30.00th=[ 3344], 40.00th=[ 3440], 50.00th=[ 3696], 60.00th=[ 5920],
| 70.00th=[ 6048], 80.00th=[ 6176], 90.00th=[ 6368], 95.00th=[ 6496],
| 99.00th=[ 6688], 99.50th=[ 6880], 99.90th=[ 7968]
bw (KB/s) : min= 0, max=155768, per=8.36%, avg=53397.35, stdev=7730.15
lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.03%
lat (usec) : 1000=0.03%
lat (msec) : 2=0.25%, 4=49.89%, 10=49.78%, 20=0.01%
cpu : usr=3.56%, sys=12.37%, ctx=867899, majf=0, minf=3732
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=105.4%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued : total=r=28757389/w=28759031/d=0, short=r=0/w=0/d=0
Run status group 0 (all jobs):
READ: io=112337MB, aggrb=639052KB/s, minb=654389KB/s, maxb=654389KB/s, mint=180005msec, maxt=180005msec
WRITE: io=112343MB, aggrb=639088KB/s, minb=654426KB/s, maxb=654426KB/s, mint=180005msec, maxt=180005msec
Disk stats (read/write):
sdb: ios=1882423/1870026, merge=655365/638486, ticks=7598597/7455054, in_queue=11516156, util=100.00%
sdc: ios=1875581/1859758, merge=666514/652191, ticks=7771243/7603780, in_queue=11852784, util=100.00%
sda: ios=1861462/1870446, merge=648061/669440, ticks=7741259/7625233, in_queue=11829088, util=100.00%
sde: ios=1859610/1871711, merge=648268/669318, ticks=7731183/7621478, in_queue=11840624, util=100.00%
sdd: ios=1875063/1861567, merge=668113/651005, ticks=7768061/7604671, in_queue=11861292, util=100.00%
sdf: ios=1875610/1860054, merge=669361/648896, ticks=7766338/7597224, in_queue=11844000, util=100.00%
sdh: ios=1860733/1872170, merge=648073/669182, ticks=7738473/7627724, in_queue=11853960, util=100.00%
sdi: ios=1866027/1878153, merge=638707/658066, ticks=7562550/7470393, in_queue=11523588, util=100.00%
sdj: ios=1865874/1878173, merge=640898/662800, ticks=7607518/7511112, in_queue=11621248, util=100.00%
sdg: ios=1859649/1872425, merge=646653/664821, ticks=7735257/7641005, in_queue=11860448, util=100.00%
sdl: ios=1875534/1861691, merge=668426/649576, ticks=7761596/7601379, in_queue=11831976, util=100.00%
sdk: ios=1879027/1863265, merge=666816/651064, ticks=7771003/7608856, in_queue=11837400, util=100.00%
root@(none)$ rmmod hisi_sas_v3_hw.ko
[ 601.784539] hisi_sas_v3_hw 0000:74:02.0: dev[14:1] is gone
[ 601.784539] hisi_sas_v3_hw 0000:74:02.0: dev[14:1] is gone
[ 601.795921] sd 3:0:11:0: [sdl] Synchronizing SCSI cache
[ 601.795921] sd 3:0:11:0: [sdl] Synchronizing SCSI cache
[ 601.852572] hisi_sas_v3_hw 0000:74:02.0: dev[13:1] is gone
[ 601.852572] hisi_sas_v3_hw 0000:74:02.0: dev[13:1] is gone
[ 601.863924] sd 3:0:10:0: [sdk] Synchronizing SCSI cache
[ 601.863924] sd 3:0:10:0: [sdk] Synchronizing SCSI cache
[ 601.920588] hisi_sas_v3_hw 0000:74:02.0: dev[12:1] is gone
[ 601.920588] hisi_sas_v3_hw 0000:74:02.0: dev[12:1] is gone
[ 601.931985] sd 3:0:9:0: [sdj] Synchronizing SCSI cache
[ 601.931985] sd 3:0:9:0: [sdj] Synchronizing SCSI cache
[ 602.004558] hisi_sas_v3_hw 0000:74:02.0: dev[11:1] is gone
[ 602.004558] hisi_sas_v3_hw 0000:74:02.0: dev[11:1] is gone
[ 602.015951] sd 3:0:8:0: [sdi] Synchronizing SCSI cache
[ 602.015951] sd 3:0:8:0: [sdi] Synchronizing SCSI cache
[ 602.076536] hisi_sas_v3_hw 0000:74:02.0: dev[10:1] is gone
[ 602.076536] hisi_sas_v3_hw 0000:74:02.0: dev[10:1] is gone
[ 602.087930] sd 3:0:7:0: [sdh] Synchronizing SCSI cache
[ 602.087930] sd 3:0:7:0: [sdh] Synchronizing SCSI cache
[ 602.132543] hisi_sas_v3_hw 0000:74:02.0: dev[9:1] is gone
[ 602.132543] hisi_sas_v3_hw 0000:74:02.0: dev[9:1] is gone
[ 602.143723] sd 3:0:6:0: [sdg] Synchronizing SCSI cache
[ 602.143723] sd 3:0:6:0: [sdg] Synchronizing SCSI cache
[ 602.200555] hisi_sas_v3_hw 0000:74:02.0: dev[8:1] is gone
[ 602.200555] hisi_sas_v3_hw 0000:74:02.0: dev[8:1] is gone
[ 602.211742] sd 3:0:5:0: [sdf] Synchronizing SCSI cache
[ 602.211742] sd 3:0:5:0: [sdf] Synchronizing SCSI cache
[ 602.252523] hisi_sas_v3_hw 0000:74:02.0: dev[7:1] is gone
[ 602.252523] hisi_sas_v3_hw 0000:74:02.0: dev[7:1] is gone
[ 602.263700] sd 3:0:4:0: [sde] Synchronizing SCSI cache
[ 602.263700] sd 3:0:4:0: [sde] Synchronizing SCSI cache
[ 602.312494] hisi_sas_v3_hw 0000:74:02.0: dev[6:1] is gone
[ 602.312494] hisi_sas_v3_hw 0000:74:02.0: dev[6:1] is gone
[ 602.323660] sd 3:0:3:0: [sdd] Synchronizing SCSI cache
[ 602.323660] sd 3:0:3:0: [sdd] Synchronizing SCSI cache
[ 602.376485] hisi_sas_v3_hw 0000:74:02.0: dev[5:1] is gone
[ 602.376485] hisi_sas_v3_hw 0000:74:02.0: dev[5:1] is gone
[ 602.387642] sd 3:0:2:0: [sdc] Synchronizing SCSI cache
[ 602.387642] sd 3:0:2:0: [sdc] Synchronizing SCSI cache
[ 602.440474] hisi_sas_v3_hw 0000:74:02.0: dev[4:1] is gone
[ 602.440474] hisi_sas_v3_hw 0000:74:02.0: dev[4:1] is gone
[ 602.451671] sd 3:0:1:0: [sdb] Synchronizing SCSI cache
[ 602.451671] sd 3:0:1:0: [sdb] Synchronizing SCSI cache
[ 602.512430] hisi_sas_v3_hw 0000:74:02.0: dev[3:1] is gone
[ 602.512430] hisi_sas_v3_hw 0000:74:02.0: dev[3:1] is gone
[ 602.523649] sd 3:0:0:0: [sda] Synchronizing SCSI cache
[ 602.523649] sd 3:0:0:0: [sda] Synchronizing SCSI cache
[ 602.580517] hisi_sas_v3_hw 0000:74:02.0: dev[2:1] is gone
[ 602.580517] hisi_sas_v3_hw 0000:74:02.0: dev[2:1] is gone
[ 602.592684] hisi_sas_v3_hw 0000:74:02.0: dev[1:2] is gone
[ 602.592684] hisi_sas_v3_hw 0000:74:02.0: dev[1:2] is gone
root@(none)$
root@(none)$
root@(none)$
root@(none)$ insmod hisi_sas_v3_hw.ko enable_dif_dix=1
[ 614.945604] scsi host3: hisi_sas_v3_hw
[ 614.945604] scsi host3: hisi_sas_v3_hw
[ 616.159646] hisi_sas_v3_hw 0000:74:02.0: Registering for DIF/DIX type 1/2/3 protection.
[ 616.159646] hisi_sas_v3_hw 0000:74:02.0: Registering for DIF/DIX type 1/2/3 protection.
[ 616.233223] hisi_sas_v3_hw 0000:74:02.0: phyup: phy0 link_rate=11
[ 616.233223] hisi_sas_v3_hw 0000:74:02.0: phyup: phy0 link_rate=11
[ 616.245423] hisi_sas_v3_hw 0000:74:02.0: phyup: phy1 link_rate=11
[ 616.245423] hisi_sas_v3_hw 0000:74:02.0: phyup: phy1 link_rate=11
[ 616.257620] hisi_sas_v3_hw 0000:74:02.0: phyup: phy2 link_rate=11
[ 616.257620] hisi_sas_v3_hw 0000:74:02.0: phyup: phy2 link_rate=11
[ 616.269816] hisi_sas_v3_hw 0000:74:02.0: phyup: phy4 link_rate=11
[ 616.269816] hisi_sas_v3_hw 0000:74:02.0: phyup: phy4 link_rate=11
[ 616.282141] hisi_sas_v3_hw 0000:74:02.0: dev[1:2] found
[ 616.282141] hisi_sas_v3_hw 0000:74:02.0: dev[1:2] found
[ 616.296076] hisi_sas_v3_hw 0000:74:02.0: dev[2:1] found
[ 616.296076] hisi_sas_v3_hw 0000:74:02.0: dev[2:1] found
[ 616.306620] hisi_sas_v3_hw 0000:74:02.0: dev[3:1] found
[ 616.306620] hisi_sas_v3_hw 0000:74:02.0: dev[3:1] found
[ 616.317160] hisi_sas_v3_hw 0000:74:02.0: dev[4:1] found
[ 616.317160] hisi_sas_v3_hw 0000:74:02.0: dev[4:1] found
[ 616.327695] hisi_sas_v3_hw 0000:74:02.0: dev[5:1] found
[ 616.327695] hisi_sas_v3_hw 0000:74:02.0: dev[5:1] found
[ 616.338227] hisi_sas_v3_hw 0000:74:02.0: dev[6:1] found
[ 616.338227] hisi_sas_v3_hw 0000:74:02.0: dev[6:1] found
[ 616.348762] hisi_sas_v3_hw 0000:74:02.0: dev[7:1] found
[ 616.348762] hisi_sas_v3_hw 0000:74:02.0: dev[7:1] found
[ 616.359295] hisi_sas_v3_hw 0000:74:02.0: dev[8:1] found
[ 616.359295] hisi_sas_v3_hw 0000:74:02.0: dev[8:1] found
[ 616.369829] hisi_sas_v3_hw 0000:74:02.0: dev[9:1] found
[ 616.369829] hisi_sas_v3_hw 0000:74:02.0: dev[9:1] found
[ 616.380364] hisi_sas_v3_hw 0000:74:02.0: dev[10:1] found
[ 616.380364] hisi_sas_v3_hw 0000:74:02.0: dev[10:1] found
[ 616.391071] hisi_sas_v3_hw 0000:74:02.0: dev[11:1] found
[ 616.391071] hisi_sas_v3_hw 0000:74:02.0: dev[11:1] found
[ 616.401779] hisi_sas_v3_hw 0000:74:02.0: dev[12:1] found
[ 616.401779] hisi_sas_v3_hw 0000:74:02.0: dev[12:1] found
[ 616.412489] hisi_sas_v3_hw 0000:74:02.0: dev[13:1] found
[ 616.412489] hisi_sas_v3_hw 0000:74:02.0: dev[13:1] found
[ 616.423219] hisi_sas_v3_hw 0000:74:02.0: dev[14:1] found
[ 616.423219] hisi_sas_v3_hw 0000:74:02.0: dev[14:1] found
[ 616.434412] scsi 3:0:0:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 616.434412] scsi 3:0:0:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 616.450976] sd 3:0:0:0: Attached scsi generic sg0 type 0
[ 616.450976] sd 3:0:0:0: Attached scsi generic sg0 type 0
[ 616.451286] sd 3:0:0:0: [sda] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 616.451286] sd 3:0:0:0: [sda] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 616.462192] scsi 3:0:1:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 616.462192] scsi 3:0:1:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 616.476597] sd 3:0:0:0: [sda] 4096-byte physical blocks
[ 616.476597] sd 3:0:0:0: [sda] 4096-byte physical blocks
[ 616.493127] sd 3:0:1:0: Attached scsi generic sg1 type 0
[ 616.493127] sd 3:0:1:0: Attached scsi generic sg1 type 0
[ 616.493450] sd 3:0:1:0: [sdb] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 616.493450] sd 3:0:1:0: [sdb] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 616.493452] sd 3:0:1:0: [sdb] 4096-byte physical blocks
[ 616.493452] sd 3:0:1:0: [sdb] 4096-byte physical blocks
[ 616.493551] sd 3:0:1:0: [sdb] Write Protect is off
[ 616.493551] sd 3:0:1:0: [sdb] Write Protect is off
[ 616.493680] sd 3:0:1:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 616.493680] sd 3:0:1:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 616.500387] sd 3:0:1:0: [sdb] Attached SCSI disk
[ 616.500387] sd 3:0:1:0: [sdb] Attached SCSI disk
[ 616.503354] sd 3:0:0:0: [sda] Write Protect is off
[ 616.503354] sd 3:0:0:0: [sda] Write Protect is off
[ 616.514415] scsi 3:0:2:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 616.514415] scsi 3:0:2:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 616.528992] sd 3:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 616.528992] sd 3:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 616.539672] sd 3:0:2:0: Attached scsi generic sg2 type 0
[ 616.539672] sd 3:0:2:0: Attached scsi generic sg2 type 0
[ 616.539953] sd 3:0:2:0: [sdc] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 616.539953] sd 3:0:2:0: [sdc] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 616.539955] sd 3:0:2:0: [sdc] 4096-byte physical blocks
[ 616.539955] sd 3:0:2:0: [sdc] 4096-byte physical blocks
[ 616.540052] sd 3:0:2:0: [sdc] Write Protect is off
[ 616.540052] sd 3:0:2:0: [sdc] Write Protect is off
[ 616.540176] sd 3:0:2:0: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 616.540176] sd 3:0:2:0: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 616.546808] sd 3:0:2:0: [sdc] Attached SCSI disk
[ 616.546808] sd 3:0:2:0: [sdc] Attached SCSI disk
[ 616.555549] sd 3:0:0:0: [sda] Attached SCSI disk
[ 616.555549] sd 3:0:0:0: [sda] Attached SCSI disk
[ 616.566511] scsi 3:0:3:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 616.566511] scsi 3:0:3:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 616.715976] sd 3:0:3:0: Attached scsi generic sg3 type 0
[ 616.715976] sd 3:0:3:0: Attached scsi generic sg3 type 0
[ 616.716279] sd 3:0:3:0: [sdd] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 616.716279] sd 3:0:3:0: [sdd] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 616.727163] scsi 3:0:4:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 616.727163] scsi 3:0:4:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 616.741609] sd 3:0:3:0: [sdd] 4096-byte physical blocks
[ 616.741609] sd 3:0:3:0: [sdd] 4096-byte physical blocks
[ 616.758131] sd 3:0:4:0: Attached scsi generic sg4 type 0
[ 616.758131] sd 3:0:4:0: Attached scsi generic sg4 type 0
[ 616.758492] sd 3:0:4:0: [sde] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 616.758492] sd 3:0:4:0: [sde] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 616.758496] sd 3:0:4:0: [sde] 4096-byte physical blocks
[ 616.758496] sd 3:0:4:0: [sde] 4096-byte physical blocks
[ 616.758597] sd 3:0:4:0: [sde] Write Protect is off
[ 616.758597] sd 3:0:4:0: [sde] Write Protect is off
[ 616.758727] sd 3:0:4:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 616.758727] sd 3:0:4:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 616.766077] sd 3:0:4:0: [sde] Attached SCSI disk
[ 616.766077] sd 3:0:4:0: [sde] Attached SCSI disk
[ 616.768431] sd 3:0:3:0: [sdd] Write Protect is off
[ 616.768431] sd 3:0:3:0: [sdd] Write Protect is off
[ 616.779362] scsi 3:0:5:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 616.779362] scsi 3:0:5:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 616.804667] sd 3:0:5:0: Attached scsi generic sg5 type 0
[ 616.804667] sd 3:0:5:0: Attached scsi generic sg5 type 0
[ 616.804700] sd 3:0:5:0: [sdf] Enabling DIF Type 1 protection
[ 616.804700] sd 3:0:5:0: [sdf] Enabling DIF Type 1 protection
[ 616.804977] sd 3:0:5:0: [sdf] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 616.804977] sd 3:0:5:0: [sdf] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 616.804979] sd 3:0:5:0: [sdf] 4096-byte physical blocks
[ 616.804979] sd 3:0:5:0: [sdf] 4096-byte physical blocks
[ 616.805078] sd 3:0:5:0: [sdf] Write Protect is off
[ 616.805078] sd 3:0:5:0: [sdf] Write Protect is off
[ 616.805206] sd 3:0:5:0: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 616.805206] sd 3:0:5:0: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 616.809934] sd 3:0:5:0: [sdf] Enabling DIX T10-DIF-TYPE1-CRC protection
[ 616.809934] sd 3:0:5:0: [sdf] Enabling DIX T10-DIF-TYPE1-CRC protection
[ 616.811961] sd 3:0:5:0: [sdf] Attached SCSI disk
[ 616.811961] sd 3:0:5:0: [sdf] Attached SCSI disk
[ 616.814040] sd 3:0:3:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 616.814040] sd 3:0:3:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 616.831533] scsi 3:0:6:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 616.831533] scsi 3:0:6:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 616.847197] sd 3:0:3:0: [sdd] Attached SCSI disk
[ 616.847197] sd 3:0:3:0: [sdd] Attached SCSI disk
[ 616.850133] sd 3:0:6:0: Attached scsi generic sg6 type 0
[ 616.850133] sd 3:0:6:0: Attached scsi generic sg6 type 0
[ 616.850174] sd 3:0:6:0: [sdg] Enabling DIF Type 1 protection
[ 616.850174] sd 3:0:6:0: [sdg] Enabling DIF Type 1 protection
[ 616.850451] sd 3:0:6:0: [sdg] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 616.850451] sd 3:0:6:0: [sdg] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 616.850453] sd 3:0:6:0: [sdg] 4096-byte physical blocks
[ 616.850453] sd 3:0:6:0: [sdg] 4096-byte physical blocks
[ 616.850552] sd 3:0:6:0: [sdg] Write Protect is off
[ 616.850552] sd 3:0:6:0: [sdg] Write Protect is off
[ 616.850681] sd 3:0:6:0: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 616.850681] sd 3:0:6:0: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 616.855312] sd 3:0:6:0: [sdg] Enabling DIX T10-DIF-TYPE1-CRC protection
[ 616.855312] sd 3:0:6:0: [sdg] Enabling DIX T10-DIF-TYPE1-CRC protection
[ 616.857340] sd 3:0:6:0: [sdg] Attached SCSI disk
[ 616.857340] sd 3:0:6:0: [sdg] Attached SCSI disk
[ 617.102088] scsi 3:0:7:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 617.102088] scsi 3:0:7:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 617.118639] sd 3:0:7:0: Attached scsi generic sg7 type 0
[ 617.118639] sd 3:0:7:0: Attached scsi generic sg7 type 0
[ 617.118951] sd 3:0:7:0: [sdh] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 617.118951] sd 3:0:7:0: [sdh] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 617.129747] scsi 3:0:8:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 617.129747] scsi 3:0:8:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 617.144257] sd 3:0:7:0: [sdh] 4096-byte physical blocks
[ 617.144257] sd 3:0:7:0: [sdh] 4096-byte physical blocks
[ 617.144364] sd 3:0:7:0: [sdh] Write Protect is off
[ 617.144364] sd 3:0:7:0: [sdh] Write Protect is off
[ 617.160903] sd 3:0:8:0: Attached scsi generic sg8 type 0
[ 617.160903] sd 3:0:8:0: Attached scsi generic sg8 type 0
[ 617.160944] sd 3:0:8:0: [sdi] Enabling DIF Type 1 protection
[ 617.160944] sd 3:0:8:0: [sdi] Enabling DIF Type 1 protection
[ 617.161227] sd 3:0:8:0: [sdi] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 617.161227] sd 3:0:8:0: [sdi] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 617.161229] sd 3:0:8:0: [sdi] 4096-byte physical blocks
[ 617.161229] sd 3:0:8:0: [sdi] 4096-byte physical blocks
[ 617.161329] sd 3:0:8:0: [sdi] Write Protect is off
[ 617.161329] sd 3:0:8:0: [sdi] Write Protect is off
[ 617.161458] sd 3:0:8:0: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 617.161458] sd 3:0:8:0: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 617.166257] sd 3:0:8:0: [sdi] Enabling DIX T10-DIF-TYPE1-CRC protection
[ 617.166257] sd 3:0:8:0: [sdi] Enabling DIX T10-DIF-TYPE1-CRC protection
[ 617.168273] sd 3:0:8:0: [sdi] Attached SCSI disk
[ 617.168273] sd 3:0:8:0: [sdi] Attached SCSI disk
[ 617.180972] scsi 3:0:9:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 617.180972] scsi 3:0:9:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 617.191261] sd 3:0:7:0: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 617.191261] sd 3:0:7:0: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 617.202791] sd 3:0:9:0: Attached scsi generic sg9 type 0
[ 617.202791] sd 3:0:9:0: Attached scsi generic sg9 type 0
[ 617.203116] sd 3:0:9:0: [sdj] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 617.203116] sd 3:0:9:0: [sdj] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 617.203118] sd 3:0:9:0: [sdj] 4096-byte physical blocks
[ 617.203118] sd 3:0:9:0: [sdj] 4096-byte physical blocks
[ 617.203218] sd 3:0:9:0: [sdj] Write Protect is off
[ 617.203218] sd 3:0:9:0: [sdj] Write Protect is off
[ 617.203347] sd 3:0:9:0: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 617.203347] sd 3:0:9:0: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 617.209937] sd 3:0:9:0: [sdj] Attached SCSI disk
[ 617.209937] sd 3:0:9:0: [sdj] Attached SCSI disk
[ 617.223990] sd 3:0:7:0: [sdh] Attached SCSI disk
[ 617.223990] sd 3:0:7:0: [sdh] Attached SCSI disk
[ 617.228375] scsi 3:0:10:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 617.228375] scsi 3:0:10:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 617.408424] sd 3:0:10:0: Attached scsi generic sg10 type 0
[ 617.408424] sd 3:0:10:0: Attached scsi generic sg10 type 0
[ 617.408754] sd 3:0:10:0: [sdk] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 617.408754] sd 3:0:10:0: [sdk] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 617.420278] scsi 3:0:11:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 617.420278] scsi 3:0:11:0: Direct-Access HGST HUSMM1640ASS204 C2D0 PQ: 0 ANSI: 6
[ 617.434673] sd 3:0:10:0: [sdk] 4096-byte physical blocks
[ 617.434673] sd 3:0:10:0: [sdk] 4096-byte physical blocks
[ 617.451488] sd 3:0:11:0: Attached scsi generic sg11 type 0
[ 617.451488] sd 3:0:11:0: Attached scsi generic sg11 type 0
[ 617.451838] sd 3:0:11:0: [sdl] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 617.451838] sd 3:0:11:0: [sdl] 781422768 512-byte logical blocks: (400 GB/373 GiB)
[ 617.451840] sd 3:0:11:0: [sdl] 4096-byte physical blocks
[ 617.451840] sd 3:0:11:0: [sdl] 4096-byte physical blocks
[ 617.451944] sd 3:0:11:0: [sdl] Write Protect is off
[ 617.451944] sd 3:0:11:0: [sdl] Write Protect is off
[ 617.452080] sd 3:0:11:0: [sdl] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 617.452080] sd 3:0:11:0: [sdl] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 617.460540] sd 3:0:11:0: [sdl] Attached SCSI disk
[ 617.460540] sd 3:0:11:0: [sdl] Attached SCSI disk
[ 617.461870] sd 3:0:10:0: [sdk] Write Protect is off
[ 617.461870] sd 3:0:10:0: [sdk] Write Protect is off
[ 617.473661] scsi 3:0:12:0: Enclosure 12G SAS Expander RevB PQ: 0 ANSI: 6
[ 617.473661] scsi 3:0:12:0: Enclosure 12G SAS Expander RevB PQ: 0 ANSI: 6
[ 617.499127] ses 3:0:12:0: Attached Enclosure device
[ 617.499127] ses 3:0:12:0: Attached Enclosure device
[ 617.508583] sd 3:0:10:0: [sdk] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 617.508583] sd 3:0:10:0: [sdk] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 617.525816] ses 3:0:12:0: Attached scsi generic sg12 type 13
[ 617.525816] ses 3:0:12:0: Attached scsi generic sg12 type 13
[ 617.542531] sd 3:0:10:0: [sdk] Attached SCSI disk
[ 617.542531] sd 3:0:10:0: [sdk] Attached SCSI disk
root@(none)$
root@(none)$
root@(none)$
root@(none)$
root@(none)$
root@(none)$
root@(none)$
root@(none)$ ./creat_fio_task.sh 4k rw 128
Creat 4k_rw_depth128_fiotest file sucessfully
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
fio 2.0.5
Starting 12 processes
[ 629.210506] Unable to handle kernel paging request at virtual address 0000ffff8027e048
[ 629.210506] Unable to handle kernel paging request at virtual address 0000ffff8027e048
[ 629.226373] Mem abort info:
[ 629.226373] Mem abort info:
[ 629.231952] ESR = 0x96000006
[ 629.231952] ESR = 0x96000006
[ 629.238052] Exception class = DABT (current EL), IL = 32 bits
[ 629.238052] Exception class = DABT (current EL), IL = 32 bits
[ 629.249898] SET = 0, FnV = 0
[ 629.249898] SET = 0, FnV = 0
[ 629.255998] EA = 0, S1PTW = 0
[ 629.255998] EA = 0, S1PTW = 0
[ 629.262272] Data abort info:
[ 629.262272] Data abort info:
[ 629.268023] ISV = 0, ISS = 0x00000006
[ 629.268023] ISV = 0, ISS = 0x00000006
[ 629.275690] CM = 0, WnR = 0
[ 629.275690] CM = 0, WnR = 0
[ 629.281617] user pgtable: 4k pages, 48-bit VAs, pgdp = 0000000085c91728
[ 629.281617] user pgtable: 4k pages, 48-bit VAs, pgdp = 0000000085c91728
[ 629.294857] [0000ffff8027e048] pgd=00000027a8644003, pud=00000027a85ea003, pmd=0000000000000000
[ 629.294857] [0000ffff8027e048] pgd=00000027a8644003, pud=00000027a85ea003, pmd=0000000000000000
[ 629.312278] Internal error: Oops: 96000006 [#1] PREEMPT SMP
[ 629.312278] Internal error: Oops: 96000006 [#1] PREEMPT SMP
[ 629.323427] Modules linked in: hisi_sas_v3_hw [last unloaded: hisi_sas_v3_hw]
[ 629.323427] Modules linked in: hisi_sas_v3_hw [last unloaded: hisi_sas_v3_hw]
[ 629.337713] CPU: 13 PID: 4465 Comm: fio Not tainted 4.20.0-rc1-15093-ge876dec #1067
[ 629.337713] CPU: 13 PID: 4465 Comm: fio Not tainted 4.20.0-rc1-15093-ge876dec #1067
[ 629.353040] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 629.353040] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 629.370633] pstate: 80400009 (Nzcv daif +PAN -UAO)
[ 629.370633] pstate: 80400009 (Nzcv daif +PAN -UAO)
[ 629.380218] pc : deadline_remove_request+0x2c/0xd0
[ 629.380218] pc : deadline_remove_request+0x2c/0xd0
[ 629.389799] lr : dd_dispatch_request+0x184/0x1e0
[ 629.389799] lr : dd_dispatch_request+0x184/0x1e0
[ 629.399031] sp : ffff0000275a3aa0
[ 629.399031] sp : ffff0000275a3aa0
[ 629.405651] x29: ffff0000275a3aa0 x28: ffff8027dc230d40
[ 629.405651] x29: ffff0000275a3aa0 x28: ffff8027dc230d40
[ 629.416278] x27: 0000000000000001 x26: ffff8027dc230d40
[ 629.416278] x27: 0000000000000001 x26: ffff8027dc230d40
[ 629.426903] x25: 0000000000000000 x24: ffff0000094f1000
[ 629.426903] x25: 0000000000000000 x24: ffff0000094f1000
[ 629.437529] x23: ffff8027a1e1d140 x22: 0000000000000006
[ 629.437529] x23: ffff8027a1e1d140 x22: 0000000000000006
[ 629.448155] x21: ffff7e009ec69842 x20: ffff8027a1e1d100
[ 629.448155] x21: ffff7e009ec69842 x20: ffff8027a1e1d100
[ 629.458781] x19: ffff8027a1e1d100 x18: 0000000000000000
[ 629.458781] x19: ffff8027a1e1d100 x18: 0000000000000000
[ 629.469406] x17: 000000009e8c9c80 x16: 00000027a3272000
[ 629.469406] x17: 000000009e8c9c80 x16: 00000027a3272000
[ 629.480032] x15: ffff7e009fff9534 x14: ffff7e009ec698c0
[ 629.480032] x15: ffff7e009fff9534 x14: ffff7e009ec698c0
[ 629.490658] x13: ffff7e009ec69880 x12: 0000000000000009
[ 629.490658] x13: ffff7e009ec69880 x12: 0000000000000009
[ 629.501284] x11: 0000000000000000 x10: ffff8027a7c03560
[ 629.501284] x11: 0000000000000000 x10: ffff8027a7c03560
[ 629.511910] x9 : ffff8027a7c007b8 x8 : 0000000000000007
[ 629.511910] x9 : ffff8027a7c007b8 x8 : 0000000000000007
[ 629.522535] x7 : ffff8027a7c011c8 x6 : 0000000000000000
[ 629.522535] x7 : ffff8027a7c011c8 x6 : 0000000000000000
[ 629.533161] x5 : 0000000000000000 x4 : 0000ffff8027e040
[ 629.533161] x5 : 0000000000000000 x4 : 0000ffff8027e040
[ 629.543787] x3 : ffff8027a1e20040 x2 : ffff8027da87f6a0
[ 629.543787] x3 : ffff8027a1e20040 x2 : ffff8027da87f6a0
[ 629.554413] x1 : ffff8027a1e1d140 x0 : ffff8027a1e1d160
[ 629.554413] x1 : ffff8027a1e1d140 x0 : ffff8027a1e1d160
[ 629.565039] Process fio (pid: 4465, stack limit = 0x000000004e5e4c33)
[ 629.565039] Process fio (pid: 4465, stack limit = 0x000000004e5e4c33)
[ 629.577929] Call trace:
[ 629.577929] Call trace:
[ 629.582808] deadline_remove_request+0x2c/0xd0
[ 629.582808] deadline_remove_request+0x2c/0xd0
[ 629.591693] dd_dispatch_request+0x184/0x1e0
[ 629.591693] dd_dispatch_request+0x184/0x1e0
[ 629.600229] blk_mq_do_dispatch_sched+0x4c/0x104
[ 629.600229] blk_mq_do_dispatch_sched+0x4c/0x104
[ 629.609461] blk_mq_sched_dispatch_requests+0x114/0x18c
[ 629.609461] blk_mq_sched_dispatch_requests+0x114/0x18c
[ 629.619913] __blk_mq_run_hw_queue+0xa4/0x11c
[ 629.619913] __blk_mq_run_hw_queue+0xa4/0x11c
[ 629.628624] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 629.628624] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 629.638553] blk_mq_run_hw_queue+0x9c/0xf8
[ 629.638553] blk_mq_run_hw_queue+0x9c/0xf8
[ 629.646741] blk_mq_sched_insert_requests+0x60/0xac
[ 629.646741] blk_mq_sched_insert_requests+0x60/0xac
[ 629.656495] blk_mq_flush_plug_list+0x144/0x16c
[ 629.656495] blk_mq_flush_plug_list+0x144/0x16c
[ 629.665554] blk_flush_plug_list+0xe8/0x244
[ 629.665554] blk_flush_plug_list+0xe8/0x244
[ 629.673915] blk_finish_plug+0x30/0x40
[ 629.673915] blk_finish_plug+0x30/0x40
[ 629.681408] __arm64_sys_io_submit+0x114/0x218
[ 629.681408] __arm64_sys_io_submit+0x114/0x218
[ 629.690294] el0_svc_common+0x94/0xf0
[ 629.690294] el0_svc_common+0x94/0xf0
[ 629.697611] el0_svc_handler+0x24/0x80
[ 629.697611] el0_svc_handler+0x24/0x80
[ 629.705102] el0_svc+0x8/0x7c0
[ 629.705102] el0_svc+0x8/0x7c0
[ 629.711200] Code: 91010021 f9400ea4 f9402263 f9402662 (f9400496)
[ 629.711200] Code: 91010021 f9400ea4 f9402263 f9402662 (f9400496)
[ 629.723394] ---[ end trace 22423ac28b61ac51 ]---
[ 629.723394] ---[ end trace 22423ac28b61ac51 ]---
[ 629.732631] note: fio[4465] exited with preempt_count 2
[ 629.732631] note: fio[4465] exited with preempt_count 2
[ 629.743112] WARNING: CPU: 9 PID: 4465 at kernel/rcu/tree_plugin.h:337 rcu_note_context_switch+0x38/0x2ec
[ 629.743112] WARNING: CPU: 9 PID: 4465 at kernel/rcu/tree_plugin.h:337 rcu_note_context_switch+0x38/0x2ec
[ 629.762100] Modules linked in: hisi_sas_v3_hw [last unloaded: hisi_sas_v3_hw]
[ 629.762100] Modules linked in: hisi_sas_v3_hw [last unloaded: hisi_sas_v3_hw]
[ 629.776387] CPU: 9 PID: 4465 Comm: fio Tainted: G D 4.20.0-rc1-15093-ge876dec #1067
[ 629.776387] CPU: 9 PID: 4465 Comm: fio Tainted: G D 4.20.0-rc1-15093-ge876dec #1067
[ 629.794327] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 629.794327] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 - B601 (V6.01) 11/08/2018
[ 629.811919] pstate: 20400089 (nzCv daIf +PAN -UAO)
[ 629.811919] pstate: 20400089 (nzCv daIf +PAN -UAO)
[ 629.821500] pc : rcu_note_context_switch+0x38/0x2ec
[ 629.821500] pc : rcu_note_context_switch+0x38/0x2ec
[ 629.831258] lr : __schedule+0x88/0x5f8
[ 629.831258] lr : __schedule+0x88/0x5f8
Jo[bs: 12 (f=0)\r 629.838748] sp : ffff0000275a34d0
[ 629.838748] sp : ffff0000275a34d0
[ 629.846670] x29: ffff0000275a34d0 x28: ffff8027dc230d40
[ 629.846670] x29: ffff0000275a34d0 x28: ffff8027dc230d40
[ 629.857296] x27: 0000000000000000 x26: ffff000008da18e0
[ 629.857296] x27: 0000000000000000 x26: ffff000008da18e0
[ 629.867922] x25: ffff0000094f1000 x24: ffff8027dc230d40
[ 629.867922] x25: ffff0000094f1000 x24: ffff8027dc230d40
[ 629.878548] x23: ffff0000094ca000 x22: ffff0000094d6000
[ 629.878548] x23: ffff0000094ca000 x22: ffff0000094d6000
[ 629.889174] x21: ffff8027e0ab3b00 x20: ffff8027dc230d40
[ 629.889174] x21: ffff8027e0ab3b00 x20: ffff8027dc230d40
[ 629.899800] x19: ffff8027e0ab4700 x18: 0000000000000000
[ 629.899800] x19: ffff8027e0ab4700 x18: 0000000000000000
[ 629.910425] x17: 0000000000000000 x16: 0000000000000000
[ 629.910425] x17: 0000000000000000 x16: 0000000000000000
[ 629.921050] x15: 0000000000000000 x14: 0000000000000000
[ 629.921050] x15: 0000000000000000 x14: 0000000000000000
[ 629.931676] x13: 0000000000000000 x12: 0000000000000000
[ 629.931676] x13: 0000000000000000 x12: 0000000000000000
[ 629.942301] x11: 0000000000000002 x10: 0000000000000970
[ 629.942301] x11: 0000000000000002 x10: 0000000000000970
[ 629.952927] x9 : ffff0000275a3660 x8 : ffff8027dc231710
[ 629.952927] x9 : ffff0000275a3660 x8 : ffff8027dc231710
[ 629.963553] x7 : 000000927fe91094 x6 : ffff8027a0308428
[ 629.963553] x7 : 000000927fe91094 x6 : ffff8027a0308428
[ 629.974178] x5 : 0000000000000000 x4 : 0000000000000001
[ 629.974178] x5 : 0000000000000000 x4 : 0000000000000001
[ 629.984804] x3 : 674238a07d45b500 x2 : 00008027d75dd000
[ 629.984804] x3 : 674238a07d45b500 x2 : 00008027d75dd000
[ 629.995429] x1 : 00008027d75dd000 x0 : 0000000000000001
[ 629.995429] x1 : 00008027d75dd000 x0 : 0000000000000001
[ 630.006055] Call trace:
[ 630.006055] Call trace:
[ 630.010934] rcu_note_context_switch+0x38/0x2ec
[ 630.010934] rcu_note_context_switch+0x38/0x2ec
[ 630.019993] __schedule+0x88/0x5f8
[ 630.019993] __schedule+0x88/0x5f8
[ 630.026788] schedule+0x38/0x9c
[ 630.026788] schedule+0x38/0x9c
[ 630.033060] schedule_timeout+0x1b8/0x320
[ 630.033060] schedule_timeout+0x1b8/0x320
[ 630.041074] wait_for_common+0xb4/0x164
[ 630.041074] wait_for_common+0xb4/0x164
[ 630.048739] wait_for_completion+0x14/0x1c
[ 630.048739] wait_for_completion+0x14/0x1c
[ 630.056928] exit_aio+0x100/0x114
[ 630.056928] exit_aio+0x100/0x114
[ 630.063549] mmput+0x48/0x118
[ 630.063549] mmput+0x48/0x118
[ 630.069473] do_exit+0x280/0x984
[ 630.069473] do_exit+0x280/0x984
[ 630.075920] die+0x1e0/0x20c
[ 630.075920] die+0x1e0/0x20c
[ 630.081670] die_kernel_fault+0x64/0x74
[ 630.081670] die_kernel_fault+0x64/0x74
[ 6Jobs: 12 (f=0)\r30.089335] __do_kernel_fault+0x84/0xac
[ 630.089335] __do_kernel_fault+0x84/0xac
[ 630.098477] do_page_fault+0x114/0x3a8
[ 630.098477] do_page_fault+0x114/0x3a8
[ 630.105968] do_translation_fault+0x68/0x74
[ 630.105968] do_translation_fault+0x68/0x74
[ 630.114330] do_mem_abort+0x3c/0x9c
[ 630.114330] do_mem_abort+0x3c/0x9c
[ 630.121298] el1_da+0x20/0x80
[ 630.121298] el1_da+0x20/0x80
[ 630.127223] deadline_remove_request+0x2c/0xd0
[ 630.127223] deadline_remove_request+0x2c/0xd0
[ 630.136108] dd_dispatch_request+0x184/0x1e0
[ 630.136108] dd_dispatch_request+0x184/0x1e0
[ 630.144644] blk_mq_do_dispatch_sched+0x4c/0x104
[ 630.144644] blk_mq_do_dispatch_sched+0x4c/0x104
[ 630.153877] blk_mq_sched_dispatch_requests+0x114/0x18c
[ 630.153877] blk_mq_sched_dispatch_requests+0x114/0x18c
[ 630.164329] __blk_mq_run_hw_queue+0xa4/0x11c
[ 630.164329] __blk_mq_run_hw_queue+0xa4/0x11c
[ 630.173040] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 630.173040] __blk_mq_delay_run_hw_queue+0x1a4/0x208
[ 630.182969] blk_mq_run_hw_queue+0x9c/0xf8
[ 630.182969] blk_mq_run_hw_queue+0x9c/0xf8
[ 630.191157] blk_mq_sched_insert_requests+0x60/0xac
[ 630.191157] blk_mq_sched_insert_requests+0x60/0xac
[ 630.200911] blk_mq_flush_plug_list+0x144/0x16c
[ 630.200911] blk_mq_flush_plug_list+0x144/0x16c
[ 630.209970] blk_flush_plug_list+0xe8/0x244
[ 630.209970] blk_flush_plug_list+0xe8/0x244
[ 630.218331] blk_finish_plug+0x30/0x40
[ 630.218331] blk_finish_plug+0x30/0x40
[ 630.225823] __arm64_sys_io_submit+0x114/0x218
[ 630.225823] __arm64_sys_io_submit+0x114/0x218
[ 630.234708] el0_svc_common+0x94/0xf0
[ 630.234708] el0_svc_common+0x94/0xf0
[ 630.242025] el0_svc_handler+0x24/0x80
[ 630.242025] el0_svc_handler+0x24/0x80
[ 630.249516] el0_svc+0x8/0x7c0
[ 630.249516] el0_svc+0x8/0x7c0
[ 630.255613] ---[ end trace 22423ac28b61ac52 ]---
[ 630.255613] ---[ end trace 22423ac28b61ac52 ]---
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: DIF/DIX issue related to config CONFIG_SCSI_MQ_DEFAULT
2018-11-27 9:55 DIF/DIX issue related to config CONFIG_SCSI_MQ_DEFAULT chenxiang (M)
@ 2018-11-27 13:08 ` Ming Lei
2018-11-28 3:37 ` chenxiang (M)
[not found] ` <6c573f36-60d8-0631-e9ac-dacd72f6c8ad@hisilicon.com>
2018-11-27 20:22 ` Ewan D. Milne
1 sibling, 2 replies; 16+ messages in thread
From: Ming Lei @ 2018-11-27 13:08 UTC (permalink / raw)
To: chenxiang (M)
Cc: James E.J. Bottomley, Martin K. Petersen, linux-scsi,
linux-block, John Garry, Linuxarm
On Tue, Nov 27, 2018 at 05:55:45PM +0800, chenxiang (M) wrote:
> Hi all,
>
> There is a issue which may be related to CONFIG_SCSI_MQ_DEFAULT: before we
> developed DIF/DIX feature on kernel 4.18 (disable CONFIG_SCSI_MQ_DEFAULT
> default), and
> it works well.
I guess you are testing hisi_sas_v3_hw, does 4.18 work with
'scsi_mod.use_blk_mq=Y'? If yes, you may run 'git bisect' to figure out
which commit is the 1st bad one.
> But when we switch to kernel 4.19-rc1 and 4.20-rc1, Call
> trace as follow occurs when running fio and if disable config
> CONFIG_SCSI_MQ_DEFAULT,
> then it works well. Also if switch ioengine=libaio to ioengine=psync, it
> seems also work well. Do you have any idea or encounter similar issue?
I tested scsi-debug via 'dix=1 dif=1', looks everything is fine, are you
using direct io or not?
>
> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> fio 2.0.5
> Starting 12 processes
> [ 629.210506] Unable to handle kernel paging request at virtual address
> 0000ffff8027e048
> [ 629.210506] Unable to handle kernel paging request at virtual address
> 0000ffff8027e048
> [ 629.226373] Mem abort info:
> [ 629.226373] Mem abort info:
> [ 629.231952] ESR = 0x96000006
> [ 629.231952] ESR = 0x96000006
> [ 629.238052] Exception class = DABT (current EL), IL = 32 bits
> [ 629.238052] Exception class = DABT (current EL), IL = 32 bits
> [ 629.249898] SET = 0, FnV = 0
> [ 629.249898] SET = 0, FnV = 0
> [ 629.255998] EA = 0, S1PTW = 0
> [ 629.255998] EA = 0, S1PTW = 0
> [ 629.262272] Data abort info:
> [ 629.262272] Data abort info:
> [ 629.268023] ISV = 0, ISS = 0x00000006
> [ 629.268023] ISV = 0, ISS = 0x00000006
> [ 629.275690] CM = 0, WnR = 0
> [ 629.275690] CM = 0, WnR = 0
> [ 629.281617] user pgtable: 4k pages, 48-bit VAs, pgdp = 0000000085c91728
> [ 629.281617] user pgtable: 4k pages, 48-bit VAs, pgdp = 0000000085c91728
> [ 629.294857] [0000ffff8027e048] pgd=00000027a8644003,
> pud=00000027a85ea003, pmd=0000000000000000
> [ 629.294857] [0000ffff8027e048] pgd=00000027a8644003,
> pud=00000027a85ea003, pmd=0000000000000000
> [ 629.312278] Internal error: Oops: 96000006 [#1] PREEMPT SMP
> [ 629.312278] Internal error: Oops: 96000006 [#1] PREEMPT SMP
> [ 629.323427] Modules linked in: hisi_sas_v3_hw [last unloaded:
> hisi_sas_v3_hw]
> [ 629.323427] Modules linked in: hisi_sas_v3_hw [last unloaded:
> hisi_sas_v3_hw]
> [ 629.337713] CPU: 13 PID: 4465 Comm: fio Not tainted
> 4.20.0-rc1-15093-ge876dec #1067
> [ 629.337713] CPU: 13 PID: 4465 Comm: fio Not tainted
> 4.20.0-rc1-15093-ge876dec #1067
> [ 629.353040] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 -
> B601 (V6.01) 11/08/2018
> [ 629.353040] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 -
> B601 (V6.01) 11/08/2018
> [ 629.370633] pstate: 80400009 (Nzcv daif +PAN -UAO)
> [ 629.370633] pstate: 80400009 (Nzcv daif +PAN -UAO)
> [ 629.380218] pc : deadline_remove_request+0x2c/0xd0
> [ 629.380218] pc : deadline_remove_request+0x2c/0xd0
Could you use gdb to find where 'deadline_remove_request+0x2c' points
to?
Thanks,
Ming
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: DIF/DIX issue related to config CONFIG_SCSI_MQ_DEFAULT
2018-11-27 9:55 DIF/DIX issue related to config CONFIG_SCSI_MQ_DEFAULT chenxiang (M)
2018-11-27 13:08 ` Ming Lei
@ 2018-11-27 20:22 ` Ewan D. Milne
2018-11-28 3:11 ` chenxiang (M)
1 sibling, 1 reply; 16+ messages in thread
From: Ewan D. Milne @ 2018-11-27 20:22 UTC (permalink / raw)
To: chenxiang (M), James E.J. Bottomley, Martin K. Petersen, Ming Lei
Cc: linux-scsi, linux-block, John Garry, Linuxarm
On Tue, 2018-11-27 at 17:55 +0800, chenxiang (M) wrote:
> [ 629.210506] Unable to handle kernel paging request at virtual address
> 0000ffff8027e048
> [ 629.210506] Unable to handle kernel paging request at virtual address
> 0000ffff8027e048
> ...
> [ 629.380218] pc : deadline_remove_request+0x2c/0xd0
> [ 629.380218] pc : deadline_remove_request+0x2c/0xd0
> [ 629.389799] lr : dd_dispatch_request+0x184/0x1e0
> [ 629.389799] lr : dd_dispatch_request+0x184/0x1e0
> [ 629.399031] sp : ffff0000275a3aa0
> [ 629.399031] sp : ffff0000275a3aa0
> [ 629.405651] x29: ffff0000275a3aa0 x28: ffff8027dc230d40
> [ 629.405651] x29: ffff0000275a3aa0 x28: ffff8027dc230d40
> [ 629.416278] x27: 0000000000000001 x26: ffff8027dc230d40
> [ 629.416278] x27: 0000000000000001 x26: ffff8027dc230d40
> [ 629.426903] x25: 0000000000000000 x24: ffff0000094f1000
> [ 629.426903] x25: 0000000000000000 x24: ffff0000094f1000
> [ 629.437529] x23: ffff8027a1e1d140 x22: 0000000000000006
> [ 629.437529] x23: ffff8027a1e1d140 x22: 0000000000000006
> [ 629.448155] x21: ffff7e009ec69842 x20: ffff8027a1e1d100
> [ 629.448155] x21: ffff7e009ec69842 x20: ffff8027a1e1d100
> [ 629.458781] x19: ffff8027a1e1d100 x18: 0000000000000000
> [ 629.458781] x19: ffff8027a1e1d100 x18: 0000000000000000
> [ 629.469406] x17: 000000009e8c9c80 x16: 00000027a3272000
> [ 629.469406] x17: 000000009e8c9c80 x16: 00000027a3272000
> [ 629.480032] x15: ffff7e009fff9534 x14: ffff7e009ec698c0
> [ 629.480032] x15: ffff7e009fff9534 x14: ffff7e009ec698c0
> [ 629.490658] x13: ffff7e009ec69880 x12: 0000000000000009
> [ 629.490658] x13: ffff7e009ec69880 x12: 0000000000000009
> [ 629.501284] x11: 0000000000000000 x10: ffff8027a7c03560
> [ 629.501284] x11: 0000000000000000 x10: ffff8027a7c03560
> [ 629.511910] x9 : ffff8027a7c007b8 x8 : 0000000000000007
> [ 629.511910] x9 : ffff8027a7c007b8 x8 : 0000000000000007
> [ 629.522535] x7 : ffff8027a7c011c8 x6 : 0000000000000000
> [ 629.522535] x7 : ffff8027a7c011c8 x6 : 0000000000000000
> [ 629.533161] x5 : 0000000000000000 x4 : 0000ffff8027e040
> [ 629.533161] x5 : 0000000000000000 x4 : 0000ffff8027e040
> [ 629.543787] x3 : ffff8027a1e20040 x2 : ffff8027da87f6a0
> [ 629.543787] x3 : ffff8027a1e20040 x2 : ffff8027da87f6a0
> [ 629.554413] x1 : ffff8027a1e1d140 x0 : ffff8027a1e1d160
> [ 629.554413] x1 : ffff8027a1e1d140 x0 : ffff8027a1e1d160
I don't have an ARM architecture machine, but the faulting
address looks like it may have been loaded from a corrupted
pointer value (0000ffff8027e048 instead of ffff8027e048----)
The code in question uses some list primitives, were there any
warning messages about list corruption prior to the crash?
-Ewan
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: DIF/DIX issue related to config CONFIG_SCSI_MQ_DEFAULT
2018-11-27 20:22 ` Ewan D. Milne
@ 2018-11-28 3:11 ` chenxiang (M)
0 siblings, 0 replies; 16+ messages in thread
From: chenxiang (M) @ 2018-11-28 3:11 UTC (permalink / raw)
To: Ewan D. Milne, James E.J. Bottomley, Martin K. Petersen, Ming Lei
Cc: linux-scsi, linux-block, John Garry, Linuxarm
Hi,
在 2018/11/28 4:22, Ewan D. Milne 写道:
> On Tue, 2018-11-27 at 17:55 +0800, chenxiang (M) wrote:
>> [ 629.210506] Unable to handle kernel paging request at virtual address
>> 0000ffff8027e048
>> [ 629.210506] Unable to handle kernel paging request at virtual address
>> 0000ffff8027e048
>> ...
>> [ 629.380218] pc : deadline_remove_request+0x2c/0xd0
>> [ 629.380218] pc : deadline_remove_request+0x2c/0xd0
>> [ 629.389799] lr : dd_dispatch_request+0x184/0x1e0
>> [ 629.389799] lr : dd_dispatch_request+0x184/0x1e0
>> [ 629.399031] sp : ffff0000275a3aa0
>> [ 629.399031] sp : ffff0000275a3aa0
>> [ 629.405651] x29: ffff0000275a3aa0 x28: ffff8027dc230d40
>> [ 629.405651] x29: ffff0000275a3aa0 x28: ffff8027dc230d40
>> [ 629.416278] x27: 0000000000000001 x26: ffff8027dc230d40
>> [ 629.416278] x27: 0000000000000001 x26: ffff8027dc230d40
>> [ 629.426903] x25: 0000000000000000 x24: ffff0000094f1000
>> [ 629.426903] x25: 0000000000000000 x24: ffff0000094f1000
>> [ 629.437529] x23: ffff8027a1e1d140 x22: 0000000000000006
>> [ 629.437529] x23: ffff8027a1e1d140 x22: 0000000000000006
>> [ 629.448155] x21: ffff7e009ec69842 x20: ffff8027a1e1d100
>> [ 629.448155] x21: ffff7e009ec69842 x20: ffff8027a1e1d100
>> [ 629.458781] x19: ffff8027a1e1d100 x18: 0000000000000000
>> [ 629.458781] x19: ffff8027a1e1d100 x18: 0000000000000000
>> [ 629.469406] x17: 000000009e8c9c80 x16: 00000027a3272000
>> [ 629.469406] x17: 000000009e8c9c80 x16: 00000027a3272000
>> [ 629.480032] x15: ffff7e009fff9534 x14: ffff7e009ec698c0
>> [ 629.480032] x15: ffff7e009fff9534 x14: ffff7e009ec698c0
>> [ 629.490658] x13: ffff7e009ec69880 x12: 0000000000000009
>> [ 629.490658] x13: ffff7e009ec69880 x12: 0000000000000009
>> [ 629.501284] x11: 0000000000000000 x10: ffff8027a7c03560
>> [ 629.501284] x11: 0000000000000000 x10: ffff8027a7c03560
>> [ 629.511910] x9 : ffff8027a7c007b8 x8 : 0000000000000007
>> [ 629.511910] x9 : ffff8027a7c007b8 x8 : 0000000000000007
>> [ 629.522535] x7 : ffff8027a7c011c8 x6 : 0000000000000000
>> [ 629.522535] x7 : ffff8027a7c011c8 x6 : 0000000000000000
>> [ 629.533161] x5 : 0000000000000000 x4 : 0000ffff8027e040
>> [ 629.533161] x5 : 0000000000000000 x4 : 0000ffff8027e040
>> [ 629.543787] x3 : ffff8027a1e20040 x2 : ffff8027da87f6a0
>> [ 629.543787] x3 : ffff8027a1e20040 x2 : ffff8027da87f6a0
>> [ 629.554413] x1 : ffff8027a1e1d140 x0 : ffff8027a1e1d160
>> [ 629.554413] x1 : ffff8027a1e1d140 x0 : ffff8027a1e1d160
> I don't have an ARM architecture machine, but the faulting
> address looks like it may have been loaded from a corrupted
> pointer value (0000ffff8027e048 instead of ffff8027e048----)
>
> The code in question uses some list primitives, were there any
> warning messages about list corruption prior to the crash?
There is no warning message abort list corruption or other corruption
prior to the crash.
>
> -Ewan
>
>
> .
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: DIF/DIX issue related to config CONFIG_SCSI_MQ_DEFAULT
2018-11-27 13:08 ` Ming Lei
@ 2018-11-28 3:37 ` chenxiang (M)
2018-11-29 18:17 ` Ming Lei
[not found] ` <6c573f36-60d8-0631-e9ac-dacd72f6c8ad@hisilicon.com>
1 sibling, 1 reply; 16+ messages in thread
From: chenxiang (M) @ 2018-11-28 3:37 UTC (permalink / raw)
To: Ming Lei
Cc: James E.J. Bottomley, Martin K. Petersen, linux-scsi,
linux-block, John Garry, Linuxarm
Hi Lei Ming,
在 2018/11/27 21:08, Ming Lei 写道:
> On Tue, Nov 27, 2018 at 05:55:45PM +0800, chenxiang (M) wrote:
>> Hi all,
>>
>> There is a issue which may be related to CONFIG_SCSI_MQ_DEFAULT: before we
>> developed DIF/DIX feature on kernel 4.18 (disable CONFIG_SCSI_MQ_DEFAULT
>> default), and
>> it works well.
> I guess you are testing hisi_sas_v3_hw, does 4.18 work with
> 'scsi_mod.use_blk_mq=Y'? If yes, you may run 'git bisect' to figure out
> which commit is the 1st bad one.
>
>> But when we switch to kernel 4.19-rc1 and 4.20-rc1, Call
>> trace as follow occurs when running fio and if disable config
>> CONFIG_SCSI_MQ_DEFAULT,
>> then it works well. Also if switch ioengine=libaio to ioengine=psync, it
>> seems also work well. Do you have any idea or encounter similar issue?
> I tested scsi-debug via 'dix=1 dif=1', looks everything is fine, are you
> using direct io or not?
>
>> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
>> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
>> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
>> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
>> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
>> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
>> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
>> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
>> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
>> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
>> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
>> job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
>> fio 2.0.5
>> Starting 12 processes
>> [ 629.210506] Unable to handle kernel paging request at virtual address
>> 0000ffff8027e048
>> [ 629.210506] Unable to handle kernel paging request at virtual address
>> 0000ffff8027e048
>> [ 629.226373] Mem abort info:
>> [ 629.226373] Mem abort info:
>> [ 629.231952] ESR = 0x96000006
>> [ 629.231952] ESR = 0x96000006
>> [ 629.238052] Exception class = DABT (current EL), IL = 32 bits
>> [ 629.238052] Exception class = DABT (current EL), IL = 32 bits
>> [ 629.249898] SET = 0, FnV = 0
>> [ 629.249898] SET = 0, FnV = 0
>> [ 629.255998] EA = 0, S1PTW = 0
>> [ 629.255998] EA = 0, S1PTW = 0
>> [ 629.262272] Data abort info:
>> [ 629.262272] Data abort info:
>> [ 629.268023] ISV = 0, ISS = 0x00000006
>> [ 629.268023] ISV = 0, ISS = 0x00000006
>> [ 629.275690] CM = 0, WnR = 0
>> [ 629.275690] CM = 0, WnR = 0
>> [ 629.281617] user pgtable: 4k pages, 48-bit VAs, pgdp = 0000000085c91728
>> [ 629.281617] user pgtable: 4k pages, 48-bit VAs, pgdp = 0000000085c91728
>> [ 629.294857] [0000ffff8027e048] pgd=00000027a8644003,
>> pud=00000027a85ea003, pmd=0000000000000000
>> [ 629.294857] [0000ffff8027e048] pgd=00000027a8644003,
>> pud=00000027a85ea003, pmd=0000000000000000
>> [ 629.312278] Internal error: Oops: 96000006 [#1] PREEMPT SMP
>> [ 629.312278] Internal error: Oops: 96000006 [#1] PREEMPT SMP
>> [ 629.323427] Modules linked in: hisi_sas_v3_hw [last unloaded:
>> hisi_sas_v3_hw]
>> [ 629.323427] Modules linked in: hisi_sas_v3_hw [last unloaded:
>> hisi_sas_v3_hw]
>> [ 629.337713] CPU: 13 PID: 4465 Comm: fio Not tainted
>> 4.20.0-rc1-15093-ge876dec #1067
>> [ 629.337713] CPU: 13 PID: 4465 Comm: fio Not tainted
>> 4.20.0-rc1-15093-ge876dec #1067
>> [ 629.353040] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 -
>> B601 (V6.01) 11/08/2018
>> [ 629.353040] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 -
>> B601 (V6.01) 11/08/2018
>> [ 629.370633] pstate: 80400009 (Nzcv daif +PAN -UAO)
>> [ 629.370633] pstate: 80400009 (Nzcv daif +PAN -UAO)
>> [ 629.380218] pc : deadline_remove_request+0x2c/0xd0
>> [ 629.380218] pc : deadline_remove_request+0x2c/0xd0
> Could you use gdb to find where 'deadline_remove_request+0x2c' points
> to?
From objdump, 'deadline_remove_request+0x2c' is on the function
__list_del -> INIT_LIST_HEAD.
>
> Thanks,
> Ming
>
> .
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: DIF/DIX issue related to config CONFIG_SCSI_MQ_DEFAULT
[not found] ` <6c573f36-60d8-0631-e9ac-dacd72f6c8ad@hisilicon.com>
@ 2018-11-29 0:54 ` Ming Lei
2018-11-30 1:19 ` Ming Lei
1 sibling, 0 replies; 16+ messages in thread
From: Ming Lei @ 2018-11-29 0:54 UTC (permalink / raw)
To: chenxiang (M)
Cc: James E.J. Bottomley, Martin K. Petersen, linux-scsi,
linux-block, John Garry, Linuxarm
On Wed, Nov 28, 2018 at 10:50:11AM +0800, chenxiang (M) wrote:
> Hi Lei Ming,
>
> 在 2018/11/27 21:08, Ming Lei 写道:
> > On Tue, Nov 27, 2018 at 05:55:45PM +0800, chenxiang (M) wrote:
> > > Hi all,
> > >
> > > There is a issue which may be related to CONFIG_SCSI_MQ_DEFAULT: before we
> > > developed DIF/DIX feature on kernel 4.18 (disable CONFIG_SCSI_MQ_DEFAULT
> > > default), and
> > > it works well.
> > I guess you are testing hisi_sas_v3_hw, does 4.18 work with
> > 'scsi_mod.use_blk_mq=Y'? If yes, you may run 'git bisect' to figure out
> > which commit is the 1st bad one.
>
> Yes, i am testing hisi_sas_v3_hw, if 4.18 with 'scsi_mod.use_blk_mq=Y' ,
> there is still a issue, and
> the log is as attached ;
OK, looks not a regression.
>
> >
> > > But when we switch to kernel 4.19-rc1 and 4.20-rc1, Call
> > > trace as follow occurs when running fio and if disable config
> > > CONFIG_SCSI_MQ_DEFAULT,
> > > then it works well. Also if switch ioengine=libaio to ioengine=psync, it
> > > seems also work well. Do you have any idea or encounter similar issue?
> > I tested scsi-debug via 'dix=1 dif=1', looks everything is fine, are you
> > using direct io or not?
> Yes, i use direct io.
> I also test scsi-debug on kernel 4.19, and there seems be some other issue,
> and i am not sure
> whethere there is something wrong my operation:
>
> root@(none)$ insmod scsi_debug.ko dif=1 dix=1 virtual_gb=120
> [ 319.255051] scsi_debug:scsi_debug_init: dif_storep 131072 bytes @
> 00000000ba5f1637
> [ 319.255051] scsi_debug:scsi_debug_init: dif_storep 131072 bytes @
> 00000000ba5f1637
> [ 319.270661] scsi_debug:sdebug_driver_probe: host protection DIF1 DIX1
> [ 319.270661] scsi_debug:sdebug_driver_probe: host protection DIF1 DIX1
> [ 319.283562] scsi host3: scsi_debug: version 0188 [20180128]
> [ 319.283562] dev_size_mb=8, opts=0x0, submit_queues=1, statistics=0
> [ 319.283562] scsi host3: scsi_debug: version 0188 [20180128]
> [ 319.283562] dev_size_mb=8, opts=0x0, submit_queues=1, statistics=0
> [ 319.307871] scsi 3:0:0:0: Direct-Access Linux scsi_debug 0188
> PQ: 0 ANSI: 7
> [ 319.307871] scsi 3:0:0:0: Direct-Access Linux scsi_debug 0188
> PQ: 0 ANSI: 7
> [ 319.324173] sd 3:0:0:0: Power-on or device reset occurred
> [ 319.324173] sd 3:0:0:0: Power-on or device reset occurred
> [ 319.324242] sd 3:0:0:0: Attached scsi generic sg0 type 0
> [ 319.324242] sd 3:0:0:0: Attached scsi generic sg0 type 0
> [ 319.343021] sd 3:0:0:0: [sda] Enabling DIF Type 1 protection
> [ 319.343021] sd 3:0:0:0: [sda] Enabling DIF Type 1 protection
> [ 319.356952] sd 3:0:0:0: [sda] 251658240 512-byte logical blocks: (129
> GB/120 GiB)
> [ 319.356952] sd 3:0:0:0: [sda] 251658240 512-byte logical blocks: (129
> GB/120 GiB)
> [ 319.375948] sd 3:0:0:0: [sda] Write Protect is off
> [ 319.375948] sd 3:0:0:0: [sda] Write Protect is off
> [ 319.393551] sd 3:0:0:0: [sda] Write cache: enabled, read cache: enabled,
> supports DPO and FUA
> [ 319.393551] sd 3:0:0:0: [sda] Write cache: enabled, read cache: enabled,
> supports DPO and FUA
> [ 319.467940] sd 3:0:0:0: [sda] Enabling DIX T10-DIF-TYPE1-CRC protection
> [ 319.467940] sd 3:0:0:0: [sda] Enabling DIX T10-DIF-TYPE1-CRC protection
> [ 319.481189] sd 3:0:0:0: [sda] DIF application tag size 2
> [ 319.481189] sd 3:0:0:0: [sda] DIF application tag size 2
> [ 319.527902] sd 3:0:0:0: [sda] Attached SCSI disk
> [ 319.527902] sd 3:0:0:0: [sda] Attached SCSI disk
> root@(none)$
> root@(none)$
> root@(none)$
> root@(none)$
> root@(none)$ fio --ioengine=libaio --bs=4k --rw=rw --filename=/dev/sd a
> --size=10G --runtime=100 --direct=1 -numjobs=1 -iodepth=1 --time_based
> --nam e=job
> job: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=1
> fio 2.0.5
> Starting 1 process
> Jobs: 1 (f=1)
> Jobs: 1 (f=1)
> Jobs: 1 (f=1): [M] [4.0% done] [495K/520K /s] [121 /127 iops] [eta 01m:37s]
> Jobs: 1 (f=1): [M] [5.0% done] [507K/507K /s] [124 /124 iops] [eta 01m:36s]
> Jobs: 1 (f=1): [M] [5.9% done] [516K/499K /s] [126 /122 iops] [eta 01m:35s]
> Jobs: 1 (f=1): [M] [6.9% done] [479K/536K /s] [117 /131 iops] [eta 01m:34s]
> Jobs: 1 (f=1): [M] [7.9% done] [495K/520K /s] [121 /127 iops] [eta 01m:33s]
> Jobs: 1 (f=1): [M] [8.9% done] [499K/516K /s] [122 /126 iops] [eta 01m:32s]
> [ 353.298158] scsi_debug:dif_verify: REF check failed on sector 16456
> [ 353.298158] scsi_debug:dif_verify: REF check failed on sector 16456
That is the issue, I guess it is one ARM64 dependent issue. I will try
to find a arm64 machine to debug it, hope I am lucky. Or you may try to
investigate it and see why the REF check fails.
thanks,
Ming
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: DIF/DIX issue related to config CONFIG_SCSI_MQ_DEFAULT
2018-11-28 3:37 ` chenxiang (M)
@ 2018-11-29 18:17 ` Ming Lei
0 siblings, 0 replies; 16+ messages in thread
From: Ming Lei @ 2018-11-29 18:17 UTC (permalink / raw)
To: chenxiang (M)
Cc: James E.J. Bottomley, Martin K. Petersen, linux-scsi,
linux-block, John Garry, Linuxarm
On Wed, Nov 28, 2018 at 11:37:23AM +0800, chenxiang (M) wrote:
> Hi Lei Ming,
>
> 在 2018/11/27 21:08, Ming Lei 写道:
> > On Tue, Nov 27, 2018 at 05:55:45PM +0800, chenxiang (M) wrote:
> > > Hi all,
> > >
> > > There is a issue which may be related to CONFIG_SCSI_MQ_DEFAULT: before we
> > > developed DIF/DIX feature on kernel 4.18 (disable CONFIG_SCSI_MQ_DEFAULT
> > > default), and
> > > it works well.
> > I guess you are testing hisi_sas_v3_hw, does 4.18 work with
> > 'scsi_mod.use_blk_mq=Y'? If yes, you may run 'git bisect' to figure out
> > which commit is the 1st bad one.
> >
> > > But when we switch to kernel 4.19-rc1 and 4.20-rc1, Call
> > > trace as follow occurs when running fio and if disable config
> > > CONFIG_SCSI_MQ_DEFAULT,
> > > then it works well. Also if switch ioengine=libaio to ioengine=psync, it
> > > seems also work well. Do you have any idea or encounter similar issue?
> > I tested scsi-debug via 'dix=1 dif=1', looks everything is fine, are you
> > using direct io or not?
> >
> > > job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> > > job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> > > job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> > > job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> > > job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> > > job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> > > job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> > > job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> > > job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> > > job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> > > job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> > > job1: (g=0): rw=rw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=128
> > > fio 2.0.5
> > > Starting 12 processes
> > > [ 629.210506] Unable to handle kernel paging request at virtual address
> > > 0000ffff8027e048
> > > [ 629.210506] Unable to handle kernel paging request at virtual address
> > > 0000ffff8027e048
> > > [ 629.226373] Mem abort info:
> > > [ 629.226373] Mem abort info:
> > > [ 629.231952] ESR = 0x96000006
> > > [ 629.231952] ESR = 0x96000006
> > > [ 629.238052] Exception class = DABT (current EL), IL = 32 bits
> > > [ 629.238052] Exception class = DABT (current EL), IL = 32 bits
> > > [ 629.249898] SET = 0, FnV = 0
> > > [ 629.249898] SET = 0, FnV = 0
> > > [ 629.255998] EA = 0, S1PTW = 0
> > > [ 629.255998] EA = 0, S1PTW = 0
> > > [ 629.262272] Data abort info:
> > > [ 629.262272] Data abort info:
> > > [ 629.268023] ISV = 0, ISS = 0x00000006
> > > [ 629.268023] ISV = 0, ISS = 0x00000006
> > > [ 629.275690] CM = 0, WnR = 0
> > > [ 629.275690] CM = 0, WnR = 0
> > > [ 629.281617] user pgtable: 4k pages, 48-bit VAs, pgdp = 0000000085c91728
> > > [ 629.281617] user pgtable: 4k pages, 48-bit VAs, pgdp = 0000000085c91728
> > > [ 629.294857] [0000ffff8027e048] pgd=00000027a8644003,
> > > pud=00000027a85ea003, pmd=0000000000000000
> > > [ 629.294857] [0000ffff8027e048] pgd=00000027a8644003,
> > > pud=00000027a85ea003, pmd=0000000000000000
> > > [ 629.312278] Internal error: Oops: 96000006 [#1] PREEMPT SMP
> > > [ 629.312278] Internal error: Oops: 96000006 [#1] PREEMPT SMP
> > > [ 629.323427] Modules linked in: hisi_sas_v3_hw [last unloaded:
> > > hisi_sas_v3_hw]
> > > [ 629.323427] Modules linked in: hisi_sas_v3_hw [last unloaded:
> > > hisi_sas_v3_hw]
> > > [ 629.337713] CPU: 13 PID: 4465 Comm: fio Not tainted
> > > 4.20.0-rc1-15093-ge876dec #1067
> > > [ 629.337713] CPU: 13 PID: 4465 Comm: fio Not tainted
> > > 4.20.0-rc1-15093-ge876dec #1067
> > > [ 629.353040] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 -
> > > B601 (V6.01) 11/08/2018
> > > [ 629.353040] Hardware name: Huawei D06/D06, BIOS Hisilicon D06 UEFI RC0 -
> > > B601 (V6.01) 11/08/2018
> > > [ 629.370633] pstate: 80400009 (Nzcv daif +PAN -UAO)
> > > [ 629.370633] pstate: 80400009 (Nzcv daif +PAN -UAO)
> > > [ 629.380218] pc : deadline_remove_request+0x2c/0xd0
> > > [ 629.380218] pc : deadline_remove_request+0x2c/0xd0
> > Could you use gdb to find where 'deadline_remove_request+0x2c' points
> > to?
>
> From objdump, 'deadline_remove_request+0x2c' is on the function __list_del
> -> INIT_LIST_HEAD.
You may enable 'Kernel hacking/Debug linked list manipulation' config
option and see what the dumped log is.
Also it might be related with the following recent report too:
https://marc.info/?l=linux-scsi&m=154283686812846&w=2
Thanks,
Ming
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: DIF/DIX issue related to config CONFIG_SCSI_MQ_DEFAULT
[not found] ` <6c573f36-60d8-0631-e9ac-dacd72f6c8ad@hisilicon.com>
2018-11-29 0:54 ` Ming Lei
@ 2018-11-30 1:19 ` Ming Lei
2018-11-30 11:26 ` John Garry
2018-12-05 2:56 ` Martin K. Petersen
1 sibling, 2 replies; 16+ messages in thread
From: Ming Lei @ 2018-11-30 1:19 UTC (permalink / raw)
To: chenxiang (M)
Cc: James E.J. Bottomley, Martin K. Petersen, linux-scsi,
linux-block, John Garry, Linuxarm
On Wed, Nov 28, 2018 at 10:50:11AM +0800, chenxiang (M) wrote:
> Hi Lei Ming,
>
> 在 2018/11/27 21:08, Ming Lei 写道:
> > On Tue, Nov 27, 2018 at 05:55:45PM +0800, chenxiang (M) wrote:
> > > Hi all,
> > >
> > > There is a issue which may be related to CONFIG_SCSI_MQ_DEFAULT: before we
> > > developed DIF/DIX feature on kernel 4.18 (disable CONFIG_SCSI_MQ_DEFAULT
> > > default), and
> > > it works well.
> > I guess you are testing hisi_sas_v3_hw, does 4.18 work with
> > 'scsi_mod.use_blk_mq=Y'? If yes, you may run 'git bisect' to figure out
> > which commit is the 1st bad one.
>
> Yes, i am testing hisi_sas_v3_hw, if 4.18 with 'scsi_mod.use_blk_mq=Y' ,
> there is still a issue, and
> the log is as attached ;
>
> >
> > > But when we switch to kernel 4.19-rc1 and 4.20-rc1, Call
> > > trace as follow occurs when running fio and if disable config
> > > CONFIG_SCSI_MQ_DEFAULT,
> > > then it works well. Also if switch ioengine=libaio to ioengine=psync, it
> > > seems also work well. Do you have any idea or encounter similar issue?
> > I tested scsi-debug via 'dix=1 dif=1', looks everything is fine, are you
> > using direct io or not?
> Yes, i use direct io.
> I also test scsi-debug on kernel 4.19, and there seems be some other issue,
> and i am not sure
> whethere there is something wrong my operation:
>
> root@(none)$ insmod scsi_debug.ko dif=1 dix=1 virtual_gb=120
Hi chenxiang,
For DIX/DIF test, I think you need to pass 'dev_size_mb=XXX' instead of
'virtual_gb'.
I just run FIO over 'scsi_debug dev_size_mb=896 max_luns=1 dif=1 dix=1'
on one apm-mustang-ev3 board, and looks everything is fine. You may run
the same test on your D06 and see if it is fine.
Please check if you may reproduce the list corruption issue on v4.18 with
hisi_sas_v3_hw.
Thanks,
Ming
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: DIF/DIX issue related to config CONFIG_SCSI_MQ_DEFAULT
2018-11-30 1:19 ` Ming Lei
@ 2018-11-30 11:26 ` John Garry
2018-12-04 3:55 ` Martin K. Petersen
2018-12-05 2:56 ` Martin K. Petersen
1 sibling, 1 reply; 16+ messages in thread
From: John Garry @ 2018-11-30 11:26 UTC (permalink / raw)
To: Ming Lei, chenxiang (M)
Cc: James E.J. Bottomley, Martin K. Petersen, linux-scsi,
linux-block, Linuxarm, Steffen Maier
+
On 30/11/2018 01:19, Ming Lei wrote:
> On Wed, Nov 28, 2018 at 10:50:11AM +0800, chenxiang (M) wrote:
>> Hi Lei Ming,
>>
>> 在 2018/11/27 21:08, Ming Lei 写道:
>>> On Tue, Nov 27, 2018 at 05:55:45PM +0800, chenxiang (M) wrote:
>>>> Hi all,
>>>>
>>>> There is a issue which may be related to CONFIG_SCSI_MQ_DEFAULT: before we
>>>> developed DIF/DIX feature on kernel 4.18 (disable CONFIG_SCSI_MQ_DEFAULT
>>>> default), and
>>>> it works well.
>>> I guess you are testing hisi_sas_v3_hw, does 4.18 work with
>>> 'scsi_mod.use_blk_mq=Y'? If yes, you may run 'git bisect' to figure out
>>> which commit is the 1st bad one.
>>
>> Yes, i am testing hisi_sas_v3_hw, if 4.18 with 'scsi_mod.use_blk_mq=Y' ,
>> there is still a issue, and
>> the log is as attached ;
>>
>>>
>>>> But when we switch to kernel 4.19-rc1 and 4.20-rc1, Call
>>>> trace as follow occurs when running fio and if disable config
>>>> CONFIG_SCSI_MQ_DEFAULT,
>>>> then it works well. Also if switch ioengine=libaio to ioengine=psync, it
>>>> seems also work well. Do you have any idea or encounter similar issue?
>>> I tested scsi-debug via 'dix=1 dif=1', looks everything is fine, are you
>>> using direct io or not?
>> Yes, i use direct io.
>> I also test scsi-debug on kernel 4.19, and there seems be some other issue,
>> and i am not sure
>> whethere there is something wrong my operation:
>>
>> root@(none)$ insmod scsi_debug.ko dif=1 dix=1 virtual_gb=120
>
> Hi chenxiang,
>
> For DIX/DIF test, I think you need to pass 'dev_size_mb=XXX' instead of
> 'virtual_gb'.
>
> I just run FIO over 'scsi_debug dev_size_mb=896 max_luns=1 dif=1 dix=1'
> on one apm-mustang-ev3 board, and looks everything is fine. You may run
> the same test on your D06 and see if it is fine.
>
> Please check if you may reproduce the list corruption issue on v4.18 with
> hisi_sas_v3_hw.
>
Hi Ming,
Thanks for your support.
We have also noticed that if we just enable DIF in hisi_sas (with MQ),
and not DIX, then no issue.
I did also noticed mail "[PATCH v2 01/23] zfcp: make DIX experimental,
disabled, and independent of DIF", where DIX is made experimental.
For now we may not support DIX. It seems to have issues. We wanted to
try 3008 card on our system, but it does not seem to support DIX 0-3.
BTW, chen xiang said your commnand works ok on our HW. He can confirm.
Thanks again,
John
> Thanks,
> Ming
>
> .
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: DIF/DIX issue related to config CONFIG_SCSI_MQ_DEFAULT
2018-11-30 11:26 ` John Garry
@ 2018-12-04 3:55 ` Martin K. Petersen
[not found] ` <45193ec6-6398-3953-4833-88ca2057971a@huawei.com>
0 siblings, 1 reply; 16+ messages in thread
From: Martin K. Petersen @ 2018-12-04 3:55 UTC (permalink / raw)
To: John Garry
Cc: Ming Lei, chenxiang (M),
James E.J. Bottomley, Martin K. Petersen, linux-scsi,
linux-block, Linuxarm, Steffen Maier
Hi John,
> We have also noticed that if we just enable DIF in hisi_sas (with MQ),
> and not DIX, then no issue.
Enabling DIF doesn't really do anything on the kernel side other than
setting PROTECT=1 in the READ/WRITE CDB and telling the driver which DIX
protection operation the HBA should use. Since protection information is
invisible to the kernel and only sent on the wire between initiator and
target, enabling DIF doesn't really have the ability to interfere with
anything on the kernel side. We're basically just setting flags asking
HBA and storage to enable protected transfers.
> I did also noticed mail "[PATCH v2 01/23] zfcp: make DIX experimental,
> disabled, and independent of DIF", where DIX is made experimental.
...for the zfcp driver on zSeries.
Just nitpicking on terminology here:
T10 Protection Information (formerly known as DIF) describes how to
generate and verify 8 bytes of extra information that's sent trailing
each logical block on the wire between an initiator and target. The T10
PI spec is focused on the target device implementation of this and
largely ignores the initiator side.
DIX tries to remedy this deficiency. It is a spec that describes a set
of logical operations an initiator must implement to facilitate sending
and receiving the T10 protection information to/from host memory instead
of terminating it at the HBA. The DIX spec isn't experimental, it's
about a decade old and hasn't changed in years.
The Linux kernel support for data integrity passthrough in the block
layer and SCSI isn't experimental either. It's also a decade old and
used extensively in production.
So I object to the notion of "DIX being made experimental". An
ASIC/firmware/driver implementation of DIX may be experimental. And of
course I can't rule out regressions in the kernel block integrity
implementation as a result of some of the recent MQ changes (will be
happy to work with you guys to figure those out).
But DIX isn't experimental, nor is the kernel support for passing
protection information to an HBA.
> For now we may not support DIX. It seems to have issues. We wanted to
> try 3008 card on our system, but it does not seem to support DIX 0-3.
For some reason Broadcom have not upstreamed their DIX support. It's
supposedly available in their outbox driver.
--
Martin K. Petersen Oracle Linux Engineering
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: DIF/DIX issue related to config CONFIG_SCSI_MQ_DEFAULT
[not found] ` <45193ec6-6398-3953-4833-88ca2057971a@huawei.com>
@ 2018-12-05 2:22 ` Martin K. Petersen
2018-12-05 15:27 ` John Garry
0 siblings, 1 reply; 16+ messages in thread
From: Martin K. Petersen @ 2018-12-05 2:22 UTC (permalink / raw)
To: John Garry
Cc: Martin K. Petersen, Ming Lei, chenxiang (M),
James E.J. Bottomley, linux-scsi, linux-block, Linuxarm,
Steffen Maier
John,
> OK, great. Happy to help. So far we have hisi_sas fio issue to go on
> plus knowledge that this issue seems to be exposed/triggered by
> enabling SCSI MQ.
I'll have to go revisit the archives for your exact fio bits.
But I successfully ran our DIX/T10 PI qualification tooling without any
errors on the latest SCSI tree. On both scsi_debug and FC hardware.
> My colleague chenxiang also tried DIF on 3008 with our same platform,
> and we are seeing an issue there also; here's a snippet:
Reproduced this. Turns out to be a recently introduced bug that
exclusively affects DIF-only setups (so primarily mpt3sas due to lack of
DIX support). It's a regression caused by a commit that went in through
block a few months ago. I'll send a patch...
--
Martin K. Petersen Oracle Linux Engineering
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: DIF/DIX issue related to config CONFIG_SCSI_MQ_DEFAULT
2018-11-30 1:19 ` Ming Lei
2018-11-30 11:26 ` John Garry
@ 2018-12-05 2:56 ` Martin K. Petersen
1 sibling, 0 replies; 16+ messages in thread
From: Martin K. Petersen @ 2018-12-05 2:56 UTC (permalink / raw)
To: Ming Lei
Cc: chenxiang (M),
James E.J. Bottomley, Martin K. Petersen, linux-scsi,
linux-block, John Garry, Linuxarm
Ming,
> For DIX/DIF test, I think you need to pass 'dev_size_mb=XXX' instead of
> 'virtual_gb'.
Correct!
--
Martin K. Petersen Oracle Linux Engineering
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: DIF/DIX issue related to config CONFIG_SCSI_MQ_DEFAULT
2018-12-05 2:22 ` Martin K. Petersen
@ 2018-12-05 15:27 ` John Garry
2018-12-06 4:22 ` Martin K. Petersen
0 siblings, 1 reply; 16+ messages in thread
From: John Garry @ 2018-12-05 15:27 UTC (permalink / raw)
To: Martin K. Petersen
Cc: Ming Lei, chenxiang (M),
James E.J. Bottomley, linux-scsi, linux-block, Linuxarm,
Steffen Maier
On 05/12/2018 02:22, Martin K. Petersen wrote:
>
Hi Martin,
> John,
>
>> OK, great. Happy to help. So far we have hisi_sas fio issue to go on
>> plus knowledge that this issue seems to be exposed/triggered by
>> enabling SCSI MQ.
>
> I'll have to go revisit the archives for your exact fio bits.
>
> But I successfully ran our DIX/T10 PI qualification tooling without any
> errors on the latest SCSI tree. On both scsi_debug and FC hardware.
I'm not fimilar with this tooling. Is it internal to your company?
Unfortunately it seems that no other SAS HBA kernel driver supports DIX,
so we have nothing to test against for this transport.
>
>> My colleague chenxiang also tried DIF on 3008 with our same platform,
>> and we are seeing an issue there also; here's a snippet:
>
> Reproduced this. Turns out to be a recently introduced bug that
> exclusively affects DIF-only setups (so primarily mpt3sas due to lack of
> DIX support). It's a regression caused by a commit that went in through
> block a few months ago. I'll send a patch...
>
ok, great.
thanks,
John
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: DIF/DIX issue related to config CONFIG_SCSI_MQ_DEFAULT
2018-12-05 15:27 ` John Garry
@ 2018-12-06 4:22 ` Martin K. Petersen
2018-12-06 17:33 ` John Garry
0 siblings, 1 reply; 16+ messages in thread
From: Martin K. Petersen @ 2018-12-06 4:22 UTC (permalink / raw)
To: John Garry
Cc: Martin K. Petersen, Ming Lei, chenxiang (M),
James E.J. Bottomley, linux-scsi, linux-block, Linuxarm,
Steffen Maier
John,
>> I'll have to go revisit the archives for your exact fio bits.
Your fio test case ran to completion on all my test systems.
>> But I successfully ran our DIX/T10 PI qualification tooling without any
>> errors on the latest SCSI tree. On both scsi_debug and FC hardware.
>
> I'm not fimilar with this tooling. Is it internal to your company?
Sadly, yes :( We have not had much luck upstreaming the necessary aio
integrity passthrough enhancements.
> Unfortunately it seems that no other SAS HBA kernel driver supports
> DIX, so we have nothing to test against for this transport.
We did a preliminary qual of LSISAS3008 many moons ago. But we had no
immediate use for it in our offerings so it never went beyond that. I
know other companies are using it, though.
--
Martin K. Petersen Oracle Linux Engineering
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: DIF/DIX issue related to config CONFIG_SCSI_MQ_DEFAULT
2018-12-06 4:22 ` Martin K. Petersen
@ 2018-12-06 17:33 ` John Garry
2018-12-07 3:20 ` Martin K. Petersen
0 siblings, 1 reply; 16+ messages in thread
From: John Garry @ 2018-12-06 17:33 UTC (permalink / raw)
To: Martin K. Petersen
Cc: Ming Lei, chenxiang (M),
James E.J. Bottomley, linux-scsi, linux-block, Linuxarm,
Steffen Maier
On 06/12/2018 04:22, Martin K. Petersen wrote:
>
Hi Martin,
> John,
>
>>> I'll have to go revisit the archives for your exact fio bits.
>
> Your fio test case ran to completion on all my test systems.
>
>>> But I successfully ran our DIX/T10 PI qualification tooling without any
>>> errors on the latest SCSI tree. On both scsi_debug and FC hardware.
>>
>> I'm not fimilar with this tooling. Is it internal to your company?
>
> Sadly, yes :( We have not had much luck upstreaming the necessary aio
> integrity passthrough enhancements.
>
>> Unfortunately it seems that no other SAS HBA kernel driver supports
>> DIX, so we have nothing to test against for this transport.
>
> We did a preliminary qual of LSISAS3008 many moons ago. But we had no
> immediate use for it in our offerings so it never went beyond that. I
> know other companies are using it, though.
>
It seems from https://www.spinics.net/lists/linux-scsi/msg97818.html
that 3008 mainline driver does support DIX:
"DIX is supported in mpt3sas , but not by default and hence while
loading mpt3sas driver set module parameter "prot_mask=0x7F" to enable
DIX support. "
Maybe we can try it.
BTW, on a loosely related topic, in drivers/scsi/sd.h, I noticed that
the arrays in sd_prot_op() and sd_prot_flag_mask() could be made static.
In doing so I found sd.o object code shrunk by ~100B.
Thanks,
John
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: DIF/DIX issue related to config CONFIG_SCSI_MQ_DEFAULT
2018-12-06 17:33 ` John Garry
@ 2018-12-07 3:20 ` Martin K. Petersen
0 siblings, 0 replies; 16+ messages in thread
From: Martin K. Petersen @ 2018-12-07 3:20 UTC (permalink / raw)
To: John Garry
Cc: Martin K. Petersen, Ming Lei, chenxiang (M),
James E.J. Bottomley, linux-scsi, linux-block, Linuxarm,
Steffen Maier
John,
> "DIX is supported in mpt3sas , but not by default and hence while
> loading mpt3sas driver set module parameter "prot_mask=0x7F" to enable
> DIX support. "
The plumbing is there but it doesn't work. I tried earlier today and I
still can't get a single I/O to complete. I'll try to hack away at it
tomorrow.
> BTW, on a loosely related topic, in drivers/scsi/sd.h, I noticed that
> the arrays in sd_prot_op() and sd_prot_flag_mask() could be made
> static. In doing so I found sd.o object code shrunk by ~100B.
Will fix, thanks!
--
Martin K. Petersen Oracle Linux Engineering
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2018-12-07 3:20 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-11-27 9:55 DIF/DIX issue related to config CONFIG_SCSI_MQ_DEFAULT chenxiang (M)
2018-11-27 13:08 ` Ming Lei
2018-11-28 3:37 ` chenxiang (M)
2018-11-29 18:17 ` Ming Lei
[not found] ` <6c573f36-60d8-0631-e9ac-dacd72f6c8ad@hisilicon.com>
2018-11-29 0:54 ` Ming Lei
2018-11-30 1:19 ` Ming Lei
2018-11-30 11:26 ` John Garry
2018-12-04 3:55 ` Martin K. Petersen
[not found] ` <45193ec6-6398-3953-4833-88ca2057971a@huawei.com>
2018-12-05 2:22 ` Martin K. Petersen
2018-12-05 15:27 ` John Garry
2018-12-06 4:22 ` Martin K. Petersen
2018-12-06 17:33 ` John Garry
2018-12-07 3:20 ` Martin K. Petersen
2018-12-05 2:56 ` Martin K. Petersen
2018-11-27 20:22 ` Ewan D. Milne
2018-11-28 3:11 ` chenxiang (M)
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).