在 2019/6/27 上午11:41, Ming Lei 写道: > On Thu, Jun 27, 2019 at 08:59:51AM +0800, Liu Yiding wrote: >> Hi, Ming. >> >> 在 2019/6/26 上午11:32, Ming Lei 写道: >>> On Wed, Jun 26, 2019 at 10:42:52AM +0800, Liu Yiding wrote: >>>> sorry, something wrong with attatched file. >>>> >>>> ------ >>>> >>>> wipefs -a --force /dev/sdb1 >>>> mkfs -t xfs /dev/sdb1 >>>> mkdir -p /fs/sdb1 >>>> mount -t xfs -o inode64 /dev/sdb1 /fs/sdb1 >>>> >>>> for cpu_dir in /sys/devices/system/cpu/cpu[0-9]* >>>> do >>>>     online_file="$cpu_dir"/online >>>>     [ -f "$online_file" ] && [ "$(cat "$online_file")" -eq 0 ] && continue >>>> >>>>     file="$cpu_dir"/cpufreq/scaling_governor >>>>     [ -f "$file" ] && echo "performance" > "$file" >>>> done >>>> >>>> echo '[global] >>>> bs=2M >>>> ioengine=sync >>>> iodepth=32 >>>> size=68719476736 >>>> direct=0 >>>> runtime=300 >>>> invalidate=1 >>>> fallocate=posix >>>> group_reporting >>>> >>>> [task_0] >>>> rw=randread >>>> directory=/fs/sdb1 >>>> numjobs=8' | fio --output-format=json - >>>> umount /fs/sdb1 >>> Hi Liu Yiding, >>> >>> Thanks for providing me the script. >>> >>> It may take a while for me to reproduce the issue, so could you apply >>> the following debug patch and post the log first? >> >> >> ------ >> >> 2412 [  112.558155] XFS (sdb1): Ending clean mount >> 2413 [ 1317.330820] WARNING: CPU: 63 PID: 1495 at >> drivers/scsi/scsi_lib.c:1034 scsi_init_io+0x11a/0x2e0 >> 2414 [ 1317.344000] Modules linked in: xfs dm_mod btrfs xor zstd_decompress >> zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac >> x86_pkg_temp_thermal intel_powerclamp coretemp kvm_i     ntel kvm irqbypass >> crct10dif_pclmul crc32_pclmul crc32c_intel mgag200 snd_pcm >> ghash_clmulni_intel ttm snd_timer drm_kms_helper ipmi_si ahci snd >> aesni_intel syscopyarea libahci ipmi_devi     ntf sysfillrect sysimgblt >> crypto_simd mei_me fb_sys_fops soundcore ioatdma cryptd ipmi_msghandler >> glue_helper pcspkr drm libata mei joydev lpc_ich i2c_i801 dca wmi >> acpi_power_meter acpi     _pad pcc_cpufreq ip_tables >> 2415 [ 1317.408135] CPU: 63 PID: 1495 Comm: kworker/63:1H Tainted: >> G          I       5.2.0-rc5-00001-gc2d55e2 #1 >> 2416 [ 1317.420350] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS >> SE5C610.86B.01.01.0008.021120151325 02/11/2015 >> 2417 [ 1317.433545] Workqueue: kblockd blk_mq_run_work_fn >> 2418 [ 1317.440295] RIP: 0010:scsi_init_io+0x11a/0x2e0 >> 2419 [ 1317.446746] Code: be 01 00 00 00 e9 44 ff ff ff 8b 45 70 89 83 08 01 >> 00 00 f6 45 1a 01 75 8a 31 c0 48 83 c4 08 5b 5d 41 5c 41 5d 41 5e 41 5f c3 >> <0f> 0b 8b 4d 18 8b 55 28 45 31 c9 48      c7 c6 98 da 69 82 48 c7 c7 f0 >> 2420 [ 1317.470863] RSP: 0018:ffffa5fa4f7e7c68 EFLAGS: 00010246 >> 2421 [ 1317.478294] RAX: 000000003d17d1b8 RBX: ffff8ae6cb742690 RCX: >> 0000000000000000 >> 2422 [ 1317.487878] RDX: 0000000000000000 RSI: 0000000000000000 RDI: >> ffff8ae6cb742690 >> 2423 [ 1317.497469] RBP: ffff8ae6cb742580 R08: 0000000000023003 R09: >> ffff8ae6cb7426c8 >> 2424 [ 1317.507057] R10: ffffa5fa4c7d7da0 R11: fefefefefefefeff R12: >> 0000000000000801 >> 2425 [ 1317.516650] R13: 0000000000000000 R14: 0000000000000000 R15: >> ffff8ac800bd1800 >> 2426 [ 1317.526253] FS:  0000000000000000(0000) GS:ffff8b077fac0000(0000) >> knlGS:0000000000000000 >> 2427 [ 1317.536918] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> 2428 [ 1317.544952] CR2: 00007f39a5818004 CR3: 000000343440a006 CR4: >> 00000000001606e0 >> 2429 [ 1317.554554] Call Trace: >> 2430 [ 1317.558921]  sd_init_command+0x2ee/0xaf0 [sd_mod] >> 2431 [ 1317.565800]  scsi_queue_rq+0x559/0xa80 >> 2432 [ 1317.571607]  blk_mq_dispatch_rq_list+0x3da/0x5b0 >> 2433 [ 1317.578375]  ? syscall_return_via_sysret+0x10/0x7f >> 2434 [ 1317.585339]  ? __switch_to_asm+0x40/0x70 >> 2435 [ 1317.591321]  ? elv_rb_del+0x1f/0x30 >> 2436 [ 1317.596807]  ? deadline_remove_request+0x55/0xc0 >> 2437 [ 1317.603560]  blk_mq_do_dispatch_sched+0x76/0x120 >> 2438 [ 1317.610302]  blk_mq_sched_dispatch_requests+0x100/0x170 >> 2439 [ 1317.617733]  __blk_mq_run_hw_queue+0x60/0x130 >> 2440 [ 1317.624200]  process_one_work+0x19c/0x3c0 >> 2441 [ 1317.630258]  worker_thread+0x3c/0x3b0 >> 2442 [ 1317.635937]  ? process_one_work+0x3c0/0x3c0 >> 2443 [ 1317.642183]  kthread+0x11e/0x140 >> 2444 [ 1317.647346]  ? kthread_park+0x90/0x90 >> 2445 [ 1317.652986]  ret_from_fork+0x35/0x40 >> 2446 [ 1317.658523] ---[ end trace 61026ff97261da6a ]--- >> 2447 [ 1317.665327] scsi_dump_rq: rq 6(f:801/23003, seg: 0) >> 2448 [ 1317.671601] bio 0: d602/801 0 1024971192 0 >> 2449 [ 1317.676991] print_req_error: I/O error, dev sdb, sector 1024971192 >> flags 801 >> 2450 [ 1317.685722] scsi_dump_rq: rq 8(f:801/23003, seg: 0) >> 2451 [ 1317.691994] bio 0: d602/801 0 1033359800 0 > The above line shows the bio is zero-size, and this bio is submitted > from bio, since the bio->bi_flags indicates that BIO_CLONED isn't set. > > Please apply the following debug patch, and we may see where is wrong. > > diff --git a/block/blk-core.c b/block/blk-core.c > index 8340f69670d8..86bb61b0df83 100644 > --- a/block/blk-core.c > +++ b/block/blk-core.c > @@ -1125,6 +1125,7 @@ blk_qc_t submit_bio(struct bio *bio) > bio_devname(bio, b), count); > } > } > + WARN_ON(bio->bi_iter.bi_size == 0 && bio_op(bio) == REQ_OP_WRITE); > > return generic_make_request(bio); Sorry for delay, ------ [  116.619136] 2019-06-29 16:12:41 mkdir -p /fs/sdb1 [  116.619138] [  116.628743] 2019-06-29 16:12:41 mount -t xfs -o inode64 /dev/sdb1 /fs/sdb1 [  116.628745] [  116.634699] SGI XFS with ACLs, security attributes, realtime, scrub, debug enabled [  116.653110] XFS (sdb1): Mounting V5 Filesystem [  116.670081] XFS (sdb1): Ending clean mount [  690.106779] WARNING: CPU: 53 PID: 1477 at block/blk-core.c:1128 submit_bio+0x93/0x150 [  690.117972] Modules linked in: xfs dm_mod btrfs xor zstd_decompress zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass mgag200 crct10dif_pclmul crc32_pclmul ttm crc32c_intel snd_pcm ghash_clmulni_intel drm_kms_helper snd_timer ahci syscopyarea sysfillrect snd aesni_intel sysimgblt libahci ipmi_si soundcore fb_sys_fops mei_me crypto_simd ipmi_devintf ioatdma cryptd pcspkr drm ipmi_msghandler glue_helper libata mei joydev lpc_ich i2c_i801 dca wmi acpi_power_meter acpi_pad pcc_cpufreq ip_tables [  690.176967] CPU: 53 PID: 1477 Comm: fio Tainted: G        W I       5.2.0-rc5-00002-g9fa09bd #1 [  690.187198] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS SE5C610.86B.01.01.0008.021120151325 02/11/2015 [  690.199375] RIP: 0010:submit_bio+0x93/0x150 [  690.204583] Code: ee 74 45 83 e1 01 74 26 65 48 01 35 bf 39 3f 45 8b 0d ad 96 c4 01 85 c9 75 56 8b 43 28 85 c0 75 b3 0f b6 53 10 83 fa 01 75 aa <0f> 0b eb a6 65 48 8b 14 25 c0 6b 01 00 65 48 01 35 88 39 3f 45 48 [  690.226668] RSP: 0018:ffffb2d90efc3b40 EFLAGS: 00010246 [  690.233072] RAX: 0000000000000000 RBX: ffff9654c6ad9f40 RCX: 0000000000000801 [  690.241617] RDX: 0000000000000001 RSI: ffff9654c6ad9f00 RDI: ffff9654c6ad9f40 [  690.250146] RBP: 0000000000000000 R08: 0000000000000002 R09: 0000000000000000 [  690.258667] R10: 00000000000303a0 R11: ffff96757ffd4000 R12: ffffb2d90efc3e08 [  690.267183] R13: ffff9654c6ad9f00 R14: ffffb2d90efc3e08 R15: 0000000000000000 [  690.275696] FS:  00007fa8e005da80(0000) GS:ffff9654ffac0000(0000) knlGS:0000000000000000 [  690.285282] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [  690.292252] CR2: 00007fa8cc05b3c0 CR3: 0000001fc8518001 CR4: 00000000001606e0 [  690.300770] Call Trace: [  690.304123]  xfs_submit_ioend+0x84/0x1e0 [xfs] [  690.309668]  xfs_do_writepage+0x246/0x5a0 [xfs] [  690.315279]  write_cache_pages+0x16b/0x440 [  690.320420]  ? xfs_vm_releasepage+0x80/0x80 [xfs] [  690.326238]  xfs_vm_writepages+0x6b/0xa0 [xfs] [  690.331742]  do_writepages+0x4b/0xf0 [  690.336275]  ? iomap_file_buffered_write+0x62/0x90 [  690.342162]  ? iomap_write_end+0x240/0x240 [  690.347295]  ? xfs_iunlock+0x165/0x220 [xfs] [  690.352595]  ? __filemap_fdatawrite_range+0xc1/0x100 [  690.358673]  __filemap_fdatawrite_range+0xc1/0x100 [  690.364571]  ? __switch_to_asm+0x34/0x70 [  690.369484]  file_write_and_wait_range+0x4c/0xa0 [  690.375198]  xfs_file_fsync+0x5d/0x230 [xfs] [  690.380499]  do_fsync+0x38/0x60 [  690.384534]  __x64_sys_fsync+0x10/0x20 [  690.389243]  do_syscall_64+0x5b/0x1e0 [  690.393857]  entry_SYSCALL_64_after_hwframe+0x44/0xa9 [  690.400020] RIP: 0033:0x7fa8cbe4f70d [  690.404533] Code: bc 20 00 00 75 10 b8 4a 00 00 00 0f 05 48 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 6e f7 ff ff 48 89 04 24 b8 4a 00 00 00 0f 05 <48> 8b 3c 24 48 89 c2 e8 b7 f7 ff ff 48 89 d0 48 83 c4 08 48 3d 01 [  690.426581] RSP: 002b:00007ffed6954160 EFLAGS: 00000293 ORIG_RAX: 000000000000004a [  690.435583] RAX: ffffffffffffffda RBX: 00005573363790f0 RCX: 00007fa8cbe4f70d [  690.444102] RDX: 0000000000200000 RSI: 00005573363790f0 RDI: 0000000000000003 [  690.452615] RBP: 00007fa8b337f418 R08: 0000000000000008 R09: 0000000000000001 [  690.461124] R10: 00000000897b0000 R11: 0000000000000293 R12: 0000000000000000 [  690.469631] R13: 0000001000000000 R14: 00007fa8c81dda50 R15: 0000000000000000 [  690.478130] ---[ end trace 90dd4ee6d5cdfcb8 ]--- [  690.483850] WARNING: CPU: 53 PID: 1562 at drivers/scsi/scsi_lib.c:1034 scsi_init_io+0x11a/0x2e0 [  690.495957] Modules linked in: xfs dm_mod btrfs xor zstd_decompress zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass mgag200 crct10dif_pclmul crc32_pclmul ttm crc32c_intel snd_pcm ghash_clmulni_intel drm_kms_helper snd_timer ahci syscopyarea sysfillrect snd aesni_intel sysimgblt libahci ipmi_si soundcore fb_sys_fops mei_me crypto_simd ipmi_devintf ioatdma cryptd pcspkr drm ipmi_msghandler glue_helper libata mei joydev lpc_ich i2c_i801 dca wmi acpi_power_meter acpi_pad pcc_cpufreq ip_tables [  690.555277] CPU: 53 PID: 1562 Comm: kworker/53:1H Tainted: G        W I       5.2.0-rc5-00002-g9fa09bd #1 [  690.566507] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS SE5C610.86B.01.01.0008.021120151325 02/11/2015 [  690.578729] Workqueue: kblockd blk_mq_run_work_fn [  690.584527] RIP: 0010:scsi_init_io+0x11a/0x2e0 [  690.590031] Code: be 01 00 00 00 e9 44 ff ff ff 8b 45 70 89 83 08 01 00 00 f6 45 1a 01 75 8a 31 c0 48 83 c4 08 5b 5d 41 5c 41 5d 41 5e 41 5f c3 <0f> 0b 8b 4d 18 8b 55 28 45 31 c9 48 c7 c6 98 da 89 bb 48 c7 c7 f0 [  690.612092] RSP: 0018:ffffb2d90f513c68 EFLAGS: 00010246 [  690.618484] RAX: 000000001d0c2838 RBX: ffff9654be21a9d0 RCX: 0000000000000000 [  690.627018] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9654be21a9d0 [  690.635558] RBP: ffff9654be21a8c0 R08: 0000000000023003 R09: ffff9654be21aa08 [  690.644080] R10: 0000000000000000 R11: fefefefefefefeff R12: 0000000000000801 [  690.652615] R13: 0000000000000000 R14: 0000000000000000 R15: ffff967572532000 [  690.661142] FS:  0000000000000000(0000) GS:ffff9654ffac0000(0000) knlGS:0000000000000000 [  690.670748] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [  690.677760] CR2: 00007fa8cc05b3c0 CR3: 0000001fc8518001 CR4: 00000000001606e0 [  690.686268] Call Trace: [  690.689574]  sd_init_command+0x2ee/0xaf0 [sd_mod] [  690.695387]  scsi_queue_rq+0x559/0xa80 [  690.700148]  blk_mq_dispatch_rq_list+0x3da/0x5b0 [  690.705889]  ? syscall_return_via_sysret+0x10/0x7f [  690.711840]  ? __switch_to_asm+0x40/0x70 [  690.716827]  ? elv_rb_del+0x1f/0x30 [  690.721275]  ? deadline_remove_request+0x55/0xc0 [  690.726966]  blk_mq_do_dispatch_sched+0x76/0x120 [  690.732657]  blk_mq_sched_dispatch_requests+0x100/0x170 [  690.739030]  __blk_mq_run_hw_queue+0x60/0x130 [  690.744454]  process_one_work+0x19c/0x3c0 [  690.749484]  worker_thread+0x3c/0x3b0 [  690.754097]  ? process_one_work+0x3c0/0x3c0 [  690.759304]  kthread+0x11e/0x140 [  690.763452]  ? kthread_park+0x90/0x90 [  690.768092]  ret_from_fork+0x35/0x40 [  690.772613] ---[ end trace 90dd4ee6d5cdfcb9 ]--- [  690.778316] scsi_dump_rq: rq 6(f:801/23003, seg: 0) [  690.784295] bio 0: d602/801 0 487335992 0 [  690.789280] print_req_error: I/O error, dev sdb, sector 487335992 flags 801 [  690.923742] XFS (sdb1): writeback error on sector 487335992 [  744.116327] fio: pid=0, err=5/file:filesetup.c:190, func=fsync, error=Input/output error [  744.116330] ------ > } > thanks, > Ming > >