Re: 07173c3ec2: stderr.fio:pid=#, err=#/file:filesetup.c:#, func=fsync, error=Input/output_error

From: Liu Yiding <liuyd.fnst@cn.fujitsu.com>
To: lkp@lists.01.org
Subject: Re: 07173c3ec2: stderr.fio:pid=#, err=#/file:filesetup.c:#, func=fsync, error=Input/output_error
Date: Wed, 26 Jun 2019 10:28:10 +0800	[thread overview]
Message-ID: <80a11f3b-5954-246c-a083-1d718863c8f1@cn.fujitsu.com> (raw)
In-Reply-To: <35b3504b-e1dc-f345-76a4-ebf1c5f9ac57@cn.fujitsu.com>

[-- Attachment #1: Type: text/plain, Size: 6423 bytes --]

attatch reproduce.sh


在 2019/6/26 上午10:21, Liu Yiding 写道:
>
> 在 2019/6/26 上午10:05, Ming Lei 写道:
>> On Wed, Jun 26, 2019 at 09:18:36AM +0800, Liu Yiding wrote:
>>> Hi, Ming Lei.
>>>
>>>
>>> 在 2019/6/25 下午6:47, Ming Lei 写道:
>>>> Hi rong.a.chen,
>>>>
>>>> Thanks for your report!
>>>>
>>>> On Tue, Jun 25, 2019 at 04:32:48PM +0800, kernel test robot wrote:
>>>>> FYI, we noticed the following commit (built with gcc-7):
>>>>>
>>>>> commit: 07173c3ec276cbb18dc0e0687d37d310e98a1480 ("block: enable 
>>>>> multipage bvecs")
>>>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git  
>>>>> master
>>>>>
>>>>> in testcase: fio-basic
>>>>> on test machine: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 
>>>>> 2.30GHz with 256G memory
>>>>> with following parameters:
>>>>>
>>>>>     runtime: 300s
>>>>>     nr_task: 8t
>>>>>     disk: 1SSD
>>>>>     fs: xfs
>>>>>     rw: randread
>>>>>     bs: 2M
>>>>>     ioengine: sync
>>>>>     test_size: 512g
>>>>>
>>>>> caused below changes (please refer to attached dmesg/kmsg for 
>>>>> entire log/backtrace):
>>>>>
>>>>> fio: pid=0, err=5/file:filesetup.c:190, func=fsync, 
>>>>> error=Input/output error
>>>>> fio: pid=0, err=5/file:filesetup.c:190, func=fsync, 
>>>>> error=Input/output error
>>>> I just tried the fio(randread, bs:2M, sync, nvme, xfs and block 
>>>> device) test on
>>>> 5.2-rc6 kernel, looks it can't be reproduced.
>>>>
>>>> For multi-page bvec change, I'd suggest to run test on 5.1 at least,
>>>> since there are several follow-up fixes.
>>>>
>>>> Also the dmesg log seems not intact, looks it only shows booting.
>>>>
>>>> Please let us know if the issue can be triggered on 5.1 or 5.2.
>>>
>>> This issue can be triggered on v5.1 and v5,2-rc* (include v5.2-rc6)
>>>
>>> on v5.2-rc6 run 20 times, 10 times trigger out this issue.
>> OK, thanks!
>>
>> It is EIO from fsync(), which should have something dumped in dmesg log,
>> could you provide me the dmesg log after the issue is triggered?
>
>
> i have attatched the whole dmesg.xz on email.
>
>
>>
>> BTW, can you provide me the exact fio command in your test? I need to
>> find a big machine like yours for reproducing it.
>
> lkp will do some test env setting for fio test,  it seems that this 
> error happend in mount.
>
> ------
>
> [  107.314810] 2019-06-19 03:37:09 mount -t xfs -o inode64 /dev/sdb1 
> /fs/sdb1
> [  107.314812]
> [  107.316017] SGI XFS with ACLs, security attributes, realtime, 
> scrub, debug enabled
> [  107.338674] XFS (sdb1): Mounting V5 Filesystem
> [  107.356339] XFS (sdb1): Ending clean mount
> [  683.136603] WARNING: CPU: 1 PID: 1484 at 
> drivers/scsi/scsi_lib.c:1010 scsi_init_io+0x103/0x170
> [  683.148708] Modules linked in: xfs dm_mod btrfs xor zstd_decompress 
> zstd_compress raid6_pq libcrc32c sd_mod sg intel_rapl sb_edac 
> x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm mgag200 
> ttm irqbypass drm_kms_helper crct10dif_pclmul crc32_pclmul 
> crc32c_intel syscopyarea ghash_clmulni_intel snd_pcm sysfillrect 
> ipmi_si snd_timer sysimgblt aesni_intel ahci ipmi_devintf crypto_simd 
> snd fb_sys_fops libahci soundcore mei_me cryptd glue_helper 
> ipmi_msghandler pcspkr drm libata mei ioatdma joydev lpc_ich i2c_i801 
> dca wmi acpi_power_meter acpi_pad pcc_cpufreq ip_tables
> [  683.207865] CPU: 1 PID: 1484 Comm: kworker/1:1H Tainted: G          
> I       5.2.0-rc5 #1
> [  683.217421] Hardware name: Intel Corporation S2600WTT/S2600WTT, 
> BIOS SE5C610.86B.01.01.0008.021120151325 02/11/2015
> [  683.229600] Workqueue: kblockd blk_mq_run_work_fn
> [  683.235364] RIP: 0010:scsi_init_io+0x103/0x170
> [  683.240834] Code: 5c 41 5d c3 be 01 00 00 00 e9 49 ff ff ff 8b 45 
> 70 89 83 08 01 00 00 f6 45 1a 01 75 92 31 c0 48 83 c4 08 5b 5d 41 5c 
> 41 5d c3 <0f> 0b 48 83 c4 08 b8 0a 00 00 00 5b 5d 41 5c 41 5d c3 48 8b 
> 75 38
> [  683.262867] RSP: 0018:ffffb8f1cf853c78 EFLAGS: 00010246
> [  683.269260] RAX: 000000001d338288 RBX: ffff8d2af2612690 RCX: 
> 0000000000000000
> [  683.277768] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 
> ffff8d2af2612690
> [  683.286278] RBP: ffff8d2af2612580 R08: 0000000000000000 R09: 
> ffff8d2af26126c8
> [  683.294785] R10: 0000000000000000 R11: fefefefefefefeff R12: 
> 0000000000000801
> [  683.303295] R13: 0000000000000000 R14: 0000000000000000 R15: 
> ffff8d2b0b99a800
> [  683.311804] FS:  0000000000000000(0000) GS:ffff8d2b3f240000(0000) 
> knlGS:0000000000000000
> [  683.321379] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  683.328331] CR2: 00007faa9d9697d0 CR3: 000000407a8a8003 CR4: 
> 00000000001606e0
> [  683.336843] Call Trace:
> [  683.340122]  sd_init_command+0x2ee/0xaf0 [sd_mod]
> [  683.345917]  scsi_queue_rq+0x559/0xa80
> [  683.350639]  blk_mq_dispatch_rq_list+0x3da/0x5b0
> [  683.356333]  ? syscall_return_via_sysret+0x10/0x7f
> [  683.362276]  ? __switch_to_asm+0x40/0x70
> [  683.367187]  ? elv_rb_del+0x1f/0x30
> [  683.371607]  ? deadline_remove_request+0x55/0xc0
> [  683.377290]  blk_mq_do_dispatch_sched+0x76/0x120
> [  683.382990]  blk_mq_sched_dispatch_requests+0x100/0x170
> [  683.389349]  __blk_mq_run_hw_queue+0x60/0x130
> [  683.394745]  process_one_work+0x19c/0x3c0
> [  683.399746]  worker_thread+0x3c/0x3b0
> [  683.404392]  ? process_one_work+0x3c0/0x3c0
> [  683.409580]  kthread+0x11e/0x140
> [  683.413734]  ? kthread_park+0x90/0x90
> [  683.418335]  ret_from_fork+0x35/0x40
> [  683.422841] ---[ end trace f691cac020928424 ]---
> [  683.428614] print_req_error: I/O error, dev sdb, sector 489915016 
> flags 801
> [  683.561809] XFS (sdb1): writeback error on sector 489915016
> [  732.579401] fio: pid=0, err=5/file:filesetup.c:190, func=fsync, 
> error=Input/output error
> [  732.579403]
> [ 1022.580119] print_req_error: I/O error, dev sdb, sector 795883128 
> flags 801
> [ 1022.933676] XFS (sdb1): writeback error on sector 795883128
> [ 1035.041938] fio: pid=0, err=5/file:filesetup.c:190, func=fsync, 
> error=Input/output error
>
> ------
>
>
>>
>> Thanks,
>> Ming
>>
>>
>
>
>
> _______________________________________________
> LKP mailing list
> LKP(a)lists.01.org
> https://lists.01.org/mailman/listinfo/lkp


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 9742 bytes --]

[-- Attachment #3: reproduce.sh --]
[-- Type: application/x-shellscript, Size: 609 bytes --]