From: Andrea Vai <andrea.vai@unipv.it>
To: Ming Lei <ming.lei@redhat.com>
Cc: Damien Le Moal <Damien.LeMoal@wdc.com>,
Alan Stern <stern@rowland.harvard.edu>,
Jens Axboe <axboe@kernel.dk>,
Johannes Thumshirn <jthumshirn@suse.de>,
USB list <linux-usb@vger.kernel.org>,
SCSI development list <linux-scsi@vger.kernel.org>,
Himanshu Madhani <himanshu.madhani@cavium.com>,
Hannes Reinecke <hare@suse.com>, Omar Sandoval <osandov@fb.com>,
"Martin K. Petersen" <martin.petersen@oracle.com>,
Greg KH <gregkh@linuxfoundation.org>,
Hans Holmberg <Hans.Holmberg@wdc.com>,
Kernel development list <linux-kernel@vger.kernel.org>
Subject: Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
Date: Wed, 27 Nov 2019 16:01:28 +0100 [thread overview]
Message-ID: <f9c04d802c99fca3886c6d46215f39a4ba171eee.camel@unipv.it> (raw)
In-Reply-To: <20191127130854.GA12140@ming.t460p>
[-- Attachment #1: Type: text/plain, Size: 3496 bytes --]
Il giorno mer, 27/11/2019 alle 21.08 +0800, Ming Lei ha scritto:
> On Wed, Nov 27, 2019 at 10:39:40AM +0100, Andrea Vai wrote:
> > Il giorno mer, 27/11/2019 alle 10.05 +0800, Ming Lei ha scritto:
> > >
> > >
> > > It can be workaround via the following change:
> > >
> > > /lib/modules/5.4.0+/build/include/generated/autoconf.h:
> > >
> > > //#define CONFIG_CC_HAS_ASM_INLINE 1
> >
> > Thanks, it worked, trace attached. Produced by: start the trace
> script
> > (with the pendrive already plugged), wait some seconds, run the
> test
> > (1 trial, 1 GB), wait for the test to finish, stop the trace.
> >
> > The copy took 2659 seconds, roughly as already seen before.
>
> Thanks for collecting the log.
>
> From the log, some of write IOs are out-of-order, such as, the 1st
> one
> is 378880.
>
> 16.41240 2 266 266 kworker/2:1H block_rq_issue b'W'
> 370656 240
> 16.41961 3 485 485 kworker/3:1H block_rq_issue b'W'
> 378880 240
> 16.73729 2 266 266 kworker/2:1H block_rq_issue b'W'
> 370896 240
> 17.71161 2 266 266 kworker/2:1H block_rq_issue b'W'
> 379120 240
> 18.02344 2 266 266 kworker/2:1H block_rq_issue b'W'
> 371136 240
> 18.94314 3 485 485 kworker/3:1H block_rq_issue b'W'
> 379360 240
> 19.25624 2 266 266 kworker/2:1H block_rq_issue b'W'
> 371376 240
>
> IO latency is increased a lot since the 1st out-of-order request(usb
> storage HBA is single queue depth, one request can be issued only
> if
> the previous issued request is completed).
>
> The reason is that there are two kind of tasks which inserts rq to
> device.
> One is the 'cp' process, the other is kworker/u8:*. The out-of-
> order
> happens during the two task's interleaving.
>
> Under such situation, I believe that the old legacy IO path may not
> guarantee the order too. In blk_queue_bio(), after get_request()
> allocates one request, the queue lock is released. And request is
> actually inserted & issued from blk_flush_plug_list() under the
> branch of 'if (plug)'. If requests are from two tasks, then request
> is inserted/issued from two plug list, and no order can be
> guaranteed.
>
> In my test, except for several requests from the beginning, all
> other
> requests are inserted via the kworker thread(guess it is writeback
> wq),
> that is why I can't observe the issue in my test.
>
> As Schmid suggested, you may run the same test on old kernel with
> legacy io path, and see if the performance is still good.
>
> Also, could you share the following info about your machine? So that
> I can build my VM guest in this setting for reproducing your
> situation
> (requests are inserted from two types of threads).
>
> - lscpu
attached,
> - free -h
total used free shared buff/cache available
Mem: 23Gi 4,2Gi 11Gi 448Mi 7,0Gi 18Gi
Swap: 3,7Gi 0B 3,7Gi
> - lsblk -d $USB_DISK
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdg 8:96 1 28,8G 0 disk
> - exact commands for mount the disk, and running the copy operation
I attached the whole script to this thread, I attach it again to this
message and copy the relevant lines here:
mount UUID=$uuid /mnt/pendrive 2>&1 |tee -a $logfile
SECONDS=0
cp $testfile /mnt/pendrive 2>&1 |tee -a $logfile
umount /mnt/pendrive 2>&1 |tee -a $logfile
Meanwhile, I am going on with the further tests as suggested
Thanks,
Andrea
[-- Attachment #2: lscpu.txt --]
[-- Type: text/plain, Size: 1371 bytes --]
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 39 bits physical, 48 bits virtual
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 60
Model name: Intel(R) Core(TM) i5-4430 CPU @ 3.00GHz
Stepping: 3
CPU MHz: 1674.727
CPU max MHz: 3200,0000
CPU min MHz: 800,0000
BogoMIPS: 5986.16
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 6144K
NUMA node0 CPU(s): 0-3
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts md_clear flush_l1d
[-- Attachment #3: test --]
[-- Type: application/x-shellscript, Size: 1137 bytes --]
next prev parent reply other threads:[~2019-11-27 15:01 UTC|newest]
Thread overview: 120+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <307581a490b610c3025ee80f79a465a89d68ed19.camel@unipv.it>
2019-08-20 17:13 ` Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6 Alan Stern
2019-08-23 10:39 ` Andrea Vai
2019-08-23 20:42 ` Alan Stern
2019-08-26 6:09 ` Andrea Vai
2019-08-26 16:33 ` Alan Stern
2019-09-18 15:25 ` Andrea Vai
2019-09-18 16:30 ` Alan Stern
2019-09-19 7:33 ` Andrea Vai
2019-09-19 17:54 ` Alan Stern
2019-09-20 7:25 ` Andrea Vai
2019-09-20 7:44 ` Greg KH
2019-09-19 8:26 ` Damien Le Moal
2019-09-19 8:55 ` Ming Lei
2019-09-19 9:09 ` Damien Le Moal
2019-09-19 9:21 ` Ming Lei
2019-09-19 14:01 ` Alan Stern
2019-09-19 14:14 ` Damien Le Moal
2019-09-20 7:03 ` Andrea Vai
2019-09-25 19:30 ` Alan Stern
2019-09-25 19:36 ` Jens Axboe
2019-09-27 15:47 ` Andrea Vai
2019-11-04 16:00 ` Andrea Vai
2019-11-04 18:20 ` Alan Stern
2019-11-05 11:48 ` Andrea Vai
2019-11-05 18:31 ` Alan Stern
2019-11-05 23:29 ` Jens Axboe
2019-11-06 16:03 ` Alan Stern
2019-11-06 22:13 ` Damien Le Moal
2019-11-07 7:04 ` Andrea Vai
2019-11-07 7:54 ` Damien Le Moal
2019-11-07 18:59 ` Andrea Vai
2019-11-08 8:42 ` Damien Le Moal
2019-11-08 14:33 ` Jens Axboe
2019-11-11 10:46 ` Andrea Vai
2019-11-09 10:09 ` Ming Lei
2019-11-09 22:28 ` Ming Lei
2019-11-11 10:50 ` Andrea Vai
2019-11-11 11:05 ` Ming Lei
2019-11-11 11:13 ` Andrea Vai
2019-11-22 19:16 ` Andrea Vai
2019-11-23 7:28 ` Ming Lei
2019-11-23 15:44 ` Andrea Vai
2019-11-25 3:54 ` Ming Lei
2019-11-25 10:11 ` Andrea Vai
2019-11-25 10:29 ` Ming Lei
2019-11-25 14:58 ` Andrea Vai
2019-11-25 15:15 ` Ming Lei
2019-11-25 18:51 ` Andrea Vai
2019-11-26 2:32 ` Ming Lei
2019-11-26 7:46 ` Andrea Vai
2019-11-26 9:15 ` Ming Lei
2019-11-26 10:24 ` Ming Lei
2019-11-26 11:14 ` Andrea Vai
2019-11-27 2:05 ` Ming Lei
2019-11-27 9:39 ` Andrea Vai
2019-11-27 13:08 ` Ming Lei
2019-11-27 15:01 ` Andrea Vai [this message]
2019-11-27 0:21 ` Finn Thain
2019-11-27 8:14 ` AW: " Schmid, Carsten
2019-11-27 21:49 ` Finn Thain
2019-11-28 7:46 ` Andrea Vai
2019-11-28 8:12 ` AW: " Schmid, Carsten
2019-11-28 11:40 ` Andrea Vai
2019-11-28 17:39 ` Alan Stern
2019-11-28 9:17 ` Ming Lei
2019-11-28 17:34 ` Andrea Vai
2019-11-29 0:57 ` Ming Lei
2019-11-29 2:35 ` Ming Lei
2019-11-29 14:41 ` Andrea Vai
2019-12-03 2:23 ` Ming Lei
2019-12-10 7:35 ` Andrea Vai
2019-12-10 8:05 ` Ming Lei
2019-12-11 2:41 ` Theodore Y. Ts'o
2019-12-11 4:00 ` Ming Lei
2019-12-11 16:07 ` Theodore Y. Ts'o
2019-12-11 21:33 ` Ming Lei
2019-12-12 7:34 ` Andrea Vai
2019-12-18 8:25 ` Andrea Vai
2019-12-18 9:48 ` Ming Lei
[not found] ` <b1b6a0e9d690ecd9432025acd2db4ac09f834040.camel@unipv.it>
2019-12-23 13:08 ` Ming Lei
2019-12-23 14:02 ` Andrea Vai
2019-12-24 1:32 ` Ming Lei
2019-12-24 8:04 ` Andrea Vai
2019-12-24 8:47 ` Ming Lei
2019-12-23 16:26 ` Theodore Y. Ts'o
2019-12-23 16:29 ` Andrea Vai
2019-12-23 17:22 ` Theodore Y. Ts'o
2019-12-23 18:45 ` Andrea Vai
2019-12-23 19:53 ` Theodore Y. Ts'o
2019-12-24 1:27 ` Ming Lei
2019-12-24 6:49 ` Andrea Vai
2019-12-24 8:51 ` Andrea Vai
2019-12-24 9:35 ` Ming Lei
2019-12-25 5:17 ` Theodore Y. Ts'o
2019-12-26 2:27 ` Ming Lei
2019-12-26 3:30 ` Theodore Y. Ts'o
2019-12-26 8:37 ` Ming Lei
2020-01-07 7:51 ` Andrea Vai
[not found] ` <20200101074310.10904-1-hdanton@sina.com>
2020-01-01 13:53 ` slow IO on USB media Ming Lei
2019-11-29 11:44 ` AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6 Bernd Schubert
2019-12-02 7:01 ` Andrea Vai
2019-11-28 17:10 ` Andrea Vai
[not found] <e3f87757f7a0fdf551e911ad32fc8122eebe04c7.camel@unipv.it>
2019-08-13 19:52 ` Alan Stern
2019-07-02 10:46 Andrea Vai
2019-07-02 11:51 ` Johannes Thumshirn
2019-07-02 22:36 ` Andrea Vai
2019-07-03 7:29 ` Johannes Thumshirn
2019-07-03 14:23 ` Alan Stern
2019-07-06 22:06 ` Andrea Vai
2019-07-08 15:38 ` Alan Stern
2019-07-02 12:01 ` Ming Lei
2019-07-02 22:39 ` Andrea Vai
2019-07-03 2:01 ` Ming Lei
2019-07-03 5:11 ` Andrea Vai
2019-07-03 6:36 ` Ming Lei
2019-07-03 15:27 ` Chris Murphy
2019-07-06 9:33 ` Andrea Vai
2019-07-08 1:01 ` Ming Lei
2019-07-09 21:18 ` Andrea Vai
2019-07-10 2:44 ` Ming Lei
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=f9c04d802c99fca3886c6d46215f39a4ba171eee.camel@unipv.it \
--to=andrea.vai@unipv.it \
--cc=Damien.LeMoal@wdc.com \
--cc=Hans.Holmberg@wdc.com \
--cc=axboe@kernel.dk \
--cc=gregkh@linuxfoundation.org \
--cc=hare@suse.com \
--cc=himanshu.madhani@cavium.com \
--cc=jthumshirn@suse.de \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
--cc=linux-usb@vger.kernel.org \
--cc=martin.petersen@oracle.com \
--cc=ming.lei@redhat.com \
--cc=osandov@fb.com \
--cc=stern@rowland.harvard.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).