Linux-USB Archive on lore.kernel.org
 help / color / Atom feed
From: Damien Le Moal <Damien.LeMoal@wdc.com>
To: Alan Stern <stern@rowland.harvard.edu>, Jens Axboe <axboe@kernel.dk>
Cc: Andrea Vai <andrea.vai@unipv.it>,
	Johannes Thumshirn <jthumshirn@suse.de>,
	USB list <linux-usb@vger.kernel.org>,
	SCSI development list <linux-scsi@vger.kernel.org>,
	Himanshu Madhani <himanshu.madhani@cavium.com>,
	Hannes Reinecke <hare@suse.com>, Ming Lei <ming.lei@redhat.com>,
	Omar Sandoval <osandov@fb.com>,
	"Martin K. Petersen" <martin.petersen@oracle.com>,
	Greg KH <gregkh@linuxfoundation.org>,
	Hans Holmberg <Hans.Holmberg@wdc.com>,
	Kernel development list <linux-kernel@vger.kernel.org>
Subject: Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
Date: Wed, 6 Nov 2019 22:13:24 +0000
Message-ID: <BYAPR04MB5816640CEF40CB52430BBD3AE7790@BYAPR04MB5816.namprd04.prod.outlook.com> (raw)
In-Reply-To: <Pine.LNX.4.44L0.1911061044070.1694-100000@iolanthe.rowland.org>

On 2019/11/07 1:04, Alan Stern wrote:
> On Tue, 5 Nov 2019, Jens Axboe wrote:
> 
>> On 11/5/19 11:31 AM, Alan Stern wrote:
>>> On Tue, 5 Nov 2019, Andrea Vai wrote:
>>>
>>>> Il giorno lun, 04/11/2019 alle 13.20 -0500, Alan Stern ha scritto:
>>>
>>>>> You should be able to do something like this:
>>>>>
>>>>>          cd linux
>>>>>          patch -p1 </path/to/patch2
>>>>>
>>>>> and that should work with no errors.  You don't need to use git to
>>>>> apply a patch.
>>>>>
>>>>> In case that patch2 file was mangled somewhere along the way, I
>>>>> have
>>>>> attached a copy to this message.
>>>>
>>>> Ok, so the "patch" command worked, the kernel compiled and ran, but
>>>> the test still failed (273, 108, 104, 260, 177, 236, 179, 1123, 289,
>>>> 873 seconds to copy a 500MB file, vs. ~30 seconds with the "good"
>>>> kernel).
>>>>
>>>> Let me know what else could I do,
>>>
>>> I'm out of suggestions.  If anyone else knows how to make a kernel with
>>> no legacy queuing support -- only multiqueue -- issue I/O requests
>>> sequentially, please speak up.
>>
>> Do we know for a fact that the device needs strictly serialized requests
>> to not stall?
> 
> Not exactly, but that is far and away the most likely explanation for
> the device's behavior.  We tried making a bunch of changes, some of
> which helped a little bit, but all of them left a very large
> performance gap.  I/O monitoring showed that the only noticeable
> difference in the kernel-device interaction caused by the $SUBJECT
> commit was the non-sequential access pattern.
> 
>> And writes in particular?
> 
> Andrea has tested only the write behavior.  Possibly reading will be
> affected too, but my guess is that it won't be.
> 
>> I won't comment on how broken
>> that is, just trying to establish this as the problem that's making this
>> particular device be slow?
> 
> It seems reasonable that the access pattern could make a significant
> difference.  The device's behavior suggests that it buffers incoming
> data and pauses from time to time to write the accumulated data into
> non-volatile storage.  If its algorithm for allocating, erasing, and
> writing data blocks is optimized for the sequential case, you can
> easily imagine that non-sequential accesses would cause it to pause
> more often and for longer times -- which is exactly what we observed.
> These extra pauses are what resulted in the overall performance 
> decrease.
> 
> So far we have had no way to perform a direct test.  That is, we don't
> know of any setting that would change a single kernel between
> sequential and non-sequential access.  If you can suggest a simple way
> to force a kernel without the $SUBJECT commit to do non-sequential
> writes, I'm sure Andrea will be happy to try it out and see if it
> causes a slowdown.
> 
>> I've lost track of this thread, but has mq-deadline been tried as the
>> IO scheduler? We do have support for strictly serialized (writes)
>> since that's required for zoned device, wouldn't be hard at all to make
>> this cover a blacklisted device like this one.
> 
> Please spell out the exact procedure in detail so that Andrea can try 
> it.  He's not a kernel hacker, and I know very little about the block 
> layer.

Please simply try your write tests after doing this:

echo mq-deadline > /sys/block/<name of your USB disk>/queue/scheduler

And confirm that mq-deadline is selected with:

cat /sys/block/<name of your USB disk>/queue/scheduler
[mq-deadline] kyber bfq none


> 
> Alan Stern
> 
> 


-- 
Damien Le Moal
Western Digital Research

  reply index

Thread overview: 91+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <307581a490b610c3025ee80f79a465a89d68ed19.camel@unipv.it>
2019-08-20 17:13 ` Alan Stern
2019-08-23 10:39   ` Andrea Vai
2019-08-23 20:42     ` Alan Stern
2019-08-26  6:09       ` Andrea Vai
2019-08-26 16:33         ` Alan Stern
2019-09-18 15:25           ` Andrea Vai
2019-09-18 16:30             ` Alan Stern
2019-09-19  7:33               ` Andrea Vai
2019-09-19 17:54                 ` Alan Stern
2019-09-20  7:25                   ` Andrea Vai
2019-09-20  7:44                     ` Greg KH
2019-09-19  8:26               ` Damien Le Moal
2019-09-19  8:55                 ` Ming Lei
2019-09-19  9:09                   ` Damien Le Moal
2019-09-19  9:21                     ` Ming Lei
2019-09-19 14:01                 ` Alan Stern
2019-09-19 14:14                   ` Damien Le Moal
2019-09-20  7:03                     ` Andrea Vai
2019-09-25 19:30                       ` Alan Stern
2019-09-25 19:36                         ` Jens Axboe
2019-09-27 15:47                           ` Andrea Vai
2019-11-04 16:00                             ` Andrea Vai
2019-11-04 18:20                               ` Alan Stern
2019-11-05 11:48                                 ` Andrea Vai
2019-11-05 18:31                                   ` Alan Stern
2019-11-05 23:29                                     ` Jens Axboe
2019-11-06 16:03                                       ` Alan Stern
2019-11-06 22:13                                         ` Damien Le Moal [this message]
2019-11-07  7:04                                           ` Andrea Vai
2019-11-07  7:54                                             ` Damien Le Moal
2019-11-07 18:59                                               ` Andrea Vai
2019-11-08  8:42                                                 ` Damien Le Moal
2019-11-08 14:33                                                   ` Jens Axboe
2019-11-11 10:46                                                     ` Andrea Vai
2019-11-09 10:09                                                   ` Ming Lei
2019-11-09 22:28                                                 ` Ming Lei
2019-11-11 10:50                                                   ` Andrea Vai
2019-11-11 11:05                                                     ` Ming Lei
2019-11-11 11:13                                                       ` Andrea Vai
2019-11-22 19:16                                                   ` Andrea Vai
2019-11-23  7:28                                                     ` Ming Lei
2019-11-23 15:44                                                       ` Andrea Vai
2019-11-25  3:54                                                         ` Ming Lei
2019-11-25 10:11                                                           ` Andrea Vai
2019-11-25 10:29                                                             ` Ming Lei
2019-11-25 14:58                                                               ` Andrea Vai
2019-11-25 15:15                                                                 ` Ming Lei
2019-11-25 18:51                                                                   ` Andrea Vai
2019-11-26  2:32                                                                     ` Ming Lei
2019-11-26  7:46                                                                       ` Andrea Vai
2019-11-26  9:15                                                                         ` Ming Lei
2019-11-26 10:24                                                                           ` Ming Lei
2019-11-26 11:14                                                                           ` Andrea Vai
2019-11-27  2:05                                                                             ` Ming Lei
2019-11-27  9:39                                                                               ` Andrea Vai
2019-11-27 13:08                                                                                 ` Ming Lei
2019-11-27 15:01                                                                                   ` Andrea Vai
2019-11-27  0:21                                                                         ` Finn Thain
2019-11-27  8:14                                                                           ` AW: " Schmid, Carsten
2019-11-27 21:49                                                                             ` Finn Thain
2019-11-28  7:46                                                                             ` Andrea Vai
2019-11-28  8:12                                                                               ` AW: " Schmid, Carsten
2019-11-28 11:40                                                                                 ` Andrea Vai
2019-11-28 17:39                                                                                 ` Alan Stern
2019-11-28  9:17                                                                               ` Ming Lei
2019-11-28 17:34                                                                                 ` Andrea Vai
2019-11-29  0:57                                                                                   ` Ming Lei
2019-11-29  2:35                                                                                     ` Ming Lei
2019-11-29 14:41                                                                                       ` Andrea Vai
2019-12-03  2:23                                                                                         ` Ming Lei
2019-11-29 11:44                                                                                     ` Bernd Schubert
2019-12-02  7:01                                                                                       ` Andrea Vai
2019-11-28 17:10                                                                           ` Andrea Vai
     [not found] <e3f87757f7a0fdf551e911ad32fc8122eebe04c7.camel@unipv.it>
2019-08-13 19:52 ` Alan Stern
2019-07-02 10:46 Andrea Vai
2019-07-02 11:51 ` Johannes Thumshirn
2019-07-02 22:36   ` Andrea Vai
2019-07-03  7:29     ` Johannes Thumshirn
2019-07-03 14:23       ` Alan Stern
2019-07-06 22:06         ` Andrea Vai
2019-07-08 15:38           ` Alan Stern
2019-07-02 12:01 ` Ming Lei
2019-07-02 22:39   ` Andrea Vai
2019-07-03  2:01     ` Ming Lei
2019-07-03  5:11       ` Andrea Vai
2019-07-03  6:36         ` Ming Lei
2019-07-03 15:27           ` Chris Murphy
2019-07-06  9:33           ` Andrea Vai
2019-07-08  1:01             ` Ming Lei
2019-07-09 21:18               ` Andrea Vai
2019-07-10  2:44                 ` Ming Lei

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=BYAPR04MB5816640CEF40CB52430BBD3AE7790@BYAPR04MB5816.namprd04.prod.outlook.com \
    --to=damien.lemoal@wdc.com \
    --cc=Hans.Holmberg@wdc.com \
    --cc=andrea.vai@unipv.it \
    --cc=axboe@kernel.dk \
    --cc=gregkh@linuxfoundation.org \
    --cc=hare@suse.com \
    --cc=himanshu.madhani@cavium.com \
    --cc=jthumshirn@suse.de \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-scsi@vger.kernel.org \
    --cc=linux-usb@vger.kernel.org \
    --cc=martin.petersen@oracle.com \
    --cc=ming.lei@redhat.com \
    --cc=osandov@fb.com \
    --cc=stern@rowland.harvard.edu \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-USB Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-usb/0 linux-usb/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-usb linux-usb/ https://lore.kernel.org/linux-usb \
		linux-usb@vger.kernel.org
	public-inbox-index linux-usb

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-usb


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git