* Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
@ 2019-07-02 10:46 Andrea Vai
2019-07-02 11:51 ` Johannes Thumshirn
2019-07-02 12:01 ` Ming Lei
0 siblings, 2 replies; 77+ messages in thread
From: Andrea Vai @ 2019-07-02 10:46 UTC (permalink / raw)
To: Jens Axboe
Cc: linux-usb, linux-scsi, Himanshu Madhani, Hannes Reinecke,
Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH, Alan Stern
Hi,
I have a problem writing data to a USB pendrive, and it seems
kernel-related. With the help of Greg an Alan (thanks) and some
bisect, I found out the offending commit being
commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
Author: Jens Axboe <axboe@kernel.dk>
Date: Thu Nov 1 16:36:27 2018 -0600
scsi: kill off the legacy IO path
This removes the legacy (non-mq) IO path for SCSI.
So, here I am to notify you about the problem and ask you if I can
help in any way to work it out and fix it.
The problem is that if I copy a file from the internal SATA HD to the
pendrive, it takes ~10 times to complete (in respect of the time
needed with the patch reverted).
The test script, which I use to detect if the problem triggers or not,
is:
#!/bin/bash
logfile=...
uname -a | tee -a $logfile
echo -n "Begin: " | tee -a $logfile
date | tee -a $logfile
touch inizio
SECONDS=0
mount UUID="05141239-4ea5-494d-aa91-acd67db89ce5" /mnt/pendrive
cp /NoBackup/buttare/testfile /mnt/pendrive
umount /mnt/pendrive
tempo=$SECONDS
touch fine
echo -n "...end: " | tee -a $logfile
date | tee -a $logfile
echo "It took $tempo seconds!" | tee -a $logfile
If I run the test with a 512MB file it takes >10min vs. half a minute.
The problem is still present in last tested git (cloned today in the
morning).
You can see the previous discussion that lead to these results at
https://marc.info/?t=155922230700001&r=1&w=2
Thanks, and bye
Andrea
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-07-02 10:46 Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6 Andrea Vai
@ 2019-07-02 11:51 ` Johannes Thumshirn
2019-07-02 22:36 ` Andrea Vai
2019-07-02 12:01 ` Ming Lei
1 sibling, 1 reply; 77+ messages in thread
From: Johannes Thumshirn @ 2019-07-02 11:51 UTC (permalink / raw)
To: Andrea Vai
Cc: Jens Axboe, linux-usb, linux-scsi, Himanshu Madhani,
Hannes Reinecke, Ming Lei, Omar Sandoval, Martin K. Petersen,
Greg KH, Alan Stern
On Tue, Jul 02, 2019 at 12:46:45PM +0200, Andrea Vai wrote:
> Hi,
> I have a problem writing data to a USB pendrive, and it seems
> kernel-related. With the help of Greg an Alan (thanks) and some
> bisect, I found out the offending commit being
>
> commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
> Author: Jens Axboe <axboe@kernel.dk>
> Date: Thu Nov 1 16:36:27 2018 -0600
>
> scsi: kill off the legacy IO path
>
> This removes the legacy (non-mq) IO path for SCSI.
>
> So, here I am to notify you about the problem and ask you if I can
> help in any way to work it out and fix it.
>
> The problem is that if I copy a file from the internal SATA HD to the
> pendrive, it takes ~10 times to complete (in respect of the time
> needed with the patch reverted).
>
> The test script, which I use to detect if the problem triggers or not,
> is:
>
> #!/bin/bash
> logfile=...
> uname -a | tee -a $logfile
> echo -n "Begin: " | tee -a $logfile
> date | tee -a $logfile
> touch inizio
> SECONDS=0
> mount UUID="05141239-4ea5-494d-aa91-acd67db89ce5" /mnt/pendrive
> cp /NoBackup/buttare/testfile /mnt/pendrive
> umount /mnt/pendrive
> tempo=$SECONDS
> touch fine
> echo -n "...end: " | tee -a $logfile
> date | tee -a $logfile
> echo "It took $tempo seconds!" | tee -a $logfile
>
> If I run the test with a 512MB file it takes >10min vs. half a minute.
>
> The problem is still present in last tested git (cloned today in the
> morning).
>
> You can see the previous discussion that lead to these results at
>
> https://marc.info/?t=155922230700001&r=1&w=2
Hi,
Can you please check what IO scheduler you have set for your USB pendrive?
i.e. with:
cat /sys/block/$DISK/queue/scheduler
Thanks,
Johannes
--
Johannes Thumshirn SUSE Labs Filesystems
jthumshirn@suse.de +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-07-02 10:46 Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6 Andrea Vai
2019-07-02 11:51 ` Johannes Thumshirn
@ 2019-07-02 12:01 ` Ming Lei
2019-07-02 22:39 ` Andrea Vai
1 sibling, 1 reply; 77+ messages in thread
From: Ming Lei @ 2019-07-02 12:01 UTC (permalink / raw)
To: Andrea Vai
Cc: Jens Axboe, linux-usb, linux-scsi, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Alan Stern
On Tue, Jul 02, 2019 at 12:46:45PM +0200, Andrea Vai wrote:
> Hi,
> I have a problem writing data to a USB pendrive, and it seems
> kernel-related. With the help of Greg an Alan (thanks) and some
> bisect, I found out the offending commit being
>
> commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
> Author: Jens Axboe <axboe@kernel.dk>
> Date: Thu Nov 1 16:36:27 2018 -0600
>
> scsi: kill off the legacy IO path
>
> This removes the legacy (non-mq) IO path for SCSI.
>
> So, here I am to notify you about the problem and ask you if I can
> help in any way to work it out and fix it.
>
> The problem is that if I copy a file from the internal SATA HD to the
> pendrive, it takes ~10 times to complete (in respect of the time
> needed with the patch reverted).
>
> The test script, which I use to detect if the problem triggers or not,
> is:
>
> #!/bin/bash
> logfile=...
> uname -a | tee -a $logfile
> echo -n "Begin: " | tee -a $logfile
> date | tee -a $logfile
> touch inizio
> SECONDS=0
> mount UUID="05141239-4ea5-494d-aa91-acd67db89ce5" /mnt/pendrive
> cp /NoBackup/buttare/testfile /mnt/pendrive
> umount /mnt/pendrive
> tempo=$SECONDS
> touch fine
> echo -n "...end: " | tee -a $logfile
> date | tee -a $logfile
> echo "It took $tempo seconds!" | tee -a $logfile
>
> If I run the test with a 512MB file it takes >10min vs. half a minute.
>
> The problem is still present in last tested git (cloned today in the
> morning).
>
> You can see the previous discussion that lead to these results at
>
> https://marc.info/?t=155922230700001&r=1&w=2
One possible reason may be related with too small 'nr_requests', could
you apply the following command and see if any difference can be made?
echo 32 > /sys/block/sdN/queue/nr_requests
Thanks,
Ming
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-07-02 11:51 ` Johannes Thumshirn
@ 2019-07-02 22:36 ` Andrea Vai
2019-07-03 7:29 ` Johannes Thumshirn
0 siblings, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-07-02 22:36 UTC (permalink / raw)
To: Johannes Thumshirn
Cc: Jens Axboe, linux-usb, linux-scsi, Himanshu Madhani,
Hannes Reinecke, Ming Lei, Omar Sandoval, Martin K. Petersen,
Greg KH, Alan Stern
On 02/07/19 13:51:17, Johannes Thumshirn wrote:
> On Tue, Jul 02, 2019 at 12:46:45PM +0200, Andrea Vai wrote:
> > Hi,
> > I have a problem writing data to a USB pendrive, and it seems
> > kernel-related. With the help of Greg an Alan (thanks) and some
> > bisect, I found out the offending commit being
> >
> > commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
> >
> > [...]
>
> Hi,
>
> Can you please check what IO scheduler you have set for your USB pendrive?
>
> i.e. with:
> cat /sys/block/$DISK/queue/scheduler
>
# cat /sys/block/sdf/queue/scheduler
[mq-deadline] none
Many thanks,
Andrea
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-07-02 12:01 ` Ming Lei
@ 2019-07-02 22:39 ` Andrea Vai
2019-07-03 2:01 ` Ming Lei
0 siblings, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-07-02 22:39 UTC (permalink / raw)
To: Ming Lei
Cc: Jens Axboe, linux-usb, linux-scsi, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Alan Stern
On 02/07/19 20:01:13, Ming Lei wrote:
> On Tue, Jul 02, 2019 at 12:46:45PM +0200, Andrea Vai wrote:
> > Hi,
> > I have a problem writing data to a USB pendrive, and it seems
> > kernel-related. With the help of Greg an Alan (thanks) and some
> > bisect, I found out the offending commit being
> >
> > commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
> >
> > [...]
> >
>
> One possible reason may be related with too small 'nr_requests', could
> you apply the following command and see if any difference can be made?
>
> echo 32 > /sys/block/sdN/queue/nr_requests
I applied it (echo 32 > /sys/block/sdf/queue/nr_requests), ran the test again, and still failed. I assumed I didn't have to build the kernel again, did I? (sorry but I am not skilled)
Many thanks,
Andrea
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-07-02 22:39 ` Andrea Vai
@ 2019-07-03 2:01 ` Ming Lei
2019-07-03 5:11 ` Andrea Vai
0 siblings, 1 reply; 77+ messages in thread
From: Ming Lei @ 2019-07-03 2:01 UTC (permalink / raw)
To: Andrea Vai
Cc: Jens Axboe, linux-usb, linux-scsi, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Alan Stern
On Wed, Jul 03, 2019 at 12:39:31AM +0200, Andrea Vai wrote:
> On 02/07/19 20:01:13, Ming Lei wrote:
> > On Tue, Jul 02, 2019 at 12:46:45PM +0200, Andrea Vai wrote:
> > > Hi,
> > > I have a problem writing data to a USB pendrive, and it seems
> > > kernel-related. With the help of Greg an Alan (thanks) and some
> > > bisect, I found out the offending commit being
> > >
> > > commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
> > >
> > > [...]
> > >
> >
> > One possible reason may be related with too small 'nr_requests', could
> > you apply the following command and see if any difference can be made?
> >
> > echo 32 > /sys/block/sdN/queue/nr_requests
>
> I applied it (echo 32 > /sys/block/sdf/queue/nr_requests), ran the test again, and still failed. I assumed I didn't have to build the kernel again, did I? (sorry but I am not skilled)
>
You don't need to build kernel.
I just run same write test on one slow usb drive in my laptop, which
runs '5.1.11-200.fc29.x86_64', and can't reproduce your issue, maybe it
depends on your drive.
Could you collect the queue limits sysfs log via the following command?
find /sys/block/sdN/queue -type f -exec grep -aH . {} \;
Thanks,
Ming
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-07-03 2:01 ` Ming Lei
@ 2019-07-03 5:11 ` Andrea Vai
2019-07-03 6:36 ` Ming Lei
0 siblings, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-07-03 5:11 UTC (permalink / raw)
To: Ming Lei
Cc: Jens Axboe, linux-usb, linux-scsi, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Alan Stern
On 03/07/19 10:01:23, Ming Lei wrote:
> On Wed, Jul 03, 2019 at 12:39:31AM +0200, Andrea Vai wrote:
> > On 02/07/19 20:01:13, Ming Lei wrote:
> > > On Tue, Jul 02, 2019 at 12:46:45PM +0200, Andrea Vai wrote:
> > > > Hi,
> > > > I have a problem writing data to a USB pendrive, and it seems
> > > > kernel-related. With the help of Greg an Alan (thanks) and some
> > > > bisect, I found out the offending commit being
> > > >
> > > > commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
> > > >
> > > > [...]
> > > >
> > >
> > > One possible reason may be related with too small 'nr_requests', could
> > > you apply the following command and see if any difference can be made?
> > >
> > > echo 32 > /sys/block/sdN/queue/nr_requests
> >
> > I applied it (echo 32 > /sys/block/sdf/queue/nr_requests), ran the test again, and still failed. I assumed I didn't have to build the kernel again, did I? (sorry but I am not skilled)
> >
>
> You don't need to build kernel.
>
> I just run same write test on one slow usb drive in my laptop, which
> runs '5.1.11-200.fc29.x86_64', and can't reproduce your issue, maybe it
> depends on your drive.
>
> Could you collect the queue limits sysfs log via the following command?
>
> find /sys/block/sdN/queue -type f -exec grep -aH . {} \;
>
# find /sys/block/sdf/queue -type f -exec grep -aH . {} ;
/sys/block/sdf/queue/io_poll_delay:-1
/sys/block/sdf/queue/max_integrity_segments:0
/sys/block/sdf/queue/zoned:none
/sys/block/sdf/queue/scheduler:[mq-deadline] none
/sys/block/sdf/queue/io_poll:0
/sys/block/sdf/queue/discard_zeroes_data:0
/sys/block/sdf/queue/minimum_io_size:512
/sys/block/sdf/queue/nr_zones:0
/sys/block/sdf/queue/write_same_max_bytes:0
/sys/block/sdf/queue/max_segments:2048
/sys/block/sdf/queue/dax:0
/sys/block/sdf/queue/physical_block_size:512
/sys/block/sdf/queue/logical_block_size:512
/sys/block/sdf/queue/io_timeout:30000
/sys/block/sdf/queue/nr_requests:2
/sys/block/sdf/queue/write_cache:write through/sys/block/sdf/queue/max_segment_size:4294967295
/sys/block/sdf/queue/rotational:1
/sys/block/sdf/queue/discard_max_bytes:0
/sys/block/sdf/queue/add_random:1
/sys/block/sdf/queue/discard_max_hw_bytes:0
/sys/block/sdf/queue/optimal_io_size:0
/sys/block/sdf/queue/chunk_sectors:0
/sys/block/sdf/queue/iosched/front_merges:1
/sys/block/sdf/queue/iosched/read_expire:500
/sys/block/sdf/queue/iosched/fifo_batch:16
/sys/block/sdf/queue/iosched/write_expire:5000/sys/block/sdf/queue/iosched/writes_starved:2
/sys/block/sdf/queue/read_ahead_kb:128
/sys/block/sdf/queue/max_discard_segments:1
/sys/block/sdf/queue/write_zeroes_max_bytes:0
/sys/block/sdf/queue/nomerges:0
/sys/block/sdf/queue/wbt_lat_usec:75000
/sys/block/sdf/queue/fua:0
/sys/block/sdf/queue/discard_granularity:0
/sys/block/sdf/queue/rq_affinity:1
/sys/block/sdf/queue/max_sectors_kb:120
/sys/block/sdf/queue/hw_sector_size:512
/sys/block/sdf/queue/max_hw_sectors_kb:120
/sys/block/sdf/queue/iostats:1
Thanks, and bye,
Andrea
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-07-03 5:11 ` Andrea Vai
@ 2019-07-03 6:36 ` Ming Lei
2019-07-03 15:27 ` Chris Murphy
2019-07-06 9:33 ` Andrea Vai
0 siblings, 2 replies; 77+ messages in thread
From: Ming Lei @ 2019-07-03 6:36 UTC (permalink / raw)
To: Andrea Vai
Cc: Jens Axboe, linux-usb, linux-scsi, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Alan Stern
On Wed, Jul 03, 2019 at 07:11:17AM +0200, Andrea Vai wrote:
> On 03/07/19 10:01:23, Ming Lei wrote:
> > On Wed, Jul 03, 2019 at 12:39:31AM +0200, Andrea Vai wrote:
> > > On 02/07/19 20:01:13, Ming Lei wrote:
> > > > On Tue, Jul 02, 2019 at 12:46:45PM +0200, Andrea Vai wrote:
> > > > > Hi,
> > > > > I have a problem writing data to a USB pendrive, and it seems
> > > > > kernel-related. With the help of Greg an Alan (thanks) and some
> > > > > bisect, I found out the offending commit being
> > > > >
> > > > > commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
> > > > >
> > > > > [...]
> > > > >
> > > >
> > > > One possible reason may be related with too small 'nr_requests', could
> > > > you apply the following command and see if any difference can be made?
> > > >
> > > > echo 32 > /sys/block/sdN/queue/nr_requests
> > >
> > > I applied it (echo 32 > /sys/block/sdf/queue/nr_requests), ran the test again, and still failed. I assumed I didn't have to build the kernel again, did I? (sorry but I am not skilled)
> > >
> >
> > You don't need to build kernel.
> >
> > I just run same write test on one slow usb drive in my laptop, which
> > runs '5.1.11-200.fc29.x86_64', and can't reproduce your issue, maybe it
> > depends on your drive.
> >
> > Could you collect the queue limits sysfs log via the following command?
> >
> > find /sys/block/sdN/queue -type f -exec grep -aH . {} \;
> >
>
> # find /sys/block/sdf/queue -type f -exec grep -aH . {} ;
> /sys/block/sdf/queue/io_poll_delay:-1
> /sys/block/sdf/queue/max_integrity_segments:0
> /sys/block/sdf/queue/zoned:none
> /sys/block/sdf/queue/scheduler:[mq-deadline] none
> /sys/block/sdf/queue/io_poll:0
> /sys/block/sdf/queue/discard_zeroes_data:0
> /sys/block/sdf/queue/minimum_io_size:512
> /sys/block/sdf/queue/nr_zones:0
> /sys/block/sdf/queue/write_same_max_bytes:0
> /sys/block/sdf/queue/max_segments:2048
> /sys/block/sdf/queue/dax:0
> /sys/block/sdf/queue/physical_block_size:512
> /sys/block/sdf/queue/logical_block_size:512
> /sys/block/sdf/queue/io_timeout:30000
> /sys/block/sdf/queue/nr_requests:2
> /sys/block/sdf/queue/write_cache:write through/sys/block/sdf/queue/max_segment_size:4294967295
> /sys/block/sdf/queue/rotational:1
> /sys/block/sdf/queue/discard_max_bytes:0
> /sys/block/sdf/queue/add_random:1
> /sys/block/sdf/queue/discard_max_hw_bytes:0
> /sys/block/sdf/queue/optimal_io_size:0
> /sys/block/sdf/queue/chunk_sectors:0
> /sys/block/sdf/queue/iosched/front_merges:1
> /sys/block/sdf/queue/iosched/read_expire:500
> /sys/block/sdf/queue/iosched/fifo_batch:16
> /sys/block/sdf/queue/iosched/write_expire:5000/sys/block/sdf/queue/iosched/writes_starved:2
> /sys/block/sdf/queue/read_ahead_kb:128
> /sys/block/sdf/queue/max_discard_segments:1
> /sys/block/sdf/queue/write_zeroes_max_bytes:0
> /sys/block/sdf/queue/nomerges:0
> /sys/block/sdf/queue/wbt_lat_usec:75000
> /sys/block/sdf/queue/fua:0
> /sys/block/sdf/queue/discard_granularity:0
> /sys/block/sdf/queue/rq_affinity:1
> /sys/block/sdf/queue/max_sectors_kb:120
> /sys/block/sdf/queue/hw_sector_size:512
> /sys/block/sdf/queue/max_hw_sectors_kb:120
> /sys/block/sdf/queue/iostats:1
The above is basically same with my USB drive's queue setting, and looks
all are fine.
BTW, 'rotational' shouldn't be set for USB drive, except for USB HDD,
but that shouldn't be related with your issue.
Then could you install bcc package and collect the IO trace?
sudo /usr/share/bcc/tools/biosnoop | grep sdN
sdN is your USB disk device name.
Thanks,
Ming
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-07-02 22:36 ` Andrea Vai
@ 2019-07-03 7:29 ` Johannes Thumshirn
2019-07-03 14:23 ` Alan Stern
0 siblings, 1 reply; 77+ messages in thread
From: Johannes Thumshirn @ 2019-07-03 7:29 UTC (permalink / raw)
To: Andrea Vai
Cc: Jens Axboe, linux-usb, linux-scsi, Himanshu Madhani,
Hannes Reinecke, Ming Lei, Omar Sandoval, Martin K. Petersen,
Greg KH, Alan Stern
On Wed, Jul 03, 2019 at 12:36:30AM +0200, Andrea Vai wrote:
> On 02/07/19 13:51:17, Johannes Thumshirn wrote:
> > On Tue, Jul 02, 2019 at 12:46:45PM +0200, Andrea Vai wrote:
> > > Hi,
> > > I have a problem writing data to a USB pendrive, and it seems
> > > kernel-related. With the help of Greg an Alan (thanks) and some
> > > bisect, I found out the offending commit being
> > >
> > > commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
> > >
> > > [...]
> >
> > Hi,
> >
> > Can you please check what IO scheduler you have set for your USB pendrive?
> >
> > i.e. with:
> > cat /sys/block/$DISK/queue/scheduler
> >
>
> # cat /sys/block/sdf/queue/scheduler
> [mq-deadline] none
One thing you can try as well is building a kernel with CONFIG_IOSCHED_BFQ and
use it. Deadline is probably not the best choice for a slow drive.
Byte,
Johannes
--
Johannes Thumshirn SUSE Labs Filesystems
jthumshirn@suse.de +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-07-03 7:29 ` Johannes Thumshirn
@ 2019-07-03 14:23 ` Alan Stern
2019-07-06 22:06 ` Andrea Vai
0 siblings, 1 reply; 77+ messages in thread
From: Alan Stern @ 2019-07-03 14:23 UTC (permalink / raw)
To: Johannes Thumshirn
Cc: Andrea Vai, Jens Axboe, linux-usb, linux-scsi, Himanshu Madhani,
Hannes Reinecke, Ming Lei, Omar Sandoval, Martin K. Petersen,
Greg KH
On Wed, 3 Jul 2019, Johannes Thumshirn wrote:
> On Wed, Jul 03, 2019 at 12:36:30AM +0200, Andrea Vai wrote:
> > On 02/07/19 13:51:17, Johannes Thumshirn wrote:
> > > On Tue, Jul 02, 2019 at 12:46:45PM +0200, Andrea Vai wrote:
> > > > Hi,
> > > > I have a problem writing data to a USB pendrive, and it seems
> > > > kernel-related. With the help of Greg an Alan (thanks) and some
> > > > bisect, I found out the offending commit being
> > > >
> > > > commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
> > > >
> > > > [...]
> > >
> > > Hi,
> > >
> > > Can you please check what IO scheduler you have set for your USB pendrive?
> > >
> > > i.e. with:
> > > cat /sys/block/$DISK/queue/scheduler
> > >
> >
> > # cat /sys/block/sdf/queue/scheduler
> > [mq-deadline] none
>
> One thing you can try as well is building a kernel with CONFIG_IOSCHED_BFQ and
> use it. Deadline is probably not the best choice for a slow drive.
Andrea, another thing you could try is to collect a usbmon trace under
one of the "slow" kernels. Follow the instructions in
Documentation/usb/usbmon.txt. I think you could kill the file-copy
operation after just a couple of seconds; that should provide enough
trace information to help see what causes the slowdown.
(If you want, do the same test with a "fast" kernel and then we'll
compare the results.)
Alan Stern
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-07-03 6:36 ` Ming Lei
@ 2019-07-03 15:27 ` Chris Murphy
2019-07-06 9:33 ` Andrea Vai
1 sibling, 0 replies; 77+ messages in thread
From: Chris Murphy @ 2019-07-03 15:27 UTC (permalink / raw)
To: Ming Lei
Cc: Andrea Vai, Jens Axboe, linux-usb, linux-scsi, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Alan Stern
On Wed, Jul 3, 2019 at 12:36 AM Ming Lei <ming.lei@redhat.com> wrote:
>
> BTW, 'rotational' shouldn't be set for USB drive, except for USB HDD,
> but that shouldn't be related with your issue.
I get
/sys/block/sdb/queue/rotational:1
for all USB flash drives on all computers with every kernel since
forever, including 5.2rc7. Lexar, Samsung, Kingston.
--
Chris Murphy
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-07-03 6:36 ` Ming Lei
2019-07-03 15:27 ` Chris Murphy
@ 2019-07-06 9:33 ` Andrea Vai
2019-07-08 1:01 ` Ming Lei
1 sibling, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-07-06 9:33 UTC (permalink / raw)
To: Ming Lei
Cc: Jens Axboe, linux-usb, linux-scsi, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Alan Stern
On 03/07/19 14:36:05, Ming Lei wrote:
> On Wed, Jul 03, 2019 at 07:11:17AM +0200, Andrea Vai wrote:
> > On 03/07/19 10:01:23, Ming Lei wrote:
> > > On Wed, Jul 03, 2019 at 12:39:31AM +0200, Andrea Vai wrote:
> > > > On 02/07/19 20:01:13, Ming Lei wrote:
> > > > > On Tue, Jul 02, 2019 at 12:46:45PM +0200, Andrea Vai wrote:
> > > > > > Hi,
> > > > > > I have a problem writing data to a USB pendrive, and it seems
> > > > > > kernel-related. With the help of Greg an Alan (thanks) and some
> > > > > > bisect, I found out the offending commit being
> > > > > >
> > > > > > commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
> > > > > >
> > > > > > [...]
> > > > > >
> [...]
> Then could you install bcc package and collect the IO trace?
>
> sudo /usr/share/bcc/tools/biosnoop | grep sdN
>
> sdN is your USB disk device name.
The command runs forever (or at least for hours) without giving any output through "|grep sdf". The device is connected, but not mounted. Maybe I should run the command with the device mounted? Or while performing the test?
The command itself seems to work, as /usr/share/bcc/tools/biosnoop | tee -a biosnoop.txt produces an output file sized about some MB in some hours.
What should I do?
Thanks,
Andrea
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-07-03 14:23 ` Alan Stern
@ 2019-07-06 22:06 ` Andrea Vai
2019-07-08 15:38 ` Alan Stern
0 siblings, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-07-06 22:06 UTC (permalink / raw)
To: Alan Stern
Cc: Johannes Thumshirn, Jens Axboe, linux-usb, linux-scsi,
Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
Martin K. Petersen, Greg KH
[-- Attachment #1: Type: text/plain, Size: 768 bytes --]
On 03/07/19 10:23:13, Alan Stern wrote:
>
> [...]
> Andrea, another thing you could try is to collect a usbmon trace under
> one of the "slow" kernels. Follow the instructions in
> Documentation/usb/usbmon.txt. I think you could kill the file-copy
> operation after just a couple of seconds; that should provide enough
> trace information to help see what causes the slowdown.
>
> (If you want, do the same test with a "fast" kernel and then we'll
> compare the results.)
>
> Alan Stern
>
Thanks Alan,
so I attach two ouputs, one for a "good" and one for a "bad" kernel.
Both killed after roughly 20 seconds, using the command
cat /sys/kernel/debug/usb/usbmon/6u > file
(my pendrive turns out to be attached to bus #6)
Hope it helps.
Thanks,
Andrea
[-- Attachment #2: GOOD_4.20.0-rc1+.mon.out.tar --]
[-- Type: application/x-tar, Size: 66712 bytes --]
[-- Attachment #3: BAD_5.2.0-rc7.mon.out.tar --]
[-- Type: application/x-tar, Size: 9548 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-07-06 9:33 ` Andrea Vai
@ 2019-07-08 1:01 ` Ming Lei
2019-07-09 21:18 ` Andrea Vai
0 siblings, 1 reply; 77+ messages in thread
From: Ming Lei @ 2019-07-08 1:01 UTC (permalink / raw)
To: Andrea Vai
Cc: Jens Axboe, linux-usb, linux-scsi, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Alan Stern
On Sat, Jul 06, 2019 at 11:33:27AM +0200, Andrea Vai wrote:
> On 03/07/19 14:36:05, Ming Lei wrote:
> > On Wed, Jul 03, 2019 at 07:11:17AM +0200, Andrea Vai wrote:
> > > On 03/07/19 10:01:23, Ming Lei wrote:
> > > > On Wed, Jul 03, 2019 at 12:39:31AM +0200, Andrea Vai wrote:
> > > > > On 02/07/19 20:01:13, Ming Lei wrote:
> > > > > > On Tue, Jul 02, 2019 at 12:46:45PM +0200, Andrea Vai wrote:
> > > > > > > Hi,
> > > > > > > I have a problem writing data to a USB pendrive, and it seems
> > > > > > > kernel-related. With the help of Greg an Alan (thanks) and some
> > > > > > > bisect, I found out the offending commit being
> > > > > > >
> > > > > > > commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
> > > > > > >
> > > > > > > [...]
> > > > > > >
> > [...]
> > Then could you install bcc package and collect the IO trace?
> >
> > sudo /usr/share/bcc/tools/biosnoop | grep sdN
> >
> > sdN is your USB disk device name.
>
> The command runs forever (or at least for hours) without giving any output through "|grep sdf". The device is connected, but not mounted. Maybe I should run the command with the device mounted? Or while performing the test?
> The command itself seems to work, as /usr/share/bcc/tools/biosnoop | tee -a biosnoop.txt produces an output file sized about some MB in some hours.
>
> What should I do?
1) run the bcc biosnoop trace in one terminal after mounting the fs on the USB dirve
2) start the write test in another teminal
3) wait for 10 seconds, and stop the bcc trace via ctrl^ + C, then post the bcc biosnoop
trace log
Thanks,
Ming
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-07-06 22:06 ` Andrea Vai
@ 2019-07-08 15:38 ` Alan Stern
0 siblings, 0 replies; 77+ messages in thread
From: Alan Stern @ 2019-07-08 15:38 UTC (permalink / raw)
To: Andrea Vai
Cc: Johannes Thumshirn, Jens Axboe, linux-usb, linux-scsi,
Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
Martin K. Petersen, Greg KH
On Sun, 7 Jul 2019, Andrea Vai wrote:
> On 03/07/19 10:23:13, Alan Stern wrote:
> >
> > [...]
> > Andrea, another thing you could try is to collect a usbmon trace under
> > one of the "slow" kernels. Follow the instructions in
> > Documentation/usb/usbmon.txt. I think you could kill the file-copy
> > operation after just a couple of seconds; that should provide enough
> > trace information to help see what causes the slowdown.
> >
> > (If you want, do the same test with a "fast" kernel and then we'll
> > compare the results.)
> >
> > Alan Stern
> >
>
> Thanks Alan,
> so I attach two ouputs, one for a "good" and one for a "bad" kernel.
>
> Both killed after roughly 20 seconds, using the command
>
> cat /sys/kernel/debug/usb/usbmon/6u > file
>
> (my pendrive turns out to be attached to bus #6)
>
> Hope it helps.
I don't know what the results mean, but I _can_ tell you what's
happening. Every so often (at intervals of about a second) the pen
drive completely stops communicating with the "bad" kernel for about
one second and then starts up again.
Here's a short example from the "bad" trace:
ffff9169f0d399c0 513072808 S Bo:6:008:2 -115 122880 = 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
ffff9169f0d399c0 514262176 C Bo:6:008:2 0 122880 >
The second column is a timestamp (in microseconds). This little
extract shows a 120-KB write starting at time 513.072 and ending at
514.262, more than a second later. Normally such a write would
complete in about 0.06 s.
The cumulative effect of all these delays is to slow the transfer
drastically. The "good" kernel trace shows a few delays like this, but
only one or two.
I have no idea how commit f664a3cc17b7, getting rid of the legacy IO
path, could have caused these delays. It seems more likely that the
pen drive itself is the cause, perhaps because it is flushing buffers
more often under the "bad" kernel.
I'd like you to try doing another pair of usbmon tests. This time,
start collecting the usbmon trace _before_ you plug in the pen drive,
and stop the trace shortly after the pen drive has been mounted.
Don't try to transfer any data. Perhaps the two kernels are
initializing the pen drive with different settings and that accounts
for the different behaviors.
Alan Stern
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-07-08 1:01 ` Ming Lei
@ 2019-07-09 21:18 ` Andrea Vai
2019-07-10 2:44 ` Ming Lei
0 siblings, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-07-09 21:18 UTC (permalink / raw)
To: Ming Lei
Cc: Jens Axboe, linux-usb, linux-scsi, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Alan Stern
[-- Attachment #1: Type: text/plain, Size: 442 bytes --]
On 08/07/19 09:01:35, Ming Lei wrote:
> > > > > > > >
> > > > > > > > commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
> > > > > > > >
> > > > > > > > [...]
>
> 1) run the bcc biosnoop trace in one terminal after mounting the fs on the USB dirve
>
> 2) start the write test in another teminal
>
> 3) wait for 10 seconds, and stop the bcc trace via ctrl^ + C, then post the bcc biosnoop
> trace log
>
Done, attached.
Thanks,
Andrea
[-- Attachment #2: biosnoop_sdf.txt --]
[-- Type: text/plain, Size: 12210 bytes --]
5.420198000 jbd2/sdf1-8 6775 sdf W 29624320 4096 0.37
5.423601000 jbd2/sdf1-8 6775 sdf W 29624328 12288 3.29
5.424009000 jbd2/sdf1-8 6775 sdf W 29624352 4096 0.30
6.665732000 kworker/u8:0 6401 sdf R 10248 4096 841.64
6.666891000 kworker/u8:0 6401 sdf R 10256 4096 0.98
6.668141000 kworker/u8:0 6401 sdf R 4196352 4096 1.07
6.669312000 kworker/u8:0 6401 sdf R 8390656 4096 0.99
6.670568000 kworker/u8:0 6401 sdf R 12584960 4096 1.06
6.674257000 kworker/u8:0 6401 sdf R 16779264 4096 3.50
6.675510000 kworker/u8:0 6401 sdf R 20973568 4096 1.06
6.678344000 kworker/u8:0 6401 sdf R 25167872 4096 2.64
6.680384000 kworker/u8:0 6401 sdf R 29362176 4096 1.83
6.681622000 kworker/u8:0 6401 sdf R 33556480 4096 1.03
6.682713000 kworker/u8:0 6401 sdf R 37750784 4096 0.92
6.686360000 kworker/u8:0 6401 sdf R 41945088 4096 3.48
6.687589000 kworker/u8:0 6401 sdf R 46139392 4096 1.04
6.688639000 kworker/u8:0 6401 sdf R 46139424 4096 0.88
6.766336000 kworker/u8:0 6401 sdf W 47351808 122880 77.17
6.773563000 kworker/u8:0 6401 sdf W 47352048 122880 7.08
6.780980000 kworker/u8:0 6401 sdf W 47352288 122880 7.29
6.786366000 kworker/u8:0 6401 sdf W 47352528 122880 5.27
6.793831000 kworker/u8:0 6401 sdf W 47352768 122880 7.38
6.801085000 kworker/u8:0 6401 sdf W 47353008 122880 7.20
6.808405000 kworker/u8:0 6401 sdf W 47353248 122880 7.27
6.813799000 kworker/u8:0 6401 sdf W 47353488 122880 5.35
6.821103000 kworker/u8:0 6401 sdf W 47353728 118784 7.25
6.828459000 kworker/u8:0 6401 sdf W 47353960 122880 7.09
6.835722000 kworker/u8:0 6401 sdf W 47354200 122880 7.20
6.842813000 kworker/u8:0 6401 sdf W 47354440 122880 7.05
6.850184000 kworker/u8:0 6401 sdf W 47354680 122880 7.32
6.857452000 kworker/u8:0 6401 sdf W 47354920 122880 7.22
6.864874000 kworker/u8:0 6401 sdf W 47355160 122880 7.38
6.871841000 kworker/u8:0 6401 sdf W 47355400 122880 6.92
6.879380000 kworker/u8:0 6401 sdf W 47355640 122880 7.50
6.886497000 kworker/u8:0 6401 sdf W 47355880 122880 7.04
6.893940000 kworker/u8:0 6401 sdf W 47356120 122880 7.40
6.899304000 kworker/u8:0 6401 sdf W 47356360 122880 5.33
6.906635000 kworker/u8:0 6401 sdf W 47356600 122880 7.29
6.913957000 kworker/u8:0 6401 sdf W 47356840 122880 7.29
6.921308000 kworker/u8:0 6401 sdf W 47357080 122880 7.31
6.924391000 kworker/u8:0 6401 sdf W 47357320 53248 3.03
6.931971000 kworker/u8:0 6401 sdf W 47357424 122880 7.22
6.939169000 kworker/u8:0 6401 sdf W 47357664 122880 7.14
6.944985000 kworker/u8:0 6401 sdf W 47357904 122880 5.76
6.952113000 kworker/u8:0 6401 sdf W 47358144 122880 7.06
6.959608000 kworker/u8:0 6401 sdf W 47358384 122880 7.45
6.966820000 kworker/u8:0 6401 sdf W 47358624 122880 7.16
6.972393000 kworker/u8:0 6401 sdf W 47358864 122880 5.53
6.979550000 kworker/u8:0 6401 sdf W 47359104 122880 7.09
6.986879000 kworker/u8:0 6401 sdf W 47359344 122880 7.28
6.994306000 kworker/u8:0 6401 sdf W 47359584 122880 7.40
7.001338000 kworker/u8:0 6401 sdf W 47359824 122880 6.98
7.008740000 kworker/u8:0 6401 sdf W 47360064 122880 7.37
7.016114000 kworker/u8:0 6401 sdf W 47360304 122880 7.31
7.023546000 kworker/u8:0 6401 sdf W 47360544 122880 7.39
7.030437000 kworker/u8:0 6401 sdf W 47360784 122880 6.82
7.037834000 kworker/u8:0 6401 sdf W 47361024 122880 7.35
7.045224000 kworker/u8:0 6401 sdf W 47361264 122880 7.32
7.052687000 kworker/u8:0 6401 sdf W 47361504 122880 7.39
7.057977000 kworker/u8:0 6401 sdf W 47361744 122880 5.01
7.065390000 kworker/u8:0 6401 sdf W 47361984 122880 7.35
7.072749000 kworker/u8:0 6401 sdf W 47362224 122880 7.28
7.080174000 kworker/u8:0 6401 sdf W 47362464 122880 7.36
7.085335000 kworker/u8:0 6401 sdf W 47362704 122880 5.08
7.092881000 kworker/u8:0 6401 sdf W 47362944 122880 7.50
7.100197000 kworker/u8:0 6401 sdf W 47363184 122880 7.23
7.107507000 kworker/u8:0 6401 sdf W 47363424 122880 7.24
7.114684000 kworker/u8:0 6401 sdf W 47363664 122880 7.09
7.121936000 kworker/u8:0 6401 sdf W 47363904 122880 7.18
7.129299000 kworker/u8:0 6401 sdf W 47364144 122880 7.29
7.136645000 kworker/u8:0 6401 sdf W 47364384 122880 7.27
7.143733000 kworker/u8:0 6401 sdf W 47364624 122880 7.02
7.151112000 kworker/u8:0 6401 sdf W 47364864 122880 7.31
7.158509000 kworker/u8:0 6401 sdf W 47365104 122880 7.33
7.165735000 kworker/u8:0 6401 sdf W 47365344 122880 7.14
7.171205000 kworker/u8:0 6401 sdf W 47365584 122880 5.40
7.178625000 kworker/u8:0 6401 sdf W 47365824 122880 7.37
7.185980000 kworker/u8:0 6401 sdf W 47366064 122880 7.31
7.193593000 kworker/u8:0 6401 sdf W 47366304 122880 7.57
7.198674000 kworker/u8:0 6401 sdf W 47366544 122880 5.02
7.205996000 kworker/u8:0 6401 sdf W 47366784 122880 7.25
7.213419000 kworker/u8:0 6401 sdf W 47367024 122880 7.37
7.221005000 kworker/u8:0 6401 sdf W 47367264 122880 7.52
7.227906000 kworker/u8:0 6401 sdf W 47367504 122880 6.83
7.235251000 kworker/u8:0 6401 sdf W 47367744 122880 7.28
7.241892000 kworker/u8:0 6401 sdf W 47367984 106496 6.59
7.249617000 kworker/u8:0 6401 sdf W 47368192 122880 6.88
7.256892000 kworker/u8:0 6401 sdf W 47368432 122880 7.20
7.264223000 kworker/u8:0 6401 sdf W 47368672 122880 7.26
7.269672000 kworker/u8:0 6401 sdf W 47368912 122880 5.39
7.277066000 kworker/u8:0 6401 sdf W 47369152 122880 7.36
7.284630000 kworker/u8:0 6401 sdf W 47369392 122880 7.52
7.291891000 kworker/u8:0 6401 sdf W 47369632 122880 7.20
7.297211000 kworker/u8:0 6401 sdf W 47369872 122880 5.29
7.304665000 kworker/u8:0 6401 sdf W 47370112 122880 7.41
7.312050000 kworker/u8:0 6401 sdf W 47370352 122880 7.33
7.319520000 kworker/u8:0 6401 sdf W 47370592 122880 7.41
7.326494000 kworker/u8:0 6401 sdf W 47370832 122880 6.89
7.334082000 kworker/u8:0 6401 sdf W 47371072 122880 7.52
7.341505000 kworker/u8:0 6401 sdf W 47371312 122880 7.34
7.348600000 kworker/u8:0 6401 sdf W 47371552 122880 7.01
7.355806000 kworker/u8:0 6401 sdf W 47371792 122880 7.16
7.363147000 kworker/u8:0 6401 sdf W 47372032 122880 7.27
7.370377000 kworker/u8:0 6401 sdf W 47372272 122880 7.16
7.377770000 kworker/u8:0 6401 sdf W 47372512 122880 7.36
7.383323000 kworker/u8:0 6401 sdf W 47372752 122880 5.50
7.390651000 kworker/u8:0 6401 sdf W 47372992 122880 7.27
7.398245000 kworker/u8:0 6401 sdf W 47373232 122880 7.52
7.405347000 kworker/u8:0 6401 sdf W 47373472 122880 7.04
7.410761000 kworker/u8:0 6401 sdf W 47373712 122880 5.36
7.418075000 kworker/u8:0 6401 sdf W 47373952 122880 7.26
7.425451000 kworker/u8:0 6401 sdf W 47374192 122880 7.33
7.432864000 kworker/u8:0 6401 sdf W 47374432 122880 7.36
7.439913000 kworker/u8:0 6401 sdf W 47374672 122880 6.99
7.447312000 kworker/u8:0 6401 sdf W 47374912 122880 7.35
7.454721000 kworker/u8:0 6401 sdf W 47375152 122880 7.35
7.461955000 kworker/u8:0 6401 sdf W 47375392 122880 7.19
7.469040000 kworker/u8:0 6401 sdf W 47375632 122880 7.04
7.476416000 kworker/u8:0 6401 sdf W 47375872 122880 7.31
7.483856000 kworker/u8:0 6401 sdf W 47376112 122880 7.38
7.491270000 kworker/u8:0 6401 sdf W 47376352 122880 7.37
7.496452000 kworker/u8:0 6401 sdf W 47376592 122880 5.12
7.504011000 kworker/u8:0 6401 sdf W 47376832 122880 7.52
7.511217000 kworker/u8:0 6401 sdf W 47377072 122880 7.12
7.518511000 kworker/u8:0 6401 sdf W 47377312 122880 7.23
7.523956000 kworker/u8:0 6401 sdf W 47377552 122880 5.41
7.531197000 kworker/u8:0 6401 sdf W 47377792 122880 7.21
7.538546000 kworker/u8:0 6401 sdf W 47378032 122880 7.31
7.545757000 kworker/u8:0 6401 sdf W 47378272 122880 7.17
7.552919000 kworker/u8:0 6401 sdf W 47378512 122880 7.13
7.560269000 kworker/u8:0 6401 sdf W 47378752 122880 7.31
7.567555000 kworker/u8:0 6401 sdf W 47378992 122880 7.25
7.574863000 kworker/u8:0 6401 sdf W 47379232 122880 7.27
7.581802000 kworker/u8:0 6401 sdf W 47379472 122880 6.87
7.588979000 kworker/u8:0 6401 sdf W 47379712 122880 7.13
7.596343000 kworker/u8:0 6401 sdf W 47379952 122880 7.30
7.603224000 kworker/u8:0 6401 sdf W 47380192 122880 6.84
7.608655000 kworker/u8:0 6401 sdf W 47380432 122880 5.38
7.615811000 kworker/u8:0 6401 sdf W 47380672 122880 7.12
7.623028000 kworker/u8:0 6401 sdf W 47380912 122880 7.18
7.630413000 kworker/u8:0 6401 sdf W 47381152 122880 7.34
7.635223000 kworker/u8:0 6401 sdf W 47381392 122880 4.75
7.642268000 kworker/u8:0 6401 sdf W 47381632 122880 6.98
7.649383000 kworker/u8:0 6401 sdf W 47381872 122880 7.04
7.656619000 kworker/u8:0 6401 sdf W 47382112 122880 7.19
7.661918000 kworker/u8:0 6401 sdf W 47382352 122880 5.25
8.268039000 kworker/u8:0 6401 sdf W 47382592 122880 606.07
8.280414000 kworker/u8:0 6401 sdf W 47382832 122880 12.30
8.286503000 kworker/u8:0 6401 sdf W 47383072 122880 6.02
8.293627000 kworker/u8:0 6401 sdf W 47383312 122880 7.07
8.300777000 kworker/u8:0 6401 sdf W 47383552 122880 7.08
8.307733000 kworker/u8:0 6401 sdf W 47383792 122880 6.89
8.315212000 kworker/u8:0 6401 sdf W 47384032 122880 7.43
8.320697000 kworker/u8:0 6401 sdf W 47384272 122880 5.41
8.323335000 kworker/u8:0 6401 sdf W 47384512 32768 2.58
8.331116000 kworker/u8:0 6401 sdf W 47384576 122880 6.58
8.338586000 kworker/u8:0 6401 sdf W 47384816 122880 7.40
8.346113000 kworker/u8:0 6401 sdf W 47385056 122880 7.46
8.351558000 kworker/u8:0 6401 sdf W 47385296 122880 5.37
8.359030000 kworker/u8:0 6401 sdf W 47385536 122880 7.43
8.366425000 kworker/u8:0 6401 sdf W 47385776 122880 7.34
8.373712000 kworker/u8:0 6401 sdf W 47386016 122880 7.22
8.940689000 cp 6828 sdf W 47400960 122880 566.92
9.614647000 kworker/u8:0 6401 sdf W 47386256 122880 673.87
10.179120000 cp 6828 sdf W 47401200 122880 564.38
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-07-09 21:18 ` Andrea Vai
@ 2019-07-10 2:44 ` Ming Lei
0 siblings, 0 replies; 77+ messages in thread
From: Ming Lei @ 2019-07-10 2:44 UTC (permalink / raw)
To: Andrea Vai
Cc: Jens Axboe, linux-usb, linux-scsi, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Alan Stern
On Tue, Jul 09, 2019 at 11:18:38PM +0200, Andrea Vai wrote:
> On 08/07/19 09:01:35, Ming Lei wrote:
> > > > > > > > >
> > > > > > > > > commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
> > > > > > > > >
> > > > > > > > > [...]
> >
> > 1) run the bcc biosnoop trace in one terminal after mounting the fs on the USB dirve
> >
> > 2) start the write test in another teminal
> >
> > 3) wait for 10 seconds, and stop the bcc trace via ctrl^ + C, then post the bcc biosnoop
> > trace log
> >
>
> Done, attached.
>
> Thanks,
> Andrea
> 5.420198000 jbd2/sdf1-8 6775 sdf W 29624320 4096 0.37
> 5.423601000 jbd2/sdf1-8 6775 sdf W 29624328 12288 3.29
> 5.424009000 jbd2/sdf1-8 6775 sdf W 29624352 4096 0.30
> 6.665732000 kworker/u8:0 6401 sdf R 10248 4096 841.64
> 6.666891000 kworker/u8:0 6401 sdf R 10256 4096 0.98
> 6.668141000 kworker/u8:0 6401 sdf R 4196352 4096 1.07
> 6.669312000 kworker/u8:0 6401 sdf R 8390656 4096 0.99
> 6.670568000 kworker/u8:0 6401 sdf R 12584960 4096 1.06
> 6.674257000 kworker/u8:0 6401 sdf R 16779264 4096 3.50
> 6.675510000 kworker/u8:0 6401 sdf R 20973568 4096 1.06
> 6.678344000 kworker/u8:0 6401 sdf R 25167872 4096 2.64
> 6.680384000 kworker/u8:0 6401 sdf R 29362176 4096 1.83
> 6.681622000 kworker/u8:0 6401 sdf R 33556480 4096 1.03
> 6.682713000 kworker/u8:0 6401 sdf R 37750784 4096 0.92
> 6.686360000 kworker/u8:0 6401 sdf R 41945088 4096 3.48
> 6.687589000 kworker/u8:0 6401 sdf R 46139392 4096 1.04
> 6.688639000 kworker/u8:0 6401 sdf R 46139424 4096 0.88
> 6.766336000 kworker/u8:0 6401 sdf W 47351808 122880 77.17
> 6.773563000 kworker/u8:0 6401 sdf W 47352048 122880 7.08
> 6.780980000 kworker/u8:0 6401 sdf W 47352288 122880 7.29
> 6.786366000 kworker/u8:0 6401 sdf W 47352528 122880 5.27
> 6.793831000 kworker/u8:0 6401 sdf W 47352768 122880 7.38
> 6.801085000 kworker/u8:0 6401 sdf W 47353008 122880 7.20
> 6.808405000 kworker/u8:0 6401 sdf W 47353248 122880 7.27
> 6.813799000 kworker/u8:0 6401 sdf W 47353488 122880 5.35
> 6.821103000 kworker/u8:0 6401 sdf W 47353728 118784 7.25
> 6.828459000 kworker/u8:0 6401 sdf W 47353960 122880 7.09
> 6.835722000 kworker/u8:0 6401 sdf W 47354200 122880 7.20
> 6.842813000 kworker/u8:0 6401 sdf W 47354440 122880 7.05
> 6.850184000 kworker/u8:0 6401 sdf W 47354680 122880 7.32
> 6.857452000 kworker/u8:0 6401 sdf W 47354920 122880 7.22
> 6.864874000 kworker/u8:0 6401 sdf W 47355160 122880 7.38
> 6.871841000 kworker/u8:0 6401 sdf W 47355400 122880 6.92
> 6.879380000 kworker/u8:0 6401 sdf W 47355640 122880 7.50
> 6.886497000 kworker/u8:0 6401 sdf W 47355880 122880 7.04
> 6.893940000 kworker/u8:0 6401 sdf W 47356120 122880 7.40
> 6.899304000 kworker/u8:0 6401 sdf W 47356360 122880 5.33
> 6.906635000 kworker/u8:0 6401 sdf W 47356600 122880 7.29
> 6.913957000 kworker/u8:0 6401 sdf W 47356840 122880 7.29
> 6.921308000 kworker/u8:0 6401 sdf W 47357080 122880 7.31
> 6.924391000 kworker/u8:0 6401 sdf W 47357320 53248 3.03
> 6.931971000 kworker/u8:0 6401 sdf W 47357424 122880 7.22
> 6.939169000 kworker/u8:0 6401 sdf W 47357664 122880 7.14
> 6.944985000 kworker/u8:0 6401 sdf W 47357904 122880 5.76
> 6.952113000 kworker/u8:0 6401 sdf W 47358144 122880 7.06
> 6.959608000 kworker/u8:0 6401 sdf W 47358384 122880 7.45
> 6.966820000 kworker/u8:0 6401 sdf W 47358624 122880 7.16
> 6.972393000 kworker/u8:0 6401 sdf W 47358864 122880 5.53
> 6.979550000 kworker/u8:0 6401 sdf W 47359104 122880 7.09
> 6.986879000 kworker/u8:0 6401 sdf W 47359344 122880 7.28
> 6.994306000 kworker/u8:0 6401 sdf W 47359584 122880 7.40
> 7.001338000 kworker/u8:0 6401 sdf W 47359824 122880 6.98
> 7.008740000 kworker/u8:0 6401 sdf W 47360064 122880 7.37
> 7.016114000 kworker/u8:0 6401 sdf W 47360304 122880 7.31
> 7.023546000 kworker/u8:0 6401 sdf W 47360544 122880 7.39
> 7.030437000 kworker/u8:0 6401 sdf W 47360784 122880 6.82
> 7.037834000 kworker/u8:0 6401 sdf W 47361024 122880 7.35
> 7.045224000 kworker/u8:0 6401 sdf W 47361264 122880 7.32
> 7.052687000 kworker/u8:0 6401 sdf W 47361504 122880 7.39
> 7.057977000 kworker/u8:0 6401 sdf W 47361744 122880 5.01
> 7.065390000 kworker/u8:0 6401 sdf W 47361984 122880 7.35
> 7.072749000 kworker/u8:0 6401 sdf W 47362224 122880 7.28
> 7.080174000 kworker/u8:0 6401 sdf W 47362464 122880 7.36
> 7.085335000 kworker/u8:0 6401 sdf W 47362704 122880 5.08
> 7.092881000 kworker/u8:0 6401 sdf W 47362944 122880 7.50
> 7.100197000 kworker/u8:0 6401 sdf W 47363184 122880 7.23
> 7.107507000 kworker/u8:0 6401 sdf W 47363424 122880 7.24
> 7.114684000 kworker/u8:0 6401 sdf W 47363664 122880 7.09
> 7.121936000 kworker/u8:0 6401 sdf W 47363904 122880 7.18
> 7.129299000 kworker/u8:0 6401 sdf W 47364144 122880 7.29
> 7.136645000 kworker/u8:0 6401 sdf W 47364384 122880 7.27
> 7.143733000 kworker/u8:0 6401 sdf W 47364624 122880 7.02
> 7.151112000 kworker/u8:0 6401 sdf W 47364864 122880 7.31
> 7.158509000 kworker/u8:0 6401 sdf W 47365104 122880 7.33
> 7.165735000 kworker/u8:0 6401 sdf W 47365344 122880 7.14
> 7.171205000 kworker/u8:0 6401 sdf W 47365584 122880 5.40
> 7.178625000 kworker/u8:0 6401 sdf W 47365824 122880 7.37
> 7.185980000 kworker/u8:0 6401 sdf W 47366064 122880 7.31
> 7.193593000 kworker/u8:0 6401 sdf W 47366304 122880 7.57
> 7.198674000 kworker/u8:0 6401 sdf W 47366544 122880 5.02
> 7.205996000 kworker/u8:0 6401 sdf W 47366784 122880 7.25
> 7.213419000 kworker/u8:0 6401 sdf W 47367024 122880 7.37
> 7.221005000 kworker/u8:0 6401 sdf W 47367264 122880 7.52
> 7.227906000 kworker/u8:0 6401 sdf W 47367504 122880 6.83
> 7.235251000 kworker/u8:0 6401 sdf W 47367744 122880 7.28
> 7.241892000 kworker/u8:0 6401 sdf W 47367984 106496 6.59
> 7.249617000 kworker/u8:0 6401 sdf W 47368192 122880 6.88
> 7.256892000 kworker/u8:0 6401 sdf W 47368432 122880 7.20
> 7.264223000 kworker/u8:0 6401 sdf W 47368672 122880 7.26
> 7.269672000 kworker/u8:0 6401 sdf W 47368912 122880 5.39
> 7.277066000 kworker/u8:0 6401 sdf W 47369152 122880 7.36
> 7.284630000 kworker/u8:0 6401 sdf W 47369392 122880 7.52
> 7.291891000 kworker/u8:0 6401 sdf W 47369632 122880 7.20
> 7.297211000 kworker/u8:0 6401 sdf W 47369872 122880 5.29
> 7.304665000 kworker/u8:0 6401 sdf W 47370112 122880 7.41
> 7.312050000 kworker/u8:0 6401 sdf W 47370352 122880 7.33
> 7.319520000 kworker/u8:0 6401 sdf W 47370592 122880 7.41
> 7.326494000 kworker/u8:0 6401 sdf W 47370832 122880 6.89
> 7.334082000 kworker/u8:0 6401 sdf W 47371072 122880 7.52
> 7.341505000 kworker/u8:0 6401 sdf W 47371312 122880 7.34
> 7.348600000 kworker/u8:0 6401 sdf W 47371552 122880 7.01
> 7.355806000 kworker/u8:0 6401 sdf W 47371792 122880 7.16
> 7.363147000 kworker/u8:0 6401 sdf W 47372032 122880 7.27
> 7.370377000 kworker/u8:0 6401 sdf W 47372272 122880 7.16
> 7.377770000 kworker/u8:0 6401 sdf W 47372512 122880 7.36
> 7.383323000 kworker/u8:0 6401 sdf W 47372752 122880 5.50
> 7.390651000 kworker/u8:0 6401 sdf W 47372992 122880 7.27
> 7.398245000 kworker/u8:0 6401 sdf W 47373232 122880 7.52
> 7.405347000 kworker/u8:0 6401 sdf W 47373472 122880 7.04
> 7.410761000 kworker/u8:0 6401 sdf W 47373712 122880 5.36
> 7.418075000 kworker/u8:0 6401 sdf W 47373952 122880 7.26
> 7.425451000 kworker/u8:0 6401 sdf W 47374192 122880 7.33
> 7.432864000 kworker/u8:0 6401 sdf W 47374432 122880 7.36
> 7.439913000 kworker/u8:0 6401 sdf W 47374672 122880 6.99
> 7.447312000 kworker/u8:0 6401 sdf W 47374912 122880 7.35
> 7.454721000 kworker/u8:0 6401 sdf W 47375152 122880 7.35
> 7.461955000 kworker/u8:0 6401 sdf W 47375392 122880 7.19
> 7.469040000 kworker/u8:0 6401 sdf W 47375632 122880 7.04
> 7.476416000 kworker/u8:0 6401 sdf W 47375872 122880 7.31
> 7.483856000 kworker/u8:0 6401 sdf W 47376112 122880 7.38
> 7.491270000 kworker/u8:0 6401 sdf W 47376352 122880 7.37
> 7.496452000 kworker/u8:0 6401 sdf W 47376592 122880 5.12
> 7.504011000 kworker/u8:0 6401 sdf W 47376832 122880 7.52
> 7.511217000 kworker/u8:0 6401 sdf W 47377072 122880 7.12
> 7.518511000 kworker/u8:0 6401 sdf W 47377312 122880 7.23
> 7.523956000 kworker/u8:0 6401 sdf W 47377552 122880 5.41
> 7.531197000 kworker/u8:0 6401 sdf W 47377792 122880 7.21
> 7.538546000 kworker/u8:0 6401 sdf W 47378032 122880 7.31
> 7.545757000 kworker/u8:0 6401 sdf W 47378272 122880 7.17
> 7.552919000 kworker/u8:0 6401 sdf W 47378512 122880 7.13
> 7.560269000 kworker/u8:0 6401 sdf W 47378752 122880 7.31
> 7.567555000 kworker/u8:0 6401 sdf W 47378992 122880 7.25
> 7.574863000 kworker/u8:0 6401 sdf W 47379232 122880 7.27
> 7.581802000 kworker/u8:0 6401 sdf W 47379472 122880 6.87
> 7.588979000 kworker/u8:0 6401 sdf W 47379712 122880 7.13
> 7.596343000 kworker/u8:0 6401 sdf W 47379952 122880 7.30
> 7.603224000 kworker/u8:0 6401 sdf W 47380192 122880 6.84
> 7.608655000 kworker/u8:0 6401 sdf W 47380432 122880 5.38
> 7.615811000 kworker/u8:0 6401 sdf W 47380672 122880 7.12
> 7.623028000 kworker/u8:0 6401 sdf W 47380912 122880 7.18
> 7.630413000 kworker/u8:0 6401 sdf W 47381152 122880 7.34
> 7.635223000 kworker/u8:0 6401 sdf W 47381392 122880 4.75
> 7.642268000 kworker/u8:0 6401 sdf W 47381632 122880 6.98
> 7.649383000 kworker/u8:0 6401 sdf W 47381872 122880 7.04
> 7.656619000 kworker/u8:0 6401 sdf W 47382112 122880 7.19
> 7.661918000 kworker/u8:0 6401 sdf W 47382352 122880 5.25
> 8.268039000 kworker/u8:0 6401 sdf W 47382592 122880 606.07
> 8.280414000 kworker/u8:0 6401 sdf W 47382832 122880 12.30
> 8.286503000 kworker/u8:0 6401 sdf W 47383072 122880 6.02
> 8.293627000 kworker/u8:0 6401 sdf W 47383312 122880 7.07
> 8.300777000 kworker/u8:0 6401 sdf W 47383552 122880 7.08
> 8.307733000 kworker/u8:0 6401 sdf W 47383792 122880 6.89
> 8.315212000 kworker/u8:0 6401 sdf W 47384032 122880 7.43
> 8.320697000 kworker/u8:0 6401 sdf W 47384272 122880 5.41
> 8.323335000 kworker/u8:0 6401 sdf W 47384512 32768 2.58
> 8.331116000 kworker/u8:0 6401 sdf W 47384576 122880 6.58
> 8.338586000 kworker/u8:0 6401 sdf W 47384816 122880 7.40
> 8.346113000 kworker/u8:0 6401 sdf W 47385056 122880 7.46
> 8.351558000 kworker/u8:0 6401 sdf W 47385296 122880 5.37
> 8.359030000 kworker/u8:0 6401 sdf W 47385536 122880 7.43
> 8.366425000 kworker/u8:0 6401 sdf W 47385776 122880 7.34
> 8.373712000 kworker/u8:0 6401 sdf W 47386016 122880 7.22
> 8.940689000 cp 6828 sdf W 47400960 122880 566.92
> 9.614647000 kworker/u8:0 6401 sdf W 47386256 122880 673.87
> 10.179120000 cp 6828 sdf W 47401200 122880 564.38
5 of 165 IO requests(1 4096 READ and 4 120K WRITE) take more than 500ms, and
most of others only needs <10ms. That is the reason why the performance
drops, and it is highly related with your USB pen drive.
biosnoop collects the trace data in blk_mq_start_request() and
blk_account_io_completion(), both happens during dispatching request to
LLD & device.
From my understanding, scsi-mq should only change the timing, or maybe
some setting is changed(still very unlikely), you may follow Alan's suggestion
to collect usbmon on 'bad' & 'good' kernel, and check if the setting is
changed.
If the initialization setting is same, maybe you can try US_FL_GO_SLOW
quirk.
Thanks,
Ming
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-27 0:21 ` Finn Thain
@ 2019-11-28 17:10 ` Andrea Vai
0 siblings, 0 replies; 77+ messages in thread
From: Andrea Vai @ 2019-11-28 17:10 UTC (permalink / raw)
To: Finn Thain
Cc: Ming Lei, Damien Le Moal, Alan Stern, Jens Axboe,
Johannes Thumshirn, USB list, SCSI development list,
Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
Martin K. Petersen, Greg KH, Hans Holmberg,
Kernel development list
Il giorno mer, 27/11/2019 alle 11.21 +1100, Finn Thain ha scritto:
> On Tue, 26 Nov 2019, Andrea Vai wrote:
>
> > Then I started another set of 100 trials and let them run tonight,
> and
> > the first 10 trials were around 1000s, then gradually decreased
> to
> > ~300s, and finally settled around 200s with some trials below 70-
> 80s.
> > This to say, times are extremely variable and for the first time
> I
> > noticed a sort of "performance increase" with time.
> >
>
> The sheer volume of testing (probably some terabytes by now) would
> exercise the wear leveling algorithm in the FTL.
>
> This in itself seems unlikely to improve performance significantly.
> But if
> the flash memory came from a bad batch, perhaps it would have that
> effect.
>
> To find out, someone may need to source another (genuine) Kingston
> DataTraveller device.
I own another device (let's refer to it as "black odd"), identical to
the "slow" one (call it "black even"), and used it as well to do the
tests, especially in the beginning of this story, because I suspected
the problem could be related to a faulty pen drive. At a certain time
I realized that the tests I performed didn't show any difference
between the two flash drives, so since that time I kept using just the
"black even". They were bought together, so of course both of them
probably belong to the same "maybe-bad batch".
But I have another Kingston DataTraveler ("White"), externally
slightly different from the other twos (it's white instead of black,
and labeled G4 instead of G3), though lsusb shows the same IDs:
0951:1666. It had been purchased some months after the other twos
(well, actually, it may be the result of an RMA exchange).
I have just ran one test on this White one, with the new (patched)
kernel, and it took an average of 200seconds (st.dev=46s), which is
not "good", but less "bad" than the real "bad" case of the "black"
ones (>1000 seconds).
I have also tried the "WHITE" one with the old fast kernel, and the
behavior is almost the same as with the new kernel, though a little
bit better (mean=173; st.dev.=11).
Feel free to let me know if I should do other tries,
thanks,
Andrea
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-27 13:08 ` Ming Lei
@ 2019-11-27 15:01 ` Andrea Vai
0 siblings, 0 replies; 77+ messages in thread
From: Andrea Vai @ 2019-11-27 15:01 UTC (permalink / raw)
To: Ming Lei
Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
USB list, SCSI development list, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
[-- Attachment #1: Type: text/plain, Size: 3496 bytes --]
Il giorno mer, 27/11/2019 alle 21.08 +0800, Ming Lei ha scritto:
> On Wed, Nov 27, 2019 at 10:39:40AM +0100, Andrea Vai wrote:
> > Il giorno mer, 27/11/2019 alle 10.05 +0800, Ming Lei ha scritto:
> > >
> > >
> > > It can be workaround via the following change:
> > >
> > > /lib/modules/5.4.0+/build/include/generated/autoconf.h:
> > >
> > > //#define CONFIG_CC_HAS_ASM_INLINE 1
> >
> > Thanks, it worked, trace attached. Produced by: start the trace
> script
> > (with the pendrive already plugged), wait some seconds, run the
> test
> > (1 trial, 1 GB), wait for the test to finish, stop the trace.
> >
> > The copy took 2659 seconds, roughly as already seen before.
>
> Thanks for collecting the log.
>
> From the log, some of write IOs are out-of-order, such as, the 1st
> one
> is 378880.
>
> 16.41240 2 266 266 kworker/2:1H block_rq_issue b'W'
> 370656 240
> 16.41961 3 485 485 kworker/3:1H block_rq_issue b'W'
> 378880 240
> 16.73729 2 266 266 kworker/2:1H block_rq_issue b'W'
> 370896 240
> 17.71161 2 266 266 kworker/2:1H block_rq_issue b'W'
> 379120 240
> 18.02344 2 266 266 kworker/2:1H block_rq_issue b'W'
> 371136 240
> 18.94314 3 485 485 kworker/3:1H block_rq_issue b'W'
> 379360 240
> 19.25624 2 266 266 kworker/2:1H block_rq_issue b'W'
> 371376 240
>
> IO latency is increased a lot since the 1st out-of-order request(usb
> storage HBA is single queue depth, one request can be issued only
> if
> the previous issued request is completed).
>
> The reason is that there are two kind of tasks which inserts rq to
> device.
> One is the 'cp' process, the other is kworker/u8:*. The out-of-
> order
> happens during the two task's interleaving.
>
> Under such situation, I believe that the old legacy IO path may not
> guarantee the order too. In blk_queue_bio(), after get_request()
> allocates one request, the queue lock is released. And request is
> actually inserted & issued from blk_flush_plug_list() under the
> branch of 'if (plug)'. If requests are from two tasks, then request
> is inserted/issued from two plug list, and no order can be
> guaranteed.
>
> In my test, except for several requests from the beginning, all
> other
> requests are inserted via the kworker thread(guess it is writeback
> wq),
> that is why I can't observe the issue in my test.
>
> As Schmid suggested, you may run the same test on old kernel with
> legacy io path, and see if the performance is still good.
>
> Also, could you share the following info about your machine? So that
> I can build my VM guest in this setting for reproducing your
> situation
> (requests are inserted from two types of threads).
>
> - lscpu
attached,
> - free -h
total used free shared buff/cache available
Mem: 23Gi 4,2Gi 11Gi 448Mi 7,0Gi 18Gi
Swap: 3,7Gi 0B 3,7Gi
> - lsblk -d $USB_DISK
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdg 8:96 1 28,8G 0 disk
> - exact commands for mount the disk, and running the copy operation
I attached the whole script to this thread, I attach it again to this
message and copy the relevant lines here:
mount UUID=$uuid /mnt/pendrive 2>&1 |tee -a $logfile
SECONDS=0
cp $testfile /mnt/pendrive 2>&1 |tee -a $logfile
umount /mnt/pendrive 2>&1 |tee -a $logfile
Meanwhile, I am going on with the further tests as suggested
Thanks,
Andrea
[-- Attachment #2: lscpu.txt --]
[-- Type: text/plain, Size: 1371 bytes --]
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 39 bits physical, 48 bits virtual
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 60
Model name: Intel(R) Core(TM) i5-4430 CPU @ 3.00GHz
Stepping: 3
CPU MHz: 1674.727
CPU max MHz: 3200,0000
CPU min MHz: 800,0000
BogoMIPS: 5986.16
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 6144K
NUMA node0 CPU(s): 0-3
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts md_clear flush_l1d
[-- Attachment #3: test --]
[-- Type: application/x-shellscript, Size: 1137 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-27 9:39 ` Andrea Vai
@ 2019-11-27 13:08 ` Ming Lei
2019-11-27 15:01 ` Andrea Vai
0 siblings, 1 reply; 77+ messages in thread
From: Ming Lei @ 2019-11-27 13:08 UTC (permalink / raw)
To: Andrea Vai
Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
USB list, SCSI development list, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
On Wed, Nov 27, 2019 at 10:39:40AM +0100, Andrea Vai wrote:
> Il giorno mer, 27/11/2019 alle 10.05 +0800, Ming Lei ha scritto:
> >
> >
> > It can be workaround via the following change:
> >
> > /lib/modules/5.4.0+/build/include/generated/autoconf.h:
> >
> > //#define CONFIG_CC_HAS_ASM_INLINE 1
>
> Thanks, it worked, trace attached. Produced by: start the trace script
> (with the pendrive already plugged), wait some seconds, run the test
> (1 trial, 1 GB), wait for the test to finish, stop the trace.
>
> The copy took 2659 seconds, roughly as already seen before.
Thanks for collecting the log.
From the log, some of write IOs are out-of-order, such as, the 1st one
is 378880.
16.41240 2 266 266 kworker/2:1H block_rq_issue b'W' 370656 240
16.41961 3 485 485 kworker/3:1H block_rq_issue b'W' 378880 240
16.73729 2 266 266 kworker/2:1H block_rq_issue b'W' 370896 240
17.71161 2 266 266 kworker/2:1H block_rq_issue b'W' 379120 240
18.02344 2 266 266 kworker/2:1H block_rq_issue b'W' 371136 240
18.94314 3 485 485 kworker/3:1H block_rq_issue b'W' 379360 240
19.25624 2 266 266 kworker/2:1H block_rq_issue b'W' 371376 240
IO latency is increased a lot since the 1st out-of-order request(usb
storage HBA is single queue depth, one request can be issued only if
the previous issued request is completed).
The reason is that there are two kind of tasks which inserts rq to device.
One is the 'cp' process, the other is kworker/u8:*. The out-of-order
happens during the two task's interleaving.
Under such situation, I believe that the old legacy IO path may not
guarantee the order too. In blk_queue_bio(), after get_request()
allocates one request, the queue lock is released. And request is
actually inserted & issued from blk_flush_plug_list() under the
branch of 'if (plug)'. If requests are from two tasks, then request
is inserted/issued from two plug list, and no order can be guaranteed.
In my test, except for several requests from the beginning, all other
requests are inserted via the kworker thread(guess it is writeback wq),
that is why I can't observe the issue in my test.
As Schmid suggested, you may run the same test on old kernel with
legacy io path, and see if the performance is still good.
Also, could you share the following info about your machine? So that
I can build my VM guest in this setting for reproducing your situation
(requests are inserted from two types of threads).
- lscpu
- free -h
- lsblk -d $USB_DISK
- exact commands for mount the disk, and running the copy operation
Thanks,
Ming
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-27 2:05 ` Ming Lei
@ 2019-11-27 9:39 ` Andrea Vai
2019-11-27 13:08 ` Ming Lei
0 siblings, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-11-27 9:39 UTC (permalink / raw)
To: Ming Lei
Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
USB list, SCSI development list, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
[-- Attachment #1: Type: text/plain, Size: 503 bytes --]
Il giorno mer, 27/11/2019 alle 10.05 +0800, Ming Lei ha scritto:
>
>
> It can be workaround via the following change:
>
> /lib/modules/5.4.0+/build/include/generated/autoconf.h:
>
> //#define CONFIG_CC_HAS_ASM_INLINE 1
Thanks, it worked, trace attached. Produced by: start the trace script
(with the pendrive already plugged), wait some seconds, run the test
(1 trial, 1 GB), wait for the test to finish, stop the trace.
The copy took 2659 seconds, roughly as already seen before.
Thanks,
Andrea
[-- Attachment #2: log_ming.zip --]
[-- Type: application/zip, Size: 111658 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-26 11:14 ` Andrea Vai
@ 2019-11-27 2:05 ` Ming Lei
2019-11-27 9:39 ` Andrea Vai
0 siblings, 1 reply; 77+ messages in thread
From: Ming Lei @ 2019-11-27 2:05 UTC (permalink / raw)
To: Andrea Vai
Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
USB list, SCSI development list, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
On Tue, Nov 26, 2019 at 12:14:19PM +0100, Andrea Vai wrote:
> Il giorno mar, 26/11/2019 alle 17.15 +0800, Ming Lei ha scritto:
> > On Tue, Nov 26, 2019 at 08:46:07AM +0100, Andrea Vai wrote:
> > > Il giorno mar, 26/11/2019 alle 10.32 +0800, Ming Lei ha scritto:
> > > > On Mon, Nov 25, 2019 at 07:51:33PM +0100, Andrea Vai wrote:
> > > > > Il giorno lun, 25/11/2019 alle 23.15 +0800, Ming Lei ha
> > scritto:
> > > > > > On Mon, Nov 25, 2019 at 03:58:34PM +0100, Andrea Vai wrote:
> > > > > >
> > > > > > [...]
> > > > > >
> > > > > > > What to try next?
> > > > > >
> > > > > > 1) cat /sys/kernel/debug/block/$DISK/hctx0/flags
> > > > > result:
> > > > >
> > > > > alloc_policy=FIFO SHOULD_MERGE|2
> > > > >
> > > > > >
> > > > > >
> > > > > > 2) echo 128 > /sys/block/$DISK/queue/nr_requests and run
> > your
> > > > copy
> > > > > > 1GB
> > > > > > test again.
> > > > >
> > > > > done, and still fails. What to try next?
> > > >
> > > > I just run 256M cp test
> > >
> > > I would like to point out that 256MB is a filesize that usually
> > don't
> > > trigger the issue (don't know if it matters, sorry).
> >
> > OK.
> >
> > I tested 256M because IO timeout is often triggered in case of
> > qemu-ehci, and it is a long-term issue. When setting up the disk
> > via xhci-qemu, the max request size is increased to 1MB from 120KB,
> > and IO pattern changed too. When the disk is connected via uhci-
> > qemu,
> > the transfer is too slow(1MB/s) because max endpoint size is too
> > small.
> >
> > However, I just waited 16min and collected all the 1GB IO log by
> > connecting disk over uhci-qemu, but the sector of each data IO
> > is still in order.
> >
> > >
> > > Another info I would provide is about another strange behavior I
> > > noticed: yesterday I ran the test two times (as usual with 1GB
> > > filesize) and took 2370s, 1786s, and a third test was going on
> > when I
> > > stopped it. Then I started another set of 100 trials and let them
> > run
> > > tonight, and the first 10 trials were around 1000s, then gradually
> > > decreased to ~300s, and finally settled around 200s with some
> > trials
> > > below 70-80s. This to say, times are extremely variable and for
> > the
> > > first time I noticed a sort of "performance increase" with time.
> >
> > The 'cp' test is buffered IO, can you reproduce it every time by
> > running copy just after fresh mount on the USB disk?
>
> yes, every time my test script (attached) mounts, copy, unmount (but I
> don't unplug and replug the pendrive each time). Is this enough?
>
> >
> > >
> > > > to one USB storage device on patched kernel,
> > > > and WRITE data IO is really in ascending order. The filesystem
> > is
> > > > ext4,
> > > > and mount without '-o sync'. From previous discussion, looks
> > that is
> > > > exactly your test setting. The order can be observed via the
> > > > following script:
> > > >
> > > > #!/bin/sh
> > > > MAJ=$1
> > > > MIN=$2
> > > > MAJ=$(( $MAJ << 20 ))
> > > > DEV=$(( $MAJ | $MIN ))
> > > > /usr/share/bcc/tools/trace -t -C \
> > > > 't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d",
> > args-
> > > > >rwbs, args->sector, args->nr_sector'
> > > >
> > > > $MAJ & $MIN can be retrieved via lsblk for your USB storage
> > disk.
>
> ok, so I try:
>
> # lsblk /dev/sdf
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> sdf 8:80 1 28,8G 0 disk
> └─sdf1 8:81 1 28,8G 0 part
>
> so I ran your script (the second one, which you sent me in the next
> email message) with:
>
> ./test_ming 8 80
>
> but it fails to run (terminal output is in attached errors.txt).
> What am I doing wrong?
>
> It's still not clear to me if I need to start the trace script and
> then the test, or the opposite (or doesn't matter). The above errors
> are in the former case (I didn't even start the test, actually)
>
> Thanks,
> Andrea
> In file included from /virtual/main.c:2:
> In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
> In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:14:
> In file included from /lib/modules/5.4.0+/build/include/linux/pid.h:5:
> In file included from /lib/modules/5.4.0+/build/include/linux/rculist.h:11:
> In file included from /lib/modules/5.4.0+/build/include/linux/rcupdate.h:26:
> In file included from /lib/modules/5.4.0+/build/include/linux/irqflags.h:16:
> In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/irqflags.h:9:
> In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/nospec-branch.h:314:
> /lib/modules/5.4.0+/build/arch/x86/include/asm/segment.h:254:2: error: expected '(' after 'asm'
> alternative_io ("lsl %[seg],%[p]",
> ^
> /lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:240:2: note: expanded from macro 'alternative_io'
> asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature) \
> ^
> /lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
> #define asm_inline asm __inline
> ^
> In file included from /virtual/main.c:2:
> In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
> In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:14:
> In file included from /lib/modules/5.4.0+/build/include/linux/pid.h:5:
> In file included from /lib/modules/5.4.0+/build/include/linux/rculist.h:11:
> In file included from /lib/modules/5.4.0+/build/include/linux/rcupdate.h:27:
> In file included from /lib/modules/5.4.0+/build/include/linux/preempt.h:78:
> In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/preempt.h:7:
> In file included from /lib/modules/5.4.0+/build/include/linux/thread_info.h:38:
> In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/thread_info.h:12:
> In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/page.h:12:
> /lib/modules/5.4.0+/build/arch/x86/include/asm/page_64.h:49:2: error: expected '(' after 'asm'
> alternative_call_2(clear_page_orig,
> ^
> /lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:256:2: note: expanded from macro 'alternative_call_2'
> asm_inline volatile (ALTERNATIVE_2("call %P[old]", "call %P[new1]", feature1,\
> ^
> /lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
> #define asm_inline asm __inline
It can be workaround via the following change:
/lib/modules/5.4.0+/build/include/generated/autoconf.h:
//#define CONFIG_CC_HAS_ASM_INLINE 1
Thanks,
Ming
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-26 7:46 ` Andrea Vai
2019-11-26 9:15 ` Ming Lei
@ 2019-11-27 0:21 ` Finn Thain
2019-11-28 17:10 ` Andrea Vai
1 sibling, 1 reply; 77+ messages in thread
From: Finn Thain @ 2019-11-27 0:21 UTC (permalink / raw)
To: Andrea Vai
Cc: Ming Lei, Damien Le Moal, Alan Stern, Jens Axboe,
Johannes Thumshirn, USB list, SCSI development list,
Himanshu Madhani, Hannes Reinecke, Omar Sandoval,
Martin K. Petersen, Greg KH, Hans Holmberg,
Kernel development list
On Tue, 26 Nov 2019, Andrea Vai wrote:
> Then I started another set of 100 trials and let them run tonight, and
> the first 10 trials were around 1000s, then gradually decreased to
> ~300s, and finally settled around 200s with some trials below 70-80s.
> This to say, times are extremely variable and for the first time I
> noticed a sort of "performance increase" with time.
>
The sheer volume of testing (probably some terabytes by now) would
exercise the wear leveling algorithm in the FTL.
This in itself seems unlikely to improve performance significantly. But if
the flash memory came from a bad batch, perhaps it would have that effect.
To find out, someone may need to source another (genuine) Kingston
DataTraveller device.
--
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-26 9:15 ` Ming Lei
2019-11-26 10:24 ` Ming Lei
@ 2019-11-26 11:14 ` Andrea Vai
2019-11-27 2:05 ` Ming Lei
1 sibling, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-11-26 11:14 UTC (permalink / raw)
To: Ming Lei
Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
USB list, SCSI development list, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
[-- Attachment #1: Type: text/plain, Size: 3585 bytes --]
Il giorno mar, 26/11/2019 alle 17.15 +0800, Ming Lei ha scritto:
> On Tue, Nov 26, 2019 at 08:46:07AM +0100, Andrea Vai wrote:
> > Il giorno mar, 26/11/2019 alle 10.32 +0800, Ming Lei ha scritto:
> > > On Mon, Nov 25, 2019 at 07:51:33PM +0100, Andrea Vai wrote:
> > > > Il giorno lun, 25/11/2019 alle 23.15 +0800, Ming Lei ha
> scritto:
> > > > > On Mon, Nov 25, 2019 at 03:58:34PM +0100, Andrea Vai wrote:
> > > > >
> > > > > [...]
> > > > >
> > > > > > What to try next?
> > > > >
> > > > > 1) cat /sys/kernel/debug/block/$DISK/hctx0/flags
> > > > result:
> > > >
> > > > alloc_policy=FIFO SHOULD_MERGE|2
> > > >
> > > > >
> > > > >
> > > > > 2) echo 128 > /sys/block/$DISK/queue/nr_requests and run
> your
> > > copy
> > > > > 1GB
> > > > > test again.
> > > >
> > > > done, and still fails. What to try next?
> > >
> > > I just run 256M cp test
> >
> > I would like to point out that 256MB is a filesize that usually
> don't
> > trigger the issue (don't know if it matters, sorry).
>
> OK.
>
> I tested 256M because IO timeout is often triggered in case of
> qemu-ehci, and it is a long-term issue. When setting up the disk
> via xhci-qemu, the max request size is increased to 1MB from 120KB,
> and IO pattern changed too. When the disk is connected via uhci-
> qemu,
> the transfer is too slow(1MB/s) because max endpoint size is too
> small.
>
> However, I just waited 16min and collected all the 1GB IO log by
> connecting disk over uhci-qemu, but the sector of each data IO
> is still in order.
>
> >
> > Another info I would provide is about another strange behavior I
> > noticed: yesterday I ran the test two times (as usual with 1GB
> > filesize) and took 2370s, 1786s, and a third test was going on
> when I
> > stopped it. Then I started another set of 100 trials and let them
> run
> > tonight, and the first 10 trials were around 1000s, then gradually
> > decreased to ~300s, and finally settled around 200s with some
> trials
> > below 70-80s. This to say, times are extremely variable and for
> the
> > first time I noticed a sort of "performance increase" with time.
>
> The 'cp' test is buffered IO, can you reproduce it every time by
> running copy just after fresh mount on the USB disk?
yes, every time my test script (attached) mounts, copy, unmount (but I
don't unplug and replug the pendrive each time). Is this enough?
>
> >
> > > to one USB storage device on patched kernel,
> > > and WRITE data IO is really in ascending order. The filesystem
> is
> > > ext4,
> > > and mount without '-o sync'. From previous discussion, looks
> that is
> > > exactly your test setting. The order can be observed via the
> > > following script:
> > >
> > > #!/bin/sh
> > > MAJ=$1
> > > MIN=$2
> > > MAJ=$(( $MAJ << 20 ))
> > > DEV=$(( $MAJ | $MIN ))
> > > /usr/share/bcc/tools/trace -t -C \
> > > 't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d",
> args-
> > > >rwbs, args->sector, args->nr_sector'
> > >
> > > $MAJ & $MIN can be retrieved via lsblk for your USB storage
> disk.
ok, so I try:
# lsblk /dev/sdf
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdf 8:80 1 28,8G 0 disk
└─sdf1 8:81 1 28,8G 0 part
so I ran your script (the second one, which you sent me in the next
email message) with:
./test_ming 8 80
but it fails to run (terminal output is in attached errors.txt).
What am I doing wrong?
It's still not clear to me if I need to start the trace script and
then the test, or the opposite (or doesn't matter). The above errors
are in the former case (I didn't even start the test, actually)
Thanks,
Andrea
[-- Attachment #2: errors.txt --]
[-- Type: text/plain, Size: 60490 bytes --]
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:14:
In file included from /lib/modules/5.4.0+/build/include/linux/pid.h:5:
In file included from /lib/modules/5.4.0+/build/include/linux/rculist.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/rcupdate.h:26:
In file included from /lib/modules/5.4.0+/build/include/linux/irqflags.h:16:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/irqflags.h:9:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/nospec-branch.h:314:
/lib/modules/5.4.0+/build/arch/x86/include/asm/segment.h:254:2: error: expected '(' after 'asm'
alternative_io ("lsl %[seg],%[p]",
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:240:2: note: expanded from macro 'alternative_io'
asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature) \
^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:14:
In file included from /lib/modules/5.4.0+/build/include/linux/pid.h:5:
In file included from /lib/modules/5.4.0+/build/include/linux/rculist.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/rcupdate.h:27:
In file included from /lib/modules/5.4.0+/build/include/linux/preempt.h:78:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/preempt.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/thread_info.h:38:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/thread_info.h:12:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/page.h:12:
/lib/modules/5.4.0+/build/arch/x86/include/asm/page_64.h:49:2: error: expected '(' after 'asm'
alternative_call_2(clear_page_orig,
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:256:2: note: expanded from macro 'alternative_call_2'
asm_inline volatile (ALTERNATIVE_2("call %P[old]", "call %P[new1]", feature1,\
^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:14:
In file included from /lib/modules/5.4.0+/build/include/linux/pid.h:5:
In file included from /lib/modules/5.4.0+/build/include/linux/rculist.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/rcupdate.h:27:
In file included from /lib/modules/5.4.0+/build/include/linux/preempt.h:78:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/preempt.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/thread_info.h:38:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/thread_info.h:53:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/cpufeature.h:5:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/processor.h:24:
/lib/modules/5.4.0+/build/arch/x86/include/asm/special_insns.h:205:2: error: expected '(' after 'asm'
alternative_io(".byte " __stringify(NOP_DS_PREFIX) "; clflush %P0",
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:240:2: note: expanded from macro 'alternative_io'
asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature) \
^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:14:
In file included from /lib/modules/5.4.0+/build/include/linux/pid.h:5:
In file included from /lib/modules/5.4.0+/build/include/linux/rculist.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/rcupdate.h:27:
In file included from /lib/modules/5.4.0+/build/include/linux/preempt.h:78:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/preempt.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/thread_info.h:38:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/thread_info.h:53:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/cpufeature.h:5:
/lib/modules/5.4.0+/build/arch/x86/include/asm/processor.h:795:2: error: expected '(' after 'asm'
alternative_input(BASE_PREFETCH, "prefetchnta %P1",
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:221:2: note: expanded from macro 'alternative_input'
asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature) \
^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:14:
In file included from /lib/modules/5.4.0+/build/include/linux/pid.h:5:
In file included from /lib/modules/5.4.0+/build/include/linux/rculist.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/rcupdate.h:27:
In file included from /lib/modules/5.4.0+/build/include/linux/preempt.h:78:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/preempt.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/thread_info.h:38:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/thread_info.h:53:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/cpufeature.h:5:
/lib/modules/5.4.0+/build/arch/x86/include/asm/processor.h:807:2: error: expected '(' after 'asm'
alternative_input(BASE_PREFETCH, "prefetchw %P1",
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:221:2: note: expanded from macro 'alternative_input'
asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature) \
^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:14:
In file included from /lib/modules/5.4.0+/build/include/linux/pid.h:5:
In file included from /lib/modules/5.4.0+/build/include/linux/rculist.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/rcupdate.h:27:
In file included from /lib/modules/5.4.0+/build/include/linux/preempt.h:78:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/preempt.h:7:
/lib/modules/5.4.0+/build/include/linux/thread_info.h:134:2: error: expected '(' after 'asm'
WARN(1, "Buffer overflow detected (%d < %lu)!\n", size, count);
^
/lib/modules/5.4.0+/build/include/asm-generic/bug.h:124:3: note: expanded from macro 'WARN'
__WARN_printf(TAINT_WARN, format); \
^
/lib/modules/5.4.0+/build/include/asm-generic/bug.h:93:3: note: expanded from macro '__WARN_printf'
__WARN_FLAGS(BUGFLAG_NO_CUT_HERE | BUGFLAG_TAINT(taint));\
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/bug.h:79:2: note: expanded from macro '__WARN_FLAGS'
_BUG_FLAGS(ASM_UD2, BUGFLAG_WARNING|(flags)); \
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/bug.h:35:2: note: expanded from macro '_BUG_FLAGS'
asm_inline volatile("1:\t" ins "\n" \
^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:14:
In file included from /lib/modules/5.4.0+/build/include/linux/pid.h:5:
In file included from /lib/modules/5.4.0+/build/include/linux/rculist.h:11:
/lib/modules/5.4.0+/build/include/linux/rcupdate.h:893:2: error: expected '(' after 'asm'
WARN_ON_ONCE(func != (rcu_callback_t)~0L);
^
/lib/modules/5.4.0+/build/include/asm-generic/bug.h:98:3: note: expanded from macro 'WARN_ON_ONCE'
__WARN_FLAGS(BUGFLAG_ONCE | \
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/bug.h:79:2: note: expanded from macro '__WARN_FLAGS'
_BUG_FLAGS(ASM_UD2, BUGFLAG_WARNING|(flags)); \
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/bug.h:35:2: note: expanded from macro '_BUG_FLAGS'
asm_inline volatile("1:\t" ins "\n" \
^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:15:
In file included from /lib/modules/5.4.0+/build/include/linux/sem.h:5:
In file included from /lib/modules/5.4.0+/build/include/uapi/linux/sem.h:5:
In file included from /lib/modules/5.4.0+/build/include/linux/ipc.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/rhashtable-types.h:15:
In file included from /lib/modules/5.4.0+/build/include/linux/workqueue.h:9:
In file included from /lib/modules/5.4.0+/build/include/linux/timer.h:6:
/lib/modules/5.4.0+/build/include/linux/ktime.h:171:2: error: expected '(' after 'asm'
WARN_ON(div < 0);
^
/lib/modules/5.4.0+/build/include/asm-generic/bug.h:115:3: note: expanded from macro 'WARN_ON'
__WARN(); \
^
/lib/modules/5.4.0+/build/include/asm-generic/bug.h:90:19: note: expanded from macro '__WARN'
#define __WARN() __WARN_FLAGS(BUGFLAG_TAINT(TAINT_WARN))
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/bug.h:79:2: note: expanded from macro '__WARN_FLAGS'
_BUG_FLAGS(ASM_UD2, BUGFLAG_WARNING|(flags)); \
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/bug.h:35:2: note: expanded from macro '_BUG_FLAGS'
asm_inline volatile("1:\t" ins "\n" \
^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:20:
In file included from /lib/modules/5.4.0+/build/include/linux/hrtimer.h:19:
In file included from /lib/modules/5.4.0+/build/include/linux/percpu.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/smp.h:68:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/smp.h:13:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/apic.h:11:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/fixmap.h:190:
In file included from /lib/modules/5.4.0+/build/include/asm-generic/fixmap.h:19:
In file included from /lib/modules/5.4.0+/build/include/linux/mm_types.h:14:
In file included from /lib/modules/5.4.0+/build/include/linux/uprobes.h:49:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/uprobes.h:13:
In file included from /lib/modules/5.4.0+/build/include/linux/notifier.h:16:
/lib/modules/5.4.0+/build/include/linux/srcu.h:179:2: error: expected '(' after 'asm'
WARN_ON_ONCE(idx & ~0x1);
^
/lib/modules/5.4.0+/build/include/asm-generic/bug.h:98:3: note: expanded from macro 'WARN_ON_ONCE'
__WARN_FLAGS(BUGFLAG_ONCE | \
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/bug.h:79:2: note: expanded from macro '__WARN_FLAGS'
_BUG_FLAGS(ASM_UD2, BUGFLAG_WARNING|(flags)); \
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/bug.h:35:2: note: expanded from macro '_BUG_FLAGS'
asm_inline volatile("1:\t" ins "\n" \
^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:20:
In file included from /lib/modules/5.4.0+/build/include/linux/hrtimer.h:19:
In file included from /lib/modules/5.4.0+/build/include/linux/percpu.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/smp.h:68:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/smp.h:13:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/apic.h:11:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/fixmap.h:190:
/lib/modules/5.4.0+/build/include/asm-generic/fixmap.h:38:2: error: expected '(' after 'asm'
BUG_ON(vaddr >= FIXADDR_TOP || vaddr < FIXADDR_START);
^
/lib/modules/5.4.0+/build/include/asm-generic/bug.h:62:57: note: expanded from macro 'BUG_ON'
#define BUG_ON(condition) do { if (unlikely(condition)) BUG(); } while (0)
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/bug.h:73:2: note: expanded from macro 'BUG'
_BUG_FLAGS(ASM_UD2, 0); \
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/bug.h:35:2: note: expanded from macro '_BUG_FLAGS'
asm_inline volatile("1:\t" ins "\n" \
^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:6:
In file included from /lib/modules/5.4.0+/build/include/linux/sched.h:20:
In file included from /lib/modules/5.4.0+/build/include/linux/hrtimer.h:19:
In file included from /lib/modules/5.4.0+/build/include/linux/percpu.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/smp.h:68:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/smp.h:13:
/lib/modules/5.4.0+/build/arch/x86/include/asm/apic.h:107:2: error: expected '(' after 'asm'
alternative_io("movl %0, %P1", "xchgl %0, %P1", X86_BUG_11AP,
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:240:2: note: expanded from macro 'alternative_io'
asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature) \
^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:87:11: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
return (set->sig[3] | set->sig[2] |
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:87:25: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
return (set->sig[3] | set->sig[2] |
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:88:4: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
set->sig[1] | set->sig[0]) == 0;
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:90:11: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
return (set->sig[1] | set->sig[0]) == 0;
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:103:11: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
return (set1->sig[3] == set2->sig[3]) &&
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:103:27: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
return (set1->sig[3] == set2->sig[3]) &&
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:104:5: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
(set1->sig[2] == set2->sig[2]) &&
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:104:21: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
(set1->sig[2] == set2->sig[2]) &&
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:105:5: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
(set1->sig[1] == set2->sig[1]) &&
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:105:21: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
(set1->sig[1] == set2->sig[1]) &&
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:108:11: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
return (set1->sig[1] == set2->sig[1]) &&
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:108:27: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
return (set1->sig[1] == set2->sig[1]) &&
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:147:1: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigorsets, _sig_or)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:128:8: note: expanded from macro '_SIG_SET_BINOP'
a3 = a->sig[3]; a2 = a->sig[2]; \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:147:1: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigorsets, _sig_or)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:128:24: note: expanded from macro '_SIG_SET_BINOP'
a3 = a->sig[3]; a2 = a->sig[2]; \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:147:1: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigorsets, _sig_or)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:129:8: note: expanded from macro '_SIG_SET_BINOP'
b3 = b->sig[3]; b2 = b->sig[2]; \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:147:1: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigorsets, _sig_or)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:129:24: note: expanded from macro '_SIG_SET_BINOP'
b3 = b->sig[3]; b2 = b->sig[2]; \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:147:1: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigorsets, _sig_or)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:130:3: note: expanded from macro '_SIG_SET_BINOP'
r->sig[3] = op(a3, b3); \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:147:1: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigorsets, _sig_or)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:131:3: note: expanded from macro '_SIG_SET_BINOP'
r->sig[2] = op(a2, b2); \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:147:1: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigorsets, _sig_or)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:134:8: note: expanded from macro '_SIG_SET_BINOP'
a1 = a->sig[1]; b1 = b->sig[1]; \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:147:1: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigorsets, _sig_or)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:134:24: note: expanded from macro '_SIG_SET_BINOP'
a1 = a->sig[1]; b1 = b->sig[1]; \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:147:1: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigorsets, _sig_or)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:135:3: note: expanded from macro '_SIG_SET_BINOP'
r->sig[1] = op(a1, b1); \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:150:1: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandsets, _sig_and)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:128:8: note: expanded from macro '_SIG_SET_BINOP'
a3 = a->sig[3]; a2 = a->sig[2]; \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:150:1: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandsets, _sig_and)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:128:24: note: expanded from macro '_SIG_SET_BINOP'
a3 = a->sig[3]; a2 = a->sig[2]; \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:150:1: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandsets, _sig_and)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:129:8: note: expanded from macro '_SIG_SET_BINOP'
b3 = b->sig[3]; b2 = b->sig[2]; \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:150:1: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandsets, _sig_and)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:129:24: note: expanded from macro '_SIG_SET_BINOP'
b3 = b->sig[3]; b2 = b->sig[2]; \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:150:1: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandsets, _sig_and)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:130:3: note: expanded from macro '_SIG_SET_BINOP'
r->sig[3] = op(a3, b3); \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:150:1: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandsets, _sig_and)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:131:3: note: expanded from macro '_SIG_SET_BINOP'
r->sig[2] = op(a2, b2); \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:150:1: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandsets, _sig_and)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:134:8: note: expanded from macro '_SIG_SET_BINOP'
a1 = a->sig[1]; b1 = b->sig[1]; \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:150:1: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandsets, _sig_and)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:134:24: note: expanded from macro '_SIG_SET_BINOP'
a1 = a->sig[1]; b1 = b->sig[1]; \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:150:1: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandsets, _sig_and)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:135:3: note: expanded from macro '_SIG_SET_BINOP'
r->sig[1] = op(a1, b1); \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:153:1: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandnsets, _sig_andn)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:128:8: note: expanded from macro '_SIG_SET_BINOP'
a3 = a->sig[3]; a2 = a->sig[2]; \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:153:1: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandnsets, _sig_andn)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:128:24: note: expanded from macro '_SIG_SET_BINOP'
a3 = a->sig[3]; a2 = a->sig[2]; \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:153:1: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandnsets, _sig_andn)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:129:8: note: expanded from macro '_SIG_SET_BINOP'
b3 = b->sig[3]; b2 = b->sig[2]; \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:153:1: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandnsets, _sig_andn)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:129:24: note: expanded from macro '_SIG_SET_BINOP'
b3 = b->sig[3]; b2 = b->sig[2]; \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:153:1: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandnsets, _sig_andn)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:130:3: note: expanded from macro '_SIG_SET_BINOP'
r->sig[3] = op(a3, b3); \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:153:1: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandnsets, _sig_andn)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:131:3: note: expanded from macro '_SIG_SET_BINOP'
r->sig[2] = op(a2, b2); \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:153:1: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandnsets, _sig_andn)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:134:8: note: expanded from macro '_SIG_SET_BINOP'
a1 = a->sig[1]; b1 = b->sig[1]; \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:153:1: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandnsets, _sig_andn)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:134:24: note: expanded from macro '_SIG_SET_BINOP'
a1 = a->sig[1]; b1 = b->sig[1]; \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:153:1: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_BINOP(sigandnsets, _sig_andn)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:135:3: note: expanded from macro '_SIG_SET_BINOP'
r->sig[1] = op(a1, b1); \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:177:1: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_OP(signotset, _sig_not)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:164:27: note: expanded from macro '_SIG_SET_OP'
case 4: set->sig[3] = op(set->sig[3]); \
^ ~
/lib/modules/5.4.0+/build/include/linux/signal.h:176:24: note: expanded from macro '_sig_not'
#define _sig_not(x) (~(x))
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:177:1: warning: array index 3 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_OP(signotset, _sig_not)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:164:10: note: expanded from macro '_SIG_SET_OP'
case 4: set->sig[3] = op(set->sig[3]); \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:177:1: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_OP(signotset, _sig_not)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:165:20: note: expanded from macro '_SIG_SET_OP'
set->sig[2] = op(set->sig[2]); \
^ ~
/lib/modules/5.4.0+/build/include/linux/signal.h:176:24: note: expanded from macro '_sig_not'
#define _sig_not(x) (~(x))
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:177:1: warning: array index 2 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_OP(signotset, _sig_not)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:165:3: note: expanded from macro '_SIG_SET_OP'
set->sig[2] = op(set->sig[2]); \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:177:1: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_OP(signotset, _sig_not)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:167:27: note: expanded from macro '_SIG_SET_OP'
case 2: set->sig[1] = op(set->sig[1]); \
^ ~
/lib/modules/5.4.0+/build/include/linux/signal.h:176:24: note: expanded from macro '_sig_not'
#define _sig_not(x) (~(x))
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:177:1: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
_SIG_SET_OP(signotset, _sig_not)
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/lib/modules/5.4.0+/build/include/linux/signal.h:167:10: note: expanded from macro '_SIG_SET_OP'
case 2: set->sig[1] = op(set->sig[1]); \
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:188:10: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
case 2: set->sig[1] = 0;
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:201:10: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
case 2: set->sig[1] = -1;
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:232:10: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
case 2: set->sig[1] = 0;
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:6:
/lib/modules/5.4.0+/build/include/linux/signal.h:244:10: warning: array index 1 is past the end of the array (which contains 1 element) [-Warray-bounds]
case 2: set->sig[1] = -1;
^ ~
/lib/modules/5.4.0+/build/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here
unsigned long sig[_NSIG_WORDS];
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:9:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/task.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/uaccess.h:11:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:12:
/lib/modules/5.4.0+/build/arch/x86/include/asm/smap.h:47:2: error: expected '(' after 'asm'
alternative("", __ASM_CLAC, X86_FEATURE_SMAP);
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:204:2: note: expanded from macro 'alternative'
asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature) : : : "memory")
^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:9:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/task.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/uaccess.h:11:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:12:
/lib/modules/5.4.0+/build/arch/x86/include/asm/smap.h:53:2: error: expected '(' after 'asm'
alternative("", __ASM_STAC, X86_FEATURE_SMAP);
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:204:2: note: expanded from macro 'alternative'
asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature) : : : "memory")
^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:9:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/task.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/uaccess.h:11:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:694:
/lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess_64.h:37:2: error: expected '(' after 'asm'
alternative_call_2(copy_user_generic_unrolled,
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:256:2: note: expanded from macro 'alternative_call_2'
asm_inline volatile (ALTERNATIVE_2("call %P[old]", "call %P[new1]", feature1,\
^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:9:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/task.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/uaccess.h:11:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:694:
/lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess_64.h:74:3: error: expected '(' after 'asm'
__uaccess_begin_nospec();
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:125:2: note: expanded from macro '__uaccess_begin_nospec'
barrier_nospec(); \
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/barrier.h:52:26: note: expanded from macro 'barrier_nospec'
#define barrier_nospec() alternative("", "lfence", X86_FEATURE_LFENCE_RDTSC)
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:204:2: note: expanded from macro 'alternative'
asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature) : : : "memory")
^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:9:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/task.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/uaccess.h:11:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:694:
/lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess_64.h:80:3: error: expected '(' after 'asm'
__uaccess_begin_nospec();
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:125:2: note: expanded from macro '__uaccess_begin_nospec'
barrier_nospec(); \
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/barrier.h:52:26: note: expanded from macro 'barrier_nospec'
#define barrier_nospec() alternative("", "lfence", X86_FEATURE_LFENCE_RDTSC)
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:204:2: note: expanded from macro 'alternative'
asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature) : : : "memory")
^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:9:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/task.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/uaccess.h:11:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:694:
/lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess_64.h:86:3: error: expected '(' after 'asm'
__uaccess_begin_nospec();
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:125:2: note: expanded from macro '__uaccess_begin_nospec'
barrier_nospec(); \
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/barrier.h:52:26: note: expanded from macro 'barrier_nospec'
#define barrier_nospec() alternative("", "lfence", X86_FEATURE_LFENCE_RDTSC)
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:204:2: note: expanded from macro 'alternative'
asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature) : : : "memory")
^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:9:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/task.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/uaccess.h:11:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:694:
/lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess_64.h:92:3: error: expected '(' after 'asm'
__uaccess_begin_nospec();
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:125:2: note: expanded from macro '__uaccess_begin_nospec'
barrier_nospec(); \
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/barrier.h:52:26: note: expanded from macro 'barrier_nospec'
#define barrier_nospec() alternative("", "lfence", X86_FEATURE_LFENCE_RDTSC)
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:204:2: note: expanded from macro 'alternative'
asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature) : : : "memory")
^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/5.4.0+/build/include/linux/ptrace.h:7:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/signal.h:9:
In file included from /lib/modules/5.4.0+/build/include/linux/sched/task.h:11:
In file included from /lib/modules/5.4.0+/build/include/linux/uaccess.h:11:
In file included from /lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:694:
/lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess_64.h:98:3: error: expected '(' after 'asm'
__uaccess_begin_nospec();
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/uaccess.h:125:2: note: expanded from macro '__uaccess_begin_nospec'
barrier_nospec(); \
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/barrier.h:52:26: note: expanded from macro 'barrier_nospec'
#define barrier_nospec() alternative("", "lfence", X86_FEATURE_LFENCE_RDTSC)
^
/lib/modules/5.4.0+/build/arch/x86/include/asm/alternative.h:204:2: note: expanded from macro 'alternative'
asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, feature) : : : "memory")
^
/lib/modules/5.4.0+/build/include/linux/compiler_types.h:210:24: note: expanded from macro 'asm_inline'
#define asm_inline asm __inline
^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
49 warnings and 20 errors generated.
Failed to compile BPF text
[-- Attachment #3: test --]
[-- Type: application/x-shellscript, Size: 1080 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-26 9:15 ` Ming Lei
@ 2019-11-26 10:24 ` Ming Lei
2019-11-26 11:14 ` Andrea Vai
1 sibling, 0 replies; 77+ messages in thread
From: Ming Lei @ 2019-11-26 10:24 UTC (permalink / raw)
To: Andrea Vai
Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
USB list, SCSI development list, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
On Tue, Nov 26, 2019 at 05:15:33PM +0800, Ming Lei wrote:
> On Tue, Nov 26, 2019 at 08:46:07AM +0100, Andrea Vai wrote:
> > Il giorno mar, 26/11/2019 alle 10.32 +0800, Ming Lei ha scritto:
> > > On Mon, Nov 25, 2019 at 07:51:33PM +0100, Andrea Vai wrote:
> > > > Il giorno lun, 25/11/2019 alle 23.15 +0800, Ming Lei ha scritto:
> > > > > On Mon, Nov 25, 2019 at 03:58:34PM +0100, Andrea Vai wrote:
> > > > >
> > > > > [...]
> > > > >
> > > > > > What to try next?
> > > > >
> > > > > 1) cat /sys/kernel/debug/block/$DISK/hctx0/flags
> > > > result:
> > > >
> > > > alloc_policy=FIFO SHOULD_MERGE|2
> > > >
> > > > >
> > > > >
> > > > > 2) echo 128 > /sys/block/$DISK/queue/nr_requests and run your
> > > copy
> > > > > 1GB
> > > > > test again.
> > > >
> > > > done, and still fails. What to try next?
> > >
> > > I just run 256M cp test
> >
> > I would like to point out that 256MB is a filesize that usually don't
> > trigger the issue (don't know if it matters, sorry).
>
> OK.
>
> I tested 256M because IO timeout is often triggered in case of
> qemu-ehci, and it is a long-term issue. When setting up the disk
> via xhci-qemu, the max request size is increased to 1MB from 120KB,
> and IO pattern changed too. When the disk is connected via uhci-qemu,
> the transfer is too slow(1MB/s) because max endpoint size is too small.
>
> However, I just waited 16min and collected all the 1GB IO log by
> connecting disk over uhci-qemu, but the sector of each data IO
> is still in order.
>
> >
> > Another info I would provide is about another strange behavior I
> > noticed: yesterday I ran the test two times (as usual with 1GB
> > filesize) and took 2370s, 1786s, and a third test was going on when I
> > stopped it. Then I started another set of 100 trials and let them run
> > tonight, and the first 10 trials were around 1000s, then gradually
> > decreased to ~300s, and finally settled around 200s with some trials
> > below 70-80s. This to say, times are extremely variable and for the
> > first time I noticed a sort of "performance increase" with time.
>
> The 'cp' test is buffered IO, can you reproduce it every time by
> running copy just after fresh mount on the USB disk?
>
> >
> > > to one USB storage device on patched kernel,
> > > and WRITE data IO is really in ascending order. The filesystem is
> > > ext4,
> > > and mount without '-o sync'. From previous discussion, looks that is
> > > exactly your test setting. The order can be observed via the
> > > following script:
> > >
> > > #!/bin/sh
> > > MAJ=$1
> > > MIN=$2
> > > MAJ=$(( $MAJ << 20 ))
> > > DEV=$(( $MAJ | $MIN ))
> > > /usr/share/bcc/tools/trace -t -C \
> > > 't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d", args-
> > > >rwbs, args->sector, args->nr_sector'
> > >
> > > $MAJ & $MIN can be retrieved via lsblk for your USB storage disk.
> > >
> > > So I think we need to check if the patch is applied correctly first.
> > >
> > > If your kernel tree is managed via git,
> > yes it is,
> >
> > > please post 'git diff'.
> > attached. Is it correctly patched? thanks.
>
> Yeah, it should be correct except for the change on __blk_mq_delay_run_hw_queue()
> is duplicated.
>
> >
> >
> > > Otherwise, share us your kernel version,
> > btw, is 5.4.0+
> >
> > > and I will send you one
> > > backported patch on the kernel version.
> > >
> > > Meantime, you can collect IO order log via the above script as you
> > > did last
> > > time, then send us the log.
> >
> > ok, will try; is it just required to run it for a short period of time
> > (say, some seconds) during the copy, or should I run it before the
> > beginning (or before the mount?), and terminate it after the end of
> > the copy? (Please note that in the latter case a large amount of time
> > (and data, I suppose) would be involved, because, as said, to be sure
> > the problem triggers I have to use a large file... but we can try to
> > better understand and tune this. If it can help, you can get an ods
> > file with the complete statistic at [1] (look at the "prove_nov19"
> > sheet)).
>
> The data won't be very big, each line covers 120KB, and ~10K line
> is enough for cover 1GB transfer. Then ~300KB compressed file should
> hold all the trace.
Also use the following trace script this time:
#!/bin/sh
MAJ=$1
MIN=$2
MAJ=$(( $MAJ << 20 ))
DEV=$(( $MAJ | $MIN ))
/usr/share/bcc/tools/trace -t -C \
't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d", args->rwbs, args->sector, args->nr_sector' \
't:block:block_rq_insert (args->dev == '$DEV') "%s %d %d", args->rwbs, args->sector, args->nr_sector'
Thanks,
Ming
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-26 7:46 ` Andrea Vai
@ 2019-11-26 9:15 ` Ming Lei
2019-11-26 10:24 ` Ming Lei
2019-11-26 11:14 ` Andrea Vai
2019-11-27 0:21 ` Finn Thain
1 sibling, 2 replies; 77+ messages in thread
From: Ming Lei @ 2019-11-26 9:15 UTC (permalink / raw)
To: Andrea Vai
Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
USB list, SCSI development list, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
On Tue, Nov 26, 2019 at 08:46:07AM +0100, Andrea Vai wrote:
> Il giorno mar, 26/11/2019 alle 10.32 +0800, Ming Lei ha scritto:
> > On Mon, Nov 25, 2019 at 07:51:33PM +0100, Andrea Vai wrote:
> > > Il giorno lun, 25/11/2019 alle 23.15 +0800, Ming Lei ha scritto:
> > > > On Mon, Nov 25, 2019 at 03:58:34PM +0100, Andrea Vai wrote:
> > > >
> > > > [...]
> > > >
> > > > > What to try next?
> > > >
> > > > 1) cat /sys/kernel/debug/block/$DISK/hctx0/flags
> > > result:
> > >
> > > alloc_policy=FIFO SHOULD_MERGE|2
> > >
> > > >
> > > >
> > > > 2) echo 128 > /sys/block/$DISK/queue/nr_requests and run your
> > copy
> > > > 1GB
> > > > test again.
> > >
> > > done, and still fails. What to try next?
> >
> > I just run 256M cp test
>
> I would like to point out that 256MB is a filesize that usually don't
> trigger the issue (don't know if it matters, sorry).
OK.
I tested 256M because IO timeout is often triggered in case of
qemu-ehci, and it is a long-term issue. When setting up the disk
via xhci-qemu, the max request size is increased to 1MB from 120KB,
and IO pattern changed too. When the disk is connected via uhci-qemu,
the transfer is too slow(1MB/s) because max endpoint size is too small.
However, I just waited 16min and collected all the 1GB IO log by
connecting disk over uhci-qemu, but the sector of each data IO
is still in order.
>
> Another info I would provide is about another strange behavior I
> noticed: yesterday I ran the test two times (as usual with 1GB
> filesize) and took 2370s, 1786s, and a third test was going on when I
> stopped it. Then I started another set of 100 trials and let them run
> tonight, and the first 10 trials were around 1000s, then gradually
> decreased to ~300s, and finally settled around 200s with some trials
> below 70-80s. This to say, times are extremely variable and for the
> first time I noticed a sort of "performance increase" with time.
The 'cp' test is buffered IO, can you reproduce it every time by
running copy just after fresh mount on the USB disk?
>
> > to one USB storage device on patched kernel,
> > and WRITE data IO is really in ascending order. The filesystem is
> > ext4,
> > and mount without '-o sync'. From previous discussion, looks that is
> > exactly your test setting. The order can be observed via the
> > following script:
> >
> > #!/bin/sh
> > MAJ=$1
> > MIN=$2
> > MAJ=$(( $MAJ << 20 ))
> > DEV=$(( $MAJ | $MIN ))
> > /usr/share/bcc/tools/trace -t -C \
> > 't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d", args-
> > >rwbs, args->sector, args->nr_sector'
> >
> > $MAJ & $MIN can be retrieved via lsblk for your USB storage disk.
> >
> > So I think we need to check if the patch is applied correctly first.
> >
> > If your kernel tree is managed via git,
> yes it is,
>
> > please post 'git diff'.
> attached. Is it correctly patched? thanks.
Yeah, it should be correct except for the change on __blk_mq_delay_run_hw_queue()
is duplicated.
>
>
> > Otherwise, share us your kernel version,
> btw, is 5.4.0+
>
> > and I will send you one
> > backported patch on the kernel version.
> >
> > Meantime, you can collect IO order log via the above script as you
> > did last
> > time, then send us the log.
>
> ok, will try; is it just required to run it for a short period of time
> (say, some seconds) during the copy, or should I run it before the
> beginning (or before the mount?), and terminate it after the end of
> the copy? (Please note that in the latter case a large amount of time
> (and data, I suppose) would be involved, because, as said, to be sure
> the problem triggers I have to use a large file... but we can try to
> better understand and tune this. If it can help, you can get an ods
> file with the complete statistic at [1] (look at the "prove_nov19"
> sheet)).
The data won't be very big, each line covers 120KB, and ~10K line
is enough for cover 1GB transfer. Then ~300KB compressed file should
hold all the trace.
Thanks,
Ming
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-26 2:32 ` Ming Lei
@ 2019-11-26 7:46 ` Andrea Vai
2019-11-26 9:15 ` Ming Lei
2019-11-27 0:21 ` Finn Thain
0 siblings, 2 replies; 77+ messages in thread
From: Andrea Vai @ 2019-11-26 7:46 UTC (permalink / raw)
To: Ming Lei
Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
USB list, SCSI development list, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
[-- Attachment #1: Type: text/plain, Size: 2944 bytes --]
Il giorno mar, 26/11/2019 alle 10.32 +0800, Ming Lei ha scritto:
> On Mon, Nov 25, 2019 at 07:51:33PM +0100, Andrea Vai wrote:
> > Il giorno lun, 25/11/2019 alle 23.15 +0800, Ming Lei ha scritto:
> > > On Mon, Nov 25, 2019 at 03:58:34PM +0100, Andrea Vai wrote:
> > >
> > > [...]
> > >
> > > > What to try next?
> > >
> > > 1) cat /sys/kernel/debug/block/$DISK/hctx0/flags
> > result:
> >
> > alloc_policy=FIFO SHOULD_MERGE|2
> >
> > >
> > >
> > > 2) echo 128 > /sys/block/$DISK/queue/nr_requests and run your
> copy
> > > 1GB
> > > test again.
> >
> > done, and still fails. What to try next?
>
> I just run 256M cp test
I would like to point out that 256MB is a filesize that usually don't
trigger the issue (don't know if it matters, sorry).
Another info I would provide is about another strange behavior I
noticed: yesterday I ran the test two times (as usual with 1GB
filesize) and took 2370s, 1786s, and a third test was going on when I
stopped it. Then I started another set of 100 trials and let them run
tonight, and the first 10 trials were around 1000s, then gradually
decreased to ~300s, and finally settled around 200s with some trials
below 70-80s. This to say, times are extremely variable and for the
first time I noticed a sort of "performance increase" with time.
> to one USB storage device on patched kernel,
> and WRITE data IO is really in ascending order. The filesystem is
> ext4,
> and mount without '-o sync'. From previous discussion, looks that is
> exactly your test setting. The order can be observed via the
> following script:
>
> #!/bin/sh
> MAJ=$1
> MIN=$2
> MAJ=$(( $MAJ << 20 ))
> DEV=$(( $MAJ | $MIN ))
> /usr/share/bcc/tools/trace -t -C \
> 't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d", args-
> >rwbs, args->sector, args->nr_sector'
>
> $MAJ & $MIN can be retrieved via lsblk for your USB storage disk.
>
> So I think we need to check if the patch is applied correctly first.
>
> If your kernel tree is managed via git,
yes it is,
> please post 'git diff'.
attached. Is it correctly patched? thanks.
> Otherwise, share us your kernel version,
btw, is 5.4.0+
> and I will send you one
> backported patch on the kernel version.
>
> Meantime, you can collect IO order log via the above script as you
> did last
> time, then send us the log.
ok, will try; is it just required to run it for a short period of time
(say, some seconds) during the copy, or should I run it before the
beginning (or before the mount?), and terminate it after the end of
the copy? (Please note that in the latter case a large amount of time
(and data, I suppose) would be involved, because, as said, to be sure
the problem triggers I have to use a large file... but we can try to
better understand and tune this. If it can help, you can get an ods
file with the complete statistic at [1] (look at the "prove_nov19"
sheet)).
Thanks,
Andrea
[1]: http://fisica.unipv.it/transfer/kernelstats.zip
[-- Attachment #2: git_diff.txt --]
[-- Type: text/plain, Size: 6424 bytes --]
# git diff
diff --git a/block/blk-mq.c b/block/blk-mq.c
index ec791156e9cc..92d60a5e1d15 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -1465,7 +1465,13 @@ static void __blk_mq_delay_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async,
if (unlikely(blk_mq_hctx_stopped(hctx)))
return;
- if (!async && !(hctx->flags & BLK_MQ_F_BLOCKING)) {
+ /*
+ * Some single-queue devices may need to dispatch IO in order
+ * which was guaranteed for the legacy queue via the big queue
+ * lock. Now we reply on single hctx->run_work for that.
+ */
+ if (!async && !(hctx->flags & (BLK_MQ_F_BLOCKING |
+ BLK_MQ_F_STRICT_DISPATCH_ORDER))) {
int cpu = get_cpu();
if (cpumask_test_cpu(cpu, hctx->cpumask)) {
__blk_mq_run_hw_queue(hctx);
@@ -3055,6 +3061,10 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set)
if (!set->ops->get_budget ^ !set->ops->put_budget)
return -EINVAL;
+ if (set->queue_depth > 1 && (set->flags &
+ BLK_MQ_F_STRICT_DISPATCH_ORDER))
+ return -EINVAL;
+
if (set->queue_depth > BLK_MQ_MAX_DEPTH) {
pr_info("blk-mq: reduced tag depth to %u\n",
BLK_MQ_MAX_DEPTH);
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 91c007d26c1e..f013630275c9 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -1902,6 +1902,9 @@ int scsi_mq_setup_tags(struct Scsi_Host *shost)
shost->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
shost->tag_set.flags |=
BLK_ALLOC_POLICY_TO_MQ_FLAG(shost->hostt->tag_alloc_policy);
+ if (shost->hostt->strict_dispatch_order)
+ shost->tag_set.flags |= BLK_MQ_F_STRICT_DISPATCH_ORDER;
+
shost->tag_set.driver_data = shost;
return blk_mq_alloc_tag_set(&shost->tag_set);
diff --git a/drivers/usb/storage/scsiglue.c b/drivers/usb/storage/scsiglue.c
index 54a3c8195c96..df1674d7f0fc 100644
--- a/drivers/usb/storage/scsiglue.c
+++ b/drivers/usb/storage/scsiglue.c
@@ -651,6 +651,18 @@ static const struct scsi_host_template usb_stor_host_template = {
/* we do our own delay after a device or bus reset */
.skip_settle_delay = 1,
diff --git a/block/blk-mq.c b/block/blk-mq.c
index ec791156e9cc..92d60a5e1d15 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -1465,7 +1465,13 @@ static void __blk_mq_delay_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async,
if (unlikely(blk_mq_hctx_stopped(hctx)))
return;
- if (!async && !(hctx->flags & BLK_MQ_F_BLOCKING)) {
+ /*
+ * Some single-queue devices may need to dispatch IO in order
+ * which was guaranteed for the legacy queue via the big queue
+ * lock. Now we reply on single hctx->run_work for that.
+ */
+ if (!async && !(hctx->flags & (BLK_MQ_F_BLOCKING |
+ BLK_MQ_F_STRICT_DISPATCH_ORDER))) {
int cpu = get_cpu();
if (cpumask_test_cpu(cpu, hctx->cpumask)) {
__blk_mq_run_hw_queue(hctx);
@@ -3055,6 +3061,10 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set)
if (!set->ops->get_budget ^ !set->ops->put_budget)
return -EINVAL;
+ if (set->queue_depth > 1 && (set->flags &
+ BLK_MQ_F_STRICT_DISPATCH_ORDER))
+ return -EINVAL;
+
if (set->queue_depth > BLK_MQ_MAX_DEPTH) {
pr_info("blk-mq: reduced tag depth to %u\n",
BLK_MQ_MAX_DEPTH);
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 91c007d26c1e..f013630275c9 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -1902,6 +1902,9 @@ int scsi_mq_setup_tags(struct Scsi_Host *shost)
shost->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
shost->tag_set.flags |=
BLK_ALLOC_POLICY_TO_MQ_FLAG(shost->hostt->tag_alloc_policy);
+ if (shost->hostt->strict_dispatch_order)
+ shost->tag_set.flags |= BLK_MQ_F_STRICT_DISPATCH_ORDER;
+
shost->tag_set.driver_data = shost;
return blk_mq_alloc_tag_set(&shost->tag_set);
diff --git a/drivers/usb/storage/scsiglue.c b/drivers/usb/storage/scsiglue.c
index 54a3c8195c96..df1674d7f0fc 100644
--- a/drivers/usb/storage/scsiglue.c
+++ b/drivers/usb/storage/scsiglue.c
@@ -651,6 +651,18 @@ static const struct scsi_host_template usb_stor_host_template = {
/* we do our own delay after a device or bus reset */
.skip_settle_delay = 1,
+
+ /*
+ * Some USB storage, such as Kingston Technology DataTraveler 100
+ * G3/G4/SE9 G2(ID 0951:1666), requires IO dispatched in the
+ * sequential order, otherwise IO performance may drop drastically.
+ *
+ * can_queue is always 1, so we set .strict_dispatch_order for
+ * USB mass storage HBA. Another reason is that there can be such
+ * kind of devices too.
+ */
+ .strict_dispatch_order = 1,
+
/* sysfs device attributes */
.sdev_attrs = sysfs_device_attr_list,
diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
index 0bf056de5cc3..89b1c28da36a 100644
--- a/include/linux/blk-mq.h
+++ b/include/linux/blk-mq.h
@@ -226,6 +226,7 @@ struct blk_mq_ops {
enum {
BLK_MQ_F_SHOULD_MERGE = 1 << 0,
BLK_MQ_F_TAG_SHARED = 1 << 1,
+ BLK_MQ_F_STRICT_DISPATCH_ORDER = 1 << 2,
BLK_MQ_F_BLOCKING = 1 << 5,
BLK_MQ_F_NO_SCHED = 1 << 6,
BLK_MQ_F_ALLOC_POLICY_START_BIT = 8,
diff --git a/include/scsi/scsi_host.h b/include/scsi/scsi_host.h
index 31e0d6ca1eba..dbcbc9ef6993 100644
--- a/include/scsi/scsi_host.h
+++ b/include/scsi/scsi_host.h
@@ -442,6 +442,13 @@ struct scsi_host_template {
/* True if the low-level driver supports blk-mq only */
unsigned force_blk_mq:1;
+ /*
+ * True if the low-level driver needs IO to be dispatched in
+ * the order provided by legacy IO path. The flag is only
+ * valid for single queue device.
+ */
+ unsigned strict_dispatch_order:1;
+
/*
* Countdown for host blocking with no commands outstanding.
*/
^ permalink raw reply related [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-25 18:51 ` Andrea Vai
@ 2019-11-26 2:32 ` Ming Lei
2019-11-26 7:46 ` Andrea Vai
0 siblings, 1 reply; 77+ messages in thread
From: Ming Lei @ 2019-11-26 2:32 UTC (permalink / raw)
To: Andrea Vai
Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
USB list, SCSI development list, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
On Mon, Nov 25, 2019 at 07:51:33PM +0100, Andrea Vai wrote:
> Il giorno lun, 25/11/2019 alle 23.15 +0800, Ming Lei ha scritto:
> > On Mon, Nov 25, 2019 at 03:58:34PM +0100, Andrea Vai wrote:
> >
> > [...]
> >
> > > What to try next?
> >
> > 1) cat /sys/kernel/debug/block/$DISK/hctx0/flags
> result:
>
> alloc_policy=FIFO SHOULD_MERGE|2
>
> >
> >
> > 2) echo 128 > /sys/block/$DISK/queue/nr_requests and run your copy
> > 1GB
> > test again.
>
> done, and still fails. What to try next?
I just run 256M cp test to one USB storage device on patched kernel,
and WRITE data IO is really in ascending order. The filesystem is ext4,
and mount without '-o sync'. From previous discussion, looks that is
exactly your test setting. The order can be observed via the following script:
#!/bin/sh
MAJ=$1
MIN=$2
MAJ=$(( $MAJ << 20 ))
DEV=$(( $MAJ | $MIN ))
/usr/share/bcc/tools/trace -t -C \
't:block:block_rq_issue (args->dev == '$DEV') "%s %d %d", args->rwbs, args->sector, args->nr_sector'
$MAJ & $MIN can be retrieved via lsblk for your USB storage disk.
So I think we need to check if the patch is applied correctly first.
If your kernel tree is managed via git, please post 'git diff'.
Otherwise, share us your kernel version, and I will send you one
backported patch on the kernel version.
Meantime, you can collect IO order log via the above script as you did last
time, then send us the log.
Thanks,
Ming
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-25 15:15 ` Ming Lei
@ 2019-11-25 18:51 ` Andrea Vai
2019-11-26 2:32 ` Ming Lei
0 siblings, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-11-25 18:51 UTC (permalink / raw)
To: Ming Lei
Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
USB list, SCSI development list, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
Il giorno lun, 25/11/2019 alle 23.15 +0800, Ming Lei ha scritto:
> On Mon, Nov 25, 2019 at 03:58:34PM +0100, Andrea Vai wrote:
>
> [...]
>
> > What to try next?
>
> 1) cat /sys/kernel/debug/block/$DISK/hctx0/flags
result:
alloc_policy=FIFO SHOULD_MERGE|2
>
>
> 2) echo 128 > /sys/block/$DISK/queue/nr_requests and run your copy
> 1GB
> test again.
done, and still fails. What to try next?
Thanks,
Andrea
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-25 14:58 ` Andrea Vai
@ 2019-11-25 15:15 ` Ming Lei
2019-11-25 18:51 ` Andrea Vai
0 siblings, 1 reply; 77+ messages in thread
From: Ming Lei @ 2019-11-25 15:15 UTC (permalink / raw)
To: Andrea Vai
Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
USB list, SCSI development list, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
On Mon, Nov 25, 2019 at 03:58:34PM +0100, Andrea Vai wrote:
> Il giorno lun, 25/11/2019 alle 18.29 +0800, Ming Lei ha scritto:
> > On Mon, Nov 25, 2019 at 11:11:00AM +0100, Andrea Vai wrote:
> > > Il giorno lun, 25/11/2019 alle 11.54 +0800, Ming Lei ha scritto:
> > > > On Sat, Nov 23, 2019 at 04:44:55PM +0100, Andrea Vai wrote:
> > > > > Il giorno sab, 23/11/2019 alle 15.28 +0800, Ming Lei ha
> > scritto:
> > > > > >
> > > > > > Please post the log of 'lsusb -v', and I will try to make a
> > > > patch
> > > > > > for
> > > > > > addressing the issue.
> > > > >
> > > > > attached,
> > > >
> > > > Please apply the attached patch, and re-build & install & reboot
> > > > kernel.
> > > >
> > > > This time, please don't switch io scheduler.
> > >
> > > # patch -p1 < usb.patch outputs:
> > >
> > > (Stripping trailing CRs from patch; use --binary to disable.)
> > > patching file block/blk-mq.c
> > > Hunk #1 succeeded at 1465 (offset 29 lines).
> > > Hunk #2 succeeded at 3061 (offset 13 lines).
> > > (Stripping trailing CRs from patch; use --binary to disable.)
> > > patching file drivers/scsi/scsi_lib.c
> > > Hunk #1 succeeded at 1902 (offset -37 lines).
> > > (Stripping trailing CRs from patch; use --binary to disable.)
> > > patching file drivers/usb/storage/scsiglue.c
> > > Hunk #1 succeeded at 651 (offset -10 lines).
> > > (Stripping trailing CRs from patch; use --binary to disable.)
> > > patching file include/linux/blk-mq.h
> > > Hunk #1 succeeded at 226 (offset -162 lines).
> > > (Stripping trailing CRs from patch; use --binary to disable.)
> > > patching file include/scsi/scsi_host.h
> > > patch unexpectedly ends in middle of line
> > > patch unexpectedly ends in middle of line
> > >
> > > Just to be sure I have to go on, is this correct? Sounds like an
> > error
> > > but I don't know if it is important.
> >
> > Looks there is small conflict, however it has been fixed by patch,
> > so
> > it is correct, please go on your test.
>
> Done, it still fails (2000 seconds or more to copy 1GB) :-(
>
> cat /sys/block/sdf/queue/scheduler outputs:
> [mq-deadline] none
>
> What to try next?
1) cat /sys/kernel/debug/block/$DISK/hctx0/flags
note: replace $DISK with disk name of your usb drive, such as, if it is
/dev/sdb, pass $DISK as sdb.
2) echo 128 > /sys/block/$DISK/queue/nr_requests and run your copy 1GB
test again.
Thanks,
Ming
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-25 10:29 ` Ming Lei
@ 2019-11-25 14:58 ` Andrea Vai
2019-11-25 15:15 ` Ming Lei
0 siblings, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-11-25 14:58 UTC (permalink / raw)
To: Ming Lei
Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
USB list, SCSI development list, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
Il giorno lun, 25/11/2019 alle 18.29 +0800, Ming Lei ha scritto:
> On Mon, Nov 25, 2019 at 11:11:00AM +0100, Andrea Vai wrote:
> > Il giorno lun, 25/11/2019 alle 11.54 +0800, Ming Lei ha scritto:
> > > On Sat, Nov 23, 2019 at 04:44:55PM +0100, Andrea Vai wrote:
> > > > Il giorno sab, 23/11/2019 alle 15.28 +0800, Ming Lei ha
> scritto:
> > > > >
> > > > > Please post the log of 'lsusb -v', and I will try to make a
> > > patch
> > > > > for
> > > > > addressing the issue.
> > > >
> > > > attached,
> > >
> > > Please apply the attached patch, and re-build & install & reboot
> > > kernel.
> > >
> > > This time, please don't switch io scheduler.
> >
> > # patch -p1 < usb.patch outputs:
> >
> > (Stripping trailing CRs from patch; use --binary to disable.)
> > patching file block/blk-mq.c
> > Hunk #1 succeeded at 1465 (offset 29 lines).
> > Hunk #2 succeeded at 3061 (offset 13 lines).
> > (Stripping trailing CRs from patch; use --binary to disable.)
> > patching file drivers/scsi/scsi_lib.c
> > Hunk #1 succeeded at 1902 (offset -37 lines).
> > (Stripping trailing CRs from patch; use --binary to disable.)
> > patching file drivers/usb/storage/scsiglue.c
> > Hunk #1 succeeded at 651 (offset -10 lines).
> > (Stripping trailing CRs from patch; use --binary to disable.)
> > patching file include/linux/blk-mq.h
> > Hunk #1 succeeded at 226 (offset -162 lines).
> > (Stripping trailing CRs from patch; use --binary to disable.)
> > patching file include/scsi/scsi_host.h
> > patch unexpectedly ends in middle of line
> > patch unexpectedly ends in middle of line
> >
> > Just to be sure I have to go on, is this correct? Sounds like an
> error
> > but I don't know if it is important.
>
> Looks there is small conflict, however it has been fixed by patch,
> so
> it is correct, please go on your test.
Done, it still fails (2000 seconds or more to copy 1GB) :-(
cat /sys/block/sdf/queue/scheduler outputs:
[mq-deadline] none
What to try next?
Thanks,
Andrea
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-25 10:11 ` Andrea Vai
@ 2019-11-25 10:29 ` Ming Lei
2019-11-25 14:58 ` Andrea Vai
0 siblings, 1 reply; 77+ messages in thread
From: Ming Lei @ 2019-11-25 10:29 UTC (permalink / raw)
To: Andrea Vai
Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
USB list, SCSI development list, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
On Mon, Nov 25, 2019 at 11:11:00AM +0100, Andrea Vai wrote:
> Il giorno lun, 25/11/2019 alle 11.54 +0800, Ming Lei ha scritto:
> > On Sat, Nov 23, 2019 at 04:44:55PM +0100, Andrea Vai wrote:
> > > Il giorno sab, 23/11/2019 alle 15.28 +0800, Ming Lei ha scritto:
> > > >
> > > > Please post the log of 'lsusb -v', and I will try to make a
> > patch
> > > > for
> > > > addressing the issue.
> > >
> > > attached,
> >
> > Please apply the attached patch, and re-build & install & reboot
> > kernel.
> >
> > This time, please don't switch io scheduler.
>
> # patch -p1 < usb.patch outputs:
>
> (Stripping trailing CRs from patch; use --binary to disable.)
> patching file block/blk-mq.c
> Hunk #1 succeeded at 1465 (offset 29 lines).
> Hunk #2 succeeded at 3061 (offset 13 lines).
> (Stripping trailing CRs from patch; use --binary to disable.)
> patching file drivers/scsi/scsi_lib.c
> Hunk #1 succeeded at 1902 (offset -37 lines).
> (Stripping trailing CRs from patch; use --binary to disable.)
> patching file drivers/usb/storage/scsiglue.c
> Hunk #1 succeeded at 651 (offset -10 lines).
> (Stripping trailing CRs from patch; use --binary to disable.)
> patching file include/linux/blk-mq.h
> Hunk #1 succeeded at 226 (offset -162 lines).
> (Stripping trailing CRs from patch; use --binary to disable.)
> patching file include/scsi/scsi_host.h
> patch unexpectedly ends in middle of line
> patch unexpectedly ends in middle of line
>
> Just to be sure I have to go on, is this correct? Sounds like an error
> but I don't know if it is important.
Looks there is small conflict, however it has been fixed by patch, so
it is correct, please go on your test.
Thanks,
Ming
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-25 3:54 ` Ming Lei
@ 2019-11-25 10:11 ` Andrea Vai
2019-11-25 10:29 ` Ming Lei
0 siblings, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-11-25 10:11 UTC (permalink / raw)
To: Ming Lei
Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
USB list, SCSI development list, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
Il giorno lun, 25/11/2019 alle 11.54 +0800, Ming Lei ha scritto:
> On Sat, Nov 23, 2019 at 04:44:55PM +0100, Andrea Vai wrote:
> > Il giorno sab, 23/11/2019 alle 15.28 +0800, Ming Lei ha scritto:
> > >
> > > Please post the log of 'lsusb -v', and I will try to make a
> patch
> > > for
> > > addressing the issue.
> >
> > attached,
>
> Please apply the attached patch, and re-build & install & reboot
> kernel.
>
> This time, please don't switch io scheduler.
# patch -p1 < usb.patch outputs:
(Stripping trailing CRs from patch; use --binary to disable.)
patching file block/blk-mq.c
Hunk #1 succeeded at 1465 (offset 29 lines).
Hunk #2 succeeded at 3061 (offset 13 lines).
(Stripping trailing CRs from patch; use --binary to disable.)
patching file drivers/scsi/scsi_lib.c
Hunk #1 succeeded at 1902 (offset -37 lines).
(Stripping trailing CRs from patch; use --binary to disable.)
patching file drivers/usb/storage/scsiglue.c
Hunk #1 succeeded at 651 (offset -10 lines).
(Stripping trailing CRs from patch; use --binary to disable.)
patching file include/linux/blk-mq.h
Hunk #1 succeeded at 226 (offset -162 lines).
(Stripping trailing CRs from patch; use --binary to disable.)
patching file include/scsi/scsi_host.h
patch unexpectedly ends in middle of line
patch unexpectedly ends in middle of line
Just to be sure I have to go on, is this correct? Sounds like an error
but I don't know if it is important.
Thanks,
Andrea
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-23 15:44 ` Andrea Vai
@ 2019-11-25 3:54 ` Ming Lei
2019-11-25 10:11 ` Andrea Vai
0 siblings, 1 reply; 77+ messages in thread
From: Ming Lei @ 2019-11-25 3:54 UTC (permalink / raw)
To: Andrea Vai
Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
USB list, SCSI development list, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
[-- Attachment #1: Type: text/plain, Size: 400 bytes --]
On Sat, Nov 23, 2019 at 04:44:55PM +0100, Andrea Vai wrote:
> Il giorno sab, 23/11/2019 alle 15.28 +0800, Ming Lei ha scritto:
> >
> > Please post the log of 'lsusb -v', and I will try to make a patch
> > for
> > addressing the issue.
>
> attached,
Please apply the attached patch, and re-build & install & reboot kernel.
This time, please don't switch io scheduler.
Thanks,
Ming
[-- Attachment #2: usb.patch --]
[-- Type: text/plain, Size: 3616 bytes --]
diff --git a/block/blk-mq.c b/block/blk-mq.c
index 5c9adcaa27ac..eecb46020bfb 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -1436,7 +1436,13 @@ static void __blk_mq_delay_run_hw_queue(struct blk_mq_hw_ctx *hctx, bool async,
if (unlikely(blk_mq_hctx_stopped(hctx)))
return;
- if (!async && !(hctx->flags & BLK_MQ_F_BLOCKING)) {
+ /*
+ * Some single-queue devices may need to dispatch IO in order
+ * which was guaranteed for the legacy queue via the big queue
+ * lock. Now we reply on single hctx->run_work for that.
+ */
+ if (!async && !(hctx->flags & (BLK_MQ_F_BLOCKING |
+ BLK_MQ_F_STRICT_DISPATCH_ORDER))) {
int cpu = get_cpu();
if (cpumask_test_cpu(cpu, hctx->cpumask)) {
__blk_mq_run_hw_queue(hctx);
@@ -3042,6 +3048,10 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set)
if (!set->ops->get_budget ^ !set->ops->put_budget)
return -EINVAL;
+ if (set->queue_depth > 1 && (set->flags &
+ BLK_MQ_F_STRICT_DISPATCH_ORDER))
+ return -EINVAL;
+
if (set->queue_depth > BLK_MQ_MAX_DEPTH) {
pr_info("blk-mq: reduced tag depth to %u\n",
BLK_MQ_MAX_DEPTH);
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index d3d237a09a78..563188844143 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -1939,6 +1939,9 @@ int scsi_mq_setup_tags(struct Scsi_Host *shost)
shost->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
shost->tag_set.flags |=
BLK_ALLOC_POLICY_TO_MQ_FLAG(shost->hostt->tag_alloc_policy);
+ if (shost->hostt->strict_dispatch_order)
+ shost->tag_set.flags |= BLK_MQ_F_STRICT_DISPATCH_ORDER;
+
shost->tag_set.driver_data = shost;
return blk_mq_alloc_tag_set(&shost->tag_set);
diff --git a/drivers/usb/storage/scsiglue.c b/drivers/usb/storage/scsiglue.c
index 6737fab94959..77795edad8e8 100644
--- a/drivers/usb/storage/scsiglue.c
+++ b/drivers/usb/storage/scsiglue.c
@@ -661,6 +661,18 @@ static const struct scsi_host_template usb_stor_host_template = {
/* we do our own delay after a device or bus reset */
.skip_settle_delay = 1,
+
+ /*
+ * Some USB storage, such as Kingston Technology DataTraveler 100
+ * G3/G4/SE9 G2(ID 0951:1666), requires IO dispatched in the
+ * sequential order, otherwise IO performance may drop drastically.
+ *
+ * can_queue is always 1, so we set .strict_dispatch_order for
+ * USB mass storage HBA. Another reason is that there can be such
+ * kind of devices too.
+ */
+ .strict_dispatch_order = 1,
+
/* sysfs device attributes */
.sdev_attrs = sysfs_device_attr_list,
diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
index dc03e059fdff..844539690a27 100644
--- a/include/linux/blk-mq.h
+++ b/include/linux/blk-mq.h
@@ -388,6 +388,7 @@ struct blk_mq_ops {
enum {
BLK_MQ_F_SHOULD_MERGE = 1 << 0,
BLK_MQ_F_TAG_SHARED = 1 << 1,
+ BLK_MQ_F_STRICT_DISPATCH_ORDER = 1 << 2,
BLK_MQ_F_BLOCKING = 1 << 5,
BLK_MQ_F_NO_SCHED = 1 << 6,
BLK_MQ_F_ALLOC_POLICY_START_BIT = 8,
diff --git a/include/scsi/scsi_host.h b/include/scsi/scsi_host.h
index d4452d0ea3c7..f932d6fa1a4c 100644
--- a/include/scsi/scsi_host.h
+++ b/include/scsi/scsi_host.h
@@ -442,6 +442,13 @@ struct scsi_host_template {
/* True if the low-level driver supports blk-mq only */
unsigned force_blk_mq:1;
+ /*
+ * True if the low-level driver needs IO to be dispatched in
+ * the order provided by legacy IO path. The flag is only
+ * valid for single queue device.
+ */
+ unsigned strict_dispatch_order:1;
+
/*
* Countdown for host blocking with no commands outstanding.
*/
^ permalink raw reply related [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-23 7:28 ` Ming Lei
@ 2019-11-23 15:44 ` Andrea Vai
2019-11-25 3:54 ` Ming Lei
0 siblings, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-11-23 15:44 UTC (permalink / raw)
To: Ming Lei
Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
USB list, SCSI development list, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
[-- Attachment #1: Type: text/plain, Size: 201 bytes --]
Il giorno sab, 23/11/2019 alle 15.28 +0800, Ming Lei ha scritto:
>
> Please post the log of 'lsusb -v', and I will try to make a patch
> for
> addressing the issue.
attached,
Thanks, and bye
Andrea
[-- Attachment #2: lsusb.txt --]
[-- Type: text/plain, Size: 41247 bytes --]
can't get debug descriptor: Resource temporarily unavailable
Bus 002 Device 002: ID 8087:8000 Intel Corp.
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.00
bDeviceClass 9 Hub
bDeviceSubClass 0
bDeviceProtocol 1 Single TT
bMaxPacketSize0 64
idVendor 0x8087 Intel Corp.
idProduct 0x8000
bcdDevice 0.04
iManufacturer 0
iProduct 0
iSerial 0
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0019
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xe0
Self Powered
Remote Wakeup
MaxPower 0mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 1
bInterfaceClass 9 Hub
bInterfaceSubClass 0
bInterfaceProtocol 0 Full speed (or root) hub
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0001 1x 1 bytes
bInterval 12
Hub Descriptor:
bLength 9
bDescriptorType 41
nNbrPorts 6
wHubCharacteristic 0x0009
Per-port power switching
Per-port overcurrent protection
TT think time 8 FS bits
bPwrOn2PwrGood 0 * 2 milli seconds
bHubContrCurrent 0 milli Ampere
DeviceRemovable 0x00
PortPwrCtrlMask 0xff
Hub Port Status:
Port 1: 0000.0100 power
Port 2: 0000.0100 power
Port 3: 0000.0100 power
Port 4: 0000.0100 power
Port 5: 0000.0100 power
Port 6: 0000.0100 power
Device Qualifier (for other device speed):
bLength 10
bDescriptorType 6
bcdUSB 2.00
bDeviceClass 9 Hub
bDeviceSubClass 0
bDeviceProtocol 0 Full speed (or root) hub
bMaxPacketSize0 64
bNumConfigurations 1
Device Status: 0x0001
Self Powered
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.00
bDeviceClass 9 Hub
bDeviceSubClass 0
bDeviceProtocol 0 Full speed (or root) hub
bMaxPacketSize0 64
idVendor 0x1d6b Linux Foundation
idProduct 0x0002 2.0 root hub
bcdDevice 5.03
iManufacturer 3 Linux 5.3.8-200.fc30.x86_64 ehci_hcd
iProduct 2 EHCI Host Controller
iSerial 1 0000:00:1d.0
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0019
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xe0
Self Powered
Remote Wakeup
MaxPower 0mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 1
bInterfaceClass 9 Hub
bInterfaceSubClass 0
bInterfaceProtocol 0 Full speed (or root) hub
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0004 1x 4 bytes
bInterval 12
Hub Descriptor:
bLength 9
bDescriptorType 41
can't get device qualifier: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
can't get device qualifier: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
nNbrPorts 2
wHubCharacteristic 0x000a
No power switching (usb 1.0)
Per-port overcurrent protection
bPwrOn2PwrGood 10 * 2 milli seconds
bHubContrCurrent 0 milli Ampere
DeviceRemovable 0x02
PortPwrCtrlMask 0xff
Hub Port Status:
Port 1: 0000.0507 highspeed power suspend enable connect
Port 2: 0000.0100 power
Device Status: 0x0001
Self Powered
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.00
bDeviceClass 9 Hub
bDeviceSubClass 0
bDeviceProtocol 0 Full speed (or root) hub
bMaxPacketSize0 64
idVendor 0x1d6b Linux Foundation
idProduct 0x0002 2.0 root hub
bcdDevice 5.03
iManufacturer 3 Linux 5.3.8-200.fc30.x86_64 ehci_hcd
iProduct 2 EHCI Host Controller
iSerial 1 0000:05:00.2
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0019
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xe0
Self Powered
Remote Wakeup
MaxPower 0mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 1
bInterfaceClass 9 Hub
bInterfaceSubClass 0
bInterfaceProtocol 0 Full speed (or root) hub
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0004 1x 4 bytes
bInterval 12
Hub Descriptor:
bLength 9
bDescriptorType 41
nNbrPorts 4
wHubCharacteristic 0x000a
No power switching (usb 1.0)
Per-port overcurrent protection
bPwrOn2PwrGood 10 * 2 milli seconds
bHubContrCurrent 0 milli Ampere
DeviceRemovable 0x00
PortPwrCtrlMask 0xff
Hub Port Status:
Port 1: 0000.0100 power
Port 2: 0000.0100 power
Port 3: 0000.0100 power
Port 4: 0000.0100 power
Device Status: 0x0001
Self Powered
Bus 005 Device 002: ID 04a9:2206 Canon, Inc. CanoScan N650U/N656U
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 1.00
bDeviceClass 0
bDeviceSubClass 0
bDeviceProtocol 0
bMaxPacketSize0 8
idVendor 0x04a9 Canon, Inc.
idProduct 0x2206 CanoScan N650U/N656U
bcdDevice 1.00
iManufacturer 64 Canon
iProduct 77 CanoScan
iSerial 0
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0027
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0x80
(Bus Powered)
MaxPower 500mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 3
bInterfaceClass 255 Vendor Specific Class
bInterfaceSubClass 0
bInterfaceProtocol 255
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0001 1x 1 bytes
bInterval 16
Endpoint Descriptorcan't get debug descriptor: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
:
bLength 7
bDescriptorType 5
bEndpointAddress 0x82 EP 2 IN
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0040 1x 64 bytes
bInterval 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x03 EP 3 OUT
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0040 1x 64 bytes
bInterval 0
Device Status: 0x0000
(Bus Powered)
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 1.10
bDeviceClass 9 Hub
bDeviceSubClass 0
bDeviceProtocol 0 Full speed (or root) hub
bMaxPacketSize0 64
idVendor 0x1d6b Linux Foundation
idProduct 0x0001 1.1 root hub
bcdDevice 5.03
iManufacturer 3 Linux 5.3.8-200.fc30.x86_64 uhci_hcd
iProduct 2 UHCI Host Controller
iSerial 1 0000:05:00.1
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0019
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xe0
Self Powered
Remote Wakeup
MaxPower 0mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 1
bInterfaceClass 9 Hub
bInterfaceSubClass 0
bInterfaceProtocol 0 Full speed (or root) hub
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0002 1x 2 bytes
bInterval 255
Hub Descriptor:
bLength 9
bDescriptorType 41
nNbrPorts 2
wHubCharacteristic 0x000a
No power switching (usb 1.0)
Per-port overcurrent protection
bPwrOn2PwrGood 1 * 2 milli seconds
bHubContrCurrent 0 milli Ampere
DeviceRemovable 0x00
PortPwrCtrlMask 0xff
Hub Port Status:
Port 1: 0000.0100 power
Port 2: 0000.0103 power enable connect
Device Status: 0x0001
Self Powered
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 1.10
bDeviceClass 9 Hub
bDeviceSubClass 0
bDeviceProtocol 0 Full speed (or root) hub
bMaxPacketSize0 64
idVendor 0x1d6b Linux Foundation
idProduct 0x0001 1.1 root hub
bcdDevice 5.03
iManufacturer 3 Linux 5.3.8-200.fc30.x86_64 uhci_hcd
iProduct 2 UHCI Host Controller
iSerial 1 0000:05:00.0
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0019
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xe0
Self Powered
Remote Wakeup
MaxPower 0mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 1
bInterfaceClass 9 Hub
bInterfaceSubClass 0
bInterfaceProtocol 0 Full speed (or root) hub
iIntercan't get debug descriptor: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
face 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0002 1x 2 bytes
bInterval 255
Hub Descriptor:
bLength 9
bDescriptorType 41
nNbrPorts 2
wHubCharacteristic 0x000a
No power switching (usb 1.0)
Per-port overcurrent protection
bPwrOn2PwrGood 1 * 2 milli seconds
bHubContrCurrent 0 milli Ampere
DeviceRemovable 0x00
PortPwrCtrlMask 0xff
Hub Port Status:
Port 1: 0000.0100 power
Port 2: 0000.0100 power
Device Status: 0x0001
Self Powered
Bus 001 Device 002: ID 8087:8008 Intel Corp.
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.00
bDeviceClass 9 Hub
bDeviceSubClass 0
bDeviceProtocol 1 Single TT
bMaxPacketSize0 64
idVendor 0x8087 Intel Corp.
idProduct 0x8008
bcdDevice 0.04
iManufacturer 0
iProduct 0
iSerial 0
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0019
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xe0
Self Powered
Remote Wakeup
MaxPower 0mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 1
bInterfaceClass 9 Hub
bInterfaceSubClass 0
bInterfaceProtocol 0 Full speed (or root) hub
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0001 1x 1 bytes
bInterval 12
Hub Descriptor:
bLength 9
bDescriptorType 41
nNbrPorts 6
wHubCharacteristic 0x0009
Per-port power switching
Per-port overcurrent protection
TT think time 8 FS bits
bPwrOn2PwrGood 0 * 2 milli seconds
bHubContrCurrent 0 milli Ampere
DeviceRemovable 0x00
PortPwrCtrlMask 0xff
Hub Port Status:
Port 1: 0000.0100 power
Port 2: 0000.0100 power
Port 3: 0000.0100 power
Port 4: 0000.0100 power
Port 5: 0000.0100 power
Port 6: 0000.0100 power
Device Qualifier (for other device speed):
bLength 10
bDescriptorType 6
bcdUSB 2.00
bDeviceClass 9 Hub
bDeviceSubClass 0
bDeviceProtocol 0 Full speed (or root) hub
bMaxPacketSize0 64
bNumConfigurations 1
Device Status: 0x0001
Self Powered
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.00
bDeviceClass 9 Hub
bDeviceSubClass 0
bDeviceProtocol 0 Full speed (or root) hub
bMaxPacketSize0 64
idVendor 0x1d6b Linux Foundation
idProduct 0x0002 2.0 root hub
bcdDevice 5.03
iManufacturer 3 Linux 5.3.8-200.fc30.x86_64 ehci_hcd
iProduct 2 EHCI Host Controller
iSerial 1 0000:00:1a.0
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0019
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xe0
Self Powered
can't get device qualifier: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
Remote Wakeup
MaxPower 0mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 1
bInterfaceClass 9 Hub
bInterfaceSubClass 0
bInterfaceProtocol 0 Full speed (or root) hub
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0004 1x 4 bytes
bInterval 12
Hub Descriptor:
bLength 9
bDescriptorType 41
nNbrPorts 2
wHubCharacteristic 0x000a
No power switching (usb 1.0)
Per-port overcurrent protection
bPwrOn2PwrGood 10 * 2 milli seconds
bHubContrCurrent 0 milli Ampere
DeviceRemovable 0x02
PortPwrCtrlMask 0xff
Hub Port Status:
Port 1: 0000.0507 highspeed power suspend enable connect
Port 2: 0000.0100 power
Device Status: 0x0001
Self Powered
Bus 007 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 3.00
bDeviceClass 9 Hub
bDeviceSubClass 0
bDeviceProtocol 3
bMaxPacketSize0 9
idVendor 0x1d6b Linux Foundation
idProduct 0x0003 3.0 root hub
bcdDevice 5.03
iManufacturer 3 Linux 5.3.8-200.fc30.x86_64 xhci-hcd
iProduct 2 xHCI Host Controller
iSerial 1 0000:00:14.0
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x001f
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xe0
Self Powered
Remote Wakeup
MaxPower 0mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 1
bInterfaceClass 9 Hub
bInterfaceSubClass 0
bInterfaceProtocol 0 Full speed (or root) hub
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0004 1x 4 bytes
bInterval 12
bMaxBurst 0
Hub Descriptor:
bLength 12
bDescriptorType 42
nNbrPorts 6
wHubCharacteristic 0x000a
No power switching (usb 1.0)
Per-port overcurrent protection
bPwrOn2PwrGood 10 * 2 milli seconds
bHubContrCurrent 0 milli Ampere
bHubDecLat 0.0 micro seconds
wHubDelay 0 nano seconds
DeviceRemovable 0x00
Hub Port Status:
Port 1: 0000.02a0 5Gbps power Rx.Detect
Port 2: 0000.02a0 5Gbps power Rx.Detect
Port 3: 0000.02a0 5Gbps power Rx.Detect
Port 4: 0000.02a0 5Gbps power Rx.Detect
Port 5: 0000.02a0 5Gbps power Rx.Detect
Port 6: 0000.02a0 5Gbps power Rx.Detect
Binary Object Store Descriptor:
bLength 5
bDescriptorType 15
wTotalLength 0x000f
bNumDeviceCaps 1
SuperSpeed USB Device Capability:
bLength 10
bDescriptorType 16
bDevCapabilityType 3
bmAttributes 0x02
Latency Tolerance Messages (LTM) Supported
wSpeedsSupported 0x0008
Device can operate at SuperSpeed (5Gbps)
bFunctionalitySupport 3
Lowest fully-functional device speed is SuperSpeed (5Gbps)
bU1DevExitLat 10 mican't get debug descriptor: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
cannot read device status, Resource temporarily unavailable (11)
can't get debug descriptor: Resource temporarily unavailable
cro seconds
bU2DevExitLat 512 micro seconds
Device Status: 0x0001
Self Powered
Bus 006 Device 007: ID 0aec:3050 Neodio Technologies Corp. ND3050 8-in-1 Card Reader
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 1.10
bDeviceClass 0
bDeviceSubClass 0
bDeviceProtocol 0
bMaxPacketSize0 16
idVendor 0x0aec Neodio Technologies Corp.
idProduct 0x3050 ND3050 8-in-1 Card Reader
bcdDevice 1.00
iManufacturer 1 (error)
iProduct 2 (error)
iSerial 3 (error)
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0020
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0x80
(Bus Powered)
MaxPower 100mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 2
bInterfaceClass 8 Mass Storage
bInterfaceSubClass 6 SCSI
bInterfaceProtocol 80 Bulk-Only
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x01 EP 1 OUT
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0040 1x 64 bytes
bInterval 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x82 EP 2 IN
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0040 1x 64 bytes
bInterval 0
Bus 006 Device 005: ID 051d:0002 American Power Conversion Uninterruptible Power Supply
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 1.10
bDeviceClass 0
bDeviceSubClass 0
bDeviceProtocol 0
bMaxPacketSize0 8
idVendor 0x051d American Power Conversion
idProduct 0x0002 Uninterruptible Power Supply
bcdDevice 1.06
iManufacturer 3 American Power Conversion
iProduct 1 Back-UPS XS 700U FW:924.Z3 .I USB FW:Z3
iSerial 2 3B1828X60578
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0022
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xe0
Self Powered
Remote Wakeup
MaxPower 24mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 1
bInterfaceClass 3 Human Interface Device
bInterfaceSubClass 0
bInterfaceProtocol 0
iInterface 0
HID Device Descriptor:
bLength 9
bDescriptorType 33
bcdHID 1.10
bCountryCode 33 US
bNumDescriptors 1
bDescriptorType 34 Report
wDescriptorLength 1029
Report Descriptors:
** UNAVAILABLE **
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0006 1x 6 bytes
bInterval 10
Device Status: 0xcan't get device qualifier: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
cannot read device status, Resource temporarily unavailable (11)
0000
(Bus Powered)
Bus 006 Device 003: ID 04e8:330e Samsung Electronics Co., Ltd ML-2950 Series
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.00
bDeviceClass 0
bDeviceSubClass 0
bDeviceProtocol 0
bMaxPacketSize0 64
idVendor 0x04e8 Samsung Electronics Co., Ltd
idProduct 0x330e
bcdDevice 1.00
iManufacturer 1 (error)
iProduct 2 (error)
iSerial 3 (error)
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0020
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xc0
Self Powered
MaxPower 2mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 2
bInterfaceClass 7 Printer
bInterfaceSubClass 1 Printer
bInterfaceProtocol 2 Bidirectional
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x02 EP 2 OUT
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0200 1x 512 bytes
bInterval 10
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0200 1x 512 bytes
bInterval 10
Bus 006 Device 004: ID 046d:c52f Logitech, Inc. Unifying Receiver
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.00
bDeviceClass 0
bDeviceSubClass 0
bDeviceProtocol 0
bMaxPacketSize0 8
idVendor 0x046d Logitech, Inc.
idProduct 0xc52f Unifying Receiver
bcdDevice 22.00
iManufacturer 1 Logitech
iProduct 2 USB Receiver
iSerial 0
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x003b
bNumInterfaces 2
bConfigurationValue 1
iConfiguration 4 RQR22.00_B0005
bmAttributes 0xa0
(Bus Powered)
Remote Wakeup
MaxPower 98mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 1
bInterfaceClass 3 Human Interface Device
bInterfaceSubClass 1 Boot Interface Subclass
bInterfaceProtocol 2 Mouse
iInterface 0
HID Device Descriptor:
bLength 9
bDescriptorType 33
bcdHID 1.11
bCountryCode 0 Not supported
bNumDescriptors 1
bDescriptorType 34 Report
wDescriptorLength 67
Report Descriptors:
** UNAVAILABLE **
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0008 1x 8 bytes
bInterval 2
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 1
bAlternateSetting 0
bNumEndpoints can't get device qualifier: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
1
bInterfaceClass 3 Human Interface Device
bInterfaceSubClass 0
bInterfaceProtocol 0
iInterface 0
HID Device Descriptor:
bLength 9
bDescriptorType 33
bcdHID 1.11
bCountryCode 0 Not supported
bNumDescriptors 1
bDescriptorType 34 Report
wDescriptorLength 79
Report Descriptors:
** UNAVAILABLE **
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x82 EP 2 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0014 1x 20 bytes
bInterval 2
Device Status: 0x0000
(Bus Powered)
Bus 006 Device 008: ID 0951:1666 Kingston Technology DataTraveler 100 G3/G4/SE9 G2
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.10
bDeviceClass 0
bDeviceSubClass 0
bDeviceProtocol 0
bMaxPacketSize0 64
idVendor 0x0951 Kingston Technology
idProduct 0x1666 DataTraveler 100 G3/G4/SE9 G2
bcdDevice 0.01
iManufacturer 1 Kingston
iProduct 2 DataTraveler 3.0
iSerial 3 60A44C4139D4FF70899506DC
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0020
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0x80
(Bus Powered)
MaxPower 500mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 2
bInterfaceClass 8 Mass Storage
bInterfaceSubClass 6 SCSI
bInterfaceProtocol 80 Bulk-Only
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0200 1x 512 bytes
bInterval 255
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x02 EP 2 OUT
bmAttributes 2
Transfer Type Bulk
Synch Type None
Usage Type Data
wMaxPacketSize 0x0200 1x 512 bytes
bInterval 255
Binary Object Store Descriptor:
bLength 5
bDescriptorType 15
wTotalLength 0x0016
bNumDeviceCaps 2
USB 2.0 Extension Device Capability:
bLength 7
bDescriptorType 16
bDevCapabilityType 2
bmAttributes 0x00000006
BESL Link Power Management (LPM) Supported
SuperSpeed USB Device Capability:
bLength 10
bDescriptorType 16
bDevCapabilityType 3
bmAttributes 0x00
wSpeedsSupported 0x000e
Device can operate at Full Speed (12Mbps)
Device can operate at High Speed (480Mbps)
Device can operate at SuperSpeed (5Gbps)
bFunctionalitySupport 2
Lowest fully-functional device speed is High Speed (480Mbps)
bU1DevExitLat 10 micro seconds
bU2DevExitLat 2047 micro seconds
Device Status: 0x0000
(Bus Powered)
Bus 006 Device 006: ID 045b:0209 Hitachi, Ltd
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.00
bDeviceClass 9 Hub
bDeviceSubClass 0
bDeviceProtocol 1 Single TT
bMaxPcan't get debug descriptor: Resource temporarily unavailable
acketSize0 64
idVendor 0x045b Hitachi, Ltd
idProduct 0x0209
bcdDevice 1.00
iManufacturer 0
iProduct 0
iSerial 0
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0019
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xe0
Self Powered
Remote Wakeup
MaxPower 0mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 1
bInterfaceClass 9 Hub
bInterfaceSubClass 0
bInterfaceProtocol 0 Full speed (or root) hub
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0001 1x 1 bytes
bInterval 12
Hub Descriptor:
bLength 9
bDescriptorType 41
nNbrPorts 4
wHubCharacteristic 0x0029
Per-port power switching
Per-port overcurrent protection
TT think time 16 FS bits
bPwrOn2PwrGood 50 * 2 milli seconds
bHubContrCurrent 100 milli Ampere
DeviceRemovable 0x00
PortPwrCtrlMask 0xff
Hub Port Status:
Port 1: 0000.0503 highspeed power enable connect
Port 2: 0000.0100 power
Port 3: 0000.0100 power
Port 4: 0000.0100 power
Device Qualifier (for other device speed):
bLength 10
bDescriptorType 6
bcdUSB 2.00
bDeviceClass 9 Hub
bDeviceSubClass 0
bDeviceProtocol 0 Full speed (or root) hub
bMaxPacketSize0 64
bNumConfigurations 1
Device Status: 0x0001
Self Powered
Bus 006 Device 002: ID 045b:0209 Hitachi, Ltd
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.00
bDeviceClass 9 Hub
bDeviceSubClass 0
bDeviceProtocol 1 Single TT
bMaxPacketSize0 64
idVendor 0x045b Hitachi, Ltd
idProduct 0x0209
bcdDevice 1.00
iManufacturer 0
iProduct 0
iSerial 0
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0019
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xe0
Self Powered
Remote Wakeup
MaxPower 0mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 1
bInterfaceClass 9 Hub
bInterfaceSubClass 0
bInterfaceProtocol 0 Full speed (or root) hub
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0001 1x 1 bytes
bInterval 12
Hub Descriptor:
bLength 9
bDescriptorType 41
nNbrPorts 4
wHubCharacteristic 0x0029
Per-port power switching
Per-port overcurrent protection
TT think time 16 FS bits
bPwrOn2PwrGood 50 * 2 milli seconds
bHubContrCurrent 100 milli Ampere
DeviceRemovable 0x00
PortPwrCtrlMask 0xff
Hub Port Status:
Port 1: 0000.0503 highspeed power enable connect
Port 2: 0000.0100 power
Port 3: 0000.010can't get debug descriptor: Resource temporarily unavailable
can't get device qualifier: Resource temporarily unavailable
can't get debug descriptor: Resource temporarily unavailable
0 power
Port 4: 0000.0103 power enable connect
Device Qualifier (for other device speed):
bLength 10
bDescriptorType 6
bcdUSB 2.00
bDeviceClass 9 Hub
bDeviceSubClass 0
bDeviceProtocol 0 Full speed (or root) hub
bMaxPacketSize0 64
bNumConfigurations 1
Device Status: 0x0001
Self Powered
Bus 006 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.00
bDeviceClass 9 Hub
bDeviceSubClass 0
bDeviceProtocol 1 Single TT
bMaxPacketSize0 64
idVendor 0x1d6b Linux Foundation
idProduct 0x0002 2.0 root hub
bcdDevice 5.03
iManufacturer 3 Linux 5.3.8-200.fc30.x86_64 xhci-hcd
iProduct 2 xHCI Host Controller
iSerial 1 0000:00:14.0
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 0x0019
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 0
bmAttributes 0xe0
Self Powered
Remote Wakeup
MaxPower 0mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 1
bInterfaceClass 9 Hub
bInterfaceSubClass 0
bInterfaceProtocol 0 Full speed (or root) hub
iInterface 0
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 3
Transfer Type Interrupt
Synch Type None
Usage Type Data
wMaxPacketSize 0x0004 1x 4 bytes
bInterval 12
Hub Descriptor:
bLength 11
bDescriptorType 41
nNbrPorts 12
wHubCharacteristic 0x000a
No power switching (usb 1.0)
Per-port overcurrent protection
TT think time 8 FS bits
bPwrOn2PwrGood 10 * 2 milli seconds
bHubContrCurrent 0 milli Ampere
DeviceRemovable 0x00 0x00
PortPwrCtrlMask 0xff 0xff
Hub Port Status:
Port 1: 0000.0100 power
Port 2: 0000.0100 power
Port 3: 0000.0100 power
Port 4: 0000.0503 highspeed power enable connect
Port 5: 0000.0503 highspeed power enable connect
Port 6: 0000.0303 lowspeed power enable connect
Port 7: 0000.0100 power
Port 8: 0000.0103 power enable connect
Port 9: 0000.0100 power
Port 10: 0000.0100 power
Port 11: 0000.0100 power
Port 12: 0000.0100 power
Device Status: 0x0001
Self Powered
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-22 19:16 ` Andrea Vai
@ 2019-11-23 7:28 ` Ming Lei
2019-11-23 15:44 ` Andrea Vai
0 siblings, 1 reply; 77+ messages in thread
From: Ming Lei @ 2019-11-23 7:28 UTC (permalink / raw)
To: Andrea Vai
Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
USB list, SCSI development list, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
On Fri, Nov 22, 2019 at 08:16:30PM +0100, Andrea Vai wrote:
> Il giorno dom, 10/11/2019 alle 06.28 +0800, Ming Lei ha scritto:
> > Another thing we could try is to use 'none' via the following
> > command:
> >
> > echo none > /sys/block/sdh/queue/scheduler #suppose 'sdh' points
> > to the usb storage disk
> >
> > Because USB storage HBA is single hw queue, which depth is 1. This
> > way
> > should change to dispatch IO in the order of bio submission.
> >
> > Andrea, could you switch io scheduler to none and update us if
> > difference
> > can be made?
>
> Using the new kernel, there is indeed a difference because the time to
> copy a file is 1800 seconds with [mq-deadline], and 340 seconds with
> [none]. But that is still far away from the old kernel, which performs
> the copy of the same file in 76 seconds.
Please post the log of 'lsusb -v', and I will try to make a patch for
addressing the issue.
thanks,
Ming
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-09 22:28 ` Ming Lei
2019-11-11 10:50 ` Andrea Vai
@ 2019-11-22 19:16 ` Andrea Vai
2019-11-23 7:28 ` Ming Lei
1 sibling, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-11-22 19:16 UTC (permalink / raw)
To: Ming Lei
Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
USB list, SCSI development list, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
Il giorno dom, 10/11/2019 alle 06.28 +0800, Ming Lei ha scritto:
> Another thing we could try is to use 'none' via the following
> command:
>
> echo none > /sys/block/sdh/queue/scheduler #suppose 'sdh' points
> to the usb storage disk
>
> Because USB storage HBA is single hw queue, which depth is 1. This
> way
> should change to dispatch IO in the order of bio submission.
>
> Andrea, could you switch io scheduler to none and update us if
> difference
> can be made?
Using the new kernel, there is indeed a difference because the time to
copy a file is 1800 seconds with [mq-deadline], and 340 seconds with
[none]. But that is still far away from the old kernel, which performs
the copy of the same file in 76 seconds.
Side notes:
- The numbers above are average values calculated on 100 trials for
each different situation. As previously noticed on this thread, with
the new kernel the times are also very different among the different
trials in the same situation. With the old kernel the standard
deviation on the times in a set of 100 trials is much smaller (to give
some mean/sigma values: m=1800->s=530; m=340->s=131; m=76->s=13; ).
- The size of the transferred file has been 1GB in these trials.
Smaller files don't always give appreciable differences, but if you
want I can also provide those data. Of course, I can also provide the
raw data of each set of trials.
Thanks,
and bye,
Andrea
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-11 11:05 ` Ming Lei
@ 2019-11-11 11:13 ` Andrea Vai
0 siblings, 0 replies; 77+ messages in thread
From: Andrea Vai @ 2019-11-11 11:13 UTC (permalink / raw)
To: Ming Lei
Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
USB list, SCSI development list, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
Il giorno lun, 11/11/2019 alle 19.05 +0800, Ming Lei ha scritto:
> On Mon, Nov 11, 2019 at 11:50:49AM +0100, Andrea Vai wrote:
> > Il giorno dom, 10/11/2019 alle 06.28 +0800, Ming Lei ha scritto:
> > > On Thu, Nov 07, 2019 at 07:59:44PM +0100, Andrea Vai wrote:
> > > > [Sorry for the duplicate message, it didn't reach the lists
> due to
> > > > html formatting]
> > > > Il giorno gio 7 nov 2019 alle ore 08:54 Damien Le Moal
> > > > <Damien.LeMoal@wdc.com> ha scritto:
> > > > >
> > > > > On 2019/11/07 16:04, Andrea Vai wrote:
> > > > > > Il giorno mer, 06/11/2019 alle 22.13 +0000, Damien Le Moal
> ha
> > > scritto:
> > > > > >>
> > > > > >>
> > > > > >> Please simply try your write tests after doing this:
> > > > > >>
> > > > > >> echo mq-deadline > /sys/block/<name of your USB
> > > > > >> disk>/queue/scheduler
> > > > > >>
> > > > > >> And confirm that mq-deadline is selected with:
> > > > > >>
> > > > > >> cat /sys/block/<name of your USB disk>/queue/scheduler
> > > > > >> [mq-deadline] kyber bfq none
> > > > > >
> > > > > > ok, which kernel should I test with this: the fresh git
> > > cloned, or the
> > > > > > one just patched with Alan's patch, or doesn't matter
> which
> > > one?
> > > > >
> > > > > Probably all of them to see if there are any differences.
> > > >
> > > > with both kernels, the output of
> > > > cat /sys/block/sdh/queue/schedule
> > > >
> > > > already contains [mq-deadline]: is it correct to assume that
> the
> > > echo
> > > > command and the subsequent testing is useless? What to do now?
> > >
> > > Another thing we could try is to use 'none' via the following
> > > command:
> > >
> > > echo none > /sys/block/sdh/queue/scheduler #suppose 'sdh'
> points
> > > to the usb storage disk
> > >
> > > Because USB storage HBA is single hw queue, which depth is 1.
> This
> > > way
> > > should change to dispatch IO in the order of bio submission.
> > >
> > > Andrea, could you switch io scheduler to none and update us if
> > > difference
> > > can be made?
> >
> > Of course I would to it, but I see that with the "good" kernel the
> > output of "cat /sys/block/sdf/queue/scheduler" (yes, now it's sdf)
> is
> >
> > noop deadline [cfq]
>
> Not sure if cfq makes a difference, and I guess you may get same
> result
> with noop or deadline. However, if you only see good write
> performance with
> cfq, you may try 'bfq' and see if it works as cfq.
>
> >
> > , i.e. it doesn't show "none". Does it matter? (sorry if it's a
> silly
> > question)
>
> We are talking about new kernel in which there can't be 'noop
> deadline [cfq]'
> any more. And you should see the following output from
> '/sys/block/sdf/queue/scheduler'
> in the new kernel:
>
> [mq-deadline] kyber bfq none
>
>
ok sorry I misunderstood, assumed you wanted me to compare the "none"
setting in the old kernel with the same setting in the new kernel. Now
it's clear to me that you want me to compare the different scheduler
settings in the new kernel.
Thanks, and bye
Andrea
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-11 10:50 ` Andrea Vai
@ 2019-11-11 11:05 ` Ming Lei
2019-11-11 11:13 ` Andrea Vai
0 siblings, 1 reply; 77+ messages in thread
From: Ming Lei @ 2019-11-11 11:05 UTC (permalink / raw)
To: Andrea Vai
Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
USB list, SCSI development list, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
On Mon, Nov 11, 2019 at 11:50:49AM +0100, Andrea Vai wrote:
> Il giorno dom, 10/11/2019 alle 06.28 +0800, Ming Lei ha scritto:
> > On Thu, Nov 07, 2019 at 07:59:44PM +0100, Andrea Vai wrote:
> > > [Sorry for the duplicate message, it didn't reach the lists due to
> > > html formatting]
> > > Il giorno gio 7 nov 2019 alle ore 08:54 Damien Le Moal
> > > <Damien.LeMoal@wdc.com> ha scritto:
> > > >
> > > > On 2019/11/07 16:04, Andrea Vai wrote:
> > > > > Il giorno mer, 06/11/2019 alle 22.13 +0000, Damien Le Moal ha
> > scritto:
> > > > >>
> > > > >>
> > > > >> Please simply try your write tests after doing this:
> > > > >>
> > > > >> echo mq-deadline > /sys/block/<name of your USB
> > > > >> disk>/queue/scheduler
> > > > >>
> > > > >> And confirm that mq-deadline is selected with:
> > > > >>
> > > > >> cat /sys/block/<name of your USB disk>/queue/scheduler
> > > > >> [mq-deadline] kyber bfq none
> > > > >
> > > > > ok, which kernel should I test with this: the fresh git
> > cloned, or the
> > > > > one just patched with Alan's patch, or doesn't matter which
> > one?
> > > >
> > > > Probably all of them to see if there are any differences.
> > >
> > > with both kernels, the output of
> > > cat /sys/block/sdh/queue/schedule
> > >
> > > already contains [mq-deadline]: is it correct to assume that the
> > echo
> > > command and the subsequent testing is useless? What to do now?
> >
> > Another thing we could try is to use 'none' via the following
> > command:
> >
> > echo none > /sys/block/sdh/queue/scheduler #suppose 'sdh' points
> > to the usb storage disk
> >
> > Because USB storage HBA is single hw queue, which depth is 1. This
> > way
> > should change to dispatch IO in the order of bio submission.
> >
> > Andrea, could you switch io scheduler to none and update us if
> > difference
> > can be made?
>
> Of course I would to it, but I see that with the "good" kernel the
> output of "cat /sys/block/sdf/queue/scheduler" (yes, now it's sdf) is
>
> noop deadline [cfq]
Not sure if cfq makes a difference, and I guess you may get same result
with noop or deadline. However, if you only see good write performance with
cfq, you may try 'bfq' and see if it works as cfq.
>
> , i.e. it doesn't show "none". Does it matter? (sorry if it's a silly
> question)
We are talking about new kernel in which there can't be 'noop deadline [cfq]'
any more. And you should see the following output from '/sys/block/sdf/queue/scheduler'
in the new kernel:
[mq-deadline] kyber bfq none
thanks,
Ming
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-09 22:28 ` Ming Lei
@ 2019-11-11 10:50 ` Andrea Vai
2019-11-11 11:05 ` Ming Lei
2019-11-22 19:16 ` Andrea Vai
1 sibling, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-11-11 10:50 UTC (permalink / raw)
To: Ming Lei
Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
USB list, SCSI development list, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
Il giorno dom, 10/11/2019 alle 06.28 +0800, Ming Lei ha scritto:
> On Thu, Nov 07, 2019 at 07:59:44PM +0100, Andrea Vai wrote:
> > [Sorry for the duplicate message, it didn't reach the lists due to
> > html formatting]
> > Il giorno gio 7 nov 2019 alle ore 08:54 Damien Le Moal
> > <Damien.LeMoal@wdc.com> ha scritto:
> > >
> > > On 2019/11/07 16:04, Andrea Vai wrote:
> > > > Il giorno mer, 06/11/2019 alle 22.13 +0000, Damien Le Moal ha
> scritto:
> > > >>
> > > >>
> > > >> Please simply try your write tests after doing this:
> > > >>
> > > >> echo mq-deadline > /sys/block/<name of your USB
> > > >> disk>/queue/scheduler
> > > >>
> > > >> And confirm that mq-deadline is selected with:
> > > >>
> > > >> cat /sys/block/<name of your USB disk>/queue/scheduler
> > > >> [mq-deadline] kyber bfq none
> > > >
> > > > ok, which kernel should I test with this: the fresh git
> cloned, or the
> > > > one just patched with Alan's patch, or doesn't matter which
> one?
> > >
> > > Probably all of them to see if there are any differences.
> >
> > with both kernels, the output of
> > cat /sys/block/sdh/queue/schedule
> >
> > already contains [mq-deadline]: is it correct to assume that the
> echo
> > command and the subsequent testing is useless? What to do now?
>
> Another thing we could try is to use 'none' via the following
> command:
>
> echo none > /sys/block/sdh/queue/scheduler #suppose 'sdh' points
> to the usb storage disk
>
> Because USB storage HBA is single hw queue, which depth is 1. This
> way
> should change to dispatch IO in the order of bio submission.
>
> Andrea, could you switch io scheduler to none and update us if
> difference
> can be made?
Of course I would to it, but I see that with the "good" kernel the
output of "cat /sys/block/sdf/queue/scheduler" (yes, now it's sdf) is
noop deadline [cfq]
, i.e. it doesn't show "none". Does it matter? (sorry if it's a silly
question)
Thanks, and bye
Andrea
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-08 14:33 ` Jens Axboe
@ 2019-11-11 10:46 ` Andrea Vai
0 siblings, 0 replies; 77+ messages in thread
From: Andrea Vai @ 2019-11-11 10:46 UTC (permalink / raw)
To: Jens Axboe, Damien Le Moal
Cc: Alan Stern, Johannes Thumshirn, USB list, SCSI development list,
Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
Martin K. Petersen, Greg KH, Hans Holmberg,
Kernel development list
Il giorno ven, 08/11/2019 alle 07.33 -0700, Jens Axboe ha scritto:
> On 11/8/19 1:42 AM, Damien Le Moal wrote:
> > On 2019/11/08 4:00, Andrea Vai wrote:
> >> [Sorry for the duplicate message, it didn't reach the lists due
> to
> >> html formatting]
> >> Il giorno gio 7 nov 2019 alle ore 08:54 Damien Le Moal
> >> <Damien.LeMoal@wdc.com> ha scritto:
> >>>
> >>> On 2019/11/07 16:04, Andrea Vai wrote:
> >>>> Il giorno mer, 06/11/2019 alle 22.13 +0000, Damien Le Moal ha
> scritto:
> >>>>>
> >>>>>
> >>>>> Please simply try your write tests after doing this:
> >>>>>
> >>>>> echo mq-deadline > /sys/block/<name of your USB
> >>>>> disk>/queue/scheduler
> >>>>>
> >>>>> And confirm that mq-deadline is selected with:
> >>>>>
> >>>>> cat /sys/block/<name of your USB disk>/queue/scheduler
> >>>>> [mq-deadline] kyber bfq none
> >>>>
> >>>> ok, which kernel should I test with this: the fresh git cloned,
> or the
> >>>> one just patched with Alan's patch, or doesn't matter which
> one?
> >>>
> >>> Probably all of them to see if there are any differences.
> >>
> >> with both kernels, the output of
> >> cat /sys/block/sdh/queue/schedule
> >>
> >> already contains [mq-deadline]: is it correct to assume that the
> echo
> >> command and the subsequent testing is useless? What to do now?
> >
> > Probably, yes. Have you obtained a blktrace of the workload during
> these
> > tests ? Any significant difference in the IO pattern (IO size and
> > randomness) and IO timing (any device idle time where the device
> has no
> > command to process) ? Asking because the problem may be above the
> block
> > layer, with the file system for instance.
>
> blktrace would indeed be super useful, especially if you can do that
> with a kernel that's fast for you, and one with the current kernel
> where it's slow.
>
> Given that your device is sdh, you simply do:
>
> # blktrace /dev/sdh
>
> and then run the test, then ctrl-c the blktrace. Then do:
>
> # blkparse sdh > output
>
> and save that output file. Do both runs, and bzip2 them up. The
> shorter
> the run you can reproduce with the better, to cut down on the size
> of
> the traces.
Sorry, the next message from Ming...
-----
You may get the IO pattern via the previous trace
https://lore.kernel.org/linux-usb/20190710024439.GA2621@ming.t460p/
IMO, if it is related write order, one possibility could be that
the queue lock is killed in .make_request_fn().
-----
...made me wonder if I should really do the blkparse trace test, or
not. So please confirm if it's needed (testing is quite time-consuming
, so I'd like to do it if it's needed).
Thanks, and bye,
Andrea
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-07 18:59 ` Andrea Vai
2019-11-08 8:42 ` Damien Le Moal
@ 2019-11-09 22:28 ` Ming Lei
2019-11-11 10:50 ` Andrea Vai
2019-11-22 19:16 ` Andrea Vai
1 sibling, 2 replies; 77+ messages in thread
From: Ming Lei @ 2019-11-09 22:28 UTC (permalink / raw)
To: Andrea Vai
Cc: Damien Le Moal, Alan Stern, Jens Axboe, Johannes Thumshirn,
USB list, SCSI development list, Himanshu Madhani,
Hannes Reinecke, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
On Thu, Nov 07, 2019 at 07:59:44PM +0100, Andrea Vai wrote:
> [Sorry for the duplicate message, it didn't reach the lists due to
> html formatting]
> Il giorno gio 7 nov 2019 alle ore 08:54 Damien Le Moal
> <Damien.LeMoal@wdc.com> ha scritto:
> >
> > On 2019/11/07 16:04, Andrea Vai wrote:
> > > Il giorno mer, 06/11/2019 alle 22.13 +0000, Damien Le Moal ha scritto:
> > >>
> > >>
> > >> Please simply try your write tests after doing this:
> > >>
> > >> echo mq-deadline > /sys/block/<name of your USB
> > >> disk>/queue/scheduler
> > >>
> > >> And confirm that mq-deadline is selected with:
> > >>
> > >> cat /sys/block/<name of your USB disk>/queue/scheduler
> > >> [mq-deadline] kyber bfq none
> > >
> > > ok, which kernel should I test with this: the fresh git cloned, or the
> > > one just patched with Alan's patch, or doesn't matter which one?
> >
> > Probably all of them to see if there are any differences.
>
> with both kernels, the output of
> cat /sys/block/sdh/queue/schedule
>
> already contains [mq-deadline]: is it correct to assume that the echo
> command and the subsequent testing is useless? What to do now?
Another thing we could try is to use 'none' via the following command:
echo none > /sys/block/sdh/queue/scheduler #suppose 'sdh' points to the usb storage disk
Because USB storage HBA is single hw queue, which depth is 1. This way
should change to dispatch IO in the order of bio submission.
Andrea, could you switch io scheduler to none and update us if difference
can be made?
Thanks,
Ming
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-08 8:42 ` Damien Le Moal
2019-11-08 14:33 ` Jens Axboe
@ 2019-11-09 10:09 ` Ming Lei
1 sibling, 0 replies; 77+ messages in thread
From: Ming Lei @ 2019-11-09 10:09 UTC (permalink / raw)
To: Damien Le Moal
Cc: Andrea Vai, Alan Stern, Jens Axboe, Johannes Thumshirn, USB list,
SCSI development list, Himanshu Madhani, Hannes Reinecke,
Omar Sandoval, Martin K. Petersen, Greg KH, Hans Holmberg,
Kernel development list
On Fri, Nov 08, 2019 at 08:42:53AM +0000, Damien Le Moal wrote:
> On 2019/11/08 4:00, Andrea Vai wrote:
> > [Sorry for the duplicate message, it didn't reach the lists due to
> > html formatting]
> > Il giorno gio 7 nov 2019 alle ore 08:54 Damien Le Moal
> > <Damien.LeMoal@wdc.com> ha scritto:
> >>
> >> On 2019/11/07 16:04, Andrea Vai wrote:
> >>> Il giorno mer, 06/11/2019 alle 22.13 +0000, Damien Le Moal ha scritto:
> >>>>
> >>>>
> >>>> Please simply try your write tests after doing this:
> >>>>
> >>>> echo mq-deadline > /sys/block/<name of your USB
> >>>> disk>/queue/scheduler
> >>>>
> >>>> And confirm that mq-deadline is selected with:
> >>>>
> >>>> cat /sys/block/<name of your USB disk>/queue/scheduler
> >>>> [mq-deadline] kyber bfq none
> >>>
> >>> ok, which kernel should I test with this: the fresh git cloned, or the
> >>> one just patched with Alan's patch, or doesn't matter which one?
> >>
> >> Probably all of them to see if there are any differences.
> >
> > with both kernels, the output of
> > cat /sys/block/sdh/queue/schedule
> >
> > already contains [mq-deadline]: is it correct to assume that the echo
> > command and the subsequent testing is useless? What to do now?
>
> Probably, yes. Have you obtained a blktrace of the workload during these
> tests ? Any significant difference in the IO pattern (IO size and
> randomness) and IO timing (any device idle time where the device has no
> command to process) ? Asking because the problem may be above the block
> layer, with the file system for instance.
You may get the IO pattern via the previous trace
https://lore.kernel.org/linux-usb/20190710024439.GA2621@ming.t460p/
IMO, if it is related write order, one possibility could be that
the queue lock is killed in .make_request_fn().
Thanks,
Ming
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-08 8:42 ` Damien Le Moal
@ 2019-11-08 14:33 ` Jens Axboe
2019-11-11 10:46 ` Andrea Vai
2019-11-09 10:09 ` Ming Lei
1 sibling, 1 reply; 77+ messages in thread
From: Jens Axboe @ 2019-11-08 14:33 UTC (permalink / raw)
To: Damien Le Moal, Andrea Vai
Cc: Alan Stern, Johannes Thumshirn, USB list, SCSI development list,
Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
Martin K. Petersen, Greg KH, Hans Holmberg,
Kernel development list
On 11/8/19 1:42 AM, Damien Le Moal wrote:
> On 2019/11/08 4:00, Andrea Vai wrote:
>> [Sorry for the duplicate message, it didn't reach the lists due to
>> html formatting]
>> Il giorno gio 7 nov 2019 alle ore 08:54 Damien Le Moal
>> <Damien.LeMoal@wdc.com> ha scritto:
>>>
>>> On 2019/11/07 16:04, Andrea Vai wrote:
>>>> Il giorno mer, 06/11/2019 alle 22.13 +0000, Damien Le Moal ha scritto:
>>>>>
>>>>>
>>>>> Please simply try your write tests after doing this:
>>>>>
>>>>> echo mq-deadline > /sys/block/<name of your USB
>>>>> disk>/queue/scheduler
>>>>>
>>>>> And confirm that mq-deadline is selected with:
>>>>>
>>>>> cat /sys/block/<name of your USB disk>/queue/scheduler
>>>>> [mq-deadline] kyber bfq none
>>>>
>>>> ok, which kernel should I test with this: the fresh git cloned, or the
>>>> one just patched with Alan's patch, or doesn't matter which one?
>>>
>>> Probably all of them to see if there are any differences.
>>
>> with both kernels, the output of
>> cat /sys/block/sdh/queue/schedule
>>
>> already contains [mq-deadline]: is it correct to assume that the echo
>> command and the subsequent testing is useless? What to do now?
>
> Probably, yes. Have you obtained a blktrace of the workload during these
> tests ? Any significant difference in the IO pattern (IO size and
> randomness) and IO timing (any device idle time where the device has no
> command to process) ? Asking because the problem may be above the block
> layer, with the file system for instance.
blktrace would indeed be super useful, especially if you can do that
with a kernel that's fast for you, and one with the current kernel
where it's slow.
Given that your device is sdh, you simply do:
# blktrace /dev/sdh
and then run the test, then ctrl-c the blktrace. Then do:
# blkparse sdh > output
and save that output file. Do both runs, and bzip2 them up. The shorter
the run you can reproduce with the better, to cut down on the size of
the traces.
--
Jens Axboe
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-07 18:59 ` Andrea Vai
@ 2019-11-08 8:42 ` Damien Le Moal
2019-11-08 14:33 ` Jens Axboe
2019-11-09 10:09 ` Ming Lei
2019-11-09 22:28 ` Ming Lei
1 sibling, 2 replies; 77+ messages in thread
From: Damien Le Moal @ 2019-11-08 8:42 UTC (permalink / raw)
To: Andrea Vai
Cc: Alan Stern, Jens Axboe, Johannes Thumshirn, USB list,
SCSI development list, Himanshu Madhani, Hannes Reinecke,
Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
On 2019/11/08 4:00, Andrea Vai wrote:
> [Sorry for the duplicate message, it didn't reach the lists due to
> html formatting]
> Il giorno gio 7 nov 2019 alle ore 08:54 Damien Le Moal
> <Damien.LeMoal@wdc.com> ha scritto:
>>
>> On 2019/11/07 16:04, Andrea Vai wrote:
>>> Il giorno mer, 06/11/2019 alle 22.13 +0000, Damien Le Moal ha scritto:
>>>>
>>>>
>>>> Please simply try your write tests after doing this:
>>>>
>>>> echo mq-deadline > /sys/block/<name of your USB
>>>> disk>/queue/scheduler
>>>>
>>>> And confirm that mq-deadline is selected with:
>>>>
>>>> cat /sys/block/<name of your USB disk>/queue/scheduler
>>>> [mq-deadline] kyber bfq none
>>>
>>> ok, which kernel should I test with this: the fresh git cloned, or the
>>> one just patched with Alan's patch, or doesn't matter which one?
>>
>> Probably all of them to see if there are any differences.
>
> with both kernels, the output of
> cat /sys/block/sdh/queue/schedule
>
> already contains [mq-deadline]: is it correct to assume that the echo
> command and the subsequent testing is useless? What to do now?
Probably, yes. Have you obtained a blktrace of the workload during these
tests ? Any significant difference in the IO pattern (IO size and
randomness) and IO timing (any device idle time where the device has no
command to process) ? Asking because the problem may be above the block
layer, with the file system for instance.
>
> Thanks, and bye
> Andrea
>
--
Damien Le Moal
Western Digital Research
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-07 7:54 ` Damien Le Moal
@ 2019-11-07 18:59 ` Andrea Vai
2019-11-08 8:42 ` Damien Le Moal
2019-11-09 22:28 ` Ming Lei
0 siblings, 2 replies; 77+ messages in thread
From: Andrea Vai @ 2019-11-07 18:59 UTC (permalink / raw)
To: Damien Le Moal
Cc: Alan Stern, Jens Axboe, Johannes Thumshirn, USB list,
SCSI development list, Himanshu Madhani, Hannes Reinecke,
Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
[Sorry for the duplicate message, it didn't reach the lists due to
html formatting]
Il giorno gio 7 nov 2019 alle ore 08:54 Damien Le Moal
<Damien.LeMoal@wdc.com> ha scritto:
>
> On 2019/11/07 16:04, Andrea Vai wrote:
> > Il giorno mer, 06/11/2019 alle 22.13 +0000, Damien Le Moal ha scritto:
> >>
> >>
> >> Please simply try your write tests after doing this:
> >>
> >> echo mq-deadline > /sys/block/<name of your USB
> >> disk>/queue/scheduler
> >>
> >> And confirm that mq-deadline is selected with:
> >>
> >> cat /sys/block/<name of your USB disk>/queue/scheduler
> >> [mq-deadline] kyber bfq none
> >
> > ok, which kernel should I test with this: the fresh git cloned, or the
> > one just patched with Alan's patch, or doesn't matter which one?
>
> Probably all of them to see if there are any differences.
with both kernels, the output of
cat /sys/block/sdh/queue/schedule
already contains [mq-deadline]: is it correct to assume that the echo
command and the subsequent testing is useless? What to do now?
Thanks, and bye
Andrea
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-07 7:04 ` Andrea Vai
@ 2019-11-07 7:54 ` Damien Le Moal
2019-11-07 18:59 ` Andrea Vai
0 siblings, 1 reply; 77+ messages in thread
From: Damien Le Moal @ 2019-11-07 7:54 UTC (permalink / raw)
To: Andrea Vai, Alan Stern, Jens Axboe
Cc: Johannes Thumshirn, USB list, SCSI development list,
Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
Martin K. Petersen, Greg KH, Hans Holmberg,
Kernel development list
On 2019/11/07 16:04, Andrea Vai wrote:
> Il giorno mer, 06/11/2019 alle 22.13 +0000, Damien Le Moal ha scritto:
>>
>>
>> Please simply try your write tests after doing this:
>>
>> echo mq-deadline > /sys/block/<name of your USB
>> disk>/queue/scheduler
>>
>> And confirm that mq-deadline is selected with:
>>
>> cat /sys/block/<name of your USB disk>/queue/scheduler
>> [mq-deadline] kyber bfq none
>
> ok, which kernel should I test with this: the fresh git cloned, or the
> one just patched with Alan's patch, or doesn't matter which one?
Probably all of them to see if there are any differences.
>
> Thanks, and bye,
> Andrea
>
>
--
Damien Le Moal
Western Digital Research
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-06 22:13 ` Damien Le Moal
@ 2019-11-07 7:04 ` Andrea Vai
2019-11-07 7:54 ` Damien Le Moal
0 siblings, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-11-07 7:04 UTC (permalink / raw)
To: Damien Le Moal, Alan Stern, Jens Axboe
Cc: Johannes Thumshirn, USB list, SCSI development list,
Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
Martin K. Petersen, Greg KH, Hans Holmberg,
Kernel development list
Il giorno mer, 06/11/2019 alle 22.13 +0000, Damien Le Moal ha scritto:
>
>
> Please simply try your write tests after doing this:
>
> echo mq-deadline > /sys/block/<name of your USB
> disk>/queue/scheduler
>
> And confirm that mq-deadline is selected with:
>
> cat /sys/block/<name of your USB disk>/queue/scheduler
> [mq-deadline] kyber bfq none
ok, which kernel should I test with this: the fresh git cloned, or the
one just patched with Alan's patch, or doesn't matter which one?
Thanks, and bye,
Andrea
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-06 16:03 ` Alan Stern
@ 2019-11-06 22:13 ` Damien Le Moal
2019-11-07 7:04 ` Andrea Vai
0 siblings, 1 reply; 77+ messages in thread
From: Damien Le Moal @ 2019-11-06 22:13 UTC (permalink / raw)
To: Alan Stern, Jens Axboe
Cc: Andrea Vai, Johannes Thumshirn, USB list, SCSI development list,
Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
Martin K. Petersen, Greg KH, Hans Holmberg,
Kernel development list
On 2019/11/07 1:04, Alan Stern wrote:
> On Tue, 5 Nov 2019, Jens Axboe wrote:
>
>> On 11/5/19 11:31 AM, Alan Stern wrote:
>>> On Tue, 5 Nov 2019, Andrea Vai wrote:
>>>
>>>> Il giorno lun, 04/11/2019 alle 13.20 -0500, Alan Stern ha scritto:
>>>
>>>>> You should be able to do something like this:
>>>>>
>>>>> cd linux
>>>>> patch -p1 </path/to/patch2
>>>>>
>>>>> and that should work with no errors. You don't need to use git to
>>>>> apply a patch.
>>>>>
>>>>> In case that patch2 file was mangled somewhere along the way, I
>>>>> have
>>>>> attached a copy to this message.
>>>>
>>>> Ok, so the "patch" command worked, the kernel compiled and ran, but
>>>> the test still failed (273, 108, 104, 260, 177, 236, 179, 1123, 289,
>>>> 873 seconds to copy a 500MB file, vs. ~30 seconds with the "good"
>>>> kernel).
>>>>
>>>> Let me know what else could I do,
>>>
>>> I'm out of suggestions. If anyone else knows how to make a kernel with
>>> no legacy queuing support -- only multiqueue -- issue I/O requests
>>> sequentially, please speak up.
>>
>> Do we know for a fact that the device needs strictly serialized requests
>> to not stall?
>
> Not exactly, but that is far and away the most likely explanation for
> the device's behavior. We tried making a bunch of changes, some of
> which helped a little bit, but all of them left a very large
> performance gap. I/O monitoring showed that the only noticeable
> difference in the kernel-device interaction caused by the $SUBJECT
> commit was the non-sequential access pattern.
>
>> And writes in particular?
>
> Andrea has tested only the write behavior. Possibly reading will be
> affected too, but my guess is that it won't be.
>
>> I won't comment on how broken
>> that is, just trying to establish this as the problem that's making this
>> particular device be slow?
>
> It seems reasonable that the access pattern could make a significant
> difference. The device's behavior suggests that it buffers incoming
> data and pauses from time to time to write the accumulated data into
> non-volatile storage. If its algorithm for allocating, erasing, and
> writing data blocks is optimized for the sequential case, you can
> easily imagine that non-sequential accesses would cause it to pause
> more often and for longer times -- which is exactly what we observed.
> These extra pauses are what resulted in the overall performance
> decrease.
>
> So far we have had no way to perform a direct test. That is, we don't
> know of any setting that would change a single kernel between
> sequential and non-sequential access. If you can suggest a simple way
> to force a kernel without the $SUBJECT commit to do non-sequential
> writes, I'm sure Andrea will be happy to try it out and see if it
> causes a slowdown.
>
>> I've lost track of this thread, but has mq-deadline been tried as the
>> IO scheduler? We do have support for strictly serialized (writes)
>> since that's required for zoned device, wouldn't be hard at all to make
>> this cover a blacklisted device like this one.
>
> Please spell out the exact procedure in detail so that Andrea can try
> it. He's not a kernel hacker, and I know very little about the block
> layer.
Please simply try your write tests after doing this:
echo mq-deadline > /sys/block/<name of your USB disk>/queue/scheduler
And confirm that mq-deadline is selected with:
cat /sys/block/<name of your USB disk>/queue/scheduler
[mq-deadline] kyber bfq none
>
> Alan Stern
>
>
--
Damien Le Moal
Western Digital Research
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-05 23:29 ` Jens Axboe
@ 2019-11-06 16:03 ` Alan Stern
2019-11-06 22:13 ` Damien Le Moal
0 siblings, 1 reply; 77+ messages in thread
From: Alan Stern @ 2019-11-06 16:03 UTC (permalink / raw)
To: Jens Axboe
Cc: Andrea Vai, Damien Le Moal, Johannes Thumshirn, USB list,
SCSI development list, Himanshu Madhani, Hannes Reinecke,
Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
On Tue, 5 Nov 2019, Jens Axboe wrote:
> On 11/5/19 11:31 AM, Alan Stern wrote:
> > On Tue, 5 Nov 2019, Andrea Vai wrote:
> >
> >> Il giorno lun, 04/11/2019 alle 13.20 -0500, Alan Stern ha scritto:
> >
> >>> You should be able to do something like this:
> >>>
> >>> cd linux
> >>> patch -p1 </path/to/patch2
> >>>
> >>> and that should work with no errors. You don't need to use git to
> >>> apply a patch.
> >>>
> >>> In case that patch2 file was mangled somewhere along the way, I
> >>> have
> >>> attached a copy to this message.
> >>
> >> Ok, so the "patch" command worked, the kernel compiled and ran, but
> >> the test still failed (273, 108, 104, 260, 177, 236, 179, 1123, 289,
> >> 873 seconds to copy a 500MB file, vs. ~30 seconds with the "good"
> >> kernel).
> >>
> >> Let me know what else could I do,
> >
> > I'm out of suggestions. If anyone else knows how to make a kernel with
> > no legacy queuing support -- only multiqueue -- issue I/O requests
> > sequentially, please speak up.
>
> Do we know for a fact that the device needs strictly serialized requests
> to not stall?
Not exactly, but that is far and away the most likely explanation for
the device's behavior. We tried making a bunch of changes, some of
which helped a little bit, but all of them left a very large
performance gap. I/O monitoring showed that the only noticeable
difference in the kernel-device interaction caused by the $SUBJECT
commit was the non-sequential access pattern.
> And writes in particular?
Andrea has tested only the write behavior. Possibly reading will be
affected too, but my guess is that it won't be.
> I won't comment on how broken
> that is, just trying to establish this as the problem that's making this
> particular device be slow?
It seems reasonable that the access pattern could make a significant
difference. The device's behavior suggests that it buffers incoming
data and pauses from time to time to write the accumulated data into
non-volatile storage. If its algorithm for allocating, erasing, and
writing data blocks is optimized for the sequential case, you can
easily imagine that non-sequential accesses would cause it to pause
more often and for longer times -- which is exactly what we observed.
These extra pauses are what resulted in the overall performance
decrease.
So far we have had no way to perform a direct test. That is, we don't
know of any setting that would change a single kernel between
sequential and non-sequential access. If you can suggest a simple way
to force a kernel without the $SUBJECT commit to do non-sequential
writes, I'm sure Andrea will be happy to try it out and see if it
causes a slowdown.
> I've lost track of this thread, but has mq-deadline been tried as the
> IO scheduler? We do have support for strictly serialized (writes)
> since that's required for zoned device, wouldn't be hard at all to make
> this cover a blacklisted device like this one.
Please spell out the exact procedure in detail so that Andrea can try
it. He's not a kernel hacker, and I know very little about the block
layer.
Alan Stern
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-05 18:31 ` Alan Stern
@ 2019-11-05 23:29 ` Jens Axboe
2019-11-06 16:03 ` Alan Stern
0 siblings, 1 reply; 77+ messages in thread
From: Jens Axboe @ 2019-11-05 23:29 UTC (permalink / raw)
To: Alan Stern, Andrea Vai
Cc: Damien Le Moal, Johannes Thumshirn, USB list,
SCSI development list, Himanshu Madhani, Hannes Reinecke,
Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
On 11/5/19 11:31 AM, Alan Stern wrote:
> On Tue, 5 Nov 2019, Andrea Vai wrote:
>
>> Il giorno lun, 04/11/2019 alle 13.20 -0500, Alan Stern ha scritto:
>
>>> You should be able to do something like this:
>>>
>>> cd linux
>>> patch -p1 </path/to/patch2
>>>
>>> and that should work with no errors. You don't need to use git to
>>> apply a patch.
>>>
>>> In case that patch2 file was mangled somewhere along the way, I
>>> have
>>> attached a copy to this message.
>>
>> Ok, so the "patch" command worked, the kernel compiled and ran, but
>> the test still failed (273, 108, 104, 260, 177, 236, 179, 1123, 289,
>> 873 seconds to copy a 500MB file, vs. ~30 seconds with the "good"
>> kernel).
>>
>> Let me know what else could I do,
>
> I'm out of suggestions. If anyone else knows how to make a kernel with
> no legacy queuing support -- only multiqueue -- issue I/O requests
> sequentially, please speak up.
Do we know for a fact that the device needs strictly serialized requests
to not stall? And writes in particular? I won't comment on how broken
that is, just trying to establish this as the problem that's making this
particular device be slow?
I've lost track of this thread, but has mq-deadline been tried as the
IO scheduler? We do have support for strictly serialized (writes)
since that's required for zoned device, wouldn't be hard at all to make
this cover a blacklisted device like this one.
> In the absence of any responses, after a week or so I will submit a
> patch to revert the f664a3cc17b7 ("scsi: kill off the legacy IO path")
> commit.
That's not going to be feasible.
--
Jens Axboe
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-05 11:48 ` Andrea Vai
@ 2019-11-05 18:31 ` Alan Stern
2019-11-05 23:29 ` Jens Axboe
0 siblings, 1 reply; 77+ messages in thread
From: Alan Stern @ 2019-11-05 18:31 UTC (permalink / raw)
To: Andrea Vai
Cc: Jens Axboe, Damien Le Moal, Johannes Thumshirn, USB list,
SCSI development list, Himanshu Madhani, Hannes Reinecke,
Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg, Kernel development list
On Tue, 5 Nov 2019, Andrea Vai wrote:
> Il giorno lun, 04/11/2019 alle 13.20 -0500, Alan Stern ha scritto:
> > You should be able to do something like this:
> >
> > cd linux
> > patch -p1 </path/to/patch2
> >
> > and that should work with no errors. You don't need to use git to
> > apply a patch.
> >
> > In case that patch2 file was mangled somewhere along the way, I
> > have
> > attached a copy to this message.
>
> Ok, so the "patch" command worked, the kernel compiled and ran, but
> the test still failed (273, 108, 104, 260, 177, 236, 179, 1123, 289,
> 873 seconds to copy a 500MB file, vs. ~30 seconds with the "good"
> kernel).
>
> Let me know what else could I do,
I'm out of suggestions. If anyone else knows how to make a kernel with
no legacy queuing support -- only multiqueue -- issue I/O requests
sequentially, please speak up.
In the absence of any responses, after a week or so I will submit a
patch to revert the f664a3cc17b7 ("scsi: kill off the legacy IO path")
commit.
Alan Stern
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-04 18:20 ` Alan Stern
@ 2019-11-05 11:48 ` Andrea Vai
2019-11-05 18:31 ` Alan Stern
0 siblings, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-11-05 11:48 UTC (permalink / raw)
To: Alan Stern
Cc: Jens Axboe, Damien Le Moal, Johannes Thumshirn, USB list,
SCSI development list, Himanshu Madhani, Hannes Reinecke,
Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg
Il giorno lun, 04/11/2019 alle 13.20 -0500, Alan Stern ha scritto:
> On Mon, 4 Nov 2019, Andrea Vai wrote:
>
> > > The "linux" directory is the one generated by a fresh git clone:
> > >
> > > git clone
> > > git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> > >
> > > What am I doing wrong?
> > >
> >
> > Meanwhile, Alan tried to help me and gave me another patch
> (attached),
> > which doesn't work too, but gives a different error: "The git diff
> > header does not contain information about the file once removed 1
> > initial component of the path (row 14)" (actually, this is my
> > translation from the original message in Italian: "error:
> > l'intestazione git diff non riporta le informazioni sul file una
> volta
> > rimosso 1 componente iniziale del percorso (riga 14)")
> >
> > I tested the two patches after a fresh git clone today, a few
> minutes
> > ago.
> >
> > What can I do?
>
> You should be able to do something like this:
>
> cd linux
> patch -p1 </path/to/patch2
>
> and that should work with no errors. You don't need to use git to
> apply a patch.
>
> In case that patch2 file was mangled somewhere along the way, I
> have
> attached a copy to this message.
Ok, so the "patch" command worked, the kernel compiled and ran, but
the test still failed (273, 108, 104, 260, 177, 236, 179, 1123, 289,
873 seconds to copy a 500MB file, vs. ~30 seconds with the "good"
kernel).
Let me know what else could I do,
Thanks, and bye
Andrea
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-11-04 16:00 ` Andrea Vai
@ 2019-11-04 18:20 ` Alan Stern
2019-11-05 11:48 ` Andrea Vai
0 siblings, 1 reply; 77+ messages in thread
From: Alan Stern @ 2019-11-04 18:20 UTC (permalink / raw)
To: Andrea Vai
Cc: Jens Axboe, Damien Le Moal, Johannes Thumshirn, USB list,
SCSI development list, Himanshu Madhani, Hannes Reinecke,
Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg
[-- Attachment #1: Type: TEXT/PLAIN, Size: 1088 bytes --]
On Mon, 4 Nov 2019, Andrea Vai wrote:
> > The "linux" directory is the one generated by a fresh git clone:
> >
> > git clone
> > git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> >
> > What am I doing wrong?
> >
>
> Meanwhile, Alan tried to help me and gave me another patch (attached),
> which doesn't work too, but gives a different error: "The git diff
> header does not contain information about the file once removed 1
> initial component of the path (row 14)" (actually, this is my
> translation from the original message in Italian: "error:
> l'intestazione git diff non riporta le informazioni sul file una volta
> rimosso 1 componente iniziale del percorso (riga 14)")
>
> I tested the two patches after a fresh git clone today, a few minutes
> ago.
>
> What can I do?
You should be able to do something like this:
cd linux
patch -p1 </path/to/patch2
and that should work with no errors. You don't need to use git to
apply a patch.
In case that patch2 file was mangled somewhere along the way, I have
attached a copy to this message.
Alan Stern
[-- Attachment #2: Type: TEXT/PLAIN, Size: 1777 bytes --]
From: Hannes Reinecke <hare@suse.com>
A scheduler might be attached even for devices exposing more than
one hardware queue, so the check for the number of hardware queue
is pointless and should be removed.
Signed-off-by: Hannes Reinecke <hare@suse.com>
---
block/blk-mq.c | 7 +------
1 file changed, 1 insertion(+), 6 deletions(-)
diff --git a/block/blk-mq.c b/block/blk-mq.c
index 44ff3c1442a4..faab542e4836 100644
Index: usb-devel/block/blk-mq.c
===================================================================
--- usb-devel.orig/block/blk-mq.c
+++ usb-devel/block/blk-mq.c
@@ -1946,7 +1946,6 @@ static void blk_add_rq_to_plug(struct bl
static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio)
{
- const int is_sync = op_is_sync(bio->bi_opf);
const int is_flush_fua = op_is_flush(bio->bi_opf);
struct blk_mq_alloc_data data = { .flags = 0};
struct request *rq;
@@ -1992,8 +1991,7 @@ static blk_qc_t blk_mq_make_request(stru
/* bypass scheduler for flush rq */
blk_insert_flush(rq);
blk_mq_run_hw_queue(data.hctx, true);
- } else if (plug && (q->nr_hw_queues == 1 || q->mq_ops->commit_rqs ||
- !blk_queue_nonrot(q))) {
+ } else if (plug && (q->mq_ops->commit_rqs || !blk_queue_nonrot(q))) {
/*
* Use plugging if we have a ->commit_rqs() hook as well, as
* we know the driver uses bd->last in a smart fashion.
@@ -2041,9 +2039,6 @@ static blk_qc_t blk_mq_make_request(stru
blk_mq_try_issue_directly(data.hctx, same_queue_rq,
&cookie);
}
- } else if ((q->nr_hw_queues > 1 && is_sync) ||
- !data.hctx->dispatch_busy) {
- blk_mq_try_issue_directly(data.hctx, rq, &cookie);
} else {
blk_mq_sched_insert_request(rq, false, true, true);
}
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-09-27 15:47 ` Andrea Vai
@ 2019-11-04 16:00 ` Andrea Vai
2019-11-04 18:20 ` Alan Stern
0 siblings, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-11-04 16:00 UTC (permalink / raw)
To: Jens Axboe, Alan Stern
Cc: Damien Le Moal, Johannes Thumshirn, USB list,
SCSI development list, Himanshu Madhani, Hannes Reinecke,
Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg
[-- Attachment #1: Type: text/plain, Size: 2368 bytes --]
Il giorno ven, 27/09/2019 alle 17.47 +0200, Andrea Vai ha scritto:
> Il giorno mer, 25/09/2019 alle 21.36 +0200, Jens Axboe ha scritto:
> > On 9/25/19 9:30 PM, Alan Stern wrote:
> > [...]
> > >
> > > I have attached the two patches to this email. You should start
> > with a
> > > recent kernel source tree and apply the patches by doing:
> > >
> > > git apply patch1 patch2
> > >
> > > or something similar. Then build a kernel from the new source
> > code and
> > > test it.
> > >
> > > Ultimately, if nobody can find a way to restore the sequential
> I/O
> > > behavior we had prior to commit f664a3cc17b7, that commit may
> have
> > to
> > > be reverted.
> >
> > Don't use patch1, it's buggy. patch2 should be enough to test the
> > theory.
As I didn't have any answer, I am quoting my last reply here:
>
> Sorry, but if I cd into the "linux" directory and run the command
>
> # git apply -v patch2
>
> the result is that the patch cannot be applied correctly:
>
> --------------------------------------------------------------------
> ----------
> Controllo della patch block/blk-mq.c in corso...
> error: durante la ricerca per:
> ?
> static blk_qc_t blk_mq_make_request(struct request_queue *q, struct
> bio *bio)?
> {?
> const int is_sync = op_is_sync(bio->bi_opf);?
> const int is_flush_fua = op_is_flush(bio->bi_opf);?
> struct blk_mq_alloc_data data = { .flags = 0};?
> struct request *rq;?
>
> error: patch non riuscita: block/blk-mq.c:1931
> error: block/blk-mq.c: la patch non si applica correttamente
> --------------------------------------------------------------------
> ----------
>
> The "linux" directory is the one generated by a fresh git clone:
>
> git clone
> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>
> What am I doing wrong?
>
Meanwhile, Alan tried to help me and gave me another patch (attached),
which doesn't work too, but gives a different error: "The git diff
header does not contain information about the file once removed 1
initial component of the path (row 14)" (actually, this is my
translation from the original message in Italian: "error:
l'intestazione git diff non riporta le informazioni sul file una volta
rimosso 1 componente iniziale del percorso (riga 14)")
I tested the two patches after a fresh git clone today, a few minutes
ago.
What can I do?
Thank you,
Bye
Andrea
[-- Attachment #2: patch2_alan --]
[-- Type: message/rfc822, Size: 296 bytes --]
From: Hannes Reinecke <hare@suse.com>
Subject: No Subject
Date: Mon, 04 Nov 2019 16:58:21 +0100
Message-ID: <fe072bc69e13435573d824133c3981f8841cf2c7.camel@suse.com>
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-09-25 19:36 ` Jens Axboe
@ 2019-09-27 15:47 ` Andrea Vai
2019-11-04 16:00 ` Andrea Vai
0 siblings, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-09-27 15:47 UTC (permalink / raw)
To: Jens Axboe, Alan Stern
Cc: Damien Le Moal, Johannes Thumshirn, USB list,
SCSI development list, Himanshu Madhani, Hannes Reinecke,
Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg
Il giorno mer, 25/09/2019 alle 21.36 +0200, Jens Axboe ha scritto:
> On 9/25/19 9:30 PM, Alan Stern wrote:
> [...]
> >
> > I have attached the two patches to this email. You should start
> with a
> > recent kernel source tree and apply the patches by doing:
> >
> > git apply patch1 patch2
> >
> > or something similar. Then build a kernel from the new source
> code and
> > test it.
> >
> > Ultimately, if nobody can find a way to restore the sequential I/O
> > behavior we had prior to commit f664a3cc17b7, that commit may have
> to
> > be reverted.
>
> Don't use patch1, it's buggy. patch2 should be enough to test the
> theory.
Sorry, but if I cd into the "linux" directory and run the command
# git apply -v patch2
the result is that the patch cannot be applied correctly:
------------------------------------------------------------------------------
Controllo della patch block/blk-mq.c in corso...
error: durante la ricerca per:
?
static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio)?
{?
const int is_sync = op_is_sync(bio->bi_opf);?
const int is_flush_fua = op_is_flush(bio->bi_opf);?
struct blk_mq_alloc_data data = { .flags = 0};?
struct request *rq;?
error: patch non riuscita: block/blk-mq.c:1931
error: block/blk-mq.c: la patch non si applica correttamente
------------------------------------------------------------------------------
The "linux" directory is the one generated by a fresh git clone:
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
What am I doing wrong?
Thanks, and bye
Andrea
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-09-25 19:30 ` Alan Stern
@ 2019-09-25 19:36 ` Jens Axboe
2019-09-27 15:47 ` Andrea Vai
0 siblings, 1 reply; 77+ messages in thread
From: Jens Axboe @ 2019-09-25 19:36 UTC (permalink / raw)
To: Alan Stern, Andrea Vai
Cc: Damien Le Moal, Johannes Thumshirn, USB list,
SCSI development list, Himanshu Madhani, Hannes Reinecke,
Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg
On 9/25/19 9:30 PM, Alan Stern wrote:
> On Fri, 20 Sep 2019, Andrea Vai wrote:
>
>> Il giorno gio, 19/09/2019 alle 14.14 +0000, Damien Le Moal ha scritto:
>>> On 2019/09/19 16:01, Alan Stern wrote:
>>> [...]
>>>> No doubt Andrea will be happy to test your fix when it's ready.
>>
>> Yes, of course.
>>
>>>
>>> Hannes posted an RFC series:
>>>
>>> https://www.spinics.net/lists/linux-scsi/msg133848.html
>>>
>>> Andrea can try it.
>>
>> Ok, but I would need some instructions please, because I am not able
>> to understand how to "try it". Sorry for that.
>
> I have attached the two patches to this email. You should start with a
> recent kernel source tree and apply the patches by doing:
>
> git apply patch1 patch2
>
> or something similar. Then build a kernel from the new source code and
> test it.
>
> Ultimately, if nobody can find a way to restore the sequential I/O
> behavior we had prior to commit f664a3cc17b7, that commit may have to
> be reverted.
Don't use patch1, it's buggy. patch2 should be enough to test the theory.
--
Jens Axboe
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-09-20 7:03 ` Andrea Vai
@ 2019-09-25 19:30 ` Alan Stern
2019-09-25 19:36 ` Jens Axboe
0 siblings, 1 reply; 77+ messages in thread
From: Alan Stern @ 2019-09-25 19:30 UTC (permalink / raw)
To: Andrea Vai
Cc: Damien Le Moal, Johannes Thumshirn, Jens Axboe, USB list,
SCSI development list, Himanshu Madhani, Hannes Reinecke,
Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg
[-- Attachment #1: Type: TEXT/PLAIN, Size: 920 bytes --]
On Fri, 20 Sep 2019, Andrea Vai wrote:
> Il giorno gio, 19/09/2019 alle 14.14 +0000, Damien Le Moal ha scritto:
> > On 2019/09/19 16:01, Alan Stern wrote:
> > [...]
> > > No doubt Andrea will be happy to test your fix when it's ready.
>
> Yes, of course.
>
> >
> > Hannes posted an RFC series:
> >
> > https://www.spinics.net/lists/linux-scsi/msg133848.html
> >
> > Andrea can try it.
>
> Ok, but I would need some instructions please, because I am not able
> to understand how to "try it". Sorry for that.
I have attached the two patches to this email. You should start with a
recent kernel source tree and apply the patches by doing:
git apply patch1 patch2
or something similar. Then build a kernel from the new source code and
test it.
Ultimately, if nobody can find a way to restore the sequential I/O
behavior we had prior to commit f664a3cc17b7, that commit may have to
be reverted.
Alan Stern
[-- Attachment #2: Type: TEXT/PLAIN, Size: 980 bytes --]
From: Hannes Reinecke <hare@suse.com>
When blk_mq_request_issue_directly() returns BLK_STS_RESOURCE we
need to requeue the I/O, but adding it to the global request list
will mess up with the passed-in request list. So re-add the request
to the original list and leave it to the caller to handle situations
where the list wasn't completely emptied.
Signed-off-by: Hannes Reinecke <hare@suse.com>
---
block/blk-mq.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/block/blk-mq.c b/block/blk-mq.c
index b038ec680e84..44ff3c1442a4 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -1899,8 +1899,7 @@ void blk_mq_try_issue_list_directly(struct blk_mq_hw_ctx *hctx,
if (ret != BLK_STS_OK) {
if (ret == BLK_STS_RESOURCE ||
ret == BLK_STS_DEV_RESOURCE) {
- blk_mq_request_bypass_insert(rq,
- list_empty(list));
+ list_add(list, &rq->queuelist);
break;
}
blk_mq_end_request(rq, ret);
--
2.16.4
[-- Attachment #3: Type: TEXT/PLAIN, Size: 1721 bytes --]
From: Hannes Reinecke <hare@suse.com>
A scheduler might be attached even for devices exposing more than
one hardware queue, so the check for the number of hardware queue
is pointless and should be removed.
Signed-off-by: Hannes Reinecke <hare@suse.com>
---
block/blk-mq.c | 6 +-----
1 file changed, 1 insertion(+), 5 deletions(-)
diff --git a/block/blk-mq.c b/block/blk-mq.c
index 44ff3c1442a4..faab542e4836 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -1931,7 +1931,6 @@ static void blk_add_rq_to_plug(struct blk_plug *plug, struct request *rq)
static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio)
{
- const int is_sync = op_is_sync(bio->bi_opf);
const int is_flush_fua = op_is_flush(bio->bi_opf);
struct blk_mq_alloc_data data = { .flags = 0};
struct request *rq;
@@ -1977,7 +1976,7 @@ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio)
/* bypass scheduler for flush rq */
blk_insert_flush(rq);
blk_mq_run_hw_queue(data.hctx, true);
- } else if (plug && (q->nr_hw_queues == 1 || q->mq_ops->commit_rqs)) {
+ } else if (plug && q->mq_ops->commit_rqs) {
/*
* Use plugging if we have a ->commit_rqs() hook as well, as
* we know the driver uses bd->last in a smart fashion.
@@ -2020,9 +2019,6 @@ static blk_qc_t blk_mq_make_request(struct request_queue *q, struct bio *bio)
blk_mq_try_issue_directly(data.hctx, same_queue_rq,
&cookie);
}
- } else if ((q->nr_hw_queues > 1 && is_sync) || (!q->elevator &&
- !data.hctx->dispatch_busy)) {
- blk_mq_try_issue_directly(data.hctx, rq, &cookie);
} else {
blk_mq_sched_insert_request(rq, false, true, true);
}
--
2.16.4
^ permalink raw reply related [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-09-20 7:25 ` Andrea Vai
@ 2019-09-20 7:44 ` Greg KH
0 siblings, 0 replies; 77+ messages in thread
From: Greg KH @ 2019-09-20 7:44 UTC (permalink / raw)
To: Andrea Vai
Cc: Alan Stern, Johannes Thumshirn, Jens Axboe, USB list,
SCSI development list, Himanshu Madhani, Hannes Reinecke,
Ming Lei, Omar Sandoval, Martin K. Petersen
On Fri, Sep 20, 2019 at 09:25:17AM +0200, Andrea Vai wrote:
> Il giorno gio, 19/09/2019 alle 13.54 -0400, Alan Stern ha scritto:
> >
> > In general, USB flash drives should not be expected to work as well
> > as
> > an actual disk drive connected over USB.
>
> Ok, so I think I'll buy some different hardware. Would an SSD drive
> (connected over USB) behave like a flash drive or like an "actual disk
> drive" from this point of view?
It all depends on the drive. Some are a lot better than others, and
it's almost impossible to tell until you buy the thing and try it out :(
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-09-19 17:54 ` Alan Stern
@ 2019-09-20 7:25 ` Andrea Vai
2019-09-20 7:44 ` Greg KH
0 siblings, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-09-20 7:25 UTC (permalink / raw)
To: Alan Stern
Cc: Johannes Thumshirn, Jens Axboe, USB list, SCSI development list,
Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
Martin K. Petersen, Greg KH
Il giorno gio, 19/09/2019 alle 13.54 -0400, Alan Stern ha scritto:
>
> In general, USB flash drives should not be expected to work as well
> as
> an actual disk drive connected over USB.
Ok, so I think I'll buy some different hardware. Would an SSD drive
(connected over USB) behave like a flash drive or like an "actual disk
drive" from this point of view?
Many thanks, and bye
Andrea
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-09-19 14:14 ` Damien Le Moal
@ 2019-09-20 7:03 ` Andrea Vai
2019-09-25 19:30 ` Alan Stern
0 siblings, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-09-20 7:03 UTC (permalink / raw)
To: Damien Le Moal, Alan Stern
Cc: Johannes Thumshirn, Jens Axboe, USB list, SCSI development list,
Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
Martin K. Petersen, Greg KH, Hans Holmberg
Il giorno gio, 19/09/2019 alle 14.14 +0000, Damien Le Moal ha scritto:
> On 2019/09/19 16:01, Alan Stern wrote:
> [...]
> > No doubt Andrea will be happy to test your fix when it's ready.
Yes, of course.
>
> Hannes posted an RFC series:
>
> https://www.spinics.net/lists/linux-scsi/msg133848.html
>
> Andrea can try it.
Ok, but I would need some instructions please, because I am not able
to understand how to "try it". Sorry for that.
> Andrea,
>
> What is the device in question ? Is it a USB external HDD ? SSD ?
> Flash key ?
It is a USB flash key (a.k.a. pendrive, flash drive, etc.):
ID 0951:1666 Kingston Technology DataTraveler 100 G3/G4/SE9 G2
Thanks, and bye
Andrea
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-09-19 7:33 ` Andrea Vai
@ 2019-09-19 17:54 ` Alan Stern
2019-09-20 7:25 ` Andrea Vai
0 siblings, 1 reply; 77+ messages in thread
From: Alan Stern @ 2019-09-19 17:54 UTC (permalink / raw)
To: Andrea Vai
Cc: Johannes Thumshirn, Jens Axboe, USB list, SCSI development list,
Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
Martin K. Petersen, Greg KH
On Thu, 19 Sep 2019, Andrea Vai wrote:
> BTW, another question: Alan refers to the slow media as a "consumer-
> grade USB storage device". What could I do to identify and buy a "good
> media"? Are there any features to look for?
In general, USB flash drives should not be expected to work as well as
an actual disk drive connected over USB.
Alan Stern
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-09-19 14:01 ` Alan Stern
@ 2019-09-19 14:14 ` Damien Le Moal
2019-09-20 7:03 ` Andrea Vai
0 siblings, 1 reply; 77+ messages in thread
From: Damien Le Moal @ 2019-09-19 14:14 UTC (permalink / raw)
To: Alan Stern
Cc: Andrea Vai, Johannes Thumshirn, Jens Axboe, USB list,
SCSI development list, Himanshu Madhani, Hannes Reinecke,
Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg
On 2019/09/19 16:01, Alan Stern wrote:
> On Thu, 19 Sep 2019, Damien Le Moal wrote:
>
>>> This is exactly the sort of difference one might expect to see from
>>> the commit f664a3cc17b7 ("scsi: kill off the legacy IO path") you
>>> identified as the cause of the problem. With multiqueue I/O, it's not
>>> surprising to see multiple sequences of block numbers.
>>>
>>> Add it's not at all surprising that a consumer-grade USB storage device
>>> might do a much worse job of handling non-sequential writes than
>>> sequential ones.
>>>
>>> Which leads to a simple question for the SCSI or block-layer
>>> maintainers: Is there a sysfs setting Andrea can tweak which will
>>> effectively restrict a particular disk device down to a single I/O
>>> queue, forcing sequential access?
>>
>> The scheduling inefficiency you are seeing may be coming from the fact that the
>> block layer does a direct issue of requests, bypassing the elevator, under some
>> conditions. One of these is sync requests on a multiqueue device. We hit this
>> problem on Zoned disks which can easily return an error for write requests
>> without the elevator throttling writes per zones (zone write locking). This
>> problem was discovered by Hans (on CC).
>
> Is there any way for Andrea to check whether this is the underlying
> cause?>
>> I discussed this with Hannes yesterday and we think we have a fix, but we'll
>> need to do a lot of testing as all block devices are potentially impacted by the
>> change, including stacked drivers (DM). Performance regression is scary with any
>> change in that area (see blk_mq_make_request() and use of
>> blk_mq_try_issue_directly() vs blk_mq_sched_insert_request()).
>
> No doubt Andrea will be happy to test your fix when it's ready.
Hannes posted an RFC series:
https://www.spinics.net/lists/linux-scsi/msg133848.html
Andrea can try it. But If the USB device is not multi-queue, this fix will
probably have no effect.
Andrea,
What is the device in question ? Is it a USB external HDD ? SSD ? Flash key ?
Best regards.
--
Damien Le Moal
Western Digital Research
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-09-19 8:26 ` Damien Le Moal
2019-09-19 8:55 ` Ming Lei
@ 2019-09-19 14:01 ` Alan Stern
2019-09-19 14:14 ` Damien Le Moal
1 sibling, 1 reply; 77+ messages in thread
From: Alan Stern @ 2019-09-19 14:01 UTC (permalink / raw)
To: Damien Le Moal
Cc: Andrea Vai, Johannes Thumshirn, Jens Axboe, USB list,
SCSI development list, Himanshu Madhani, Hannes Reinecke,
Ming Lei, Omar Sandoval, Martin K. Petersen, Greg KH,
Hans Holmberg
On Thu, 19 Sep 2019, Damien Le Moal wrote:
> > This is exactly the sort of difference one might expect to see from
> > the commit f664a3cc17b7 ("scsi: kill off the legacy IO path") you
> > identified as the cause of the problem. With multiqueue I/O, it's not
> > surprising to see multiple sequences of block numbers.
> >
> > Add it's not at all surprising that a consumer-grade USB storage device
> > might do a much worse job of handling non-sequential writes than
> > sequential ones.
> >
> > Which leads to a simple question for the SCSI or block-layer
> > maintainers: Is there a sysfs setting Andrea can tweak which will
> > effectively restrict a particular disk device down to a single I/O
> > queue, forcing sequential access?
>
> The scheduling inefficiency you are seeing may be coming from the fact that the
> block layer does a direct issue of requests, bypassing the elevator, under some
> conditions. One of these is sync requests on a multiqueue device. We hit this
> problem on Zoned disks which can easily return an error for write requests
> without the elevator throttling writes per zones (zone write locking). This
> problem was discovered by Hans (on CC).
Is there any way for Andrea to check whether this is the underlying
cause?
> I discussed this with Hannes yesterday and we think we have a fix, but we'll
> need to do a lot of testing as all block devices are potentially impacted by the
> change, including stacked drivers (DM). Performance regression is scary with any
> change in that area (see blk_mq_make_request() and use of
> blk_mq_try_issue_directly() vs blk_mq_sched_insert_request()).
No doubt Andrea will be happy to test your fix when it's ready.
Alan Stern
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-09-19 9:09 ` Damien Le Moal
@ 2019-09-19 9:21 ` Ming Lei
0 siblings, 0 replies; 77+ messages in thread
From: Ming Lei @ 2019-09-19 9:21 UTC (permalink / raw)
To: Damien Le Moal
Cc: Alan Stern, Andrea Vai, Johannes Thumshirn, Jens Axboe, USB list,
SCSI development list, Himanshu Madhani, Hannes Reinecke,
Omar Sandoval, Martin K. Petersen, Greg KH, Hans Holmberg
On Thu, Sep 19, 2019 at 09:09:33AM +0000, Damien Le Moal wrote:
> On 2019/09/19 10:56, Ming Lei wrote:
> > On Thu, Sep 19, 2019 at 08:26:32AM +0000, Damien Le Moal wrote:
> >> On 2019/09/18 18:30, Alan Stern wrote:
> >>> On Wed, 18 Sep 2019, Andrea Vai wrote:
> >>>
> >>>>> Also, I wonder if the changing the size of the data transfers would
> >>>>> make any difference. This is easy to try; just write "64" to
> >>>>> /sys/block/sd?/queue/max_sectors_kb (where the ? is the appropriate
> >>>>> drive letter) after the drive is plugged in but before the test
> >>>>> starts.
> >>>>
> >>>> ok, so I duplicated the tests above for the "64" case (it was
> >>>> initially set as "120", if it is relevant to know), leading to 40 tests named as
> >>>>
> >>>> bad.mon.out_50000000_64_TIMESTAMP
> >>>> bad.mon.out_50000000_non64_TIMESTAMP
> >>>> good.mon.out_50000000_64_TIMESTAMP
> >>>> good.mon.out_50000000_non64_TIMESTAMP
> >>>>
> >>>> where "64" denotes the ones done with that value in max_sectors_kb,
> >>>> and "not64" the ones without it (as far as I can tell, it has been
> >>>> always "120").
> >>>>
> >>>> So, we have 40 traces total. Each set of 10 trials is identified by
> >>>> a text file, which contains the output log of the test script (and the
> >>>> timestamps), also available in the download zipfile.
> >>>>
> >>>> Just to summarize here the times, they are respectively (number
> >>>> expressed in seconds):
> >>>>
> >>>> BAD:
> >>>> Logs: log_10trials_50MB_BAD_NonCanc_64.txt,
> >>>> log_10trials_50MB_BAD_NonCanc_non64.txt
> >>>> 64: 34, 34, 35, 39, 37, 32, 42, 44, 43, 40
> >>>> not64: 61, 71, 59, 71, 62, 75, 62, 70, 62, 68
> >>>> GOOD:
> >>>> Logs: log_10trials_50MB_GOOD_NonCanc_64.txt,
> >>>> log_10trials_50MB_GOOD_NonCanc_non64.txt
> >>>> 64: 34, 32, 35, 34, 35, 33, 34, 33, 33, 33
> >>>> not64: 32, 30, 32, 31, 31, 30, 32, 30, 32, 31
> >>>
> >>> The improvement from using "64" with the bad kernel is quite large.
> >>> That alone would be a big help for you.
> >>>
> >>> However, I did see what appears to be a very significant difference
> >>> between the bad and good kernel traces. It has to do with the order in
> >>> which the blocks are accessed.
> >>>
> >>> Here is an extract from one of the bad traces. I have erased all the
> >>> information except for the columns containing the block numbers to be
> >>> written:
> >>>
> >>> 00019628 00
> >>> 00019667 00
> >>> 00019628 80
> >>> 00019667 80
> >>> 00019629 00
> >>> 00019668 00
> >>> 00019629 80
> >>> 00019668 80
> >>>
> >>> Here is the equivalent portion from one of the good traces:
> >>>
> >>> 00019628 00
> >>> 00019628 80
> >>> 00019629 00
> >>> 00019629 80
> >>> 0001962a 00
> >>> 0001962a 80
> >>> 0001962b 00
> >>> 0001962b 80
> >>>
> >>> Notice that under the good kernel, the block numbers increase
> >>> monotonically in a single sequence. But under the bad kernel, the
> >>> block numbers are not monotonic -- it looks like there are two separate
> >>> threads each with its own strictly increasing sequence.
> >>>
> >>> This is exactly the sort of difference one might expect to see from
> >>> the commit f664a3cc17b7 ("scsi: kill off the legacy IO path") you
> >>> identified as the cause of the problem. With multiqueue I/O, it's not
> >>> surprising to see multiple sequences of block numbers.
> >>>
> >>> Add it's not at all surprising that a consumer-grade USB storage device
> >>> might do a much worse job of handling non-sequential writes than
> >>> sequential ones.
> >>>
> >>> Which leads to a simple question for the SCSI or block-layer
> >>> maintainers: Is there a sysfs setting Andrea can tweak which will
> >>> effectively restrict a particular disk device down to a single I/O
> >>> queue, forcing sequential access?
> >>
> >> The scheduling inefficiency you are seeing may be coming from the fact that the
> >> block layer does a direct issue of requests, bypassing the elevator, under some
> >> conditions. One of these is sync requests on a multiqueue device. We hit this
> >> problem on Zoned disks which can easily return an error for write requests
> >> without the elevator throttling writes per zones (zone write locking). This
> >> problem was discovered by Hans (on CC).
> >>
> >> I discussed this with Hannes yesterday and we think we have a fix, but we'll
> >> need to do a lot of testing as all block devices are potentially impacted by the
> >> change, including stacked drivers (DM). Performance regression is scary with any
> >> change in that area (see blk_mq_make_request() and use of
> >> blk_mq_try_issue_directly() vs blk_mq_sched_insert_request()).
> >
> > Not sure this one is same with yours, for USB, mq-deadline is used at
> > default, and direct issue won't be possible. Direct issue is only used
> > in case of none or underlying queues of DM multipath.
>
> For a multi-queue zoned disk, mq-deadline is also set, but we have observed
> unaligned write IO errors for sync writes because of mq-deadline being bypassed
> and as a result zones not being write-locked.
>
> In blk_mq_make_request(), at the end, you have:
>
> } else if ((q->nr_hw_queues > 1 && is_sync) || (!q->elevator &&
> !data.hctx->dispatch_busy)) {
> blk_mq_try_issue_directly(data.hctx, rq, &cookie);
> } else {
> blk_mq_sched_insert_request(rq, false, true, true);
> }
>
> Which I read as "for a sync req on a multi-queue device, direct issue",
> regardless of the elevator being none or something else.
Yeah, looks elevator is bypassed in the above case, which seems a bug.
USB storage has only single queue.
>
> The correct test should probably be:
>
> } else if (!q->elevator &&
> ((q->nr_hw_queues > 1 && is_sync) ||
> !data.hctx->dispatch_busy))) {
> blk_mq_try_issue_directly(data.hctx, rq, &cookie);
> } else {
> blk_mq_sched_insert_request(rq, false, true, true);
> }
>
> That is, never bypass the elevator if one is set. Thoughts ?
IMO, elevator shouldn't be bypassed any time, looks it is bypassed
in the following branch too, but may not be reached for zone device.
blk_mq_make_request()
...
} else if (plug && !blk_queue_nomerges(q)) {
...
if (same_queue_rq) {
data.hctx = same_queue_rq->mq_hctx;
trace_block_unplug(q, 1, true);
blk_mq_try_issue_directly(data.hctx, same_queue_rq,
&cookie);
}
}
Thanks,
Ming
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-09-19 8:55 ` Ming Lei
@ 2019-09-19 9:09 ` Damien Le Moal
2019-09-19 9:21 ` Ming Lei
0 siblings, 1 reply; 77+ messages in thread
From: Damien Le Moal @ 2019-09-19 9:09 UTC (permalink / raw)
To: Ming Lei
Cc: Alan Stern, Andrea Vai, Johannes Thumshirn, Jens Axboe, USB list,
SCSI development list, Himanshu Madhani, Hannes Reinecke,
Omar Sandoval, Martin K. Petersen, Greg KH, Hans Holmberg
On 2019/09/19 10:56, Ming Lei wrote:
> On Thu, Sep 19, 2019 at 08:26:32AM +0000, Damien Le Moal wrote:
>> On 2019/09/18 18:30, Alan Stern wrote:
>>> On Wed, 18 Sep 2019, Andrea Vai wrote:
>>>
>>>>> Also, I wonder if the changing the size of the data transfers would
>>>>> make any difference. This is easy to try; just write "64" to
>>>>> /sys/block/sd?/queue/max_sectors_kb (where the ? is the appropriate
>>>>> drive letter) after the drive is plugged in but before the test
>>>>> starts.
>>>>
>>>> ok, so I duplicated the tests above for the "64" case (it was
>>>> initially set as "120", if it is relevant to know), leading to 40 tests named as
>>>>
>>>> bad.mon.out_50000000_64_TIMESTAMP
>>>> bad.mon.out_50000000_non64_TIMESTAMP
>>>> good.mon.out_50000000_64_TIMESTAMP
>>>> good.mon.out_50000000_non64_TIMESTAMP
>>>>
>>>> where "64" denotes the ones done with that value in max_sectors_kb,
>>>> and "not64" the ones without it (as far as I can tell, it has been
>>>> always "120").
>>>>
>>>> So, we have 40 traces total. Each set of 10 trials is identified by
>>>> a text file, which contains the output log of the test script (and the
>>>> timestamps), also available in the download zipfile.
>>>>
>>>> Just to summarize here the times, they are respectively (number
>>>> expressed in seconds):
>>>>
>>>> BAD:
>>>> Logs: log_10trials_50MB_BAD_NonCanc_64.txt,
>>>> log_10trials_50MB_BAD_NonCanc_non64.txt
>>>> 64: 34, 34, 35, 39, 37, 32, 42, 44, 43, 40
>>>> not64: 61, 71, 59, 71, 62, 75, 62, 70, 62, 68
>>>> GOOD:
>>>> Logs: log_10trials_50MB_GOOD_NonCanc_64.txt,
>>>> log_10trials_50MB_GOOD_NonCanc_non64.txt
>>>> 64: 34, 32, 35, 34, 35, 33, 34, 33, 33, 33
>>>> not64: 32, 30, 32, 31, 31, 30, 32, 30, 32, 31
>>>
>>> The improvement from using "64" with the bad kernel is quite large.
>>> That alone would be a big help for you.
>>>
>>> However, I did see what appears to be a very significant difference
>>> between the bad and good kernel traces. It has to do with the order in
>>> which the blocks are accessed.
>>>
>>> Here is an extract from one of the bad traces. I have erased all the
>>> information except for the columns containing the block numbers to be
>>> written:
>>>
>>> 00019628 00
>>> 00019667 00
>>> 00019628 80
>>> 00019667 80
>>> 00019629 00
>>> 00019668 00
>>> 00019629 80
>>> 00019668 80
>>>
>>> Here is the equivalent portion from one of the good traces:
>>>
>>> 00019628 00
>>> 00019628 80
>>> 00019629 00
>>> 00019629 80
>>> 0001962a 00
>>> 0001962a 80
>>> 0001962b 00
>>> 0001962b 80
>>>
>>> Notice that under the good kernel, the block numbers increase
>>> monotonically in a single sequence. But under the bad kernel, the
>>> block numbers are not monotonic -- it looks like there are two separate
>>> threads each with its own strictly increasing sequence.
>>>
>>> This is exactly the sort of difference one might expect to see from
>>> the commit f664a3cc17b7 ("scsi: kill off the legacy IO path") you
>>> identified as the cause of the problem. With multiqueue I/O, it's not
>>> surprising to see multiple sequences of block numbers.
>>>
>>> Add it's not at all surprising that a consumer-grade USB storage device
>>> might do a much worse job of handling non-sequential writes than
>>> sequential ones.
>>>
>>> Which leads to a simple question for the SCSI or block-layer
>>> maintainers: Is there a sysfs setting Andrea can tweak which will
>>> effectively restrict a particular disk device down to a single I/O
>>> queue, forcing sequential access?
>>
>> The scheduling inefficiency you are seeing may be coming from the fact that the
>> block layer does a direct issue of requests, bypassing the elevator, under some
>> conditions. One of these is sync requests on a multiqueue device. We hit this
>> problem on Zoned disks which can easily return an error for write requests
>> without the elevator throttling writes per zones (zone write locking). This
>> problem was discovered by Hans (on CC).
>>
>> I discussed this with Hannes yesterday and we think we have a fix, but we'll
>> need to do a lot of testing as all block devices are potentially impacted by the
>> change, including stacked drivers (DM). Performance regression is scary with any
>> change in that area (see blk_mq_make_request() and use of
>> blk_mq_try_issue_directly() vs blk_mq_sched_insert_request()).
>
> Not sure this one is same with yours, for USB, mq-deadline is used at
> default, and direct issue won't be possible. Direct issue is only used
> in case of none or underlying queues of DM multipath.
For a multi-queue zoned disk, mq-deadline is also set, but we have observed
unaligned write IO errors for sync writes because of mq-deadline being bypassed
and as a result zones not being write-locked.
In blk_mq_make_request(), at the end, you have:
} else if ((q->nr_hw_queues > 1 && is_sync) || (!q->elevator &&
!data.hctx->dispatch_busy)) {
blk_mq_try_issue_directly(data.hctx, rq, &cookie);
} else {
blk_mq_sched_insert_request(rq, false, true, true);
}
Which I read as "for a sync req on a multi-queue device, direct issue",
regardless of the elevator being none or something else.
The correct test should probably be:
} else if (!q->elevator &&
((q->nr_hw_queues > 1 && is_sync) ||
!data.hctx->dispatch_busy))) {
blk_mq_try_issue_directly(data.hctx, rq, &cookie);
} else {
blk_mq_sched_insert_request(rq, false, true, true);
}
That is, never bypass the elevator if one is set. Thoughts ?
--
Damien Le Moal
Western Digital Research
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-09-19 8:26 ` Damien Le Moal
@ 2019-09-19 8:55 ` Ming Lei
2019-09-19 9:09 ` Damien Le Moal
2019-09-19 14:01 ` Alan Stern
1 sibling, 1 reply; 77+ messages in thread
From: Ming Lei @ 2019-09-19 8:55 UTC (permalink / raw)
To: Damien Le Moal
Cc: Alan Stern, Andrea Vai, Johannes Thumshirn, Jens Axboe, USB list,
SCSI development list, Himanshu Madhani, Hannes Reinecke,
Omar Sandoval, Martin K. Petersen, Greg KH, Hans Holmberg
On Thu, Sep 19, 2019 at 08:26:32AM +0000, Damien Le Moal wrote:
> On 2019/09/18 18:30, Alan Stern wrote:
> > On Wed, 18 Sep 2019, Andrea Vai wrote:
> >
> >>> Also, I wonder if the changing the size of the data transfers would
> >>> make any difference. This is easy to try; just write "64" to
> >>> /sys/block/sd?/queue/max_sectors_kb (where the ? is the appropriate
> >>> drive letter) after the drive is plugged in but before the test
> >>> starts.
> >>
> >> ok, so I duplicated the tests above for the "64" case (it was
> >> initially set as "120", if it is relevant to know), leading to 40 tests named as
> >>
> >> bad.mon.out_50000000_64_TIMESTAMP
> >> bad.mon.out_50000000_non64_TIMESTAMP
> >> good.mon.out_50000000_64_TIMESTAMP
> >> good.mon.out_50000000_non64_TIMESTAMP
> >>
> >> where "64" denotes the ones done with that value in max_sectors_kb,
> >> and "not64" the ones without it (as far as I can tell, it has been
> >> always "120").
> >>
> >> So, we have 40 traces total. Each set of 10 trials is identified by
> >> a text file, which contains the output log of the test script (and the
> >> timestamps), also available in the download zipfile.
> >>
> >> Just to summarize here the times, they are respectively (number
> >> expressed in seconds):
> >>
> >> BAD:
> >> Logs: log_10trials_50MB_BAD_NonCanc_64.txt,
> >> log_10trials_50MB_BAD_NonCanc_non64.txt
> >> 64: 34, 34, 35, 39, 37, 32, 42, 44, 43, 40
> >> not64: 61, 71, 59, 71, 62, 75, 62, 70, 62, 68
> >> GOOD:
> >> Logs: log_10trials_50MB_GOOD_NonCanc_64.txt,
> >> log_10trials_50MB_GOOD_NonCanc_non64.txt
> >> 64: 34, 32, 35, 34, 35, 33, 34, 33, 33, 33
> >> not64: 32, 30, 32, 31, 31, 30, 32, 30, 32, 31
> >
> > The improvement from using "64" with the bad kernel is quite large.
> > That alone would be a big help for you.
> >
> > However, I did see what appears to be a very significant difference
> > between the bad and good kernel traces. It has to do with the order in
> > which the blocks are accessed.
> >
> > Here is an extract from one of the bad traces. I have erased all the
> > information except for the columns containing the block numbers to be
> > written:
> >
> > 00019628 00
> > 00019667 00
> > 00019628 80
> > 00019667 80
> > 00019629 00
> > 00019668 00
> > 00019629 80
> > 00019668 80
> >
> > Here is the equivalent portion from one of the good traces:
> >
> > 00019628 00
> > 00019628 80
> > 00019629 00
> > 00019629 80
> > 0001962a 00
> > 0001962a 80
> > 0001962b 00
> > 0001962b 80
> >
> > Notice that under the good kernel, the block numbers increase
> > monotonically in a single sequence. But under the bad kernel, the
> > block numbers are not monotonic -- it looks like there are two separate
> > threads each with its own strictly increasing sequence.
> >
> > This is exactly the sort of difference one might expect to see from
> > the commit f664a3cc17b7 ("scsi: kill off the legacy IO path") you
> > identified as the cause of the problem. With multiqueue I/O, it's not
> > surprising to see multiple sequences of block numbers.
> >
> > Add it's not at all surprising that a consumer-grade USB storage device
> > might do a much worse job of handling non-sequential writes than
> > sequential ones.
> >
> > Which leads to a simple question for the SCSI or block-layer
> > maintainers: Is there a sysfs setting Andrea can tweak which will
> > effectively restrict a particular disk device down to a single I/O
> > queue, forcing sequential access?
>
> The scheduling inefficiency you are seeing may be coming from the fact that the
> block layer does a direct issue of requests, bypassing the elevator, under some
> conditions. One of these is sync requests on a multiqueue device. We hit this
> problem on Zoned disks which can easily return an error for write requests
> without the elevator throttling writes per zones (zone write locking). This
> problem was discovered by Hans (on CC).
>
> I discussed this with Hannes yesterday and we think we have a fix, but we'll
> need to do a lot of testing as all block devices are potentially impacted by the
> change, including stacked drivers (DM). Performance regression is scary with any
> change in that area (see blk_mq_make_request() and use of
> blk_mq_try_issue_directly() vs blk_mq_sched_insert_request()).
Not sure this one is same with yours, for USB, mq-deadline is used at
default, and direct issue won't be possible. Direct issue is only used
in case of none or underlying queues of DM multipath.
thanks,
Ming
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-09-18 16:30 ` Alan Stern
2019-09-19 7:33 ` Andrea Vai
@ 2019-09-19 8:26 ` Damien Le Moal
2019-09-19 8:55 ` Ming Lei
2019-09-19 14:01 ` Alan Stern
1 sibling, 2 replies; 77+ messages in thread
From: Damien Le Moal @ 2019-09-19 8:26 UTC (permalink / raw)
To: Alan Stern, Andrea Vai
Cc: Johannes Thumshirn, Jens Axboe, USB list, SCSI development list,
Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
Martin K. Petersen, Greg KH, Hans Holmberg
On 2019/09/18 18:30, Alan Stern wrote:
> On Wed, 18 Sep 2019, Andrea Vai wrote:
>
>>> Also, I wonder if the changing the size of the data transfers would
>>> make any difference. This is easy to try; just write "64" to
>>> /sys/block/sd?/queue/max_sectors_kb (where the ? is the appropriate
>>> drive letter) after the drive is plugged in but before the test
>>> starts.
>>
>> ok, so I duplicated the tests above for the "64" case (it was
>> initially set as "120", if it is relevant to know), leading to 40 tests named as
>>
>> bad.mon.out_50000000_64_TIMESTAMP
>> bad.mon.out_50000000_non64_TIMESTAMP
>> good.mon.out_50000000_64_TIMESTAMP
>> good.mon.out_50000000_non64_TIMESTAMP
>>
>> where "64" denotes the ones done with that value in max_sectors_kb,
>> and "not64" the ones without it (as far as I can tell, it has been
>> always "120").
>>
>> So, we have 40 traces total. Each set of 10 trials is identified by
>> a text file, which contains the output log of the test script (and the
>> timestamps), also available in the download zipfile.
>>
>> Just to summarize here the times, they are respectively (number
>> expressed in seconds):
>>
>> BAD:
>> Logs: log_10trials_50MB_BAD_NonCanc_64.txt,
>> log_10trials_50MB_BAD_NonCanc_non64.txt
>> 64: 34, 34, 35, 39, 37, 32, 42, 44, 43, 40
>> not64: 61, 71, 59, 71, 62, 75, 62, 70, 62, 68
>> GOOD:
>> Logs: log_10trials_50MB_GOOD_NonCanc_64.txt,
>> log_10trials_50MB_GOOD_NonCanc_non64.txt
>> 64: 34, 32, 35, 34, 35, 33, 34, 33, 33, 33
>> not64: 32, 30, 32, 31, 31, 30, 32, 30, 32, 31
>
> The improvement from using "64" with the bad kernel is quite large.
> That alone would be a big help for you.
>
> However, I did see what appears to be a very significant difference
> between the bad and good kernel traces. It has to do with the order in
> which the blocks are accessed.
>
> Here is an extract from one of the bad traces. I have erased all the
> information except for the columns containing the block numbers to be
> written:
>
> 00019628 00
> 00019667 00
> 00019628 80
> 00019667 80
> 00019629 00
> 00019668 00
> 00019629 80
> 00019668 80
>
> Here is the equivalent portion from one of the good traces:
>
> 00019628 00
> 00019628 80
> 00019629 00
> 00019629 80
> 0001962a 00
> 0001962a 80
> 0001962b 00
> 0001962b 80
>
> Notice that under the good kernel, the block numbers increase
> monotonically in a single sequence. But under the bad kernel, the
> block numbers are not monotonic -- it looks like there are two separate
> threads each with its own strictly increasing sequence.
>
> This is exactly the sort of difference one might expect to see from
> the commit f664a3cc17b7 ("scsi: kill off the legacy IO path") you
> identified as the cause of the problem. With multiqueue I/O, it's not
> surprising to see multiple sequences of block numbers.
>
> Add it's not at all surprising that a consumer-grade USB storage device
> might do a much worse job of handling non-sequential writes than
> sequential ones.
>
> Which leads to a simple question for the SCSI or block-layer
> maintainers: Is there a sysfs setting Andrea can tweak which will
> effectively restrict a particular disk device down to a single I/O
> queue, forcing sequential access?
The scheduling inefficiency you are seeing may be coming from the fact that the
block layer does a direct issue of requests, bypassing the elevator, under some
conditions. One of these is sync requests on a multiqueue device. We hit this
problem on Zoned disks which can easily return an error for write requests
without the elevator throttling writes per zones (zone write locking). This
problem was discovered by Hans (on CC).
I discussed this with Hannes yesterday and we think we have a fix, but we'll
need to do a lot of testing as all block devices are potentially impacted by the
change, including stacked drivers (DM). Performance regression is scary with any
change in that area (see blk_mq_make_request() and use of
blk_mq_try_issue_directly() vs blk_mq_sched_insert_request()).
--
Damien Le Moal
Western Digital Research
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-09-18 16:30 ` Alan Stern
@ 2019-09-19 7:33 ` Andrea Vai
2019-09-19 17:54 ` Alan Stern
2019-09-19 8:26 ` Damien Le Moal
1 sibling, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-09-19 7:33 UTC (permalink / raw)
To: Alan Stern
Cc: Johannes Thumshirn, Jens Axboe, USB list, SCSI development list,
Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
Martin K. Petersen, Greg KH
Il giorno mer, 18/09/2019 alle 12.30 -0400, Alan Stern ha scritto:
> On Wed, 18 Sep 2019, Andrea Vai wrote:
> [...]
> > BAD:
> > Logs: log_10trials_50MB_BAD_NonCanc_64.txt,
> > log_10trials_50MB_BAD_NonCanc_non64.txt
> > 64: 34, 34, 35, 39, 37, 32, 42, 44, 43, 40
> > not64: 61, 71, 59, 71, 62, 75, 62, 70, 62, 68
> > GOOD:
> > Logs: log_10trials_50MB_GOOD_NonCanc_64.txt,
> > log_10trials_50MB_GOOD_NonCanc_non64.txt
> > 64: 34, 32, 35, 34, 35, 33, 34, 33, 33, 33
> > not64: 32, 30, 32, 31, 31, 30, 32, 30, 32, 31
>
> The improvement from using "64" with the bad kernel is quite
> large.
> That alone would be a big help for you.
Well, not so much, actually, because the backup would take twice the
time, that is quite annoying for me. But, apart from that, and from
the efforts of Alan and other people following this issue (thanks), I
would like to point out what I am not sure to have ever made clear
about my support request: I have understood that my problem is quite
specific, and don't want anyone to waste their time to help
specifically *me* (I can buy another media, use the "64" tweak, or
find any other workaround). But since we have identified the problem
as kernel-related, I am worried for other users, maybe new to linux,
that can have the same problem, and the evidence for them would be
that linux is extremely slow to copy file over some USB media. So,
among all the technical comments, I would like to make clear (if it's
not already clear) that in my opinion it would be important to solve
the problem without the need of user workarounds. Does it make sense?
Are we moving towards that goal?
BTW, another question: Alan refers to the slow media as a "consumer-
grade USB storage device". What could I do to identify and buy a "good
media"? Are there any features to look for?
Many thanks, and sorry if I ask anything obvious.
Bye,
Andrea
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-09-18 15:25 ` Andrea Vai
@ 2019-09-18 16:30 ` Alan Stern
2019-09-19 7:33 ` Andrea Vai
2019-09-19 8:26 ` Damien Le Moal
0 siblings, 2 replies; 77+ messages in thread
From: Alan Stern @ 2019-09-18 16:30 UTC (permalink / raw)
To: Andrea Vai
Cc: Johannes Thumshirn, Jens Axboe, USB list, SCSI development list,
Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
Martin K. Petersen, Greg KH
On Wed, 18 Sep 2019, Andrea Vai wrote:
> > Also, I wonder if the changing the size of the data transfers would
> > make any difference. This is easy to try; just write "64" to
> > /sys/block/sd?/queue/max_sectors_kb (where the ? is the appropriate
> > drive letter) after the drive is plugged in but before the test
> > starts.
>
> ok, so I duplicated the tests above for the "64" case (it was
> initially set as "120", if it is relevant to know), leading to 40 tests named as
>
> bad.mon.out_50000000_64_TIMESTAMP
> bad.mon.out_50000000_non64_TIMESTAMP
> good.mon.out_50000000_64_TIMESTAMP
> good.mon.out_50000000_non64_TIMESTAMP
>
> where "64" denotes the ones done with that value in max_sectors_kb,
> and "not64" the ones without it (as far as I can tell, it has been
> always "120").
>
> So, we have 40 traces total. Each set of 10 trials is identified by
> a text file, which contains the output log of the test script (and the
> timestamps), also available in the download zipfile.
>
> Just to summarize here the times, they are respectively (number
> expressed in seconds):
>
> BAD:
> Logs: log_10trials_50MB_BAD_NonCanc_64.txt,
> log_10trials_50MB_BAD_NonCanc_non64.txt
> 64: 34, 34, 35, 39, 37, 32, 42, 44, 43, 40
> not64: 61, 71, 59, 71, 62, 75, 62, 70, 62, 68
> GOOD:
> Logs: log_10trials_50MB_GOOD_NonCanc_64.txt,
> log_10trials_50MB_GOOD_NonCanc_non64.txt
> 64: 34, 32, 35, 34, 35, 33, 34, 33, 33, 33
> not64: 32, 30, 32, 31, 31, 30, 32, 30, 32, 31
The improvement from using "64" with the bad kernel is quite large.
That alone would be a big help for you.
However, I did see what appears to be a very significant difference
between the bad and good kernel traces. It has to do with the order in
which the blocks are accessed.
Here is an extract from one of the bad traces. I have erased all the
information except for the columns containing the block numbers to be
written:
00019628 00
00019667 00
00019628 80
00019667 80
00019629 00
00019668 00
00019629 80
00019668 80
Here is the equivalent portion from one of the good traces:
00019628 00
00019628 80
00019629 00
00019629 80
0001962a 00
0001962a 80
0001962b 00
0001962b 80
Notice that under the good kernel, the block numbers increase
monotonically in a single sequence. But under the bad kernel, the
block numbers are not monotonic -- it looks like there are two separate
threads each with its own strictly increasing sequence.
This is exactly the sort of difference one might expect to see from
the commit f664a3cc17b7 ("scsi: kill off the legacy IO path") you
identified as the cause of the problem. With multiqueue I/O, it's not
surprising to see multiple sequences of block numbers.
Add it's not at all surprising that a consumer-grade USB storage device
might do a much worse job of handling non-sequential writes than
sequential ones.
Which leads to a simple question for the SCSI or block-layer
maintainers: Is there a sysfs setting Andrea can tweak which will
effectively restrict a particular disk device down to a single I/O
queue, forcing sequential access?
Alan Stern
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-08-26 16:33 ` Alan Stern
@ 2019-09-18 15:25 ` Andrea Vai
2019-09-18 16:30 ` Alan Stern
0 siblings, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-09-18 15:25 UTC (permalink / raw)
To: Alan Stern
Cc: Johannes Thumshirn, Jens Axboe, linux-usb, linux-scsi,
Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
Martin K. Petersen, Greg KH
Il giorno lun 26 ago 2019 alle ore 18:33 Alan Stern <
stern@rowland.harvard.edu> ha scritto:
> [...]
> In fact, even the traces where the file doesn't exist beforehand
> show
> some delays. Just not as many delays as the traces where the file
> does
> exist. And again, each delay is in the middle of a write command,
> not
> between commands.
>
> I suppose changes to the upper software layers could affect which
> blocks are assigned when a new file is written. Perhaps one kernel
> re-uses the same old blocks that had been previously occupied and
> the
> other kernel allocates a completely new set of blocks. That might
> change the drive's behavior. The quick way to tell is to record two
> usbmon traces, one under the "good" kernel and one under the "bad"
> kernel, where each test involves writing over a file that already
> exists (say, 50 MB) -- the same file for both tests. The block
> numbers
> will appear in the traces.
ok, I performed 10 tests for each kernel, so we have 20 traces.
> Also, I wonder if the changing the size of the data transfers would
> make any difference. This is easy to try; just write "64" to
> /sys/block/sd?/queue/max_sectors_kb (where the ? is the appropriate
> drive letter) after the drive is plugged in but before the test
> starts.
ok, so I duplicated the tests above for the "64" case (it was
initially set as "120", if it is relevant to know), leading to 40 tests named as
bad.mon.out_50000000_64_TIMESTAMP
bad.mon.out_50000000_non64_TIMESTAMP
good.mon.out_50000000_64_TIMESTAMP
good.mon.out_50000000_non64_TIMESTAMP
where "64" denotes the ones done with that value in max_sectors_kb,
and "not64" the ones without it (as far as I can tell, it has been
always "120").
So, we have 40 traces total. Each set of 10 trials is identified by
a text file, which contains the output log of the test script (and the
timestamps), also available in the download zipfile.
Just to summarize here the times, they are respectively (number
expressed in seconds):
BAD:
Logs: log_10trials_50MB_BAD_NonCanc_64.txt,
log_10trials_50MB_BAD_NonCanc_non64.txt
64: 34, 34, 35, 39, 37, 32, 42, 44, 43, 40
not64: 61, 71, 59, 71, 62, 75, 62, 70, 62, 68
GOOD:
Logs: log_10trials_50MB_GOOD_NonCanc_64.txt,
log_10trials_50MB_GOOD_NonCanc_non64.txt
64: 34, 32, 35, 34, 35, 33, 34, 33, 33, 33
not64: 32, 30, 32, 31, 31, 30, 32, 30, 32, 31
Finally, one note about the workaround proposed by Alan, "delete the
file before copying". My original problem occurred while using a
backup software (dar - see http://dar.linux.free.fr/). So, I tried now
to do the backup by deleting the existing file beforehand, and it
still takes a lot of time with bad kernel: a 900 file backup takes
~160sec with GOOD kernel, and >40min with BAD kernel. I also tried the
"64" tweak in the BAD kernel and it becomes ~300s. Then, I also tried
the "64" case with good kernel, and became ~140s. Detailed data:
GOOD (not "64): 155s, 151s
GOOD ("64"): 142s, 141s
BAD (not "64"): 47minutes, 43minutes
BAD ("64"): 315s, 288s, 268s, 239s, 302s
The command ran is:
$ SECONDS=0; rm /run/media/andrea/BAK_ANDVAI/aero.1.dar && dar -c /run/media/andrea/BAK_ANDVAI/aero -R /home/andrea/Musica/MP3/Aerosmith && umount /run/media/andrea/BAK_ANDVAI; echo "Ci ho messo: $SECONDS secondi."
Speculations:
- It seems the "64" value plays a role, more evident on "bad" kernels
(~halves the time) and less (but still existing?) on "good" kernels;
- dar with the bad kernel, with the "delete beforehand" action, is
still an order of magnitude slower than with the good kernel (so, it
behaves the same way as in the "overwrite" case). Maybe it depends on
the way dar itself writes data... I don't know if you can understand
it, or we should ask for a light to the dar developer(s) about it.
You can grab the traces at
http://fisica.unipv.it/transfer/usbmon_logs_2.zip
Thanks, and bye
Andrea
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-08-26 6:09 ` Andrea Vai
@ 2019-08-26 16:33 ` Alan Stern
2019-09-18 15:25 ` Andrea Vai
0 siblings, 1 reply; 77+ messages in thread
From: Alan Stern @ 2019-08-26 16:33 UTC (permalink / raw)
To: Andrea Vai
Cc: Johannes Thumshirn, Jens Axboe, linux-usb, linux-scsi,
Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
Martin K. Petersen, Greg KH
On Mon, 26 Aug 2019, Andrea Vai wrote:
> ok, so you can grab them at
>
> http://fisica.unipv.it/transfer/usbmon_logs.zip
>
> (they will be automatically removed from there in a couple of weeks).
>
> For each size there is a .txt file (which contains the terminal
> output) and 10 bad.mon.out_.... trace files. The file suffix "NonCanc"
> means there has not been file deletion before copy; while "Canc" means
> the opposite.
>
> Each trace file name is identified by a timestamp that is also
> referenced inside the txt, so if you want to get i.e. the 39-sec trial
> for the 10MB filesize you have to open the ...10MB....txt, search for
> the 39 seconds total time string ("Dopo stop trace: 39"), look at the
> beginning of that trial, a dozen rows before, take note of the
> timestamp, and open the corresponding bad.mon.out file (of course, if
> there are more trials with the same time, you have to identify it by
> counting its position (7th in the example above)).
>
> To make it more simple:
>
> $ seconds=39; size=10MB; grep -B14 "Dopo stop trace: $seconds" log_10trials_"$size"_NonCanc.txt
>
> should show you more straightly the part(s) you need.
>
> > Odd that the delays never occur when you're writing a new file. (If
> > nothing else, that gives you a way to work around the problem!)
>
> Thank you, didn't realize that :-) I will try it.
In fact, even the traces where the file doesn't exist beforehand show
some delays. Just not as many delays as the traces where the file does
exist. And again, each delay is in the middle of a write command, not
between commands.
I suppose changes to the upper software layers could affect which
blocks are assigned when a new file is written. Perhaps one kernel
re-uses the same old blocks that had been previously occupied and the
other kernel allocates a completely new set of blocks. That might
change the drive's behavior. The quick way to tell is to record two
usbmon traces, one under the "good" kernel and one under the "bad"
kernel, where each test involves writing over a file that already
exists (say, 50 MB) -- the same file for both tests. The block numbers
will appear in the traces.
Also, I wonder if the changing the size of the data transfers would
make any difference. This is easy to try; just write "64" to
/sys/block/sd?/queue/max_sectors_kb (where the ? is the appropriate
drive letter) after the drive is plugged in but before the test starts.
Alan Stern
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-08-23 20:42 ` Alan Stern
@ 2019-08-26 6:09 ` Andrea Vai
2019-08-26 16:33 ` Alan Stern
0 siblings, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-08-26 6:09 UTC (permalink / raw)
To: Alan Stern
Cc: Johannes Thumshirn, Jens Axboe, linux-usb, linux-scsi,
Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
Martin K. Petersen, Greg KH
[Sorry, I had previously sent the message to the list but has been
rejected. Sorry for any duplicate]
Il giorno ven, 23/08/2019 alle 16.42 -0400, Alan Stern ha scritto:
> On Fri, 23 Aug 2019, Andrea Vai wrote:
>
> > Il giorno mar, 20/08/2019 alle 13.13 -0400, Alan Stern ha scritto:
> > > On Mon, 19 Aug 2019, Andrea Vai wrote:
> > >
> > > > Hi Alan,
> > > > I attach the two traces, collected as follows:
> > > >
> > > > - start the trace;
> > > > - wait 10 seconds;
> > > > - plug the drive;
> > > > - wait 5 seconds;
> > > > - mount the drive;
> > > > - wait 5 seconds;
> > > > - copy a 500 byte file;
> > > > - wait 5 seconds;
> > > > - unmount the drive;
> > > > - wait 5 seconds;
> > > > - stop the trace.
> > >
> > > Still no noticeable differences between the two traces. They
> both
> > > include a 1.2 second delay shortly after the writing starts, and
> > > the
> > > initialization sequences are the same.
> > >
> > > I really don't know where to look for this. The only thing I
> can
> > > think
> > > of at this point is to repeat this test, but using a file large
> > > enough
> > > for the difference in writing speed to show up plainly.
> > >
> > > By the way, it would be best to run the tests with the smallest
> > > possible number of other USB devices plugged in. None at all,
> if
> > > you
> > > can arrange it.
> >
> > Thanks, I went some steps further on this.
> > The following considerations all apply to the "bad" kernel.
> >
> > Increasing the filesize lead me to find out that using a file
> sized
> > less than roughly 10MB the problem does not happen.
> >
> > I found these results by making sets of 10 tries for each
> filesize,
> > using a filesize of 1kB, 10kB, 100kB, 1MB, 10MB, 100MB, 500MB (so,
> we
> > have 70 usbmon logs on these). If we define "fast" a copy that
> takes
> > (roughly(*)) no more time to complete than all the other tries in
> its
> > set, and "slow" elsewhere (=one or more tries in its set are
> > (sensibly(*)) faster), I noticed that in each set with a filesize
> of
> > 10MB or more the behavior can be very different: sometimes the
> copy is
> > still "fast", sometimes is "slow". The frequency of the "slow"
> copies
> > increases with the filesize. Also, among the "slow" copies in a
> set,
> > the time can be very different.
> >
> > Also, I found that if the file is not present on the target
> location
> > (i.e. the USB pendrive), the problem does not happen (I have ten
> > usbmon logs here, taken in the worst scenario (500MB filesize)).
> >
> > Tell me which log(s) would you like me to send you: I can sum up
> here
> > all the sets of tries, and the time their copies took to complete
> (in
> > seconds):
> >
> > 1kB: 26, 27, 26, 26, 27, 26, 26, 27, 26, 27
> > 10kB: 27, 27, 26, 26, 27, 26, 27, 26, 27, 27
> > 100kB: 26, 26, 26, 27, 26, 26, 26, 27, 27, 27
> > 1MB: 26, 27, 27, 27, 27, 27, 27, 27, 27, 26
> > 10MB: 27, 31, 37, 27, 38, 27, 39, 27, 30, 28
> > 100MB: 32, 32, 144, 32, 145, 32, 123, 32, 153, 123
> > 500MB: 56, 1396, 747, 131, 795, 764, 292, 1021, 807, 516
> >
> > Also, note that the first copy is always "fast", because each file
> was
> > initially not present on the pendrive. As said, I did one test of
> 10
> > tries by deleting the file on the pendrive before copying it
> again,
> > and the results are
> >
> > 500MB: 56, 56, 57, 57, 56, 56, 60, 25***, 55, 56 (***Note the
> "fake"
> > 25s, doesn't matter because I forgot to plug the pendrive :-/ )
> >
> > I have made a script to semi-automate all the tests I have done. I
> > attach the script here, so anyone interested could check it for
> any
> > mistake (remember I am not very skilled so I may have wrote buggy
> > code, done wrong assumptions, etc.). Please note that I decreased
> the
> > time between the trace start and the drive plugging from 10s to 5s
> > (simply to reduce the time needed to me to look at the countdown).
> Of
> > course I can do again the test(s) you need with a bigger amount of
> > $wait.
> >
> > The script has been run with the command
> >
> > # for k in {1..10}; do size=1000; ./test_usbmon $size && ping -a
> -c 5 8.8.8.8 ; done
> > (example for 1kB filesize)
> >
> > or, in the set of "delete before copy",
> >
> > # for k in {1..10}; do size=500000000; ./cancellaTestFile $size &&
> ./test_usbmon $size && ping -a -c 5 8.8.8.8 ; done
> >
> > The ping command is there just to have a sound alarm when
> finished.
> >
> > I also attach the script to delete the file ("cancellaTestFile").
> >
> > I took care to plug the pendrive exactly at the end of the
> countdown,
> > to keep the times in the logs more simple to detect and manage by
> you.
> >
> > I have also logged all the terminal output log of the script.
> >
> > Last note: I ran all the tests without any other USB device
> connected
> > but the pendrive (well, actually there is a card reader connected
> to
> > the internal USB connector, but on another bus. I didn't want to
> open
> > the case and disconnect it but of course I can do it if needed).
> > Thanks for pointing it out.
> >
> > Thanks, and bye
> > Andrea
> >
> > (*) as an example, on a set that shows the total elapsed time in
> > seconds being
> >
> > 26, 27, 27, 27, 27, 27, 27, 27, 27, 26
> >
> > I have assumed all of the copies to be "fast", while in the set
> >
> > 32, 32, 144, 32, 145, 32, 123, 32, 153, 123
> >
> > I have assumed 5 of the copies as "fast" (the ones that took 32
> > seconds) and the other "slow". Not going to deepen in some
> standard
> > deviation evaluation, etc., but if you'd like to I can provide
> some
> > more scientific detailed data :-)
>
> Wow, that sounds like a lot of work.
just a drop in the ocean, compared to yours :-)
> Let's start with the 39-second run for the 10-MB file. If you can
> put
> the trace files on a server somewhere, available for downloading,
> that
> would avoid sending a lot of uninteresting data to the mailing list.
ok, so you can grab them at
http://fisica.unipv.it/transfer/usbmon_logs.zip
(they will be automatically removed from there in a couple of weeks).
For each size there is a .txt file (which contains the terminal
output) and 10 bad.mon.out_.... trace files. The file suffix "NonCanc"
means there has not been file deletion before copy; while "Canc" means
the opposite.
Each trace file name is identified by a timestamp that is also
referenced inside the txt, so if you want to get i.e. the 39-sec trial
for the 10MB filesize you have to open the ...10MB....txt, search for
the 39 seconds total time string ("Dopo stop trace: 39"), look at the
beginning of that trial, a dozen rows before, take note of the
timestamp, and open the corresponding bad.mon.out file (of course, if
there are more trials with the same time, you have to identify it by
counting its position (7th in the example above)).
To make it more simple:
$ seconds=39; size=10MB; grep -B14 "Dopo stop trace: $seconds" log_10trials_"$size"_NonCanc.txt
should show you more straightly the part(s) you need.
> Odd that the delays never occur when you're writing a new file. (If
> nothing else, that gives you a way to work around the problem!)
Thank you, didn't realize that :-) I will try it.
Thanks, and bye
Andrea
--
Andrea Vai
University of Pavia
Department of Physics
Via Bassi, 6
27100 Pavia PV
Tel. +39 0382 987489
Mob. +39 328 3354086
http://fisica.unipv.it
http://www.andreavai.it
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-08-23 10:39 ` Andrea Vai
@ 2019-08-23 20:42 ` Alan Stern
2019-08-26 6:09 ` Andrea Vai
0 siblings, 1 reply; 77+ messages in thread
From: Alan Stern @ 2019-08-23 20:42 UTC (permalink / raw)
To: Andrea Vai
Cc: Johannes Thumshirn, Jens Axboe, linux-usb, linux-scsi,
Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
Martin K. Petersen, Greg KH
On Fri, 23 Aug 2019, Andrea Vai wrote:
> Il giorno mar, 20/08/2019 alle 13.13 -0400, Alan Stern ha scritto:
> > On Mon, 19 Aug 2019, Andrea Vai wrote:
> >
> > > Hi Alan,
> > > I attach the two traces, collected as follows:
> > >
> > > - start the trace;
> > > - wait 10 seconds;
> > > - plug the drive;
> > > - wait 5 seconds;
> > > - mount the drive;
> > > - wait 5 seconds;
> > > - copy a 500 byte file;
> > > - wait 5 seconds;
> > > - unmount the drive;
> > > - wait 5 seconds;
> > > - stop the trace.
> >
> > Still no noticeable differences between the two traces. They both
> > include a 1.2 second delay shortly after the writing starts, and
> > the
> > initialization sequences are the same.
> >
> > I really don't know where to look for this. The only thing I can
> > think
> > of at this point is to repeat this test, but using a file large
> > enough
> > for the difference in writing speed to show up plainly.
> >
> > By the way, it would be best to run the tests with the smallest
> > possible number of other USB devices plugged in. None at all, if
> > you
> > can arrange it.
>
> Thanks, I went some steps further on this.
> The following considerations all apply to the "bad" kernel.
>
> Increasing the filesize lead me to find out that using a file sized
> less than roughly 10MB the problem does not happen.
>
> I found these results by making sets of 10 tries for each filesize,
> using a filesize of 1kB, 10kB, 100kB, 1MB, 10MB, 100MB, 500MB (so, we
> have 70 usbmon logs on these). If we define "fast" a copy that takes
> (roughly(*)) no more time to complete than all the other tries in its
> set, and "slow" elsewhere (=one or more tries in its set are
> (sensibly(*)) faster), I noticed that in each set with a filesize of
> 10MB or more the behavior can be very different: sometimes the copy is
> still "fast", sometimes is "slow". The frequency of the "slow" copies
> increases with the filesize. Also, among the "slow" copies in a set,
> the time can be very different.
>
> Also, I found that if the file is not present on the target location
> (i.e. the USB pendrive), the problem does not happen (I have ten
> usbmon logs here, taken in the worst scenario (500MB filesize)).
>
> Tell me which log(s) would you like me to send you: I can sum up here
> all the sets of tries, and the time their copies took to complete (in
> seconds):
>
> 1kB: 26, 27, 26, 26, 27, 26, 26, 27, 26, 27
> 10kB: 27, 27, 26, 26, 27, 26, 27, 26, 27, 27
> 100kB: 26, 26, 26, 27, 26, 26, 26, 27, 27, 27
> 1MB: 26, 27, 27, 27, 27, 27, 27, 27, 27, 26
> 10MB: 27, 31, 37, 27, 38, 27, 39, 27, 30, 28
> 100MB: 32, 32, 144, 32, 145, 32, 123, 32, 153, 123
> 500MB: 56, 1396, 747, 131, 795, 764, 292, 1021, 807, 516
>
> Also, note that the first copy is always "fast", because each file was
> initially not present on the pendrive. As said, I did one test of 10
> tries by deleting the file on the pendrive before copying it again,
> and the results are
>
> 500MB: 56, 56, 57, 57, 56, 56, 60, 25***, 55, 56 (***Note the "fake"
> 25s, doesn't matter because I forgot to plug the pendrive :-/ )
>
> I have made a script to semi-automate all the tests I have done. I
> attach the script here, so anyone interested could check it for any
> mistake (remember I am not very skilled so I may have wrote buggy
> code, done wrong assumptions, etc.). Please note that I decreased the
> time between the trace start and the drive plugging from 10s to 5s
> (simply to reduce the time needed to me to look at the countdown). Of
> course I can do again the test(s) you need with a bigger amount of
> $wait.
>
> The script has been run with the command
>
> # for k in {1..10}; do size=1000; ./test_usbmon $size && ping -a -c 5 8.8.8.8 ; done
> (example for 1kB filesize)
>
> or, in the set of "delete before copy",
>
> # for k in {1..10}; do size=500000000; ./cancellaTestFile $size && ./test_usbmon $size && ping -a -c 5 8.8.8.8 ; done
>
> The ping command is there just to have a sound alarm when finished.
>
> I also attach the script to delete the file ("cancellaTestFile").
>
> I took care to plug the pendrive exactly at the end of the countdown,
> to keep the times in the logs more simple to detect and manage by you.
>
> I have also logged all the terminal output log of the script.
>
> Last note: I ran all the tests without any other USB device connected
> but the pendrive (well, actually there is a card reader connected to
> the internal USB connector, but on another bus. I didn't want to open
> the case and disconnect it but of course I can do it if needed).
> Thanks for pointing it out.
>
> Thanks, and bye
> Andrea
>
> (*) as an example, on a set that shows the total elapsed time in
> seconds being
>
> 26, 27, 27, 27, 27, 27, 27, 27, 27, 26
>
> I have assumed all of the copies to be "fast", while in the set
>
> 32, 32, 144, 32, 145, 32, 123, 32, 153, 123
>
> I have assumed 5 of the copies as "fast" (the ones that took 32
> seconds) and the other "slow". Not going to deepen in some standard
> deviation evaluation, etc., but if you'd like to I can provide some
> more scientific detailed data :-)
Wow, that sounds like a lot of work.
Let's start with the 39-second run for the 10-MB file. If you can put
the trace files on a server somewhere, available for downloading, that
would avoid sending a lot of uninteresting data to the mailing list.
Odd that the delays never occur when you're writing a new file. (If
nothing else, that gives you a way to work around the problem!) It's
hard to say what it means, though. Maybe the flash drive doesn't like
overwriting used blocks.
Alan Stern
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
2019-08-20 17:13 ` Alan Stern
@ 2019-08-23 10:39 ` Andrea Vai
2019-08-23 20:42 ` Alan Stern
0 siblings, 1 reply; 77+ messages in thread
From: Andrea Vai @ 2019-08-23 10:39 UTC (permalink / raw)
To: Alan Stern
Cc: Johannes Thumshirn, Jens Axboe, linux-usb, linux-scsi,
Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
Martin K. Petersen, Greg KH
[-- Attachment #1: Type: text/plain, Size: 4851 bytes --]
Il giorno mar, 20/08/2019 alle 13.13 -0400, Alan Stern ha scritto:
> On Mon, 19 Aug 2019, Andrea Vai wrote:
>
> > Hi Alan,
> > I attach the two traces, collected as follows:
> >
> > - start the trace;
> > - wait 10 seconds;
> > - plug the drive;
> > - wait 5 seconds;
> > - mount the drive;
> > - wait 5 seconds;
> > - copy a 500 byte file;
> > - wait 5 seconds;
> > - unmount the drive;
> > - wait 5 seconds;
> > - stop the trace.
>
> Still no noticeable differences between the two traces. They both
> include a 1.2 second delay shortly after the writing starts, and
> the
> initialization sequences are the same.
>
> I really don't know where to look for this. The only thing I can
> think
> of at this point is to repeat this test, but using a file large
> enough
> for the difference in writing speed to show up plainly.
>
> By the way, it would be best to run the tests with the smallest
> possible number of other USB devices plugged in. None at all, if
> you
> can arrange it.
Thanks, I went some steps further on this.
The following considerations all apply to the "bad" kernel.
Increasing the filesize lead me to find out that using a file sized
less than roughly 10MB the problem does not happen.
I found these results by making sets of 10 tries for each filesize,
using a filesize of 1kB, 10kB, 100kB, 1MB, 10MB, 100MB, 500MB (so, we
have 70 usbmon logs on these). If we define "fast" a copy that takes
(roughly(*)) no more time to complete than all the other tries in its
set, and "slow" elsewhere (=one or more tries in its set are
(sensibly(*)) faster), I noticed that in each set with a filesize of
10MB or more the behavior can be very different: sometimes the copy is
still "fast", sometimes is "slow". The frequency of the "slow" copies
increases with the filesize. Also, among the "slow" copies in a set,
the time can be very different.
Also, I found that if the file is not present on the target location
(i.e. the USB pendrive), the problem does not happen (I have ten
usbmon logs here, taken in the worst scenario (500MB filesize)).
Tell me which log(s) would you like me to send you: I can sum up here
all the sets of tries, and the time their copies took to complete (in
seconds):
1kB: 26, 27, 26, 26, 27, 26, 26, 27, 26, 27
10kB: 27, 27, 26, 26, 27, 26, 27, 26, 27, 27
100kB: 26, 26, 26, 27, 26, 26, 26, 27, 27, 27
1MB: 26, 27, 27, 27, 27, 27, 27, 27, 27, 26
10MB: 27, 31, 37, 27, 38, 27, 39, 27, 30, 28
100MB: 32, 32, 144, 32, 145, 32, 123, 32, 153, 123
500MB: 56, 1396, 747, 131, 795, 764, 292, 1021, 807, 516
Also, note that the first copy is always "fast", because each file was
initially not present on the pendrive. As said, I did one test of 10
tries by deleting the file on the pendrive before copying it again,
and the results are
500MB: 56, 56, 57, 57, 56, 56, 60, 25***, 55, 56 (***Note the "fake"
25s, doesn't matter because I forgot to plug the pendrive :-/ )
I have made a script to semi-automate all the tests I have done. I
attach the script here, so anyone interested could check it for any
mistake (remember I am not very skilled so I may have wrote buggy
code, done wrong assumptions, etc.). Please note that I decreased the
time between the trace start and the drive plugging from 10s to 5s
(simply to reduce the time needed to me to look at the countdown). Of
course I can do again the test(s) you need with a bigger amount of
$wait.
The script has been run with the command
# for k in {1..10}; do size=1000; ./test_usbmon $size && ping -a -c 5 8.8.8.8 ; done
(example for 1kB filesize)
or, in the set of "delete before copy",
# for k in {1..10}; do size=500000000; ./cancellaTestFile $size && ./test_usbmon $size && ping -a -c 5 8.8.8.8 ; done
The ping command is there just to have a sound alarm when finished.
I also attach the script to delete the file ("cancellaTestFile").
I took care to plug the pendrive exactly at the end of the countdown,
to keep the times in the logs more simple to detect and manage by you.
I have also logged all the terminal output log of the script.
Last note: I ran all the tests without any other USB device connected
but the pendrive (well, actually there is a card reader connected to
the internal USB connector, but on another bus. I didn't want to open
the case and disconnect it but of course I can do it if needed).
Thanks for pointing it out.
Thanks, and bye
Andrea
(*) as an example, on a set that shows the total elapsed time in
seconds being
26, 27, 27, 27, 27, 27, 27, 27, 27, 26
I have assumed all of the copies to be "fast", while in the set
32, 32, 144, 32, 145, 32, 123, 32, 153, 123
I have assumed 5 of the copies as "fast" (the ones that took 32
seconds) and the other "slow". Not going to deepen in some standard
deviation evaluation, etc., but if you'd like to I can provide some
more scientific detailed data :-)
[-- Attachment #2: test_usbmon --]
[-- Type: application/x-shellscript, Size: 1607 bytes --]
[-- Attachment #3: cancellaTestFile --]
[-- Type: application/x-shellscript, Size: 296 bytes --]
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
[not found] <307581a490b610c3025ee80f79a465a89d68ed19.camel@unipv.it>
@ 2019-08-20 17:13 ` Alan Stern
2019-08-23 10:39 ` Andrea Vai
0 siblings, 1 reply; 77+ messages in thread
From: Alan Stern @ 2019-08-20 17:13 UTC (permalink / raw)
To: Andrea Vai
Cc: Johannes Thumshirn, Jens Axboe, linux-usb, linux-scsi,
Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
Martin K. Petersen, Greg KH
On Mon, 19 Aug 2019, Andrea Vai wrote:
> Hi Alan,
> I attach the two traces, collected as follows:
>
> - start the trace;
> - wait 10 seconds;
> - plug the drive;
> - wait 5 seconds;
> - mount the drive;
> - wait 5 seconds;
> - copy a 500 byte file;
> - wait 5 seconds;
> - unmount the drive;
> - wait 5 seconds;
> - stop the trace.
Still no noticeable differences between the two traces. They both
include a 1.2 second delay shortly after the writing starts, and the
initialization sequences are the same.
I really don't know where to look for this. The only thing I can think
of at this point is to repeat this test, but using a file large enough
for the difference in writing speed to show up plainly.
By the way, it would be best to run the tests with the smallest
possible number of other USB devices plugged in. None at all, if you
can arrange it.
Alan Stern
^ permalink raw reply [flat|nested] 77+ messages in thread
* Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
[not found] <e3f87757f7a0fdf551e911ad32fc8122eebe04c7.camel@unipv.it>
@ 2019-08-13 19:52 ` Alan Stern
0 siblings, 0 replies; 77+ messages in thread
From: Alan Stern @ 2019-08-13 19:52 UTC (permalink / raw)
To: Andrea Vai
Cc: Johannes Thumshirn, Jens Axboe, linux-usb, linux-scsi,
Himanshu Madhani, Hannes Reinecke, Ming Lei, Omar Sandoval,
Martin K. Petersen, Greg KH
On Fri, 9 Aug 2019, Andrea Vai wrote:
> Il giorno lun, 08/07/2019 alle 11.38 -0400, Alan Stern ha scritto:
> >
> > [...]
> > Andrea, another thing you could try is to collect a usbmon trace
> > under
> > > > one of the "slow" kernels. Follow the instructions in
> > > > Documentation/usb/usbmon.txt. I think you could kill the file-
> > copy
> > > > operation after just a couple of seconds; that should provide
> > enough
> > > > trace information to help see what causes the slowdown.
> > > >
> > > > (If you want, do the same test with a "fast" kernel and then
> > we'll
> > > > compare the results.)
> > > >
> > > > Alan Stern
> > > >
> > >
> > > Thanks Alan,
> > > so I attach two ouputs, one for a "good" and one for a "bad"
> > kernel.
> >
> > [...]
> > I don't know what the results mean, but I _can_ tell you what's
> > happening. Every so often (at intervals of about a second) the pen
> > drive completely stops communicating with the "bad" kernel for about
> > one second and then starts up again.
> >
> > Here's a short example from the "bad" trace:
> >
> > ffff9169f0d399c0 513072808 S Bo:6:008:2 -115 122880 = 00000000
> > 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> > ffff9169f0d399c0 514262176 C Bo:6:008:2 0 122880 >
> >
> > The second column is a timestamp (in microseconds). This little
> > extract shows a 120-KB write starting at time 513.072 and ending at
> > 514.262, more than a second later. Normally such a write would
> > complete in about 0.06 s.
> >
> > The cumulative effect of all these delays is to slow the transfer
> > drastically. The "good" kernel trace shows a few delays like this,
> > but
> > only one or two.
> >
> > I have no idea how commit f664a3cc17b7, getting rid of the legacy
> > IO
> > path, could have caused these delays. It seems more likely that
> > the
> > pen drive itself is the cause, perhaps because it is flushing
> > buffers
> > more often under the "bad" kernel.
> >
> > I'd like you to try doing another pair of usbmon tests. This time,
> > start collecting the usbmon trace _before_ you plug in the pen
> > drive,
> > and stop the trace shortly after the pen drive has been mounted.
> > Don't try to transfer any data. Perhaps the two kernels are
> > initializing the pen drive with different settings and that accounts
> > for the different behaviors.
>
> Thanks Alan, sorry for the delay (I haven't had physical access to the
> device for a while). I attach the two new usbmon logs.
>
> Meanwhile, I can try with other pendrives, although I already tried
> two more and both behave "bad" (but one is identical to the first, and
> the other one is similar).
I looked through your two traces. Unfortunately they don't help --
they are essentially identical. Exactly the same sequence of commands
was sent to the device in both traces, except for a couple of
inconsequential TEST UNIT READY commands.
There are some timing differences, but they occur in between commands,
not during a command. They probably are the times you spent typing
stuff into the terminal.
Can you run another test? Similar to the last one, but this time start
collecting the usbmon trace at least 10 seconds after the drive is
plugged in. Then mount the drive, copy a small file (say less than 500
bytes) to it, unmount it, and then stop the trace.
Alan Stern
^ permalink raw reply [flat|nested] 77+ messages in thread
end of thread, other threads:[~2019-11-28 17:11 UTC | newest]
Thread overview: 77+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-07-02 10:46 Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6 Andrea Vai
2019-07-02 11:51 ` Johannes Thumshirn
2019-07-02 22:36 ` Andrea Vai
2019-07-03 7:29 ` Johannes Thumshirn
2019-07-03 14:23 ` Alan Stern
2019-07-06 22:06 ` Andrea Vai
2019-07-08 15:38 ` Alan Stern
2019-07-02 12:01 ` Ming Lei
2019-07-02 22:39 ` Andrea Vai
2019-07-03 2:01 ` Ming Lei
2019-07-03 5:11 ` Andrea Vai
2019-07-03 6:36 ` Ming Lei
2019-07-03 15:27 ` Chris Murphy
2019-07-06 9:33 ` Andrea Vai
2019-07-08 1:01 ` Ming Lei
2019-07-09 21:18 ` Andrea Vai
2019-07-10 2:44 ` Ming Lei
[not found] <e3f87757f7a0fdf551e911ad32fc8122eebe04c7.camel@unipv.it>
2019-08-13 19:52 ` Alan Stern
[not found] <307581a490b610c3025ee80f79a465a89d68ed19.camel@unipv.it>
2019-08-20 17:13 ` Alan Stern
2019-08-23 10:39 ` Andrea Vai
2019-08-23 20:42 ` Alan Stern
2019-08-26 6:09 ` Andrea Vai
2019-08-26 16:33 ` Alan Stern
2019-09-18 15:25 ` Andrea Vai
2019-09-18 16:30 ` Alan Stern
2019-09-19 7:33 ` Andrea Vai
2019-09-19 17:54 ` Alan Stern
2019-09-20 7:25 ` Andrea Vai
2019-09-20 7:44 ` Greg KH
2019-09-19 8:26 ` Damien Le Moal
2019-09-19 8:55 ` Ming Lei
2019-09-19 9:09 ` Damien Le Moal
2019-09-19 9:21 ` Ming Lei
2019-09-19 14:01 ` Alan Stern
2019-09-19 14:14 ` Damien Le Moal
2019-09-20 7:03 ` Andrea Vai
2019-09-25 19:30 ` Alan Stern
2019-09-25 19:36 ` Jens Axboe
2019-09-27 15:47 ` Andrea Vai
2019-11-04 16:00 ` Andrea Vai
2019-11-04 18:20 ` Alan Stern
2019-11-05 11:48 ` Andrea Vai
2019-11-05 18:31 ` Alan Stern
2019-11-05 23:29 ` Jens Axboe
2019-11-06 16:03 ` Alan Stern
2019-11-06 22:13 ` Damien Le Moal
2019-11-07 7:04 ` Andrea Vai
2019-11-07 7:54 ` Damien Le Moal
2019-11-07 18:59 ` Andrea Vai
2019-11-08 8:42 ` Damien Le Moal
2019-11-08 14:33 ` Jens Axboe
2019-11-11 10:46 ` Andrea Vai
2019-11-09 10:09 ` Ming Lei
2019-11-09 22:28 ` Ming Lei
2019-11-11 10:50 ` Andrea Vai
2019-11-11 11:05 ` Ming Lei
2019-11-11 11:13 ` Andrea Vai
2019-11-22 19:16 ` Andrea Vai
2019-11-23 7:28 ` Ming Lei
2019-11-23 15:44 ` Andrea Vai
2019-11-25 3:54 ` Ming Lei
2019-11-25 10:11 ` Andrea Vai
2019-11-25 10:29 ` Ming Lei
2019-11-25 14:58 ` Andrea Vai
2019-11-25 15:15 ` Ming Lei
2019-11-25 18:51 ` Andrea Vai
2019-11-26 2:32 ` Ming Lei
2019-11-26 7:46 ` Andrea Vai
2019-11-26 9:15 ` Ming Lei
2019-11-26 10:24 ` Ming Lei
2019-11-26 11:14 ` Andrea Vai
2019-11-27 2:05 ` Ming Lei
2019-11-27 9:39 ` Andrea Vai
2019-11-27 13:08 ` Ming Lei
2019-11-27 15:01 ` Andrea Vai
2019-11-27 0:21 ` Finn Thain
2019-11-28 17:10 ` Andrea Vai
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).