From: Ming Lei <ming.lei@redhat.com>
To: Andrea Vai <andrea.vai@unipv.it>
Cc: Jens Axboe <axboe@kernel.dk>,
linux-usb@vger.kernel.org, linux-scsi@vger.kernel.org,
Himanshu Madhani <himanshu.madhani@cavium.com>,
Hannes Reinecke <hare@suse.com>, Omar Sandoval <osandov@fb.com>,
"Martin K. Petersen" <martin.petersen@oracle.com>,
Greg KH <gregkh@linuxfoundation.org>,
Alan Stern <stern@rowland.harvard.edu>
Subject: Re: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
Date: Wed, 10 Jul 2019 10:44:40 +0800
Message-ID: <20190710024439.GA2621@ming.t460p> (raw)
In-Reply-To: <20190709211838.GA9645@brian.unipv.it>
On Tue, Jul 09, 2019 at 11:18:38PM +0200, Andrea Vai wrote:
> On 08/07/19 09:01:35, Ming Lei wrote:
> > > > > > > > >
> > > > > > > > > commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
> > > > > > > > >
> > > > > > > > > [...]
> >
> > 1) run the bcc biosnoop trace in one terminal after mounting the fs on the USB dirve
> >
> > 2) start the write test in another teminal
> >
> > 3) wait for 10 seconds, and stop the bcc trace via ctrl^ + C, then post the bcc biosnoop
> > trace log
> >
>
> Done, attached.
>
> Thanks,
> Andrea
> 5.420198000 jbd2/sdf1-8 6775 sdf W 29624320 4096 0.37
> 5.423601000 jbd2/sdf1-8 6775 sdf W 29624328 12288 3.29
> 5.424009000 jbd2/sdf1-8 6775 sdf W 29624352 4096 0.30
> 6.665732000 kworker/u8:0 6401 sdf R 10248 4096 841.64
> 6.666891000 kworker/u8:0 6401 sdf R 10256 4096 0.98
> 6.668141000 kworker/u8:0 6401 sdf R 4196352 4096 1.07
> 6.669312000 kworker/u8:0 6401 sdf R 8390656 4096 0.99
> 6.670568000 kworker/u8:0 6401 sdf R 12584960 4096 1.06
> 6.674257000 kworker/u8:0 6401 sdf R 16779264 4096 3.50
> 6.675510000 kworker/u8:0 6401 sdf R 20973568 4096 1.06
> 6.678344000 kworker/u8:0 6401 sdf R 25167872 4096 2.64
> 6.680384000 kworker/u8:0 6401 sdf R 29362176 4096 1.83
> 6.681622000 kworker/u8:0 6401 sdf R 33556480 4096 1.03
> 6.682713000 kworker/u8:0 6401 sdf R 37750784 4096 0.92
> 6.686360000 kworker/u8:0 6401 sdf R 41945088 4096 3.48
> 6.687589000 kworker/u8:0 6401 sdf R 46139392 4096 1.04
> 6.688639000 kworker/u8:0 6401 sdf R 46139424 4096 0.88
> 6.766336000 kworker/u8:0 6401 sdf W 47351808 122880 77.17
> 6.773563000 kworker/u8:0 6401 sdf W 47352048 122880 7.08
> 6.780980000 kworker/u8:0 6401 sdf W 47352288 122880 7.29
> 6.786366000 kworker/u8:0 6401 sdf W 47352528 122880 5.27
> 6.793831000 kworker/u8:0 6401 sdf W 47352768 122880 7.38
> 6.801085000 kworker/u8:0 6401 sdf W 47353008 122880 7.20
> 6.808405000 kworker/u8:0 6401 sdf W 47353248 122880 7.27
> 6.813799000 kworker/u8:0 6401 sdf W 47353488 122880 5.35
> 6.821103000 kworker/u8:0 6401 sdf W 47353728 118784 7.25
> 6.828459000 kworker/u8:0 6401 sdf W 47353960 122880 7.09
> 6.835722000 kworker/u8:0 6401 sdf W 47354200 122880 7.20
> 6.842813000 kworker/u8:0 6401 sdf W 47354440 122880 7.05
> 6.850184000 kworker/u8:0 6401 sdf W 47354680 122880 7.32
> 6.857452000 kworker/u8:0 6401 sdf W 47354920 122880 7.22
> 6.864874000 kworker/u8:0 6401 sdf W 47355160 122880 7.38
> 6.871841000 kworker/u8:0 6401 sdf W 47355400 122880 6.92
> 6.879380000 kworker/u8:0 6401 sdf W 47355640 122880 7.50
> 6.886497000 kworker/u8:0 6401 sdf W 47355880 122880 7.04
> 6.893940000 kworker/u8:0 6401 sdf W 47356120 122880 7.40
> 6.899304000 kworker/u8:0 6401 sdf W 47356360 122880 5.33
> 6.906635000 kworker/u8:0 6401 sdf W 47356600 122880 7.29
> 6.913957000 kworker/u8:0 6401 sdf W 47356840 122880 7.29
> 6.921308000 kworker/u8:0 6401 sdf W 47357080 122880 7.31
> 6.924391000 kworker/u8:0 6401 sdf W 47357320 53248 3.03
> 6.931971000 kworker/u8:0 6401 sdf W 47357424 122880 7.22
> 6.939169000 kworker/u8:0 6401 sdf W 47357664 122880 7.14
> 6.944985000 kworker/u8:0 6401 sdf W 47357904 122880 5.76
> 6.952113000 kworker/u8:0 6401 sdf W 47358144 122880 7.06
> 6.959608000 kworker/u8:0 6401 sdf W 47358384 122880 7.45
> 6.966820000 kworker/u8:0 6401 sdf W 47358624 122880 7.16
> 6.972393000 kworker/u8:0 6401 sdf W 47358864 122880 5.53
> 6.979550000 kworker/u8:0 6401 sdf W 47359104 122880 7.09
> 6.986879000 kworker/u8:0 6401 sdf W 47359344 122880 7.28
> 6.994306000 kworker/u8:0 6401 sdf W 47359584 122880 7.40
> 7.001338000 kworker/u8:0 6401 sdf W 47359824 122880 6.98
> 7.008740000 kworker/u8:0 6401 sdf W 47360064 122880 7.37
> 7.016114000 kworker/u8:0 6401 sdf W 47360304 122880 7.31
> 7.023546000 kworker/u8:0 6401 sdf W 47360544 122880 7.39
> 7.030437000 kworker/u8:0 6401 sdf W 47360784 122880 6.82
> 7.037834000 kworker/u8:0 6401 sdf W 47361024 122880 7.35
> 7.045224000 kworker/u8:0 6401 sdf W 47361264 122880 7.32
> 7.052687000 kworker/u8:0 6401 sdf W 47361504 122880 7.39
> 7.057977000 kworker/u8:0 6401 sdf W 47361744 122880 5.01
> 7.065390000 kworker/u8:0 6401 sdf W 47361984 122880 7.35
> 7.072749000 kworker/u8:0 6401 sdf W 47362224 122880 7.28
> 7.080174000 kworker/u8:0 6401 sdf W 47362464 122880 7.36
> 7.085335000 kworker/u8:0 6401 sdf W 47362704 122880 5.08
> 7.092881000 kworker/u8:0 6401 sdf W 47362944 122880 7.50
> 7.100197000 kworker/u8:0 6401 sdf W 47363184 122880 7.23
> 7.107507000 kworker/u8:0 6401 sdf W 47363424 122880 7.24
> 7.114684000 kworker/u8:0 6401 sdf W 47363664 122880 7.09
> 7.121936000 kworker/u8:0 6401 sdf W 47363904 122880 7.18
> 7.129299000 kworker/u8:0 6401 sdf W 47364144 122880 7.29
> 7.136645000 kworker/u8:0 6401 sdf W 47364384 122880 7.27
> 7.143733000 kworker/u8:0 6401 sdf W 47364624 122880 7.02
> 7.151112000 kworker/u8:0 6401 sdf W 47364864 122880 7.31
> 7.158509000 kworker/u8:0 6401 sdf W 47365104 122880 7.33
> 7.165735000 kworker/u8:0 6401 sdf W 47365344 122880 7.14
> 7.171205000 kworker/u8:0 6401 sdf W 47365584 122880 5.40
> 7.178625000 kworker/u8:0 6401 sdf W 47365824 122880 7.37
> 7.185980000 kworker/u8:0 6401 sdf W 47366064 122880 7.31
> 7.193593000 kworker/u8:0 6401 sdf W 47366304 122880 7.57
> 7.198674000 kworker/u8:0 6401 sdf W 47366544 122880 5.02
> 7.205996000 kworker/u8:0 6401 sdf W 47366784 122880 7.25
> 7.213419000 kworker/u8:0 6401 sdf W 47367024 122880 7.37
> 7.221005000 kworker/u8:0 6401 sdf W 47367264 122880 7.52
> 7.227906000 kworker/u8:0 6401 sdf W 47367504 122880 6.83
> 7.235251000 kworker/u8:0 6401 sdf W 47367744 122880 7.28
> 7.241892000 kworker/u8:0 6401 sdf W 47367984 106496 6.59
> 7.249617000 kworker/u8:0 6401 sdf W 47368192 122880 6.88
> 7.256892000 kworker/u8:0 6401 sdf W 47368432 122880 7.20
> 7.264223000 kworker/u8:0 6401 sdf W 47368672 122880 7.26
> 7.269672000 kworker/u8:0 6401 sdf W 47368912 122880 5.39
> 7.277066000 kworker/u8:0 6401 sdf W 47369152 122880 7.36
> 7.284630000 kworker/u8:0 6401 sdf W 47369392 122880 7.52
> 7.291891000 kworker/u8:0 6401 sdf W 47369632 122880 7.20
> 7.297211000 kworker/u8:0 6401 sdf W 47369872 122880 5.29
> 7.304665000 kworker/u8:0 6401 sdf W 47370112 122880 7.41
> 7.312050000 kworker/u8:0 6401 sdf W 47370352 122880 7.33
> 7.319520000 kworker/u8:0 6401 sdf W 47370592 122880 7.41
> 7.326494000 kworker/u8:0 6401 sdf W 47370832 122880 6.89
> 7.334082000 kworker/u8:0 6401 sdf W 47371072 122880 7.52
> 7.341505000 kworker/u8:0 6401 sdf W 47371312 122880 7.34
> 7.348600000 kworker/u8:0 6401 sdf W 47371552 122880 7.01
> 7.355806000 kworker/u8:0 6401 sdf W 47371792 122880 7.16
> 7.363147000 kworker/u8:0 6401 sdf W 47372032 122880 7.27
> 7.370377000 kworker/u8:0 6401 sdf W 47372272 122880 7.16
> 7.377770000 kworker/u8:0 6401 sdf W 47372512 122880 7.36
> 7.383323000 kworker/u8:0 6401 sdf W 47372752 122880 5.50
> 7.390651000 kworker/u8:0 6401 sdf W 47372992 122880 7.27
> 7.398245000 kworker/u8:0 6401 sdf W 47373232 122880 7.52
> 7.405347000 kworker/u8:0 6401 sdf W 47373472 122880 7.04
> 7.410761000 kworker/u8:0 6401 sdf W 47373712 122880 5.36
> 7.418075000 kworker/u8:0 6401 sdf W 47373952 122880 7.26
> 7.425451000 kworker/u8:0 6401 sdf W 47374192 122880 7.33
> 7.432864000 kworker/u8:0 6401 sdf W 47374432 122880 7.36
> 7.439913000 kworker/u8:0 6401 sdf W 47374672 122880 6.99
> 7.447312000 kworker/u8:0 6401 sdf W 47374912 122880 7.35
> 7.454721000 kworker/u8:0 6401 sdf W 47375152 122880 7.35
> 7.461955000 kworker/u8:0 6401 sdf W 47375392 122880 7.19
> 7.469040000 kworker/u8:0 6401 sdf W 47375632 122880 7.04
> 7.476416000 kworker/u8:0 6401 sdf W 47375872 122880 7.31
> 7.483856000 kworker/u8:0 6401 sdf W 47376112 122880 7.38
> 7.491270000 kworker/u8:0 6401 sdf W 47376352 122880 7.37
> 7.496452000 kworker/u8:0 6401 sdf W 47376592 122880 5.12
> 7.504011000 kworker/u8:0 6401 sdf W 47376832 122880 7.52
> 7.511217000 kworker/u8:0 6401 sdf W 47377072 122880 7.12
> 7.518511000 kworker/u8:0 6401 sdf W 47377312 122880 7.23
> 7.523956000 kworker/u8:0 6401 sdf W 47377552 122880 5.41
> 7.531197000 kworker/u8:0 6401 sdf W 47377792 122880 7.21
> 7.538546000 kworker/u8:0 6401 sdf W 47378032 122880 7.31
> 7.545757000 kworker/u8:0 6401 sdf W 47378272 122880 7.17
> 7.552919000 kworker/u8:0 6401 sdf W 47378512 122880 7.13
> 7.560269000 kworker/u8:0 6401 sdf W 47378752 122880 7.31
> 7.567555000 kworker/u8:0 6401 sdf W 47378992 122880 7.25
> 7.574863000 kworker/u8:0 6401 sdf W 47379232 122880 7.27
> 7.581802000 kworker/u8:0 6401 sdf W 47379472 122880 6.87
> 7.588979000 kworker/u8:0 6401 sdf W 47379712 122880 7.13
> 7.596343000 kworker/u8:0 6401 sdf W 47379952 122880 7.30
> 7.603224000 kworker/u8:0 6401 sdf W 47380192 122880 6.84
> 7.608655000 kworker/u8:0 6401 sdf W 47380432 122880 5.38
> 7.615811000 kworker/u8:0 6401 sdf W 47380672 122880 7.12
> 7.623028000 kworker/u8:0 6401 sdf W 47380912 122880 7.18
> 7.630413000 kworker/u8:0 6401 sdf W 47381152 122880 7.34
> 7.635223000 kworker/u8:0 6401 sdf W 47381392 122880 4.75
> 7.642268000 kworker/u8:0 6401 sdf W 47381632 122880 6.98
> 7.649383000 kworker/u8:0 6401 sdf W 47381872 122880 7.04
> 7.656619000 kworker/u8:0 6401 sdf W 47382112 122880 7.19
> 7.661918000 kworker/u8:0 6401 sdf W 47382352 122880 5.25
> 8.268039000 kworker/u8:0 6401 sdf W 47382592 122880 606.07
> 8.280414000 kworker/u8:0 6401 sdf W 47382832 122880 12.30
> 8.286503000 kworker/u8:0 6401 sdf W 47383072 122880 6.02
> 8.293627000 kworker/u8:0 6401 sdf W 47383312 122880 7.07
> 8.300777000 kworker/u8:0 6401 sdf W 47383552 122880 7.08
> 8.307733000 kworker/u8:0 6401 sdf W 47383792 122880 6.89
> 8.315212000 kworker/u8:0 6401 sdf W 47384032 122880 7.43
> 8.320697000 kworker/u8:0 6401 sdf W 47384272 122880 5.41
> 8.323335000 kworker/u8:0 6401 sdf W 47384512 32768 2.58
> 8.331116000 kworker/u8:0 6401 sdf W 47384576 122880 6.58
> 8.338586000 kworker/u8:0 6401 sdf W 47384816 122880 7.40
> 8.346113000 kworker/u8:0 6401 sdf W 47385056 122880 7.46
> 8.351558000 kworker/u8:0 6401 sdf W 47385296 122880 5.37
> 8.359030000 kworker/u8:0 6401 sdf W 47385536 122880 7.43
> 8.366425000 kworker/u8:0 6401 sdf W 47385776 122880 7.34
> 8.373712000 kworker/u8:0 6401 sdf W 47386016 122880 7.22
> 8.940689000 cp 6828 sdf W 47400960 122880 566.92
> 9.614647000 kworker/u8:0 6401 sdf W 47386256 122880 673.87
> 10.179120000 cp 6828 sdf W 47401200 122880 564.38
5 of 165 IO requests(1 4096 READ and 4 120K WRITE) take more than 500ms, and
most of others only needs <10ms. That is the reason why the performance
drops, and it is highly related with your USB pen drive.
biosnoop collects the trace data in blk_mq_start_request() and
blk_account_io_completion(), both happens during dispatching request to
LLD & device.
From my understanding, scsi-mq should only change the timing, or maybe
some setting is changed(still very unlikely), you may follow Alan's suggestion
to collect usbmon on 'bad' & 'good' kernel, and check if the setting is
changed.
If the initialization setting is same, maybe you can try US_FL_GO_SLOW
quirk.
Thanks,
Ming
next prev parent reply index
Thread overview: 77+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-07-02 10:46 Andrea Vai
2019-07-02 11:51 ` Johannes Thumshirn
2019-07-02 22:36 ` Andrea Vai
2019-07-03 7:29 ` Johannes Thumshirn
2019-07-03 14:23 ` Alan Stern
2019-07-06 22:06 ` Andrea Vai
2019-07-08 15:38 ` Alan Stern
2019-07-02 12:01 ` Ming Lei
2019-07-02 22:39 ` Andrea Vai
2019-07-03 2:01 ` Ming Lei
2019-07-03 5:11 ` Andrea Vai
2019-07-03 6:36 ` Ming Lei
2019-07-03 15:27 ` Chris Murphy
2019-07-06 9:33 ` Andrea Vai
2019-07-08 1:01 ` Ming Lei
2019-07-09 21:18 ` Andrea Vai
2019-07-10 2:44 ` Ming Lei [this message]
[not found] <e3f87757f7a0fdf551e911ad32fc8122eebe04c7.camel@unipv.it>
2019-08-13 19:52 ` Alan Stern
[not found] <307581a490b610c3025ee80f79a465a89d68ed19.camel@unipv.it>
2019-08-20 17:13 ` Alan Stern
2019-08-23 10:39 ` Andrea Vai
2019-08-23 20:42 ` Alan Stern
2019-08-26 6:09 ` Andrea Vai
2019-08-26 16:33 ` Alan Stern
2019-09-18 15:25 ` Andrea Vai
2019-09-18 16:30 ` Alan Stern
2019-09-19 7:33 ` Andrea Vai
2019-09-19 17:54 ` Alan Stern
2019-09-20 7:25 ` Andrea Vai
2019-09-20 7:44 ` Greg KH
2019-09-19 8:26 ` Damien Le Moal
2019-09-19 8:55 ` Ming Lei
2019-09-19 9:09 ` Damien Le Moal
2019-09-19 9:21 ` Ming Lei
2019-09-19 14:01 ` Alan Stern
2019-09-19 14:14 ` Damien Le Moal
2019-09-20 7:03 ` Andrea Vai
2019-09-25 19:30 ` Alan Stern
2019-09-25 19:36 ` Jens Axboe
2019-09-27 15:47 ` Andrea Vai
2019-11-04 16:00 ` Andrea Vai
2019-11-04 18:20 ` Alan Stern
2019-11-05 11:48 ` Andrea Vai
2019-11-05 18:31 ` Alan Stern
2019-11-05 23:29 ` Jens Axboe
2019-11-06 16:03 ` Alan Stern
2019-11-06 22:13 ` Damien Le Moal
2019-11-07 7:04 ` Andrea Vai
2019-11-07 7:54 ` Damien Le Moal
2019-11-07 18:59 ` Andrea Vai
2019-11-08 8:42 ` Damien Le Moal
2019-11-08 14:33 ` Jens Axboe
2019-11-11 10:46 ` Andrea Vai
2019-11-09 10:09 ` Ming Lei
2019-11-09 22:28 ` Ming Lei
2019-11-11 10:50 ` Andrea Vai
2019-11-11 11:05 ` Ming Lei
2019-11-11 11:13 ` Andrea Vai
2019-11-22 19:16 ` Andrea Vai
2019-11-23 7:28 ` Ming Lei
2019-11-23 15:44 ` Andrea Vai
2019-11-25 3:54 ` Ming Lei
2019-11-25 10:11 ` Andrea Vai
2019-11-25 10:29 ` Ming Lei
2019-11-25 14:58 ` Andrea Vai
2019-11-25 15:15 ` Ming Lei
2019-11-25 18:51 ` Andrea Vai
2019-11-26 2:32 ` Ming Lei
2019-11-26 7:46 ` Andrea Vai
2019-11-26 9:15 ` Ming Lei
2019-11-26 10:24 ` Ming Lei
2019-11-26 11:14 ` Andrea Vai
2019-11-27 2:05 ` Ming Lei
2019-11-27 9:39 ` Andrea Vai
2019-11-27 13:08 ` Ming Lei
2019-11-27 15:01 ` Andrea Vai
2019-11-27 0:21 ` Finn Thain
2019-11-28 17:10 ` Andrea Vai
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190710024439.GA2621@ming.t460p \
--to=ming.lei@redhat.com \
--cc=andrea.vai@unipv.it \
--cc=axboe@kernel.dk \
--cc=gregkh@linuxfoundation.org \
--cc=hare@suse.com \
--cc=himanshu.madhani@cavium.com \
--cc=linux-scsi@vger.kernel.org \
--cc=linux-usb@vger.kernel.org \
--cc=martin.petersen@oracle.com \
--cc=osandov@fb.com \
--cc=stern@rowland.harvard.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Linux-USB Archive on lore.kernel.org
Archives are clonable:
git clone --mirror https://lore.kernel.org/linux-usb/0 linux-usb/git/0.git
# If you have public-inbox 1.1+ installed, you may
# initialize and index your mirror using the following commands:
public-inbox-init -V2 linux-usb linux-usb/ https://lore.kernel.org/linux-usb \
linux-usb@vger.kernel.org
public-inbox-index linux-usb
Example config snippet for mirrors
Newsgroup available over NNTP:
nntp://nntp.lore.kernel.org/org.kernel.vger.linux-usb
AGPL code for this site: git clone https://public-inbox.org/public-inbox.git