From: Ming Lei <ming.lei@redhat.com>
To: Andrea Vai <andrea.vai@unipv.it>
Cc: "Schmid, Carsten" <Carsten_Schmid@mentor.com>,
Finn Thain <fthain@telegraphics.com.au>,
Damien Le Moal <Damien.LeMoal@wdc.com>,
Alan Stern <stern@rowland.harvard.edu>,
Jens Axboe <axboe@kernel.dk>,
Johannes Thumshirn <jthumshirn@suse.de>,
USB list <linux-usb@vger.kernel.org>,
SCSI development list <linux-scsi@vger.kernel.org>,
Himanshu Madhani <himanshu.madhani@cavium.com>,
Hannes Reinecke <hare@suse.com>, Omar Sandoval <osandov@fb.com>,
"Martin K. Petersen" <martin.petersen@oracle.com>,
Greg KH <gregkh@linuxfoundation.org>,
Hans Holmberg <Hans.Holmberg@wdc.com>,
Kernel development list <linux-kernel@vger.kernel.org>,
linux-ext4@vger.kernel.org, linux-fsdevel@vger.kernel.org,
Theodore Ts'o <tytso@mit.edu>
Subject: Re: AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6
Date: Tue, 10 Dec 2019 16:05:50 +0800 [thread overview]
Message-ID: <20191210080550.GA5699@ming.t460p> (raw)
In-Reply-To: <8196b014b1a4d91169bf3b0d68905109aeaf2191.camel@unipv.it>
On Tue, Dec 10, 2019 at 08:35:43AM +0100, Andrea Vai wrote:
> Il giorno mar, 03/12/2019 alle 10.23 +0800, Ming Lei ha scritto:
> > On Fri, Nov 29, 2019 at 03:41:01PM +0100, Andrea Vai wrote:
> > > Il giorno ven, 29/11/2019 alle 10.35 +0800, Ming Lei ha scritto:
> > > > On Fri, Nov 29, 2019 at 08:57:34AM +0800, Ming Lei wrote:
> > > >
> > > > > [...]
> > > >
> > > > > Andrea, can you collect the following log when running the
> > test
> > > > > on current new(bad) kernel?
> > > > >
> > > > > /usr/share/bcc/tools/stackcount -K
> > blk_mq_make_request
> > > >
> > > > Instead, please run the following trace, given insert may be
> > > > called from other paths, such as flush plug:
> > > >
> > > > /usr/share/bcc/tools/stackcount -K t:block:block_rq_insert
> > >
> > > Attached, for new (patched) bad kernel.
> > >
> > > Produced by: start the trace script (with the pendrive already
> > > plugged), wait some seconds, run the test (1 trial, 1 GB), wait
> > for
> > > the test to finish, stop the trace.
> > >
> > > The copy took ~1700 seconds.
> >
> > See the two path[1][2] of inserting request, and path[1] is
> > triggered
> > 4358 times, and the path[2] is triggered 5763 times.
> >
> > The path[2] is expected behaviour. Not sure path [1] is correct,
> > given
> > ext4_release_file() is supposed to be called when this inode is
> > released. That means the file is closed 4358 times during 1GB file
> > copying to usb storage.
> >
> > Cc filesystem list.
> >
> >
> > [1] insert requests when returning to user mode from syscall
> >
> > b'blk_mq_sched_request_inserted'
> > b'blk_mq_sched_request_inserted'
> > b'dd_insert_requests'
> > b'blk_mq_sched_insert_requests'
> > b'blk_mq_flush_plug_list'
> > b'blk_flush_plug_list'
> > b'io_schedule_prepare'
> > b'io_schedule'
> > b'rq_qos_wait'
> > b'wbt_wait'
> > b'__rq_qos_throttle'
> > b'blk_mq_make_request'
> > b'generic_make_request'
> > b'submit_bio'
> > b'ext4_io_submit'
> > b'ext4_writepages'
> > b'do_writepages'
> > b'__filemap_fdatawrite_range'
> > b'ext4_release_file'
> > b'__fput'
> > b'task_work_run'
> > b'exit_to_usermode_loop'
> > b'do_syscall_64'
> > b'entry_SYSCALL_64_after_hwframe'
> > 4358
> >
> > [2] insert requests from writeback wq context
> >
> > b'blk_mq_sched_request_inserted'
> > b'blk_mq_sched_request_inserted'
> > b'dd_insert_requests'
> > b'blk_mq_sched_insert_requests'
> > b'blk_mq_flush_plug_list'
> > b'blk_flush_plug_list'
> > b'io_schedule_prepare'
> > b'io_schedule'
> > b'rq_qos_wait'
> > b'wbt_wait'
> > b'__rq_qos_throttle'
> > b'blk_mq_make_request'
> > b'generic_make_request'
> > b'submit_bio'
> > b'ext4_io_submit'
> > b'ext4_bio_write_page'
> > b'mpage_submit_page'
> > b'mpage_process_page_bufs'
> > b'mpage_prepare_extent_to_map'
> > b'ext4_writepages'
> > b'do_writepages'
> > b'__writeback_single_inode'
> > b'writeback_sb_inodes'
> > b'__writeback_inodes_wb'
> > b'wb_writeback'
> > b'wb_workfn'
> > b'process_one_work'
> > b'worker_thread'
> > b'kthread'
> > b'ret_from_fork'
> > 5763
> >
> > Thanks,
> > Ming
> >
>
> Is there any update on this? Sorry if I am making noise, but I would
> like to help to improve the kernel (or fix it) if I can help.
> Otherwise, please let me know how to consider this case,
IMO, the extra write path from exit_to_usermode_loop() isn't expected,
that should be the reason why write IO order is changed, then performance
drops on your USB storage.
We need our fs/ext4 experts to take a look.
Or can you reproduce the issue on xfs or btrfs?
Thanks,
Ming
next prev parent reply other threads:[~2019-12-10 8:06 UTC|newest]
Thread overview: 102+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <307581a490b610c3025ee80f79a465a89d68ed19.camel@unipv.it>
2019-08-20 17:13 ` Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6 Alan Stern
2019-08-23 10:39 ` Andrea Vai
2019-08-23 20:42 ` Alan Stern
2019-08-26 6:09 ` Andrea Vai
2019-08-26 16:33 ` Alan Stern
2019-09-18 15:25 ` Andrea Vai
2019-09-18 16:30 ` Alan Stern
2019-09-19 7:33 ` Andrea Vai
2019-09-19 17:54 ` Alan Stern
2019-09-20 7:25 ` Andrea Vai
2019-09-20 7:44 ` Greg KH
2019-09-19 8:26 ` Damien Le Moal
2019-09-19 8:55 ` Ming Lei
2019-09-19 9:09 ` Damien Le Moal
2019-09-19 9:21 ` Ming Lei
2019-09-19 14:01 ` Alan Stern
2019-09-19 14:14 ` Damien Le Moal
2019-09-20 7:03 ` Andrea Vai
2019-09-25 19:30 ` Alan Stern
2019-09-25 19:36 ` Jens Axboe
2019-09-27 15:47 ` Andrea Vai
2019-11-04 16:00 ` Andrea Vai
2019-11-04 18:20 ` Alan Stern
2019-11-05 11:48 ` Andrea Vai
2019-11-05 18:31 ` Alan Stern
2019-11-05 23:29 ` Jens Axboe
2019-11-06 16:03 ` Alan Stern
2019-11-06 22:13 ` Damien Le Moal
2019-11-07 7:04 ` Andrea Vai
2019-11-07 7:54 ` Damien Le Moal
2019-11-07 18:59 ` Andrea Vai
2019-11-08 8:42 ` Damien Le Moal
2019-11-08 14:33 ` Jens Axboe
2019-11-11 10:46 ` Andrea Vai
2019-11-09 10:09 ` Ming Lei
2019-11-09 22:28 ` Ming Lei
2019-11-11 10:50 ` Andrea Vai
2019-11-11 11:05 ` Ming Lei
2019-11-11 11:13 ` Andrea Vai
2019-11-22 19:16 ` Andrea Vai
2019-11-23 7:28 ` Ming Lei
2019-11-23 15:44 ` Andrea Vai
2019-11-25 3:54 ` Ming Lei
2019-11-25 10:11 ` Andrea Vai
2019-11-25 10:29 ` Ming Lei
2019-11-25 14:58 ` Andrea Vai
2019-11-25 15:15 ` Ming Lei
2019-11-25 18:51 ` Andrea Vai
2019-11-26 2:32 ` Ming Lei
2019-11-26 7:46 ` Andrea Vai
2019-11-26 9:15 ` Ming Lei
2019-11-26 10:24 ` Ming Lei
2019-11-26 11:14 ` Andrea Vai
2019-11-27 2:05 ` Ming Lei
2019-11-27 9:39 ` Andrea Vai
2019-11-27 13:08 ` Ming Lei
2019-11-27 15:01 ` Andrea Vai
2019-11-27 0:21 ` Finn Thain
2019-11-27 8:14 ` AW: " Schmid, Carsten
2019-11-27 21:49 ` Finn Thain
2019-11-28 7:46 ` Andrea Vai
2019-11-28 8:12 ` AW: " Schmid, Carsten
2019-11-28 11:40 ` Andrea Vai
2019-11-28 17:39 ` Alan Stern
2019-11-28 9:17 ` Ming Lei
2019-11-28 17:34 ` Andrea Vai
2019-11-29 0:57 ` Ming Lei
2019-11-29 2:35 ` Ming Lei
2019-11-29 14:41 ` Andrea Vai
2019-12-03 2:23 ` Ming Lei
2019-12-10 7:35 ` Andrea Vai
2019-12-10 8:05 ` Ming Lei [this message]
2019-12-11 2:41 ` Theodore Y. Ts'o
2019-12-11 4:00 ` Ming Lei
2019-12-11 16:07 ` Theodore Y. Ts'o
2019-12-11 21:33 ` Ming Lei
2019-12-12 7:34 ` Andrea Vai
2019-12-18 8:25 ` Andrea Vai
2019-12-18 9:48 ` Ming Lei
[not found] ` <b1b6a0e9d690ecd9432025acd2db4ac09f834040.camel@unipv.it>
2019-12-23 13:08 ` Ming Lei
2019-12-23 14:02 ` Andrea Vai
2019-12-24 1:32 ` Ming Lei
2019-12-24 8:04 ` Andrea Vai
2019-12-24 8:47 ` Ming Lei
2019-12-23 16:26 ` Theodore Y. Ts'o
2019-12-23 16:29 ` Andrea Vai
2019-12-23 17:22 ` Theodore Y. Ts'o
2019-12-23 18:45 ` Andrea Vai
2019-12-23 19:53 ` Theodore Y. Ts'o
2019-12-24 1:27 ` Ming Lei
2019-12-24 6:49 ` Andrea Vai
2019-12-24 8:51 ` Andrea Vai
2019-12-24 9:35 ` Ming Lei
2019-12-25 5:17 ` Theodore Y. Ts'o
2019-12-26 2:27 ` Ming Lei
2019-12-26 3:30 ` Theodore Y. Ts'o
2019-12-26 8:37 ` Ming Lei
2020-01-07 7:51 ` Andrea Vai
[not found] ` <20200101074310.10904-1-hdanton@sina.com>
2020-01-01 13:53 ` slow IO on USB media Ming Lei
2019-11-29 11:44 ` AW: Slow I/O on USB media after commit f664a3cc17b7d0a2bc3b3ab96181e1029b0ec0e6 Bernd Schubert
2019-12-02 7:01 ` Andrea Vai
2019-11-28 17:10 ` Andrea Vai
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20191210080550.GA5699@ming.t460p \
--to=ming.lei@redhat.com \
--cc=Carsten_Schmid@mentor.com \
--cc=Damien.LeMoal@wdc.com \
--cc=Hans.Holmberg@wdc.com \
--cc=andrea.vai@unipv.it \
--cc=axboe@kernel.dk \
--cc=fthain@telegraphics.com.au \
--cc=gregkh@linuxfoundation.org \
--cc=hare@suse.com \
--cc=himanshu.madhani@cavium.com \
--cc=jthumshirn@suse.de \
--cc=linux-ext4@vger.kernel.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
--cc=linux-usb@vger.kernel.org \
--cc=martin.petersen@oracle.com \
--cc=osandov@fb.com \
--cc=stern@rowland.harvard.edu \
--cc=tytso@mit.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).