From: Ming Lei <ming.lei@canonical.com> To: Maxim Patlasov <mpatlasov@parallels.com> Cc: Jens Axboe <axboe@kernel.dk>, Christoph Hellwig <hch@infradead.org>, Linux Kernel Mailing List <linux-kernel@vger.kernel.org>, Andrew Morton <akpm@linux-foundation.org>, Dave Kleikamp <dave.kleikamp@oracle.com>, Zach Brown <zab@zabbo.net>, Benjamin LaHaise <bcrl@kvack.org>, Kent Overstreet <kmo@daterainc.com>, AIO <linux-aio@kvack.org>, Linux FS Devel <linux-fsdevel@vger.kernel.org>, Dave Chinner <david@fromorbit.com> Subject: Re: [PATCH v1 5/9] block: loop: convert to blk-mq Date: Thu, 28 Aug 2014 10:06:23 +0800 [thread overview] Message-ID: <CACVXFVOx6n88kFhCk60YOfVa22qw1B3Gs0+Mj_ng2T5zy1Wb3A@mail.gmail.com> (raw) In-Reply-To: <53FE029B.1030200@parallels.com> On 8/28/14, Maxim Patlasov <mpatlasov@parallels.com> wrote: > On 08/21/2014 09:44 AM, Ming Lei wrote: >> On Wed, Aug 20, 2014 at 4:50 AM, Jens Axboe <axboe@kernel.dk> wrote: >> >>> >>> Reworked a bit more: >>> >>> http://git.kernel.dk/?p=linux-block.git;a=commit;h=a323185a761b9a54dc340d383695b4205ea258b6 >> One big problem of the commit is that it is basically a serialized >> workqueue >> because of single &hctx->run_work, and per-req work_struct has to be >> used for concurrent implementation. So looks the approach isn't flexible >> enough compared with doing that in driver, or any idea about how to fix >> that? >> > > I'm interested what's the price of handling requests in a separate > thread at large. I used the following fio script: > > [global] > direct=1 > bsrange=512-512 > timeout=10 > numjobs=1 > ioengine=sync > > filename=/dev/loop0 # or /dev/nullb0 > > [f1] > rw=randwrite > > to compare the performance of: > > 1) /dev/loop0 of 3.17.0-rc1 with Ming's patches applied -- 11K iops If you enable BLK_MQ_F_WQ_CONTEXT, it isn't strange to see this result since blk-mq implements a serialized workqueue. > 2) the same as above, but call loop_queue_work() directly from > loop_queue_rq() -- 270K iops > 3) /dev/nullb0 of 3.17.0-rc1 -- 380K iops In my recent investigation and discussion with Jens, using workqueue may introduce some regression for cases like loop over null_blk, tmpfs. And 270K vs. 380K is a bit similar with my result, and it was observed that context switch is increased by more than 50% with introducing workqueue. I will post V3 which will use previous kthread, with blk-mq & kernel aio, which should make full use of blk-mq and kernel aio, and won't introduce regression for cases like above. > Taking into account so big difference (11K vs. 270K), would it be worthy > to implement pure non-blocking version of aio_kernel_submit() returning > error if blocking needed? Then loop driver (or any other in-kernel user) The kernel aio submit is very similar with user space's implementation, except for block plug&unplug usage in user space aio submit path. If it is blocked in aio_kernel_submit(), you should observe similar thing with io_submit() too. Thanks, -- Ming Lei
WARNING: multiple messages have this Message-ID (diff)
From: Ming Lei <ming.lei@canonical.com> To: Maxim Patlasov <mpatlasov@parallels.com> Cc: Jens Axboe <axboe@kernel.dk>, Christoph Hellwig <hch@infradead.org>, Linux Kernel Mailing List <linux-kernel@vger.kernel.org>, Andrew Morton <akpm@linux-foundation.org>, Dave Kleikamp <dave.kleikamp@oracle.com>, Zach Brown <zab@zabbo.net>, Benjamin LaHaise <bcrl@kvack.org>, Kent Overstreet <kmo@daterainc.com>, AIO <linux-aio@kvack.org>, Linux FS Devel <linux-fsdevel@vger.kernel.org>, Dave Chinner <david@fromorbit.com> Subject: Re: [PATCH v1 5/9] block: loop: convert to blk-mq Date: Thu, 28 Aug 2014 10:06:23 +0800 [thread overview] Message-ID: <CACVXFVOx6n88kFhCk60YOfVa22qw1B3Gs0+Mj_ng2T5zy1Wb3A@mail.gmail.com> (raw) In-Reply-To: <53FE029B.1030200@parallels.com> On 8/28/14, Maxim Patlasov <mpatlasov@parallels.com> wrote: > On 08/21/2014 09:44 AM, Ming Lei wrote: >> On Wed, Aug 20, 2014 at 4:50 AM, Jens Axboe <axboe@kernel.dk> wrote: >> >>> >>> Reworked a bit more: >>> >>> http://git.kernel.dk/?p=linux-block.git;a=commit;h=a323185a761b9a54dc340d383695b4205ea258b6 >> One big problem of the commit is that it is basically a serialized >> workqueue >> because of single &hctx->run_work, and per-req work_struct has to be >> used for concurrent implementation. So looks the approach isn't flexible >> enough compared with doing that in driver, or any idea about how to fix >> that? >> > > I'm interested what's the price of handling requests in a separate > thread at large. I used the following fio script: > > [global] > direct=1 > bsrange=512-512 > timeout=10 > numjobs=1 > ioengine=sync > > filename=/dev/loop0 # or /dev/nullb0 > > [f1] > rw=randwrite > > to compare the performance of: > > 1) /dev/loop0 of 3.17.0-rc1 with Ming's patches applied -- 11K iops If you enable BLK_MQ_F_WQ_CONTEXT, it isn't strange to see this result since blk-mq implements a serialized workqueue. > 2) the same as above, but call loop_queue_work() directly from > loop_queue_rq() -- 270K iops > 3) /dev/nullb0 of 3.17.0-rc1 -- 380K iops In my recent investigation and discussion with Jens, using workqueue may introduce some regression for cases like loop over null_blk, tmpfs. And 270K vs. 380K is a bit similar with my result, and it was observed that context switch is increased by more than 50% with introducing workqueue. I will post V3 which will use previous kthread, with blk-mq & kernel aio, which should make full use of blk-mq and kernel aio, and won't introduce regression for cases like above. > Taking into account so big difference (11K vs. 270K), would it be worthy > to implement pure non-blocking version of aio_kernel_submit() returning > error if blocking needed? Then loop driver (or any other in-kernel user) The kernel aio submit is very similar with user space's implementation, except for block plug&unplug usage in user space aio submit path. If it is blocked in aio_kernel_submit(), you should observe similar thing with io_submit() too. Thanks, -- Ming Lei -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to majordomo@kvack.org. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>
next prev parent reply other threads:[~2014-08-28 2:06 UTC|newest] Thread overview: 75+ messages / expand[flat|nested] mbox.gz Atom feed top 2014-08-14 15:50 [PATCH v1 0/9] block & aio: kernel aio and loop mq conversion Ming Lei 2014-08-14 15:50 ` Ming Lei 2014-08-14 15:50 ` [PATCH v1 1/9] aio: add aio_kernel_() interface Ming Lei 2014-08-14 15:50 ` Ming Lei 2014-08-14 18:07 ` Zach Brown 2014-08-14 18:07 ` Zach Brown 2014-08-15 13:20 ` Ming Lei 2014-08-15 13:20 ` Ming Lei 2014-08-14 15:50 ` [PATCH v1 2/9] fd/direct-io: introduce should_dirty for kernel aio Ming Lei 2014-08-14 15:50 ` Ming Lei 2014-08-14 15:50 ` [PATCH v1 3/9] blk-mq: export blk_mq_freeze_queue and blk_mq_unfreeze_queue Ming Lei 2014-08-14 15:50 ` Ming Lei 2014-08-14 15:50 ` [PATCH v1 4/9] blk-mq: introduce init_flush_rq_fn callback in 'blk_mq_ops' Ming Lei 2014-08-14 15:50 ` Ming Lei 2014-08-15 16:19 ` Jens Axboe 2014-08-15 16:19 ` Jens Axboe 2014-08-16 7:49 ` Ming Lei 2014-08-16 7:49 ` Ming Lei 2014-08-17 18:39 ` Jens Axboe 2014-08-17 18:39 ` Jens Axboe 2014-08-14 15:50 ` [PATCH v1 5/9] block: loop: convert to blk-mq Ming Lei 2014-08-14 15:50 ` Ming Lei 2014-08-15 16:31 ` Christoph Hellwig 2014-08-15 16:31 ` Christoph Hellwig 2014-08-15 16:36 ` Jens Axboe 2014-08-15 16:46 ` Jens Axboe 2014-08-16 8:06 ` Ming Lei 2014-08-16 8:06 ` Ming Lei 2014-08-17 17:48 ` Jens Axboe 2014-08-17 17:48 ` Jens Axboe 2014-08-18 1:22 ` Ming Lei 2014-08-18 1:22 ` Ming Lei 2014-08-18 11:53 ` Ming Lei 2014-08-18 11:53 ` Ming Lei 2014-08-19 20:50 ` Jens Axboe 2014-08-20 1:23 ` Ming Lei 2014-08-20 16:09 ` Jens Axboe 2014-08-21 2:54 ` Ming Lei 2014-08-21 2:58 ` Jens Axboe 2014-08-21 3:13 ` Ming Lei 2014-08-21 3:15 ` Ming Lei 2014-08-21 3:16 ` Jens Axboe 2014-08-21 3:34 ` Ming Lei 2014-08-21 5:44 ` Ming Lei 2014-08-27 16:08 ` Maxim Patlasov 2014-08-27 16:08 ` Maxim Patlasov 2014-08-27 16:29 ` Benjamin LaHaise 2014-08-27 16:29 ` Benjamin LaHaise 2014-08-27 17:19 ` Maxim Patlasov 2014-08-27 17:19 ` Maxim Patlasov 2014-08-27 17:56 ` Zach Brown 2014-08-27 17:56 ` Zach Brown 2014-08-28 2:10 ` Ming Lei 2014-08-28 2:10 ` Ming Lei 2014-08-28 2:06 ` Ming Lei [this message] 2014-08-28 2:06 ` Ming Lei 2014-08-29 11:14 ` Maxim Patlasov 2014-08-29 11:14 ` Maxim Patlasov 2014-08-14 15:50 ` [PATCH v1 6/9] block: loop: say goodby to bio Ming Lei 2014-08-14 15:50 ` Ming Lei 2014-08-14 15:50 ` [PATCH v1 7/9] block: loop: introduce lo_discard() and lo_req_flush() Ming Lei 2014-08-14 15:50 ` Ming Lei 2014-08-14 15:50 ` [PATCH v1 8/9] block: loop: don't handle REQ_FUA explicitly Ming Lei 2014-08-14 15:50 ` Ming Lei 2014-08-14 15:50 ` [PATCH v1 9/9] block: loop: support to submit I/O via kernel aio based Ming Lei 2014-08-14 15:50 ` Ming Lei 2014-08-14 16:53 ` [PATCH v1 0/9] block & aio: kernel aio and loop mq conversion Jens Axboe 2014-08-14 16:53 ` Jens Axboe 2014-08-15 12:59 ` Ming Lei 2014-08-15 12:59 ` Ming Lei 2014-08-15 13:11 ` Christoph Hellwig 2014-08-15 13:11 ` Christoph Hellwig 2014-08-15 14:32 ` Ming Lei 2014-08-15 14:32 ` Ming Lei 2014-08-29 10:41 [PATCH v1 5/9] block: loop: convert to blk-mq Maxim Patlasov
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=CACVXFVOx6n88kFhCk60YOfVa22qw1B3Gs0+Mj_ng2T5zy1Wb3A@mail.gmail.com \ --to=ming.lei@canonical.com \ --cc=akpm@linux-foundation.org \ --cc=axboe@kernel.dk \ --cc=bcrl@kvack.org \ --cc=dave.kleikamp@oracle.com \ --cc=david@fromorbit.com \ --cc=hch@infradead.org \ --cc=kmo@daterainc.com \ --cc=linux-aio@kvack.org \ --cc=linux-fsdevel@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=mpatlasov@parallels.com \ --cc=zab@zabbo.net \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.