From: Pavel Begunkov <asml.silence@gmail.com>
To: linux-block@vger.kernel.org
Cc: Jens Axboe <axboe@kernel.dk>, Pavel Begunkov <asml.silence@gmail.com>
Subject: [PATCH 12/16] block: add single bio async direct IO helper
Date: Tue, 19 Oct 2021 22:24:21 +0100 [thread overview]
Message-ID: <c8d2d919894fd0112f21723a9cb50b6c7cbd9613.1634676157.git.asml.silence@gmail.com> (raw)
In-Reply-To: <cover.1634676157.git.asml.silence@gmail.com>
As with __blkdev_direct_IO_simple(), we can implement direct IO more
efficiently if there is only one bio. Add __blkdev_direct_IO_async() and
blkdev_bio_end_io_async(). This patch brings me from 4.45-4.5 MIOPS with
nullblk to 4.7+.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
block/fops.c | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 87 insertions(+), 3 deletions(-)
diff --git a/block/fops.c b/block/fops.c
index 7cf98db0595a..0f1332374756 100644
--- a/block/fops.c
+++ b/block/fops.c
@@ -305,6 +305,88 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter,
return ret;
}
+static void blkdev_bio_end_io_async(struct bio *bio)
+{
+ struct blkdev_dio *dio = container_of(bio, struct blkdev_dio, bio);
+ struct kiocb *iocb = dio->iocb;
+ ssize_t ret;
+
+ if (likely(!bio->bi_status)) {
+ ret = dio->size;
+ iocb->ki_pos += ret;
+ } else {
+ ret = blk_status_to_errno(bio->bi_status);
+ }
+
+ iocb->ki_complete(iocb, ret, 0);
+
+ if (dio->flags & DIO_SHOULD_DIRTY) {
+ bio_check_pages_dirty(bio);
+ } else {
+ bio_release_pages(bio, false);
+ bio_put(bio);
+ }
+}
+
+static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb,
+ struct iov_iter *iter,
+ unsigned int nr_pages)
+{
+ struct block_device *bdev = iocb->ki_filp->private_data;
+ struct blkdev_dio *dio;
+ struct bio *bio;
+ loff_t pos = iocb->ki_pos;
+ int ret = 0;
+
+ if ((pos | iov_iter_alignment(iter)) &
+ (bdev_logical_block_size(bdev) - 1))
+ return -EINVAL;
+
+ bio = bio_alloc_kiocb(iocb, nr_pages, &blkdev_dio_pool);
+ dio = container_of(bio, struct blkdev_dio, bio);
+ __bio_set_dev(bio, bdev);
+ bio->bi_iter.bi_sector = pos >> 9;
+ bio->bi_write_hint = iocb->ki_hint;
+ bio->bi_end_io = blkdev_bio_end_io_async;
+ bio->bi_ioprio = iocb->ki_ioprio;
+ dio->flags = 0;
+ dio->iocb = iocb;
+
+ ret = bio_iov_iter_get_pages(bio, iter);
+ if (unlikely(ret)) {
+ bio->bi_status = BLK_STS_IOERR;
+ bio_endio(bio);
+ return BLK_STS_IOERR;
+ }
+ dio->size = bio->bi_iter.bi_size;
+
+ if (iov_iter_rw(iter) == READ) {
+ bio->bi_opf = REQ_OP_READ;
+ if (iter_is_iovec(iter)) {
+ dio->flags |= DIO_SHOULD_DIRTY;
+ bio_set_pages_dirty(bio);
+ }
+ } else {
+ bio->bi_opf = dio_bio_write_op(iocb);
+ task_io_account_write(bio->bi_iter.bi_size);
+ }
+
+ if (iocb->ki_flags & IOCB_NOWAIT)
+ bio->bi_opf |= REQ_NOWAIT;
+ /*
+ * Don't plug for HIPRI/polled IO, as those should go straight
+ * to issue
+ */
+ if (iocb->ki_flags & IOCB_HIPRI) {
+ bio_set_polled(bio, iocb);
+ submit_bio(bio);
+ WRITE_ONCE(iocb->private, bio);
+ } else {
+ submit_bio(bio);
+ }
+ return -EIOCBQUEUED;
+}
+
static ssize_t blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter)
{
unsigned int nr_pages;
@@ -313,9 +395,11 @@ static ssize_t blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter)
return 0;
nr_pages = bio_iov_vecs_to_alloc(iter, BIO_MAX_VECS + 1);
- if (is_sync_kiocb(iocb) && nr_pages <= BIO_MAX_VECS)
- return __blkdev_direct_IO_simple(iocb, iter, nr_pages);
-
+ if (likely(nr_pages <= BIO_MAX_VECS)) {
+ if (is_sync_kiocb(iocb))
+ return __blkdev_direct_IO_simple(iocb, iter, nr_pages);
+ return __blkdev_direct_IO_async(iocb, iter, nr_pages);
+ }
return __blkdev_direct_IO(iocb, iter, bio_max_segs(nr_pages));
}
--
2.33.1
next prev parent reply other threads:[~2021-10-19 21:24 UTC|newest]
Thread overview: 57+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-10-19 21:24 [PATCH 00/16] block optimisation round Pavel Begunkov
2021-10-19 21:24 ` [PATCH 01/16] block: turn macro helpers into inline functions Pavel Begunkov
2021-10-20 6:09 ` Christoph Hellwig
2021-10-19 21:24 ` [PATCH 02/16] block: convert leftovers to bdev_get_queue Pavel Begunkov
2021-10-19 22:34 ` Chaitanya Kulkarni
2021-10-19 21:24 ` [PATCH 03/16] block: optimise req_bio_endio() Pavel Begunkov
2021-10-20 6:11 ` Christoph Hellwig
2021-10-22 9:58 ` Shinichiro Kawasaki
2021-10-22 10:58 ` Pavel Begunkov
2021-10-19 21:24 ` [PATCH 04/16] block: don't bloat enter_queue with percpu_ref Pavel Begunkov
2021-10-19 22:32 ` Chaitanya Kulkarni
2021-10-20 6:12 ` Christoph Hellwig
2021-10-20 12:08 ` Pavel Begunkov
2021-10-19 21:24 ` [PATCH 05/16] block: inline a part of bio_release_pages() Pavel Begunkov
2021-10-20 6:13 ` Christoph Hellwig
2021-10-20 12:19 ` Pavel Begunkov
2021-10-20 14:15 ` Jens Axboe
2021-10-20 17:29 ` Christoph Hellwig
2021-10-19 21:24 ` [PATCH 06/16] block: clean up blk_mq_submit_bio() merging Pavel Begunkov
2021-10-20 6:16 ` Christoph Hellwig
2021-10-20 12:20 ` Pavel Begunkov
2021-10-19 21:24 ` [PATCH 07/16] blocK: move plug flush functions to blk-mq.c Pavel Begunkov
2021-10-19 22:34 ` Chaitanya Kulkarni
2021-10-20 6:17 ` Christoph Hellwig
2021-10-20 12:23 ` Pavel Begunkov
2021-10-20 12:37 ` Christoph Hellwig
2021-10-20 13:18 ` Pavel Begunkov
2021-10-19 21:24 ` [PATCH 08/16] block: optimise blk_flush_plug_list Pavel Begunkov
2021-10-20 6:29 ` Christoph Hellwig
2021-10-20 12:26 ` Pavel Begunkov
2021-10-19 21:24 ` [PATCH 09/16] block: optimise boundary blkdev_read_iter's checks Pavel Begunkov
2021-10-20 6:29 ` Christoph Hellwig
2021-10-19 21:24 ` [PATCH 10/16] block: optimise blkdev_bio_end_io() Pavel Begunkov
2021-10-20 6:30 ` Christoph Hellwig
2021-10-20 12:29 ` Pavel Begunkov
2021-10-19 21:24 ` [PATCH 11/16] block: add optimised version bio_set_dev() Pavel Begunkov
2021-10-19 22:36 ` Chaitanya Kulkarni
2021-10-20 6:20 ` Christoph Hellwig
2021-10-20 12:29 ` Pavel Begunkov
2021-10-19 21:24 ` Pavel Begunkov [this message]
2021-10-20 6:36 ` [PATCH 12/16] block: add single bio async direct IO helper Christoph Hellwig
2021-10-20 12:35 ` Pavel Begunkov
2021-10-19 21:24 ` [PATCH 13/16] block: add async version of bio_set_polled Pavel Begunkov
2021-10-20 6:37 ` Christoph Hellwig
2021-10-20 12:58 ` Pavel Begunkov
2021-10-19 21:24 ` [PATCH 14/16] block: skip advance when async and not needed Pavel Begunkov
2021-10-20 6:41 ` Christoph Hellwig
2021-10-19 21:24 ` [PATCH 15/16] block: optimise blk_may_split for normal rw Pavel Begunkov
2021-10-20 6:25 ` Christoph Hellwig
2021-10-20 13:38 ` Pavel Begunkov
2021-10-19 21:24 ` [PATCH 16/16] block: optimise submit_bio_checks " Pavel Begunkov
2021-10-20 6:26 ` Christoph Hellwig
2021-10-19 23:31 ` [PATCH 00/16] block optimisation round Jens Axboe
2021-10-20 0:21 ` Jens Axboe
2021-10-20 0:22 ` Jens Axboe
2021-10-20 14:12 ` (subset) " Jens Axboe
2021-10-20 14:54 ` Pavel Begunkov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=c8d2d919894fd0112f21723a9cb50b6c7cbd9613.1634676157.git.asml.silence@gmail.com \
--to=asml.silence@gmail.com \
--cc=axboe@kernel.dk \
--cc=linux-block@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).