From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-it1-f193.google.com ([209.85.166.193]:51634 "EHLO mail-it1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726679AbeLRPnK (ORCPT ); Tue, 18 Dec 2018 10:43:10 -0500 Received: by mail-it1-f193.google.com with SMTP id w18so4472571ite.1 for ; Tue, 18 Dec 2018 07:43:09 -0800 (PST) From: Jens Axboe To: linux-fsdevel@vger.kernel.org, linux-aio@kvack.org Cc: hch@lst.de, viro@zeniv.linux.org.uk, Jens Axboe Subject: [PATCH 17/22] block: implement bio helper to add iter bvec pages to bio Date: Tue, 18 Dec 2018 08:42:25 -0700 Message-Id: <20181218154230.3120-18-axboe@kernel.dk> In-Reply-To: <20181218154230.3120-1-axboe@kernel.dk> References: <20181218154230.3120-1-axboe@kernel.dk> Sender: linux-fsdevel-owner@vger.kernel.org List-ID: For an ITER_BVEC, we can just iterate the iov and add the pages to the bio directly. The current two callers of bio_iov_iter_get_pages() are updated to check if they need to release pages on completion. This makes them work with bvecs that contain kernel mapped pages already. Signed-off-by: Jens Axboe --- block/bio.c | 50 ++++++++++++++++++++++++++++++++++++++++++-------- fs/block_dev.c | 5 +++-- fs/iomap.c | 5 +++-- 3 files changed, 48 insertions(+), 12 deletions(-) diff --git a/block/bio.c b/block/bio.c index a475949afd70..398412d46bb1 100644 --- a/block/bio.c +++ b/block/bio.c @@ -828,6 +828,20 @@ int bio_add_page(struct bio *bio, struct page *page, } EXPORT_SYMBOL(bio_add_page); +static int __bio_iov_bvec_add_pages(struct bio *bio, struct iov_iter *iter) +{ + const struct bio_vec *bv = iter->bvec + iter->iov_offset; + size_t size; + + size = bio_add_page(bio, bv->bv_page, bv->bv_len, bv->bv_offset); + if (size == bv->bv_len) { + iov_iter_advance(iter, size); + return 0; + } + + return -EINVAL; +} + #define PAGE_PTRS_PER_BVEC (sizeof(struct bio_vec) / sizeof(struct page *)) /** @@ -876,23 +890,43 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) } /** - * bio_iov_iter_get_pages - pin user or kernel pages and add them to a bio + * bio_iov_iter_get_pages - add user or kernel pages to a bio * @bio: bio to add pages to - * @iter: iov iterator describing the region to be mapped + * @iter: iov iterator describing the region to be added + * + * This takes either an iterator pointing to user memory, or one pointing to + * kernel pages (BVEC iterator). If we're adding user pages, we pin them and + * map them into the kernel. On IO completion, the caller should put those + * pages. If we're adding kernel pages, we just have to add the pages to the + * bio directly. We don't grab an extra reference to those pages (the user + * should already have that), and we don't put the page on IO completion. + * The caller needs to check if the bio is flagged BIO_HOLD_PAGES on IO + * completion. If it isn't, then pages should be released. * - * Pins pages from *iter and appends them to @bio's bvec array. The - * pages will have to be released using put_page() when done. * The function tries, but does not guarantee, to pin as many pages as - * fit into the bio, or are requested in *iter, whatever is smaller. - * If MM encounters an error pinning the requested pages, it stops. - * Error is returned only if 0 pages could be pinned. + * fit into the bio, or are requested in *iter, whatever is smaller. If + * MM encounters an error pinning the requested pages, it stops. Error + * is returned only if 0 pages could be pinned. */ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) { + const bool is_bvec = iov_iter_is_bvec(iter); unsigned short orig_vcnt = bio->bi_vcnt; + /* + * If this is a BVEC iter, then the pages are kernel pages. Don't + * release them on IO completion. + */ + if (is_bvec) + bio_set_flag(bio, BIO_HOLD_PAGES); + do { - int ret = __bio_iov_iter_get_pages(bio, iter); + int ret; + + if (is_bvec) + ret = __bio_iov_bvec_add_pages(bio, iter); + else + ret = __bio_iov_iter_get_pages(bio, iter); if (unlikely(ret)) return bio->bi_vcnt > orig_vcnt ? 0 : ret; diff --git a/fs/block_dev.c b/fs/block_dev.c index b8f574615792..cbf846706f57 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -326,8 +326,9 @@ static void blkdev_bio_end_io(struct bio *bio) struct bio_vec *bvec; int i; - bio_for_each_segment_all(bvec, bio, i) - put_page(bvec->bv_page); + if (!bio_flagged(bio, BIO_HOLD_PAGES)) + bio_for_each_segment_all(bvec, bio, i) + put_page(bvec->bv_page); bio_put(bio); } } diff --git a/fs/iomap.c b/fs/iomap.c index 199324826da6..ded89f5f2e79 100644 --- a/fs/iomap.c +++ b/fs/iomap.c @@ -1580,8 +1580,9 @@ static void iomap_dio_bio_end_io(struct bio *bio) struct bio_vec *bvec; int i; - bio_for_each_segment_all(bvec, bio, i) - put_page(bvec->bv_page); + if (!bio_flagged(bio, BIO_HOLD_PAGES)) + bio_for_each_segment_all(bvec, bio, i) + put_page(bvec->bv_page); bio_put(bio); } } -- 2.17.1