From: Sagi Grimberg <sagi@grimberg.me>
To: Ming Lei <ming.lei@redhat.com>
Cc: Christoph Hellwig <hch@lst.de>, Jens Axboe <axboe@kernel.dk>,
linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-mm@kvack.org, Dave Chinner <dchinner@redhat.com>,
Kent Overstreet <kent.overstreet@gmail.com>,
Mike Snitzer <snitzer@redhat.com>,
dm-devel@redhat.com, Alexander Viro <viro@zeniv.linux.org.uk>,
linux-fsdevel@vger.kernel.org, Shaohua Li <shli@kernel.org>,
linux-raid@vger.kernel.org, linux-erofs@lists.ozlabs.org,
David Sterba <dsterba@suse.com>,
linux-btrfs@vger.kernel.org,
"Darrick J . Wong" <darrick.wong@oracle.com>,
linux-xfs@vger.kernel.org, Gao Xiang <gaoxiang25@huawei.com>,
Theodore Ts'o <tytso@mit.edu>,
linux-ext4@vger.kernel.org, Coly Li <colyli@suse.de>,
linux-bcache@vger.kernel.org, Boaz Harrosh <ooo@electrozaur.com>,
Bob Peterson <rpeterso@redhat.com>,
cluster-devel@redhat.com
Subject: Re: [PATCH V10 09/19] block: introduce bio_bvecs()
Date: Tue, 20 Nov 2018 20:25:46 -0800 [thread overview]
Message-ID: <2a47d336-c19b-6bf4-c247-d7382871eeea@grimberg.me> (raw)
In-Reply-To: <20181121034415.GA8408@ming.t460p>
>> I would like to avoid growing bvec tables and keep everything
>> preallocated. Plus, a bvec_iter operates on a bvec which means
>> we'll need a table there as well... Not liking it so far...
>
> In case of bios in one request, we can't know how many bvecs there
> are except for calling rq_bvecs(), so it may not be suitable to
> preallocate the table. If you have to send the IO request in one send(),
> runtime allocation may be inevitable.
I don't want to do that, I want to work on a single bvec at a time like
the current implementation does.
> If you don't require to send the IO request in one send(), you may send
> one bio in one time, and just uses the bio's bvec table directly,
> such as the single bio case in lo_rw_aio().
we'd need some indication that we need to reinit my iter with the
new bvec, today we do:
static inline void nvme_tcp_advance_req(struct nvme_tcp_request *req,
int len)
{
req->snd.data_sent += len;
req->pdu_sent += len;
iov_iter_advance(&req->snd.iter, len);
if (!iov_iter_count(&req->snd.iter) &&
req->snd.data_sent < req->data_len) {
req->snd.curr_bio = req->snd.curr_bio->bi_next;
nvme_tcp_init_send_iter(req);
}
}
and initialize the send iter. I imagine that now I will need to
switch to the next bvec and only if I'm on the last I need to
use the next bio...
Do you offer an API for that?
>>> can this way avoid your blocking issue? You may see this
>>> example in branch 'rq->bio != rq->biotail' of lo_rw_aio().
>>
>> This is exactly an example of not ignoring the bios...
>
> Yeah, that is the most common example, given merge is enabled
> in most of cases. If the driver or device doesn't care merge,
> you can disable it and always get single bio request, then the
> bio's bvec table can be reused for send().
Does bvec_iter span bvecs with your patches? I didn't see that change?
>> I'm not sure how this helps me either. Unless we can set a bvec_iter to
>> span bvecs or have an abstract bio crossing when we re-initialize the
>> bvec_iter I don't see how I can ignore bios completely...
>
> rq_for_each_bvec() will iterate over all bvecs from all bios, so you
> needn't to see any bio in this req.
But I don't need this iteration, I need a transparent API like;
bvec2 = rq_bvec_next(rq, bvec)
This way I can simply always reinit my iter without thinking about how
the request/bios/bvecs are constructed...
> rq_bvecs() will return how many bvecs there are in this request(cover
> all bios in this req)
Still not very useful given that I don't want to use a table...
>>> So looks nvme-tcp host driver might be the 2nd driver which benefits
>>> from multi-page bvec directly.
>>>
>>> The multi-page bvec V11 has passed my tests and addressed almost
>>> all the comments during review on V10. I removed bio_vecs() in V11,
>>> but it won't be big deal, we can introduce them anytime when there
>>> is the requirement.
>>
>> multipage-bvecs and nvme-tcp are going to conflict, so it would be good
>> to coordinate on this. I think that nvme-tcp host needs some adjustments
>> as setting a bvec_iter. I'm under the impression that the change is rather
>> small and self-contained, but I'm not sure I have the full
>> picture here.
>
> I guess I may not get your exact requirement on block io iterator from nvme-tcp
> too, :-(
They are pretty much listed above. Today nvme-tcp sets an iterator with:
vec = __bvec_iter_bvec(bio->bi_io_vec, bio->bi_iter);
nsegs = bio_segments(bio);
size = bio->bi_iter.bi_size;
offset = bio->bi_iter.bi_bvec_done;
iov_iter_bvec(&req->snd.iter, WRITE, vec, nsegs, size);
and when done, iterate to the next bio and do the same.
With multipage bvec it would be great if we can simply have
something like rq_bvec_next() that would pretty much satisfy
the requirements from the nvme-tcp side...
next prev parent reply other threads:[~2018-11-21 4:25 UTC|newest]
Thread overview: 103+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-11-15 8:52 [PATCH V10 00/19] block: support multi-page bvec Ming Lei
2018-11-15 8:52 ` [PATCH V10 01/19] block: introduce multi-page page bvec helpers Ming Lei
2018-11-15 18:25 ` Omar Sandoval
2018-11-19 2:25 ` Ming Lei
2018-11-16 13:13 ` Christoph Hellwig
2018-11-19 2:23 ` Ming Lei
2018-11-19 3:10 ` Jens Axboe
2018-11-19 3:35 ` Ming Lei
2018-11-15 8:52 ` [PATCH V10 02/19] block: introduce bio_for_each_bvec() Ming Lei
2018-11-15 18:28 ` Omar Sandoval
2018-11-16 13:30 ` Christoph Hellwig
2018-11-19 3:31 ` Ming Lei
2018-11-15 8:52 ` [PATCH V10 03/19] block: use bio_for_each_bvec() to compute multi-page bvec count Ming Lei
2018-11-15 20:20 ` Omar Sandoval
2018-11-15 21:05 ` Mike Snitzer
2018-11-15 22:18 ` Omar Sandoval
2018-11-16 9:19 ` Christoph Hellwig
2018-11-16 9:41 ` Gao Xiang
2018-11-16 16:04 ` Omar Sandoval
2018-11-19 7:50 ` Ming Lei
2018-11-15 8:52 ` [PATCH V10 04/19] block: use bio_for_each_bvec() to map sg Ming Lei
2018-11-15 22:33 ` Omar Sandoval
2018-11-16 13:33 ` Christoph Hellwig
2018-11-19 7:51 ` Ming Lei
2018-11-15 8:52 ` [PATCH V10 05/19] block: introduce bvec_last_segment() Ming Lei
2018-11-15 23:23 ` Omar Sandoval
2018-11-19 7:57 ` Ming Lei
2018-11-16 13:34 ` Christoph Hellwig
2018-11-15 8:52 ` [PATCH V10 06/19] fs/buffer.c: use bvec iterator to truncate the bio Ming Lei
2018-11-16 0:20 ` Omar Sandoval
2018-11-16 13:36 ` Christoph Hellwig
2018-11-15 8:52 ` [PATCH V10 07/19] btrfs: use bvec_last_segment to get bio's last page Ming Lei
2018-11-16 0:21 ` Omar Sandoval
2018-11-16 13:37 ` Christoph Hellwig
2018-11-19 8:09 ` Ming Lei
2018-11-15 8:52 ` [PATCH V10 08/19] btrfs: move bio_pages_all() to btrfs Ming Lei
2018-11-16 0:23 ` Omar Sandoval
2018-11-19 8:15 ` Ming Lei
2018-11-16 13:38 ` Christoph Hellwig
2018-11-19 8:19 ` Ming Lei
2018-11-19 8:24 ` Christoph Hellwig
2018-11-15 8:52 ` [PATCH V10 09/19] block: introduce bio_bvecs() Ming Lei
2018-11-16 0:26 ` Omar Sandoval
2018-11-16 13:45 ` Christoph Hellwig
2018-11-19 8:21 ` Ming Lei
2018-11-20 0:49 ` Sagi Grimberg
2018-11-20 16:16 ` Christoph Hellwig
2018-11-20 20:11 ` Sagi Grimberg
2018-11-21 0:59 ` Ming Lei
2018-11-21 3:20 ` Sagi Grimberg
2018-11-21 3:44 ` Ming Lei
2018-11-21 4:25 ` Sagi Grimberg [this message]
2018-11-21 4:42 ` Sagi Grimberg
2018-11-21 5:04 ` Ming Lei
2018-11-21 5:35 ` Sagi Grimberg
2018-11-21 8:46 ` Christoph Hellwig
2018-11-21 10:19 ` Ming Lei
2018-11-15 8:52 ` [PATCH V10 10/19] block: loop: pass multi-page bvec to iov_iter Ming Lei
2018-11-16 0:40 ` Omar Sandoval
2018-11-19 8:25 ` Ming Lei
2018-11-15 8:52 ` [PATCH V10 11/19] bcache: avoid to use bio_for_each_segment_all() in bch_bio_alloc_pages() Ming Lei
2018-11-16 0:44 ` Omar Sandoval
2018-11-19 8:27 ` Ming Lei
2018-11-16 13:46 ` Christoph Hellwig
2018-11-19 8:28 ` Ming Lei
2018-11-15 8:52 ` [PATCH V10 12/19] block: allow bio_for_each_segment_all() to iterate over multi-page bvec Ming Lei
2018-11-15 12:42 ` David Sterba
2018-11-19 8:29 ` Ming Lei
2018-11-16 1:22 ` Omar Sandoval
2018-11-19 8:32 ` Ming Lei
2018-11-15 8:53 ` [PATCH V10 13/19] iomap & xfs: only account for new added page Ming Lei
2018-11-16 1:46 ` Omar Sandoval
2018-11-19 8:35 ` Ming Lei
2018-11-16 13:49 ` Christoph Hellwig
2018-11-19 8:39 ` Ming Lei
2018-11-15 8:53 ` [PATCH V10 14/19] block: enable multipage bvecs Ming Lei
2018-11-16 1:56 ` Omar Sandoval
2018-11-19 8:45 ` Ming Lei
2018-11-16 13:53 ` Christoph Hellwig
2018-11-19 9:00 ` Ming Lei
2018-11-15 8:53 ` [PATCH V10 15/19] block: always define BIO_MAX_PAGES as 256 Ming Lei
2018-11-16 1:59 ` Omar Sandoval
2018-11-19 9:04 ` Ming Lei
2018-11-20 2:45 ` Huang, Ying
2018-11-16 13:53 ` Christoph Hellwig
2018-11-15 8:53 ` [PATCH V10 16/19] block: document usage of bio iterator helpers Ming Lei
2018-11-16 2:05 ` Omar Sandoval
2018-11-15 8:53 ` [PATCH V10 17/19] block: don't use bio->bi_vcnt to figure out segment number Ming Lei
2018-11-16 2:11 ` Omar Sandoval
2018-11-19 9:06 ` Ming Lei
2018-11-16 13:55 ` Christoph Hellwig
2018-11-15 8:53 ` [PATCH V10 18/19] block: kill QUEUE_FLAG_NO_SG_MERGE Ming Lei
2018-11-16 2:18 ` Omar Sandoval
2018-11-16 13:59 ` Christoph Hellwig
2018-11-16 16:40 ` Omar Sandoval
2018-11-19 9:17 ` Ming Lei
2018-11-16 13:58 ` Christoph Hellwig
2018-11-19 9:20 ` Ming Lei
2018-11-15 8:53 ` [PATCH V10 19/19] block: kill BLK_MQ_F_SG_MERGE Ming Lei
2018-11-16 13:59 ` Christoph Hellwig
2018-11-16 16:40 ` Omar Sandoval
2018-11-16 14:03 ` [PATCH V10 00/19] block: support multi-page bvec Christoph Hellwig
2018-11-17 2:42 ` Ming Lei
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=2a47d336-c19b-6bf4-c247-d7382871eeea@grimberg.me \
--to=sagi@grimberg.me \
--cc=axboe@kernel.dk \
--cc=cluster-devel@redhat.com \
--cc=colyli@suse.de \
--cc=darrick.wong@oracle.com \
--cc=dchinner@redhat.com \
--cc=dm-devel@redhat.com \
--cc=dsterba@suse.com \
--cc=gaoxiang25@huawei.com \
--cc=hch@lst.de \
--cc=kent.overstreet@gmail.com \
--cc=linux-bcache@vger.kernel.org \
--cc=linux-block@vger.kernel.org \
--cc=linux-btrfs@vger.kernel.org \
--cc=linux-erofs@lists.ozlabs.org \
--cc=linux-ext4@vger.kernel.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-raid@vger.kernel.org \
--cc=linux-xfs@vger.kernel.org \
--cc=ming.lei@redhat.com \
--cc=ooo@electrozaur.com \
--cc=rpeterso@redhat.com \
--cc=shli@kernel.org \
--cc=snitzer@redhat.com \
--cc=tytso@mit.edu \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).