From: Jens Axboe <axboe@kernel.dk> To: Ming Lei <ming.lei@redhat.com> Cc: linux-block@vger.kernel.org, linux-nvme@lists.infradead.org Subject: Re: [PATCH 5/8] virtio_blk: implement mq_ops->commit_rqs() hook Date: Wed, 28 Nov 2018 19:19:09 -0700 [thread overview] Message-ID: <e937451d-57dd-41a4-fc4d-b5bbdb10869f@kernel.dk> (raw) In-Reply-To: <20181129012342.GB23249@ming.t460p> On 11/28/18 6:23 PM, Ming Lei wrote: > On Tue, Nov 27, 2018 at 07:34:51PM -0700, Jens Axboe wrote: >> On 11/27/18 7:10 PM, Ming Lei wrote: >>> On Mon, Nov 26, 2018 at 09:35:53AM -0700, Jens Axboe wrote: >>>> We need this for blk-mq to kick things into gear, if we told it that >>>> we had more IO coming, but then failed to deliver on that promise. >>>> >>>> Signed-off-by: Jens Axboe <axboe@kernel.dk> >>>> --- >>>> drivers/block/virtio_blk.c | 15 +++++++++++++++ >>>> 1 file changed, 15 insertions(+) >>>> >>>> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c >>>> index 6e869d05f91e..b49c57e77780 100644 >>>> --- a/drivers/block/virtio_blk.c >>>> +++ b/drivers/block/virtio_blk.c >>>> @@ -214,6 +214,20 @@ static void virtblk_done(struct virtqueue *vq) >>>> spin_unlock_irqrestore(&vblk->vqs[qid].lock, flags); >>>> } >>>> >>>> +static void virtio_commit_rqs(struct blk_mq_hw_ctx *hctx) >>>> +{ >>>> + struct virtio_blk *vblk = hctx->queue->queuedata; >>>> + int qid = hctx->queue_num; >>>> + bool kick; >>>> + >>>> + spin_lock_irq(&vblk->vqs[qid].lock); >>>> + kick = virtqueue_kick_prepare(vblk->vqs[qid].vq); >>>> + spin_unlock_irq(&vblk->vqs[qid].lock); >>>> + >>>> + if (kick) >>>> + virtqueue_notify(vblk->vqs[qid].vq); >>>> +} >>>> + >>>> static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, >>>> const struct blk_mq_queue_data *bd) >>>> { >>>> @@ -638,6 +652,7 @@ static void virtblk_initialize_rq(struct request *req) >>>> >>>> static const struct blk_mq_ops virtio_mq_ops = { >>>> .queue_rq = virtio_queue_rq, >>>> + .commit_rqs = virtio_commit_rqs, >>>> .complete = virtblk_request_done, >>>> .init_request = virtblk_init_request, >>>> #ifdef CONFIG_VIRTIO_BLK_SCSI >>>> -- >>>> 2.17.1 >>>> >>> >>> If .commit_rqs() is implemented, virtqueue_notify() in virtio_queue_rq() >>> should have been removed for saving the world switch per .queue_rq() >> >> ->commits_rqs() is only for the case where bd->last is set to false, >> and we never make it to the end and flag bd->last == true. If bd->last >> is true, the driver should kick things into gear. > > OK, looks I misunderstood it. However, virtio-blk doesn't need this > change since virtio_queue_rq() can handle it well. This patch may introduce > one unnecessary VM world switch in case of queue busy. Not it won't, it may in the case of some failure outside of the driver. The only reason that virtio-blk doesn't currently hang is because it has restart logic, and the failure case only happens in the if we already have IO in-flight. For the NVMe variant, that's not going to be the case. > IMO bd->last won't work well in case of io scheduler given the rq_list > only includes one single request. But that's a fake limitation that definitely should just be lifted, the fact that blk-mq-sched is _currently_ just doing single requests is woefully inefficient. > I wrote this kind of patch(never posted) before to use sort of > ->commits_rqs() to replace the current bd->last mechanism which need > one extra driver tag, which may improve the above case, also code gets > cleaned up. It doesn't need one extra driver tag, we currently get an extra one just to flag ->last correctly. That's not a requirement, that's a limitation of the current implementation. We could get rid of that, and it it proves to be an issue, that's not hard to do. I prefer to deal with actual issues and fix those, not in hypotheticals. -- Jens Axboe
WARNING: multiple messages have this Message-ID (diff)
From: axboe@kernel.dk (Jens Axboe) Subject: [PATCH 5/8] virtio_blk: implement mq_ops->commit_rqs() hook Date: Wed, 28 Nov 2018 19:19:09 -0700 [thread overview] Message-ID: <e937451d-57dd-41a4-fc4d-b5bbdb10869f@kernel.dk> (raw) In-Reply-To: <20181129012342.GB23249@ming.t460p> On 11/28/18 6:23 PM, Ming Lei wrote: > On Tue, Nov 27, 2018@07:34:51PM -0700, Jens Axboe wrote: >> On 11/27/18 7:10 PM, Ming Lei wrote: >>> On Mon, Nov 26, 2018@09:35:53AM -0700, Jens Axboe wrote: >>>> We need this for blk-mq to kick things into gear, if we told it that >>>> we had more IO coming, but then failed to deliver on that promise. >>>> >>>> Signed-off-by: Jens Axboe <axboe at kernel.dk> >>>> --- >>>> drivers/block/virtio_blk.c | 15 +++++++++++++++ >>>> 1 file changed, 15 insertions(+) >>>> >>>> diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c >>>> index 6e869d05f91e..b49c57e77780 100644 >>>> --- a/drivers/block/virtio_blk.c >>>> +++ b/drivers/block/virtio_blk.c >>>> @@ -214,6 +214,20 @@ static void virtblk_done(struct virtqueue *vq) >>>> spin_unlock_irqrestore(&vblk->vqs[qid].lock, flags); >>>> } >>>> >>>> +static void virtio_commit_rqs(struct blk_mq_hw_ctx *hctx) >>>> +{ >>>> + struct virtio_blk *vblk = hctx->queue->queuedata; >>>> + int qid = hctx->queue_num; >>>> + bool kick; >>>> + >>>> + spin_lock_irq(&vblk->vqs[qid].lock); >>>> + kick = virtqueue_kick_prepare(vblk->vqs[qid].vq); >>>> + spin_unlock_irq(&vblk->vqs[qid].lock); >>>> + >>>> + if (kick) >>>> + virtqueue_notify(vblk->vqs[qid].vq); >>>> +} >>>> + >>>> static blk_status_t virtio_queue_rq(struct blk_mq_hw_ctx *hctx, >>>> const struct blk_mq_queue_data *bd) >>>> { >>>> @@ -638,6 +652,7 @@ static void virtblk_initialize_rq(struct request *req) >>>> >>>> static const struct blk_mq_ops virtio_mq_ops = { >>>> .queue_rq = virtio_queue_rq, >>>> + .commit_rqs = virtio_commit_rqs, >>>> .complete = virtblk_request_done, >>>> .init_request = virtblk_init_request, >>>> #ifdef CONFIG_VIRTIO_BLK_SCSI >>>> -- >>>> 2.17.1 >>>> >>> >>> If .commit_rqs() is implemented, virtqueue_notify() in virtio_queue_rq() >>> should have been removed for saving the world switch per .queue_rq() >> >> ->commits_rqs() is only for the case where bd->last is set to false, >> and we never make it to the end and flag bd->last == true. If bd->last >> is true, the driver should kick things into gear. > > OK, looks I misunderstood it. However, virtio-blk doesn't need this > change since virtio_queue_rq() can handle it well. This patch may introduce > one unnecessary VM world switch in case of queue busy. Not it won't, it may in the case of some failure outside of the driver. The only reason that virtio-blk doesn't currently hang is because it has restart logic, and the failure case only happens in the if we already have IO in-flight. For the NVMe variant, that's not going to be the case. > IMO bd->last won't work well in case of io scheduler given the rq_list > only includes one single request. But that's a fake limitation that definitely should just be lifted, the fact that blk-mq-sched is _currently_ just doing single requests is woefully inefficient. > I wrote this kind of patch(never posted) before to use sort of > ->commits_rqs() to replace the current bd->last mechanism which need > one extra driver tag, which may improve the above case, also code gets > cleaned up. It doesn't need one extra driver tag, we currently get an extra one just to flag ->last correctly. That's not a requirement, that's a limitation of the current implementation. We could get rid of that, and it it proves to be an issue, that's not hard to do. I prefer to deal with actual issues and fix those, not in hypotheticals. -- Jens Axboe
next prev parent reply other threads:[~2018-11-29 2:19 UTC|newest] Thread overview: 84+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-11-26 16:35 [PATCHSET 0/8] block plugging improvements Jens Axboe 2018-11-26 16:35 ` Jens Axboe 2018-11-26 16:35 ` [PATCH 1/8] block: sum requests in the plug structure Jens Axboe 2018-11-26 16:35 ` Jens Axboe 2018-11-26 17:02 ` Christoph Hellwig 2018-11-26 17:02 ` Christoph Hellwig 2018-11-26 16:35 ` [PATCH 2/8] block: improve logic around when to sort a plug list Jens Axboe 2018-11-26 16:35 ` Jens Axboe 2018-11-27 23:31 ` Omar Sandoval 2018-11-27 23:31 ` Omar Sandoval 2018-11-27 23:49 ` Jens Axboe 2018-11-27 23:49 ` Jens Axboe 2018-11-27 23:55 ` Omar Sandoval 2018-11-27 23:55 ` Omar Sandoval 2018-11-27 23:59 ` Jens Axboe 2018-11-27 23:59 ` Jens Axboe 2018-11-28 0:05 ` Omar Sandoval 2018-11-28 0:05 ` Omar Sandoval 2018-11-28 0:16 ` Jens Axboe 2018-11-28 0:16 ` Jens Axboe 2018-11-26 16:35 ` [PATCH 3/8] blk-mq: add mq_ops->commit_rqs() Jens Axboe 2018-11-26 16:35 ` Jens Axboe 2018-11-27 23:43 ` Omar Sandoval 2018-11-27 23:43 ` Omar Sandoval 2018-11-28 1:38 ` Ming Lei 2018-11-28 1:38 ` Ming Lei 2018-11-28 7:16 ` Christoph Hellwig 2018-11-28 7:16 ` Christoph Hellwig 2018-11-28 12:54 ` Jens Axboe 2018-11-28 12:54 ` Jens Axboe 2018-11-26 16:35 ` [PATCH 4/8] nvme: implement mq_ops->commit_rqs() hook Jens Axboe 2018-11-26 16:35 ` Jens Axboe 2018-11-28 7:20 ` Christoph Hellwig 2018-11-28 7:20 ` Christoph Hellwig 2018-11-28 13:07 ` Jens Axboe 2018-11-28 13:07 ` Jens Axboe 2018-11-26 16:35 ` [PATCH 5/8] virtio_blk: " Jens Axboe 2018-11-26 16:35 ` Jens Axboe 2018-11-27 23:45 ` Omar Sandoval 2018-11-27 23:45 ` Omar Sandoval 2018-11-28 3:05 ` Michael S. Tsirkin 2018-11-28 3:05 ` Michael S. Tsirkin 2018-11-28 2:10 ` Ming Lei 2018-11-28 2:10 ` Ming Lei 2018-11-28 2:34 ` Jens Axboe 2018-11-28 2:34 ` Jens Axboe 2018-11-29 1:23 ` Ming Lei 2018-11-29 1:23 ` Ming Lei 2018-11-29 2:19 ` Jens Axboe [this message] 2018-11-29 2:19 ` Jens Axboe 2018-11-29 2:51 ` Ming Lei 2018-11-29 2:51 ` Ming Lei 2018-11-29 3:13 ` Jens Axboe 2018-11-29 3:13 ` Jens Axboe 2018-11-29 3:27 ` Ming Lei 2018-11-29 3:27 ` Ming Lei 2018-11-29 3:53 ` Jens Axboe 2018-11-29 3:53 ` Jens Axboe 2018-11-28 7:21 ` Christoph Hellwig 2018-11-28 7:21 ` Christoph Hellwig 2018-11-26 16:35 ` [PATCH 6/8] ataflop: " Jens Axboe 2018-11-26 16:35 ` Jens Axboe 2018-11-27 23:46 ` Omar Sandoval 2018-11-27 23:46 ` Omar Sandoval 2018-11-28 7:22 ` Christoph Hellwig 2018-11-28 7:22 ` Christoph Hellwig 2018-11-28 13:09 ` Jens Axboe 2018-11-28 13:09 ` Jens Axboe 2018-11-26 16:35 ` [PATCH 7/8] blk-mq: use bd->last == true for list inserts Jens Axboe 2018-11-26 16:35 ` Jens Axboe 2018-11-27 23:49 ` Omar Sandoval 2018-11-27 23:49 ` Omar Sandoval 2018-11-27 23:51 ` Jens Axboe 2018-11-27 23:51 ` Jens Axboe 2018-11-28 1:49 ` Ming Lei 2018-11-28 1:49 ` Ming Lei 2018-11-28 2:37 ` Jens Axboe 2018-11-28 2:37 ` Jens Axboe 2018-11-26 16:35 ` [PATCH 8/8] blk-mq: add plug case for devices that implement ->commits_rqs() Jens Axboe 2018-11-26 16:35 ` Jens Axboe 2018-11-28 7:26 ` Christoph Hellwig 2018-11-28 7:26 ` Christoph Hellwig 2018-11-28 13:11 ` Jens Axboe 2018-11-28 13:11 ` Jens Axboe
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=e937451d-57dd-41a4-fc4d-b5bbdb10869f@kernel.dk \ --to=axboe@kernel.dk \ --cc=linux-block@vger.kernel.org \ --cc=linux-nvme@lists.infradead.org \ --cc=ming.lei@redhat.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.