* [PATCH 1/2] null_blk: prevent timer handler running on a different CPU where started
@ 2015-06-01 23:35 Akinobu Mita
2015-06-01 23:35 ` [PATCH 2/2] null_blk: restart request processing on completion handler Akinobu Mita
2015-06-02 14:44 ` [PATCH 1/2] null_blk: prevent timer handler running on a different CPU where started Jens Axboe
0 siblings, 2 replies; 4+ messages in thread
From: Akinobu Mita @ 2015-06-01 23:35 UTC (permalink / raw)
To: linux-kernel; +Cc: Akinobu Mita, Jens Axboe
When irqmode=2 (IRQ completion handler is timer), timer handler should
be called on the same CPU where the timer has been started.
Since completion_queues are per-cpu and the completion handler only
touches completion_queue for local CPU, we need to prevent the handler
from running on a different CPU where the timer has been started.
Otherwise, the IO cannot be completed until another completion handler
is executed on that CPU.
Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Cc: Jens Axboe <axboe@fb.com>
---
drivers/block/null_blk.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/block/null_blk.c b/drivers/block/null_blk.c
index 65cd61a..6f0a58e 100644
--- a/drivers/block/null_blk.c
+++ b/drivers/block/null_blk.c
@@ -257,7 +257,7 @@ static void null_cmd_end_timer(struct nullb_cmd *cmd)
if (llist_add(&cmd->ll_list, &cq->list)) {
ktime_t kt = ktime_set(0, completion_nsec);
- hrtimer_start(&cq->timer, kt, HRTIMER_MODE_REL);
+ hrtimer_start(&cq->timer, kt, HRTIMER_MODE_REL_PINNED);
}
put_cpu();
--
1.9.1
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [PATCH 2/2] null_blk: restart request processing on completion handler
2015-06-01 23:35 [PATCH 1/2] null_blk: prevent timer handler running on a different CPU where started Akinobu Mita
@ 2015-06-01 23:35 ` Akinobu Mita
2015-06-02 15:53 ` Akinobu Mita
2015-06-02 14:44 ` [PATCH 1/2] null_blk: prevent timer handler running on a different CPU where started Jens Axboe
1 sibling, 1 reply; 4+ messages in thread
From: Akinobu Mita @ 2015-06-01 23:35 UTC (permalink / raw)
To: linux-kernel; +Cc: Akinobu Mita, Jens Axboe
When irqmode=2 (IRQ completion handler is timer) and queue_mode=1
(Block interface to use is rq), the completion handler should restart
request handling for any pending requests on a queue because request
processing stops when the number of commands are queued more than
hw_queue_depth (null_rq_prep_fn returns BLKPREP_DEFER).
Without this change, the following command cannot finish.
# modprobe null_blk irqmode=2 queue_mode=1 hw_queue_depth=1
# fio --name=t --rw=read --size=1g --direct=1 \
--ioengine=libaio --iodepth=64 --filename=/dev/nullb0
Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Cc: Jens Axboe <axboe@fb.com>
---
drivers/block/null_blk.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/drivers/block/null_blk.c b/drivers/block/null_blk.c
index 6f0a58e..6f9b753 100644
--- a/drivers/block/null_blk.c
+++ b/drivers/block/null_blk.c
@@ -243,6 +243,17 @@ static enum hrtimer_restart null_cmd_timer_expired(struct hrtimer *timer)
cmd = container_of(entry, struct nullb_cmd, ll_list);
entry = entry->next;
end_cmd(cmd);
+
+ if (cmd->rq) {
+ struct request_queue *q = cmd->rq->q;
+
+ if (!q->mq_ops && blk_queue_stopped(q)) {
+ spin_lock(q->queue_lock);
+ if (blk_queue_stopped(q))
+ blk_start_queue(q);
+ spin_unlock(q->queue_lock);
+ }
+ }
} while (entry);
}
@@ -334,6 +345,7 @@ static int null_rq_prep_fn(struct request_queue *q, struct request *req)
req->special = cmd;
return BLKPREP_OK;
}
+ blk_stop_queue(q);
return BLKPREP_DEFER;
}
--
1.9.1
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH 1/2] null_blk: prevent timer handler running on a different CPU where started
2015-06-01 23:35 [PATCH 1/2] null_blk: prevent timer handler running on a different CPU where started Akinobu Mita
2015-06-01 23:35 ` [PATCH 2/2] null_blk: restart request processing on completion handler Akinobu Mita
@ 2015-06-02 14:44 ` Jens Axboe
1 sibling, 0 replies; 4+ messages in thread
From: Jens Axboe @ 2015-06-02 14:44 UTC (permalink / raw)
To: Akinobu Mita, linux-kernel
On 06/01/2015 05:35 PM, Akinobu Mita wrote:
> When irqmode=2 (IRQ completion handler is timer), timer handler should
> be called on the same CPU where the timer has been started.
>
> Since completion_queues are per-cpu and the completion handler only
> touches completion_queue for local CPU, we need to prevent the handler
> from running on a different CPU where the timer has been started.
> Otherwise, the IO cannot be completed until another completion handler
> is executed on that CPU.
Thanks, applied this and 2/2, the latter seems to have been forgotten
after our initial discussion about it.
--
Jens Axboe
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH 2/2] null_blk: restart request processing on completion handler
2015-06-01 23:35 ` [PATCH 2/2] null_blk: restart request processing on completion handler Akinobu Mita
@ 2015-06-02 15:53 ` Akinobu Mita
0 siblings, 0 replies; 4+ messages in thread
From: Akinobu Mita @ 2015-06-02 15:53 UTC (permalink / raw)
To: LKML; +Cc: Akinobu Mita, Jens Axboe
2015-06-02 8:35 GMT+09:00 Akinobu Mita <akinobu.mita@gmail.com>:
> When irqmode=2 (IRQ completion handler is timer) and queue_mode=1
> (Block interface to use is rq), the completion handler should restart
> request handling for any pending requests on a queue because request
> processing stops when the number of commands are queued more than
> hw_queue_depth (null_rq_prep_fn returns BLKPREP_DEFER).
>
> Without this change, the following command cannot finish.
>
> # modprobe null_blk irqmode=2 queue_mode=1 hw_queue_depth=1
> # fio --name=t --rw=read --size=1g --direct=1 \
> --ioengine=libaio --iodepth=64 --filename=/dev/nullb0
>
> Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
> Cc: Jens Axboe <axboe@fb.com>
> ---
> drivers/block/null_blk.c | 12 ++++++++++++
> 1 file changed, 12 insertions(+)
>
> diff --git a/drivers/block/null_blk.c b/drivers/block/null_blk.c
> index 6f0a58e..6f9b753 100644
> --- a/drivers/block/null_blk.c
> +++ b/drivers/block/null_blk.c
> @@ -243,6 +243,17 @@ static enum hrtimer_restart null_cmd_timer_expired(struct hrtimer *timer)
> cmd = container_of(entry, struct nullb_cmd, ll_list);
> entry = entry->next;
> end_cmd(cmd);
> +
> + if (cmd->rq) {
> + struct request_queue *q = cmd->rq->q;
> +
> + if (!q->mq_ops && blk_queue_stopped(q)) {
> + spin_lock(q->queue_lock);
> + if (blk_queue_stopped(q))
> + blk_start_queue(q);
Should 'blk_start_queue(q)' be:
queue_flag_clear(QUEUE_FLAG_STOPPED, q);
blk_run_queue_async(q);
?
Otherwise, all pending requests on a queue are completed by this timer
handler execution without needing next timer expiry.
> + spin_unlock(q->queue_lock);
> + }
> + }
> } while (entry);
> }
>
> @@ -334,6 +345,7 @@ static int null_rq_prep_fn(struct request_queue *q, struct request *req)
> req->special = cmd;
> return BLKPREP_OK;
> }
> + blk_stop_queue(q);
>
> return BLKPREP_DEFER;
> }
> --
> 1.9.1
>
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2015-06-02 15:54 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-06-01 23:35 [PATCH 1/2] null_blk: prevent timer handler running on a different CPU where started Akinobu Mita
2015-06-01 23:35 ` [PATCH 2/2] null_blk: restart request processing on completion handler Akinobu Mita
2015-06-02 15:53 ` Akinobu Mita
2015-06-02 14:44 ` [PATCH 1/2] null_blk: prevent timer handler running on a different CPU where started Jens Axboe
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).