* confusion about nr of pending I/O requests
@ 2018-12-18 12:45 Paolo Valente
2018-12-18 12:49 ` Paolo Valente
2018-12-18 18:50 ` Jens Axboe
0 siblings, 2 replies; 8+ messages in thread
From: Paolo Valente @ 2018-12-18 12:45 UTC (permalink / raw)
To: Jens Axboe, linux-block; +Cc: Linus Walleij, Mark Brown, Ulf Hansson
Hi Jens,
sorry for the following silly question, but maybe you can solve very
quickly a doubt for which I'd spend much more time investigating.
While doing some tests with scsi_debug, I've just seen that (at least)
with direct I/O, the maximum number of pending I/O requests (at least
in the I/O schedulers) is equal, unexpectedly, to the queue depth of
the drive and not to
/sys/block/<dev>/queue/nr_requests
For example, after:
sudo modprobe scsi_debug max_queue=4
and with fio executed as follows:
job: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=20
I get this periodic trace, where four insertions are followed by four
completions, and so on, till the end of the I/O. This trace is taken
with none, but the result is the same with bfq.
fio-20275 [001] d... 7560.655213: 8,48 I R 281088 + 8 [fio]
fio-20275 [001] d... 7560.655288: 8,48 I R 281096 + 8 [fio]
fio-20275 [001] d... 7560.655311: 8,48 I R 281104 + 8 [fio]
fio-20275 [001] d... 7560.655331: 8,48 I R 281112 + 8 [fio]
<idle>-0 [001] d.h. 7560.749868: 8,48 C R 281088 + 8 [0]
<idle>-0 [001] dNh. 7560.749912: 8,48 C R 281096 + 8 [0]
<idle>-0 [001] dNh. 7560.749928: 8,48 C R 281104 + 8 [0]
<idle>-0 [001] dNh. 7560.749934: 8,48 C R 281112 + 8 [0]
fio-20275 [001] d... 7560.750023: 8,48 I R 281120 + 8 [fio]
fio-20275 [001] d... 7560.750196: 8,48 I R 281128 + 8 [fio]
fio-20275 [001] d... 7560.750229: 8,48 I R 281136 + 8 [fio]
fio-20275 [001] d... 7560.750250: 8,48 I R 281144 + 8 [fio]
<idle>-0 [001] d.h. 7560.842510: 8,48 C R 281120 + 8 [0]
<idle>-0 [001] dNh. 7560.842551: 8,48 C R 281128 + 8 [0]
<idle>-0 [001] dNh. 7560.842556: 8,48 C R 281136 + 8 [0]
<idle>-0 [001] dNh. 7560.842562: 8,48 C R 281144 + 8 [0]
Shouldn't the total number of pending requests reach
/sys/block/<dev>/queue/nr_requests ?
The latter is of course equal to 8.
Thanks,
Paolo
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: confusion about nr of pending I/O requests 2018-12-18 12:45 confusion about nr of pending I/O requests Paolo Valente @ 2018-12-18 12:49 ` Paolo Valente 2018-12-18 18:50 ` Jens Axboe 1 sibling, 0 replies; 8+ messages in thread From: Paolo Valente @ 2018-12-18 12:49 UTC (permalink / raw) To: Jens Axboe, linux-block; +Cc: Linus Walleij, Mark Brown, Ulf Hansson > Il giorno 18 dic 2018, alle ore 13:45, Paolo Valente <paolo.valente@linaro.org> ha scritto: > > Hi Jens, > sorry for the following silly question, but maybe you can solve very > quickly a doubt for which I'd spend much more time investigating. > > While doing some tests with scsi_debug, I've just seen that (at least) > with direct I/O, the maximum number of pending I/O requests (at least > in the I/O schedulers) is equal, unexpectedly, to the queue depth of > the drive and not to > /sys/block/<dev>/queue/nr_requests > > For example, after: > > sudo modprobe scsi_debug max_queue=4 > > and with fio executed as follows: > > job: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=20 > > I get this periodic trace, where four insertions are followed by four > completions, and so on, till the end of the I/O. This trace is taken > with none, but the result is the same with bfq. > > fio-20275 [001] d... 7560.655213: 8,48 I R 281088 + 8 [fio] > fio-20275 [001] d... 7560.655288: 8,48 I R 281096 + 8 [fio] > fio-20275 [001] d... 7560.655311: 8,48 I R 281104 + 8 [fio] > fio-20275 [001] d... 7560.655331: 8,48 I R 281112 + 8 [fio] > <idle>-0 [001] d.h. 7560.749868: 8,48 C R 281088 + 8 [0] > <idle>-0 [001] dNh. 7560.749912: 8,48 C R 281096 + 8 [0] > <idle>-0 [001] dNh. 7560.749928: 8,48 C R 281104 + 8 [0] > <idle>-0 [001] dNh. 7560.749934: 8,48 C R 281112 + 8 [0] > fio-20275 [001] d... 7560.750023: 8,48 I R 281120 + 8 [fio] > fio-20275 [001] d... 7560.750196: 8,48 I R 281128 + 8 [fio] > fio-20275 [001] d... 7560.750229: 8,48 I R 281136 + 8 [fio] > fio-20275 [001] d... 7560.750250: 8,48 I R 281144 + 8 [fio] > <idle>-0 [001] d.h. 7560.842510: 8,48 C R 281120 + 8 [0] > <idle>-0 [001] dNh. 7560.842551: 8,48 C R 281128 + 8 [0] > <idle>-0 [001] dNh. 7560.842556: 8,48 C R 281136 + 8 [0] > <idle>-0 [001] dNh. 7560.842562: 8,48 C R 281144 + 8 [0] > > Shouldn't the total number of pending requests reach > /sys/block/<dev>/queue/nr_requests ? > > The latter is of course equal to 8. > I forgot: I did this with a 4.18. Maybe this is something that has been changed/fixed? Thanks, Paolo > Thanks, > Paolo > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: confusion about nr of pending I/O requests 2018-12-18 12:45 confusion about nr of pending I/O requests Paolo Valente 2018-12-18 12:49 ` Paolo Valente @ 2018-12-18 18:50 ` Jens Axboe 2018-12-18 23:35 ` Paolo Valente 2018-12-19 3:45 ` Ming Lei 1 sibling, 2 replies; 8+ messages in thread From: Jens Axboe @ 2018-12-18 18:50 UTC (permalink / raw) To: Paolo Valente, linux-block; +Cc: Linus Walleij, Mark Brown, Ulf Hansson On 12/18/18 5:45 AM, Paolo Valente wrote: > Hi Jens, > sorry for the following silly question, but maybe you can solve very > quickly a doubt for which I'd spend much more time investigating. > > While doing some tests with scsi_debug, I've just seen that (at least) > with direct I/O, the maximum number of pending I/O requests (at least > in the I/O schedulers) is equal, unexpectedly, to the queue depth of > the drive and not to > /sys/block/<dev>/queue/nr_requests > > For example, after: > > sudo modprobe scsi_debug max_queue=4 > > and with fio executed as follows: > > job: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=20 > > I get this periodic trace, where four insertions are followed by four > completions, and so on, till the end of the I/O. This trace is taken > with none, but the result is the same with bfq. > > fio-20275 [001] d... 7560.655213: 8,48 I R 281088 + 8 [fio] > fio-20275 [001] d... 7560.655288: 8,48 I R 281096 + 8 [fio] > fio-20275 [001] d... 7560.655311: 8,48 I R 281104 + 8 [fio] > fio-20275 [001] d... 7560.655331: 8,48 I R 281112 + 8 [fio] > <idle>-0 [001] d.h. 7560.749868: 8,48 C R 281088 + 8 [0] > <idle>-0 [001] dNh. 7560.749912: 8,48 C R 281096 + 8 [0] > <idle>-0 [001] dNh. 7560.749928: 8,48 C R 281104 + 8 [0] > <idle>-0 [001] dNh. 7560.749934: 8,48 C R 281112 + 8 [0] > fio-20275 [001] d... 7560.750023: 8,48 I R 281120 + 8 [fio] > fio-20275 [001] d... 7560.750196: 8,48 I R 281128 + 8 [fio] > fio-20275 [001] d... 7560.750229: 8,48 I R 281136 + 8 [fio] > fio-20275 [001] d... 7560.750250: 8,48 I R 281144 + 8 [fio] > <idle>-0 [001] d.h. 7560.842510: 8,48 C R 281120 + 8 [0] > <idle>-0 [001] dNh. 7560.842551: 8,48 C R 281128 + 8 [0] > <idle>-0 [001] dNh. 7560.842556: 8,48 C R 281136 + 8 [0] > <idle>-0 [001] dNh. 7560.842562: 8,48 C R 281144 + 8 [0] > > Shouldn't the total number of pending requests reach > /sys/block/<dev>/queue/nr_requests ? > > The latter is of course equal to 8. With a scheduler, the depth is what the scheduler provides. You cannot exceed the hardware queue depth in any situation. You just have 8 requests available for scheduling, with a max of 4 being inflight on the device side. If both were 4, for instance, then you would have nothing to schedule with, as all of them could reside on the hardware side. That's why the scheduler defaults to twice the hardware queue depth. -- Jens Axboe ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: confusion about nr of pending I/O requests 2018-12-18 18:50 ` Jens Axboe @ 2018-12-18 23:35 ` Paolo Valente 2018-12-19 3:45 ` Ming Lei 1 sibling, 0 replies; 8+ messages in thread From: Paolo Valente @ 2018-12-18 23:35 UTC (permalink / raw) To: Jens Axboe; +Cc: linux-block, Linus Walleij, Mark Brown, Ulf Hansson > Il giorno 18 dic 2018, alle ore 19:50, Jens Axboe <axboe@kernel.dk> ha scritto: > > On 12/18/18 5:45 AM, Paolo Valente wrote: >> Hi Jens, >> sorry for the following silly question, but maybe you can solve very >> quickly a doubt for which I'd spend much more time investigating. >> >> While doing some tests with scsi_debug, I've just seen that (at least) >> with direct I/O, the maximum number of pending I/O requests (at least >> in the I/O schedulers) is equal, unexpectedly, to the queue depth of >> the drive and not to >> /sys/block/<dev>/queue/nr_requests >> >> For example, after: >> >> sudo modprobe scsi_debug max_queue=4 >> >> and with fio executed as follows: >> >> job: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=20 >> >> I get this periodic trace, where four insertions are followed by four >> completions, and so on, till the end of the I/O. This trace is taken >> with none, but the result is the same with bfq. >> >> fio-20275 [001] d... 7560.655213: 8,48 I R 281088 + 8 [fio] >> fio-20275 [001] d... 7560.655288: 8,48 I R 281096 + 8 [fio] >> fio-20275 [001] d... 7560.655311: 8,48 I R 281104 + 8 [fio] >> fio-20275 [001] d... 7560.655331: 8,48 I R 281112 + 8 [fio] >> <idle>-0 [001] d.h. 7560.749868: 8,48 C R 281088 + 8 [0] >> <idle>-0 [001] dNh. 7560.749912: 8,48 C R 281096 + 8 [0] >> <idle>-0 [001] dNh. 7560.749928: 8,48 C R 281104 + 8 [0] >> <idle>-0 [001] dNh. 7560.749934: 8,48 C R 281112 + 8 [0] >> fio-20275 [001] d... 7560.750023: 8,48 I R 281120 + 8 [fio] >> fio-20275 [001] d... 7560.750196: 8,48 I R 281128 + 8 [fio] >> fio-20275 [001] d... 7560.750229: 8,48 I R 281136 + 8 [fio] >> fio-20275 [001] d... 7560.750250: 8,48 I R 281144 + 8 [fio] >> <idle>-0 [001] d.h. 7560.842510: 8,48 C R 281120 + 8 [0] >> <idle>-0 [001] dNh. 7560.842551: 8,48 C R 281128 + 8 [0] >> <idle>-0 [001] dNh. 7560.842556: 8,48 C R 281136 + 8 [0] >> <idle>-0 [001] dNh. 7560.842562: 8,48 C R 281144 + 8 [0] >> >> Shouldn't the total number of pending requests reach >> /sys/block/<dev>/queue/nr_requests ? >> >> The latter is of course equal to 8. > > With a scheduler, the depth is what the scheduler provides. You cannot > exceed the hardware queue depth in any situation. You just have 8 > requests available for scheduling, with a max of 4 being inflight on > the device side. > > If both were 4, for instance, then you would have nothing to schedule > with, as all of them could reside on the hardware side. That's why > the scheduler defaults to twice the hardware queue depth. > That's exactly what I expected, thanks. But not what happened. Let me add also dispatch lines to my filtered-trace snippet. The repetitive pattern becomes: fio-5180 [001] d... 786.931956: 8,48 I WS 333824 + 1024 [fio] fio-5180 [001] d... 786.932010: 8,48 D WS 333824 + 1024 [fio] fio-5180 [001] d... 786.932137: 8,48 I WS 334848 + 1024 [fio] fio-5180 [001] d... 786.932160: 8,48 D WS 334848 + 1024 [fio] fio-5180 [001] d... 786.932318: 8,48 I WS 335872 + 1024 [fio] fio-5180 [001] d... 786.932354: 8,48 D WS 335872 + 1024 [fio] fio-5180 [001] d... 786.932467: 8,48 I WS 336896 + 1024 [fio] fio-5180 [001] d... 786.932489: 8,48 D WS 336896 + 1024 [fio] <idle>-0 [001] d.h. 787.023945: 8,48 C WS 333824 + 1024 [0] <idle>-0 [001] d.h. 787.023978: 8,48 C WS 334848 + 1024 [0] <idle>-0 [001] d.h. 787.024080: 8,48 C WS 335872 + 1024 [0] <idle>-0 [001] d.h. 787.024237: 8,48 C WS 336896 + 1024 [0] So, after the four dispatches, there are 0 requests in the scheduler, and 4 requests inflight. But *no* new request is inserted into the scheduler before *all* four inflight requests are completed. So the total number of requests available seems 4 and not 8. Am I missing something? Thanks, Paolo > -- > Jens Axboe ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: confusion about nr of pending I/O requests 2018-12-18 18:50 ` Jens Axboe 2018-12-18 23:35 ` Paolo Valente @ 2018-12-19 3:45 ` Ming Lei 2018-12-19 6:17 ` Paolo Valente 1 sibling, 1 reply; 8+ messages in thread From: Ming Lei @ 2018-12-19 3:45 UTC (permalink / raw) To: Jens Axboe Cc: Paolo Valente, linux-block, Linus Walleij, Mark Brown, Ulf Hansson On Wed, Dec 19, 2018 at 2:52 AM Jens Axboe <axboe@kernel.dk> wrote: > > On 12/18/18 5:45 AM, Paolo Valente wrote: > > Hi Jens, > > sorry for the following silly question, but maybe you can solve very > > quickly a doubt for which I'd spend much more time investigating. > > > > While doing some tests with scsi_debug, I've just seen that (at least) > > with direct I/O, the maximum number of pending I/O requests (at least > > in the I/O schedulers) is equal, unexpectedly, to the queue depth of > > the drive and not to > > /sys/block/<dev>/queue/nr_requests > > > > For example, after: > > > > sudo modprobe scsi_debug max_queue=4 > > > > and with fio executed as follows: > > > > job: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=20 > > > > I get this periodic trace, where four insertions are followed by four > > completions, and so on, till the end of the I/O. This trace is taken > > with none, but the result is the same with bfq. > > > > fio-20275 [001] d... 7560.655213: 8,48 I R 281088 + 8 [fio] > > fio-20275 [001] d... 7560.655288: 8,48 I R 281096 + 8 [fio] > > fio-20275 [001] d... 7560.655311: 8,48 I R 281104 + 8 [fio] > > fio-20275 [001] d... 7560.655331: 8,48 I R 281112 + 8 [fio] > > <idle>-0 [001] d.h. 7560.749868: 8,48 C R 281088 + 8 [0] > > <idle>-0 [001] dNh. 7560.749912: 8,48 C R 281096 + 8 [0] > > <idle>-0 [001] dNh. 7560.749928: 8,48 C R 281104 + 8 [0] > > <idle>-0 [001] dNh. 7560.749934: 8,48 C R 281112 + 8 [0] > > fio-20275 [001] d... 7560.750023: 8,48 I R 281120 + 8 [fio] > > fio-20275 [001] d... 7560.750196: 8,48 I R 281128 + 8 [fio] > > fio-20275 [001] d... 7560.750229: 8,48 I R 281136 + 8 [fio] > > fio-20275 [001] d... 7560.750250: 8,48 I R 281144 + 8 [fio] > > <idle>-0 [001] d.h. 7560.842510: 8,48 C R 281120 + 8 [0] > > <idle>-0 [001] dNh. 7560.842551: 8,48 C R 281128 + 8 [0] > > <idle>-0 [001] dNh. 7560.842556: 8,48 C R 281136 + 8 [0] > > <idle>-0 [001] dNh. 7560.842562: 8,48 C R 281144 + 8 [0] > > > > Shouldn't the total number of pending requests reach > > /sys/block/<dev>/queue/nr_requests ? > > > > The latter is of course equal to 8. > > With a scheduler, the depth is what the scheduler provides. You cannot > exceed the hardware queue depth in any situation. You just have 8 > requests available for scheduling, with a max of 4 being inflight on > the device side. > > If both were 4, for instance, then you would have nothing to schedule > with, as all of them could reside on the hardware side. That's why > the scheduler defaults to twice the hardware queue depth. The default twice to hw queue depth might not be reasonable for multi LUN. Maybe it should be set twice of sdev->queue_depth for SCSI or hw queue depth/hctx->nr_active. But either way may become complicated because both can be adjusted runtime. Thanks, Ming Lei ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: confusion about nr of pending I/O requests 2018-12-19 3:45 ` Ming Lei @ 2018-12-19 6:17 ` Paolo Valente 2018-12-19 10:32 ` Ming Lei 0 siblings, 1 reply; 8+ messages in thread From: Paolo Valente @ 2018-12-19 6:17 UTC (permalink / raw) To: Ming Lei; +Cc: Jens Axboe, linux-block, Linus Walleij, Mark Brown, Ulf Hansson > Il giorno 19 dic 2018, alle ore 04:45, Ming Lei <tom.leiming@gmail.com> ha scritto: > > On Wed, Dec 19, 2018 at 2:52 AM Jens Axboe <axboe@kernel.dk> wrote: >> >> On 12/18/18 5:45 AM, Paolo Valente wrote: >>> Hi Jens, >>> sorry for the following silly question, but maybe you can solve very >>> quickly a doubt for which I'd spend much more time investigating. >>> >>> While doing some tests with scsi_debug, I've just seen that (at least) >>> with direct I/O, the maximum number of pending I/O requests (at least >>> in the I/O schedulers) is equal, unexpectedly, to the queue depth of >>> the drive and not to >>> /sys/block/<dev>/queue/nr_requests >>> >>> For example, after: >>> >>> sudo modprobe scsi_debug max_queue=4 >>> >>> and with fio executed as follows: >>> >>> job: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=20 >>> >>> I get this periodic trace, where four insertions are followed by four >>> completions, and so on, till the end of the I/O. This trace is taken >>> with none, but the result is the same with bfq. >>> >>> fio-20275 [001] d... 7560.655213: 8,48 I R 281088 + 8 [fio] >>> fio-20275 [001] d... 7560.655288: 8,48 I R 281096 + 8 [fio] >>> fio-20275 [001] d... 7560.655311: 8,48 I R 281104 + 8 [fio] >>> fio-20275 [001] d... 7560.655331: 8,48 I R 281112 + 8 [fio] >>> <idle>-0 [001] d.h. 7560.749868: 8,48 C R 281088 + 8 [0] >>> <idle>-0 [001] dNh. 7560.749912: 8,48 C R 281096 + 8 [0] >>> <idle>-0 [001] dNh. 7560.749928: 8,48 C R 281104 + 8 [0] >>> <idle>-0 [001] dNh. 7560.749934: 8,48 C R 281112 + 8 [0] >>> fio-20275 [001] d... 7560.750023: 8,48 I R 281120 + 8 [fio] >>> fio-20275 [001] d... 7560.750196: 8,48 I R 281128 + 8 [fio] >>> fio-20275 [001] d... 7560.750229: 8,48 I R 281136 + 8 [fio] >>> fio-20275 [001] d... 7560.750250: 8,48 I R 281144 + 8 [fio] >>> <idle>-0 [001] d.h. 7560.842510: 8,48 C R 281120 + 8 [0] >>> <idle>-0 [001] dNh. 7560.842551: 8,48 C R 281128 + 8 [0] >>> <idle>-0 [001] dNh. 7560.842556: 8,48 C R 281136 + 8 [0] >>> <idle>-0 [001] dNh. 7560.842562: 8,48 C R 281144 + 8 [0] >>> >>> Shouldn't the total number of pending requests reach >>> /sys/block/<dev>/queue/nr_requests ? >>> >>> The latter is of course equal to 8. >> >> With a scheduler, the depth is what the scheduler provides. You cannot >> exceed the hardware queue depth in any situation. You just have 8 >> requests available for scheduling, with a max of 4 being inflight on >> the device side. >> >> If both were 4, for instance, then you would have nothing to schedule >> with, as all of them could reside on the hardware side. That's why >> the scheduler defaults to twice the hardware queue depth. > > The default twice to hw queue depth might not be reasonable for multi LUN. > > Maybe it should be set twice of sdev->queue_depth for SCSI or > hw queue depth/hctx->nr_active. But either way may become complicated > because both can be adjusted runtime. > Could you please explain why it is not working (if it is not working) in my example, where there should be only one LUN? Thanks, Paolo > Thanks, > Ming Lei ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: confusion about nr of pending I/O requests 2018-12-19 6:17 ` Paolo Valente @ 2018-12-19 10:32 ` Ming Lei 2018-12-19 11:45 ` Paolo Valente 0 siblings, 1 reply; 8+ messages in thread From: Ming Lei @ 2018-12-19 10:32 UTC (permalink / raw) To: Paolo Valente Cc: Jens Axboe, linux-block, Linus Walleij, Mark Brown, Ulf Hansson On Wed, Dec 19, 2018 at 2:18 PM Paolo Valente <paolo.valente@linaro.org> wrote: > > > > > Il giorno 19 dic 2018, alle ore 04:45, Ming Lei <tom.leiming@gmail.com> ha scritto: > > > > On Wed, Dec 19, 2018 at 2:52 AM Jens Axboe <axboe@kernel.dk> wrote: > >> > >> On 12/18/18 5:45 AM, Paolo Valente wrote: > >>> Hi Jens, > >>> sorry for the following silly question, but maybe you can solve very > >>> quickly a doubt for which I'd spend much more time investigating. > >>> > >>> While doing some tests with scsi_debug, I've just seen that (at least) > >>> with direct I/O, the maximum number of pending I/O requests (at least > >>> in the I/O schedulers) is equal, unexpectedly, to the queue depth of > >>> the drive and not to > >>> /sys/block/<dev>/queue/nr_requests > >>> > >>> For example, after: > >>> > >>> sudo modprobe scsi_debug max_queue=4 > >>> > >>> and with fio executed as follows: > >>> > >>> job: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=20 > >>> > >>> I get this periodic trace, where four insertions are followed by four > >>> completions, and so on, till the end of the I/O. This trace is taken > >>> with none, but the result is the same with bfq. > >>> > >>> fio-20275 [001] d... 7560.655213: 8,48 I R 281088 + 8 [fio] > >>> fio-20275 [001] d... 7560.655288: 8,48 I R 281096 + 8 [fio] > >>> fio-20275 [001] d... 7560.655311: 8,48 I R 281104 + 8 [fio] > >>> fio-20275 [001] d... 7560.655331: 8,48 I R 281112 + 8 [fio] > >>> <idle>-0 [001] d.h. 7560.749868: 8,48 C R 281088 + 8 [0] > >>> <idle>-0 [001] dNh. 7560.749912: 8,48 C R 281096 + 8 [0] > >>> <idle>-0 [001] dNh. 7560.749928: 8,48 C R 281104 + 8 [0] > >>> <idle>-0 [001] dNh. 7560.749934: 8,48 C R 281112 + 8 [0] > >>> fio-20275 [001] d... 7560.750023: 8,48 I R 281120 + 8 [fio] > >>> fio-20275 [001] d... 7560.750196: 8,48 I R 281128 + 8 [fio] > >>> fio-20275 [001] d... 7560.750229: 8,48 I R 281136 + 8 [fio] > >>> fio-20275 [001] d... 7560.750250: 8,48 I R 281144 + 8 [fio] > >>> <idle>-0 [001] d.h. 7560.842510: 8,48 C R 281120 + 8 [0] > >>> <idle>-0 [001] dNh. 7560.842551: 8,48 C R 281128 + 8 [0] > >>> <idle>-0 [001] dNh. 7560.842556: 8,48 C R 281136 + 8 [0] > >>> <idle>-0 [001] dNh. 7560.842562: 8,48 C R 281144 + 8 [0] > >>> > >>> Shouldn't the total number of pending requests reach > >>> /sys/block/<dev>/queue/nr_requests ? > >>> > >>> The latter is of course equal to 8. > >> > >> With a scheduler, the depth is what the scheduler provides. You cannot > >> exceed the hardware queue depth in any situation. You just have 8 > >> requests available for scheduling, with a max of 4 being inflight on > >> the device side. > >> > >> If both were 4, for instance, then you would have nothing to schedule > >> with, as all of them could reside on the hardware side. That's why > >> the scheduler defaults to twice the hardware queue depth. > > > > The default twice to hw queue depth might not be reasonable for multi LUN. > > > > Maybe it should be set twice of sdev->queue_depth for SCSI or > > hw queue depth/hctx->nr_active. But either way may become complicated > > because both can be adjusted runtime. > > > > Could you please explain why it is not working (if it is not working) > in my example, where there should be only one LUN? I didn't say it isn't working, and I mean it isn't perfect. The hardware queue depth is host-wide, that means it is shared by all LUNs. Of course, lots of LUNs may be attached to one single HBA. You can setup this setting via 'modprobe scsi_debug max_luns=16 max_queue=4' easily, then all 4 LUNs share the 4 tags. Thanks, Ming Lei ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: confusion about nr of pending I/O requests 2018-12-19 10:32 ` Ming Lei @ 2018-12-19 11:45 ` Paolo Valente 0 siblings, 0 replies; 8+ messages in thread From: Paolo Valente @ 2018-12-19 11:45 UTC (permalink / raw) To: Ming Lei; +Cc: Jens Axboe, linux-block, Linus Walleij, Mark Brown, Ulf Hansson > Il giorno 19 dic 2018, alle ore 11:32, Ming Lei <tom.leiming@gmail.com> ha scritto: > > On Wed, Dec 19, 2018 at 2:18 PM Paolo Valente <paolo.valente@linaro.org> wrote: >> >> >> >>> Il giorno 19 dic 2018, alle ore 04:45, Ming Lei <tom.leiming@gmail.com> ha scritto: >>> >>> On Wed, Dec 19, 2018 at 2:52 AM Jens Axboe <axboe@kernel.dk> wrote: >>>> >>>> On 12/18/18 5:45 AM, Paolo Valente wrote: >>>>> Hi Jens, >>>>> sorry for the following silly question, but maybe you can solve very >>>>> quickly a doubt for which I'd spend much more time investigating. >>>>> >>>>> While doing some tests with scsi_debug, I've just seen that (at least) >>>>> with direct I/O, the maximum number of pending I/O requests (at least >>>>> in the I/O schedulers) is equal, unexpectedly, to the queue depth of >>>>> the drive and not to >>>>> /sys/block/<dev>/queue/nr_requests >>>>> >>>>> For example, after: >>>>> >>>>> sudo modprobe scsi_debug max_queue=4 >>>>> >>>>> and with fio executed as follows: >>>>> >>>>> job: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=20 >>>>> >>>>> I get this periodic trace, where four insertions are followed by four >>>>> completions, and so on, till the end of the I/O. This trace is taken >>>>> with none, but the result is the same with bfq. >>>>> >>>>> fio-20275 [001] d... 7560.655213: 8,48 I R 281088 + 8 [fio] >>>>> fio-20275 [001] d... 7560.655288: 8,48 I R 281096 + 8 [fio] >>>>> fio-20275 [001] d... 7560.655311: 8,48 I R 281104 + 8 [fio] >>>>> fio-20275 [001] d... 7560.655331: 8,48 I R 281112 + 8 [fio] >>>>> <idle>-0 [001] d.h. 7560.749868: 8,48 C R 281088 + 8 [0] >>>>> <idle>-0 [001] dNh. 7560.749912: 8,48 C R 281096 + 8 [0] >>>>> <idle>-0 [001] dNh. 7560.749928: 8,48 C R 281104 + 8 [0] >>>>> <idle>-0 [001] dNh. 7560.749934: 8,48 C R 281112 + 8 [0] >>>>> fio-20275 [001] d... 7560.750023: 8,48 I R 281120 + 8 [fio] >>>>> fio-20275 [001] d... 7560.750196: 8,48 I R 281128 + 8 [fio] >>>>> fio-20275 [001] d... 7560.750229: 8,48 I R 281136 + 8 [fio] >>>>> fio-20275 [001] d... 7560.750250: 8,48 I R 281144 + 8 [fio] >>>>> <idle>-0 [001] d.h. 7560.842510: 8,48 C R 281120 + 8 [0] >>>>> <idle>-0 [001] dNh. 7560.842551: 8,48 C R 281128 + 8 [0] >>>>> <idle>-0 [001] dNh. 7560.842556: 8,48 C R 281136 + 8 [0] >>>>> <idle>-0 [001] dNh. 7560.842562: 8,48 C R 281144 + 8 [0] >>>>> >>>>> Shouldn't the total number of pending requests reach >>>>> /sys/block/<dev>/queue/nr_requests ? >>>>> >>>>> The latter is of course equal to 8. >>>> >>>> With a scheduler, the depth is what the scheduler provides. You cannot >>>> exceed the hardware queue depth in any situation. You just have 8 >>>> requests available for scheduling, with a max of 4 being inflight on >>>> the device side. >>>> >>>> If both were 4, for instance, then you would have nothing to schedule >>>> with, as all of them could reside on the hardware side. That's why >>>> the scheduler defaults to twice the hardware queue depth. >>> >>> The default twice to hw queue depth might not be reasonable for multi LUN. >>> >>> Maybe it should be set twice of sdev->queue_depth for SCSI or >>> hw queue depth/hctx->nr_active. But either way may become complicated >>> because both can be adjusted runtime. >>> >> >> Could you please explain why it is not working (if it is not working) >> in my example, where there should be only one LUN? > > I didn't say it isn't working, and I mean it isn't perfect. > > The hardware queue depth is host-wide, that means it is shared by all LUNs. > Of course, lots of LUNs may be attached to one single HBA. You can setup > this setting via 'modprobe scsi_debug max_luns=16 max_queue=4' easily, > then all 4 LUNs share the 4 tags. > Ok, so you are talking about the opposite problem, in a sense. What I'm saying here is that tags had to be 8, as Jens pointed out, but they are 4. Thanks, Paolo > Thanks, > Ming Lei ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2018-12-19 11:45 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-12-18 12:45 confusion about nr of pending I/O requests Paolo Valente 2018-12-18 12:49 ` Paolo Valente 2018-12-18 18:50 ` Jens Axboe 2018-12-18 23:35 ` Paolo Valente 2018-12-19 3:45 ` Ming Lei 2018-12-19 6:17 ` Paolo Valente 2018-12-19 10:32 ` Ming Lei 2018-12-19 11:45 ` Paolo Valente
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).