io_submit depth

io_submit depth

[Julien Desfossez]

Hi,

When stracing fio running on Linux (4.13) with libaio engine, I saw that 
only one request is sent in the io_submit system call regardless of the 
iodepth parameter value.

To see this:
strace -f fio --name=tmp --filename=/tmp/fio --filesize=8G \
--ioengine=libaio --bs=4k --iodepth=32 --rw=randwrite \
--runtime=10 2>&1 | grep io_submit

All of the lines look like this:
io_submit(140287955259392, 1, [{pwrite, fildes=3, str="...", 
nbytes=4096, offset=1035223040}]) = 1

I would have expected to see the second parameter to be 32.

I see that the iodepth parameter has an impact in how many requests are 
sent at the beginning (in this example, I see 32 requests sent), but 
after that the requests are sent one by one as soon as one completes. I 
understand this is done to keep the number of "requests in flight" 
steady, but this is not what I expected when setting iodepth.

So is there a way to run fio so that we can send multiple requests per 
io_submit call ?

Thanks,

Julien

Re: io_submit depth

[Julien Desfossez]

On 05/15/2018 12:32 PM, Sitsofe Wheeler wrote:
> Hello Julien,
> 
> On 15 May 2018 at 16:20, Julien Desfossez <ju@klipix.org> wrote:
>> Hi,
>>
>> When stracing fio running on Linux (4.13) with libaio engine, I saw that
>> only one request is sent in the io_submit system call regardless of the
>> iodepth parameter value.
>>
>> To see this:
>> strace -f fio --name=tmp --filename=/tmp/fio --filesize=8G \
>> --ioengine=libaio --bs=4k --iodepth=32 --rw=randwrite \
>> --runtime=10 2>&1 | grep io_submit
>>
>> All of the lines look like this:
>> io_submit(140287955259392, 1, [{pwrite, fildes=3, str="...", nbytes=4096,
>> offset=1035223040}]) = 1
>>
>> I would have expected to see the second parameter to be 32.
> 
> By the by, did you see the warnings related to libaio in the
> manual/HOWTO (http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-ioengine
> and http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-iodepth
> ) regarding buffering / not setting direct=1?
Yes I did and I usually benchmark with direct=1, I just removed 
parameters while doing this test.

>> I see that the iodepth parameter has an impact in how many requests are sent
>> at the beginning (in this example, I see 32 requests sent), but after that
>> the requests are sent one by one as soon as one completes. I understand this
>> is done to keep the number of "requests in flight" steady, but this is not
>> what I expected when setting iodepth.
>>
>> So is there a way to run fio so that we can send multiple requests per
>> io_submit call ?
> 
> Is iodepth_batch_submit
> (http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-arg-iodepth-batch-submit
> ) and following what you're looking for?
> 
Indeed it helps !

Apparently what I am looking for is:
--iodepth=32 --iodepth_batch=32 --iodepth_batch_complete=32

Thanks !

Julien