Fio docs say io_submit_mode=offload with async IO is not reliable?

Fio docs say io_submit_mode=offload with async IO is not reliable?

[Nick Neumann]

I'm learning about io_submit_mode and avoiding the coordinated
omission problem. The fio docs cover the mode and give two examples of
using it. But there is a comment in the docs that says

"Note that this option cannot reliably be used with async IO engines."

I don't really understand why? Further, both examples in the docs of
using the option specify the libaio engine, which is asynchronous,
adding to my confusion.

Perhaps the comment means to say "sync" and not "async" - I could see
how pushing I/O to a pool of threads in the sync case could be odd,
but not necessarily "unreliable", so maybe not. Just trying to
understand what danger the comment is trying to warn of.

Thanks,
Nick

Re: Fio docs say io_submit_mode=offload with async IO is not reliable?

[Niklas Cassel]

On Tue, Feb 08, 2022 at 01:10:33PM -0600, Nick Neumann wrote:
> I'm learning about io_submit_mode and avoiding the coordinated
> omission problem. The fio docs cover the mode and give two examples of
> using it. But there is a comment in the docs that says
> 
> "Note that this option cannot reliably be used with async IO engines."
> 
> I don't really understand why? Further, both examples in the docs of
> using the option specify the libaio engine, which is asynchronous,
> adding to my confusion.
> 
> Perhaps the comment means to say "sync" and not "async" - I could see
> how pushing I/O to a pool of threads in the sync case could be odd,
> but not necessarily "unreliable", so maybe not. Just trying to
> understand what danger the comment is trying to warn of.

Original commit disallowed offload for all async ioengines:
https://git.kernel.dk/cgit/fio/commit/?id=abfd235a05330eae8444f0e4c0039bbdf9f160a3

Later it was changed to be disallowed only for io_uring:
https://git.kernel.dk/cgit/fio/commit/?id=8bfe330eb42739d503d35d0b7d96f98c5c544204

So I assume that offload works fine with libaio.

Perhaps the HOWTO, man page, and the comment in check_engine_ops()
should simply be updated?


Kind regards,
Niklas

Re: Fio docs say io_submit_mode=offload with async IO is not reliable?

[Nick Neumann]

On Tue, Feb 8, 2022 at 2:12 PM Niklas Cassel <Niklas.Cassel@wdc.com> wrote:
> Original commit disallowed offload for all async ioengines:
> https://git.kernel.dk/cgit/fio/commit/?id=abfd235a05330eae8444f0e4c0039bbdf9f160a3
>
> Later it was changed to be disallowed only for io_uring:
> https://git.kernel.dk/cgit/fio/commit/?id=8bfe330eb42739d503d35d0b7d96f98c5c544204
>
> So I assume that offload works fine with libaio.
>
> Perhaps the HOWTO, man page, and the comment in check_engine_ops()
> should simply be updated?
>
>
> Kind regards,
> Niklas

Ah, thank you very much for finding and sharing those. That is very
helpful. If Jens or Jeff Furlong has any memories of some of the cases
with io_uring that are problematic, it would be great to know - I
could include mention of them while updating the howto/man/comment
Niklas mentioned.

At a higher level, would it be considered "unusual" or "non-standard"
to use offload with libaio (or really any async engine)? I guess in
some sense it is, because when your using offload, you've got a pool
of iodepth threads/workers running to perform the io, so using libaio
is not going to help those workers, but will really just be adding
another layer of async overhead after the async interface already
provided by the offload pool. Further, a real world app that is using
a threadpool to handle its IO is probably not using libaio as well.
Does that line of thinking (async engine with offload option is
unusual) make sense?

Thanks,
Nick

Re: Fio docs say io_submit_mode=offload with async IO is not reliable?

[Nick Neumann]

On Tue, Feb 8, 2022 at 3:20 PM Nick Neumann <nick@pcpartpicker.com> wrote:
> Ah, thank you very much for finding and sharing those. That is very
> helpful. If Jens or Jeff Furlong has any memories of some of the cases
> with io_uring that are problematic, it would be great to know - I
> could include mention of them while updating the howto/man/comment
> Niklas mentioned.

I think I'm hitting one of those problematic cases now trying to run
one of the docs samples, but on windows. That is, this job file
(targeted at a disk device):

[fixed-rate-submit]
size=128m
rw=read
ioengine=windowsaio
iodepth=32
direct=1
# by setting the submit mode to offload, we can guarantee a fixed rate of
# submission regardless of what the device completion rate is.
io_submit_mode=offload
rate_iops=1000

just spins forever, never performing any reads.

Changing ioengine to sync (and deleting direct=1, necessary on
windows) allows the job to run to completion.