Problems with check_min_rate and io_submit_mode=offload ?

Problems with check_min_rate and io_submit_mode=offload ?

[Nick Neumann]

I started trying to use the rate_min and rate_iops_min options. I was
trying to use them with io_submit_mode=offload. There appear to be
some significant issues with it though.

The first is that it will intermittently assert and generally ignore
any minimum specified rate. Why? Because in do_io in backend.c, in the
IO_OFFLOAD case, comp_time is never set. This means that later when
check_min_rate is passed comp_time, it is checking using a time for
"now" that is an uninitialized variable. Sometimes that will trip the
assert in rel_time_since, sometimes not. And when not, any rate
checking is not going to be right.

So I added a call to fio_gettime in the IO_OFFLOAD case under the
if(should_check_rate(td)...)

Now the job will catch when rate_min is not met, but something goes
wrong in the shutdown. Fio ends up in a state where IO appears to
stop, but fio doesn't exit, and ctrl-C will not cause an exit either.
This is because td->error is (rightly) set, so the code after the main
loop in do_io will enter the "else cleanup_pending_aio(td)" and hang
in its second call to io_u_queued_complete.

I think the right fix is to call workqueue_flush before
cleanup_pending_aio (in the error, offload case), so that there is
nothing left in the work queues and cleanup can proceed as it does in
the non-offload case.

Two questions before I submit a PR for this:
1) Does adding the fio_gettime() call in the offload case seem right?
2) Does adding the workqueue_flush in the error case before cleanup seem right?

Thanks,
Nick