A fio job that just waits?

A fio job that just waits?

[Nick Neumann]

Is there a clean way to create a fio job that just waits for a fixed
time period? I'm fairly new to fio, and want a delay between two
consecutive jobs. I use "wait_for" to serialize as needed, and
expected "startdelay" to allow me to put the desired delay between the
two. But it looks like the startdelay for a job begins when fio starts
and not when the job referenced in a "wait_for" finishes.

I can play games with a job with a very low bandwidth limit (e.g.
1B/s) and "runtime" and "time_based", but it feels hacky.

Am I thinking about how to get a delay between two jobs wrong? Is
there a better way? I realize I could just run fio multiple times with
command line delays between the runs, but I'd like the results of the
runs to share the same time basis and output/log files.

Thanks,
Nick

Re: A fio job that just waits?

[Damien Le Moal]

On 2022/01/27 8:08, Nick Neumann wrote:
> Is there a clean way to create a fio job that just waits for a fixed
> time period? I'm fairly new to fio, and want a delay between two
> consecutive jobs. I use "wait_for" to serialize as needed, and
> expected "startdelay" to allow me to put the desired delay between the
> two. But it looks like the startdelay for a job begins when fio starts
> and not when the job referenced in a "wait_for" finishes.

The job that needs to be started after a delay should have both wait_for and
startdelay options. That should work.

> 
> I can play games with a job with a very low bandwidth limit (e.g.
> 1B/s) and "runtime" and "time_based", but it feels hacky.
> 
> Am I thinking about how to get a delay between two jobs wrong? Is
> there a better way? I realize I could just run fio multiple times with
> command line delays between the runs, but I'd like the results of the
> runs to share the same time basis and output/log files.
> 
> Thanks,
> Nick


-- 
Damien Le Moal
Western Digital Research

Re: A fio job that just waits?

[Nick Neumann]

On Wed, Jan 26, 2022 at 5:41 PM Damien Le Moal
<damien.lemoal@opensource.wdc.com> wrote:
> The job that needs to be started after a delay should have both wait_for and
> startdelay options. That should work.

So that's what I expected and hoped. But it doesn't seem to work that
way. For example, I tried this:

time sudo fio --ioengine=libaio --direct=1 --filename=/dev/nvme1n1
--rw=write --iodepth=16 --bs=128k --name=job1 --size=15GB
--runtime=10s --time_based --name=job2 -size=15GB --runtime=10s
--time_based --wait_for=job1 --startdelay=5s

The first job runs for 10 seconds writing. The second job does the
same but both waiting for job1 to complete, and with a startdelay of
5s. So I would expect the total command to take about 25 seconds = 10s
job 1 + 5s startdelay for job2 + 10s job2 . But instead the startdelay
appears to begin counting when fio begins, not when job2 becomes
eligible to run due to job1 completion. The output for the runtime of
the command above is 20.760s, with the second job starting immediately
when job1 completes.

Re: A fio job that just waits?

[Damien Le Moal]

On 1/29/22 02:02, Nick Neumann wrote:
> On Wed, Jan 26, 2022 at 5:41 PM Damien Le Moal
> <damien.lemoal@opensource.wdc.com> wrote:
>> The job that needs to be started after a delay should have both wait_for and
>> startdelay options. That should work.
> 
> So that's what I expected and hoped. But it doesn't seem to work that
> way. For example, I tried this:
> 
> time sudo fio --ioengine=libaio --direct=1 --filename=/dev/nvme1n1
> --rw=write --iodepth=16 --bs=128k --name=job1 --size=15GB
> --runtime=10s --time_based --name=job2 -size=15GB --runtime=10s
> --time_based --wait_for=job1 --startdelay=5s
> 
> The first job runs for 10 seconds writing. The second job does the
> same but both waiting for job1 to complete, and with a startdelay of
> 5s. So I would expect the total command to take about 25 seconds = 10s
> job 1 + 5s startdelay for job2 + 10s job2 . But instead the startdelay
> appears to begin counting when fio begins, not when job2 becomes
> eligible to run due to job1 completion. The output for the runtime of
> the command above is 20.760s, with the second job starting immediately
> when job1 completes.

OK. There may be a bug then, startdelay is not accounting for the fact
that the wait_for option is being used. Can you try the stonewall option
instead of wait_for ?


-- 
Damien Le Moal
Western Digital Research

Re: A fio job that just waits?

[Nick Neumann]

On Fri, Jan 28, 2022 at 6:16 PM Damien Le Moal
<damien.lemoal@opensource.wdc.com> wrote:
> OK. There may be a bug then, startdelay is not accounting for the fact
> that the wait_for option is being used. Can you try the stonewall option
> instead of wait_for ?

I'm seeing the same behavior when I replace wait_for with stonewall.
E.g., a time of just over 20 seconds for:
time sudo fio --ioengine=libaio --direct=1 --filename=/dev/nvme1n1
--rw=write --iodepth=16 --bs=128k --name=job1 --size=15GB
--runtime=10s --time_based --name=job2 -size=15GB --runtime=10s
--time_based --stonewall --startdelay=5s

This is also the behavior I'd expect (but not want of course) based on
the code in backend.c .

It's also the behavior I see when I put a single job with a start
delay in a file and invoke fio with the same file twice. E.g., foo.fio
with contents of:
[job]
ioengine=libaio
direct=1
filename=/dev/nvme1n1
rw=write
iodepth=16
bs=128k
size=15GB
runtime=10s
time_based=1
startdelay=5s

and then invoking

time sudo fio foo.fio foo.fio

gives a runtime of 25 seconds, when I'd expect 30 =  (5 second wait +
10 second execution ) * 2

In most of the cases it seems like the ETA is confused by the behavior
two.  ETA jumps instantaneously from 15 down to 10 when the first run
of foo.fio is complete and the second one begins.

Re: A fio job that just waits?

[Nick Neumann]

On Fri, Jan 28, 2022 at 5:09 PM Nick Neumann <nick@pcpartpicker.com> wrote:
> I was hopeful it would
> work on windows as well. While a cpuio task runs fine on windows,
> there is something wrong when you try to run a task after it.

I went ahead and filed a bug for the windows-only behavior where two
different engines don't get along:
https://github.com/axboe/fio/issues/1331

On Sat, Jan 29, 2022 at 12:52 AM Nick Neumann <nick@pcpartpicker.com> wrote:
> In most of the cases it seems like the ETA is confused by the behavior
> two.  ETA jumps instantaneously from 15 down to 10 when the first run
> of foo.fio is complete and the second one begins.

Thoughts on filing a bug for the original issue with the startdelay?
On the one hand, the current behavior seems wrong, or at least
unexpected, to me. But on the other, based on how the code in
backend.c is written, unless it used to be structured very
differently, I imagine this has been the behavior for a while. There
could be jobs/scripts depending on the behavior, so I wouldn't be
surprised if changing the behavior to what I want/expect isn't really
an option - instead the best I might hope for is a new option
(interjobdelay?) that provides the behavior I want.

Re: A fio job that just waits?

[Erwan Velu]

Le 27/01/2022 à 00:08, Nick Neumann a écrit :
> Is there a clean way to create a fio job that just waits for a fixed
> time period? I'm fairly new to fio, and want a delay between two
> consecutive jobs. I use "wait_for" to serialize as needed, and
> expected "startdelay" to allow me to put the desired delay between the
> two. But it looks like the startdelay for a job begins when fio starts
> and not when the job referenced in a "wait_for" finishes.
>
> I can play games with a job with a very low bandwidth limit (e.g.
> 1B/s) and "runtime" and "time_based", but it feels hacky.
>
> Am I thinking about how to get a delay between two jobs wrong? Is
> there a better way? I realize I could just run fio multiple times with
> command line delays between the runs, but I'd like the results of the
> runs to share the same time basis and output/log files.

Nick,

I've been adding an exec engine in fio that could be used to do any kind 
of behavior.

There is an example where I wanted to perform explicit sleeps between 
jobs to cool down the server : 
https://github.com/axboe/fio/blob/master/examples/exec.fio

You may try something like that 
https://github.com/axboe/fio/tree/master/tools/fiograph can help you 
graphing how jobs are interlaced.

If you look at 
https://github.com/axboe/fio/blob/master/examples/exec.png, it gives a 
visual representation of the sleep action in this example.


That may help you,

Erwan,

Re: A fio job that just waits?

[Nick Neumann]

On Thu, Jan 27, 2022 at 8:03 AM Erwan Velu <e.velu@criteo.com> wrote:
> I've been adding an exec engine in fio that could be used to do any kind
> of behavior.
>
> There is an example where I wanted to perform explicit sleeps between
> jobs to cool down the server :
> https://github.com/axboe/fio/blob/master/examples/exec.fio
>
> You may try something like that
> https://github.com/axboe/fio/tree/master/tools/fiograph can help you
> graphing how jobs are interlaced.
>
> If you look at
> https://github.com/axboe/fio/blob/master/examples/exec.png, it gives a
> visual representation of the sleep action in this example.

Oooh, that looks really nice, and so flexible too. I will definitely
give it a try. On linux I'm running ubuntu 20 so it's an older version
of fio by default, but I can definitely deal with that. I'm also
running fio on windows too though, so unfortunatley it doesn't help
there yet.

Thanks very much for pointing me to this.

Re: A fio job that just waits?

[Nick Neumann]

On Fri, Jan 28, 2022 at 11:11 AM Nick Neumann <nick@pcpartpicker.com> wrote:
> Oooh, that looks really nice, and so flexible too. I will definitely
> give it a try. On linux I'm running ubuntu 20 so it's an older version
> of fio by default, but I can definitely deal with that. I'm also
> running fio on windows too though, so unfortunately it doesn't help
> there yet.
>
> Thanks very much for pointing me to this.

Your pointer to the exec engine got me to look more carefully at the
other engines. While not nearly as extensible or pure, there is the
"cpuio" ioengine. Setting its cpuload to 1 and making a time_based job
with desired runtime seems to work pretty well. I was hopeful it would
work on windows as well. While a cpuio task runs fine on windows,
there is something wrong when you try to run a task after it. For
example, running this:

time sudo fio --ioengine=libaio --direct=1 --filename=/dev/nvme1n1
--rw=write --iodepth=16 --bs=128k --name=job1 --size=15GB
--runtime=10s --time_based --name=job2 --ioengine=cpuio --cpuload=1
--runtime=5s --time_based --wait_for=job1 --name=job3 --size=15GB
--runtime=10s --time_based --wait_for=job2

does exactly what I expect on linux. 10 seconds of write for job 1,
then 5 seconds of idle for job 2, then 10 seconds of write for job 3.
Total runtime 25 seconds. But the equivalent on windows (just changing
the global ioengine and filename):

.\fio.exe --thread --ioengine=windowsaio
--filename='\\.\PhysicalDrive1' --rw=write --iodepth=16 --bs=128k
--name=job1 --size=15GB --runtime=10s --time_based --name=job2
--ioengine=null --size=1GB --runtime=5s --time_based --wait_for=job1
--name=job3 --size=15GB --runtime=10s --time_based --wait_for=job2

finishes and never even starts job3. Total runtime 15 seconds.

It seems to be something with the cpuio option, because changing it to
the null engine with --size=1GB gives the expected idle delay (but
pollutes logs/stats with job2 which doesn't really write).

Also, just examining the code in backend.c, it indeed looks like
startdelay is evaluated based on time since fio began, not since the
job itself became eligible to run due to a wait_for finishing..