Details about the usage of flow

Details about the usage of flow

[Tom Cameron]

All,

I recently filed https://github.com/axboe/fio/issues/1483 on github
describing my usage of multiple concurrent jobs to split random read
and random write so I can have a different number of reader and writer
threads. I was informed that what I actually want is to use "flow=" to
specify the ratio of activity between jobs.

My issue while testing the use of flow, is that it seems to be off by
a factor of 10x. For example, the following has two jobs, one for
readers and one for writers. The writer, as can be seen, has "numjobs"
set to 8, while the reader has "numjobs" set to 64. Flow is set to
"100" for reader, and "1" for writer. But from the output below we can
see the actual split is more like 786 to 1.

Jobs: 72 (f=72):
[w(8),r(64)][26.6%][r=1254MiB/s,w=1632KiB/s][r=321k,w=408 IOPS][eta
44m:11s]

Should I be dividing the flow value by the ratio of "numjobs" as well?
The command running obviously has an 8:1 ratio of readers to writers,
and I've defined a 100:1 ratio in the value of "flow", so am I
actually seeing an 800:1 here?

I suspect I'm missing a key piece of information, but the
documentation doesn't explicitly call this out. The line that makes me
think I may need to adjust for "numjobs" within each job is "The flow
parameter stands for how much should be added or subtracted to the
flow counter on each iteration of the main I/O loop.". This makes me
suspect that each of the threads spawned under a job definition is
spawned with it's own 100:1 ratio, and since there's an 8:1 numjobs
ratio, I end up seeing the above.

The command, in total, is below.

fio --output=fio-logs/m2test-01/20221102/manual-flow-test.write-1.qd-32/fio.manual-flow-test.log
\
--exitall_on_error --bs=4Ki --direct=1 --fadvise_hint=random
--filename=/mnt/testing/fio \
--group_reporting=1 --iodepth=32 --ioengine=libaio
--ioscheduler=mq-deadline --log_avg_msec=1000 \
--log_compression=1m --log_store_compressed=1 --ramp_time=10
--randrepeat=0 --runtime=1h --thread \
--time_based --write_bw_log=fio-logs/m2test-01/20221102/manual-flow-test.write-1.qd-32/fio-stress
\
--write_iops_log=fio-logs/m2test-01/20221102/manual-flow-test.write-1.qd-32/fio-stress
\
--write_lat_log=fio-logs/m2test-01/20221102/manual-flow-test.write-1.qd-32/fio-stress
\
--name=writer --new_group --stonewall --numjobs=8 --rw=randwrite
--flow=1 --exitall \
--name=reader --new_group --numjobs=64 --rw=randread --flow=100


Thanks for any advice,
Tom

Re: Details about the usage of flow

[Vincent Fu]

On 11/2/22 16:34, Tom Cameron wrote:
 > All,
 >
 > I recently filed https://github.com/axboe/fio/issues/1483 on github
 > describing my usage of multiple concurrent jobs to split random read
 > and random write so I can have a different number of reader and writer
 > threads. I was informed that what I actually want is to use "flow=" to
 > specify the ratio of activity between jobs.
 >
 > My issue while testing the use of flow, is that it seems to be off by
 > a factor of 10x. For example, the following has two jobs, one for
 > readers and one for writers. The writer, as can be seen, has "numjobs"
 > set to 8, while the reader has "numjobs" set to 64. Flow is set to
 > "100" for reader, and "1" for writer. But from the output below we can
 > see the actual split is more like 786 to 1.
 >
 > Jobs: 72 (f=72):
 > [w(8),r(64)][26.6%][r=1254MiB/s,w=1632KiB/s][r=321k,w=408 IOPS][eta
 > 44m:11s]
 >
 > Should I be dividing the flow value by the ratio of "numjobs" as well?
 > The command running obviously has an 8:1 ratio of readers to writers,
 > and I've defined a 100:1 ratio in the value of "flow", so am I
 > actually seeing an 800:1 here?
 >
 > I suspect I'm missing a key piece of information, but the
 > documentation doesn't explicitly call this out. The line that makes me
 > think I may need to adjust for "numjobs" within each job is "The flow
 > parameter stands for how much should be added or subtracted to the
 > flow counter on each iteration of the main I/O loop.". This makes me
 > suspect that each of the threads spawned under a job definition is
 > spawned with it's own 100:1 ratio, and since there's an 8:1 numjobs
 > ratio, I end up seeing the above.
 >
 > The command, in total, is below.
 >
 > fio 
--output=fio-logs/m2test-01/20221102/manual-flow-test.write-1.qd-32/fio.manual-flow-test.log
 > \
 > --exitall_on_error --bs=4Ki --direct=1 --fadvise_hint=random
 > --filename=/mnt/testing/fio \
 > --group_reporting=1 --iodepth=32 --ioengine=libaio
 > --ioscheduler=mq-deadline --log_avg_msec=1000 \
 > --log_compression=1m --log_store_compressed=1 --ramp_time=10
 > --randrepeat=0 --runtime=1h --thread \
 > --time_based 
--write_bw_log=fio-logs/m2test-01/20221102/manual-flow-test.write-1.qd-32/fio-stress
 > \
 > 
--write_iops_log=fio-logs/m2test-01/20221102/manual-flow-test.write-1.qd-32/fio-stress
 > \
 > 
--write_lat_log=fio-logs/m2test-01/20221102/manual-flow-test.write-1.qd-32/fio-stress
 > \
 > --name=writer --new_group --stonewall --numjobs=8 --rw=randwrite
 > --flow=1 --exitall \
 > --name=reader --new_group --numjobs=64 --rw=randread --flow=100
 >
 >
 > Thanks for any advice,
 > Tom

numjobs=X directs fio to behave as if you had specified the same job 
configuration X times. So your options create 8 writer jobs with flow=1 
and 64 reader jobs with flow=100.

If you want a 100:1 r/w ratio you will need to use rw=randrw and rwmixread.

The flow feature does not guarantee a specified r/w mix because your 
device may have different read and write performance. What flow does is 
to direct jobs to skip opportunities to issue IO using the specified 
weights. The resulting r/w mix depends on this of course but also 
depends on the relative r/w performance of your device.

Vincent