Re: fio question

[Vincent Fu <vincentfu@gmail.com>]

On 12/27/22 18:50, M Kelly wrote:
> Hi,
> 
> Thank you for fio.
> Apologies if this is the wrong place to ask.
> 
> I am running fio on a laptop with an nvme ssd and have a question
> about differences between results.
> 
> test.1 - test.8 are each 900 Mb files of random data created beforehand
> 
> run 1 -
> 
> echo 3 | sudo tee /proc/sys/vm/drop_caches
> 
> fio --name=fiotest
> --filename=test.1:test.2:test.3:test.4:test.5:test.6:test.7:test.8
> -gtod_reduce=1 --size=900M --rw=randread --bs=8K --direct=0
> --numjobs=8 --invalidate=0 --ioengine=psync
> 
> results:
> 
>     iops        : min= 6794, max=86407, avg=26989.80, stdev=33609.35, samples=5
>     iops        : min= 6786, max=84998, avg=26818.40, stdev=32972.35, samples=5
>     iops        : min= 6816, max=83273, avg=26422.20, stdev=32200.02, samples=5
>     iops        : min= 6838, max=88227, avg=27462.20, stdev=34405.56, samples=5
>     iops        : min= 6868, max=85788, avg=26870.00, stdev=33332.10, samples=5
>     iops        : min= 6818, max=84768, avg=26719.60, stdev=32880.23, samples=5
>     iops        : min= 6810, max=86326, avg=27005.60, stdev=33581.89, samples=5
>     iops        : min= 6856, max=81411, avg=26165.00, stdev=31345.85, samples=5
>     READ: bw=2644MiB/s (2773MB/s), 331MiB/s-332MiB/s (347MB/s-348MB/s),
> io=7200MiB (7550MB), run=2709-2723msec
> 
> run 2 -
> 
> echo 3 | sudo tee /proc/sys/vm/drop_caches
> 
> fio --name=fiotest1 --filename=test.1 -gtod_reduce=1 --size=900M
> --rw=randread --bs=8K --direct=0 --numjobs=1 --invalidate=0
> --ioengine=psync &
> fio --name=fiotest2 --filename=test.2 -gtod_reduce=1 --size=900M
> --rw=randread --bs=8K --direct=0 --numjobs=1 --invalidate=0
> --ioengine=psync &
> fio --name=fiotest3 --filename=test.3 -gtod_reduce=1 --size=900M
> --rw=randread --bs=8K --direct=0 --numjobs=1 --invalidate=0
> --ioengine=psync &
> fio --name=fiotest4 --filename=test.4 -gtod_reduce=1 --size=900M
> --rw=randread --bs=8K --direct=0 --numjobs=1 --invalidate=0
> --ioengine=psync &
> fio --name=fiotest5 --filename=test.5 -gtod_reduce=1 --size=900M
> --rw=randread --bs=8K --direct=0 --numjobs=1 --invalidate=0
> --ioengine=psync &
> fio --name=fiotest6 --filename=test.6 -gtod_reduce=1 --size=900M
> --rw=randread --bs=8K --direct=0 --numjobs=1 --invalidate=0
> --ioengine=psync &
> fio --name=fiotest7 --filename=test.7 -gtod_reduce=1 --size=900M
> --rw=randread --bs=8K --direct=0 --numjobs=1 --invalidate=0
> --ioengine=psync &
> fio --name=fiotest8 --filename=test.8 -gtod_reduce=1 --size=900M
> --rw=randread --bs=8K --direct=0 --numjobs=1 --invalidate=0
> --ioengine=psync &
> wait
> 
> results (from each) -
> 
>    iops        : min= 5836, max= 6790, avg=6481.37, stdev=140.90, samples=35
>    READ: bw=50.6MiB/s (53.1MB/s), 50.6MiB/s-50.6MiB/s
> (53.1MB/s-53.1MB/s), io=900MiB (944MB), run=17777-17777msec
>    iops        : min= 5924, max= 6784, avg=6476.54, stdev=126.85, samples=35
>    READ: bw=50.6MiB/s (53.1MB/s), 50.6MiB/s-50.6MiB/s
> (53.1MB/s-53.1MB/s), io=900MiB (944MB), run=17785-17785msec
>    iops        : min= 5878, max= 6766, avg=6475.23, stdev=138.57, samples=35
>    READ: bw=50.6MiB/s (53.0MB/s), 50.6MiB/s-50.6MiB/s
> (53.0MB/s-53.0MB/s), io=900MiB (944MB), run=17792-17792msec
>    iops        : min= 5866, max= 6788, avg=6478.74, stdev=139.18, samples=35
>    READ: bw=50.6MiB/s (53.1MB/s), 50.6MiB/s-50.6MiB/s
> (53.1MB/s-53.1MB/s), io=900MiB (944MB), run=17782-17782msec
>    iops        : min= 5890, max= 6782, avg=6477.91, stdev=135.21, samples=35
>    READ: bw=50.6MiB/s (53.1MB/s), 50.6MiB/s-50.6MiB/s
> (53.1MB/s-53.1MB/s), io=900MiB (944MB), run=17783-17783msec
>    iops        : min= 5838, max= 6790, avg=6450.11, stdev=137.17, samples=35
>    READ: bw=50.3MiB/s (52.7MB/s), 50.3MiB/s-50.3MiB/s
> (52.7MB/s-52.7MB/s), io=900MiB (944MB), run=17892-17892msec
>    iops        : min= 5922, max= 6750, avg=6454.06, stdev=125.76, samples=35
>    READ: bw=50.3MiB/s (52.8MB/s), 50.3MiB/s-50.3MiB/s
> (52.8MB/s-52.8MB/s), io=900MiB (944MB), run=17880-17880msec
>    iops        : min= 5916, max= 6724, avg=6465.94, stdev=127.54, samples=35
>    READ: bw=50.5MiB/s (52.0MB/s), 50.5MiB/s-50.5MiB/s
> (52.0MB/s-52.0MB/s), io=900MiB (944MB), run=17816-17816msec
> 
> Question: if I am really doing 900 MiB of random reads from 8 separate
> files in both tests, and the page-cache was empty before each test,
> does this difference in performance make sense or am I reading the
> results incorrectly ?
> 
> thank you for any info/ advice, suggestions,
> -mark


Your first test directs fio to create a single job that reads from the 
files in a round robin fashion.

The second test has eight independent jobs reading separately from the 
eight files as fast as they can.

It's not a surprise that the results are different.

Vincent