Re: Guidance to learn more about fio tool

[Erwan Velu <e.velu@criteo.com>]

Le 31/07/2021 à 11:39, CED17I048 SJ HARINI a écrit :
> Hi,
> [...]
Good to see new comers.
>
> What have I done so far to understand the tool and the source code better:
> 1) Tried running  fio (installed using : sudo apt-get install fio)
> from command line (ubuntu) with --debug=all.
>      A lot of log information was printed in the terminal.
> 2) Cloned the source code from github fio repository. Installed
> sourcetrail and indexed all the files in fio-master. But there were
> many errors while indexing the files, so the visualization was
> somewhat incomplete. In case anyone has tried using sourcetrail, would
> love to know your experience.
I never used it sorry. I'm using atom or vim directly on the source tree.
> 3) Tried running many test jobs using fio.
> 4) Read up various resources available online about using fio and
> understanding its output.

That's a good start !


>
> Some doubts:
> 1) To understand the underlying logic about how the tool works, what
> concepts should I learn about prior? - From my reading about the fio
> tool, I came across AIO engine, nvme and learned that the hard
> disk/storage device also contains a hardware cache. So sometimes when
> we want to stress the hard disk/spin disk, we are actually stressing
> the cache. How do I disable the hardware cache using the fio tool ?
> Will --validate or --direct parameters take care of this ?

First you need to differentiate if you test the filesystem 
(/mnt/something) or the block device (/dev/sdx).

And yes, --direct bypass the caching.


> 2) I also came across some tools like IOStat, IO Meter, for storage
> device performance. How is FIO superior to these tools?

To me, it's real opensource, it has many backends, it's scriptable.

So barely we can test any kind of configuration with fio and if one is 
missing, let's add it.

> 3) When I run fio for random read and set the file option : example :
> --size = 20M, a file is created, which is not readable (i'm guessing
> it is a binary file). What does this file contain?
> And why is a file getting created even if rw is set to read? How to
> read this test file created by fio? My understanding is that FIO reads
> and writes random data using a random number generator, so shouldn't
> the data be readable.

To read a file, it needs to be created first right ?

I would say, why do you care about the content of it ?

If you want to check the consistency of the file, there is a 
verification mode in fio.

You can read the documentation around it if interested : 
https://fio.readthedocs.io/en/latest/fio_doc.html#verification

> 4) Is there any design diagram or documentation to understand the
> logic of how fio works?
I don't think so
> 5) How do I create a debug version of fio source code ? I was hoping
> that using breakpoints and debugging may help me identify the flow of
> the code.
You can do that, but reading the code is probably easier. With all the 
threads, the debugging will not be really trivial I think.
> 6) How to find the optimal io depth? Is queue depth same as fio iodepth?

The iodepth is usually linked to the device you target. Starts from the 
hardware specs and make your tries ;)

https://fio.readthedocs.io/en/latest/fio_doc.html#i-o-depth

https://www.kernel.org/doc/html/latest/block/blk-mq.html

> 7) For zonemode = strided, does zonesize represent the stride length?
> My understanding of stride length is the gap between each read/write.
I have no experience here.
> 8) I wanted to check if whatever data was written is correct or not,
> so I ran the basic-verify.fio job file. But I didn't get any output
> saying the data written was also read, and that there were no errors.
> How do I know if there were errors ?
I don't know the expected behavior here.
>
> If you could recommend any useful resources to better my
> understanding, that would be highly appreciated.
> Even though I do not have a clear understanding of how the tool works,
> I am eager to learn as much as possible and hope to gain an enriching
> learning experience.
> I hope to gain a good understanding of the tool and want to create a
> documentation with flowchart of how fio works as my first contribution
> (so it's easier for complete beginners like me to understand better)
> Thank you for taking the time and reading this mail, awaiting your response.

That's a great idea to tackle the points that you face when entering a 
project so the following ones will have a better experience.


Erwan,