FIO performance measurement between volumes

FIO performance measurement between volumes

[Robert Balogh]

hello FIO experts,

I am a beginner in FIO, and I met with a problem.

Currently in my Ubuntu based server the "/dev/vdb1" volume is attached
to the "/home/batman/fio/cindervolume" path. FIO job is configured
like this

    [global]
    filename=fio-172_20_2_13
    directory=/home/batman/fio/cindervolume rw=rw
    rwmixread=60
    rwmixwrite=40
    bs=4k
    rate=500k
    direct=1
    numjobs=1
    time_based=1
    runtime=14d
    verify=crc32c
    continue_on_error=all
    group_reporting=1

    [file1]
    iodepth=1
    ; -- end job file --

FIO is started like this and works well:
    /usr/bin/fio --size=10G
--output=/home/batman/fio/cindervolume/fio-172_20_2_13-process.log
/home/batman/fioApp/fio-seq-RW.job &

FIO version I use: fio-3.25

My next step would be to attach a 2nd volume for example "/dev/vdc" to
"/home/batman/fio/cindervolume-2" path, and I would do performance
measurement between the volumes. I was checking the FIO user’s guide,
https://fio.readthedocs.io/en/latest/fio_doc.html but unfortunately I
cannot figure out which parameter might help me to solve this topic.

By-the-way is this possible to do with FIO? If so, could you please
help me by giving some direction/hints?

thanks for your help,

/Robi

Re: FIO performance measurement between volumes

[Damien Le Moal]

On 1/13/22 20:47, Robert Balogh wrote:
> hello FIO experts,
> 
> I am a beginner in FIO, and I met with a problem.
> 
> Currently in my Ubuntu based server the "/dev/vdb1" volume is attached
> to the "/home/batman/fio/cindervolume" path. FIO job is configured
> like this
> 
>     [global]
>     filename=fio-172_20_2_13
>     directory=/home/batman/fio/cindervolume rw=rw
>     rwmixread=60
>     rwmixwrite=40
>     bs=4k
>     rate=500k
>     direct=1
>     numjobs=1
>     time_based=1
>     runtime=14d
>     verify=crc32c
>     continue_on_error=all
>     group_reporting=1
> 
>     [file1]
>     iodepth=1
>     ; -- end job file --
> 
> FIO is started like this and works well:
>     /usr/bin/fio --size=10G
> --output=/home/batman/fio/cindervolume/fio-172_20_2_13-process.log
> /home/batman/fioApp/fio-seq-RW.job &
> 
> FIO version I use: fio-3.25
> 
> My next step would be to attach a 2nd volume for example "/dev/vdc" to
> "/home/batman/fio/cindervolume-2" path, and I would do performance
> measurement between the volumes. I was checking the FIO user’s guide,
> https://fio.readthedocs.io/en/latest/fio_doc.html but unfortunately I
> cannot figure out which parameter might help me to solve this topic.
> 
> By-the-way is this possible to do with FIO? If so, could you please
> help me by giving some direction/hints?

Move the directory=/... option from global section into the job section
and add a job for your other volume with directory= again added in that
job section to point to the new volume directory. The 2 jobs will run
simultaneously targetting different volumes, unless you add "stonewall",
in which case the jobs will run one after the other.

Or you could just write another fio script for the second volume.

It is not clear if you want to run the perf tests on the 2 volumes in
parallel or one after the other.

> 
> thanks for your help,
> 
> /Robi


-- 
Damien Le Moal
Western Digital Research

Re: FIO performance measurement between volumes

[Robert Balogh]

hello Damien,

Thanks for your quick reply. I am afraid I made a mistake to not
describe clearly what my goal is. So I can perform the FIO test for
the 2nd attached volume similar to what is done for the 1st volume.
But now I would like to do a performance measurement between the
volumes.

So my idea is, there is a e.g: 10G size file stored in volume1, and
this file is going to write to volume2, then read back. But now, I am
not sure this can be done with the FIO tool.

Thanks for your help.
/Robi

On Fri, Jan 14, 2022 at 12:20 AM Damien Le Moal
<damien.lemoal@opensource.wdc.com> wrote:
>
> On 1/13/22 20:47, Robert Balogh wrote:
> > hello FIO experts,
> >
> > I am a beginner in FIO, and I met with a problem.
> >
> > Currently in my Ubuntu based server the "/dev/vdb1" volume is attached
> > to the "/home/batman/fio/cindervolume" path. FIO job is configured
> > like this
> >
> >     [global]
> >     filename=fio-172_20_2_13
> >     directory=/home/batman/fio/cindervolume rw=rw
> >     rwmixread=60
> >     rwmixwrite=40
> >     bs=4k
> >     rate=500k
> >     direct=1
> >     numjobs=1
> >     time_based=1
> >     runtime=14d
> >     verify=crc32c
> >     continue_on_error=all
> >     group_reporting=1
> >
> >     [file1]
> >     iodepth=1
> >     ; -- end job file --
> >
> > FIO is started like this and works well:
> >     /usr/bin/fio --size=10G
> > --output=/home/batman/fio/cindervolume/fio-172_20_2_13-process.log
> > /home/batman/fioApp/fio-seq-RW.job &
> >
> > FIO version I use: fio-3.25
> >
> > My next step would be to attach a 2nd volume for example "/dev/vdc" to
> > "/home/batman/fio/cindervolume-2" path, and I would do performance
> > measurement between the volumes. I was checking the FIO user’s guide,
> > https://fio.readthedocs.io/en/latest/fio_doc.html but unfortunately I
> > cannot figure out which parameter might help me to solve this topic.
> >
> > By-the-way is this possible to do with FIO? If so, could you please
> > help me by giving some direction/hints?
>
> Move the directory=/... option from global section into the job section
> and add a job for your other volume with directory= again added in that
> job section to point to the new volume directory. The 2 jobs will run
> simultaneously targetting different volumes, unless you add "stonewall",
> in which case the jobs will run one after the other.
>
> Or you could just write another fio script for the second volume.
>
> It is not clear if you want to run the perf tests on the 2 volumes in
> parallel or one after the other.
>
> >
> > thanks for your help,
> >
> > /Robi
>
>
> --
> Damien Le Moal
> Western Digital Research

Re: FIO performance measurement between volumes

[Robert Balogh]

hello Ben,

Thanks for your reply. I will try out how dd works for me.

thanks again,
/Robi

On Fri, Jan 14, 2022 at 2:59 PM Ben England <bengland@redhat.com> wrote:
>
> dd can do this.   Beware of caching effects, you can drop cache + use dd conv=fsync  to ensure the data makes it to the target volume.
>
> On Fri, Jan 14, 2022 at 2:48 AM Robert Balogh <ethrbh@gmail.com> wrote:
>>
>> hello Damien,
>>
>> Thanks for your quick reply. I am afraid I made a mistake to not
>> describe clearly what my goal is. So I can perform the FIO test for
>> the 2nd attached volume similar to what is done for the 1st volume.
>> But now I would like to do a performance measurement between the
>> volumes.
>>
>> So my idea is, there is a e.g: 10G size file stored in volume1, and
>> this file is going to write to volume2, then read back. But now, I am
>> not sure this can be done with the FIO tool.
>>
>> Thanks for your help.
>> /Robi
>>
>> On Fri, Jan 14, 2022 at 12:20 AM Damien Le Moal
>> <damien.lemoal@opensource.wdc.com> wrote:
>> >
>> > On 1/13/22 20:47, Robert Balogh wrote:
>> > > hello FIO experts,
>> > >
>> > > I am a beginner in FIO, and I met with a problem.
>> > >
>> > > Currently in my Ubuntu based server the "/dev/vdb1" volume is attached
>> > > to the "/home/batman/fio/cindervolume" path. FIO job is configured
>> > > like this
>> > >
>> > >     [global]
>> > >     filename=fio-172_20_2_13
>> > >     directory=/home/batman/fio/cindervolume rw=rw
>> > >     rwmixread=60
>> > >     rwmixwrite=40
>> > >     bs=4k
>> > >     rate=500k
>> > >     direct=1
>> > >     numjobs=1
>> > >     time_based=1
>> > >     runtime=14d
>> > >     verify=crc32c
>> > >     continue_on_error=all
>> > >     group_reporting=1
>> > >
>> > >     [file1]
>> > >     iodepth=1
>> > >     ; -- end job file --
>> > >
>> > > FIO is started like this and works well:
>> > >     /usr/bin/fio --size=10G
>> > > --output=/home/batman/fio/cindervolume/fio-172_20_2_13-process.log
>> > > /home/batman/fioApp/fio-seq-RW.job &
>> > >
>> > > FIO version I use: fio-3.25
>> > >
>> > > My next step would be to attach a 2nd volume for example "/dev/vdc" to
>> > > "/home/batman/fio/cindervolume-2" path, and I would do performance
>> > > measurement between the volumes. I was checking the FIO user’s guide,
>> > > https://fio.readthedocs.io/en/latest/fio_doc.html but unfortunately I
>> > > cannot figure out which parameter might help me to solve this topic.
>> > >
>> > > By-the-way is this possible to do with FIO? If so, could you please
>> > > help me by giving some direction/hints?
>> >
>> > Move the directory=/... option from global section into the job section
>> > and add a job for your other volume with directory= again added in that
>> > job section to point to the new volume directory. The 2 jobs will run
>> > simultaneously targetting different volumes, unless you add "stonewall",
>> > in which case the jobs will run one after the other.
>> >
>> > Or you could just write another fio script for the second volume.
>> >
>> > It is not clear if you want to run the perf tests on the 2 volumes in
>> > parallel or one after the other.
>> >
>> > >
>> > > thanks for your help,
>> > >
>> > > /Robi
>> >
>> >
>> > --
>> > Damien Le Moal
>> > Western Digital Research
>>