Total latency (lat) of a workload test

Total latency (lat) of a workload test

[Eliezer Nebab]

Hello,

I have the following workload:

fio --randrepeat=1 --ioengine=libaio --direct=1 --sync=1 --name=test --filename=random_readwrite_test.fio --overwrite=1 --iodepth=64 --size=5GB --readwrite=randrw --rwmixread=50 --rwmixwrite=50 --bs=32k --runtime=60 --time_based --ramp_time=15

With the following result:

read : io=1949.4MB, bw=33249KB/s, iops=1038 , runt= 60035msec
    slat (usec): min=4 , max=30100 , avg=23.92, stdev=240.73
    clat (msec): min=1 , max=80 , avg=31.59, stdev= 8.94
     lat (msec): min=1 , max=80 , avg=31.61, stdev= 8.94

 write: io=1830.3MB, bw=31219KB/s, iops=975 , runt= 60035msec
    slat (usec): min=5 , max=30150 , avg=35.61, stdev=293.88
    clat (msec): min=1 , max=80 , avg=31.91, stdev= 8.82
     lat (msec): min=1 , max=80 , avg=31.95, stdev= 8.81

This is a 2000 iops iscsi. The read operation shows a 31.61 (msec) lat and the write operation at 31.95 (msec).  What is the correct way to interpret the total latency of this fio workload?  Do you add the read lat (31.61) plus write lat (31.95) to get the overall latency for this workload?

Thanks in advance for your input.

Eliezer

Re: Total latency (lat) of a workload test

[Sitsofe Wheeler]

Hi,

On Mon, 19 Aug 2019 at 00:02, Eliezer Nebab
<eliezer@integritech.solutions> wrote:
>
> Hello,
>
> I have the following workload:
>
> fio --randrepeat=1 --ioengine=libaio --direct=1 --sync=1 --name=test --filename=random_readwrite_test.fio --overwrite=1 --iodepth=64 --size=5GB --readwrite=randrw --rwmixread=50 --rwmixwrite=50 --bs=32k --runtime=60 --time_based --ramp_time=15
>
> With the following result:
>
> read : io=1949.4MB, bw=33249KB/s, iops=1038 , runt= 60035msec
>     slat (usec): min=4 , max=30100 , avg=23.92, stdev=240.73
>     clat (msec): min=1 , max=80 , avg=31.59, stdev= 8.94
>      lat (msec): min=1 , max=80 , avg=31.61, stdev= 8.94
>
>  write: io=1830.3MB, bw=31219KB/s, iops=975 , runt= 60035msec
>     slat (usec): min=5 , max=30150 , avg=35.61, stdev=293.88
>     clat (msec): min=1 , max=80 , avg=31.91, stdev= 8.82
>      lat (msec): min=1 , max=80 , avg=31.95, stdev= 8.81
>
> This is a 2000 iops iscsi. The read operation shows a 31.61 (msec) lat and the write operation at 31.95 (msec).  What is the correct way to interpret the total latency of this fio workload?  Do you add the read lat (31.61) plus write lat (31.95) to get the overall latency for this workload?

I'm afraid I don't understand the question. I guess you're saying "can
I flatten the read and write latencies into one number by summation"
and my feeling would be no you can't because then it's not per I/O
data that you are talking about - you have two I/Os worth of latency
(I'm not statician though). I'd probably make some statement like "the
average per I/O latency with workload XYZ was approximately 32
milliseconds regardless of direction" (if you're about to publish
results don't forget about
https://github.com/axboe/fio/blob/master/MORAL-LICENSE ). Possibly
averaging the average is slightly more meaningful or taking the higher
of the two two numbers.

Just be aware it looks like the workload you're doing has a filesystem
between you and your disk and by default fio is going to be making
that "random_readwrite_test.fio" file in the current working
directory. If that's not a filesystem on the iSCSI disk you actually
wanted to test or it is a filesystem that ignores the direct flag
(https://stackoverflow.com/a/46377629/ ) then results may not be as
you expect etc. I'd recommend at least checking what you get doing I/O
against the block device directly but be aware that doing so will
DESTROY any data (e.g. like a filesystem) already there!

-- 
Sitsofe | http://sucs.org/~sits/

RE: Total latency (lat) of a workload test

[eliezer]

I forgot to mention that the  "--filename=random_readwrite_test.fio" is being written to the ISCSI filesystem, so thank you for bringing that up.

From your response, I should not sum the avg read latency and the avg write latency.  Instead, I should average the average or take the higher of the two two numbers as the total latency of my fio test.  Correct?

Thank you in advance.

-----Original Message-----
From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On Behalf Of Sitsofe Wheeler
Sent: Monday, August 19, 2019 1:03 AM
To: Eliezer Nebab
Cc: fio
Subject: Re: Total latency (lat) of a workload test

Hi,

On Mon, 19 Aug 2019 at 00:02, Eliezer Nebab
<eliezer@integritech.solutions> wrote:
>
> Hello,
>
> I have the following workload:
>
> fio --randrepeat=1 --ioengine=libaio --direct=1 --sync=1 --name=test --filename=random_readwrite_test.fio --overwrite=1 --iodepth=64 --size=5GB --readwrite=randrw --rwmixread=50 --rwmixwrite=50 --bs=32k --runtime=60 --time_based --ramp_time=15
>
> With the following result:
>
> read : io=1949.4MB, bw=33249KB/s, iops=1038 , runt= 60035msec
>     slat (usec): min=4 , max=30100 , avg=23.92, stdev=240.73
>     clat (msec): min=1 , max=80 , avg=31.59, stdev= 8.94
>      lat (msec): min=1 , max=80 , avg=31.61, stdev= 8.94
>
>  write: io=1830.3MB, bw=31219KB/s, iops=975 , runt= 60035msec
>     slat (usec): min=5 , max=30150 , avg=35.61, stdev=293.88
>     clat (msec): min=1 , max=80 , avg=31.91, stdev= 8.82
>      lat (msec): min=1 , max=80 , avg=31.95, stdev= 8.81
>
> This is a 2000 iops iscsi. The read operation shows a 31.61 (msec) lat and the write operation at 31.95 (msec).  What is the correct way to interpret the total latency of this fio workload?  Do you add the read lat (31.61) plus write lat (31.95) to get the overall latency for this workload?

I'm afraid I don't understand the question. I guess you're saying "can
I flatten the read and write latencies into one number by summation"
and my feeling would be no you can't because then it's not per I/O
data that you are talking about - you have two I/Os worth of latency
(I'm not statician though). I'd probably make some statement like "the
average per I/O latency with workload XYZ was approximately 32
milliseconds regardless of direction" (if you're about to publish
results don't forget about
https://github.com/axboe/fio/blob/master/MORAL-LICENSE ). Possibly
averaging the average is slightly more meaningful or taking the higher
of the two two numbers.

Just be aware it looks like the workload you're doing has a filesystem
between you and your disk and by default fio is going to be making
that "random_readwrite_test.fio" file in the current working
directory. If that's not a filesystem on the iSCSI disk you actually
wanted to test or it is a filesystem that ignores the direct flag
(https://stackoverflow.com/a/46377629/ ) then results may not be as
you expect etc. I'd recommend at least checking what you get doing I/O
against the block device directly but be aware that doing so will
DESTROY any data (e.g. like a filesystem) already there!

-- 
Sitsofe | http://sucs.org/~sits/

Re: Total latency (lat) of a workload test

[Bryan Gurney]

On Mon, Aug 19, 2019 at 2:03 AM Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>
> Hi,
>
> On Mon, 19 Aug 2019 at 00:02, Eliezer Nebab
> <eliezer@integritech.solutions> wrote:
> >
> > Hello,
> >
> > I have the following workload:
> >
> > fio --randrepeat=1 --ioengine=libaio --direct=1 --sync=1 --name=test --filename=random_readwrite_test.fio --overwrite=1 --iodepth=64 --size=5GB --readwrite=randrw --rwmixread=50 --rwmixwrite=50 --bs=32k --runtime=60 --time_based --ramp_time=15
> >
> > With the following result:
> >
> > read : io=1949.4MB, bw=33249KB/s, iops=1038 , runt= 60035msec
> >     slat (usec): min=4 , max=30100 , avg=23.92, stdev=240.73
> >     clat (msec): min=1 , max=80 , avg=31.59, stdev= 8.94
> >      lat (msec): min=1 , max=80 , avg=31.61, stdev= 8.94
> >
> >  write: io=1830.3MB, bw=31219KB/s, iops=975 , runt= 60035msec
> >     slat (usec): min=5 , max=30150 , avg=35.61, stdev=293.88
> >     clat (msec): min=1 , max=80 , avg=31.91, stdev= 8.82
> >      lat (msec): min=1 , max=80 , avg=31.95, stdev= 8.81
> >
> > This is a 2000 iops iscsi. The read operation shows a 31.61 (msec) lat and the write operation at 31.95 (msec).  What is the correct way to interpret the total latency of this fio workload?  Do you add the read lat (31.61) plus write lat (31.95) to get the overall latency for this workload?
>
> I'm afraid I don't understand the question. I guess you're saying "can
> I flatten the read and write latencies into one number by summation"
> and my feeling would be no you can't because then it's not per I/O
> data that you are talking about - you have two I/Os worth of latency
> (I'm not statician though). I'd probably make some statement like "the
> average per I/O latency with workload XYZ was approximately 32
> milliseconds regardless of direction" (if you're about to publish
> results don't forget about
> https://github.com/axboe/fio/blob/master/MORAL-LICENSE ). Possibly
> averaging the average is slightly more meaningful or taking the higher
> of the two two numbers.
>
> Just be aware it looks like the workload you're doing has a filesystem
> between you and your disk and by default fio is going to be making
> that "random_readwrite_test.fio" file in the current working
> directory.

The job in this example contains "--overwrite=1", which I'm guessing
would simply reuse the existing file and overwrite its contents during
the test.

However, I had a look at the fio HOWTO file, and I see this:

(begin excerpt from https://github.com/axboe/fio/blob/master/HOWTO )
.. option:: overwrite=bool
If true, writes to a file will always overwrite existing data. If the
file doesn't already exist, it will be created before the write phase
begins. If the file exists and is large enough for the specified write
phase, nothing will be done. Default: false.
(end excerpt)

I'm guessing that the phrase "nothing will be done" means that the
existing file will be used before the write phase begins, without
attempting to delete the file.  Is this correct?


Thanks,

Bryan


> If that's not a filesystem on the iSCSI disk you actually
> wanted to test or it is a filesystem that ignores the direct flag
> (https://stackoverflow.com/a/46377629/ ) then results may not be as
> you expect etc. I'd recommend at least checking what you get doing I/O
> against the block device directly but be aware that doing so will
> DESTROY any data (e.g. like a filesystem) already there!
>
> --
> Sitsofe | http://sucs.org/~sits/