Ques on FIO output (how to interpret iops)

Ques on FIO output (how to interpret iops)

[Learner Study]

[-- Attachment #1: Type: text/plain, Size: 2807 bytes --]

Hi:

How do I calculate total IOPS done by different FIO threads/jobs?

Following is output from FIO (with two threads)...

Each thread shows "bw" and "iops" numbers. Towards the end of output log,
there is "aggrb" which is sum of individual "bw" of threads, but "io" field
(3 line from the bottom of output below) doesn't seem to be sum of
individual thread iops....

For example, Thread_1 has bw: 15k iops: 29366
Thread_2 has bw: 15k iops: 29445
But, the summary towards the end (third line from bottom of output), says
"io=1,723MiB, aggrb=30,105KiB/s".

In this aggrb is good but io doesn't match up (it should be 58781) ! Can
someone please educate what am I missing?

Thanks a lot!

================================== FIO Output
=========================================
[root@localhost fio-1.24]# ./1.sh
job1: (g=0): rw=read, bs=512-512/512-512, ioengine=libaio, iodepth=4000
job2: (g=0): rw=read, bs=512-512/512-512, ioengine=libaio, iodepth=4000
Starting 2 processes
Jobs: 2 (f=2): [RR] [100.0% done] [ 30609/     0 kb/s] [eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=3389
  read : io=860MiB, bw=15,035KiB/s, iops=29,366, runt= 60003msec
    slat (usec): min=7, max=72,825, avg=20.43, stdev=219.35
    clat (msec): min=1, max=505, avg=136.08, stdev=43.51
    bw (KiB/s) : min= 4047, max=23724, per=49.91%, avg=15025.60,
stdev=3869.13
  cpu          : usr=14.77%, sys=70.29%, ctx=36920, majf=0, minf=514
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%,
>=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.1%
     issued r/w: total=1762085/0, short=0/0

     lat (msec): 2=0.01%, 4=0.01%, 10=0.01%, 20=0.02%, 50=0.03%
     lat (msec): 100=28.57%, 250=70.55%, 500=0.68%, 750=0.14%
job2: (groupid=0, jobs=1): err= 0: pid=3390
  read : io=863MiB, bw=15,076KiB/s, iops=29,445, runt= 60030msec
    slat (usec): min=7, max=90,307, avg=20.00, stdev=233.75
    clat (msec): min=28, max=617, avg=135.63, stdev=47.85
    bw (KiB/s) : min= 5039, max=23341, per=49.98%, avg=15047.44,
stdev=4046.40
  cpu          : usr=14.86%, sys=66.24%, ctx=35149, majf=0, minf=514
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%,
>=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.1%
     issued r/w: total=1767629/0, short=0/0

     lat (msec): 50=0.21%, 100=24.98%, 250=72.56%, 500=2.03%, 750=0.21%

Run status group 0 (all jobs):
   READ: io=1,723MiB, aggrb=30,105KiB/s, minb=15,035KiB/s, maxb=15,076KiB/s,
mint=60003msec, maxt=60030msec

Disk stats (read/write):
  sdb: ios=76944/0, merge=3452770/0, ticks=454753/0, in_queue=454712,
util=99.86%

[-- Attachment #2: Type: text/html, Size: 3082 bytes --]

Re: Ques on FIO output (how to interpret iops)

[Jens Axboe]

On Fri, Jun 19 2009, Learner Study wrote:
> Hi:
> 
> How do I calculate total IOPS done by different FIO threads/jobs?
> 
> Following is output from FIO (with two threads)...
> 
> Each thread shows "bw" and "iops" numbers. Towards the end of output log,
> there is "aggrb" which is sum of individual "bw" of threads, but "io" field
> (3 line from the bottom of output below) doesn't seem to be sum of
> individual thread iops....
> 
> For example, Thread_1 has bw: 15k iops: 29366
> Thread_2 has bw: 15k iops: 29445
> But, the summary towards the end (third line from bottom of output), says
> "io=1,723MiB, aggrb=30,105KiB/s".
> 
> In this aggrb is good but io doesn't match up (it should be 58781) ! Can
> someone please educate what am I missing?

Can you try a newer version? If the problem is still there, the best way
to get it fixed is to send my a simple job file that reproduces the
problem, and then I can easily fix it for you.

-- 
Jens Axboe

Re: Ques on FIO output (how to interpret iops)

[John Cagle]

[-- Attachment #1: Type: text/plain, Size: 1251 bytes --]

On Sat, Jun 20, 2009 at 1:27 AM, Jens Axboe <jens.axboe@oracle.com> wrote:

> On Fri, Jun 19 2009, Learner Study wrote:
> > Hi:
> >
> > How do I calculate total IOPS done by different FIO threads/jobs?
> >
> > Following is output from FIO (with two threads)...
> >
> > Each thread shows "bw" and "iops" numbers. Towards the end of output log,
> > there is "aggrb" which is sum of individual "bw" of threads, but "io"
> field
> > (3 line from the bottom of output below) doesn't seem to be sum of
> > individual thread iops....
> >
> > For example, Thread_1 has bw: 15k iops: 29366
> > Thread_2 has bw: 15k iops: 29445
> > But, the summary towards the end (third line from bottom of output), says
> > "io=1,723MiB, aggrb=30,105KiB/s".
> >
> > In this aggrb is good but io doesn't match up (it should be 58781) ! Can
> > someone please educate what am I missing?
>
> Can you try a newer version? If the problem is still there, the best way
> to get it fixed is to send my a simple job file that reproduces the
> problem, and then I can easily fix it for you.
>

I thought the "io=" summary at the end was the total amount of I/O performed
(in megabytes), not the total number of IOPS.

What were your block size and length of run for the example?

John

[-- Attachment #2: Type: text/html, Size: 1720 bytes --]

Re: Ques on FIO output (how to interpret iops)

[Jens Axboe]

[-- Attachment #1: Type: text/plain, Size: 1328 bytes --]

On 20/06/2009, at 15.21, John Cagle <jcagle@gmail.com> wrote:

> On Sat, Jun 20, 2009 at 1:27 AM, Jens Axboe <jens.axboe@oracle.com>  
> wrote:
> On Fri, Jun 19 2009, Learner Study wrote:
> > Hi:
> >
> > How do I calculate total IOPS done by different FIO threads/jobs?
> >
> > Following is output from FIO (with two threads)...
> >
> > Each thread shows "bw" and "iops" numbers. Towards the end of  
> output log,
> > there is "aggrb" which is sum of individual "bw" of threads, but  
> "io" field
> > (3 line from the bottom of output below) doesn't seem to be sum of
> > individual thread iops....
> >
> > For example, Thread_1 has bw: 15k iops: 29366
> > Thread_2 has bw: 15k iops: 29445
> > But, the summary towards the end (third line from bottom of  
> output), says
> > "io=1,723MiB, aggrb=30,105KiB/s".
> >
> > In this aggrb is good but io doesn't match up (it should be  
> 58781) ! Can
> > someone please educate what am I missing?
>
> Can you try a newer version? If the problem is still there, the best  
> way
> to get it fixed is to send my a simple job file that reproduces the
> problem, and then I can easily fix it for you.
>
> I thought the "io=" summary at the end was the total amount of I/O  
> performed (in megabytes), not the total number of IOPS.

It is, you are correct. Seems I misread the problem.

[-- Attachment #2: Type: text/html, Size: 2115 bytes --]

Re: Ques on FIO output (how to interpret iops)

[Learner Study]

[-- Attachment #1: Type: text/plain, Size: 1676 bytes --]

runtime = 60 and blocksize=512

The exact cmd is:
./fio --name=global --runtime=60 --rw=read --bs=512 --direct=1
--ioengine=aio --iodepth=4000 --loops=100 --name=job1 --filename=/dev/sdb
--name=job2 --filename=/dev/sdb

Thanks!


On Sat, Jun 20, 2009 at 7:12 AM, Jens Axboe <axboe@kernel.dk> wrote:

> On 20/06/2009, at 15.21, John Cagle <jcagle@gmail.com> wrote:
>
> On Sat, Jun 20, 2009 at 1:27 AM, Jens Axboe < <jens.axboe@oracle.com>
> jens.axboe@oracle.com> wrote:
>
>> On Fri, Jun 19 2009, Learner Study wrote:
>> > Hi:
>> >
>> > How do I calculate total IOPS done by different FIO threads/jobs?
>> >
>> > Following is output from FIO (with two threads)...
>> >
>> > Each thread shows "bw" and "iops" numbers. Towards the end of output
>> log,
>> > there is "aggrb" which is sum of individual "bw" of threads, but "io"
>> field
>> > (3 line from the bottom of output below) doesn't seem to be sum of
>> > individual thread iops....
>> >
>> > For example, Thread_1 has bw: 15k iops: 29366
>> > Thread_2 has bw: 15k iops: 29445
>> > But, the summary towards the end (third line from bottom of output),
>> says
>> > "io=1,723MiB, aggrb=30,105KiB/s".
>> >
>> > In this aggrb is good but io doesn't match up (it should be 58781) ! Can
>> > someone please educate what am I missing?
>>
>> Can you try a newer version? If the problem is still there, the best way
>> to get it fixed is to send my a simple job file that reproduces the
>> problem, and then I can easily fix it for you.
>>
>
> I thought the "io=" summary at the end was the total amount of I/O
> performed (in megabytes), not the total number of IOPS.
>
>
> It is, you are correct. Seems I misread the problem.
>

[-- Attachment #2: Type: text/html, Size: 2669 bytes --]

Re: Ques on FIO output (how to interpret iops)

[Learner Study]

[-- Attachment #1: Type: text/plain, Size: 2012 bytes --]

The summary says, io=1,723MiB, aggrb=30,105KiB/s

So, aggrb = io/runtime => 1723/60 => 28.716 => 28, 716 KiB/s (assuming 1KiB
= 1000 B)....still != aggrb shown in the result...

Thanks!


On Sat, Jun 20, 2009 at 1:38 PM, Learner Study <learner.study@gmail.com>wrote:

> runtime = 60 and blocksize=512
>
> The exact cmd is:
> ./fio --name=global --runtime=60 --rw=read --bs=512 --direct=1
> --ioengine=aio --iodepth=4000 --loops=100 --name=job1 --filename=/dev/sdb
> --name=job2 --filename=/dev/sdb
>
> Thanks!
>
>
>
> On Sat, Jun 20, 2009 at 7:12 AM, Jens Axboe <axboe@kernel.dk> wrote:
>
>> On 20/06/2009, at 15.21, John Cagle <jcagle@gmail.com> wrote:
>>
>> On Sat, Jun 20, 2009 at 1:27 AM, Jens Axboe < <jens.axboe@oracle.com>
>> jens.axboe@oracle.com> wrote:
>>
>>> On Fri, Jun 19 2009, Learner Study wrote:
>>> > Hi:
>>> >
>>> > How do I calculate total IOPS done by different FIO threads/jobs?
>>> >
>>> > Following is output from FIO (with two threads)...
>>> >
>>> > Each thread shows "bw" and "iops" numbers. Towards the end of output
>>> log,
>>> > there is "aggrb" which is sum of individual "bw" of threads, but "io"
>>> field
>>> > (3 line from the bottom of output below) doesn't seem to be sum of
>>> > individual thread iops....
>>> >
>>> > For example, Thread_1 has bw: 15k iops: 29366
>>> > Thread_2 has bw: 15k iops: 29445
>>> > But, the summary towards the end (third line from bottom of output),
>>> says
>>> > "io=1,723MiB, aggrb=30,105KiB/s".
>>> >
>>> > In this aggrb is good but io doesn't match up (it should be 58781) !
>>> Can
>>> > someone please educate what am I missing?
>>>
>>> Can you try a newer version? If the problem is still there, the best way
>>> to get it fixed is to send my a simple job file that reproduces the
>>> problem, and then I can easily fix it for you.
>>>
>>
>> I thought the "io=" summary at the end was the total amount of I/O
>> performed (in megabytes), not the total number of IOPS.
>>
>>
>> It is, you are correct. Seems I misread the problem.
>>
>
>

[-- Attachment #2: Type: text/html, Size: 3282 bytes --]

Re: Ques on FIO output (how to interpret iops)

[Learner Study]

[-- Attachment #1: Type: text/plain, Size: 2688 bytes --]

I'm using 1.24 version - does latest 1.27 fix this issue?

Thanks!

On Sat, Jun 20, 2009 at 8:31 PM, John Cagle <jcagle@gmail.com> wrote:

> Actually, 1KiB is 1024 bytes, but you're right -- the io values and the
> aggrb values don't seem to agree.
>
> What version of fio are you using?  There was a bug in one of the earlier
> version (around 1.24 or 1.25) where it was using 1000 instead of 1024 when
> doing some calculations.
>
> John
>
>
> On Sat, Jun 20, 2009 at 4:17 PM, Learner Study <learner.study@gmail.com>wrote:
>
>> The summary says, io=1,723MiB, aggrb=30,105KiB/s
>>
>> So, aggrb = io/runtime => 1723/60 => 28.716 => 28, 716 KiB/s (assuming
>> 1KiB = 1000 B)....still != aggrb shown in the result...
>>
>> Thanks!
>>
>>
>>
>> On Sat, Jun 20, 2009 at 1:38 PM, Learner Study <learner.study@gmail.com>wrote:
>>
>>> runtime = 60 and blocksize=512
>>>
>>> The exact cmd is:
>>> ./fio --name=global --runtime=60 --rw=read --bs=512 --direct=1
>>> --ioengine=aio --iodepth=4000 --loops=100 --name=job1 --filename=/dev/sdb
>>> --name=job2 --filename=/dev/sdb
>>>
>>> Thanks!
>>>
>>>
>>>
>>> On Sat, Jun 20, 2009 at 7:12 AM, Jens Axboe <axboe@kernel.dk> wrote:
>>>
>>>> On 20/06/2009, at 15.21, John Cagle <jcagle@gmail.com> wrote:
>>>>
>>>> On Sat, Jun 20, 2009 at 1:27 AM, Jens Axboe < <jens.axboe@oracle.com>
>>>> jens.axboe@oracle.com> wrote:
>>>>
>>>>> On Fri, Jun 19 2009, Learner Study wrote:
>>>>> > Hi:
>>>>> >
>>>>> > How do I calculate total IOPS done by different FIO threads/jobs?
>>>>> >
>>>>> > Following is output from FIO (with two threads)...
>>>>> >
>>>>> > Each thread shows "bw" and "iops" numbers. Towards the end of output
>>>>> log,
>>>>> > there is "aggrb" which is sum of individual "bw" of threads, but "io"
>>>>> field
>>>>> > (3 line from the bottom of output below) doesn't seem to be sum of
>>>>> > individual thread iops....
>>>>> >
>>>>> > For example, Thread_1 has bw: 15k iops: 29366
>>>>> > Thread_2 has bw: 15k iops: 29445
>>>>> > But, the summary towards the end (third line from bottom of output),
>>>>> says
>>>>> > "io=1,723MiB, aggrb=30,105KiB/s".
>>>>> >
>>>>> > In this aggrb is good but io doesn't match up (it should be 58781) !
>>>>> Can
>>>>> > someone please educate what am I missing?
>>>>>
>>>>> Can you try a newer version? If the problem is still there, the best
>>>>> way
>>>>> to get it fixed is to send my a simple job file that reproduces the
>>>>> problem, and then I can easily fix it for you.
>>>>>
>>>>
>>>> I thought the "io=" summary at the end was the total amount of I/O
>>>> performed (in megabytes), not the total number of IOPS.
>>>>
>>>>
>>>> It is, you are correct. Seems I misread the problem.
>>>>
>>>
>>>
>>
>

[-- Attachment #2: Type: text/html, Size: 4492 bytes --]

Re: Ques on FIO output (how to interpret iops)

[John Cagle]

[-- Attachment #1: Type: text/plain, Size: 2890 bytes --]

Probably.  Give it a try.

On Sun, Jun 21, 2009 at 12:15 AM, Learner Study <learner.study@gmail.com>wrote:

> I'm using 1.24 version - does latest 1.27 fix this issue?
>
> Thanks!
>
>
> On Sat, Jun 20, 2009 at 8:31 PM, John Cagle <jcagle@gmail.com> wrote:
>
>> Actually, 1KiB is 1024 bytes, but you're right -- the io values and the
>> aggrb values don't seem to agree.
>>
>> What version of fio are you using?  There was a bug in one of the earlier
>> version (around 1.24 or 1.25) where it was using 1000 instead of 1024 when
>> doing some calculations.
>>
>> John
>>
>>
>> On Sat, Jun 20, 2009 at 4:17 PM, Learner Study <learner.study@gmail.com>wrote:
>>
>>> The summary says, io=1,723MiB, aggrb=30,105KiB/s
>>>
>>> So, aggrb = io/runtime => 1723/60 => 28.716 => 28, 716 KiB/s (assuming
>>> 1KiB = 1000 B)....still != aggrb shown in the result...
>>>
>>> Thanks!
>>>
>>>
>>>
>>> On Sat, Jun 20, 2009 at 1:38 PM, Learner Study <learner.study@gmail.com>wrote:
>>>
>>>> runtime = 60 and blocksize=512
>>>>
>>>> The exact cmd is:
>>>> ./fio --name=global --runtime=60 --rw=read --bs=512 --direct=1
>>>> --ioengine=aio --iodepth=4000 --loops=100 --name=job1 --filename=/dev/sdb
>>>> --name=job2 --filename=/dev/sdb
>>>>
>>>> Thanks!
>>>>
>>>>
>>>>
>>>> On Sat, Jun 20, 2009 at 7:12 AM, Jens Axboe <axboe@kernel.dk> wrote:
>>>>
>>>>> On 20/06/2009, at 15.21, John Cagle <jcagle@gmail.com> wrote:
>>>>>
>>>>> On Sat, Jun 20, 2009 at 1:27 AM, Jens Axboe < <jens.axboe@oracle.com>
>>>>> jens.axboe@oracle.com> wrote:
>>>>>
>>>>>> On Fri, Jun 19 2009, Learner Study wrote:
>>>>>> > Hi:
>>>>>> >
>>>>>> > How do I calculate total IOPS done by different FIO threads/jobs?
>>>>>> >
>>>>>> > Following is output from FIO (with two threads)...
>>>>>> >
>>>>>> > Each thread shows "bw" and "iops" numbers. Towards the end of output
>>>>>> log,
>>>>>> > there is "aggrb" which is sum of individual "bw" of threads, but
>>>>>> "io" field
>>>>>> > (3 line from the bottom of output below) doesn't seem to be sum of
>>>>>> > individual thread iops....
>>>>>> >
>>>>>> > For example, Thread_1 has bw: 15k iops: 29366
>>>>>> > Thread_2 has bw: 15k iops: 29445
>>>>>> > But, the summary towards the end (third line from bottom of output),
>>>>>> says
>>>>>> > "io=1,723MiB, aggrb=30,105KiB/s".
>>>>>> >
>>>>>> > In this aggrb is good but io doesn't match up (it should be 58781) !
>>>>>> Can
>>>>>> > someone please educate what am I missing?
>>>>>>
>>>>>> Can you try a newer version? If the problem is still there, the best
>>>>>> way
>>>>>> to get it fixed is to send my a simple job file that reproduces the
>>>>>> problem, and then I can easily fix it for you.
>>>>>>
>>>>>
>>>>> I thought the "io=" summary at the end was the total amount of I/O
>>>>> performed (in megabytes), not the total number of IOPS.
>>>>>
>>>>>
>>>>> It is, you are correct. Seems I misread the problem.
>>>>>
>>>>
>>>>
>>>
>>
>

[-- Attachment #2: Type: text/html, Size: 4929 bytes --]

Re: Ques on FIO output (how to interpret iops)

[Jens Axboe]

On 20/06/2009, at 15.21, John Cagle <jcagle@gmail.com> wrote:

> On Sat, Jun 20, 2009 at 1:27 AM, Jens Axboe <jens.axboe@oracle.com>  
> wrote:
> On Fri, Jun 19 2009, Learner Study wrote:
> > Hi:
> >
> > How do I calculate total IOPS done by different FIO threads/jobs?
> >
> > Following is output from FIO (with two threads)...
> >
> > Each thread shows "bw" and "iops" numbers. Towards the end of  
> output log,
> > there is "aggrb" which is sum of individual "bw" of threads, but  
> "io" field
> > (3 line from the bottom of output below) doesn't seem to be sum of
> > individual thread iops....
> >
> > For example, Thread_1 has bw: 15k iops: 29366
> > Thread_2 has bw: 15k iops: 29445
> > But, the summary towards the end (third line from bottom of  
> output), says
> > "io=1,723MiB, aggrb=30,105KiB/s".
> >
> > In this aggrb is good but io doesn't match up (it should be  
> 58781) ! Can
> > someone please educate what am I missing?
>
> Can you try a newer version? If the problem is still there, the best  
> way
> to get it fixed is to send my a simple job file that reproduces the
> problem, and then I can easily fix it for you.
>
> I thought the "io=" summary at the end was the total amount of I/O  
> performed (in megabytes), not the total number of IOPS.

It is, you are correct. Seems I misread the problem.