Monitor IO while fio is running

Monitor IO while fio is running

[Kurt_Dorsey]

I may just not know how to do this
I know about the ability to touch /tmp/fio-dump-status which will dump the output of fio to a file or stdout while the test is running.  What I am looking for is to be able to dump the write_iops_log while fio is running so that I can see the point in time average IOps for a given interval
For example if I have log_avg_msec=60000 it will write an iops.log file with 60second intervals, unfortunately this file doesn't get created until fio exits.  Is there a way to get this to dump on a regular basis (or by touching a file in /tmp) so that I can tail the output?

I am looking for a way to get point in time status via a script and the output from tmp-dump-status only gives me averages from the start of time rather than the last interval

Thank you,
Kurt

RE: Monitor IO while fio is running

[Jeff Furlong]

You might add --eta=always and --eta-newline=1 to get the current IOPs rate every 1s, printed to stdout.  You can parse that current IOPs rate via a script and do something.  I believe the IOPs rate is recalculated every 1s, not the average from beginning of test.

Not sure why --eta and --eta-newline is not in the latest HOWTO notes.

Regards,
Jeff

-----Original Message-----
From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On Behalf Of Kurt_Dorsey@Dell.com
Sent: Saturday, March 12, 2016 2:20 PM
To: fio@vger.kernel.org
Subject: Monitor IO while fio is running

I may just not know how to do this
I know about the ability to touch /tmp/fio-dump-status which will dump the output of fio to a file or stdout while the test is running.  What I am looking for is to be able to dump the write_iops_log while fio is running so that I can see the point in time average IOps for a given interval For example if I have log_avg_msec=60000 it will write an iops.log file with 60second intervals, unfortunately this file doesn't get created until fio exits.  Is there a way to get this to dump on a regular basis (or by touching a file in /tmp) so that I can tail the output?

I am looking for a way to get point in time status via a script and the output from tmp-dump-status only gives me averages from the start of time rather than the last interval

Thank you,
Kurt
--
To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:

This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system.

RE: Monitor IO while fio is running

[Kurt_Dorsey]

Thank you Jeff for the reply.

I have been playing around with this but unfortunately I cannot get the output to be outputted anywhere other than the screen / terminal.  I  have tried redirecting it to a txt file and parsing with with a python script but nothing ever gets output. (It only outputs to the screen)

I haven't done C programming in years but I am trying to figure out from the code why this is.  I see in the eta.c file

void display_thread_status(struct jobs_eta *je)
{
...
       printf("%s", output);

        if (!eta_new_line_init) {
                fio_gettime(&disp_eta_new_line, NULL);
                eta_new_line_init = 1;
        } else if (eta_new_line && mtime_since_now(&disp_eta_new_line) > eta_new_line) {
                fio_gettime(&disp_eta_new_line, NULL);
                eta_new_line_pending = 1;
        }

        fflush(stdout);


It prints everything to the screen but I cannot redirect it at all

Thoughts?
Thank you,
Kurt


-----Original Message-----
From: Jeff Furlong [mailto:jeff.furlong@hgst.com] 
Sent: Monday, March 14, 2016 12:56 PM
To: Dorsey, Kurt ; fio@vger.kernel.org
Subject: RE: Monitor IO while fio is running

You might add --eta=always and --eta-newline=1 to get the current IOPs rate every 1s, printed to stdout. You can parse that current IOPs rate via a script and do something. I believe the IOPs rate is recalculated every 1s, not the average from beginning of test.

Not sure why --eta and --eta-newline is not in the latest HOWTO notes.

Regards,
Jeff

-----Original Message-----
From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On Behalf Of Kurt_Dorsey@Dell.com
Sent: Saturday, March 12, 2016 2:20 PM
To: fio@vger.kernel.org
Subject: Monitor IO while fio is running

I may just not know how to do this
I know about the ability to touch /tmp/fio-dump-status which will dump the output of fio to a file or stdout while the test is running. What I am looking for is to be able to dump the write_iops_log while fio is running so that I can see the point in time average IOps for a given interval For example if I have log_avg_msec=60000 it will write an iops.log file with 60second intervals, unfortunately this file doesn't get created until fio exits. Is there a way to get this to dump on a regular basis (or by touching a file in /tmp) so that I can tail the output?

I am looking for a way to get point in time status via a script and the output from tmp-dump-status only gives me averages from the start of time rather than the last interval

Thank you,
Kurt
--
To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:

This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system.

Re: Monitor IO while fio is running

[Kudryavtsev, Andrey O]

That�s how FIO Visualizer was designed. 01.org/fio-visualizer
It parses in realtime the --minimum output string.



-- 
Andrey Kudryavtsev,

SSD Solution Architect
Intel Corp. 
inet: 83564353
work: +1-916-356-4353
mobile: +1-916-221-2281




On 3/14/16, 10:55 AM, "fio-owner@vger.kernel.org on behalf of Jeff
Furlong" <fio-owner@vger.kernel.org on behalf of jeff.furlong@hgst.com>
wrote:

>You might add --eta=always and --eta-newline=1 to get the current IOPs
>rate every 1s, printed to stdout.  You can parse that current IOPs rate
>via a script and do something.  I believe the IOPs rate is recalculated
>every 1s, not the average from beginning of test.
>
>Not sure why --eta and --eta-newline is not in the latest HOWTO notes.
>
>Regards,
>Jeff
>
>-----Original Message-----
>From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On
>Behalf Of Kurt_Dorsey@Dell.com
>Sent: Saturday, March 12, 2016 2:20 PM
>To: fio@vger.kernel.org
>Subject: Monitor IO while fio is running
>
>I may just not know how to do this
>I know about the ability to touch /tmp/fio-dump-status which will dump
>the output of fio to a file or stdout while the test is running.  What I
>am looking for is to be able to dump the write_iops_log while fio is
>running so that I can see the point in time average IOps for a given
>interval For example if I have log_avg_msec=60000 it will write an
>iops.log file with 60second intervals, unfortunately this file doesn't
>get created until fio exits.  Is there a way to get this to dump on a
>regular basis (or by touching a file in /tmp) so that I can tail the
>output?
>
>I am looking for a way to get point in time status via a script and the
>output from tmp-dump-status only gives me averages from the start of time
>rather than the last interval
>
>Thank you,
>Kurt
>--
>To unsubscribe from this list: send the line "unsubscribe fio" in the
>body of a message to majordomo@vger.kernel.org More majordomo info at
>http://vger.kernel.org/majordomo-info.html
>Western Digital Corporation (and its subsidiaries) E-mail Confidentiality
>Notice & Disclaimer:
>
>This e-mail and any files transmitted with it may contain confidential or
>legally privileged information of WDC and/or its affiliates, and are
>intended solely for the use of the individual or entity to which they are
>addressed. If you are not the intended recipient, any disclosure,
>copying, distribution or any action taken or omitted to be taken in
>reliance on it, is prohibited. If you have received this e-mail in error,
>please notify the sender immediately and delete the e-mail in its
>entirety from your system.
>
>--
>To unsubscribe from this list: send the line "unsubscribe fio" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Monitor IO while fio is running

[Jeff Furlong]

No need to redirect if you're in python.  You can invoke fio from subprocess, then read the subprocess's output in realtime:

subprocess.stdout.readline().strip()

would get you the eta newline outputs, which you can analyze, print to screen, etc. from python.

Regards,
Jeff


-----Original Message-----
From: Kurt_Dorsey@Dell.com [mailto:Kurt_Dorsey@Dell.com] 
Sent: Monday, March 14, 2016 11:03 AM
To: Jeff Furlong <jeff.furlong@hgst.com>; fio@vger.kernel.org
Subject: RE: Monitor IO while fio is running

Thank you Jeff for the reply.

I have been playing around with this but unfortunately I cannot get the output to be outputted anywhere other than the screen / terminal.  I  have tried redirecting it to a txt file and parsing with with a python script but nothing ever gets output. (It only outputs to the screen)

I haven't done C programming in years but I am trying to figure out from the code why this is.  I see in the eta.c file

void display_thread_status(struct jobs_eta *je) { ...
       printf("%s", output);

        if (!eta_new_line_init) {
                fio_gettime(&disp_eta_new_line, NULL);
                eta_new_line_init = 1;
        } else if (eta_new_line && mtime_since_now(&disp_eta_new_line) > eta_new_line) {
                fio_gettime(&disp_eta_new_line, NULL);
                eta_new_line_pending = 1;
        }

        fflush(stdout);


It prints everything to the screen but I cannot redirect it at all

Thoughts?
Thank you,
Kurt


-----Original Message-----
From: Jeff Furlong [mailto:jeff.furlong@hgst.com]
Sent: Monday, March 14, 2016 12:56 PM
To: Dorsey, Kurt ; fio@vger.kernel.org
Subject: RE: Monitor IO while fio is running

You might add --eta=always and --eta-newline=1 to get the current IOPs rate every 1s, printed to stdout. You can parse that current IOPs rate via a script and do something. I believe the IOPs rate is recalculated every 1s, not the average from beginning of test.

Not sure why --eta and --eta-newline is not in the latest HOWTO notes.

Regards,
Jeff

-----Original Message-----
From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On Behalf Of Kurt_Dorsey@Dell.com
Sent: Saturday, March 12, 2016 2:20 PM
To: fio@vger.kernel.org
Subject: Monitor IO while fio is running

I may just not know how to do this
I know about the ability to touch /tmp/fio-dump-status which will dump the output of fio to a file or stdout while the test is running. What I am looking for is to be able to dump the write_iops_log while fio is running so that I can see the point in time average IOps for a given interval For example if I have log_avg_msec=60000 it will write an iops.log file with 60second intervals, unfortunately this file doesn't get created until fio exits. Is there a way to get this to dump on a regular basis (or by touching a file in /tmp) so that I can tail the output?

I am looking for a way to get point in time status via a script and the output from tmp-dump-status only gives me averages from the start of time rather than the last interval

Thank you,
Kurt
--
To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:

This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system.
Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:

This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system.

Re: Monitor IO while fio is running

[Alireza Haghdoost]

You should be able to redirect it to a file. Would you mind share with
us how you set the redirection?

On Mon, Mar 14, 2016 at 1:03 PM,  <Kurt_Dorsey@dell.com> wrote:
> Thank you Jeff for the reply.
>
> I have been playing around with this but unfortunately I cannot get the output to be outputted anywhere other than the screen / terminal.  I  have tried redirecting it to a txt file and parsing with with a python script but nothing ever gets output. (It only outputs to the screen)
>
> I haven't done C programming in years but I am trying to figure out from the code why this is.  I see in the eta.c file
>
> void display_thread_status(struct jobs_eta *je)
> {
> ...
>        printf("%s", output);
>
>         if (!eta_new_line_init) {
>                 fio_gettime(&disp_eta_new_line, NULL);
>                 eta_new_line_init = 1;
>         } else if (eta_new_line && mtime_since_now(&disp_eta_new_line) > eta_new_line) {
>                 fio_gettime(&disp_eta_new_line, NULL);
>                 eta_new_line_pending = 1;
>         }
>
>         fflush(stdout);
>
>
> It prints everything to the screen but I cannot redirect it at all
>
> Thoughts?
> Thank you,
> Kurt
>
>
> -----Original Message-----
> From: Jeff Furlong [mailto:jeff.furlong@hgst.com]
> Sent: Monday, March 14, 2016 12:56 PM
> To: Dorsey, Kurt ; fio@vger.kernel.org
> Subject: RE: Monitor IO while fio is running
>
> You might add --eta=always and --eta-newline=1 to get the current IOPs rate every 1s, printed to stdout. You can parse that current IOPs rate via a script and do something. I believe the IOPs rate is recalculated every 1s, not the average from beginning of test.
>
> Not sure why --eta and --eta-newline is not in the latest HOWTO notes.
>
> Regards,
> Jeff
>
> -----Original Message-----
> From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On Behalf Of Kurt_Dorsey@Dell.com
> Sent: Saturday, March 12, 2016 2:20 PM
> To: fio@vger.kernel.org
> Subject: Monitor IO while fio is running
>
> I may just not know how to do this
> I know about the ability to touch /tmp/fio-dump-status which will dump the output of fio to a file or stdout while the test is running. What I am looking for is to be able to dump the write_iops_log while fio is running so that I can see the point in time average IOps for a given interval For example if I have log_avg_msec=60000 it will write an iops.log file with 60second intervals, unfortunately this file doesn't get created until fio exits. Is there a way to get this to dump on a regular basis (or by touching a file in /tmp) so that I can tail the output?
>
> I am looking for a way to get point in time status via a script and the output from tmp-dump-status only gives me averages from the start of time rather than the last interval
>
> Thank you,
> Kurt
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
> Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:
>
> This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system.
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: Monitor IO while fio is running

[Kurt_Dorsey]

Hi Alireza, Thank you for the response


If I run the command on the bash command line I get this
(full output with 1 second intervals for the status
/usr/local/bin/fio --group_reporting=1 --name=test_workload  --time_based --runtime=10 --rw=randread --bs=16K --filename=/dev/sdb  --eta-newline=1 
test_workload: (g=0): rw=randread, bs=16K-16K/16K-16K/16K-16K, ioengine=sync, iodepth=1
fio-2.7-5-g5f2f
Starting 1 process
Jobs: 1 (f=1): [r(1)] [36.4% done] [78176KB/0KB/0KB /s] [4886/0/0 iops] [eta 00m:07s]
Jobs: 1 (f=1): [r(1)] [54.5% done] [79072KB/0KB/0KB /s] [4942/0/0 iops] [eta 00m:05s]
Jobs: 1 (f=1): [r(1)] [72.7% done] [79088KB/0KB/0KB /s] [4943/0/0 iops] [eta 00m:03s]
Jobs: 1 (f=1): [r(1)] [90.9% done] [81056KB/0KB/0KB /s] [5066/0/0 iops] [eta 00m:01s]
Jobs: 1 (f=1): [r(1)] [100.0% done] [80304KB/0KB/0KB /s] [5019/0/0 iops] [eta 00m:00s]
test_workload: (groupid=0, jobs=1): err= 0: pid=75170: Mon Mar 14 14:21:47 2016
  read : io=795824KB, bw=79574KB/s, iops=4973, runt= 10001msec
    clat (usec): min=136, max=7872, avg=198.30, stdev=59.80
     lat (usec): min=136, max=7872, avg=198.49, stdev=59.81
    clat percentiles (usec):

But if I redirect to a file the file is missing the Jobs:1 intervals
/usr/local/bin/fio --group_reporting=1 --name=test_workload  --time_based --runtime=10 --rw=randread --bs=16K --filename=/dev/sdb  --eta-newline=1  > test.out
cat test.out
test_workload: (g=0): rw=randread, bs=16K-16K/16K-16K/16K-16K, ioengine=sync, iodepth=1
fio-2.7-5-g5f2f
Starting 1 process

test_workload: (groupid=0, jobs=1): err= 0: pid=75251: Mon Mar 14 14:23:30 2016
  read : io=796416KB, bw=79634KB/s, iops=4977, runt= 10001msec
    clat (usec): min=136, max=519, avg=198.15, stdev=30.34
     lat (usec): min=136, max=519, avg=198.34, stdev=30.35
    clat percentiles (usec):
     |  1.00th=[  145],  5.00th=[  161], 10.00th=[  163], 20.00th=[  169],
     | 30.00th=[  179], 40.00th=[  183], 50.00th=[  193], 60.00th=[  207],
     | 70.00th=[  211], 80.00th=[  235], 90.00th=[  243], 95.00th=[  247],
     | 99.00th=[  262], 99.50th=[  270], 99.90th=[  282], 99.95th=[  290],
     | 99.99th=[  314]
    bw (KB  /s): min=78304, max=80960, per=100.00%, avg=79654.74, stdev=673.82
    lat (usec) : 250=96.25%, 500=3.75%, 750=0.01%
  cpu          : usr=2.49%, sys=11.79%, ctx=49808, majf=0, minf=37
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=49776/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: io=796416KB, aggrb=79633KB/s, minb=79633KB/s, maxb=79633KB/s, mint=10001msec, maxt=10001msec

Disk stats (read/write):
  sdb: ios=49309/0, merge=147930/0, ticks=8643/0, in_queue=8627, util=86.28%


I have also tried
| tee test.out

With the same resules.  
Looking at the FIO source I see that when Jen's uses log_buf (  it will output to a file and I can also watch the output inside python but when uses printf( it will only output to the screen.

Thank you,
Kurt

-----Original Message-----
From: Alireza Haghdoost [mailto:alireza@cs.umn.edu] 
Sent: Monday, March 14, 2016 1:37 PM
To: Dorsey, Kurt 
Cc: jeff.furlong@hgst.com; fio@vger.kernel.org
Subject: Re: Monitor IO while fio is running

You should be able to redirect it to a file. Would you mind share with us how you set the redirection?

On Mon, Mar 14, 2016 at 1:03 PM, wrote:
> Thank you Jeff for the reply.
>
> I have been playing around with this but unfortunately I cannot get 
> the output to be outputted anywhere other than the screen / terminal. 
> I have tried redirecting it to a txt file and parsing with with a 
> python script but nothing ever gets output. (It only outputs to the 
> screen)
>
> I haven't done C programming in years but I am trying to figure out 
> from the code why this is. I see in the eta.c file
>
> void display_thread_status(struct jobs_eta *je) { ...
> printf("%s", output);
>
> if (!eta_new_line_init) {
> fio_gettime(&disp_eta_new_line, NULL);
> eta_new_line_init = 1;
> } else if (eta_new_line && mtime_since_now(&disp_eta_new_line) > eta_new_line) {
> fio_gettime(&disp_eta_new_line, NULL);
> eta_new_line_pending = 1;
> }
>
> fflush(stdout);
>
>
> It prints everything to the screen but I cannot redirect it at all
>
> Thoughts?
> Thank you,
> Kurt
>
>
> -----Original Message-----
> From: Jeff Furlong [mailto:jeff.furlong@hgst.com]
> Sent: Monday, March 14, 2016 12:56 PM
> To: Dorsey, Kurt ; fio@vger.kernel.org
> Subject: RE: Monitor IO while fio is running
>
> You might add --eta=always and --eta-newline=1 to get the current IOPs rate every 1s, printed to stdout. You can parse that current IOPs rate via a script and do something. I believe the IOPs rate is recalculated every 1s, not the average from beginning of test.
>
> Not sure why --eta and --eta-newline is not in the latest HOWTO notes.
>
> Regards,
> Jeff
>
> -----Original Message-----
> From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On 
> Behalf Of Kurt_Dorsey@Dell.com
> Sent: Saturday, March 12, 2016 2:20 PM
> To: fio@vger.kernel.org
> Subject: Monitor IO while fio is running
>
> I may just not know how to do this
> I know about the ability to touch /tmp/fio-dump-status which will dump the output of fio to a file or stdout while the test is running. What I am looking for is to be able to dump the write_iops_log while fio is running so that I can see the point in time average IOps for a given interval For example if I have log_avg_msec=60000 it will write an iops.log file with 60second intervals, unfortunately this file doesn't get created until fio exits. Is there a way to get this to dump on a regular basis (or by touching a file in /tmp) so that I can tail the output?
>
> I am looking for a way to get point in time status via a script and 
> the output from tmp-dump-status only gives me averages from the start 
> of time rather than the last interval
>
> Thank you,
> Kurt
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in the 
> body of a message to majordomo@vger.kernel.org More majordomo info at 
> http://vger.kernel.org/majordomo-info.html
> Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:
>
> This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system.
> --
> To unsubscribe from this list: send the line "unsubscribe fio" in the 
> body of a message to majordomo@vger.kernel.org More majordomo info at 
> http://vger.kernel.org/majordomo-info.html

RE: Monitor IO while fio is running

[Wolford, Jeff (HPE Servers - Web Scale performance)]

Stupid question time:

Have you used the "--output filename.log" option ?
(that is the option I always use)

Or if you want to stay with std redirection, have you tried to redirect "stderr" ?

Jeff



| -----Original Message-----
| From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On Behalf Of
| Kurt_Dorsey@Dell.com
| Sent: Monday, March 14, 2016 2:27 PM
| To: alireza@cs.umn.edu
| Cc: jeff.furlong@hgst.com; fio@vger.kernel.org
| Subject: RE: Monitor IO while fio is running
| 
| Hi Alireza, Thank you for the response
| 
| 
| If I run the command on the bash command line I get this (full output with 1 second
| intervals for the status /usr/local/bin/fio --group_reporting=1 --name=test_workload
| --time_based --runtime=10 --rw=randread --bs=16K --filename=/dev/sdb  --eta-
| newline=1
| test_workload: (g=0): rw=randread, bs=16K-16K/16K-16K/16K-16K,
| ioengine=sync, iodepth=1 fio-2.7-5-g5f2f Starting 1 process
| Jobs: 1 (f=1): [r(1)] [36.4% done] [78176KB/0KB/0KB /s] [4886/0/0 iops] [eta
| 00m:07s]
| Jobs: 1 (f=1): [r(1)] [54.5% done] [79072KB/0KB/0KB /s] [4942/0/0 iops] [eta
| 00m:05s]
| Jobs: 1 (f=1): [r(1)] [72.7% done] [79088KB/0KB/0KB /s] [4943/0/0 iops] [eta
| 00m:03s]
| Jobs: 1 (f=1): [r(1)] [90.9% done] [81056KB/0KB/0KB /s] [5066/0/0 iops] [eta
| 00m:01s]
| Jobs: 1 (f=1): [r(1)] [100.0% done] [80304KB/0KB/0KB /s] [5019/0/0 iops] [eta
| 00m:00s]
| test_workload: (groupid=0, jobs=1): err= 0: pid=75170: Mon Mar 14 14:21:47 2016
|   read : io=795824KB, bw=79574KB/s, iops=4973, runt= 10001msec
|     clat (usec): min=136, max=7872, avg=198.30, stdev=59.80
|      lat (usec): min=136, max=7872, avg=198.49, stdev=59.81
|     clat percentiles (usec):
| 
| But if I redirect to a file the file is missing the Jobs:1 intervals /usr/local/bin/fio --
| group_reporting=1 --name=test_workload  --time_based --runtime=10 --
| rw=randread --bs=16K --filename=/dev/sdb  --eta-newline=1  > test.out cat test.out
| test_workload: (g=0): rw=randread, bs=16K-16K/16K-16K/16K-16K,
| ioengine=sync, iodepth=1 fio-2.7-5-g5f2f Starting 1 process
| 
| test_workload: (groupid=0, jobs=1): err= 0: pid=75251: Mon Mar 14 14:23:30 2016
|   read : io=796416KB, bw=79634KB/s, iops=4977, runt= 10001msec
|     clat (usec): min=136, max=519, avg=198.15, stdev=30.34
|      lat (usec): min=136, max=519, avg=198.34, stdev=30.35
|     clat percentiles (usec):
|      |  1.00th=[  145],  5.00th=[  161], 10.00th=[  163], 20.00th=[  169],
|      | 30.00th=[  179], 40.00th=[  183], 50.00th=[  193], 60.00th=[  207],
|      | 70.00th=[  211], 80.00th=[  235], 90.00th=[  243], 95.00th=[  247],
|      | 99.00th=[  262], 99.50th=[  270], 99.90th=[  282], 99.95th=[  290],
|      | 99.99th=[  314]
|     bw (KB  /s): min=78304, max=80960, per=100.00%, avg=79654.74,
| stdev=673.82
|     lat (usec) : 250=96.25%, 500=3.75%, 750=0.01%
|   cpu          : usr=2.49%, sys=11.79%, ctx=49808, majf=0, minf=37
|   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
| >=64=0.0%
|      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
| >=64=0.0%
|      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
| >=64=0.0%
|      issued    : total=r=49776/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
|      latency   : target=0, window=0, percentile=100.00%, depth=1
| 
| Run status group 0 (all jobs):
|    READ: io=796416KB, aggrb=79633KB/s, minb=79633KB/s, maxb=79633KB/s,
| mint=10001msec, maxt=10001msec
| 
| Disk stats (read/write):
|   sdb: ios=49309/0, merge=147930/0, ticks=8643/0, in_queue=8627, util=86.28%
| 
| 
| I have also tried
| | tee test.out
| 
| With the same resules.
| Looking at the FIO source I see that when Jen's uses log_buf (  it will output to a
| file and I can also watch the output inside python but when uses printf( it will only
| output to the screen.
| 
| Thank you,
| Kurt
| 
| -----Original Message-----
| From: Alireza Haghdoost [mailto:alireza@cs.umn.edu]
| Sent: Monday, March 14, 2016 1:37 PM
| To: Dorsey, Kurt
| Cc: jeff.furlong@hgst.com; fio@vger.kernel.org
| Subject: Re: Monitor IO while fio is running
| 
| You should be able to redirect it to a file. Would you mind share with us how you
| set the redirection?
| 
| On Mon, Mar 14, 2016 at 1:03 PM, wrote:
| > Thank you Jeff for the reply.
| >
| > I have been playing around with this but unfortunately I cannot get
| > the output to be outputted anywhere other than the screen / terminal.
| > I have tried redirecting it to a txt file and parsing with with a
| > python script but nothing ever gets output. (It only outputs to the
| > screen)
| >
| > I haven't done C programming in years but I am trying to figure out
| > from the code why this is. I see in the eta.c file
| >
| > void display_thread_status(struct jobs_eta *je) { ...
| > printf("%s", output);
| >
| > if (!eta_new_line_init) {
| > fio_gettime(&disp_eta_new_line, NULL); eta_new_line_init = 1; } else
| > if (eta_new_line && mtime_since_now(&disp_eta_new_line) >
| > eta_new_line) { fio_gettime(&disp_eta_new_line, NULL);
| > eta_new_line_pending = 1; }
| >
| > fflush(stdout);
| >
| >
| > It prints everything to the screen but I cannot redirect it at all
| >
| > Thoughts?
| > Thank you,
| > Kurt
| >
| >
| > -----Original Message-----
| > From: Jeff Furlong [mailto:jeff.furlong@hgst.com]
| > Sent: Monday, March 14, 2016 12:56 PM
| > To: Dorsey, Kurt ; fio@vger.kernel.org
| > Subject: RE: Monitor IO while fio is running
| >
| > You might add --eta=always and --eta-newline=1 to get the current IOPs rate
| every 1s, printed to stdout. You can parse that current IOPs rate via a script and
| do something. I believe the IOPs rate is recalculated every 1s, not the average
| from beginning of test.
| >
| > Not sure why --eta and --eta-newline is not in the latest HOWTO notes.
| >
| > Regards,
| > Jeff
| >
| > -----Original Message-----
| > From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On
| > Behalf Of Kurt_Dorsey@Dell.com
| > Sent: Saturday, March 12, 2016 2:20 PM
| > To: fio@vger.kernel.org
| > Subject: Monitor IO while fio is running
| >
| > I may just not know how to do this
| > I know about the ability to touch /tmp/fio-dump-status which will dump the output
| of fio to a file or stdout while the test is running. What I am looking for is to be able
| to dump the write_iops_log while fio is running so that I can see the point in time
| average IOps for a given interval For example if I have log_avg_msec=60000 it will
| write an iops.log file with 60second intervals, unfortunately this file doesn't get
| created until fio exits. Is there a way to get this to dump on a regular basis (or by
| touching a file in /tmp) so that I can tail the output?
| >
| > I am looking for a way to get point in time status via a script and
| > the output from tmp-dump-status only gives me averages from the start
| > of time rather than the last interval
| >
| > Thank you,
| > Kurt
| > --
| > To unsubscribe from this list: send the line "unsubscribe fio" in the
| > body of a message to majordomo@vger.kernel.org More majordomo info at
| > http://vger.kernel.org/majordomo-info.html
| > Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice &
| Disclaimer:
| >
| > This e-mail and any files transmitted with it may contain confidential or legally
| privileged information of WDC and/or its affiliates, and are intended solely for the
| use of the individual or entity to which they are addressed. If you are not the
| intended recipient, any disclosure, copying, distribution or any action taken or
| omitted to be taken in reliance on it, is prohibited. If you have received this e-mail
| in error, please notify the sender immediately and delete the e-mail in its entirety
| from your system.
| > --
| > To unsubscribe from this list: send the line "unsubscribe fio" in the
| > body of a message to majordomo@vger.kernel.org More majordomo info at
| > http://vger.kernel.org/majordomo-info.html
| --
| To unsubscribe from this list: send the line "unsubscribe fio" in the body of a
| message to majordomo@vger.kernel.org More majordomo info at
| http://vger.kernel.org/majordomo-info.html

RE: Monitor IO while fio is running

[Kurt_Dorsey]

Thank you all for the help.

--eta=always 

was the fix!



-----Original Message-----
From: Wolford, Jeff (HPE Servers - Web Scale performance) [mailto:jeff.wolford@hpe.com] 
Sent: Monday, March 14, 2016 3:40 PM
To: Dorsey, Kurt ; alireza@cs.umn.edu
Cc: jeff.furlong@hgst.com; fio@vger.kernel.org
Subject: RE: Monitor IO while fio is running

Stupid question time:

Have you used the "--output filename.log" option ?
(that is the option I always use)

Or if you want to stay with std redirection, have you tried to redirect "stderr" ?

Jeff



| -----Original Message-----
| From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On 
| Behalf Of Kurt_Dorsey@Dell.com
| Sent: Monday, March 14, 2016 2:27 PM
| To: alireza@cs.umn.edu
| Cc: jeff.furlong@hgst.com; fio@vger.kernel.org
| Subject: RE: Monitor IO while fio is running
| 
| Hi Alireza, Thank you for the response
| 
| 
| If I run the command on the bash command line I get this (full output 
| with 1 second intervals for the status /usr/local/bin/fio 
| --group_reporting=1 --name=test_workload --time_based --runtime=10 
| --rw=randread --bs=16K --filename=/dev/sdb --eta-
| newline=1
| test_workload: (g=0): rw=randread, bs=16K-16K/16K-16K/16K-16K, 
| ioengine=sync, iodepth=1 fio-2.7-5-g5f2f Starting 1 process
| Jobs: 1 (f=1): [r(1)] [36.4% done] [78176KB/0KB/0KB /s] [4886/0/0 
| iops] [eta 00m:07s]
| Jobs: 1 (f=1): [r(1)] [54.5% done] [79072KB/0KB/0KB /s] [4942/0/0 
| iops] [eta 00m:05s]
| Jobs: 1 (f=1): [r(1)] [72.7% done] [79088KB/0KB/0KB /s] [4943/0/0 
| iops] [eta 00m:03s]
| Jobs: 1 (f=1): [r(1)] [90.9% done] [81056KB/0KB/0KB /s] [5066/0/0 
| iops] [eta 00m:01s]
| Jobs: 1 (f=1): [r(1)] [100.0% done] [80304KB/0KB/0KB /s] [5019/0/0 
| iops] [eta 00m:00s]
| test_workload: (groupid=0, jobs=1): err= 0: pid=75170: Mon Mar 14 14:21:47 2016
| read : io=795824KB, bw=79574KB/s, iops=4973, runt= 10001msec
| clat (usec): min=136, max=7872, avg=198.30, stdev=59.80
| lat (usec): min=136, max=7872, avg=198.49, stdev=59.81
| clat percentiles (usec):
| 
| But if I redirect to a file the file is missing the Jobs:1 intervals 
| /usr/local/bin/fio --
| group_reporting=1 --name=test_workload --time_based --runtime=10 -- 
| rw=randread --bs=16K --filename=/dev/sdb --eta-newline=1 > test.out 
| cat test.out
| test_workload: (g=0): rw=randread, bs=16K-16K/16K-16K/16K-16K, 
| ioengine=sync, iodepth=1 fio-2.7-5-g5f2f Starting 1 process
| 
| test_workload: (groupid=0, jobs=1): err= 0: pid=75251: Mon Mar 14 14:23:30 2016
| read : io=796416KB, bw=79634KB/s, iops=4977, runt= 10001msec
| clat (usec): min=136, max=519, avg=198.15, stdev=30.34
| lat (usec): min=136, max=519, avg=198.34, stdev=30.35
| clat percentiles (usec):
| | 1.00th=[ 145], 5.00th=[ 161], 10.00th=[ 163], 20.00th=[ 169],
| | 30.00th=[ 179], 40.00th=[ 183], 50.00th=[ 193], 60.00th=[ 207],
| | 70.00th=[ 211], 80.00th=[ 235], 90.00th=[ 243], 95.00th=[ 247],
| | 99.00th=[ 262], 99.50th=[ 270], 99.90th=[ 282], 99.95th=[ 290],
| | 99.99th=[ 314]
| bw (KB /s): min=78304, max=80960, per=100.00%, avg=79654.74,
| stdev=673.82
| lat (usec) : 250=96.25%, 500=3.75%, 750=0.01%
| cpu : usr=2.49%, sys=11.79%, ctx=49808, majf=0, minf=37
| IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
| >=64=0.0%
| submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
| >=64=0.0%
| complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
| >=64=0.0%
| issued : total=r=49776/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
| latency : target=0, window=0, percentile=100.00%, depth=1
| 
| Run status group 0 (all jobs):
| READ: io=796416KB, aggrb=79633KB/s, minb=79633KB/s, maxb=79633KB/s, 
| mint=10001msec, maxt=10001msec
| 
| Disk stats (read/write):
| sdb: ios=49309/0, merge=147930/0, ticks=8643/0, in_queue=8627, 
| util=86.28%
| 
| 
| I have also tried
| | tee test.out
| 
| With the same resules.
| Looking at the FIO source I see that when Jen's uses log_buf ( it 
| will output to a file and I can also watch the output inside python 
| but when uses printf( it will only output to the screen.
| 
| Thank you,
| Kurt
| 
| -----Original Message-----
| From: Alireza Haghdoost [mailto:alireza@cs.umn.edu]
| Sent: Monday, March 14, 2016 1:37 PM
| To: Dorsey, Kurt
| Cc: jeff.furlong@hgst.com; fio@vger.kernel.org
| Subject: Re: Monitor IO while fio is running
| 
| You should be able to redirect it to a file. Would you mind share with 
| us how you set the redirection?
| 
| On Mon, Mar 14, 2016 at 1:03 PM, wrote:
| > Thank you Jeff for the reply.
| >
| > I have been playing around with this but unfortunately I cannot get 
| > the output to be outputted anywhere other than the screen / terminal.
| > I have tried redirecting it to a txt file and parsing with with a 
| > python script but nothing ever gets output. (It only outputs to the
| > screen)
| >
| > I haven't done C programming in years but I am trying to figure out 
| > from the code why this is. I see in the eta.c file
| >
| > void display_thread_status(struct jobs_eta *je) { ...
| > printf("%s", output);
| >
| > if (!eta_new_line_init) {
| > fio_gettime(&disp_eta_new_line, NULL); eta_new_line_init = 1; } else 
| > if (eta_new_line && mtime_since_now(&disp_eta_new_line) >
| > eta_new_line) { fio_gettime(&disp_eta_new_line, NULL); 
| > eta_new_line_pending = 1; }
| >
| > fflush(stdout);
| >
| >
| > It prints everything to the screen but I cannot redirect it at all
| >
| > Thoughts?
| > Thank you,
| > Kurt
| >
| >
| > -----Original Message-----
| > From: Jeff Furlong [mailto:jeff.furlong@hgst.com]
| > Sent: Monday, March 14, 2016 12:56 PM
| > To: Dorsey, Kurt ; fio@vger.kernel.org
| > Subject: RE: Monitor IO while fio is running
| >
| > You might add --eta=always and --eta-newline=1 to get the current 
| > IOPs rate
| every 1s, printed to stdout. You can parse that current IOPs rate via 
| a script and do something. I believe the IOPs rate is recalculated 
| every 1s, not the average from beginning of test.
| >
| > Not sure why --eta and --eta-newline is not in the latest HOWTO notes.
| >
| > Regards,
| > Jeff
| >
| > -----Original Message-----
| > From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] 
| > On Behalf Of Kurt_Dorsey@Dell.com
| > Sent: Saturday, March 12, 2016 2:20 PM
| > To: fio@vger.kernel.org
| > Subject: Monitor IO while fio is running
| >
| > I may just not know how to do this
| > I know about the ability to touch /tmp/fio-dump-status which will 
| > dump the output
| of fio to a file or stdout while the test is running. What I am 
| looking for is to be able to dump the write_iops_log while fio is 
| running so that I can see the point in time average IOps for a given 
| interval For example if I have log_avg_msec=60000 it will write an 
| iops.log file with 60second intervals, unfortunately this file doesn't 
| get created until fio exits. Is there a way to get this to dump on a regular basis (or by touching a file in /tmp) so that I can tail the output?
| >
| > I am looking for a way to get point in time status via a script and 
| > the output from tmp-dump-status only gives me averages from the 
| > start of time rather than the last interval
| >
| > Thank you,
| > Kurt
| > --
| > To unsubscribe from this list: send the line "unsubscribe fio" in 
| > the body of a message to majordomo@vger.kernel.org More majordomo 
| > info at http://vger.kernel.org/majordomo-info.html
| > Western Digital Corporation (and its subsidiaries) E-mail 
| > Confidentiality Notice &
| Disclaimer:
| >
| > This e-mail and any files transmitted with it may contain 
| > confidential or legally
| privileged information of WDC and/or its affiliates, and are intended 
| solely for the use of the individual or entity to which they are 
| addressed. If you are not the intended recipient, any disclosure, 
| copying, distribution or any action taken or omitted to be taken in 
| reliance on it, is prohibited. If you have received this e-mail in 
| error, please notify the sender immediately and delete the e-mail in its entirety from your system.
| > --
| > To unsubscribe from this list: send the line "unsubscribe fio" in 
| > the body of a message to majordomo@vger.kernel.org More majordomo 
| > info at http://vger.kernel.org/majordomo-info.html
| --
| To unsubscribe from this list: send the line "unsubscribe fio" in the 
| body of a message to majordomo@vger.kernel.org More majordomo info at 
| http://vger.kernel.org/majordomo-info.html