[PATCH] idle-prof: append output cpu idleness data to terse log

[PATCH] idle-prof: append output cpu idleness data to terse log

[Friendy.Su]

cpu idleness data only output to normal and json log,
also output to terse log.

Signed-off-by: friendy-su <friendy.su@sony.com>
---
 HOWTO      |  4 +++-
 idletime.c | 19 +++++++++++++++++++
 stat.c     |  2 ++
 3 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/HOWTO b/HOWTO
index 70eed280..497e8cb1 100644
--- a/HOWTO
+++ b/HOWTO
@@ -3564,8 +3564,10 @@ will be a disk utilization section.
 Below is a single line containing short names for each of the fields in the
 minimal output v3, separated by semicolons::

-        terse_version_3;fio_version;jobname;groupid;error;read_kb;read_bandwidth;read_iops;read_runtime_ms;read_slat_min;read_slat_max;read_slat_mean;read_slat_dev;read_clat_min;read_clat_max;read_clat_mean;read_clat_dev;read_clat_pct01;read_clat_pct02;read_clat_pct03;read_clat_pct04;read_clat_pct05;read_clat_pct06;read_clat_pct07;read_clat_pct08;read_clat_pct09;read_clat_pct10;read_clat_pct11;read_clat_pct12;read_clat_pct13;read_clat_pct14;read_clat_pct15;read_clat_pct16;read_clat_pct17;read_clat_pct18;read_clat_pct19;read_clat_pct20;read_tlat_min;read_lat_max;read_lat_mean;read_lat_dev;read_bw_min;read_bw_max;read_bw_agg_pct;read_bw_mean;read_bw_dev;write_kb;write_bandwidth;write_iops;write_runtime_ms;write_slat_min;write_slat_max;write_slat_mean;write_slat_dev;write_clat_min;write_clat_max;write_clat_mean;write_clat_dev;write_clat_pct01;write_clat_pct02;write_clat_pct03;write_clat_pct04;write_clat_pct05;write_clat_pct06;write_clat_pct07;write_clat_pct08;write_clat_pct09;write_clat_pct10;write_clat_pct11;write_clat_pct12;write_clat_pct13;write_clat_pct14;write_clat_pct15;write_clat_pct16;write_clat_pct17;write_clat_pct18;write_clat_pct19;write_clat_pct20;write_tlat_min;write_lat_max;write_lat_mean;write_lat_dev;write_bw_min;write_bw_max;write_bw_agg_pct;write_bw_mean;write_bw_dev;cpu_user;cpu_sys;cpu_csw;cpu_mjf;cpu_minf;iodepth_1;iodepth_2;iodepth_4;iodepth_8;iodepth_16;iodepth_32;iodepth_64;lat_2us;lat_4us;lat_10us;lat_20us;lat_50us;lat_100us;lat_250us;lat_500us;lat_750us;lat_1000us;lat_2ms;lat_4ms;lat_10ms;lat_20ms;lat_50ms;lat_100ms;lat_250ms;lat_500ms;lat_750ms;lat_1000ms;lat_2000ms;lat_over_2000ms;disk_name;disk_read_iops;disk_write_iops;disk_read_merges;disk_write_merges;disk_read_ticks;write_ticks;disk_queue_time;disk_util
+        terse_version_3;fio_version;jobname;groupid;error;read_kb;read_bandwidth;read_iops;read_runtime_ms;read_slat_min;read_slat_max;read_slat_mean;read_slat_dev;read_clat_min;read_clat_max;read_clat_mean;read_clat_dev;read_clat_pct01;read_clat_pct02;read_clat_pct03;read_clat_pct04;read_clat_pct05;read_clat_pct06;read_clat_pct07;read_clat_pct08;read_clat_pct09;read_clat_pct10;read_clat_pct11;read_clat_pct12;read_clat_pct13;read_clat_pct14;read_clat_pct15;read_clat_pct16;read_clat_pct17;read_clat_pct18;read_clat_pct19;read_clat_pct20;read_tlat_min;read_lat_max;read_lat_mean;read_lat_dev;read_bw_min;read_bw_max;read_bw_agg_pct;read_bw_mean;read_bw_dev;write_kb;write_bandwidth;write_iops;write_runtime_ms;write_slat_min;write_slat_max;write_slat_mean;write_slat_dev;write_clat_min;write_clat_max;write_clat_mean;write_clat_dev;write_clat_pct01;write_clat_pct02;write_clat_pct03;write_clat_pct04;write_clat_pct05;write_clat_pct06;write_clat_pct07;write_clat_pct08;write_clat_pct09;write_clat_pct10;write_clat_pct11;write_clat_pct12;write_clat_pct13;write_clat_pct14;write_clat_pct15;write_clat_pct16;write_clat_pct17;write_clat_pct18;write_clat_pct19;write_clat_pct20;write_tlat_min;write_lat_max;write_lat_mean;write_lat_dev;write_bw_min;write_bw_max;write_bw_agg_pct;write_bw_mean;write_bw_dev;cpu_user;cpu_sys;cpu_csw;cpu_mjf;cpu_minf;iodepth_1;iodepth_2;iodepth_4;iodepth_8;iodepth_16;iodepth_32;iodepth_64;lat_2us;lat_4us;lat_10us;lat_20us;lat_50us;lat_100us;lat_250us;lat_500us;lat_750us;lat_1000us;lat_2ms;lat_4ms;lat_10ms;lat_20ms;lat_50ms;lat_100ms;lat_250ms;lat_500ms;lat_750ms;lat_1000ms;lat_2000ms;lat_over_2000ms;disk_name;disk_read_iops;disk_write_iops;disk_read_merges;disk_write_merges;disk_read_ticks;write_ticks;disk_queue_time;disk_util;cpu_idle_system;cpu_idle_cpu[0];cpu_idle_cpu[1];.....;cpu_idle_cpu[n-1];cpu_idle_uw_mean;cpu_idle_uw_stddev

+cpu_idle_XXX are only available when run with --idle-prof.
+cpu_idle_cpu[] are only available when --idle-prof=percpu.

 JSON output
 ------------
diff --git a/idletime.c b/idletime.c
index 2f59f510..94c16a13 100644
--- a/idletime.c
+++ b/idletime.c
@@ -450,6 +450,25 @@ void show_idle_prof_stats(int output, struct json_object *parent,
        struct json_object *tmp;
        char s[MAX_CPU_STR_LEN];

+       if (output == FIO_OUTPUT_TERSE) {
+
+               if (ipc.opt == IDLE_PROF_OPT_NONE)
+                       return;
+
+               if (ipc.opt >= IDLE_PROF_OPT_SYSTEM)
+                       log_buf(out, ";%3.2f%%", fio_idle_prof_cpu_stat(-1));
+
+               if (ipc.opt == IDLE_PROF_OPT_PERCPU) {
+                       for (i = 0; i < nr_cpus; i++)
+                               log_buf(out, ";%3.2f%%", fio_idle_prof_cpu_stat(i));
+               }
+
+               if (ipc.opt >= IDLE_PROF_OPT_CALI)
+                       log_buf(out, ";%3.2fus;%3.2f", ipc.cali_mean, ipc.cali_stddev);
+
+               return;
+       }
+
        if (output == FIO_OUTPUT_NORMAL) {
                if (ipc.opt > IDLE_PROF_OPT_CALI)
                        log_buf(out, "\nCPU idleness:\n");
diff --git a/stat.c b/stat.c
index a308eb88..76deb4ea 100644
--- a/stat.c
+++ b/stat.c
@@ -1208,6 +1208,8 @@ static void show_thread_status_terse_all(struct thread_stat *ts,
        if (ver >= 3)
                show_disk_util(1, NULL, out);

+       show_idle_prof_stats(FIO_OUTPUT_TERSE, NULL, out);
+
        /* Additional output if continue_on_error set - default off*/
        if (ts->continue_on_error)
                log_buf(out, ";%llu;%d", (unsigned long long) ts->total_err_count, ts->first_error);
--
2.18.0


________________________________

This email is confidential and intended only for the use of the individual or entity named above and may contain information that is privileged. If you are not the intended recipient, you are notified that any dissemination, distribution or copying of this email is strictly prohibited. If you have received this email in error, please notify us immediately by return email or telephone and destroy the original message. - This mail is sent via Sony Asia Pacific Mail Gateway..

RE: [PATCH] idle-prof: append output cpu idleness data to terse log

[Elliott, Robert (Persistent Memory)]

> -----Original Message-----
> From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf
> Of Friendy.Su@sony.com
> Sent: Tuesday, July 17, 2018 2:46 AM
> To: fio@vger.kernel.org
> Cc: No.Tanaka@sony.com; Hajime.Tomura@sony.com; Friendy.Su@sony.com
> Subject: [PATCH] idle-prof: append output cpu idleness data to terse
> log
> 
> cpu idleness data only output to normal and json log,
> also output to terse log.
> 
> Signed-off-by: friendy-su <friendy.su@sony.com>
> ---
>  HOWTO      |  4 +++-
>  idletime.c | 19 +++++++++++++++++++
>  stat.c     |  2 ++
>  3 files changed, 24 insertions(+), 1 deletion(-)
> 
> diff --git a/HOWTO b/HOWTO
> index 70eed280..497e8cb1 100644
> --- a/HOWTO
> +++ b/HOWTO
> @@ -3564,8 +3564,10 @@ will be a disk utilization section.
>  Below is a single line containing short names for each of the fields
> in the
>  minimal output v3, separated by semicolons::
> 
> -
> terse_version_3;fio_version;jobname;groupid;error;read_kb;read_bandwi
...
> +
> terse_version_3;fio_version;jobname;groupid;error;read_kb;read_bandwi
...

Since the terse format is not self-describing, new fields require
a version bump (see -terse-version).

Maybe one of the versions should print out the field names as the first
line, so it becomes self-describing.  I guess the json format already
provides that functionality, so it might not be necessary.

---
Robert Elliott, HPE Persistent Memory

Re: [PATCH] idle-prof: append output cpu idleness data to terse log

[Alireza Haghdoost]

On Tue, Jul 17, 2018 at 9:18 AM, Elliott, Robert (Persistent Memory)
<elliott@hpe.com> wrote:
>
>
>> -----Original Message-----
>> From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf
>> Of Friendy.Su@sony.com
>> Sent: Tuesday, July 17, 2018 2:46 AM
>> To: fio@vger.kernel.org
>> Cc: No.Tanaka@sony.com; Hajime.Tomura@sony.com; Friendy.Su@sony.com
>> Subject: [PATCH] idle-prof: append output cpu idleness data to terse
>> log
>>
>> cpu idleness data only output to normal and json log,
>> also output to terse log.
>>
>> Signed-off-by: friendy-su <friendy.su@sony.com>
>> ---
>>  HOWTO      |  4 +++-
>>  idletime.c | 19 +++++++++++++++++++
>>  stat.c     |  2 ++
>>  3 files changed, 24 insertions(+), 1 deletion(-)
>>
>> diff --git a/HOWTO b/HOWTO
>> index 70eed280..497e8cb1 100644
>> --- a/HOWTO
>> +++ b/HOWTO
>> @@ -3564,8 +3564,10 @@ will be a disk utilization section.
>>  Below is a single line containing short names for each of the fields
>> in the
>>  minimal output v3, separated by semicolons::
>>
>> -
>> terse_version_3;fio_version;jobname;groupid;error;read_kb;read_bandwi
> ...
>> +
>> terse_version_3;fio_version;jobname;groupid;error;read_kb;read_bandwi
> ...
>
> Since the terse format is not self-describing, new fields require
> a version bump (see -terse-version).
>
> Maybe one of the versions should print out the field names as the first
> line, so it becomes self-describing.  I guess the json format already
> provides that functionality, so it might not be necessary.
>
How about adding an extra command option like --terse-header that
prints out the header only? It is hard to keep track of these version
changes in one place at the user manual or some blog post.
with a terse header command option, I can ignore the terse version and
adjust my test scripts to query the header first before running the
test, then pull the right column after the test finishes.

Re: [PATCH] idle-prof: append output cpu idleness data to terse log

[Sitsofe Wheeler]

On 17 July 2018 at 15:30, Alireza Haghdoost <haghdoost@gmail.com> wrote:
> On Tue, Jul 17, 2018 at 9:18 AM, Elliott, Robert (Persistent Memory)
> <elliott@hpe.com> wrote:
>>
>>
>>> -----Original Message-----
>>> From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf
>>> Of Friendy.Su@sony.com
>>> Sent: Tuesday, July 17, 2018 2:46 AM
>>> To: fio@vger.kernel.org
>>> Cc: No.Tanaka@sony.com; Hajime.Tomura@sony.com; Friendy.Su@sony.com
>>> Subject: [PATCH] idle-prof: append output cpu idleness data to terse
>>> log
>>>
>>> cpu idleness data only output to normal and json log,
>>> also output to terse log.
>>>
>>> Signed-off-by: friendy-su <friendy.su@sony.com>
>>> ---
>>>  HOWTO      |  4 +++-
>>>  idletime.c | 19 +++++++++++++++++++
>>>  stat.c     |  2 ++
>>>  3 files changed, 24 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/HOWTO b/HOWTO
>>> index 70eed280..497e8cb1 100644
>>> --- a/HOWTO
>>> +++ b/HOWTO
>>> @@ -3564,8 +3564,10 @@ will be a disk utilization section.
>>>  Below is a single line containing short names for each of the fields
>>> in the
>>>  minimal output v3, separated by semicolons::
>>>
>>> -
>>> terse_version_3;fio_version;jobname;groupid;error;read_kb;read_bandwi
>> ...
>>> +
>>> terse_version_3;fio_version;jobname;groupid;error;read_kb;read_bandwi
>> ...
>>
>> Since the terse format is not self-describing, new fields require
>> a version bump (see -terse-version).
>>
>> Maybe one of the versions should print out the field names as the first
>> line, so it becomes self-describing.  I guess the json format already
>> provides that functionality, so it might not be necessary.
>>
> How about adding an extra command option like --terse-header that
> prints out the header only? It is hard to keep track of these version
> changes in one place at the user manual or some blog post.
> with a terse header command option, I can ignore the terse version and
> adjust my test scripts to query the header first before running the
> test, then pull the right column after the test finishes.

You would then need to document all the terse formats because you
never know which one someone will use (it looks like there are four:
http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-terse-version
). At this stage shouldn't we be nudging people towards JSON which is
more self describing and easier to extend in a backwards compatible
manner?

-- 
Sitsofe | http://sucs.org/~sits/

Re: [PATCH] idle-prof: append output cpu idleness data to terse log

[Alireza Haghdoost]

On Tue, Jul 17, 2018 at 2:13 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>
> On 17 July 2018 at 15:30, Alireza Haghdoost <haghdoost@gmail.com> wrote:
> > On Tue, Jul 17, 2018 at 9:18 AM, Elliott, Robert (Persistent Memory)
> > <elliott@hpe.com> wrote:
> >>
> >>
> >>> -----Original Message-----
> >>> From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf
> >>> Of Friendy.Su@sony.com
> >>> Sent: Tuesday, July 17, 2018 2:46 AM
> >>> To: fio@vger.kernel.org
> >>> Cc: No.Tanaka@sony.com; Hajime.Tomura@sony.com; Friendy.Su@sony.com
> >>> Subject: [PATCH] idle-prof: append output cpu idleness data to terse
> >>> log
> >>>
> >>> cpu idleness data only output to normal and json log,
> >>> also output to terse log.
> >>>
> >>> Signed-off-by: friendy-su <friendy.su@sony.com>
> >>> ---
> >>>  HOWTO      |  4 +++-
> >>>  idletime.c | 19 +++++++++++++++++++
> >>>  stat.c     |  2 ++
> >>>  3 files changed, 24 insertions(+), 1 deletion(-)
> >>>
> >>> diff --git a/HOWTO b/HOWTO
> >>> index 70eed280..497e8cb1 100644
> >>> --- a/HOWTO
> >>> +++ b/HOWTO
> >>> @@ -3564,8 +3564,10 @@ will be a disk utilization section.
> >>>  Below is a single line containing short names for each of the fields
> >>> in the
> >>>  minimal output v3, separated by semicolons::
> >>>
> >>> -
> >>> terse_version_3;fio_version;jobname;groupid;error;read_kb;read_bandwi
> >> ...
> >>> +
> >>> terse_version_3;fio_version;jobname;groupid;error;read_kb;read_bandwi
> >> ...
> >>
> >> Since the terse format is not self-describing, new fields require
> >> a version bump (see -terse-version).
> >>
> >> Maybe one of the versions should print out the field names as the first
> >> line, so it becomes self-describing.  I guess the json format already
> >> provides that functionality, so it might not be necessary.
> >>
> > How about adding an extra command option like --terse-header that
> > prints out the header only? It is hard to keep track of these version
> > changes in one place at the user manual or some blog post.
> > with a terse header command option, I can ignore the terse version and
> > adjust my test scripts to query the header first before running the
> > test, then pull the right column after the test finishes.
>
> You would then need to document all the terse formats because you
> never know which one someone will use (it looks like there are four:
> http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-terse-version
> ). At this stage shouldn't we be nudging people towards JSON which is
> more self describing and easier to extend in a backwards compatible
> manner?

I am on board with your suggestion but not every user might have a
time/resource to re-implement their existing test script to work with
json. We should stop adding more versions of the terse. Otherwise, If
we want to continue supporting the terse format, I recommend to figure
out a programmatic way to handle its complexity.

RE: [PATCH] idle-prof: append output cpu idleness data to terse log

[Elliott, Robert (Persistent Memory)]

> -----Original Message-----
> From: Alireza Haghdoost <haghdoost@gmail.com>
> Sent: Tuesday, July 17, 2018 2:25 PM
> To: Sitsofe Wheeler <sitsofe@gmail.com>
> Cc: Elliott, Robert (Persistent Memory) <elliott@hpe.com>;
> Friendy.Su@sony.com; fio@vger.kernel.org; No.Tanaka@sony.com;
> Hajime.Tomura@sony.com
> Subject: Re: [PATCH] idle-prof: append output cpu idleness data to
> terse log
> 
> On Tue, Jul 17, 2018 at 2:13 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> >
> > On 17 July 2018 at 15:30, Alireza Haghdoost <haghdoost@gmail.com> wrote:
> > > On Tue, Jul 17, 2018 at 9:18 AM, Elliott, Robert (Persistent Memory) ...
> > >> Since the terse format is not self-describing, new fields require
> > >> a version bump (see -terse-version).
> > >>
> > >> Maybe one of the versions should print out the field names as the first
> > >> line, so it becomes self-describing.  I guess the json format already
> > >> provides that functionality, so it might not be necessary.
> > >>
> > > How about adding an extra command option like --terse-header that
> > > prints out the header only? It is hard to keep track of these version
> > > changes in one place at the user manual or some blog post.
> > > with a terse header command option, I can ignore the terse version and
> > > adjust my test scripts to query the header first before running the
> > > test, then pull the right column after the test finishes.
> >
> > You would then need to document all the terse formats because you
> > never know which one someone will use (it looks like there are four:
> > http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-terse-version
> > ). At this stage shouldn't we be nudging people towards JSON which is
> > more self describing and easier to extend in a backwards compatible
> > manner?
> 
> I am on board with your suggestion but not every user might have a
> time/resource to re-implement their existing test script to work with
> json. We should stop adding more versions of the terse. Otherwise, If
> we want to continue supporting the terse format, I recommend to
> figure out a programmatic way to handle its complexity.

How about defining that the next terse version always includes
a header line, so it'll be self-describing and can be the final
terse version.


---
Robert Elliott, HPE Persistent Memory

RE: [PATCH] idle-prof: append output cpu idleness data to terse log

[Friendy.Su]

In practice, terse output without self-header makes it trouble for a parsing script. Not only terse version, but also variable field count. Some fields existing or not depends on options in job file or ioengine's configuration, e.g disk utilization data. Self-header a variable length terse looks not so easy.

> -----Original Message-----
> From: Alireza Haghdoost [mailto:haghdoost@gmail.com]
> Sent: Wednesday, July 18, 2018 3:25 AM
> To: Sitsofe Wheeler
> Cc: Elliott, Robert (Persistent Memory); Su, Friendy; fio@vger.kernel.org; Tanaka,
> Nobuyuki (Sony); Tomura, Hajime (Sony)
> Subject: Re: [PATCH] idle-prof: append output cpu idleness data to terse log
>
> On Tue, Jul 17, 2018 at 2:13 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
> >
> > On 17 July 2018 at 15:30, Alireza Haghdoost <haghdoost@gmail.com> wrote:
> > > On Tue, Jul 17, 2018 at 9:18 AM, Elliott, Robert (Persistent Memory)
> > > <elliott@hpe.com> wrote:
> > >>
> > >>
> > >>> -----Original Message-----
> > >>> From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On
> Behalf
> > >>> Of Friendy.Su@sony.com
> > >>> Sent: Tuesday, July 17, 2018 2:46 AM
> > >>> To: fio@vger.kernel.org
> > >>> Cc: No.Tanaka@sony.com; Hajime.Tomura@sony.com;
> Friendy.Su@sony.com
> > >>> Subject: [PATCH] idle-prof: append output cpu idleness data to terse
> > >>> log
> > >>>
> > >>> cpu idleness data only output to normal and json log,
> > >>> also output to terse log.
> > >>>
> > >>> Signed-off-by: friendy-su <friendy.su@sony.com>
> > >>> ---
> > >>>  HOWTO      |  4 +++-
> > >>>  idletime.c | 19 +++++++++++++++++++
> > >>>  stat.c     |  2 ++
> > >>>  3 files changed, 24 insertions(+), 1 deletion(-)
> > >>>
> > >>> diff --git a/HOWTO b/HOWTO
> > >>> index 70eed280..497e8cb1 100644
> > >>> --- a/HOWTO
> > >>> +++ b/HOWTO
> > >>> @@ -3564,8 +3564,10 @@ will be a disk utilization section.
> > >>>  Below is a single line containing short names for each of the fields
> > >>> in the
> > >>>  minimal output v3, separated by semicolons::
> > >>>
> > >>> -
> > >>> terse_version_3;fio_version;jobname;groupid;error;read_kb;read_bandwi
> > >> ...
> > >>> +
> > >>> terse_version_3;fio_version;jobname;groupid;error;read_kb;read_bandwi
> > >> ...
> > >>
> > >> Since the terse format is not self-describing, new fields require
> > >> a version bump (see -terse-version).
> > >>
> > >> Maybe one of the versions should print out the field names as the first
> > >> line, so it becomes self-describing.  I guess the json format already
> > >> provides that functionality, so it might not be necessary.
> > >>
> > > How about adding an extra command option like --terse-header that
> > > prints out the header only? It is hard to keep track of these version
> > > changes in one place at the user manual or some blog post.
> > > with a terse header command option, I can ignore the terse version and
> > > adjust my test scripts to query the header first before running the
> > > test, then pull the right column after the test finishes.
> >
> > You would then need to document all the terse formats because you
> > never know which one someone will use (it looks like there are four:
> > http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-terse-version
> > ). At this stage shouldn't we be nudging people towards JSON which is
> > more self describing and easier to extend in a backwards compatible
> > manner?
>
> I am on board with your suggestion but not every user might have a
> time/resource to re-implement their existing test script to work with
> json. We should stop adding more versions of the terse. Otherwise, If
> we want to continue supporting the terse format, I recommend to figure
> out a programmatic way to handle its complexity.

________________________________

This email is confidential and intended only for the use of the individual or entity named above and may contain information that is privileged. If you are not the intended recipient, you are notified that any dissemination, distribution or copying of this email is strictly prohibited. If you have received this email in error, please notify us immediately by return email or telephone and destroy the original message. - This mail is sent via Sony Asia Pacific Mail Gateway..

Re: [PATCH] idle-prof: append output cpu idleness data to terse log

[Jens Axboe]

On 7/17/18 9:18 PM, Elliott, Robert (Persistent Memory) wrote:
> 
> 
>> -----Original Message-----
>> From: Alireza Haghdoost <haghdoost@gmail.com>
>> Sent: Tuesday, July 17, 2018 2:25 PM
>> To: Sitsofe Wheeler <sitsofe@gmail.com>
>> Cc: Elliott, Robert (Persistent Memory) <elliott@hpe.com>;
>> Friendy.Su@sony.com; fio@vger.kernel.org; No.Tanaka@sony.com;
>> Hajime.Tomura@sony.com
>> Subject: Re: [PATCH] idle-prof: append output cpu idleness data to
>> terse log
>>
>> On Tue, Jul 17, 2018 at 2:13 PM, Sitsofe Wheeler <sitsofe@gmail.com> wrote:
>>>
>>> On 17 July 2018 at 15:30, Alireza Haghdoost <haghdoost@gmail.com> wrote:
>>>> On Tue, Jul 17, 2018 at 9:18 AM, Elliott, Robert (Persistent Memory) ...
>>>>> Since the terse format is not self-describing, new fields require
>>>>> a version bump (see -terse-version).
>>>>>
>>>>> Maybe one of the versions should print out the field names as the first
>>>>> line, so it becomes self-describing.  I guess the json format already
>>>>> provides that functionality, so it might not be necessary.
>>>>>
>>>> How about adding an extra command option like --terse-header that
>>>> prints out the header only? It is hard to keep track of these version
>>>> changes in one place at the user manual or some blog post.
>>>> with a terse header command option, I can ignore the terse version and
>>>> adjust my test scripts to query the header first before running the
>>>> test, then pull the right column after the test finishes.
>>>
>>> You would then need to document all the terse formats because you
>>> never know which one someone will use (it looks like there are four:
>>> http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-terse-version
>>> ). At this stage shouldn't we be nudging people towards JSON which is
>>> more self describing and easier to extend in a backwards compatible
>>> manner?
>>
>> I am on board with your suggestion but not every user might have a
>> time/resource to re-implement their existing test script to work with
>> json. We should stop adding more versions of the terse. Otherwise, If
>> we want to continue supporting the terse format, I recommend to
>> figure out a programmatic way to handle its complexity.
> 
> How about defining that the next terse version always includes
> a header line, so it'll be self-describing and can be the final
> terse version.

I like that idea. We already have a version at the front, proper scripts
should be checking that, and we can further make it safer by adding
new fields at the end.

Honestly, I really dislike the terse format. But it's there and some
people like to use it, and it's a shame if we can add new additions. It's
already trailing the json/normal output in various cases, for instance
it doesn't include fsync output (and I'm sure others). If at all possible,
I always push people towards json instead of the nasty terse output.

-- 
Jens Axboe

RE: [PATCH] idle-prof: append output cpu idleness data to terse log

[Friendy.Su]

Many people are still used to process, plot data by excel. As a format which can be opened directly by excel, terse is welcome.
Exactly terse log has already trailing the normal/json log, improving it inside fio looks a redundant work. Instead, how about provide a script to convert json to terse under tools/?Meanwhile, in HOWTO doc, tell user to use that script to get a 'perfect' terse.

> -----Original Message-----
> From: Jens Axboe [mailto:axboe@kernel.dk]
> Sent: Tuesday, July 24, 2018 11:41 PM
> To: Elliott, Robert (Persistent Memory); Alireza Haghdoost; Sitsofe Wheeler
> Cc: Su, Friendy; fio@vger.kernel.org; Tanaka, Nobuyuki (Sony); Tomura, Hajime
> (Sony)
> Subject: Re: [PATCH] idle-prof: append output cpu idleness data to terse log
>
> On 7/17/18 9:18 PM, Elliott, Robert (Persistent Memory) wrote:
> >
> >
> >> -----Original Message-----
> >> From: Alireza Haghdoost <haghdoost@gmail.com>
> >> Sent: Tuesday, July 17, 2018 2:25 PM
> >> To: Sitsofe Wheeler <sitsofe@gmail.com>
> >> Cc: Elliott, Robert (Persistent Memory) <elliott@hpe.com>;
> >> Friendy.Su@sony.com; fio@vger.kernel.org; No.Tanaka@sony.com;
> >> Hajime.Tomura@sony.com
> >> Subject: Re: [PATCH] idle-prof: append output cpu idleness data to
> >> terse log
> >>
> >> On Tue, Jul 17, 2018 at 2:13 PM, Sitsofe Wheeler <sitsofe@gmail.com>
> wrote:
> >>>
> >>> On 17 July 2018 at 15:30, Alireza Haghdoost <haghdoost@gmail.com>
> wrote:
> >>>> On Tue, Jul 17, 2018 at 9:18 AM, Elliott, Robert (Persistent Memory) ...
> >>>>> Since the terse format is not self-describing, new fields require
> >>>>> a version bump (see -terse-version).
> >>>>>
> >>>>> Maybe one of the versions should print out the field names as the first
> >>>>> line, so it becomes self-describing.  I guess the json format already
> >>>>> provides that functionality, so it might not be necessary.
> >>>>>
> >>>> How about adding an extra command option like --terse-header that
> >>>> prints out the header only? It is hard to keep track of these version
> >>>> changes in one place at the user manual or some blog post.
> >>>> with a terse header command option, I can ignore the terse version and
> >>>> adjust my test scripts to query the header first before running the
> >>>> test, then pull the right column after the test finishes.
> >>>
> >>> You would then need to document all the terse formats because you
> >>> never know which one someone will use (it looks like there are four:
> >>> http://fio.readthedocs.io/en/latest/fio_doc.html#cmdoption-terse-version
> >>> ). At this stage shouldn't we be nudging people towards JSON which is
> >>> more self describing and easier to extend in a backwards compatible
> >>> manner?
> >>
> >> I am on board with your suggestion but not every user might have a
> >> time/resource to re-implement their existing test script to work with
> >> json. We should stop adding more versions of the terse. Otherwise, If
> >> we want to continue supporting the terse format, I recommend to
> >> figure out a programmatic way to handle its complexity.
> >
> > How about defining that the next terse version always includes
> > a header line, so it'll be self-describing and can be the final
> > terse version.
>
> I like that idea. We already have a version at the front, proper scripts
> should be checking that, and we can further make it safer by adding
> new fields at the end.
>
> Honestly, I really dislike the terse format. But it's there and some
> people like to use it, and it's a shame if we can add new additions. It's
> already trailing the json/normal output in various cases, for instance
> it doesn't include fsync output (and I'm sure others). If at all possible,
> I always push people towards json instead of the nasty terse output.
>
> --
> Jens Axboe


________________________________

This email is confidential and intended only for the use of the individual or entity named above and may contain information that is privileged. If you are not the intended recipient, you are notified that any dissemination, distribution or copying of this email is strictly prohibited. If you have received this email in error, please notify us immediately by return email or telephone and destroy the original message. - This mail is sent via Sony Asia Pacific Mail Gateway..

Re: [PATCH] idle-prof: append output cpu idleness data to terse log

[Jens Axboe]

On 7/25/18 1:47 AM, Friendy.Su@sony.com wrote:
> Many people are still used to process, plot data by excel. As a format which can be opened directly by excel, terse is welcome.
> Exactly terse log has already trailing the normal/json log, improving it inside fio looks a redundant work. Instead, how about provide a script to convert json to terse under tools/?Meanwhile, in HOWTO doc, tell user to use that script to get a 'perfect' terse.

Yeah, I was thinking the exact same thing while mulling over this yesterday.
I think that'd be great, and would allow us to kill the terse code in fio
proper.

-- 
Jens Axboe

RE: [PATCH] idle-prof: append output cpu idleness data to terse log

[Kris Davis]

I don't really use the terse output myself, but the terse labels from the HOWTO include separate values for read and writes.  However, the json output will only include 'mixed' results when --unified_rw_reporting is set.   Also, the terse reported clat percentiles seem to be different than what is indicated in --percentiles option in json. So with a script generating terse results from the json data, the results wouldn't be strictly identical, but with headers included using scripts could be flexible.  

For what it's worth, I couple of months ago, I created a python3 module and class which reads fio v3+ and can produce a normal "like" result and a 'terse' like result with headers, along with some methods to interrogate overall iops, mips, etc (for use in a scripts).  I later discovered I could list all three (json,normal and terse) in the output option, and so mostly use that in scripts to have "human" readable and json script readable.  So I haven't really completely tested all this code.  I mostly use it to check the iops, mibs, latencies in scripts, and then strip out the "normal" output for display.  However, it also include 'terse' like results.  I've done some cursory testing.

I've added arguments to the modules main section to turn into a script, and it may be a starting point.  It would also need help test the corner cases, since there are so many options within Fio.  I'm working on some other things which have priority at the moment, but anyone is welcome to run with it if you need something quickly.    

I put a starting proposal script call fioJasonInterpreter.py at https://github.com/shimrot/fio/tree/json_output_conversion_script/tools . I imagine a script for public consumption might want a different name.

Thanks

Kris Davis

-----Original Message-----
From: fio-owner@vger.kernel.org [mailto:fio-owner@vger.kernel.org] On Behalf Of Jens Axboe
Sent: Wednesday, July 25, 2018 9:48 AM
To: Friendy.Su@sony.com; elliott@hpe.com; haghdoost@gmail.com; sitsofe@gmail.com
Cc: fio@vger.kernel.org; No.Tanaka@sony.com; Hajime.Tomura@sony.com
Subject: Re: [PATCH] idle-prof: append output cpu idleness data to terse log

On 7/25/18 1:47 AM, Friendy.Su@sony.com wrote:
> Many people are still used to process, plot data by excel. As a format which can be opened directly by excel, terse is welcome.
> Exactly terse log has already trailing the normal/json log, improving it inside fio looks a redundant work. Instead, how about provide a script to convert json to terse under tools/?Meanwhile, in HOWTO doc, tell user to use that script to get a 'perfect' terse.

Yeah, I was thinking the exact same thing while mulling over this yesterday.
I think that'd be great, and would allow us to kill the terse code in fio proper.

--
Jens Axboe