[Fuego] Problem of adding a benchmark result table

[Fuego] Problem of adding a benchmark result table

[Li, Xiaoming]

[-- Attachment #1: Type: text/plain, Size: 832 bytes --]



Hi all，



I want to add a table of building results for every single benchmark in Jenkins.



My solutions is to use jenkins's plugin groovy postbuild, which can add a hyperlink in build history.



But groovy postbuild requies jenkins‘s version of 2.6.0 or later, as fuego's default is 2.32.1.



I also have to install  java8 to run higher version jenkins.



Could anyone give me a better idea?









BR

Li



--------------------------------------------------

Li, Xiaoming

Development Dept.III

Nanjing Fujitsu Nanda Software Tech. Co., Ltd.(FNST) No. 6 Wenzhu Road, Nanjing, 210012, China

TEL: +86+25-86630566-8451

FAX: +86+25-83317685

MAIL: lixm.fnst@cn.fujitsu.com<mailto:lixm.fnst@cn.fujitsu.com>

--------------------------------------------------






[-- Attachment #2: Type: text/html, Size: 5161 bytes --]

Re: [Fuego] Problem of adding a benchmark result table

[Daniel Sangorrin]

Hi Li,

> <fuego-bounces@lists.linuxfoundation.org> On Behalf Of Li, Xiaoming
 > I want to add a table of building results for every single benchmark in Jenkins.

If you want to use a table to show the building results instead of a plot, you can change the chart_type from chart_config.json to testset_summary_table. If you run the benchmark again, you should get a table instead of a plot.

That functionality is implemented at:
scripts/parser/prepare_chart_data.py
scripts/mod.js

I believe that code needs some improvements:
   - ability to override chart_config.json with a per-board configuration
   - I have seen in Dhrystone that the table shows pass=1 but also err=1
   - The tables only show building results (pass/fail), you may want to add the benchmark measures

> My solutions is to use jenkins's plugin groovy postbuild, which can add a hyperlink
> in build history.
> But groovy postbuild requies jenkins‘s version of 2.6.0 or later, as fuego's default is
> 2.32.1.

Updating jenkins regularly is important. However, we have had bad experiences with Jenkins' plugins. Each time we upgraded Jenkins a number of plugins where not available anymore causing many breakages. For that reason, we decided to limit the usage of plugins to the minimum necessary.

> I also have to install  java8 to run higher version jenkins.
This would require changing the Dockerfile. It might be a hard job if it can't be done with Debian Jessie.

> Could anyone give me a better idea?

Apart from using chart_config.son, there is another option. We have an SQL-like interface to fetch results from Fuego. You can use that interface with ftc gen-report and create an html/xls/pdf report with all of the build results that you want. Maybe you could even call to ftc gen-report in a postbuild action and then add a link to the generated html from the "Set build description" post action.

Also, soon we will be able to post test results to kernelci and squad instances.

Best regards,
Daniel

> --------------------------------------------------
> 
> Li, Xiaoming
> 
> Development Dept.III
> 
> Nanjing Fujitsu Nanda Software Tech. Co., Ltd.(FNST) No. 6 Wenzhu Road, Nanjing,
> 210012, China
> 
> TEL: +86+25-86630566-8451
> 
> FAX: +86+25-83317685
> 
> MAIL: lixm.fnst@cn.fujitsu.com <mailto:lixm.fnst@cn.fujitsu.com>
> 
> --------------------------------------------------
> 
> 

Re: [Fuego] Problem of adding a benchmark result table

[Li, Xiaoming]

Hi Daniel,

> -----Original Message-----
> From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp]
> Sent: Friday, August 24, 2018 11:11 AM
> To: Li, Xiaoming/李 霄鸣; fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] Problem of adding a benchmark result table
> 
> Hi Li,
> 
> > <fuego-bounces@lists.linuxfoundation.org> On Behalf Of Li, Xiaoming
>  > I want to add a table of building results for every single benchmark in
> Jenkins.
> 
> If you want to use a table to show the building results instead of a plot, you
> can change the chart_type from chart_config.json to testset_summary_table.
> If you run the benchmark again, you should get a table instead of a plot.

> 
> That functionality is implemented at:
> scripts/parser/prepare_chart_data.py
> scripts/mod.js
> 
> I believe that code needs some improvements:
>    - ability to override chart_config.json with a per-board configuration
>    - I have seen in Dhrystone that the table shows pass=1 but also err=1
>    - The tables only show building results (pass/fail), you may want to add the
> benchmark measures
> 
> > My solutions is to use jenkins's plugin groovy postbuild, which can
> > add a hyperlink in build history.
> > But groovy postbuild requies jenkins‘s version of 2.6.0 or later, as
> > fuego's default is 2.32.1.
> 
> Updating jenkins regularly is important. However, we have had bad
> experiences with Jenkins' plugins. Each time we upgraded Jenkins a number
> of plugins where not available anymore causing many breakages. For that
> reason, we decided to limit the usage of plugins to the minimum necessary.
> 
> > I also have to install  java8 to run higher version jenkins.
> This would require changing the Dockerfile. It might be a hard job if it can't be
> done with Debian Jessie.
> 
> > Could anyone give me a better idea?
> 
> Apart from using chart_config.son, there is another option. We have an SQL-
> like interface to fetch results from Fuego. You can use that interface with ftc
> gen-report and create an html/xls/pdf report with all of the build results that
> you want. Maybe you could even call to ftc gen-report in a postbuild action
> and then add a link to the generated html from the "Set build description"
> post action.

I have tried it , and it works.
But the  --where batch_id arg  can not work temporarily,   I will  investigate it sometime later.
On the whole,  it is a  wonderful  method.



> 
> Also, soon we will be able to post test results to kernelci and squad instances.
> 
> Best regards,
> Daniel
> 
> > --------------------------------------------------
> >
> > Li, Xiaoming
> >
> > Development Dept.III
> >
> > Nanjing Fujitsu Nanda Software Tech. Co., Ltd.(FNST) No. 6 Wenzhu
> > Road, Nanjing, 210012, China
> >
> > TEL: +86+25-86630566-8451
> >
> > FAX: +86+25-83317685
> >
> > MAIL: lixm.fnst@cn.fujitsu.com <mailto:lixm.fnst@cn.fujitsu.com>
> >
> > --------------------------------------------------
> >
> >
> 
> 
> 


Thx 
Li

Re: [Fuego] Problem of adding a benchmark result table

[Tim.Bird]

[-- Attachment #1: Type: text/plain, Size: 4575 bytes --]

On the whole, I'd like to extend our current charting mechanisms to support what you need. scripts/mod.js can support arbitrary HTML generated by prepare_chart_data.py, so we should be able to make it work.

With regard to batch_id, that feature is not finished yet. Filtering on it is trivial to add to ftc and prepare_chart_data.py.

But I don't think the field is recorded in run.json and the other files used for charting. The key issue outstanding was the mechanism for assigning the batch_id for runs. The idea was to have a test plan job set the batch_id for the jobs it started. But there's not a good Jenkins mechanism to do this. When we finished having Jenkins call ftc run-test to run test jobs, my plan was to add a new ftc run-plan, and have Jenkins call that for executing batch jobs. Thus would allow me to set a batch ID for the jobs associated with the batch run, I think.

Anyway, there just needs to be some batch_id on the job command line, or in an environment variable, that gets put into the run.json file.

I'll take a look at this next week.

Regards,
- - Tim


-------- Original Message --------
Subject: Re: [Fuego] Problem of adding a benchmark result table
From: "Li, Xiaoming" <lixm.fnst@cn.fujitsu.com>
Date: Aug 24, 2018, 12:57 AM
To: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>,fuego@lists.linuxfoundation.org
Hi Daniel,

> -----Original Message-----
> From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp]
> Sent: Friday, August 24, 2018 11:11 AM
> To: Li, Xiaoming/李 霄鸣; fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] Problem of adding a benchmark result table
>
> Hi Li,
>
> > <fuego-bounces@lists.linuxfoundation.org> On Behalf Of Li, Xiaoming
>  > I want to add a table of building results for every single benchmark in
> Jenkins.
>
> If you want to use a table to show the building results instead of a plot, you
> can change the chart_type from chart_config.json to testset_summary_table.
> If you run the benchmark again, you should get a table instead of a plot.

>
> That functionality is implemented at:
> scripts/parser/prepare_chart_data.py
> scripts/mod.js
>
> I believe that code needs some improvements:
>    - ability to override chart_config.json with a per-board configuration
>    - I have seen in Dhrystone that the table shows pass=1 but also err=1
>    - The tables only show building results (pass/fail), you may want to add the
> benchmark measures
>
> > My solutions is to use jenkins's plugin groovy postbuild, which can
> > add a hyperlink in build history.
> > But groovy postbuild requies jenkins‘s version of 2.6.0 or later, as
> > fuego's default is 2.32.1.
>
> Updating jenkins regularly is important. However, we have had bad
> experiences with Jenkins' plugins. Each time we upgraded Jenkins a number
> of plugins where not available anymore causing many breakages. For that
> reason, we decided to limit the usage of plugins to the minimum necessary.
>
> > I also have to install  java8 to run higher version jenkins.
> This would require changing the Dockerfile. It might be a hard job if it can't be
> done with Debian Jessie.
>
> > Could anyone give me a better idea?
>
> Apart from using chart_config.son, there is another option. We have an SQL-
> like interface to fetch results from Fuego. You can use that interface with ftc
> gen-report and create an html/xls/pdf report with all of the build results that
> you want. Maybe you could even call to ftc gen-report in a postbuild action
> and then add a link to the generated html from the "Set build description"
> post action.

I have tried it , and it works.
But the  --where batch_id arg  can not work temporarily,   I will  investigate it sometime later.
On the whole,  it is a  wonderful  method.



>
> Also, soon we will be able to post test results to kernelci and squad instances.
>
> Best regards,
> Daniel
>
> > --------------------------------------------------
> >
> > Li, Xiaoming
> >
> > Development Dept.III
> >
> > Nanjing Fujitsu Nanda Software Tech. Co., Ltd.(FNST) No. 6 Wenzhu
> > Road, Nanjing, 210012, China
> >
> > TEL: +86+25-86630566-8451
> >
> > FAX: +86+25-83317685
> >
> > MAIL: lixm.fnst@cn.fujitsu.com <mailto:lixm.fnst@cn.fujitsu.com>
> >
> > --------------------------------------------------
> >
> >
>
>
>


Thx
Li


_______________________________________________
Fuego mailing list
Fuego@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/fuego

[-- Attachment #2: Type: text/html, Size: 6315 bytes --]

Re: [Fuego] Problem of adding a benchmark result table

[Li, Xiaoming]

[-- Attachment #1: Type: text/plain, Size: 5572 bytes --]

Hi Tim,

As I am urge of implement this function, could u give me some hints or advice  on questions below?

1)If modiying the mod.js to generate a single benchmark report, does it need the batch_id arg? or parse the report file with own algorithm?

2)My plan is to add a chart on top of test suite's flot graph, with a menu of selecting build no. (sorry, I am not an expert of web design)

3)Do u have any schedule of re-investigating the fuego chart（or graph,table...） related mechanism?

THX
Li

From: Tim.Bird@sony.com [mailto:Tim.Bird@sony.com]
Sent: Friday, August 24, 2018 11:59 PM
To: Li, Xiaoming/李 霄鸣; daniel.sangorrin@toshiba.co.jp; fuego@lists.linuxfoundation.org
Subject: Re: [Fuego] Problem of adding a benchmark result table

On the whole, I'd like to extend our current charting mechanisms to support what you need. scripts/mod.js can support arbitrary HTML generated by prepare_chart_data.py, so we should be able to make it work.

With regard to batch_id, that feature is not finished yet. Filtering on it is trivial to add to ftc and prepare_chart_data.py.

But I don't think the field is recorded in run.json and the other files used for charting. The key issue outstanding was the mechanism for assigning the batch_id for runs. The idea was to have a test plan job set the batch_id for the jobs it started. But there's not a good Jenkins mechanism to do this. When we finished having Jenkins call ftc run-test to run test jobs, my plan was to add a new ftc run-plan, and have Jenkins call that for executing batch jobs. Thus would allow me to set a batch ID for the jobs associated with the batch run, I think.

Anyway, there just needs to be some batch_id on the job command line, or in an environment variable, that gets put into the run.json file.

I'll take a look at this next week.

Regards,
- - Tim

-------- Original Message --------
Subject: Re: [Fuego] Problem of adding a benchmark result table
From: "Li, Xiaoming" <lixm.fnst@cn.fujitsu.com<mailto:lixm.fnst@cn.fujitsu.com>>
Date: Aug 24, 2018, 12:57 AM
To: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp<mailto:daniel.sangorrin@toshiba.co.jp>>,fuego@lists.linuxfoundation.org
Hi Daniel,

> -----Original Message-----
> From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp]
> Sent: Friday, August 24, 2018 11:11 AM
> To: Li, Xiaoming/李 霄鸣; fuego@lists.linuxfoundation.org<mailto:fuego@lists.linuxfoundation.org>
> Subject: RE: [Fuego] Problem of adding a benchmark result table
>
> Hi Li,
>
> > <fuego-bounces@lists.linuxfoundation.org<mailto:fuego-bounces@lists.linuxfoundation.org>> On Behalf Of Li, Xiaoming
>  > I want to add a table of building results for every single benchmark in
> Jenkins.
>
> If you want to use a table to show the building results instead of a plot, you
> can change the chart_type from chart_config.json to testset_summary_table.
> If you run the benchmark again, you should get a table instead of a plot.

>
> That functionality is implemented at:
> scripts/parser/prepare_chart_data.py
> scripts/mod.js
>
> I believe that code needs some improvements:
>    - ability to override chart_config.json with a per-board configuration
>    - I have seen in Dhrystone that the table shows pass=1 but also err=1
>    - The tables only show building results (pass/fail), you may want to add the
> benchmark measures
>
> > My solutions is to use jenkins's plugin groovy postbuild, which can
> > add a hyperlink in build history.
> > But groovy postbuild requies jenkins‘s version of 2.6.0 or later, as
> > fuego's default is 2.32.1.
>
> Updating jenkins regularly is important. However, we have had bad
> experiences with Jenkins' plugins. Each time we upgraded Jenkins a number
> of plugins where not available anymore causing many breakages. For that
> reason, we decided to limit the usage of plugins to the minimum necessary.
>
> > I also have to install  java8 to run higher version jenkins.
> This would require changing the Dockerfile. It might be a hard job if it can't be
> done with Debian Jessie.
>
> > Could anyone give me a better idea?
>
> Apart from using chart_config.son, there is another option. We have an SQL-
> like interface to fetch results from Fuego. You can use that interface with ftc
> gen-report and create an html/xls/pdf report with all of the build results that
> you want. Maybe you could even call to ftc gen-report in a postbuild action
> and then add a link to the generated html from the "Set build description"
> post action.

I have tried it , and it works.
But the  --where batch_id arg  can not work temporarily,   I will  investigate it sometime later.
On the whole,  it is a  wonderful  method.



>
> Also, soon we will be able to post test results to kernelci and squad instances.
>
> Best regards,
> Daniel
>
> > --------------------------------------------------
> >
> > Li, Xiaoming
> >
> > Development Dept.III
> >
> > Nanjing Fujitsu Nanda Software Tech. Co., Ltd.(FNST) No. 6 Wenzhu
> > Road, Nanjing, 210012, China
> >
> > TEL: +86+25-86630566-8451
> >
> > FAX: +86+25-83317685
> >
> > MAIL: lixm.fnst@cn.fujitsu.com<mailto:lixm.fnst@cn.fujitsu.com> <mailto:lixm.fnst@cn.fujitsu.com>
> >
> > --------------------------------------------------
> >
> >
>
>
>


Thx
Li


_______________________________________________
Fuego mailing list
Fuego@lists.linuxfoundation.org<mailto:Fuego@lists.linuxfoundation.org>
https://lists.linuxfoundation.org/mailman/listinfo/fuego



[-- Attachment #2: Type: text/html, Size: 12963 bytes --]

Re: [Fuego] Problem of adding a benchmark result table

[Tim.Bird]

> -----Original Message-----
> From: Li, Xiaoming 
> 
> Hi Tim,
> 
> 
> 
> As I am urge of implement this function, could u give me some hints or advice
> on questions below?

You should look at:
http://fuegotest.org/wiki/Jenkins_Visualization
It has some information about the charting features of Fuego, and the integration
with Jenkins.

> 
> 
> 
> 1)If modiying the mod.js to generate a single benchmark report, does it need
> the batch_id arg? or parse the report file with own algorithm?

Currently, Fuego does not have a batch_id implemented.  It does not appear
in run.json, or in flat_plot_data.txt, or flot_chart-data.json.  Although it is mentioned
in the ftc command-line help for --where, it isn't actually implemented yet.
(Note that if run.json *had* a batch_id field, the code in ftc that handled --where
clauses should function properly for filtering this.)

prepare_chart_data.py parses the run.json data with its own algorithm.
 
> 
> 2)My plan is to add a chart on top of test suite's flot graph, with a menu of
> selecting build no. (sorry, I am not an expert of web design)
This sounds useful, but might be complicated.
Does this 'chart on top of ...the flog graph' contain data, or is it just
a list of checkboxes for selecting the data to include in the graph?

Just to clarify the terminology, I use the following (hopefully consistently):
chart = visualization item, which can be a table or a plot (graph)

There are three types of charts:
1) measure_plot = a 'flot' plot of benchmark measures
2) testcase_table = an HTML table of testcase results
3) testcase_summary_table = an HTML table of summary counts per testset
 
I wanted for the system (prepare_chart_data.py and mod.js) to support
being able to show multiple charts for the same test, and being able to
support pre-filtering (by prepare_chart_data.py) and visualization-time
filtering (by mod.js using flot capabilities to turn items on and off in the
Jenkins user interface).

I'm not sure, however, of the status of this code.  It looks like mod.js
can read multiple charts from the flot_chart_data.json file, but that
prepare_chart_data.py only writes one chart (or one kind of chart)
to that file.

> 
> 
> 3)Do u have any schedule of re-investigating the fuego chart（or
> graph,table...） related mechanism?
I don't have a schedule.  I might be able to take a look at it this week,
but I am WAY behind in my patch review for new tests and test fixes.  Don't get
me wrong - I am grateful for all the work by Fujitsu engineers, but
I need to catch up on about 20 to 30 patches.  It will take me probably a few
days at least to get to this.

 -- Tim


> From: Tim.Bird@sony.com [mailto:Tim.Bird@sony.com]
> 
> 
> On the whole, I'd like to extend our current charting mechanisms to support
> what you need. scripts/mod.js can support arbitrary HTML generated by
> prepare_chart_data.py, so we should be able to make it work.
> 
> 
> 
> With regard to batch_id, that feature is not finished yet. Filtering on it is trivial
> to add to ftc and prepare_chart_data.py.
> 
> 
> 
> But I don't think the field is recorded in run.json and the other files used for
> charting. The key issue outstanding was the mechanism for assigning the
> batch_id for runs. The idea was to have a test plan job set the batch_id for
> the jobs it started. But there's not a good Jenkins mechanism to do this.
> When we finished having Jenkins call ftc run-test to run test jobs, my plan
> was to add a new ftc run-plan, and have Jenkins call that for executing batch
> jobs. Thus would allow me to set a batch ID for the jobs associated with the
> batch run, I think.
> 
> 
> 
> Anyway, there just needs to be some batch_id on the job command line, or
> in an environment variable, that gets put into the run.json file.
> 
> 
> 
> I'll take a look at this next week.
> 
> 
> 
> Regards,
> 
> - - Tim
> 
> 
> 
> -------- Original Message --------
> 
> Subject: Re: [Fuego] Problem of adding a benchmark result table
> 
> From: "Li, Xiaoming" <lixm.fnst@cn.fujitsu.com
> <mailto:lixm.fnst@cn.fujitsu.com> >
> 
> Date: Aug 24, 2018, 12:57 AM
> 
> To: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp
> <mailto:daniel.sangorrin@toshiba.co.jp> >,fuego@lists.linuxfoundation.org
> 
> Hi Daniel,
> 
> > -----Original Message-----
> > From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp]
> > Sent: Friday, August 24, 2018 11:11 AM
> > To: Li, Xiaoming/李 霄鸣; fuego@lists.linuxfoundation.org
> <mailto:fuego@lists.linuxfoundation.org>
> > Subject: RE: [Fuego] Problem of adding a benchmark result table
> >
> > Hi Li,
> >
> > > <fuego-bounces@lists.linuxfoundation.org <mailto:fuego-
> bounces@lists.linuxfoundation.org> > On Behalf Of Li, Xiaoming
> >  > I want to add a table of building results for every single benchmark in
> > Jenkins.
> >
> > If you want to use a table to show the building results instead of a plot, you
> > can change the chart_type from chart_config.json to
> testset_summary_table.
> > If you run the benchmark again, you should get a table instead of a plot.
> 
> >
> > That functionality is implemented at:
> > scripts/parser/prepare_chart_data.py
> > scripts/mod.js
> >
> > I believe that code needs some improvements:
> >    - ability to override chart_config.json with a per-board configuration
> >    - I have seen in Dhrystone that the table shows pass=1 but also err=1
> >    - The tables only show building results (pass/fail), you may want to add
> the
> > benchmark measures
> >
> > > My solutions is to use jenkins's plugin groovy postbuild, which can
> > > add a hyperlink in build history.
> > > But groovy postbuild requies jenkins‘s version of 2.6.0 or later, as
> > > fuego's default is 2.32.1.
> >
> > Updating jenkins regularly is important. However, we have had bad
> > experiences with Jenkins' plugins. Each time we upgraded Jenkins a
> number
> > of plugins where not available anymore causing many breakages. For that
> > reason, we decided to limit the usage of plugins to the minimum necessary.
> >
> > > I also have to install  java8 to run higher version jenkins.
> > This would require changing the Dockerfile. It might be a hard job if it can't
> be
> > done with Debian Jessie.
> >
> > > Could anyone give me a better idea?
> >
> > Apart from using chart_config.son, there is another option. We have an
> SQL-
> > like interface to fetch results from Fuego. You can use that interface with
> ftc
> > gen-report and create an html/xls/pdf report with all of the build results
> that
> > you want. Maybe you could even call to ftc gen-report in a postbuild action
> > and then add a link to the generated html from the "Set build description"
> > post action.
> 
> I have tried it , and it works.
> But the  --where batch_id arg  can not work temporarily,   I will  investigate it
> sometime later.
> On the whole,  it is a  wonderful  method.
> 
> 
> 
> >
> > Also, soon we will be able to post test results to kernelci and squad
> instances.
> >
> > Best regards,
> > Daniel
> >
> > > --------------------------------------------------
> > >
> > > Li, Xiaoming
> > >
> > > Development Dept.III
> > >
> > > Nanjing Fujitsu Nanda Software Tech. Co., Ltd.(FNST) No. 6 Wenzhu
> > > Road, Nanjing, 210012, China
> > >
> > > TEL: +86+25-86630566-8451
> > >
> > > FAX: +86+25-83317685
> > >
> > > MAIL: lixm.fnst@cn.fujitsu.com <mailto:lixm.fnst@cn.fujitsu.com>
> <mailto:lixm.fnst@cn.fujitsu.com>
> > >
> > > --------------------------------------------------
> > >
> > >
> >
> >
> >
> 
> 
> Thx
> Li
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org <mailto:Fuego@lists.linuxfoundation.org>
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] Problem of adding a benchmark result table

[Li, Xiaoming]

Hi Tim,

THX for your prompt instruction.

My  reply is below.

> -----Original Message-----
> From: Tim.Bird@sony.com [mailto:Tim.Bird@sony.com]
> Sent: Tuesday, August 28, 2018 7:49 AM
> To: Li, Xiaoming/李 霄鸣; fuego@lists.linuxfoundation.org
> Cc: daniel.sangorrin@toshiba.co.jp
> Subject: RE: [Fuego] Problem of adding a benchmark result table
> 
> 
> 
> > -----Original Message-----
> > From: Li, Xiaoming
> >
> > Hi Tim,
> >
> >
> >
> > As I am urge of implement this function, could u give me some hints or
> > advice on questions below?
> 
> You should look at:
> http://fuegotest.org/wiki/Jenkins_Visualization
> It has some information about the charting features of Fuego, and the
> integration with Jenkins.

Great, I found much valuable info there.

> 
> >
> >
> >
> > 1)If modiying the mod.js to generate a single benchmark report, does
> > it need the batch_id arg? or parse the report file with own algorithm?
> 
> Currently, Fuego does not have a batch_id implemented.  It does not appear
> in run.json, or in flat_plot_data.txt, or flot_chart-data.json.  Although it is
> mentioned in the ftc command-line help for --where, it isn't actually
> implemented yet.
> (Note that if run.json *had* a batch_id field, the code in ftc that handled --
> where clauses should function properly for filtering this.)
> 
> prepare_chart_data.py parses the run.json data with its own algorithm.
> 
> >
> > 2)My plan is to add a chart on top of test suite's flot graph, with a
> > menu of selecting build no. (sorry, I am not an expert of web design)
> This sounds useful, but might be complicated.
> Does this 'chart on top of ...the flog graph' contain data, or is it just a list of
> checkboxes for selecting the data to include in the graph?
> 
> Just to clarify the terminology, I use the following (hopefully consistently):
> chart = visualization item, which can be a table or a plot (graph)
> 
> There are three types of charts:
> 1) measure_plot = a 'flot' plot of benchmark measures
> 2) testcase_table = an HTML table of testcase results
> 3) testcase_summary_table = an HTML table of summary counts per testset
> 
> I wanted for the system (prepare_chart_data.py and mod.js) to support
> being able to show multiple charts for the same test, and being able to
> support pre-filtering (by prepare_chart_data.py) and visualization-time
> filtering (by mod.js using flot capabilities to turn items on and off in the
> Jenkins user interface).
> 
> I'm not sure, however, of the status of this code.  It looks like mod.js can
> read multiple charts from the flot_chart_data.json file, but that
> prepare_chart_data.py only writes one chart (or one kind of chart) to that
> file.

My head is messed up by their complex relation.
Maybe I'd better  read the code fully before starting new work. 

> 
> >
> >
> > 3)Do u have any schedule of re-investigating the fuego chart（or
> > graph,table...） related mechanism?
> I don't have a schedule.  I might be able to take a look at it this week, but I am
> WAY behind in my patch review for new tests and test fixes.  Don't get me
> wrong - I am grateful for all the work by Fujitsu engineers, but I need to catch
> up on about 20 to 30 patches.  It will take me probably a few days at least to
> get to this.

Please forgive my rudeness,  which is probably due to my poor English(or differences between culture ). 
It is not my intention.


> 
>  -- Tim
> 
> 
> > From: Tim.Bird@sony.com [mailto:Tim.Bird@sony.com]
> >
> >
> > On the whole, I'd like to extend our current charting mechanisms to
> > support what you need. scripts/mod.js can support arbitrary HTML
> > generated by prepare_chart_data.py, so we should be able to make it
> work.
> >
> >
> >
> > With regard to batch_id, that feature is not finished yet. Filtering
> > on it is trivial to add to ftc and prepare_chart_data.py.
> >
> >
> >
> > But I don't think the field is recorded in run.json and the other
> > files used for charting. The key issue outstanding was the mechanism
> > for assigning the batch_id for runs. The idea was to have a test plan
> > job set the batch_id for the jobs it started. But there's not a good Jenkins
> mechanism to do this.
> > When we finished having Jenkins call ftc run-test to run test jobs, my
> > plan was to add a new ftc run-plan, and have Jenkins call that for
> > executing batch jobs. Thus would allow me to set a batch ID for the
> > jobs associated with the batch run, I think.
> >
> >
> >
> > Anyway, there just needs to be some batch_id on the job command line,
> > or in an environment variable, that gets put into the run.json file.
> >
> >
> >
> > I'll take a look at this next week.
> >
> >
> >
> > Regards,
> >
> > - - Tim
> >
> >
> >
> > -------- Original Message --------
> >
> > Subject: Re: [Fuego] Problem of adding a benchmark result table
> >
> > From: "Li, Xiaoming" <lixm.fnst@cn.fujitsu.com
> > <mailto:lixm.fnst@cn.fujitsu.com> >
> >
> > Date: Aug 24, 2018, 12:57 AM
> >
> > To: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp
> > <mailto:daniel.sangorrin@toshiba.co.jp>
> > >,fuego@lists.linuxfoundation.org
> >
> > Hi Daniel,
> >
> > > -----Original Message-----
> > > From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp]
> > > Sent: Friday, August 24, 2018 11:11 AM
> > > To: Li, Xiaoming/李 霄鸣; fuego@lists.linuxfoundation.org
> > <mailto:fuego@lists.linuxfoundation.org>
> > > Subject: RE: [Fuego] Problem of adding a benchmark result table
> > >
> > > Hi Li,
> > >
> > > > <fuego-bounces@lists.linuxfoundation.org <mailto:fuego-
> > bounces@lists.linuxfoundation.org> > On Behalf Of Li, Xiaoming
> > >  > I want to add a table of building results for every single
> > > benchmark in Jenkins.
> > >
> > > If you want to use a table to show the building results instead of a
> > > plot, you can change the chart_type from chart_config.json to
> > testset_summary_table.
> > > If you run the benchmark again, you should get a table instead of a plot.
> >
> > >
> > > That functionality is implemented at:
> > > scripts/parser/prepare_chart_data.py
> > > scripts/mod.js
> > >
> > > I believe that code needs some improvements:
> > >    - ability to override chart_config.json with a per-board configuration
> > >    - I have seen in Dhrystone that the table shows pass=1 but also err=1
> > >    - The tables only show building results (pass/fail), you may want
> > > to add
> > the
> > > benchmark measures
> > >
> > > > My solutions is to use jenkins's plugin groovy postbuild, which
> > > > can add a hyperlink in build history.
> > > > But groovy postbuild requies jenkins‘s version of 2.6.0 or later,
> > > > as fuego's default is 2.32.1.
> > >
> > > Updating jenkins regularly is important. However, we have had bad
> > > experiences with Jenkins' plugins. Each time we upgraded Jenkins a
> > number
> > > of plugins where not available anymore causing many breakages. For
> > > that reason, we decided to limit the usage of plugins to the minimum
> necessary.
> > >
> > > > I also have to install  java8 to run higher version jenkins.
> > > This would require changing the Dockerfile. It might be a hard job
> > > if it can't
> > be
> > > done with Debian Jessie.
> > >
> > > > Could anyone give me a better idea?
> > >
> > > Apart from using chart_config.son, there is another option. We have
> > > an
> > SQL-
> > > like interface to fetch results from Fuego. You can use that
> > > interface with
> > ftc
> > > gen-report and create an html/xls/pdf report with all of the build
> > > results
> > that
> > > you want. Maybe you could even call to ftc gen-report in a postbuild
> > > action and then add a link to the generated html from the "Set build
> description"
> > > post action.
> >
> > I have tried it , and it works.
> > But the  --where batch_id arg  can not work temporarily,   I will  investigate
> it
> > sometime later.
> > On the whole,  it is a  wonderful  method.
> >
> >
> >
> > >
> > > Also, soon we will be able to post test results to kernelci and
> > > squad
> > instances.
> > >
> > > Best regards,
> > > Daniel
> > >
> > > > --------------------------------------------------
> > > >
> > > > Li, Xiaoming
> > > >
> > > > Development Dept.III
> > > >
> > > > Nanjing Fujitsu Nanda Software Tech. Co., Ltd.(FNST) No. 6 Wenzhu
> > > > Road, Nanjing, 210012, China
> > > >
> > > > TEL: +86+25-86630566-8451
> > > >
> > > > FAX: +86+25-83317685
> > > >
> > > > MAIL: lixm.fnst@cn.fujitsu.com <mailto:lixm.fnst@cn.fujitsu.com>
> > <mailto:lixm.fnst@cn.fujitsu.com>
> > > >
> > > > --------------------------------------------------
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> > Thx
> > Li
> >
> >
> > _______________________________________________
> > Fuego mailing list
> > Fuego@lists.linuxfoundation.org
> > <mailto:Fuego@lists.linuxfoundation.org>
> > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> 
> 


Best regard 
Li

Re: [Fuego] Problem of adding a benchmark result table

[Tim.Bird]

A few more comments inline below...

> -----Original Message-----
> From: Li, Xiaoming 
...
> My  reply is below.
> 
> > -----Original Message-----
> > From: Tim.Bird@sony.com 
> > > -----Original Message-----
> > > From: Li, Xiaoming
> > >
> > > Hi Tim,
> > >
> > >
> > >
> > > As I am urge of implement this function, could u give me some hints or
> > > advice on questions below?
> >
> > You should look at:
> > http://fuegotest.org/wiki/Jenkins_Visualization
> > It has some information about the charting features of Fuego, and the
> > integration with Jenkins.
> 
> Great, I found much valuable info there.
> 
> >
> > >
> > >
> > >
> > > 1)If modiying the mod.js to generate a single benchmark report, does
> > > it need the batch_id arg? or parse the report file with own algorithm?
> >
> > Currently, Fuego does not have a batch_id implemented.  It does not
> appear
> > in run.json, or in flat_plot_data.txt, or flot_chart-data.json.  Although it is
> > mentioned in the ftc command-line help for --where, it isn't actually
> > implemented yet.
> > (Note that if run.json *had* a batch_id field, the code in ftc that handled --
> > where clauses should function properly for filtering this.)
> >
> > prepare_chart_data.py parses the run.json data with its own algorithm.
> >
> > >
> > > 2)My plan is to add a chart on top of test suite's flot graph, with a
> > > menu of selecting build no. (sorry, I am not an expert of web design)
> > This sounds useful, but might be complicated.
> > Does this 'chart on top of ...the flog graph' contain data, or is it just a list of
> > checkboxes for selecting the data to include in the graph?
> >
> > Just to clarify the terminology, I use the following (hopefully consistently):
> > chart = visualization item, which can be a table or a plot (graph)
> >
> > There are three types of charts:
> > 1) measure_plot = a 'flot' plot of benchmark measures
> > 2) testcase_table = an HTML table of testcase results
> > 3) testcase_summary_table = an HTML table of summary counts per
> testset
> >
> > I wanted for the system (prepare_chart_data.py and mod.js) to support
> > being able to show multiple charts for the same test, and being able to
> > support pre-filtering (by prepare_chart_data.py) and visualization-time
> > filtering (by mod.js using flot capabilities to turn items on and off in the
> > Jenkins user interface).
> >
> > I'm not sure, however, of the status of this code.  It looks like mod.js can
> > read multiple charts from the flot_chart_data.json file, but that
> > prepare_chart_data.py only writes one chart (or one kind of chart) to that
> > file.
> 
> My head is messed up by their complex relation.
> Maybe I'd better  read the code fully before starting new work.

It would probably help to look at the files before and after
a run of a simple test (with multiple test cases) or benchmark.

The key files are:
 testlog.txt - contains the un-parsed standard output from the test, generated by the test program on the DUT
   - one per run
 run.json - contains the data for a single test run, generated by Fuego parser from testlog.txt
   - one per run
 chart_config.json - contains configuration for the charts for a test - created by test author and is in test home directory
   - one per test
 flat_plot_data.txt - contains a set of lines for all runs for a particular test, generated by prepare_chart_data.py
   - one per test, one line per testcase
 flot_chart_data.json - contains data for display by mod.js - either HTML tables or data needed by flot Javascript graphing system
  - one per test, one structure per chart
 
> 
> >
> > >
> > >
> > > 3)Do u have any schedule of re-investigating the fuego chart（or
> > > graph,table...） related mechanism?
> > I don't have a schedule.  I might be able to take a look at it this week, but I
> am
> > WAY behind in my patch review for new tests and test fixes.  Don't get me
> > wrong - I am grateful for all the work by Fujitsu engineers, but I need to
> catch
> > up on about 20 to 30 patches.  It will take me probably a few days at least to
> > get to this.
> 
> Please forgive my rudeness,  which is probably due to my poor English(or
> differences between culture ).
> It is not my intention.

I did not interpret anything you said as rude.  I was just trying
to explain why I might not be able to work on this for a while.
I am grateful for Fujitsu's great contributions to Fuego.
 -- Tim

[Fuego] 答复: Problem of adding a benchmark result table

[Li, Xiaoming]

Many thanks for the information.

I could not wait to take a dive into them


-----邮件原件-----
发件人: Tim.Bird@sony.com [mailto:Tim.Bird@sony.com] 
发送时间: 2018年8月29日 1:45
收件人: Li, Xiaoming/李 霄鸣; fuego@lists.linuxfoundation.org
抄送: daniel.sangorrin@toshiba.co.jp
主题: RE: [Fuego] Problem of adding a benchmark result table

A few more comments inline below...

> -----Original Message-----
> From: Li, Xiaoming
...
> My  reply is below.
> 
> > -----Original Message-----
> > From: Tim.Bird@sony.com
> > > -----Original Message-----
> > > From: Li, Xiaoming
> > >
> > > Hi Tim,
> > >
> > >
> > >
> > > As I am urge of implement this function, could u give me some 
> > > hints or advice on questions below?
> >
> > You should look at:
> > http://fuegotest.org/wiki/Jenkins_Visualization
> > It has some information about the charting features of Fuego, and 
> > the integration with Jenkins.
> 
> Great, I found much valuable info there.
> 
> >
> > >
> > >
> > >
> > > 1)If modiying the mod.js to generate a single benchmark report, 
> > > does it need the batch_id arg? or parse the report file with own algorithm?
> >
> > Currently, Fuego does not have a batch_id implemented.  It does not
> appear
> > in run.json, or in flat_plot_data.txt, or flot_chart-data.json.  
> > Although it is mentioned in the ftc command-line help for --where, 
> > it isn't actually implemented yet.
> > (Note that if run.json *had* a batch_id field, the code in ftc that 
> > handled -- where clauses should function properly for filtering 
> > this.)
> >
> > prepare_chart_data.py parses the run.json data with its own algorithm.
> >
> > >
> > > 2)My plan is to add a chart on top of test suite's flot graph, 
> > > with a menu of selecting build no. (sorry, I am not an expert of 
> > > web design)
> > This sounds useful, but might be complicated.
> > Does this 'chart on top of ...the flog graph' contain data, or is it 
> > just a list of checkboxes for selecting the data to include in the graph?
> >
> > Just to clarify the terminology, I use the following (hopefully consistently):
> > chart = visualization item, which can be a table or a plot (graph)
> >
> > There are three types of charts:
> > 1) measure_plot = a 'flot' plot of benchmark measures
> > 2) testcase_table = an HTML table of testcase results
> > 3) testcase_summary_table = an HTML table of summary counts per
> testset
> >
> > I wanted for the system (prepare_chart_data.py and mod.js) to 
> > support being able to show multiple charts for the same test, and 
> > being able to support pre-filtering (by prepare_chart_data.py) and 
> > visualization-time filtering (by mod.js using flot capabilities to 
> > turn items on and off in the Jenkins user interface).
> >
> > I'm not sure, however, of the status of this code.  It looks like 
> > mod.js can read multiple charts from the flot_chart_data.json file, 
> > but that prepare_chart_data.py only writes one chart (or one kind of 
> > chart) to that file.
> 
> My head is messed up by their complex relation.
> Maybe I'd better  read the code fully before starting new work.

It would probably help to look at the files before and after a run of a simple test (with multiple test cases) or benchmark.

The key files are:
 testlog.txt - contains the un-parsed standard output from the test, generated by the test program on the DUT
   - one per run
 run.json - contains the data for a single test run, generated by Fuego parser from testlog.txt
   - one per run
 chart_config.json - contains configuration for the charts for a test - created by test author and is in test home directory
   - one per test
 flat_plot_data.txt - contains a set of lines for all runs for a particular test, generated by prepare_chart_data.py
   - one per test, one line per testcase  flot_chart_data.json - contains data for display by mod.js - either HTML tables or data needed by flot Javascript graphing system
  - one per test, one structure per chart
 
> 
> >
> > >
> > >
> > > 3)Do u have any schedule of re-investigating the fuego chart（or
> > > graph,table...） related mechanism?
> > I don't have a schedule.  I might be able to take a look at it this 
> > week, but I
> am
> > WAY behind in my patch review for new tests and test fixes.  Don't 
> > get me wrong - I am grateful for all the work by Fujitsu engineers, 
> > but I need to
> catch
> > up on about 20 to 30 patches.  It will take me probably a few days 
> > at least to get to this.
> 
> Please forgive my rudeness,  which is probably due to my poor 
> English(or differences between culture ).
> It is not my intention.

I did not interpret anything you said as rude.  I was just trying to explain why I might not be able to work on this for a while.
I am grateful for Fujitsu's great contributions to Fuego.
 -- Tim