* [Fuego] Problem of adding a benchmark result table @ 2018-08-24 2:34 Li, Xiaoming 2018-08-24 3:11 ` Daniel Sangorrin 0 siblings, 1 reply; 9+ messages in thread From: Li, Xiaoming @ 2018-08-24 2:34 UTC (permalink / raw) To: fuego [-- Attachment #1: Type: text/plain, Size: 832 bytes --] Hi all， I want to add a table of building results for every single benchmark in Jenkins. My solutions is to use jenkins's plugin groovy postbuild, which can add a hyperlink in build history. But groovy postbuild requies jenkins‘s version of 2.6.0 or later, as fuego's default is 2.32.1. I also have to install java8 to run higher version jenkins. Could anyone give me a better idea? BR Li -------------------------------------------------- Li, Xiaoming Development Dept.III Nanjing Fujitsu Nanda Software Tech. Co., Ltd.(FNST) No. 6 Wenzhu Road, Nanjing, 210012, China TEL: +86+25-86630566-8451 FAX: +86+25-83317685 MAIL: lixm.fnst@cn.fujitsu.com<mailto:lixm.fnst@cn.fujitsu.com> -------------------------------------------------- [-- Attachment #2: Type: text/html, Size: 5161 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Fuego] Problem of adding a benchmark result table 2018-08-24 2:34 [Fuego] Problem of adding a benchmark result table Li, Xiaoming @ 2018-08-24 3:11 ` Daniel Sangorrin 2018-08-24 6:56 ` Li, Xiaoming 0 siblings, 1 reply; 9+ messages in thread From: Daniel Sangorrin @ 2018-08-24 3:11 UTC (permalink / raw) To: 'Li, Xiaoming', fuego Hi Li, > <fuego-bounces@lists.linuxfoundation.org> On Behalf Of Li, Xiaoming > I want to add a table of building results for every single benchmark in Jenkins. If you want to use a table to show the building results instead of a plot, you can change the chart_type from chart_config.json to testset_summary_table. If you run the benchmark again, you should get a table instead of a plot. That functionality is implemented at: scripts/parser/prepare_chart_data.py scripts/mod.js I believe that code needs some improvements: - ability to override chart_config.json with a per-board configuration - I have seen in Dhrystone that the table shows pass=1 but also err=1 - The tables only show building results (pass/fail), you may want to add the benchmark measures > My solutions is to use jenkins's plugin groovy postbuild, which can add a hyperlink > in build history. > But groovy postbuild requies jenkins‘s version of 2.6.0 or later, as fuego's default is > 2.32.1. Updating jenkins regularly is important. However, we have had bad experiences with Jenkins' plugins. Each time we upgraded Jenkins a number of plugins where not available anymore causing many breakages. For that reason, we decided to limit the usage of plugins to the minimum necessary. > I also have to install java8 to run higher version jenkins. This would require changing the Dockerfile. It might be a hard job if it can't be done with Debian Jessie. > Could anyone give me a better idea? Apart from using chart_config.son, there is another option. We have an SQL-like interface to fetch results from Fuego. You can use that interface with ftc gen-report and create an html/xls/pdf report with all of the build results that you want. Maybe you could even call to ftc gen-report in a postbuild action and then add a link to the generated html from the "Set build description" post action. Also, soon we will be able to post test results to kernelci and squad instances. Best regards, Daniel > -------------------------------------------------- > > Li, Xiaoming > > Development Dept.III > > Nanjing Fujitsu Nanda Software Tech. Co., Ltd.(FNST) No. 6 Wenzhu Road, Nanjing, > 210012, China > > TEL: +86+25-86630566-8451 > > FAX: +86+25-83317685 > > MAIL: lixm.fnst@cn.fujitsu.com <mailto:lixm.fnst@cn.fujitsu.com> > > -------------------------------------------------- > > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Fuego] Problem of adding a benchmark result table 2018-08-24 3:11 ` Daniel Sangorrin @ 2018-08-24 6:56 ` Li, Xiaoming 2018-08-24 15:58 ` Tim.Bird 0 siblings, 1 reply; 9+ messages in thread From: Li, Xiaoming @ 2018-08-24 6:56 UTC (permalink / raw) To: Daniel Sangorrin, fuego Hi Daniel, > -----Original Message----- > From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp] > Sent: Friday, August 24, 2018 11:11 AM > To: Li, Xiaoming/李 霄鸣; fuego@lists.linuxfoundation.org > Subject: RE: [Fuego] Problem of adding a benchmark result table > > Hi Li, > > > <fuego-bounces@lists.linuxfoundation.org> On Behalf Of Li, Xiaoming > > I want to add a table of building results for every single benchmark in > Jenkins. > > If you want to use a table to show the building results instead of a plot, you > can change the chart_type from chart_config.json to testset_summary_table. > If you run the benchmark again, you should get a table instead of a plot. > > That functionality is implemented at: > scripts/parser/prepare_chart_data.py > scripts/mod.js > > I believe that code needs some improvements: > - ability to override chart_config.json with a per-board configuration > - I have seen in Dhrystone that the table shows pass=1 but also err=1 > - The tables only show building results (pass/fail), you may want to add the > benchmark measures > > > My solutions is to use jenkins's plugin groovy postbuild, which can > > add a hyperlink in build history. > > But groovy postbuild requies jenkins‘s version of 2.6.0 or later, as > > fuego's default is 2.32.1. > > Updating jenkins regularly is important. However, we have had bad > experiences with Jenkins' plugins. Each time we upgraded Jenkins a number > of plugins where not available anymore causing many breakages. For that > reason, we decided to limit the usage of plugins to the minimum necessary. > > > I also have to install java8 to run higher version jenkins. > This would require changing the Dockerfile. It might be a hard job if it can't be > done with Debian Jessie. > > > Could anyone give me a better idea? > > Apart from using chart_config.son, there is another option. We have an SQL- > like interface to fetch results from Fuego. You can use that interface with ftc > gen-report and create an html/xls/pdf report with all of the build results that > you want. Maybe you could even call to ftc gen-report in a postbuild action > and then add a link to the generated html from the "Set build description" > post action. I have tried it , and it works. But the --where batch_id arg can not work temporarily, I will investigate it sometime later. On the whole, it is a wonderful method. > > Also, soon we will be able to post test results to kernelci and squad instances. > > Best regards, > Daniel > > > -------------------------------------------------- > > > > Li, Xiaoming > > > > Development Dept.III > > > > Nanjing Fujitsu Nanda Software Tech. Co., Ltd.(FNST) No. 6 Wenzhu > > Road, Nanjing, 210012, China > > > > TEL: +86+25-86630566-8451 > > > > FAX: +86+25-83317685 > > > > MAIL: lixm.fnst@cn.fujitsu.com <mailto:lixm.fnst@cn.fujitsu.com> > > > > -------------------------------------------------- > > > > > > > Thx Li ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Fuego] Problem of adding a benchmark result table 2018-08-24 6:56 ` Li, Xiaoming @ 2018-08-24 15:58 ` Tim.Bird 2018-08-27 6:14 ` Li, Xiaoming 0 siblings, 1 reply; 9+ messages in thread From: Tim.Bird @ 2018-08-24 15:58 UTC (permalink / raw) To: lixm.fnst, daniel.sangorrin, fuego [-- Attachment #1: Type: text/plain, Size: 4575 bytes --] On the whole, I'd like to extend our current charting mechanisms to support what you need. scripts/mod.js can support arbitrary HTML generated by prepare_chart_data.py, so we should be able to make it work. With regard to batch_id, that feature is not finished yet. Filtering on it is trivial to add to ftc and prepare_chart_data.py. But I don't think the field is recorded in run.json and the other files used for charting. The key issue outstanding was the mechanism for assigning the batch_id for runs. The idea was to have a test plan job set the batch_id for the jobs it started. But there's not a good Jenkins mechanism to do this. When we finished having Jenkins call ftc run-test to run test jobs, my plan was to add a new ftc run-plan, and have Jenkins call that for executing batch jobs. Thus would allow me to set a batch ID for the jobs associated with the batch run, I think. Anyway, there just needs to be some batch_id on the job command line, or in an environment variable, that gets put into the run.json file. I'll take a look at this next week. Regards, - - Tim -------- Original Message -------- Subject: Re: [Fuego] Problem of adding a benchmark result table From: "Li, Xiaoming" <lixm.fnst@cn.fujitsu.com> Date: Aug 24, 2018, 12:57 AM To: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>,fuego@lists.linuxfoundation.org Hi Daniel, > -----Original Message----- > From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp] > Sent: Friday, August 24, 2018 11:11 AM > To: Li, Xiaoming/李 霄鸣; fuego@lists.linuxfoundation.org > Subject: RE: [Fuego] Problem of adding a benchmark result table > > Hi Li, > > > <fuego-bounces@lists.linuxfoundation.org> On Behalf Of Li, Xiaoming > > I want to add a table of building results for every single benchmark in > Jenkins. > > If you want to use a table to show the building results instead of a plot, you > can change the chart_type from chart_config.json to testset_summary_table. > If you run the benchmark again, you should get a table instead of a plot. > > That functionality is implemented at: > scripts/parser/prepare_chart_data.py > scripts/mod.js > > I believe that code needs some improvements: > - ability to override chart_config.json with a per-board configuration > - I have seen in Dhrystone that the table shows pass=1 but also err=1 > - The tables only show building results (pass/fail), you may want to add the > benchmark measures > > > My solutions is to use jenkins's plugin groovy postbuild, which can > > add a hyperlink in build history. > > But groovy postbuild requies jenkins‘s version of 2.6.0 or later, as > > fuego's default is 2.32.1. > > Updating jenkins regularly is important. However, we have had bad > experiences with Jenkins' plugins. Each time we upgraded Jenkins a number > of plugins where not available anymore causing many breakages. For that > reason, we decided to limit the usage of plugins to the minimum necessary. > > > I also have to install java8 to run higher version jenkins. > This would require changing the Dockerfile. It might be a hard job if it can't be > done with Debian Jessie. > > > Could anyone give me a better idea? > > Apart from using chart_config.son, there is another option. We have an SQL- > like interface to fetch results from Fuego. You can use that interface with ftc > gen-report and create an html/xls/pdf report with all of the build results that > you want. Maybe you could even call to ftc gen-report in a postbuild action > and then add a link to the generated html from the "Set build description" > post action. I have tried it , and it works. But the --where batch_id arg can not work temporarily, I will investigate it sometime later. On the whole, it is a wonderful method. > > Also, soon we will be able to post test results to kernelci and squad instances. > > Best regards, > Daniel > > > -------------------------------------------------- > > > > Li, Xiaoming > > > > Development Dept.III > > > > Nanjing Fujitsu Nanda Software Tech. Co., Ltd.(FNST) No. 6 Wenzhu > > Road, Nanjing, 210012, China > > > > TEL: +86+25-86630566-8451 > > > > FAX: +86+25-83317685 > > > > MAIL: lixm.fnst@cn.fujitsu.com <mailto:lixm.fnst@cn.fujitsu.com> > > > > -------------------------------------------------- > > > > > > > Thx Li _______________________________________________ Fuego mailing list Fuego@lists.linuxfoundation.org https://lists.linuxfoundation.org/mailman/listinfo/fuego [-- Attachment #2: Type: text/html, Size: 6315 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Fuego] Problem of adding a benchmark result table 2018-08-24 15:58 ` Tim.Bird @ 2018-08-27 6:14 ` Li, Xiaoming 2018-08-27 23:48 ` Tim.Bird 0 siblings, 1 reply; 9+ messages in thread From: Li, Xiaoming @ 2018-08-27 6:14 UTC (permalink / raw) To: fuego [-- Attachment #1: Type: text/plain, Size: 5572 bytes --] Hi Tim, As I am urge of implement this function, could u give me some hints or advice on questions below? 1)If modiying the mod.js to generate a single benchmark report, does it need the batch_id arg? or parse the report file with own algorithm? 2)My plan is to add a chart on top of test suite's flot graph, with a menu of selecting build no. (sorry, I am not an expert of web design) 3)Do u have any schedule of re-investigating the fuego chart（or graph,table...） related mechanism? THX Li From: Tim.Bird@sony.com [mailto:Tim.Bird@sony.com] Sent: Friday, August 24, 2018 11:59 PM To: Li, Xiaoming/李 霄鸣; daniel.sangorrin@toshiba.co.jp; fuego@lists.linuxfoundation.org Subject: Re: [Fuego] Problem of adding a benchmark result table On the whole, I'd like to extend our current charting mechanisms to support what you need. scripts/mod.js can support arbitrary HTML generated by prepare_chart_data.py, so we should be able to make it work. With regard to batch_id, that feature is not finished yet. Filtering on it is trivial to add to ftc and prepare_chart_data.py. But I don't think the field is recorded in run.json and the other files used for charting. The key issue outstanding was the mechanism for assigning the batch_id for runs. The idea was to have a test plan job set the batch_id for the jobs it started. But there's not a good Jenkins mechanism to do this. When we finished having Jenkins call ftc run-test to run test jobs, my plan was to add a new ftc run-plan, and have Jenkins call that for executing batch jobs. Thus would allow me to set a batch ID for the jobs associated with the batch run, I think. Anyway, there just needs to be some batch_id on the job command line, or in an environment variable, that gets put into the run.json file. I'll take a look at this next week. Regards, - - Tim -------- Original Message -------- Subject: Re: [Fuego] Problem of adding a benchmark result table From: "Li, Xiaoming" <lixm.fnst@cn.fujitsu.com<mailto:lixm.fnst@cn.fujitsu.com>> Date: Aug 24, 2018, 12:57 AM To: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp<mailto:daniel.sangorrin@toshiba.co.jp>>,fuego@lists.linuxfoundation.org Hi Daniel, > -----Original Message----- > From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp] > Sent: Friday, August 24, 2018 11:11 AM > To: Li, Xiaoming/李 霄鸣; fuego@lists.linuxfoundation.org<mailto:fuego@lists.linuxfoundation.org> > Subject: RE: [Fuego] Problem of adding a benchmark result table > > Hi Li, > > > <fuego-bounces@lists.linuxfoundation.org<mailto:fuego-bounces@lists.linuxfoundation.org>> On Behalf Of Li, Xiaoming > > I want to add a table of building results for every single benchmark in > Jenkins. > > If you want to use a table to show the building results instead of a plot, you > can change the chart_type from chart_config.json to testset_summary_table. > If you run the benchmark again, you should get a table instead of a plot. > > That functionality is implemented at: > scripts/parser/prepare_chart_data.py > scripts/mod.js > > I believe that code needs some improvements: > - ability to override chart_config.json with a per-board configuration > - I have seen in Dhrystone that the table shows pass=1 but also err=1 > - The tables only show building results (pass/fail), you may want to add the > benchmark measures > > > My solutions is to use jenkins's plugin groovy postbuild, which can > > add a hyperlink in build history. > > But groovy postbuild requies jenkins‘s version of 2.6.0 or later, as > > fuego's default is 2.32.1. > > Updating jenkins regularly is important. However, we have had bad > experiences with Jenkins' plugins. Each time we upgraded Jenkins a number > of plugins where not available anymore causing many breakages. For that > reason, we decided to limit the usage of plugins to the minimum necessary. > > > I also have to install java8 to run higher version jenkins. > This would require changing the Dockerfile. It might be a hard job if it can't be > done with Debian Jessie. > > > Could anyone give me a better idea? > > Apart from using chart_config.son, there is another option. We have an SQL- > like interface to fetch results from Fuego. You can use that interface with ftc > gen-report and create an html/xls/pdf report with all of the build results that > you want. Maybe you could even call to ftc gen-report in a postbuild action > and then add a link to the generated html from the "Set build description" > post action. I have tried it , and it works. But the --where batch_id arg can not work temporarily, I will investigate it sometime later. On the whole, it is a wonderful method. > > Also, soon we will be able to post test results to kernelci and squad instances. > > Best regards, > Daniel > > > -------------------------------------------------- > > > > Li, Xiaoming > > > > Development Dept.III > > > > Nanjing Fujitsu Nanda Software Tech. Co., Ltd.(FNST) No. 6 Wenzhu > > Road, Nanjing, 210012, China > > > > TEL: +86+25-86630566-8451 > > > > FAX: +86+25-83317685 > > > > MAIL: lixm.fnst@cn.fujitsu.com<mailto:lixm.fnst@cn.fujitsu.com> <mailto:lixm.fnst@cn.fujitsu.com> > > > > -------------------------------------------------- > > > > > > > Thx Li _______________________________________________ Fuego mailing list Fuego@lists.linuxfoundation.org<mailto:Fuego@lists.linuxfoundation.org> https://lists.linuxfoundation.org/mailman/listinfo/fuego [-- Attachment #2: Type: text/html, Size: 12963 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Fuego] Problem of adding a benchmark result table 2018-08-27 6:14 ` Li, Xiaoming @ 2018-08-27 23:48 ` Tim.Bird 2018-08-28 6:38 ` Li, Xiaoming 0 siblings, 1 reply; 9+ messages in thread From: Tim.Bird @ 2018-08-27 23:48 UTC (permalink / raw) To: lixm.fnst, fuego > -----Original Message----- > From: Li, Xiaoming > > Hi Tim, > > > > As I am urge of implement this function, could u give me some hints or advice > on questions below? You should look at: http://fuegotest.org/wiki/Jenkins_Visualization It has some information about the charting features of Fuego, and the integration with Jenkins. > > > > 1)If modiying the mod.js to generate a single benchmark report, does it need > the batch_id arg? or parse the report file with own algorithm? Currently, Fuego does not have a batch_id implemented. It does not appear in run.json, or in flat_plot_data.txt, or flot_chart-data.json. Although it is mentioned in the ftc command-line help for --where, it isn't actually implemented yet. (Note that if run.json *had* a batch_id field, the code in ftc that handled --where clauses should function properly for filtering this.) prepare_chart_data.py parses the run.json data with its own algorithm. > > 2)My plan is to add a chart on top of test suite's flot graph, with a menu of > selecting build no. (sorry, I am not an expert of web design) This sounds useful, but might be complicated. Does this 'chart on top of ...the flog graph' contain data, or is it just a list of checkboxes for selecting the data to include in the graph? Just to clarify the terminology, I use the following (hopefully consistently): chart = visualization item, which can be a table or a plot (graph) There are three types of charts: 1) measure_plot = a 'flot' plot of benchmark measures 2) testcase_table = an HTML table of testcase results 3) testcase_summary_table = an HTML table of summary counts per testset I wanted for the system (prepare_chart_data.py and mod.js) to support being able to show multiple charts for the same test, and being able to support pre-filtering (by prepare_chart_data.py) and visualization-time filtering (by mod.js using flot capabilities to turn items on and off in the Jenkins user interface). I'm not sure, however, of the status of this code. It looks like mod.js can read multiple charts from the flot_chart_data.json file, but that prepare_chart_data.py only writes one chart (or one kind of chart) to that file. > > > 3)Do u have any schedule of re-investigating the fuego chart（or > graph,table...） related mechanism? I don't have a schedule. I might be able to take a look at it this week, but I am WAY behind in my patch review for new tests and test fixes. Don't get me wrong - I am grateful for all the work by Fujitsu engineers, but I need to catch up on about 20 to 30 patches. It will take me probably a few days at least to get to this. -- Tim > From: Tim.Bird@sony.com [mailto:Tim.Bird@sony.com] > > > On the whole, I'd like to extend our current charting mechanisms to support > what you need. scripts/mod.js can support arbitrary HTML generated by > prepare_chart_data.py, so we should be able to make it work. > > > > With regard to batch_id, that feature is not finished yet. Filtering on it is trivial > to add to ftc and prepare_chart_data.py. > > > > But I don't think the field is recorded in run.json and the other files used for > charting. The key issue outstanding was the mechanism for assigning the > batch_id for runs. The idea was to have a test plan job set the batch_id for > the jobs it started. But there's not a good Jenkins mechanism to do this. > When we finished having Jenkins call ftc run-test to run test jobs, my plan > was to add a new ftc run-plan, and have Jenkins call that for executing batch > jobs. Thus would allow me to set a batch ID for the jobs associated with the > batch run, I think. > > > > Anyway, there just needs to be some batch_id on the job command line, or > in an environment variable, that gets put into the run.json file. > > > > I'll take a look at this next week. > > > > Regards, > > - - Tim > > > > -------- Original Message -------- > > Subject: Re: [Fuego] Problem of adding a benchmark result table > > From: "Li, Xiaoming" <lixm.fnst@cn.fujitsu.com > <mailto:lixm.fnst@cn.fujitsu.com> > > > Date: Aug 24, 2018, 12:57 AM > > To: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp > <mailto:daniel.sangorrin@toshiba.co.jp> >,fuego@lists.linuxfoundation.org > > Hi Daniel, > > > -----Original Message----- > > From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp] > > Sent: Friday, August 24, 2018 11:11 AM > > To: Li, Xiaoming/李 霄鸣; fuego@lists.linuxfoundation.org > <mailto:fuego@lists.linuxfoundation.org> > > Subject: RE: [Fuego] Problem of adding a benchmark result table > > > > Hi Li, > > > > > <fuego-bounces@lists.linuxfoundation.org <mailto:fuego- > bounces@lists.linuxfoundation.org> > On Behalf Of Li, Xiaoming > > > I want to add a table of building results for every single benchmark in > > Jenkins. > > > > If you want to use a table to show the building results instead of a plot, you > > can change the chart_type from chart_config.json to > testset_summary_table. > > If you run the benchmark again, you should get a table instead of a plot. > > > > > That functionality is implemented at: > > scripts/parser/prepare_chart_data.py > > scripts/mod.js > > > > I believe that code needs some improvements: > > - ability to override chart_config.json with a per-board configuration > > - I have seen in Dhrystone that the table shows pass=1 but also err=1 > > - The tables only show building results (pass/fail), you may want to add > the > > benchmark measures > > > > > My solutions is to use jenkins's plugin groovy postbuild, which can > > > add a hyperlink in build history. > > > But groovy postbuild requies jenkins‘s version of 2.6.0 or later, as > > > fuego's default is 2.32.1. > > > > Updating jenkins regularly is important. However, we have had bad > > experiences with Jenkins' plugins. Each time we upgraded Jenkins a > number > > of plugins where not available anymore causing many breakages. For that > > reason, we decided to limit the usage of plugins to the minimum necessary. > > > > > I also have to install java8 to run higher version jenkins. > > This would require changing the Dockerfile. It might be a hard job if it can't > be > > done with Debian Jessie. > > > > > Could anyone give me a better idea? > > > > Apart from using chart_config.son, there is another option. We have an > SQL- > > like interface to fetch results from Fuego. You can use that interface with > ftc > > gen-report and create an html/xls/pdf report with all of the build results > that > > you want. Maybe you could even call to ftc gen-report in a postbuild action > > and then add a link to the generated html from the "Set build description" > > post action. > > I have tried it , and it works. > But the --where batch_id arg can not work temporarily, I will investigate it > sometime later. > On the whole, it is a wonderful method. > > > > > > > Also, soon we will be able to post test results to kernelci and squad > instances. > > > > Best regards, > > Daniel > > > > > -------------------------------------------------- > > > > > > Li, Xiaoming > > > > > > Development Dept.III > > > > > > Nanjing Fujitsu Nanda Software Tech. Co., Ltd.(FNST) No. 6 Wenzhu > > > Road, Nanjing, 210012, China > > > > > > TEL: +86+25-86630566-8451 > > > > > > FAX: +86+25-83317685 > > > > > > MAIL: lixm.fnst@cn.fujitsu.com <mailto:lixm.fnst@cn.fujitsu.com> > <mailto:lixm.fnst@cn.fujitsu.com> > > > > > > -------------------------------------------------- > > > > > > > > > > > > > > > Thx > Li > > > _______________________________________________ > Fuego mailing list > Fuego@lists.linuxfoundation.org <mailto:Fuego@lists.linuxfoundation.org> > https://lists.linuxfoundation.org/mailman/listinfo/fuego ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Fuego] Problem of adding a benchmark result table 2018-08-27 23:48 ` Tim.Bird @ 2018-08-28 6:38 ` Li, Xiaoming 2018-08-28 17:45 ` Tim.Bird 0 siblings, 1 reply; 9+ messages in thread From: Li, Xiaoming @ 2018-08-28 6:38 UTC (permalink / raw) To: Tim.Bird, fuego Hi Tim, THX for your prompt instruction. My reply is below. > -----Original Message----- > From: Tim.Bird@sony.com [mailto:Tim.Bird@sony.com] > Sent: Tuesday, August 28, 2018 7:49 AM > To: Li, Xiaoming/李 霄鸣; fuego@lists.linuxfoundation.org > Cc: daniel.sangorrin@toshiba.co.jp > Subject: RE: [Fuego] Problem of adding a benchmark result table > > > > > -----Original Message----- > > From: Li, Xiaoming > > > > Hi Tim, > > > > > > > > As I am urge of implement this function, could u give me some hints or > > advice on questions below? > > You should look at: > http://fuegotest.org/wiki/Jenkins_Visualization > It has some information about the charting features of Fuego, and the > integration with Jenkins. Great, I found much valuable info there. > > > > > > > > > 1)If modiying the mod.js to generate a single benchmark report, does > > it need the batch_id arg? or parse the report file with own algorithm? > > Currently, Fuego does not have a batch_id implemented. It does not appear > in run.json, or in flat_plot_data.txt, or flot_chart-data.json. Although it is > mentioned in the ftc command-line help for --where, it isn't actually > implemented yet. > (Note that if run.json *had* a batch_id field, the code in ftc that handled -- > where clauses should function properly for filtering this.) > > prepare_chart_data.py parses the run.json data with its own algorithm. > > > > > 2)My plan is to add a chart on top of test suite's flot graph, with a > > menu of selecting build no. (sorry, I am not an expert of web design) > This sounds useful, but might be complicated. > Does this 'chart on top of ...the flog graph' contain data, or is it just a list of > checkboxes for selecting the data to include in the graph? > > Just to clarify the terminology, I use the following (hopefully consistently): > chart = visualization item, which can be a table or a plot (graph) > > There are three types of charts: > 1) measure_plot = a 'flot' plot of benchmark measures > 2) testcase_table = an HTML table of testcase results > 3) testcase_summary_table = an HTML table of summary counts per testset > > I wanted for the system (prepare_chart_data.py and mod.js) to support > being able to show multiple charts for the same test, and being able to > support pre-filtering (by prepare_chart_data.py) and visualization-time > filtering (by mod.js using flot capabilities to turn items on and off in the > Jenkins user interface). > > I'm not sure, however, of the status of this code. It looks like mod.js can > read multiple charts from the flot_chart_data.json file, but that > prepare_chart_data.py only writes one chart (or one kind of chart) to that > file. My head is messed up by their complex relation. Maybe I'd better read the code fully before starting new work. > > > > > > > 3)Do u have any schedule of re-investigating the fuego chart（or > > graph,table...） related mechanism? > I don't have a schedule. I might be able to take a look at it this week, but I am > WAY behind in my patch review for new tests and test fixes. Don't get me > wrong - I am grateful for all the work by Fujitsu engineers, but I need to catch > up on about 20 to 30 patches. It will take me probably a few days at least to > get to this. Please forgive my rudeness, which is probably due to my poor English(or differences between culture ). It is not my intention. > > -- Tim > > > > From: Tim.Bird@sony.com [mailto:Tim.Bird@sony.com] > > > > > > On the whole, I'd like to extend our current charting mechanisms to > > support what you need. scripts/mod.js can support arbitrary HTML > > generated by prepare_chart_data.py, so we should be able to make it > work. > > > > > > > > With regard to batch_id, that feature is not finished yet. Filtering > > on it is trivial to add to ftc and prepare_chart_data.py. > > > > > > > > But I don't think the field is recorded in run.json and the other > > files used for charting. The key issue outstanding was the mechanism > > for assigning the batch_id for runs. The idea was to have a test plan > > job set the batch_id for the jobs it started. But there's not a good Jenkins > mechanism to do this. > > When we finished having Jenkins call ftc run-test to run test jobs, my > > plan was to add a new ftc run-plan, and have Jenkins call that for > > executing batch jobs. Thus would allow me to set a batch ID for the > > jobs associated with the batch run, I think. > > > > > > > > Anyway, there just needs to be some batch_id on the job command line, > > or in an environment variable, that gets put into the run.json file. > > > > > > > > I'll take a look at this next week. > > > > > > > > Regards, > > > > - - Tim > > > > > > > > -------- Original Message -------- > > > > Subject: Re: [Fuego] Problem of adding a benchmark result table > > > > From: "Li, Xiaoming" <lixm.fnst@cn.fujitsu.com > > <mailto:lixm.fnst@cn.fujitsu.com> > > > > > Date: Aug 24, 2018, 12:57 AM > > > > To: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp > > <mailto:daniel.sangorrin@toshiba.co.jp> > > >,fuego@lists.linuxfoundation.org > > > > Hi Daniel, > > > > > -----Original Message----- > > > From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp] > > > Sent: Friday, August 24, 2018 11:11 AM > > > To: Li, Xiaoming/李 霄鸣; fuego@lists.linuxfoundation.org > > <mailto:fuego@lists.linuxfoundation.org> > > > Subject: RE: [Fuego] Problem of adding a benchmark result table > > > > > > Hi Li, > > > > > > > <fuego-bounces@lists.linuxfoundation.org <mailto:fuego- > > bounces@lists.linuxfoundation.org> > On Behalf Of Li, Xiaoming > > > > I want to add a table of building results for every single > > > benchmark in Jenkins. > > > > > > If you want to use a table to show the building results instead of a > > > plot, you can change the chart_type from chart_config.json to > > testset_summary_table. > > > If you run the benchmark again, you should get a table instead of a plot. > > > > > > > > That functionality is implemented at: > > > scripts/parser/prepare_chart_data.py > > > scripts/mod.js > > > > > > I believe that code needs some improvements: > > > - ability to override chart_config.json with a per-board configuration > > > - I have seen in Dhrystone that the table shows pass=1 but also err=1 > > > - The tables only show building results (pass/fail), you may want > > > to add > > the > > > benchmark measures > > > > > > > My solutions is to use jenkins's plugin groovy postbuild, which > > > > can add a hyperlink in build history. > > > > But groovy postbuild requies jenkins‘s version of 2.6.0 or later, > > > > as fuego's default is 2.32.1. > > > > > > Updating jenkins regularly is important. However, we have had bad > > > experiences with Jenkins' plugins. Each time we upgraded Jenkins a > > number > > > of plugins where not available anymore causing many breakages. For > > > that reason, we decided to limit the usage of plugins to the minimum > necessary. > > > > > > > I also have to install java8 to run higher version jenkins. > > > This would require changing the Dockerfile. It might be a hard job > > > if it can't > > be > > > done with Debian Jessie. > > > > > > > Could anyone give me a better idea? > > > > > > Apart from using chart_config.son, there is another option. We have > > > an > > SQL- > > > like interface to fetch results from Fuego. You can use that > > > interface with > > ftc > > > gen-report and create an html/xls/pdf report with all of the build > > > results > > that > > > you want. Maybe you could even call to ftc gen-report in a postbuild > > > action and then add a link to the generated html from the "Set build > description" > > > post action. > > > > I have tried it , and it works. > > But the --where batch_id arg can not work temporarily, I will investigate > it > > sometime later. > > On the whole, it is a wonderful method. > > > > > > > > > > > > Also, soon we will be able to post test results to kernelci and > > > squad > > instances. > > > > > > Best regards, > > > Daniel > > > > > > > -------------------------------------------------- > > > > > > > > Li, Xiaoming > > > > > > > > Development Dept.III > > > > > > > > Nanjing Fujitsu Nanda Software Tech. Co., Ltd.(FNST) No. 6 Wenzhu > > > > Road, Nanjing, 210012, China > > > > > > > > TEL: +86+25-86630566-8451 > > > > > > > > FAX: +86+25-83317685 > > > > > > > > MAIL: lixm.fnst@cn.fujitsu.com <mailto:lixm.fnst@cn.fujitsu.com> > > <mailto:lixm.fnst@cn.fujitsu.com> > > > > > > > > -------------------------------------------------- > > > > > > > > > > > > > > > > > > > > > > > Thx > > Li > > > > > > _______________________________________________ > > Fuego mailing list > > Fuego@lists.linuxfoundation.org > > <mailto:Fuego@lists.linuxfoundation.org> > > https://lists.linuxfoundation.org/mailman/listinfo/fuego > > Best regard Li ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Fuego] Problem of adding a benchmark result table 2018-08-28 6:38 ` Li, Xiaoming @ 2018-08-28 17:45 ` Tim.Bird 2018-08-29 8:01 ` [Fuego] 答复: " Li, Xiaoming 0 siblings, 1 reply; 9+ messages in thread From: Tim.Bird @ 2018-08-28 17:45 UTC (permalink / raw) To: lixm.fnst, fuego A few more comments inline below... > -----Original Message----- > From: Li, Xiaoming ... > My reply is below. > > > -----Original Message----- > > From: Tim.Bird@sony.com > > > -----Original Message----- > > > From: Li, Xiaoming > > > > > > Hi Tim, > > > > > > > > > > > > As I am urge of implement this function, could u give me some hints or > > > advice on questions below? > > > > You should look at: > > http://fuegotest.org/wiki/Jenkins_Visualization > > It has some information about the charting features of Fuego, and the > > integration with Jenkins. > > Great, I found much valuable info there. > > > > > > > > > > > > > > > 1)If modiying the mod.js to generate a single benchmark report, does > > > it need the batch_id arg? or parse the report file with own algorithm? > > > > Currently, Fuego does not have a batch_id implemented. It does not > appear > > in run.json, or in flat_plot_data.txt, or flot_chart-data.json. Although it is > > mentioned in the ftc command-line help for --where, it isn't actually > > implemented yet. > > (Note that if run.json *had* a batch_id field, the code in ftc that handled -- > > where clauses should function properly for filtering this.) > > > > prepare_chart_data.py parses the run.json data with its own algorithm. > > > > > > > > 2)My plan is to add a chart on top of test suite's flot graph, with a > > > menu of selecting build no. (sorry, I am not an expert of web design) > > This sounds useful, but might be complicated. > > Does this 'chart on top of ...the flog graph' contain data, or is it just a list of > > checkboxes for selecting the data to include in the graph? > > > > Just to clarify the terminology, I use the following (hopefully consistently): > > chart = visualization item, which can be a table or a plot (graph) > > > > There are three types of charts: > > 1) measure_plot = a 'flot' plot of benchmark measures > > 2) testcase_table = an HTML table of testcase results > > 3) testcase_summary_table = an HTML table of summary counts per > testset > > > > I wanted for the system (prepare_chart_data.py and mod.js) to support > > being able to show multiple charts for the same test, and being able to > > support pre-filtering (by prepare_chart_data.py) and visualization-time > > filtering (by mod.js using flot capabilities to turn items on and off in the > > Jenkins user interface). > > > > I'm not sure, however, of the status of this code. It looks like mod.js can > > read multiple charts from the flot_chart_data.json file, but that > > prepare_chart_data.py only writes one chart (or one kind of chart) to that > > file. > > My head is messed up by their complex relation. > Maybe I'd better read the code fully before starting new work. It would probably help to look at the files before and after a run of a simple test (with multiple test cases) or benchmark. The key files are: testlog.txt - contains the un-parsed standard output from the test, generated by the test program on the DUT - one per run run.json - contains the data for a single test run, generated by Fuego parser from testlog.txt - one per run chart_config.json - contains configuration for the charts for a test - created by test author and is in test home directory - one per test flat_plot_data.txt - contains a set of lines for all runs for a particular test, generated by prepare_chart_data.py - one per test, one line per testcase flot_chart_data.json - contains data for display by mod.js - either HTML tables or data needed by flot Javascript graphing system - one per test, one structure per chart > > > > > > > > > > > > 3)Do u have any schedule of re-investigating the fuego chart（or > > > graph,table...） related mechanism? > > I don't have a schedule. I might be able to take a look at it this week, but I > am > > WAY behind in my patch review for new tests and test fixes. Don't get me > > wrong - I am grateful for all the work by Fujitsu engineers, but I need to > catch > > up on about 20 to 30 patches. It will take me probably a few days at least to > > get to this. > > Please forgive my rudeness, which is probably due to my poor English(or > differences between culture ). > It is not my intention. I did not interpret anything you said as rude. I was just trying to explain why I might not be able to work on this for a while. I am grateful for Fujitsu's great contributions to Fuego. -- Tim ^ permalink raw reply [flat|nested] 9+ messages in thread
* [Fuego] 答复: Problem of adding a benchmark result table 2018-08-28 17:45 ` Tim.Bird @ 2018-08-29 8:01 ` Li, Xiaoming 0 siblings, 0 replies; 9+ messages in thread From: Li, Xiaoming @ 2018-08-29 8:01 UTC (permalink / raw) To: Tim.Bird, fuego Many thanks for the information. I could not wait to take a dive into them -----邮件原件----- 发件人: Tim.Bird@sony.com [mailto:Tim.Bird@sony.com] 发送时间: 2018年8月29日 1:45 收件人: Li, Xiaoming/李 霄鸣; fuego@lists.linuxfoundation.org 抄送: daniel.sangorrin@toshiba.co.jp 主题: RE: [Fuego] Problem of adding a benchmark result table A few more comments inline below... > -----Original Message----- > From: Li, Xiaoming ... > My reply is below. > > > -----Original Message----- > > From: Tim.Bird@sony.com > > > -----Original Message----- > > > From: Li, Xiaoming > > > > > > Hi Tim, > > > > > > > > > > > > As I am urge of implement this function, could u give me some > > > hints or advice on questions below? > > > > You should look at: > > http://fuegotest.org/wiki/Jenkins_Visualization > > It has some information about the charting features of Fuego, and > > the integration with Jenkins. > > Great, I found much valuable info there. > > > > > > > > > > > > > > > 1)If modiying the mod.js to generate a single benchmark report, > > > does it need the batch_id arg? or parse the report file with own algorithm? > > > > Currently, Fuego does not have a batch_id implemented. It does not > appear > > in run.json, or in flat_plot_data.txt, or flot_chart-data.json. > > Although it is mentioned in the ftc command-line help for --where, > > it isn't actually implemented yet. > > (Note that if run.json *had* a batch_id field, the code in ftc that > > handled -- where clauses should function properly for filtering > > this.) > > > > prepare_chart_data.py parses the run.json data with its own algorithm. > > > > > > > > 2)My plan is to add a chart on top of test suite's flot graph, > > > with a menu of selecting build no. (sorry, I am not an expert of > > > web design) > > This sounds useful, but might be complicated. > > Does this 'chart on top of ...the flog graph' contain data, or is it > > just a list of checkboxes for selecting the data to include in the graph? > > > > Just to clarify the terminology, I use the following (hopefully consistently): > > chart = visualization item, which can be a table or a plot (graph) > > > > There are three types of charts: > > 1) measure_plot = a 'flot' plot of benchmark measures > > 2) testcase_table = an HTML table of testcase results > > 3) testcase_summary_table = an HTML table of summary counts per > testset > > > > I wanted for the system (prepare_chart_data.py and mod.js) to > > support being able to show multiple charts for the same test, and > > being able to support pre-filtering (by prepare_chart_data.py) and > > visualization-time filtering (by mod.js using flot capabilities to > > turn items on and off in the Jenkins user interface). > > > > I'm not sure, however, of the status of this code. It looks like > > mod.js can read multiple charts from the flot_chart_data.json file, > > but that prepare_chart_data.py only writes one chart (or one kind of > > chart) to that file. > > My head is messed up by their complex relation. > Maybe I'd better read the code fully before starting new work. It would probably help to look at the files before and after a run of a simple test (with multiple test cases) or benchmark. The key files are: testlog.txt - contains the un-parsed standard output from the test, generated by the test program on the DUT - one per run run.json - contains the data for a single test run, generated by Fuego parser from testlog.txt - one per run chart_config.json - contains configuration for the charts for a test - created by test author and is in test home directory - one per test flat_plot_data.txt - contains a set of lines for all runs for a particular test, generated by prepare_chart_data.py - one per test, one line per testcase flot_chart_data.json - contains data for display by mod.js - either HTML tables or data needed by flot Javascript graphing system - one per test, one structure per chart > > > > > > > > > > > > 3)Do u have any schedule of re-investigating the fuego chart（or > > > graph,table...） related mechanism? > > I don't have a schedule. I might be able to take a look at it this > > week, but I > am > > WAY behind in my patch review for new tests and test fixes. Don't > > get me wrong - I am grateful for all the work by Fujitsu engineers, > > but I need to > catch > > up on about 20 to 30 patches. It will take me probably a few days > > at least to get to this. > > Please forgive my rudeness, which is probably due to my poor > English(or differences between culture ). > It is not my intention. I did not interpret anything you said as rude. I was just trying to explain why I might not be able to work on this for a while. I am grateful for Fujitsu's great contributions to Fuego. -- Tim ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2018-08-29 8:01 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-08-24 2:34 [Fuego] Problem of adding a benchmark result table Li, Xiaoming 2018-08-24 3:11 ` Daniel Sangorrin 2018-08-24 6:56 ` Li, Xiaoming 2018-08-24 15:58 ` Tim.Bird 2018-08-27 6:14 ` Li, Xiaoming 2018-08-27 23:48 ` Tim.Bird 2018-08-28 6:38 ` Li, Xiaoming 2018-08-28 17:45 ` Tim.Bird 2018-08-29 8:01 ` [Fuego] 答复: " Li, Xiaoming
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.