Re: [Fuego] [agl-discussions] [RFC]AGL-Fuego output test result change from xml to json

["Jan-Simon Möller" <jsmoeller@linuxfoundation.org>]

AGL-JTA / Fuego sync 
Wed, Apr 12, 2017 10:00 PM - 11:00 PM PDT 

Please join my meeting from your computer, tablet or smartphone. 
https://global.gotomeeting.com/join/358810109 

You can also dial in using your phone. 
United States (Toll Free): 1 866 899 4679 
United States: +1 (312) 757-3117 

Access Code: 358-810-109 

More phone numbers 
China (Toll Free): 4008 866143 
France (Toll Free): 0 800 919 896 
France: +33 157 329 481 
Germany (Toll Free): 0 800 723 5274 
Germany: +49 69 5880 7802 72 


Am Mittwoch, 12. April 2017, 21:54:36 schrieb Jan-Simon Möller:
> I can open a conf in 5min . Link follows.
> 
> JS
> 
> Am Mittwoch, 12. April 2017, 18:37:07 schrieb Bird, Timothy:
> > Sounds good.
> > 
> > How about today at:
> > Wed. April 12, 10:00 pm (22:00) Pacific
> > Thur April 13, 1:00 pm (13:00) Beijing time
> > ?
> > 
> > That would be Thur, April 13, 5:00 am UTC.
> > 
> > Jan - let me know if you get a gotomeeting set up.  Otherwise I have a
> > conference number  I can
> 
>  provide.
> 
> > Song, let me know if this works for you.  If it does, but we don't hear
> > from Jan-Simon, then
> 
>  I'll send the conference number shortly before the call.
> 
> >  -- Tim
> >  
> > > -----Original Message-----
> > > From: automotive-discussions-bounces@lists.linuxfoundation.org
> > > [mailto:automotive-discussions-bounces@lists.linuxfoundation.org] On
> > > Behalf Of Cai, Song
> > > Sent: Wednesday, April 12, 2017 4:40 AM
> > > To: Jan-Simon Möller <jsmoeller@linuxfoundation.org>;
> > > fuego@lists.linuxfoundation.org; automotive-
> > > discussions@lists.linuxfoundation.org
> > > Subject: Re: [agl-discussions] [RFC]AGL-Fuego output test result change
> > > from
> 
>  xml to json
> 
> > > Thanks for your kindness.
> > > I can change my schedule according to Tim's one.
> > > 
> > > --------------------------------------------------
> > > B.R.
> > > Cai Song
> > > --------------------------------------------------
> > > 
> > > -----Original Message-----
> > > From: Jan-Simon Möller [mailto:jsmoeller@linuxfoundation.org]
> > > Sent: Wednesday, April 12, 2017 7:23 PM
> > > To: automotive-discussions@lists.linuxfoundation.org
> > > Cc: Cai, Song/蔡 嵩 <cais.fnst@cn.fujitsu.com>; Bird, Timothy
> > > <Tim.Bird@sony.com>; fuego@lists.linuxfoundation.org
> > > Subject: Re: [agl-discussions] [RFC]AGL-Fuego output test result change
> > > from
> 
>  xml to json
> 
> > > Hi Tim / Song,
> > > 
> > > I can host a gotomeeting bridge if needed. Just ping with the timeslot.
> > > 
> > > Best,
> > > JS
> > > 
> > > Am Mittwoch, 12. April 2017, 10:40:09 schrieb Cai, Song:
> > > > Hi Tim:
> > > >   Our job about LTP is going to draw to an end.
> > > >   So, can you take some time for us to discuss the issues "xml to json
> > > > 
> > > > recently", since both of us have done the work by ourselves before.
> > > > Thursday and Friday this week is available, or other time at your
> > > > convinces. My time zone is UTC +8. Considering the network in China,
> > > > goto meeting maybe be good choice. Looking forward to your reply.
> > > > 
> > > >   Thanks.
> > > > 
> > > > --------------------------------------------------
> > > > B.R.
> > > > Cai Song
> > > > --------------------------------------------------
> > > > 
> > > > 
> > > > 
> > > > -----Original Message-----
> > > > From: Bird, Timothy [mailto:Tim.Bird@sony.com]
> > > > Sent: Tuesday, April 04, 2017 12:21 PM
> > > > To: Cai, Song/蔡 嵩 <cais.fnst@cn.fujitsu.com>;
> > > > automotive-discussions@lists.linuxfoundation.org;
> > > > fuego@lists.linuxfoundation.org Subject: RE: [agl-discussions]
> > > > [RFC]AGL-Fuego output test result change from xml to json
> > > > 
> > > > 
> > > > 
> > > > Sounds good.  Let me know when you are ready to resume work on this
> > > > part of Fuego. I'm currently doing a lot of integration work with
> > > > Daniel Sangorrin, of Toshiba. Hopefully that will be settled down by
> > > > the end of this week, and we'll have some time to work on the AGL
> > > > integration and unified output and report generation issues. -- Tim
> > > > 
> > > > > -----Original Message-----
> > > > > From: automotive-discussions-bounces@lists.linuxfoundation.org
> > > > > [mailto:automotive-discussions-bounces@lists.linuxfoundation.org] On
> > > > > Behalf Of Cai, Song
> > > > > Sent: Friday, March 31, 2017 4:03 AM
> > > > > To: Bird, Timothy <Tim.Bird@sony.com>; automotive-
> > > > > discussions@lists.linuxfoundation.org;
> > > > > fuego@lists.linuxfoundation.org
> > > > > Subject: Re: [agl-discussions] [RFC]AGL-Fuego output test result
> > > > > change from xml to json
> > > > > 
> > > > > Hi Tim:
> > > > >   Sorry for my late reply.
> > > > >   Our team are solving the cases of LTP on 64 bits boards.
> > > > >   
> > > > >   
> > > > >   
> > > > >   And we have accepted the AGL community job "integrate AGL-Fuego to
> > > > > 
> > > > > Fuego" recently, so we need to keep progress on it after we finish
> > > > > LTP cases.>
> > > > > 
> > > > >   We can deal with the community job and the json issue at the same
> > > > > 
> > > > > time, since it is included in integration.
> > > > > 
> > > > >   Hope for your understanding.
> > > > >   Thanks.
> > > > > 
> > > > > B.R
> > > > > CaiSong
> > > > > 
> > > > > 
> > > > > 
> > > > > -----Original Message-----
> > > > > From: Bird, Timothy [mailto:Tim.Bird@sony.com]
> > > > > Sent: Friday, March 24, 2017 3:37 AM
> > > > > To: Cai, Song/蔡 嵩 <cais.fnst@cn.fujitsu.com>; automotive-
> > > > > discussions@lists.linuxfoundation.org;
> > > > > fuego@lists.linuxfoundation.org
> > > > > Subject: RE: [RFC]AFL-Fuego output test result change from xml to
> > > > > json
> > > > > 
> > > > > > -----Original Message-----
> > > > > > From: Cai, Song on Wednesday, March 22, 2017 7:19 PM Hello
> > > > > > Jan-Simon, AGL members
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > Fuego members consider that it will be more versatile if we output
> > > > > > the test results in json format,
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > Since json is widely used, and many tool (e.g. Kernel CI) is also
> > > > > > in json
> > > > > 
> > > > > format.
> > > > > 
> > > > > > I totally agree with that.
> > > > > 
> > > > > I strongly agree with this as well.  Our intermediate format for
> > > > > results should be json.  By "intermediate", I mean the format used
> > > > > to
> > > > > store results in the system, as opposed to the "display format".  
> > > > > We
> > > > > should support multiple display
> > > > > output formats (such as html, xml, pdf, or wiki markup), that can be
> > > > > generated from the json formatted data.
> > > > > 
> > > > > > And we plan to change the current result output format from xml to
> > > > > > json, which is scheduled to be finished until late May.
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > I would appreciate any comments on this activity.
> > > > > 
> > > > > Please note that Fuego already has the capability to put results in
> > > > > json format now.
> > > > > However, the work is scattered in different functions, and could use
> > > > > some better organization and coverage.
> > > > > 
> > > > > Here is some information that may be useful in working on this 
issue:
> > > > >  * data for a functional test that is executed by Jenkins is in the
> > > > > 
> > > > > file  build.xml>
> > > > > 
> > > > >     * this is converted to a "run.json" file inside the run package
> > > > > 
> > > > > if
> > > > > 
> > > > > 
> > > > > 
> > > > > you execute "ftc package-run" for the run
> > > > > 
> > > > >        * so, ftc has the capability to read the Jenkins build.xml
> > > > > 
> > > > > file
> > > > > 
> > > > > 
> > > > > 
> > > > > and produce a Fuego run.json file now
> > > > > 
> > > > >  * data for a functional test that is executed by 'ftc run-test'
> > > > > 
> > > > > creates the run.json file directly, and it is present in the log
> > > > > directory for that run
> > > > > 
> > > > >  * data for each metric of a benchmark test is placed in a file with
> > > > > 
> > > > > the suffix ".res.json"
> > > > > 
> > > > >     * this is currently only generated when Fuego is executing a
> > > > > 
> > > > > batch
> > > > > 
> > > > > 
> > > > > 
> > > > > job (this should be extended to all job execution contexts)
> > > > > 
> > > > >     * this file is created by parser.py, when the benchmark metrics
> > > > > 
> > > > > are collected from the log
> > > > > 
> > > > >  * additionally, when a batch job is executed, a "logrun" file is
> > > > > 
> > > > > created, in JSON format
> > > > > 
> > > > >     * this file is used to collect the data from the multiple tests
> > > > > 
> > > > > in
> > > > > 
> > > > > 
> > > > > 
> > > > > the batch job, so they can be formatted into a PDF by the report
> > > > > generator
> > > > > 
> > > > >  * There are also some json files used for generating plots using
> > > > > 
> > > > > the  'flot'
> > > > > 
> > > > > 
> > > > > 
> > > > > module
> > > > > 
> > > > >     * there is one file per metric in the log/<testname> directory
> > > > >     * there is an 'info.json' file in the log/<testname> directory,
> > > > > 
> > > > > which has additional data used for the plot
> > > > > 
> > > > > 
> > > > > 
> > > > > Note that some of these features are currently broken in the 'next'
> > > > > branch, as we work through the integration of the Toshiba and Sony
> > > > > code bases (the Jenkins integration refactoring).
> > > > > I hope to have some of the broken issues fixed by the 1.1 release,
> > > > > and all of them resolved by the 1.2 release.
> > > > > 
> > > > > 
> > > > > 
> > > > > Please see the following pages, which have information on the
> > > > > logrun,
> > > > > 
> > > > > res.json, and run.json files:
> > > > >  * http://bird.org/fuego/Report_Generator
> > > > >  * http://bird.org/fuego/run.json
> > > > > 
> > > > > What I'd like to see happen, is for all test result data to be put
> > > > > into the run.json file.
> > > > > This includes data that is now put into res.json, and data that is
> > > > > put into the xlsx files for tests that have lots of sub-results
> > > > > (e.g.
> > > > > LTP).
> > > > > 
> > > > > 
> > > > > 
> > > > > Also, we should make sure that the data and the run.json file are
> > > > > created in all contexts - not just when a test is run from ftc, or
> > > > > when a batch job is executed.
> > > > > 
> > > > > 
> > > > > 
> > > > > Also, it sounds like it would be handy to have the board data in
> > > > > json format. We have python code now that parses the board file
> > > > > (both in ovgen.py and in ftc), and produces python data structures
> > > > > for the board data. It might be possible to reuse this parsing code
> > > > > for your work.  In particular, the board data might be getting more
> > > > > complex in the future, as we add the data dictionary feature to
> > > > > Fuego to store dynamic variables for boards.  I can trivially add
> > > > > some code to ftc to output the board data in json format, for
> > > > > processing
> > > 
> > > by other tools.
> > > 
> > > > > Let me know if you would like me to do this.
> > > > > 
> > > > > 
> > > > > 
> > > > > Please let me know what you are planning on doing in this regard,
> > > > > and we can discuss details of the design, to make sure it fits with
> > > > > the long-term vision for Fuego results (and report generation)
> > > > > storage and display.
> > > > > 
> > > > > 
> > > > > 
> > > > > BTW - I saw your slides from the Japan Jamboree presentation.  They
> > > > > look good.
> > > > > I'll post a link to them on http://bird.org/fuego/Presentations.
> > > > > 
> > > > > 
> > > > > 
> > > > > Thanks,
> > > > > 
> > > > >  -- Tim
> > > > > 
> > > > > P.S. Note that I've made a slight modification to reports.sh in my
> > > > > 'next'
> > > > > branch, just today, based on this discussion.  The function
> > > > > set_testres_file in reports.sh used to only set the GEN_TESTRES_FILE
> > > > > when operating in batch mode.  However, I've removed this condition.
> > > > > Tests should now always create a res.json file.
> > > > > See https://bitbucket.org/tbird20d/fuego-> >
> > > > > core/commits/6cd8fe8bde38887b12b1a5b1713877a0ec754a2d
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > _______________________________________________
> > > > > automotive-discussions mailing list
> > > > > automotive-discussions@lists.linuxfoundation.org
> > > > > https://lists.linuxfoundation.org/mailman/listinfo/automotive-discus
> > > > > si
> > > > > ons
> > > > 
> > > > _______________________________________________
> > > > automotive-discussions mailing list
> > > > automotive-discussions@lists.linuxfoundation.org
> > > > https://lists.linuxfoundation.org/mailman/listinfo/automotive-discussi
> > > > ons
> > > 
> > > _______________________________________________
> > > automotive-discussions mailing list
> > > automotive-discussions@lists.linuxfoundation.org
> > > https://lists.linuxfoundation.org/mailman/listinfo/automotive-discussion
> > > s