From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; h=from:to:cc:subject:date:message-id:organization:user-agent :in-reply-to:references:mime-version:content-transfer-encoding; bh=4GUyeN6qLUbP3M2wNJDDHQIdHtmr4IcQNrQuC+oLl7w=; b=GqTHdGa2OzVyVrIu2ewIYw5//4fg2u+ISSBzDUXveOHmQ8M3sz1Rkx+y1o6Z1UDXmk ST399mNiseJ3W8oGivyngcn8GuF4taiFuazEufCKDnXpX75G/5/Tbqs/9FUTgw1Po7vL 8Fn89rojUvRGz7yLZeJogmlHz/op4FOHnTu5M= From: Jan-Simon =?ISO-8859-1?Q?M=F6ller?= Date: Wed, 12 Apr 2017 21:54:36 +0200 Message-ID: <8751900.SUhSrPJfrz@elrond> In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Subject: Re: [Fuego] [agl-discussions] [RFC]AGL-Fuego output test result change from xml to json List-Id: Mailing list for the Fuego test framework List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Bird, Timothy" Cc: "fuego@lists.linuxfoundation.org" , "automotive-discussions@lists.linuxfoundation.org" I can open a conf in 5min . Link follows. JS Am Mittwoch, 12. April 2017, 18:37:07 schrieb Bird, Timothy: > Sounds good. >=20 > How about today at: > Wed. April 12, 10:00 pm (22:00) Pacific > Thur April 13, 1:00 pm (13:00) Beijing time > ? >=20 > That would be Thur, April 13, 5:00 am UTC. >=20 > Jan - let me know if you get a gotomeeting set up. Otherwise I have = a > conference number I can=20 provide. >=20 > Song, let me know if this works for you. If it does, but we don't he= ar from > Jan-Simon, then=20 I'll send the conference number shortly before the call. > -- Tim >=20 >=20 > > -----Original Message----- > > From: automotive-discussions-bounces@lists.linuxfoundation.org > > [mailto:automotive-discussions-bounces@lists.linuxfoundation.org] O= n > > Behalf Of Cai, Song > > Sent: Wednesday, April 12, 2017 4:40 AM > > To: Jan-Simon M=C3=B6ller ; > > fuego@lists.linuxfoundation.org; automotive- > > discussions@lists.linuxfoundation.org > > Subject: Re: [agl-discussions] [RFC]AGL-Fuego output test result ch= ange > > from xml to json > >=20 > > Thanks for your kindness. > > I can change my schedule according to Tim's one. > >=20 > > -------------------------------------------------- > > B.R. > > Cai Song > > -------------------------------------------------- > >=20 > > -----Original Message----- > > From: Jan-Simon M=C3=B6ller [mailto:jsmoeller@linuxfoundation.org] > > Sent: Wednesday, April 12, 2017 7:23 PM > > To: automotive-discussions@lists.linuxfoundation.org > > Cc: Cai, Song/=E8=94=A1 =E5=B5=A9 ; Bird,= Timothy > > ; fuego@lists.linuxfoundation.org > > Subject: Re: [agl-discussions] [RFC]AGL-Fuego output test result ch= ange > > from xml to json > >=20 > > Hi Tim / Song, > >=20 > > I can host a gotomeeting bridge if needed. Just ping with the times= lot. > >=20 > > Best, > > JS > > Am Mittwoch, 12. April 2017, 10:40:09 schrieb Cai, Song: > >=20 > > > Hi Tim: > > >=20 > > > Our job about LTP is going to draw to an end. > > > So, can you take some time for us to discuss the issues "xml to= json > > >=20 > > > recently", since both of us have done the work by ourselves befor= e. > > > Thursday and Friday this week is available, or other time at your= > > > convinces. My time zone is UTC +8. Considering the network in Chi= na, > > > goto meeting maybe be good choice. Looking forward to your reply.= > > >=20 > > > Thanks. > > > > > > > > > > > > -------------------------------------------------- > > > B.R. > > > Cai Song > > > -------------------------------------------------- > > > > > > > > > > > > -----Original Message----- > > > From: Bird, Timothy [mailto:Tim.Bird@sony.com] > > > Sent: Tuesday, April 04, 2017 12:21 PM > > > To: Cai, Song/=E8=94=A1 =E5=B5=A9 ; > > > automotive-discussions@lists.linuxfoundation.org; > > > fuego@lists.linuxfoundation.org Subject: RE: [agl-discussions] > > > [RFC]AGL-Fuego output test result change from xml to json > > > > > > > > > > > > Sounds good. Let me know when you are ready to resume work on th= is > > > part of Fuego. I'm currently doing a lot of integration work with= > > > Daniel Sangorrin, of Toshiba. Hopefully that will be settled down= by > > > the end of this week, and we'll have some time to work on the AGL= > > > integration and unified output and report generation issues. -- T= im > > > > > > > > > > > > > -----Original Message----- > > > > From: automotive-discussions-bounces@lists.linuxfoundation.org > > > > [mailto:automotive-discussions-bounces@lists.linuxfoundation.or= g] On > > > > Behalf Of Cai, Song > > > > Sent: Friday, March 31, 2017 4:03 AM > > > > To: Bird, Timothy ; automotive- > > > > discussions@lists.linuxfoundation.org; > > > > fuego@lists.linuxfoundation.org > > > > Subject: Re: [agl-discussions] [RFC]AGL-Fuego output test resul= t > > > > change from xml to json > > > > > > > > > > > > > > > > Hi Tim: > > > >=20 > > > > Sorry for my late reply. > > > > Our team are solving the cases of LTP on 64 bits boards. > > > > > > > > > > > > > > > > And we have accepted the AGL community job "integrate AGL-Fue= go to > > > > > > > > > > > > > > > > Fuego" recently, so we need to keep progress on it after we fin= ish > > > > LTP cases.> > > > >=20 > > > > We can deal with the community job and the json issue at the = same > > > > > > > > > > > > > > > > time, since it is included in integration. > > > > > > > > > > > > > > > > Hope for your understanding. > > > > Thanks. > > > > > > > > > > > > > > > > B.R > > > > CaiSong > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: Bird, Timothy [mailto:Tim.Bird@sony.com] > > > > Sent: Friday, March 24, 2017 3:37 AM > > > > To: Cai, Song/=E8=94=A1 =E5=B5=A9 ; a= utomotive- > > > > discussions@lists.linuxfoundation.org; > > > > fuego@lists.linuxfoundation.org > > > > Subject: RE: [RFC]AFL-Fuego output test result change from xml = to > > > > json > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: Cai, Song on Wednesday, March 22, 2017 7:19 PM Hello > > > > > Jan-Simon, AGL members > > > > > > > > > > > > > > > > > > > > Fuego members consider that it will be more versatile if we o= utput > > > > > the test results in json format, > > > > > > > > > > > > > > > > > > > > Since json is widely used, and many tool (e.g. Kernel CI) is = also > > > > > in json > > > > > > > > > > > > > > > > format. > > > > > > > > > > > > > > > > > I totally agree with that. > > > > > > > > > > > > > > > > I strongly agree with this as well. Our intermediate format fo= r > > > > results should be json. By "intermediate", I mean the format u= sed to > > > > store results in the system, as opposed to the "display format"= . We > > > > should support multiple display > > > > output formats (such as html, xml, pdf, or wiki markup), that c= an be > > > > generated from the json formatted data. > > > > > > > > > > > > > > > > > And we plan to change the current result output format from x= ml to > > > > > json, which is scheduled to be finished until late May. > > > > > > > > > > > > > > > > > > > > I would appreciate any comments on this activity. > > > > > > > > > > > > > > > > Please note that Fuego already has the capability to put result= s in > > > > json format now. > > > > However, the work is scattered in different functions, and coul= d use > > > > some better organization and coverage. > > > > > > > > > > > > > > > > Here is some information that may be useful in working on this = issue: > > > >=20 > > > > * data for a functional test that is executed by Jenkins is in= the > > > >=20 > > > > file build.xml> > > > >=20 > > > > * this is converted to a "run.json" file inside the run pac= kage > > > >=20 > > > > if > > > > > > > > > > > > > > > > you execute "ftc package-run" for the run > > > > > > > > > > > > > > > > * so, ftc has the capability to read the Jenkins build.x= ml > > > >=20 > > > > file > > > > > > > > > > > > > > > > and produce a Fuego run.json file now > > > > > > > > > > > > > > > > * data for a functional test that is executed by 'ftc run-test= ' > > > > > > > > > > > > > > > > creates the run.json file directly, and it is present in the lo= g > > > > directory for that run > > > > > > > > > > > > > > > > * data for each metric of a benchmark test is placed in a file= with > > > > > > > > > > > > > > > > the suffix ".res.json" > > > > > > > > > > > > > > > > * this is currently only generated when Fuego is executing = a > > > >=20 > > > > batch > > > > > > > > > > > > > > > > job (this should be extended to all job execution contexts) > > > > > > > > > > > > > > > > * this file is created by parser.py, when the benchmark met= rics > > > > > > > > > > > > > > > > are collected from the log > > > > > > > > > > > > > > > > * additionally, when a batch job is executed, a "logrun" file = is > > > > > > > > > > > > > > > > created, in JSON format > > > > > > > > > > > > > > > > * this file is used to collect the data from the multiple t= ests > > > >=20 > > > > in > > > > > > > > > > > > > > > > the batch job, so they can be formatted into a PDF by the repor= t > > > > generator > > > > > > > > > > > > > > > > * There are also some json files used for generating plots usi= ng > > > >=20 > > > > the 'flot' > > > > > > > > > > > > > > > > module > > > > > > > > > > > > > > > > * there is one file per metric in the log/ direct= ory > > > > * there is an 'info.json' file in the log/ direct= ory, > > > > > > > > > > > > > > > > which has additional data used for the plot > > > > > > > > > > > > > > > > Note that some of these features are currently broken in the 'n= ext' > > > > branch, as we work through the integration of the Toshiba and S= ony > > > > code bases (the Jenkins integration refactoring). > > > > I hope to have some of the broken issues fixed by the 1.1 relea= se, > > > > and all of them resolved by the 1.2 release. > > > > > > > > > > > > > > > > Please see the following pages, which have information on the > > > > logrun, > > > > > > > > > > > > > > > > res.json, and run.json files: > > > >=20 > > > > * http://bird.org/fuego/Report_Generator > > > > * http://bird.org/fuego/run.json > > > > > > > > > > > > > > > > What I'd like to see happen, is for all test result data to be = put > > > > into the run.json file. > > > > This includes data that is now put into res.json, and data that= is > > > > put into the xlsx files for tests that have lots of sub-results= (e.g. > > > > LTP). > > > > > > > > > > > > > > > Also, we should make sure that the data and the run.json file a= re > > > > created in all contexts - not just when a test is run from ftc,= or > > > > when a batch job is executed. > > > > > > > > > > > > > > > > Also, it sounds like it would be handy to have the board data i= n > > > > json format. We have python code now that parses the board file= > > > > (both in ovgen.py and in ftc), and produces python data structu= res > > > > for the board data. It might be possible to reuse this parsing = code > > > > for your work. In particular, the board data might be getting = more > > > > complex in the future, as we add the data dictionary feature to= > > > > Fuego to store dynamic variables for boards. I can trivially a= dd > > > > some code to ftc to output the board data in json format, for > > > > processing > >=20 > > by other tools. > >=20 > > > > Let me know if you would like me to do this. > > > > > > > > > > > > > > > > Please let me know what you are planning on doing in this regar= d, > > > > and we can discuss details of the design, to make sure it fits = with > > > > the long-term vision for Fuego results (and report generation) > > > > storage and display. > > > > > > > > > > > > > > > > BTW - I saw your slides from the Japan Jamboree presentation. = They > > > > look good. > > > > I'll post a link to them on http://bird.org/fuego/Presentations= . > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > -- Tim > > > > > > > > > > > > > > > > P.S. Note that I've made a slight modification to reports.sh in= my > > > > 'next' > > > > branch, just today, based on this discussion. The function > > > > set_testres_file in reports.sh used to only set the GEN_TESTRES= _FILE > > > > when operating in batch mode. However, I've removed this condi= tion. > > > > Tests should now always create a res.json file. > > > > See https://bitbucket.org/tbird20d/fuego-> > > > > > core/commits/6cd8fe8bde38887b12b1a5b1713877a0ec754a2d > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > automotive-discussions mailing list > > > > automotive-discussions@lists.linuxfoundation.org > > > > https://lists.linuxfoundation.org/mailman/listinfo/automotive-d= iscus > > > > si > > > > ons > > > > > > > > > > > > _______________________________________________ > > > automotive-discussions mailing list > > > automotive-discussions@lists.linuxfoundation.org > > > https://lists.linuxfoundation.org/mailman/listinfo/automotive-dis= cussi > > > ons > >=20 > >=20 > >=20 > >=20 > >=20 > >=20 > > _______________________________________________ > > automotive-discussions mailing list > > automotive-discussions@lists.linuxfoundation.org > > https://lists.linuxfoundation.org/mailman/listinfo/automotive-discu= ssions