From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Sony.onmicrosoft.com; s=selector1-Sony-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=WLuSqmxzBRC0EfBbH7sEbgmGvSQ0kTuSen3Rj317JDw=; b=hwKPQ1WzU+snPF9vgmMnnSC1LpiFkZcmtFYvO1r/K4BU2d+SKcZk4nDs0dMzb0BjJOXmO/YxolSZkbhnGk97C6MqQdun4Zd6ID5NNU8uWPWNRS6G1WvVPNP3AhOQYsI5uvMKYcmMP0ugv/RRdBF2s7XBR40oM5GpZL4+PufGmjE= From: "Bird, Timothy" Date: Tue, 4 Apr 2017 04:20:34 +0000 Message-ID: References: In-Reply-To: Content-Language: en-US Content-Type: text/plain; charset="iso-2022-jp" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [Fuego] [agl-discussions] [RFC]AGL-Fuego output test result change from xml to json List-Id: Mailing list for the Fuego test framework List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Cai, Song" , "automotive-discussions@lists.linuxfoundation.org" , "fuego@lists.linuxfoundation.org" Sounds good. Let me know when you are ready to resume work on this part of= Fuego. I'm currently doing a lot of integration work with Daniel Sangorrin, of Tos= hiba. Hopefully that will be settled down by the end of this week, and we'll have= some time to work on the AGL integration and unified output and report generation iss= ues. -- Tim > -----Original Message----- > From: automotive-discussions-bounces@lists.linuxfoundation.org > [mailto:automotive-discussions-bounces@lists.linuxfoundation.org] On > Behalf Of Cai, Song > Sent: Friday, March 31, 2017 4:03 AM > To: Bird, Timothy ; automotive- > discussions@lists.linuxfoundation.org; fuego@lists.linuxfoundation.org > Subject: Re: [agl-discussions] [RFC]AGL-Fuego output test result change f= rom > xml to json >=20 > Hi Tim: > Sorry for my late reply. > Our team are solving the cases of LTP on 64 bits boards. >=20 > And we have accepted the AGL community job "integrate AGL-Fuego to > Fuego" recently, so we need to keep progress on it after we finish LTP ca= ses. >=20 > We can deal with the community job and the json issue at the same time, > since it is included in integration. >=20 > Hope for your understanding. > Thanks. > B.R > CaiSong >=20 > -----Original Message----- > From: Bird, Timothy [mailto:Tim.Bird@sony.com] > Sent: Friday, March 24, 2017 3:37 AM > To: Cai, Song/=1B$Bhq=1B(B =1B$B?s=1B(B ; autom= otive- > discussions@lists.linuxfoundation.org; fuego@lists.linuxfoundation.org > Subject: RE: [RFC]AFL-Fuego output test result change from xml to json >=20 > > -----Original Message----- > > From: Cai, Song on Wednesday, March 22, 2017 7:19 PM Hello Jan-Simon, > > AGL members > > > > Fuego members consider that it will be more versatile if we output the > > test results in json format, > > > > Since json is widely used, and many tool (e.g. Kernel CI) is also in js= on > format. > > > > I totally agree with that. > > > I strongly agree with this as well. Our intermediate format for results = should > be json. By "intermediate", I mean the format used to store results > in the system, as opposed to the "display format". We should support > multiple display > output formats (such as html, xml, pdf, or wiki markup), that can be > generated from the json formatted data. >=20 > > And we plan to change the current result output format from xml to > > json, which is scheduled to be finished until late May. > > > > I would appreciate any comments on this activity. >=20 > Please note that Fuego already has the capability to put results in json = format > now. > However, the work is scattered in different functions, and could use some > better organization and coverage. >=20 > Here is some information that may be useful in working on this issue: > * data for a functional test that is executed by Jenkins is in the file = build.xml > * this is converted to a "run.json" file inside the run package if yo= u execute > "ftc package-run" for the run > * so, ftc has the capability to read the Jenkins build.xml file an= d produce a > Fuego run.json file now > * data for a functional test that is executed by 'ftc run-test' creates = the > run.json file directly, and it is present in the log directory for that r= un > * data for each metric of a benchmark test is placed in a file with the = suffix > ".res.json" > * this is currently only generated when Fuego is executing a batch jo= b (this > should be extended to all job execution contexts) > * this file is created by parser.py, when the benchmark metrics are > collected from the log > * additionally, when a batch job is executed, a "logrun" file is created= , in > JSON format > * this file is used to collect the data from the multiple tests in th= e batch job, > so they can be formatted into a PDF by the report generator > * There are also some json files used for generating plots using the 'fl= ot' > module > * there is one file per metric in the log/ directory > * there is an 'info.json' file in the log/ directory, which= has > additional data used for the plot >=20 > Note that some of these features are currently broken in the 'next' branc= h, > as we work through the integration of the Toshiba and Sony code bases (th= e > Jenkins integration refactoring). > I hope to have some of the broken issues fixed by the 1.1 release, and al= l of > them resolved by the 1.2 release. >=20 > Please see the following pages, which have information on the logrun, > res.json, and run.json files: > * http://bird.org/fuego/Report_Generator > * http://bird.org/fuego/run.json >=20 > What I'd like to see happen, is for all test result data to be put into t= he > run.json file. > This includes data that is now put into res.json, and data that is put in= to the > xlsx files for tests that have lots of sub-results (e.g. LTP). >=20 > Also, we should make sure that the data and the run.json file are created= in > all contexts - not just when a test is run from ftc, or when a batch job = is > executed. >=20 > Also, it sounds like it would be handy to have the board data in json for= mat. > We have python code now that parses the board file (both in ovgen.py and > in ftc), and produces python data structures for the board data. It migh= t be > possible to reuse this parsing code for your work. In particular, the bo= ard > data might be getting more complex in the future, as we add the data > dictionary feature to Fuego to store dynamic variables for boards. I can > trivially add some code to ftc to output the board data in json format, f= or > processing by other tools. > Let me know if you would like me to do this. >=20 > Please let me know what you are planning on doing in this regard, and we > can discuss details of the design, to make sure it fits with the long-ter= m vision > for Fuego results (and report generation) storage and display. >=20 > BTW - I saw your slides from the Japan Jamboree presentation. They look > good. > I'll post a link to them on http://bird.org/fuego/Presentations. >=20 > Thanks, > -- Tim >=20 > P.S. Note that I've made a slight modification to reports.sh in my 'next' > branch, just today, based on this discussion. The function set_testres_f= ile in > reports.sh used to only set the GEN_TESTRES_FILE when operating in batch > mode. However, I've removed this condition. Tests should now always > create a res.json file. > See https://bitbucket.org/tbird20d/fuego- > core/commits/6cd8fe8bde38887b12b1a5b1713877a0ec754a2d >=20 >=20 >=20 >=20 >=20 > _______________________________________________ > automotive-discussions mailing list > automotive-discussions@lists.linuxfoundation.org > https://lists.linuxfoundation.org/mailman/listinfo/automotive-discussions