[Fuego] benchmark metrics changes (was RE: [PATCH] tests.info: reboot has only a time plot at the moment)

[Fuego] benchmark metrics changes (was RE: [PATCH] tests.info: reboot has only a time plot at the moment)

[Bird, Timothy]

> -----Original Message-----
> From: Daniel Sangorrin on Tuesday, October 18, 2016 7:07 PM
> 
> The memory and disk plots are not available so remove them
> until implemented.
> TODO: why are there two tests.info files?

OK, I don't know why there were two tests.info files, but from what
I could see it was a bit of a mess in there.

I did a bunch of research on how tests.info is used, and where the
plotting is done in Fuego, in order to understand how to fix this.
I have put my notes about parsing-related files (including tests.info)
on http://bird.org/fuego/Benchmark_parser_notes

Here are some interesting tidbits - we actually have two completely
different plotting systems in fuego - parser.py uses matplotlib (a python
plotting module) to generate the static plot.png that is linked to from
the Benchmark status page.  A Jenkins plugin called 'flot' (authored by Cogent,
and using the 'flot' jquery plotting module) is used to produce
the dynamic plot that appears on the status page itself.

I'm not sure why there are two different plotting systems, but the matplotlib
(plot.png) one doesn't seem configured right.  It only shows data for one run,
that I can see, and it's zoom factor is way off.

It's the 'flot' system that uses tests.info, and it's structured a bit weird.
I decided to break apart the test.info file, and have changed to a system where the
information that was collected into tests.info now appears in each test's
main directory, as 'metrics.json'.  This makes it so that each test is more
independent of the system.  The dataload.py program, which is used to
populate the files used by the flot system (which appear under /userdata/logs,
by the way), had been modified to copy metrics.json to the appropriate
directory, at runtime.  Everything should be backwards compatible with tests
that exist outside the fuego-core directory and don't have a metrics.json file.
(I'm not sure how many of these there are, but AGL-CIAT probably has several).

This helps move us in the direction of having all meta-data for a test self-contained
in a single per-test directory - instead of scattered around the system like it is now.

This is all in the 'next' branches for fuego and fuego-core on bitbucket.org/tbird20d,
and my plan is to roll it out in the next release, whenever that is.

Dmitry - I hope I haven't broken anything.  Please tell me why tests.info was
structured to have all the metric names in a single file for all the Benchmark tests, when
only one test is plotted at a time, in case there's some need for this that I'm missing.

Thanks,
 -- Tim

P.S. This took a bit longer than expected, as I had to learn about Jenkins plugins,
Javascript, jQuery, and the flot plotting module, before I could make the modifications
for this change.  There was a bit more to learn than I expected.

Re: [Fuego] benchmark metrics changes (was RE: [PATCH] tests.info: reboot has only a time plot at the moment)

[Daniel Sangorrin]

> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird@am.sony.com]
> > -----Original Message-----
> > From: Daniel Sangorrin on Tuesday, October 18, 2016 7:07 PM
> >
> > The memory and disk plots are not available so remove them
> > until implemented.
> > TODO: why are there two tests.info files?
> 
> OK, I don't know why there were two tests.info files, but from what
> I could see it was a bit of a mess in there.
> 
> I did a bunch of research on how tests.info is used, and where the
> plotting is done in Fuego, in order to understand how to fix this.
> I have put my notes about parsing-related files (including tests.info)
> on http://bird.org/fuego/Benchmark_parser_notes
> 
> Here are some interesting tidbits - we actually have two completely
> different plotting systems in fuego - parser.py uses matplotlib (a python
> plotting module) to generate the static plot.png that is linked to from
> the Benchmark status page.  A Jenkins plugin called 'flot' (authored by Cogent,
> and using the 'flot' jquery plotting module) is used to produce
> the dynamic plot that appears on the status page itself.
> 
> I'm not sure why there are two different plotting systems, but the matplotlib
> (plot.png) one doesn't seem configured right.  It only shows data for one run,
> that I can see, and it's zoom factor is way off.

Actually, if you try the "reboot" test with my patch, the plot.png 
should be working. Please let me know otherwise. 
Dhrystone was also working if I remember correctly.

> It's the 'flot' system that uses tests.info, and it's structured a bit weird.
> I decided to break apart the test.info file, and have changed to a system where the
> information that was collected into tests.info now appears in each test's
> main directory, as 'metrics.json'.  This makes it so that each test is more
> independent of the system.  The dataload.py program, which is used to
> populate the files used by the flot system (which appear under /userdata/logs,
> by the way), had been modified to copy metrics.json to the appropriate
> directory, at runtime.  Everything should be backwards compatible with tests
> that exist outside the fuego-core directory and don't have a metrics.json file.
> (I'm not sure how many of these there are, but AGL-CIAT probably has several).

From a first glance at the AGL-JTA-core repository, they have a "common" folder
that includes almost the same tests as Fuego (some tests are new and some are missing).
Most of them have unmodified parser.py files (apart from the JTA-Fuego renaming).
However, I observe that each test also has a file "create_xml_testname.py". From a quick
read it looks like they are parsing the logs into xml files, and at the same time saving information
such as timing or build number.  I haven't checked yet but my guess is they are later
converting the XML files into HTML so they can be read from the Jenkins interface.
# I will check what the XML format looks like next week and report

By the way, most of these "create_xml_testname.py" files have lots of code in common,
with just a few lines modified. Probably it needs to be re-architected.

Cheers,
Daniel

> This helps move us in the direction of having all meta-data for a test self-contained
> in a single per-test directory - instead of scattered around the system like it is now.
> 
> This is all in the 'next' branches for fuego and fuego-core on bitbucket.org/tbird20d,
> and my plan is to roll it out in the next release, whenever that is.
> 
> Dmitry - I hope I haven't broken anything.  Please tell me why tests.info was
> structured to have all the metric names in a single file for all the Benchmark tests, when
> only one test is plotted at a time, in case there's some need for this that I'm missing.
> 
> Thanks,
>  -- Tim
> 
> P.S. This took a bit longer than expected, as I had to learn about Jenkins plugins,
> Javascript, jQuery, and the flot plotting module, before I could make the modifications
> for this change.  There was a bit more to learn than I expected.
> 

Re: [Fuego] benchmark metrics changes (was RE: [PATCH] tests.info: reboot has only a time plot at the moment)

[Bird, Timothy]

> -----Original Message-----
> From: Daniel Sangorrin on  Thursday, November 17, 2016 12:56 AM> 
> > -----Original Message-----
> > From: Bird, Timothy [mailto:Tim.Bird@am.sony.com]
...
> > I'm not sure why there are two different plotting systems, but the
> matplotlib
> > (plot.png) one doesn't seem configured right.  It only shows data for one
> run,
> > that I can see, and it's zoom factor is way off.
> 
> Actually, if you try the "reboot" test with my patch, the plot.png
> should be working. Please let me know otherwise.
> Dhrystone was also working if I remember correctly.

I'll double-check this.  I keep getting my plot data messed up.
I believe it's because the test numbering re-sets every time
I build a new docker container.

I'm considering either putting some of the info from /home/jenkins/fuego/jobs
into /userdata as well, or adding a manual tool to sync nextBuildNumber
when a new container is deployed.

...
> From a first glance at the AGL-JTA-core repository, they have a "common"
> folder
> that includes almost the same tests as Fuego (some tests are new and some
> are missing).
> Most of them have unmodified parser.py files (apart from the JTA-Fuego
> renaming).
> However, I observe that each test also has a file "create_xml_testname.py".
> From a quick
> read it looks like they are parsing the logs into xml files, and at the same time
> saving information
> such as timing or build number.  I haven't checked yet but my guess is they
> are later
> converting the XML files into HTML so they can be read from the Jenkins
> interface.

OK - that's interesting.   Putting extra information in the Jenkins interface
would be really useful.  I'd like to figure out how to get direct links to the
test log file for a particular test run.  The logs are sitting there under
/var/lib/jenkins/userContent/fuego.logs, which is visible via the web
server interface.  But it would be convenient to have a direct link.

However, with regard to the JTA-CIAT solution, I'm not an XML fan. 
I'm reminded me of an old saying:
"Sometimes, when a developer is faced with a technical problem, they
decide to solve it using XML.  Now they have 2 problems."  :-)

> # I will check what the XML format looks like next week and report

Thanks.

I've been toying with converting some results to markdown, for
another front end I've been experimenting with.
This might work for reports as well, if we can settle on something
that works for both online (web) and printed (pdf) docs.

Another thing would be to color-code the logs, with some post-processing.
Scanning visually through the Jenkins console logs for the Fuego error
is time consuming. This could be helped by a post-processor that just
added '<font color="red">' and '</font>' around some Fuego messages.

 -- Tim

Re: [Fuego] benchmark metrics changes (was RE: [PATCH] tests.info: reboot has only a time plot at the moment)

[Daniel Sangorrin]

Hi Tim,

> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird@am.sony.com]
> Sent: Friday, November 18, 2016 3:17 AM
> OK - that's interesting.   Putting extra information in the Jenkins interface
> would be really useful.  I'd like to figure out how to get direct links to the
> test log file for a particular test run.  The logs are sitting there under
> /var/lib/jenkins/userContent/fuego.logs, which is visible via the web
> server interface.  But it would be convenient to have a direct link.

You can add a "Set build description" section and use these lines:
Dbench benchmark<p><b><a href="/userContent/jta.logs/${JOB_NAME}/testlogs/${NODE_NAME}.${BUILD_ID}.${BUILD_NUMBER}.log">Test Result</a></b></p>

[Note] substitute testlogs by devlogs for the complete logs
[Note] This works on AGL-JTA, I haven't tested it in Fuego yet because I'm figuring 
out how to upgrade it in a clean and maintainable way.
 
> However, with regard to the JTA-CIAT solution, I'm not an XML fan.
> I'm reminded me of an old saying:
> "Sometimes, when a developer is faced with a technical problem, they
> decide to solve it using XML.  Now they have 2 problems."  :-)

Haha well, that's a question of taste. I find python's library xmltodict (same one
as the one AGL-JTA is using) quite easy to use, and the XML format is easy to read.
The real power of XML comes from its ability to validate against an XML Schema.
Another option is to use JSON, which nowadays  has very similar functionality.

I wouldn't mind so much about the format as long as there are good mature tools for 
parsing and visualizing the test results.

> I've been toying with converting some results to markdown, for
> another front end I've been experimenting with.
> This might work for reports as well, if we can settle on something
> that works for both online (web) and printed (pdf) docs.

Sounds good. 

Once we have a single common output format (e.g. XML or JSON) we can
make several visualization tools. 

> Another thing would be to color-code the logs, with some post-processing.
> Scanning visually through the Jenkins console logs for the Fuego error
> is time consuming. This could be helped by a post-processor that just
> added '<font color="red">' and '</font>' around some Fuego messages.

That would be great.

Best regards,
Daniel