Re: [Fuego] [PATCH 3/3] cyclictest: modify parsing and test measures

["Bird, Timothy" <Tim.Bird@sony.com>]

> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Tuesday, October 31, 2017 1:32 AM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 3/3] cyclictest: modify parsing and test measures
> 
> cyclictest can have more or less result lines depending on
> how many threads are executed.
> 
> PROBLEM: Fuego currently does not support arrays of arbitrary
> size that can be populated dynamically. The reason
> is because we split reference.json from parser.py which
> could have easily solved that.
> 
> For now, I have decided to calculate the global minimum,
> average and maximum latency variables for all threads (1
> thread per core in the new default spec).

OK - I thought about this, and agree that this is a limitation we need to address.

I've created an issue for it here:
http://fuegotest.org/wiki/Issue_0061

Fuego currently has a pretty hard-coded notion of the test sets, test cases and
measures that will be produced by a test.  But there are clearly cases where
tests will generate results that are dependent on data determined at runtime
(or from other variations, like the spec - in the case of cyclictest, that affect
operation of the test.

I don't have any good answers at the moment, but I do have some ideas for
how to deal with these situations.  Just as a note, Benchmark.cyclictest is
a rather simple test (from Fuego's perspective - as it does not produce a lot
of different, complicated results), so it is a good candidate for experimenting
with solutions to the generic problem of dynamic results.

Thanks for bringing this up.
 -- Tim