[Fuego] UART/USB Connectivity, History Database, Test result format

[Fuego] UART/USB Connectivity, History Database, Test result format

[dhinakar k]

[-- Attachment #1: Type: text/plain, Size: 613 bytes --]

Hi,

I would like to know how the following are handled by Fuego.

1.Connectivity to targets which support UART or USB
   - If Fuego supports it, then how is it done? (e.g. via Perl Serial
module?)
   - How are the Request/Response handled via UART or USB connectivity?

2.How can we archive job details, execution results etc.? Does Fuego
already have support for database? If so, which database? (e.g. MySQL?)

3.What format(s) supported for test results? (json, xunit, yaml, TAP?)
   All tests integrated into Fuego will output results in the above format?

Please answer the above queries.

Regards,
Dhinakar

[-- Attachment #2: Type: text/html, Size: 842 bytes --]

Re: [Fuego] UART/USB Connectivity, History Database, Test result format

[Bird, Timothy]

> -----Original Message-----
> From dhinakar k on Tuesday, March 28, 2017 3:26 AM
> Hi,
> 
> I would like to know how the following are handled by Fuego.
> 
> 1.Connectivity to targets which support UART or USB
>    - If Fuego supports it, then how is it done? (e.g. via Perl Serial module?)
>    - How are the Request/Response handled via UART or USB connectivity?

UART connectivity is handled with the 'serial' transport, using a program
called 'serio'.  Support for this is currently in what I'd call 'alpha' format.
It works, for the most part, with a few issues that need to be resolved
before we can call it an official feature.

For USB there are two methods of support (one existing and one intended):
 * network over USB - that is supported today, but the configuration of the
USB networking is left as an exercise for the user
 * ADB - There are patches available to support ADB, but they are not
currently integrated.

> 2.How can we archive job details, execution results etc.? Does Fuego already
> have support for database? If so, which database? (e.g. MySQL?)

Currently, at a local site, job results are saved in XML in the Jenkins build.xml
file.  Fuego saves it's own parallel set of results to a run.json file (when the test
is executed with 'ftc run-test').  This will be changing shortly (likely in the 1.1 release),
to have all tests (including those executed by Jenkins) create a run.json file.

We have not decided on a centralized database yet, although there is prototype
code in the fuego wiki software to manage json files like a database (e.g. supporting filters
and queries and table generation).  However, this wiki code is not finished yet,
and is lower priority.
 
> 3.What format(s) supported for test results? (json, xunit, yaml, TAP?)
>    All tests integrated into Fuego will output results in the above format?

The target is json, with converters to other formats as needed.  Currently,
benchmarks produce json output, and there is prototype code to have
functional tests produce json output (for sub-tests) as well. The AGL group has code
to produce XML, but it appears they are interested in switching to json output,
and the intent is to unify around this format.

The intent is to have all tests integrated into Fuego produce the json output,
and have report generators that take the format as input.  We have report generators
that take the benchmark data, and functional data (in json) and produce pdf
reports (with xml and tex as intermediate formats).  However, with a big refactoring
that we're doing in the format output, there will be some churn on this part of
Fuego over the next 2 months or so.

> Please answer the above queries.

I hope this is helpful.
 -- Tim

Re: [Fuego] UART/USB Connectivity, History Database, Test result format

[dhinakar k]

[-- Attachment #1: Type: text/plain, Size: 3134 bytes --]

Hi Tim,

That was a very detailed reply.
Thank you very much.

Just one follow up question.
How does Fuego compare with Avocado?
If you had checked it out already please let me know.

Regards,
Dhinakar

On Tue, Mar 28, 2017 at 11:54 PM, Bird, Timothy <Tim.Bird@sony.com> wrote:

>
>
> > -----Original Message-----
> > From dhinakar k on Tuesday, March 28, 2017 3:26 AM
> > Hi,
> >
> > I would like to know how the following are handled by Fuego.
> >
> > 1.Connectivity to targets which support UART or USB
> >    - If Fuego supports it, then how is it done? (e.g. via Perl Serial
> module?)
> >    - How are the Request/Response handled via UART or USB connectivity?
>
> UART connectivity is handled with the 'serial' transport, using a program
> called 'serio'.  Support for this is currently in what I'd call 'alpha'
> format.
> It works, for the most part, with a few issues that need to be resolved
> before we can call it an official feature.
>
> For USB there are two methods of support (one existing and one intended):
>  * network over USB - that is supported today, but the configuration of the
> USB networking is left as an exercise for the user
>  * ADB - There are patches available to support ADB, but they are not
> currently integrated.
>
> > 2.How can we archive job details, execution results etc.? Does Fuego
> already
> > have support for database? If so, which database? (e.g. MySQL?)
>
> Currently, at a local site, job results are saved in XML in the Jenkins
> build.xml
> file.  Fuego saves it's own parallel set of results to a run.json file
> (when the test
> is executed with 'ftc run-test').  This will be changing shortly (likely
> in the 1.1 release),
> to have all tests (including those executed by Jenkins) create a run.json
> file.
>
> We have not decided on a centralized database yet, although there is
> prototype
> code in the fuego wiki software to manage json files like a database (e.g.
> supporting filters
> and queries and table generation).  However, this wiki code is not
> finished yet,
> and is lower priority.
>
> > 3.What format(s) supported for test results? (json, xunit, yaml, TAP?)
> >    All tests integrated into Fuego will output results in the above
> format?
>
> The target is json, with converters to other formats as needed.  Currently,
> benchmarks produce json output, and there is prototype code to have
> functional tests produce json output (for sub-tests) as well. The AGL
> group has code
> to produce XML, but it appears they are interested in switching to json
> output,
> and the intent is to unify around this format.
>
> The intent is to have all tests integrated into Fuego produce the json
> output,
> and have report generators that take the format as input.  We have report
> generators
> that take the benchmark data, and functional data (in json) and produce pdf
> reports (with xml and tex as intermediate formats).  However, with a big
> refactoring
> that we're doing in the format output, there will be some churn on this
> part of
> Fuego over the next 2 months or so.
>
> > Please answer the above queries.
>
> I hope this is helpful.
>  -- Tim
>
>

[-- Attachment #2: Type: text/html, Size: 3851 bytes --]

Re: [Fuego] UART/USB Connectivity, History Database, Test result format

[Bird, Timothy]

> -----Original Message-----
> From: dhinakar k on Tuesday, March 28, 2017 7:34 PM
> 
> Hi Tim,
> 
> That was a very detailed reply.
> Thank you very much.
> 
> Just one follow up question.
> How does Fuego compare with Avocado?
> If you had checked it out already please let me know.

I did take a quick look at Avacado a while back, and put
some notes at:
http://bird.org/fuego/Other_test_systems#Avacado

Avacado has some very nice features.
It seems like it excels at handling large test matrices.  It has yaml
files to define the different test variations.

They also interface well with Jenkins - a single command-line from a Jenkins
job can start an avocado test, and they product output that is easily parsed
by existing Jenkins post-processors.
e.g. 'avacado run /bin/true sleeptest passtest --xunit "$WORKSPACE/results.xml"...'

They have an 'avacado server' concept, where each host can execute jobs from
another machine (I believe - I haven't tried it out).

So far, it looks like Avacado is used for virtualization testing.  I don't see a 
lot of different tests that run on this framework.

Here are some of the major difference, that I can see (this is with limited
study of Avacado, so I apologize for any omissions of features on their part):
1) Fuego is both a test framework, and a distribution of tests themselves.
  * We have about 60 tests now, with about 90 in our 'next' branch
     * the goal is to have hundreds of tests available, for a wide variety of system features and issues
2) Fuego provides a consistent, containerized back end (docker) so that the
 builds of the test software can be uniform between different sites.
   * this will help when comparing results from different sites
       * multi-site results comparisons are not done yet, but on our roadmap
3) Fuego has a multi-node test API (host/target), that is specially geared for
embedded linux testing
  * the footprint on the target is intentionally very small, using existing POSIX features
     * IOW - there is no specialized target agent on the target board (except for a transport agent, like sshd or adb)
  * Fuego is geared towards testing of final production images of the software stacks for products
4) Fuego is behind Avacado in terms of:
   a) unified test ouput (we're working on that now)
   b) test results sharing
       * (also in-progress - 'runs' can be packaged and shared now to a central server, in our 'next' branch)
       * however, there's only rudimentary test display and analysis on the Fuego server at the moment
           * Avacado seems to be using Jenkins for this - and we should look at them more closely to see what they're doing

There are more, but hopefully this is helpful to see some of the differences.
 -- Tim

Re: [Fuego] UART/USB Connectivity, History Database, Test result format

[dhinakar k]

[-- Attachment #1: Type: text/plain, Size: 3114 bytes --]

Hi Tim,

That was quite a detailed reply.
Thanks a lot.
Fuego looks promising, I am looking forward to the future releases of Fuego.

Regards,
Dhinakar

On Thu, Mar 30, 2017 at 11:21 PM, Bird, Timothy <Tim.Bird@sony.com> wrote:

>
>
> > -----Original Message-----
> > From: dhinakar k on Tuesday, March 28, 2017 7:34 PM
> >
> > Hi Tim,
> >
> > That was a very detailed reply.
> > Thank you very much.
> >
> > Just one follow up question.
> > How does Fuego compare with Avocado?
> > If you had checked it out already please let me know.
>
> I did take a quick look at Avacado a while back, and put
> some notes at:
> http://bird.org/fuego/Other_test_systems#Avacado
>
> Avacado has some very nice features.
> It seems like it excels at handling large test matrices.  It has yaml
> files to define the different test variations.
>
> They also interface well with Jenkins - a single command-line from a
> Jenkins
> job can start an avocado test, and they product output that is easily
> parsed
> by existing Jenkins post-processors.
> e.g. 'avacado run /bin/true sleeptest passtest --xunit
> "$WORKSPACE/results.xml"...'
>
> They have an 'avacado server' concept, where each host can execute jobs
> from
> another machine (I believe - I haven't tried it out).
>
> So far, it looks like Avacado is used for virtualization testing.  I don't
> see a
> lot of different tests that run on this framework.
>
> Here are some of the major difference, that I can see (this is with limited
> study of Avacado, so I apologize for any omissions of features on their
> part):
> 1) Fuego is both a test framework, and a distribution of tests themselves.
>   * We have about 60 tests now, with about 90 in our 'next' branch
>      * the goal is to have hundreds of tests available, for a wide variety
> of system features and issues
> 2) Fuego provides a consistent, containerized back end (docker) so that the
>  builds of the test software can be uniform between different sites.
>    * this will help when comparing results from different sites
>        * multi-site results comparisons are not done yet, but on our
> roadmap
> 3) Fuego has a multi-node test API (host/target), that is specially geared
> for
> embedded linux testing
>   * the footprint on the target is intentionally very small, using
> existing POSIX features
>      * IOW - there is no specialized target agent on the target board
> (except for a transport agent, like sshd or adb)
>   * Fuego is geared towards testing of final production images of the
> software stacks for products
> 4) Fuego is behind Avacado in terms of:
>    a) unified test ouput (we're working on that now)
>    b) test results sharing
>        * (also in-progress - 'runs' can be packaged and shared now to a
> central server, in our 'next' branch)
>        * however, there's only rudimentary test display and analysis on
> the Fuego server at the moment
>            * Avacado seems to be using Jenkins for this - and we should
> look at them more closely to see what they're doing
>
> There are more, but hopefully this is helpful to see some of the
> differences.
>  -- Tim
>
>

[-- Attachment #2: Type: text/html, Size: 3900 bytes --]