Re: [Fuego] First steps with fuego

["Bird, Timothy" <Tim.Bird@sony.com>]

> From: Rafael Gago Castano on Tuesday, May 09, 2017 2:49 AM
>
> We are a group that is using a LAVA setup, but we are not completely happy
> with it and we are trying alternatives. We like from fuego that it seems to be
> more hackable, it runs the server from a container (hopefully being able to
> develop tests locally) and doesn't try to bring in a lot of Web
> technologies/languages.
Thanks for taking a look at Fuego!  I hope we can answer your questions
and address any issues you have.

> 
> So far I have some questions.
> 
> I have built the .pdf under fuego/docs (fuego-docs.pdf) and I'm trying to
> follow it on Ubuntu 16.04. I'm now blocked at step 4. The jenkins Web
> interface  is not showing anything specific to fuego, I see just a vanilla Jenkins
> server with no tabs, not something like this:
> 
> http://bird.org/fuego-files/fuego-dashboard-history.png

Yeah - the PDF  doc is unfortunately a bit out of date.  We had a major change
in the jenkins interface, and in the method of populating the interface, in our 1.1
release, which was released in March.  I'm sorry this didn't get updated for that
release.  That file on our wiki that you point to refers to the old, pre v1.1 interface.

The best location for information on getting up and running is the wiki.  This page:
http://bird.org/fuego/Fuego_Quickstart_Guide
hopefully has some good information you can use (and up-to-date screenshots).
There are also some screenshots at:
http://bird.org/fuego/Jenkins_User_Interface

> 
> I was able to submit some tests (Initial test section on the README file) by
> using the ftc tool and I saw some of them failing
> (docker.default.Functional.LTP,  docker.default.Functional.glib), but the
> Jenkins interface doesn't let me do anything. I'm not very well versed on
> Jenkins and Docker setups so I might be doing something wrong. What
> should I check?
Actually, docker is a special target for playing around with Fuego by
running tests on the Fuego docker container.  We actually discovered
a few real bugs in docker with Fuego, that we haven't been able to fix
yet.  So the LTP bug you saw is real.  There are currently 3 errors from
LTP that should be passing but are currently failing with the "docker"
board, that are under investigation now.  We may just blacklist those
particular sub-tests, so that they don't cause undue anxiety running Fuego. 

In the future, our plan is to have an 'ftc' sub-command you can run to save
your current results (including test failures that you feel you can ignore),
as the new "reference" results that future tests on your hardware will
compare against to determine success or failure.  There was a tool for this
in the past, but it was an awk script, and had some usability problems, and
we have just recently refactored a lot of our results parsing in conjunction
with a unified results output and processing, across all tests (both
Benchmark and Functional).

Unfortunately, this area of Fuego is undergoing
some churn at the moment.  You can detect errors with Fuego, using
the Jenkins interface, and you can detect functional test regressions
and benchmark results (performance) regressions.  But you can't use
the Jenkins interface to blacklist the failing tests (indicate that you
don't want to run them, or indicate that you want to ignore those
results). And actually investigating the cause of an error is still left
as an exercise to the user (although our links to the test logs should
help).

We are still investigating the docker bug with the Functional.glib test, to
see what the actual problem is.   I hope to have this resolved and/or
explained in Fuego version 1.2. To be honest, the 1.1 release was a bit
rushed, and we probably should have removed Functional.glib from the
testplans for the docker board, to avoid these issues (and then
re-enabled it in the testplans when the issue was resolved.)

You bring up an important point - which is that we should probably have
some release notes that explain known problems, especially with one of
our "demo" boards (the 'docker' pseudo-board).  One thing we are striving
for is a good out-of-the box experience, and it looks like we've failed with
the 1.1 release.

> 
> Then I saw no reference about what in LAVA is called multinode tests (just
> for someone reading this that doesn't know LAVA: a test using more than
> one board involved to be able to test e.g. communication hardware). If this is
> achievable with Fuego, how would be do?

In a way, every test in Fuego is a multi-node test, with the host being one
end of the test.  The way Fuego is structured, the base script for the test
runs on the host, inside the docker container, and communicates with
the board.  For many tests, the base script just: 1) builds the software,
2) deploys it to target, 3) executes it on the board, 4) collects the results
and 5) analyzes it.

However, we do have a few tests that perform other actions on the host,
or rely on processes running on the host.
For example, the Functional.netperf test relies on a netperf server running on
the host (an instance is started in the container if one is not already running).
A test can execute whatever steps it wants to, including communicating
with nodes, machines, or test equipment that is not on the target board.
These steps would be put into the base script's test_pre_check() or test_run()
functions.  test_pre_check is usually reserved to checking that test pre-requisites
are met, which in the case of a multi-node test might include reserving off-target
testing hardware (such as video scanners), or it might include setting up
the connections for things like bus testing (e.g. establishing the connections to the
device under test for CAN bus testing or USB testing).

Then the test_run() function should do things like initiating the actual saving
of results, or starting and stopping communication from off-board endpoints.

See http://bird.org/fuego/Adding_a_test for more information about the
functions in a test base script.

One concept in Fuego related to this, that I think is important, is that
we intend the contents of the Fuego docker container to be a
"test distribution", or in other words, a distribution of Linux, based
on Debian, that is specifically geared for testing.  Not only does Fuego
come with actual test programs, and the test packages themselves
include source (if needed) to build the tests for the target boards.
But also, the Fuego test distribution is intended to include host-side
software as well for testing, as well as (potentially) the software
needed to drive external devices and test equipment.  It is a pain
for the average developer to collect, install, and integrate this
software onto their desktop machines, and that's one aspect of
testing that we'd like to share between developers (test program
setup, hardware and tools configuration, etc.)  You can see the
first steps in this direction by the inclusion in Fuego of the netperf
server, and several utilities (like serio and ttc) to handle different
board communication methods.  We'll be expanding this in the
years to come, so that more and more tests can be run with just
the single docker installation of Fuego, requiring much less pre-test
configuration and specialized expertise by testers.

I hope this answers your questions.  Thanks again for
experimenting with Fuego and providing feedback.
 -- Tim