[Fuego] RFC - test/set/case documentation generation

[<Tim.Bird@sony.com>]

Hello all,

I'm writing to let you know about a project that I've started, having to do with
documenting tests, testsets, and test cases, as well as feeding into capabilities
for report generation.

It's a bit hard to describe the overall scope of the project, but it's quite big.
I want to document every testcase that Fuego performs.  This include a mixture
of human-written documentation, and reports of results from tests performed
on both local boards and eventually remote boards.

My plan is to add a 'docs' directory under each test directory.  Inside that
there will be template files, which describe tests, test sets, and individual test
cases.  My plan is that the fuego system can combine test result data, with
human-written text, in both HTML format (for use via the Jenkins intereface),
and PDF format (for use in PDFs that are generated as test reports).
The templates are written in reStructuredText, with macro directives
to indicate what dynamic data to add to the templates.

The template is processed by Fuego, and is output as an .rst file, which can
be used as is as a straight text file, or further processed by sphynx to convert
into either HTML (for access via the Jenkins interface) or latex (for publication
in PDF test reports).

It will likely take years to fill in all the testcases, but I expect that the list of
documented testcases will likely always be sparse.  Only the most important
testcases will be documented at first, and they can be filled in as we go along.
Having some testcases documented will be more useful than having no
testcases documented.

The first instance of this is not quite ready for usage yet, but you can get an idea
of what I have planned by looking at:
/fuego-core/engine/tests/Functional.LTP/docs/Functional.LTP.ftmp
('ftmp' stands for 'Fuego template')
This is processed by Fuego (by a program called gen-page.py) and
the output is placed in:
/fuego-rw/docs/Functional.LTP.rst
The intent is to further process this into
/fuego-rw/logs/Fucntional.LTP/Functional.LTP.html
and make that accessible via the Jenkins interface.

This is in my 'master' branch, at commit 0eba4e6

Note that the information in Functional.LTP.ftmp is a little out-of-date, and
the plan is to gather more information automatically (such as dependencies,
spec list, etc.) over time.

My idea is that using the same mechanisms, we will be able to generate reports
from test data, that are a mixture of testcase results, summary data, detailed
error reports, and human-written explanations of test operation and intent.

One problem this is intended to solve has to do with report generation.  But
another problem is that many of our tests are very opaque.  There's no
human-readable documentation on the operation of the tests, what the results
are supposed to mean, and what to do to resolve reported failures.  Creating a
documentation system like this will allow such human-generated text to be
presented alongside results, hopefully making results interpretation much easier.

Work on this is progressing slowly, due to other demands on my time.
But I wanted to present the idea to allow people to give feedback on it.

Please let me know what you think.

Regards,
 -- Tim