[Fuego] RFC - test/set/case documentation generation

[Fuego] RFC - test/set/case documentation generation

[Tim.Bird]

Hello all,

I'm writing to let you know about a project that I've started, having to do with
documenting tests, testsets, and test cases, as well as feeding into capabilities
for report generation.

It's a bit hard to describe the overall scope of the project, but it's quite big.
I want to document every testcase that Fuego performs.  This include a mixture
of human-written documentation, and reports of results from tests performed
on both local boards and eventually remote boards.

My plan is to add a 'docs' directory under each test directory.  Inside that
there will be template files, which describe tests, test sets, and individual test
cases.  My plan is that the fuego system can combine test result data, with
human-written text, in both HTML format (for use via the Jenkins intereface),
and PDF format (for use in PDFs that are generated as test reports).
The templates are written in reStructuredText, with macro directives
to indicate what dynamic data to add to the templates.

The template is processed by Fuego, and is output as an .rst file, which can
be used as is as a straight text file, or further processed by sphynx to convert
into either HTML (for access via the Jenkins interface) or latex (for publication
in PDF test reports).

It will likely take years to fill in all the testcases, but I expect that the list of
documented testcases will likely always be sparse.  Only the most important
testcases will be documented at first, and they can be filled in as we go along.
Having some testcases documented will be more useful than having no
testcases documented.

The first instance of this is not quite ready for usage yet, but you can get an idea
of what I have planned by looking at:
/fuego-core/engine/tests/Functional.LTP/docs/Functional.LTP.ftmp
('ftmp' stands for 'Fuego template')
This is processed by Fuego (by a program called gen-page.py) and
the output is placed in:
/fuego-rw/docs/Functional.LTP.rst
The intent is to further process this into
/fuego-rw/logs/Fucntional.LTP/Functional.LTP.html
and make that accessible via the Jenkins interface.

This is in my 'master' branch, at commit 0eba4e6

Note that the information in Functional.LTP.ftmp is a little out-of-date, and
the plan is to gather more information automatically (such as dependencies,
spec list, etc.) over time.

My idea is that using the same mechanisms, we will be able to generate reports
from test data, that are a mixture of testcase results, summary data, detailed
error reports, and human-written explanations of test operation and intent.

One problem this is intended to solve has to do with report generation.  But
another problem is that many of our tests are very opaque.  There's no
human-readable documentation on the operation of the tests, what the results
are supposed to mean, and what to do to resolve reported failures.  Creating a
documentation system like this will allow such human-generated text to be
presented alongside results, hopefully making results interpretation much easier.

Work on this is progressing slowly, due to other demands on my time.
But I wanted to present the idea to allow people to give feedback on it.

Please let me know what you think.

Regards,
 -- Tim

Re: [Fuego] RFC - test/set/case documentation generation

[Gustavo Sverzut Barbieri]

On Thu, Feb 8, 2018 at 4:28 AM, <Tim.Bird@sony.com> wrote:
>
> Hello all,
>
> I'm writing to let you know about a project that I've started, having to do with
> documenting tests, testsets, and test cases, as well as feeding into capabilities
> for report generation.
>
> It's a bit hard to describe the overall scope of the project, but it's quite big.
> I want to document every testcase that Fuego performs.  This include a mixture
> of human-written documentation, and reports of results from tests performed
> on both local boards and eventually remote boards.
>
> My plan is to add a 'docs' directory under each test directory.  Inside that
> there will be template files, which describe tests, test sets, and individual test
> cases.  My plan is that the fuego system can combine test result data, with
> human-written text, in both HTML format (for use via the Jenkins intereface),
> and PDF format (for use in PDFs that are generated as test reports).
> The templates are written in reStructuredText, with macro directives
> to indicate what dynamic data to add to the templates.
>
> The template is processed by Fuego, and is output as an .rst file, which can
> be used as is as a straight text file, or further processed by sphynx to convert
> into either HTML (for access via the Jenkins interface) or latex (for publication
> in PDF test reports).
>
> It will likely take years to fill in all the testcases, but I expect that the list of
> documented testcases will likely always be sparse.  Only the most important
> testcases will be documented at first, and they can be filled in as we go along.
> Having some testcases documented will be more useful than having no
> testcases documented.
>
> The first instance of this is not quite ready for usage yet, but you can get an idea
> of what I have planned by looking at:
> /fuego-core/engine/tests/Functional.LTP/docs/Functional.LTP.ftmp
> ('ftmp' stands for 'Fuego template')
> This is processed by Fuego (by a program called gen-page.py) and
> the output is placed in:
> /fuego-rw/docs/Functional.LTP.rst
> The intent is to further process this into
> /fuego-rw/logs/Fucntional.LTP/Functional.LTP.html
> and make that accessible via the Jenkins interface.
>
> This is in my 'master' branch, at commit 0eba4e6
>
> Note that the information in Functional.LTP.ftmp is a little out-of-date, and
> the plan is to gather more information automatically (such as dependencies,
> spec list, etc.) over time.
>
> My idea is that using the same mechanisms, we will be able to generate reports
> from test data, that are a mixture of testcase results, summary data, detailed
> error reports, and human-written explanations of test operation and intent.
>
> One problem this is intended to solve has to do with report generation.  But
> another problem is that many of our tests are very opaque.  There's no
> human-readable documentation on the operation of the tests, what the results
> are supposed to mean, and what to do to resolve reported failures.  Creating a
> documentation system like this will allow such human-generated text to be
> presented alongside results, hopefully making results interpretation much easier.
>
> Work on this is progressing slowly, due to other demands on my time.
> But I wanted to present the idea to allow people to give feedback on it.
>
> Please let me know what you think.

Hi Tim, looks nice but my advice is to avoid pre-processing the rst
(your ftmp format) and instead use reStructuredText and add your own
directives, it's already meant for that.

I'd even go further and say that's better to setup an sphinx for the
project and create your own extension, then you can provide those as
needed AND scan for files in the directory structure. For example
Python's autodoc and the does something like that for python modules.

Initially you can avoid much of the burden by:

 - start sphinx project in the root directory
 - manually link the child .rst until you create your own autoindex
kind of function
 - in the child .rst you .. literalinclude:: results from some
directory until you create your own directives or roles

with that, no gen-pages is needed, just run the tests (to populate the
directories with results you're including as "literalinclude") and it
should produce the contents.

then you can create your sphinx extension (inside this repository,
just list it in the sphinx config) and start to provide the directives
and macros... they can do lots of stuff, like generate tables and
other sphinx formatting (ie: there are modules that does syntax
highlight using pygments). You can receive parameters in them, and you
can get the file/path of the file being processed, so you can discover
the test name and all.


-- 
Gustavo Sverzut Barbieri
http://profusion.mobi embedded systems
--------------------------------------
MSN, GTalk, FaceTime: barbieri@gmail.com
Skype: gsbarbieri
Mobile: +55 (16) 99354-9890

Re: [Fuego] RFC - test/set/case documentation generation

[Tim.Bird]

Response at the bottom...

> -----Original Message-----
> From: Gustavo Sverzut Barbieri 
> On Thu, Feb 8, 2018 at 4:28 AM, <Tim.Bird@sony.com> wrote:
> >
> > Hello all,
> >
> > I'm writing to let you know about a project that I've started, having to do
> with
> > documenting tests, testsets, and test cases, as well as feeding into
> capabilities
> > for report generation.
> >
> > It's a bit hard to describe the overall scope of the project, but it's quite big.
> > I want to document every testcase that Fuego performs.  This include a
> mixture
> > of human-written documentation, and reports of results from tests
> performed
> > on both local boards and eventually remote boards.
> >
> > My plan is to add a 'docs' directory under each test directory.  Inside that
> > there will be template files, which describe tests, test sets, and individual
> test
> > cases.  My plan is that the fuego system can combine test result data, with
> > human-written text, in both HTML format (for use via the Jenkins
> intereface),
> > and PDF format (for use in PDFs that are generated as test reports).
> > The templates are written in reStructuredText, with macro directives
> > to indicate what dynamic data to add to the templates.
> >
> > The template is processed by Fuego, and is output as an .rst file, which can
> > be used as is as a straight text file, or further processed by sphynx to
> convert
> > into either HTML (for access via the Jenkins interface) or latex (for
> publication
> > in PDF test reports).
> >
> > It will likely take years to fill in all the testcases, but I expect that the list of
> > documented testcases will likely always be sparse.  Only the most
> important
> > testcases will be documented at first, and they can be filled in as we go
> along.
> > Having some testcases documented will be more useful than having no
> > testcases documented.
> >
> > The first instance of this is not quite ready for usage yet, but you can get an
> idea
> > of what I have planned by looking at:
> > /fuego-core/engine/tests/Functional.LTP/docs/Functional.LTP.ftmp
> > ('ftmp' stands for 'Fuego template')
> > This is processed by Fuego (by a program called gen-page.py) and
> > the output is placed in:
> > /fuego-rw/docs/Functional.LTP.rst
> > The intent is to further process this into
> > /fuego-rw/logs/Fucntional.LTP/Functional.LTP.html
> > and make that accessible via the Jenkins interface.
> >
> > This is in my 'master' branch, at commit 0eba4e6
> >
> > Note that the information in Functional.LTP.ftmp is a little out-of-date, and
> > the plan is to gather more information automatically (such as
> dependencies,
> > spec list, etc.) over time.
> >
> > My idea is that using the same mechanisms, we will be able to generate
> reports
> > from test data, that are a mixture of testcase results, summary data,
> detailed
> > error reports, and human-written explanations of test operation and
> intent.
> >
> > One problem this is intended to solve has to do with report generation.
> But
> > another problem is that many of our tests are very opaque.  There's no
> > human-readable documentation on the operation of the tests, what the
> results
> > are supposed to mean, and what to do to resolve reported failures.
> Creating a
> > documentation system like this will allow such human-generated text to be
> > presented alongside results, hopefully making results interpretation much
> easier.
> >
> > Work on this is progressing slowly, due to other demands on my time.
> > But I wanted to present the idea to allow people to give feedback on it.
> >
> > Please let me know what you think.
> 
> Hi Tim, looks nice but my advice is to avoid pre-processing the rst
> (your ftmp format) and instead use reStructuredText and add your own
> directives, it's already meant for that.

You may notice that my "macro" actually has the same syntax as
an reStructuredText directive.  This was done specifically to allow
the possibility of converting over to the flow of processing that
you mention below.

As an aside, what does Sphynx do with directives it doesn't recognize?
(Sorry - I'm too lazy to go test is right now, but if someone knows
off the top of their head please let me know.)

> 
> I'd even go further and say that's better to setup an sphinx for the
> project and create your own extension, then you can provide those as
> needed AND scan for files in the directory structure. For example
> Python's autodoc and the does something like that for python modules.
> 
> Initially you can avoid much of the burden by:
> 
>  - start sphinx project in the root directory
>  - manually link the child .rst until you create your own autoindex
> kind of function
>  - in the child .rst you .. literalinclude:: results from some
> directory until you create your own directives or roles
> 
> with that, no gen-pages is needed, just run the tests (to populate the
> directories with results you're including as "literalinclude") and it
> should produce the contents.
> 
> then you can create your sphinx extension (inside this repository,
> just list it in the sphinx config) and start to provide the directives
> and macros... they can do lots of stuff, like generate tables and
> other sphinx formatting (ie: there are modules that does syntax
> highlight using pygments). You can receive parameters in them, and you
> can get the file/path of the file being processed, so you can discover
> the test name and all.

Thanks very much for the feedback.  I am by no means sure that the
approach I have taken is optimal, so I really appreciate other ideas
for implementing this.

With regard to sphynx, I definitely want to make it a first-class tool
in our container, and possibly add fuego-specific extensions (directives) to it.
I took a look yesterday at how to write extensions for it, and they
look powerful, and straightforward to write.  I've already started moving
the regular Fuego documentation over to reStructuredText, but I don't
recall if I've made the changes to include sphynx in the Dockerfile yet.

I'll think more about your suggestion as I keep working on this.
 -- Tim

Re: [Fuego] RFC - test/set/case documentation generation

[Gustavo Sverzut Barbieri]

>> > Please let me know what you think.
>>
>> Hi Tim, looks nice but my advice is to avoid pre-processing the rst
>> (your ftmp format) and instead use reStructuredText and add your own
>> directives, it's already meant for that.
>
> You may notice that my "macro" actually has the same syntax as
> an reStructuredText directive.  This was done specifically to allow
> the possibility of converting over to the flow of processing that
> you mention below.

sure, but then keep the same extension (so github/bitucket...) handle
it properly. And also note the final "::" (two), otherwise it's
handled as comment (leading ".." without trailing "::")



> As an aside, what does Sphynx do with directives it doesn't recognize?
> (Sorry - I'm too lazy to go test is right now, but if someone knows
> off the top of their head please let me know.)

not sure, I never tried that... but I guess it will report errors and
keep going. Github/Bitbucket keep it untouched (you see the actual
text).


>> I'd even go further and say that's better to setup an sphinx for the
>> project and create your own extension, then you can provide those as
>> needed AND scan for files in the directory structure. For example
>> Python's autodoc and the does something like that for python modules.
>>
>> Initially you can avoid much of the burden by:
>>
>>  - start sphinx project in the root directory
>>  - manually link the child .rst until you create your own autoindex
>> kind of function
>>  - in the child .rst you .. literalinclude:: results from some
>> directory until you create your own directives or roles
>>
>> with that, no gen-pages is needed, just run the tests (to populate the
>> directories with results you're including as "literalinclude") and it
>> should produce the contents.
>>
>> then you can create your sphinx extension (inside this repository,
>> just list it in the sphinx config) and start to provide the directives
>> and macros... they can do lots of stuff, like generate tables and
>> other sphinx formatting (ie: there are modules that does syntax
>> highlight using pygments). You can receive parameters in them, and you
>> can get the file/path of the file being processed, so you can discover
>> the test name and all.
>
> Thanks very much for the feedback.  I am by no means sure that the
> approach I have taken is optimal, so I really appreciate other ideas
> for implementing this.
>
> With regard to sphynx, I definitely want to make it a first-class tool
> in our container, and possibly add fuego-specific extensions (directives) to it.
> I took a look yesterday at how to write extensions for it, and they
> look powerful, and straightforward to write.  I've already started moving
> the regular Fuego documentation over to reStructuredText, but I don't
> recall if I've made the changes to include sphynx in the Dockerfile yet.
>
> I'll think more about your suggestion as I keep working on this.

you're welcome. It's a bit steep learning curve to get started
(compared to plain python + readlines + regexp), but at the end it
pays off. With some days looking into that, you can get it going.
There's not much documentation, but you can look the source of similar
extensions to see how things work (that's how i learnt).

you get called with a list of options + body... after that, just
return the rst "ast"... you can either create on manually, using
elements such as "paragraph" or generate a text string and ask it to
be parsed (ie: when you do a quote block: generate a new toplevel
element yourself (blockquote) then ask it to parse the contents), this
is self.state.nested_parse() from your directive...

then everything else is handled for you, output to LaTeX/PDF, HTML,
manpage, etc...


-- 
Gustavo Sverzut Barbieri
http://profusion.mobi embedded systems
--------------------------------------
MSN, GTalk, FaceTime: barbieri@gmail.com
Skype: gsbarbieri
Mobile: +55 (16) 99354-9890

Re: [Fuego] RFC - test/set/case documentation generation

[Tim.Bird]

> -----Original Message-----
> From: Gustavo Sverzut Barbieri 
> > You may notice that my "macro" actually has the same syntax as
> > an reStructuredText directive.  This was done specifically to allow
> > the possibility of converting over to the flow of processing that
> > you mention below.
> 
> sure, but then keep the same extension (so github/bitucket...) handle
> it properly. And also note the final "::" (two), otherwise it's
> handled as comment (leading ".." without trailing "::")

I saw your comment on the commit.  Thanks for catching that.
I just fixed it.

Thanks!
 -- Tim