Tests documentation

* Tests documentation
@ 2023-07-02  7:23 Mauro Carvalho Chehab
  2023-07-04  9:45 ` David Gow
  0 siblings, 1 reply; 3+ messages in thread
From: Mauro Carvalho Chehab @ 2023-07-02  7:23 UTC (permalink / raw)
  To: Shuah Khan, Jonathan Corbet
  Cc: Nikolai Kondrashov, linux-kselftest, kunit-dev, mauro.chehab,
	linux-doc, Brendan Higgins, David Gow

Hi Jon, Shuah & others,

I'd like to discuss with you with regards to test documentation.

I had some preliminary discussions with people interested on improving 
tests during EOSS last week in Prague, as we're working to improve media
test coverage as well. During such discussions, I talked with developers
from several companies that have been collaboration and/or using Kernel
CI. I also talked with Nikolai from Red Hat, who gave a presentation about
Kernel CI, which points that one of the areas to be improved there is
documentation.

So, it seems it is worth having some discussions about how to improve 
Kernel test documentation.

While kernel_doc does a pretty decent job documenting functions and data
structures, for tests, the most important things to be documented are:

	a. what the tests do;
	b. what functionalities they are testing.

This is a lot more important than documenting functions - and the used
data structures on tests are typically the ones that are part of the
driver's kAPI or uAPI, so they should be documented somewhere else.

Usually, (b) is not so simple, as, at least for complex hardware,
the tested features are grouped on an hierarchical way, like:

	1. hardware
	1.1 DMA engine
	1.2 output ports
	...
	2. firmware
	2.1 firmware load
	2.2 firmware DMA actions
	...
	3. kernel features
	3.1 memory allocation
	3.2 mmap
	3.3 bind/unbind
	...

CI engines running the test sets usually want to produce a report that will
be providing pass rates for the tested features and functionalites that
are available at the driver's and their respective hardware and firmware.

I've doing some work at the tool we use to test DRM code [1] in order to 
have a decent documentation of the tests we have hosted there, focusing 
mostly on tests for i915 and Xe Intel drivers, also covering documentation
for DRM core tests - while providing support for other vendors to also
improve their test documentation for IGT - IGT GPU tools and tests.

The documentation tool I developed is generic enough to be used for other
test sets and I believe it could be useful as well to document Kselftest 
and KUnit.

The core of the tool (at test_list.py) is a Python class, with some callers
(igt_doc.py, xls_to_doc.py, doc_to_xls.py), being extensible enough to
also have other callers to integrate with external tools. We are 
developing internally one to integrate with our internal Grafana reports
to report the pass rate per documented feature, in an hierarchical way.

Something similar to:

	1. hardware        pass rate:  98% (98 tests passed of 100)
	1.1 DMA engine     pass rate:  80% (8  tests passed of  10)
	1.2 output ports   pass rate: 100% (10 tests passed of  10)
	...

It is based on the concept that test documentation should be placed as
close as possible to the actual code implementing the test sets. It was
also be developed in a way that the documentation grouping is flexible.
The code was written from the scratch in Python and was implemented 
inside a class that can also be re-used to do do other nice things,
like importing/exporting test documentation to spreadsheets and
integration with other tools (like Grafana).

The actual documentation tags look like this:

        /**
         * TEST: Check if new IGT test documentation logic functionality is working
         * Category: Software build block
         * Sub-category: documentation
         * Functionality: test documentation
         * Issue: none
         * Description: Complete description of this test
         *
         * SUBTEST: foo
         * Description: do foo things
         * 	description continuing on another line
         *
         * SUBTEST: bar
         * Description: do bar things
         * 	description continuing on another line
         * Functionality: bar test doc
         */

And it has support for wildcards.

There, "TEST" is associated to the contents of the file, while "SUBTEST"
refers to each specific subtest inside it. The valid fields are imported
from JSON config files, and can be placed into an hierarchical way, in
order to produce an hierarchical documentation. Fields defined at the
"TEST" level are imported on "SUBTEST", but can be overriden. 

The JSON config file looks like this:
	https://gitlab.freedesktop.org/drm/igt-gpu-tools/-/blob/158feaa20fa2b9424ee544efd2e0e0892562f8f0/tests/xe/xe_test_config.json

The output is in ReST, which can be generated in hierarchical or per-file
way. The hierarchical output looks like this:

	$ ./scripts/igt_doc.py --config tests/xe/xe_test_config.json --file fubar_tests.c

	===============================
	Implemented Tests for Xe Driver
	===============================

	Category: Software build block
	==============================

	Sub-category: documentation
	---------------------------

	Functionality: bar test doc
	^^^^^^^^^^^^^^^^^^^^^^^^^^^

	``igt@fubar_tests@bar``

	:Description: do bar things description continuing on another line
	:Issue: none

	Functionality: test documentation
	^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

	``igt@fubar_tests@foo``

	:Description: do foo things description continuing on another line
	:Issue: none

(if --file is not used, it will use all C files specified at the
configuration)

The tool already skips tags like the ones used by kernel-doc[1], so one
could have both function documentation and per-test documentation on
the same file, if needed.

While such tool was conceived to be part of IGT, it doesn't have anything
specific for it [2], and I do believe it would be a great contribution to
the Kernel to have such tool upstreamed, and integrated as a Sphinx
extension.

If we decide to go ahead adding it, I can work on a patchset to apply
it to the Kernel, modifying the scripts to better fit at the Kernel
needs and start with some documentation examples for i915,
DRM core and upcoming Xe KUnit tests.

Comments?

Regards,
Mauro

[1] It should be trivial to patch kernel-doc for it to skip TEST and 
    SUBTEST tags if we decide to integrate it to the kernel.

[2] except that tests there are named after IGT, as
    <igt <test>@<subtest>@<dynamic_subtest>, but a change to a 
    Kernel-specific namespace would be trivial

^ permalink raw reply	[flat|nested] 3+ messages in thread