Re: [PATCH] Documentation: kunit: add tips for running KUnit

[David Gow <davidgow@google.com>]

Thanks for writing this: it's good to have these things documented at last!

There are definitely a few things this document points out which still
need deciding, which does make this document lean a bit into "design
discussion" territory in a few of the notes. This doesn't bother me --
it's an accurate description of the state of things -- but I wouldn't
want this documentation held up too long because of these sorts of
TODOs (and can definitely see how having too many of them might
discourage KUnit use a bit). Particularly things like the
".kunitconfig" fragment file feature stuff: I feel that's something
better discussed on patches adding/using the feature than in the
documentation / reviews of the documentation, so I'd rather drop or
simplify those '..note:'s than bokeshed about it here (something I'm a
little guilty of below).

Otherwise, a few minor comments and nitpicks:

-- David

On Sat, Apr 10, 2021 at 2:01 AM Daniel Latypov <dlatypov@google.com> wrote:
>
> This is long overdue.
>
> There are several things that aren't nailed down (in-tree
> .kunitconfig's), or partially broken (GCOV on UML), but having them
> documented, warts and all, is better than having nothing.
>
> This covers a bunch of the more recent features
> * kunit_filter_glob
> * kunit.py run --kunitconfig
> * kunit.py run --alltests
> * slightly more detail on building tests as modules
> * CONFIG_KUNIT_DEBUGFS
>
> By my count, the only headline features now not mentioned are the KASAN
> integration and KernelCI json output support (kunit.py run --json).
>
> And then it also discusses how to get code coverage reports under UML
> and non-UML since this is a question people have repeatedly asked.
>
> Non-UML coverage collection is no differnt from normal, but we should
> probably explicitly call thsi out.

Nit: typos in 'different' and 'this'.

>
> As for UML, I was able to get it working again with two small hacks.*
> E.g. with CONFIG_KUNIT=y && CONFIG_KUNIT_ALL_TESTS=y
>   Overall coverage rate:
>     lines......: 15.1% (18294 of 120776 lines)
>     functions..: 16.8% (1860 of 11050 functions)
>
> *Switching to use gcc/gcov-6 and not using uml_abort().
> I've documented these hacks in "Notes" but left TODOs for
> brendanhiggins@google.com who tracked down the runtime issue in GCC.
> To be clear: these are not issues specific to KUnit, but rather to UML.

(We should probably note where uml_abort() needs to be replaced if
we're mentioning this, though doing so below in the more detailed
section may be more useful.)

>
> Signed-off-by: Daniel Latypov <dlatypov@google.com>
> ---
>  Documentation/dev-tools/kunit/index.rst       |   1 +
>  .../dev-tools/kunit/running_tips.rst          | 278 ++++++++++++++++++
>  Documentation/dev-tools/kunit/start.rst       |   2 +
>  3 files changed, 281 insertions(+)
>  create mode 100644 Documentation/dev-tools/kunit/running_tips.rst
>
> diff --git a/Documentation/dev-tools/kunit/index.rst b/Documentation/dev-tools/kunit/index.rst
> index 848478838347..7f7cf8d2ab20 100644
> --- a/Documentation/dev-tools/kunit/index.rst
> +++ b/Documentation/dev-tools/kunit/index.rst
> @@ -14,6 +14,7 @@ KUnit - Unit Testing for the Linux Kernel
>         style
>         faq
>         tips
> +       running_tips
>
>  What is KUnit?
>  ==============
> diff --git a/Documentation/dev-tools/kunit/running_tips.rst b/Documentation/dev-tools/kunit/running_tips.rst
> new file mode 100644
> index 000000000000..d38e665e530f
> --- /dev/null
> +++ b/Documentation/dev-tools/kunit/running_tips.rst
> @@ -0,0 +1,278 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +============================
> +Tips For Running KUnit Tests
> +============================
> +
> +Using ``kunit.py run`` ("kunit tool")
> +=====================================
> +
> +Running from any directory
> +--------------------------
> +
> +It can be handy to create a bash function like:
> +
> +.. code-block:: bash
> +
> +       function run_kunit() {
> +         ( cd "$(git rev-parse --show-toplevel)" && ./tools/testing/kunit/kunit.py run $@ )
> +       }
> +
> +.. note::
> +       Early versions of ``kunit.py`` (before 5.6) didn't work unless run from
> +       the kernel root, hence the use of a subshell and ``cd``.
> +
> +Running a subset of tests
> +-------------------------
> +
> +``kunit.py run`` accepts an optional glob argument to filter tests. Currently
> +this only matches against suite names, but this may change in the future.
> +
> +Say that we wanted to run the sysctl tests, we could do so via:
> +
> +.. code-block:: bash
> +
> +       $ echo -e 'CONFIG_KUNIT=y\nCONFIG_KUNIT_ALL_TESTS=y' > .kunit/.kunitconfig
> +       $ ./tools/testing/kunit/kunit.py run 'sysctl*'
> +
> +We're paying the cost of building more tests than we need this way, but it's
> +easier than fiddling with ``.kunitconfig`` files or commenting out
> +``kunit_suite``'s.
> +
> +However, if we wanted to define a set of tests in a less ad hoc way, the next
> +tip is useful.
> +
> +Defining a set of tests
> +-----------------------
> +
> +``kunit.py run`` (along with ``build``, and ``config``) supports a
> +``--kunitconfig`` flag. So if you have a set of tests that you want to run on a
> +regular basis (especially if they have other dependencies), you can create a
> +specific ``.kunitconfig`` for them.
> +
> +E.g. kunit has own for its tests:

Nit: 'one' for its tests (or 'its own' for its tests?)

> +
> +.. code-block:: bash
> +
> +       $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit/.kunitconfig
> +
> +Alternatively, if you're following the convention of naming your
> +file ``.kunitconfig``, you can just pass in the dir, e.g.
> +
> +.. code-block:: bash
> +
> +       $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit
> +
> +.. note::
> +       This is a relatively new feature (5.12+) so we don't have any
> +       conventions yet about on what files should be checked in versus just
> +       kept around locally. But if the tests don't have any dependencies
> +       (beyond ``CONFIG_KUNIT``), it's probably not worth writing and
> +       maintaining a ``.kunitconfig`` fragment.  Running with
> +       ``CONFIG_KUNIT_ALL_TESTS=y`` is probably easier.

I think the rule of thumb for checked-in .kunitconfig files should be
an explicit endorsement by the maintainer that these are the tests for
a particular subsystem.

> +
> +.. note::
> +       Having ``.kunitconfig`` fragments in a parent and child directory is
> +       iffy. There's discussion about adding an "import" statement in these
> +       files to make it possible to have a top-level config run tests from all
> +       child directories. But that would mean ``.kunitconfig`` files are no
> +       longer just simple .config fragments.
> +
> +       One alternative would be to have kunit tool recursively combine configs
> +       automagically, but tests could theoretically depend on incompatible
> +       options, so handling that would be tricky.
> +
> +Running with ``allyesconfig``
> +-----------------------------
> +
> +.. code-block:: bash
> +
> +       $ ./tools/testing/kunit/kunit.py run --alltests
> +
> +This will try and use ``allyesconfig``, or rather ``allyesconfig`` with a list
Excessively pedantic nit: 'try to use'
> +of UML-incompatible configs turned off. That list is maintained in
> +``tools/testing/kunit/configs/broken_on_uml.config``.
> +
> +.. note::
> +       This will take a *lot* longer to run and might be broken from time to
> +       time, especially on -next. It's not recommended to use this unless you
> +       need to or are morbidly curious.

Given that it's been the plan to run this for KernelCI, I'm not sure
we should discourage it in general to quite this
extent. I think it is broken at the moment, though, so that's
nevertheless worth noting.

> +
> +Generating code coverage reports under UML
> +------------------------------------------
> +
> +.. note::
> +       TODO(brendanhiggins@google.com): There are various issues with UML and
> +       versions of gcc 7 and up. You're likely to run into missing ``.gcda``
> +       files or compile errors. We know one `faulty GCC commit
> +       <https://github.com/gcc-mirror/gcc/commit/8c9434c2f9358b8b8bad2c1990edf10a21645f9d>`_
> +       but not how we'd go about getting this fixed. The compile errors still
> +       need some investigation.
> +
> +.. note::
> +       TODO(brendanhiggins@google.com): for recent versions of Linux
> +       (5.10-5.12, maybe earlier), there's a bug with gcov counters not being
> +       flushed in UML. This translates to very low (<1%) reported coverage. This is
> +       related to the above issue and can be worked around by replacing the
> +       one call to ``uml_abort()`` with a plain ``exit()``.

Can we be more specific than 'the one call' here? I know there is only
one call, but maybe noting that it's in arch/um/os-Linux/util.c will
make this clearer.

> +
> +
> +This is different from the "normal" way of getting coverage information that is
> +documented in Documentation/dev-tools/gcov.rst.
> +
> +Instead of enabling ``CONFIG_GCOV_KERNEL=y``, we can set these options:
> +
> +.. code-block:: none
> +
> +       CONFIG_DEBUG_KERNEL=y
> +       CONFIG_DEBUG_INFO=y
> +       CONFIG_GCOV=y
> +
> +
> +Putting it together into a copy-pastable sequence of commands:
> +
> +.. code-block:: bash
> +
> +       # Append coverage options to the current config
> +       $ echo -e "CONFIG_DEBUG_KERNEL=y\nCONFIG_DEBUG_INFO=y\nCONFIG_GCOV=y" >> .kunit/.kunitconfig
> +       $ ./tools/testing/kunit/kunit.py run
> +       # Extract the coverage information from the build dir (.kunit/)
> +       $ lcov -t "my_kunit_tests" -o coverage.info -c -d .kunit/
> +
> +       # From here on, it's the same process as with CONFIG_GCOV_KERNEL=y
> +       # E.g. can generate an HTML report in a tmp dir like so:
> +       $ genhtml -o /tmp/coverage_html coverage.info
> +
> +
> +If your installed version of gcc doesn't work, you can tweak the steps:
> +
> +.. code-block:: bash
> +
> +       # need to edit tools/testing/kunit/kunit_kernel.py to call make with 'CC=/usr/bin/gcc-6'
> +       $ $EDITOR tools/testing/kunit/kunit_kernel.py
> +
> +       $ lcov -t "my_kunit_tests" -o coverage.info -c -d .kunit/ --gcov-tool=/usr/bin/gcov-6
> +
> +
> +Running tests manually
> +======================
> +
> +Running tests without using ``kunit.py run`` is also an important use case.
> +Currently it's your only option if you want to test on architectures other than
> +UML.
> +
> +As running the tests under UML is fairly straightforward (configure and compile
> +the kernel, run the ``./linux`` binary), this section will focus on testing
> +non-UML architectures.
> +
> +
> +Running built-in tests
> +----------------------
> +
> +When setting tests to ``=y``, the tests will run as part of boot and print
> +results to dmesg in TAP format. So you just need to add your tests to your
> +``.config``, build and boot your kernel as normal.
> +
> +So if we compiled our kernel with:
> +
> +.. code-block:: none
> +
> +       CONFIG_KUNIT=y
> +       CONFIG_KUNIT_EXAMPLE_TEST=y
> +
> +Then we'd see output like this in dmesg signaling the test ran and passed:
> +
> +.. code-block:: none
> +
> +       TAP version 14
> +       1..1
> +           # Subtest: example
> +           1..1
> +           # example_simple_test: initializing
> +           ok 1 - example_simple_test
> +       ok 1 - example
> +
> +Running tests as modules
> +------------------------
> +
> +Depending on the tests, you can build them as loadable modules.
> +
> +For example, we'd change the config options from before to
> +
> +.. code-block:: none
> +
> +       CONFIG_KUNIT=y
> +       CONFIG_KUNIT_EXAMPLE_TEST=m
> +
> +Then after booting into our kernel, we can run the test via
> +
> +.. code-block:: none
> +
> +       $ modprobe kunit-example-test
> +
> +This will then cause it to print TAP output to stdout.
> +
> +.. note::
> +       The ``modprobe`` will *not* have a non-zero exit code if any test
> +       failed (as of 5.13). But ``kunit.py parse`` would, see below.
> +
> +.. note::
> +       You can set ``CONFIG_KUNIT=m`` as well, however, some features will not
> +       work and thus some tests might break. Ideally tests would specify they
> +       depend on ``KUNIT=y`` in their ``Kconfig``'s, but this is an edge case
> +       most test authors won't think about.
> +       As of 5.13, the only difference is that ``current->kunit_test`` will
> +       not exist.
> +
> +Pretty-printing results
> +-----------------------
> +
> +You can use ``kunit.py parse`` to parse dmesg for test output and print out
> +results in the same familiar format that ``kunit.py run`` does.

This also should work for the debugfs files below, so maybe reword
this to either mention that or not explicitly mention dmesg above.

> +
> +.. code-block:: bash
> +
> +       $ ./tools/testing/kunit/kunit.py parse /var/log/dmesg
> +
> +
> +Retrieving per suite results
> +----------------------------
> +
> +Regardless of how you're running your tests, you can enable
> +``CONFIG_KUNIT_DEBUGFS`` to expose per-suite TAP-formatted results:
> +
> +.. code-block:: none
> +
> +       CONFIG_KUNIT=y
> +       CONFIG_KUNIT_EXAMPLE_TEST=m
> +       CONFIG_KUNIT_DEBUGFS=y
> +
> +The results for each suite will be exposed under
> +``/sys/kernel/debug/kunit/<suite>/results``.
> +So using our example config:
> +
> +.. code-block:: bash
> +
> +       $ modprobe kunit-example-test > /dev/null
> +       $ cat /sys/kernel/debug/kunit/example/results
> +       ... <TAP output> ...
> +
> +       # After removing the module, the corresponding files will go away
> +       $ modprobe -r kunit-example-test
> +       $ cat /sys/kernel/debug/kunit/example/results
> +       /sys/kernel/debug/kunit/example/results: No such file or directory
> +
> +Generating code coverage reports
> +--------------------------------
> +
> +See Documentation/dev-tools/gcov.rst for details on how to do this.
> +
> +The only vaguely KUnit-specific advice here is that you probably want to build
> +your tests as modules. That way you can isolate the coverage from tests from
> +other code executed during boot, e.g.
> +
> +.. code-block:: bash
> +
> +       # Reset coverage counters before running the test.
> +       $ echo 0 > /sys/kernel/debug/gcov/reset
> +       $ modprobe kunit-example-test
> diff --git a/Documentation/dev-tools/kunit/start.rst b/Documentation/dev-tools/kunit/start.rst
> index 0e65cabe08eb..aa56d7ca6bfb 100644
> --- a/Documentation/dev-tools/kunit/start.rst
> +++ b/Documentation/dev-tools/kunit/start.rst
> @@ -236,5 +236,7 @@ Next Steps
>  ==========
>  *   Check out the :doc:`tips` page for tips on
>      writing idiomatic KUnit tests.
> +*   Check out the :doc:`running_tips` page for tips on
> +    how to make running KUnit tests easier.
>  *   Optional: see the :doc:`usage` page for a more
>      in-depth explanation of KUnit.
>
> base-commit: de2fcb3e62013738f22bbb42cbd757d9a242574e
> --
> 2.31.1.295.g9ea45b61b8-goog
>