Re: [PATCH v6 3/7] Documentation: KUnit: Added KUnit Architecture

[Marco Elver <elver@google.com>]

On Fri, 17 Dec 2021 at 05:49, Harinder Singh <sharinder@google.com> wrote:
>
> Describe the components of KUnit and how the kernel mode parts
> interact with kunit_tool.
>
> Signed-off-by: Harinder Singh <sharinder@google.com>

Acked-by: Marco Elver <elver@google.com>

For the .svg file, I think per
https://www.kernel.org/doc/html/latest/process/submitting-patches.html#sign-your-work-the-developer-s-certificate-of-origin
at least my Signed-off-by is required, but probably also my
Co-developed-by? In any case my SOB for kunit_suitememorydiagram.svg
is hereby given:

Signed-off-by: Marco Elver <elver@google.com>

Thanks,
-- Marco

> ---
>  .../dev-tools/kunit/architecture.rst          | 204 ++++++++++++++++++
>  Documentation/dev-tools/kunit/index.rst       |   2 +
>  .../kunit/kunit_suitememorydiagram.svg        |  81 +++++++
>  Documentation/dev-tools/kunit/start.rst       |   1 +
>  4 files changed, 288 insertions(+)
>  create mode 100644 Documentation/dev-tools/kunit/architecture.rst
>  create mode 100644 Documentation/dev-tools/kunit/kunit_suitememorydiagram.svg
>
> diff --git a/Documentation/dev-tools/kunit/architecture.rst b/Documentation/dev-tools/kunit/architecture.rst
> new file mode 100644
> index 000000000000..aa2cea821e25
> --- /dev/null
> +++ b/Documentation/dev-tools/kunit/architecture.rst
> @@ -0,0 +1,204 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +==================
> +KUnit Architecture
> +==================
> +
> +The KUnit architecture can be divided into two parts:
> +
> +- Kernel testing library
> +- kunit_tool (Command line test harness)
> +
> +In-Kernel Testing Framework
> +===========================
> +
> +The kernel testing library supports KUnit tests written in C using
> +KUnit. KUnit tests are kernel code. KUnit does several things:
> +
> +- Organizes tests
> +- Reports test results
> +- Provides test utilities
> +
> +Test Cases
> +----------
> +
> +The fundamental unit in KUnit is the test case. The KUnit test cases are
> +grouped into KUnit suites. A KUnit test case is a function with type
> +signature ``void (*)(struct kunit *test)``.
> +These test case functions are wrapped in a struct called
> +``struct kunit_case``. For code, see:
> +
> +.. kernel-doc:: include/kunit/test.h
> +       :identifiers: kunit_case
> +
> +.. note:
> +       ``generate_params`` is optional for non-parameterized tests.
> +
> +Each KUnit test case gets a ``struct kunit`` context
> +object passed to it that tracks a running test. The KUnit assertion
> +macros and other KUnit utilities use the ``struct kunit`` context
> +object. As an exception, there are two fields:
> +
> +- ``->priv``: The setup functions can use it to store arbitrary test
> +  user data.
> +
> +- ``->param_value``: It contains the parameter value which can be
> +  retrieved in the parameterized tests.
> +
> +Test Suites
> +-----------
> +
> +A KUnit suite includes a collection of test cases. The KUnit suites
> +are represented by the ``struct kunit_suite``. For example:
> +
> +.. code-block:: c
> +
> +       static struct kunit_case example_test_cases[] = {
> +               KUNIT_CASE(example_test_foo),
> +               KUNIT_CASE(example_test_bar),
> +               KUNIT_CASE(example_test_baz),
> +               {}
> +       };
> +
> +       static struct kunit_suite example_test_suite = {
> +               .name = "example",
> +               .init = example_test_init,
> +               .exit = example_test_exit,
> +               .test_cases = example_test_cases,
> +       };
> +       kunit_test_suite(example_test_suite);
> +
> +In the above example, the test suite ``example_test_suite``, runs the
> +test cases ``example_test_foo``, ``example_test_bar``, and
> +``example_test_baz``. Before running the test, the ``example_test_init``
> +is called and after running the test, ``example_test_exit`` is called.
> +The ``kunit_test_suite(example_test_suite)`` registers the test suite
> +with the KUnit test framework.
> +
> +Executor
> +--------
> +
> +The KUnit executor can list and run built-in KUnit tests on boot.
> +The Test suites are stored in a linker section
> +called ``.kunit_test_suites``. For code, see:
> +https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/asm-generic/vmlinux.lds.h?h=v5.15#n945.
> +The linker section consists of an array of pointers to
> +``struct kunit_suite``, and is populated by the ``kunit_test_suites()``
> +macro. To run all tests compiled into the kernel, the KUnit executor
> +iterates over the linker section array.
> +
> +.. kernel-figure:: kunit_suitememorydiagram.svg
> +       :alt:   KUnit Suite Memory
> +
> +       KUnit Suite Memory Diagram
> +
> +On the kernel boot, the KUnit executor uses the start and end addresses
> +of this section to iterate over and run all tests. For code, see:
> +https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/lib/kunit/executor.c
> +
> +When built as a module, the ``kunit_test_suites()`` macro defines a
> +``module_init()`` function, which runs all the tests in the compilation
> +unit instead of utilizing the executor.
> +
> +In KUnit tests, some error classes do not affect other tests
> +or parts of the kernel, each KUnit case executes in a separate thread
> +context. For code, see:
> +https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/lib/kunit/try-catch.c?h=v5.15#n58
> +
> +Assertion Macros
> +----------------
> +
> +KUnit tests verify state using expectations/assertions.
> +All expectations/assertions are formatted as:
> +``KUNIT_{EXPECT|ASSERT}_<op>[_MSG](kunit, property[, message])``
> +
> +- ``{EXPECT|ASSERT}`` determines whether the check is an assertion or an
> +  expectation.
> +
> +       - For an expectation, if the check fails, marks the test as failed
> +         and logs the failure.
> +
> +       - An assertion, on failure, causes the test case to terminate
> +         immediately.
> +
> +               - Assertions call function:
> +                 ``void __noreturn kunit_abort(struct kunit *)``.
> +
> +               - ``kunit_abort`` calls function:
> +                 ``void __noreturn kunit_try_catch_throw(struct kunit_try_catch *try_catch)``.
> +
> +               - ``kunit_try_catch_throw`` calls function:
> +                 ``void complete_and_exit(struct completion *, long) __noreturn;``
> +                 and terminates the special thread context.
> +
> +- ``<op>`` denotes a check with options: ``TRUE`` (supplied property
> +  has the boolean value “true”), ``EQ`` (two supplied properties are
> +  equal), ``NOT_ERR_OR_NULL`` (supplied pointer is not null and does not
> +  contain an “err” value).
> +
> +- ``[_MSG]`` prints a custom message on failure.
> +
> +Test Result Reporting
> +---------------------
> +KUnit prints test results in KTAP format. KTAP is based on TAP14, see:
> +https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-14-specification.md.
> +KTAP (yet to be standardized format) works with KUnit and Kselftest.
> +The KUnit executor prints KTAP results to dmesg, and debugfs
> +(if configured).
> +
> +Parameterized Tests
> +-------------------
> +
> +Each KUnit parameterized test is associated with a collection of
> +parameters. The test is invoked multiple times, once for each parameter
> +value and the parameter is stored in the ``param_value`` field.
> +The test case includes a ``KUNIT_CASE_PARAM()`` macro that accepts a
> +generator function.
> +The generator function is passed the previous parameter and returns the next
> +parameter. It also provides a macro to generate common-case generators based on
> +arrays.
> +
> +For code, see:
> +
> +.. kernel-doc:: include/kunit/test.h
> +       :identifiers: KUNIT_ARRAY_PARAM
> +
> +
> +kunit_tool (Command Line Test Harness)
> +======================================
> +
> +kunit_tool is a Python script ``(tools/testing/kunit/kunit.py)``
> +that can be used to configure, build, exec, parse and run (runs other
> +commands in order) test results. You can either run KUnit tests using
> +kunit_tool or can include KUnit in kernel and parse manually.
> +
> +- ``configure`` command generates the kernel ``.config`` from a
> +  ``.kunitconfig`` file (and any architecture-specific options).
> +  For some architectures, additional config options are specified in the
> +  ``qemu_config`` Python script
> +  (For example: ``tools/testing/kunit/qemu_configs/powerpc.py``).
> +  It parses both the existing ``.config`` and the ``.kunitconfig`` files
> +  and ensures that ``.config`` is a superset of ``.kunitconfig``.
> +  If this is not the case, it will combine the two and run
> +  ``make olddefconfig`` to regenerate the ``.config`` file. It then
> +  verifies that ``.config`` is now a superset. This checks if all
> +  Kconfig dependencies are correctly specified in ``.kunitconfig``.
> +  ``kunit_config.py`` includes the parsing Kconfigs code. The code which
> +  runs ``make olddefconfig`` is a part of ``kunit_kernel.py``. You can
> +  invoke this command via: ``./tools/testing/kunit/kunit.py config`` and
> +  generate a ``.config`` file.
> +- ``build`` runs ``make`` on the kernel tree with required options
> +  (depends on the architecture and some options, for example: build_dir)
> +  and reports any errors.
> +  To build a KUnit kernel from the current ``.config``, you can use the
> +  ``build`` argument: ``./tools/testing/kunit/kunit.py build``.
> +- ``exec`` command executes kernel results either directly (using
> +  User-mode Linux configuration), or via an emulator such
> +  as QEMU. It reads results from the log via standard
> +  output (stdout), and passes them to ``parse`` to be parsed.
> +  If you already have built a kernel with built-in KUnit tests,
> +  you can run the kernel and display the test results with the ``exec``
> +  argument: ``./tools/testing/kunit/kunit.py exec``.
> +- ``parse`` extracts the KTAP output from a kernel log, parses
> +  the test results, and prints a summary. For failed tests, any
> +  diagnostic output will be included.
> diff --git a/Documentation/dev-tools/kunit/index.rst b/Documentation/dev-tools/kunit/index.rst
> index 55d2444b0745..50d3ef9359dd 100644
> --- a/Documentation/dev-tools/kunit/index.rst
> +++ b/Documentation/dev-tools/kunit/index.rst
> @@ -9,6 +9,7 @@ KUnit - Linux Kernel Unit Testing
>         :caption: Contents:
>
>         start
> +       architecture
>         usage
>         kunit-tool
>         api/index
> @@ -96,6 +97,7 @@ How do I use it?
>  ================
>
>  *   Documentation/dev-tools/kunit/start.rst - for KUnit new users.
> +*   Documentation/dev-tools/kunit/architecture.rst - KUnit architecture.
>  *   Documentation/dev-tools/kunit/usage.rst - KUnit features.
>  *   Documentation/dev-tools/kunit/tips.rst - best practices with
>      examples.
> diff --git a/Documentation/dev-tools/kunit/kunit_suitememorydiagram.svg b/Documentation/dev-tools/kunit/kunit_suitememorydiagram.svg
> new file mode 100644
> index 000000000000..cf8fddc27500
> --- /dev/null
> +++ b/Documentation/dev-tools/kunit/kunit_suitememorydiagram.svg
> @@ -0,0 +1,81 @@
> +<?xml version="1.0" encoding="UTF-8"?>
> +<svg width="796.93" height="555.73" version="1.1" viewBox="0 0 796.93 555.73" xmlns="http://www.w3.org/2000/svg">
> +       <g transform="translate(-13.724 -17.943)">
> +               <g fill="#dad4d4" fill-opacity=".91765" stroke="#1a1a1a">
> +                       <rect x="323.56" y="18.443" width="115.75" height="41.331"/>
> +                       <rect x="323.56" y="463.09" width="115.75" height="41.331"/>
> +                       <rect x="323.56" y="531.84" width="115.75" height="41.331"/>
> +                       <rect x="323.56" y="88.931" width="115.75" height="74.231"/>
> +               </g>
> +               <g>
> +                       <rect x="323.56" y="421.76" width="115.75" height="41.331" fill="#b9dbc6" stroke="#1a1a1a"/>
> +                       <text x="328.00888" y="446.61826" fill="#000000" font-family="sans-serif" font-size="16px" style="line-height:1.25" xml:space="preserve"><tspan x="328.00888" y="446.61826" font-family="monospace" font-size="16px">kunit_suite</tspan></text>
> +               </g>
> +               <g transform="translate(0 -258.6)">
> +                       <rect x="323.56" y="421.76" width="115.75" height="41.331" fill="#b9dbc6" stroke="#1a1a1a"/>
> +                       <text x="328.00888" y="446.61826" fill="#000000" font-family="sans-serif" font-size="16px" style="line-height:1.25" xml:space="preserve"><tspan x="328.00888" y="446.61826" font-family="monospace" font-size="16px">kunit_suite</tspan></text>
> +               </g>
> +               <g transform="translate(0 -217.27)">
> +                       <rect x="323.56" y="421.76" width="115.75" height="41.331" fill="#b9dbc6" stroke="#1a1a1a"/>
> +                       <text x="328.00888" y="446.61826" fill="#000000" font-family="sans-serif" font-size="16px" style="line-height:1.25" xml:space="preserve"><tspan x="328.00888" y="446.61826" font-family="monospace" font-size="16px">kunit_suite</tspan></text>
> +               </g>
> +               <g transform="translate(0 -175.94)">
> +                       <rect x="323.56" y="421.76" width="115.75" height="41.331" fill="#b9dbc6" stroke="#1a1a1a"/>
> +                       <text x="328.00888" y="446.61826" fill="#000000" font-family="sans-serif" font-size="16px" style="line-height:1.25" xml:space="preserve"><tspan x="328.00888" y="446.61826" font-family="monospace" font-size="16px">kunit_suite</tspan></text>
> +               </g>
> +               <g transform="translate(0 -134.61)">
> +                       <rect x="323.56" y="421.76" width="115.75" height="41.331" fill="#b9dbc6" stroke="#1a1a1a"/>
> +                       <text x="328.00888" y="446.61826" fill="#000000" font-family="sans-serif" font-size="16px" style="line-height:1.25" xml:space="preserve"><tspan x="328.00888" y="446.61826" font-family="monospace" font-size="16px">kunit_suite</tspan></text>
> +               </g>
> +               <g transform="translate(0 -41.331)">
> +                       <rect x="323.56" y="421.76" width="115.75" height="41.331" fill="#b9dbc6" stroke="#1a1a1a"/>
> +                       <text x="328.00888" y="446.61826" fill="#000000" font-family="sans-serif" font-size="16px" style="line-height:1.25" xml:space="preserve"><tspan x="328.00888" y="446.61826" font-family="monospace" font-size="16px">kunit_suite</tspan></text>
> +               </g>
> +               <g transform="translate(3.4459e-5 -.71088)">
> +                       <rect x="502.19" y="143.16" width="201.13" height="41.331" fill="#dad4d4" fill-opacity=".91765" stroke="#1a1a1a"/>
> +                       <text x="512.02319" y="168.02026" font-family="sans-serif" font-size="16px" style="line-height:1.25" xml:space="preserve"><tspan x="512.02319" y="168.02026" font-family="monospace">_kunit_suites_start</tspan></text>
> +               </g>
> +               <g transform="translate(3.0518e-5 -3.1753)">
> +                       <rect x="502.19" y="445.69" width="201.13" height="41.331" fill="#dad4d4" fill-opacity=".91765" stroke="#1a1a1a"/>
> +                       <text x="521.61694" y="470.54846" font-family="sans-serif" font-size="16px" style="line-height:1.25" xml:space="preserve"><tspan x="521.61694" y="470.54846" font-family="monospace">_kunit_suites_end</tspan></text>
> +               </g>
> +               <rect x="14.224" y="277.78" width="134.47" height="41.331" fill="#dad4d4" fill-opacity=".91765" stroke="#1a1a1a"/>
> +               <text x="32.062176" y="304.41287" font-family="sans-serif" font-size="16px" style="line-height:1.25" xml:space="preserve"><tspan x="32.062176" y="304.41287" font-family="monospace">.init.data</tspan></text>
> +               <g transform="translate(217.98 145.12)" stroke="#1a1a1a">
> +                       <circle cx="149.97" cy="373.01" r="3.4012"/>
> +                       <circle cx="163.46" cy="373.01" r="3.4012"/>
> +                       <circle cx="176.95" cy="373.01" r="3.4012"/>
> +               </g>
> +               <g transform="translate(217.98 -298.66)" stroke="#1a1a1a">
> +                       <circle cx="149.97" cy="373.01" r="3.4012"/>
> +                       <circle cx="163.46" cy="373.01" r="3.4012"/>
> +                       <circle cx="176.95" cy="373.01" r="3.4012"/>
> +               </g>
> +               <g stroke="#1a1a1a">
> +                       <rect x="323.56" y="328.49" width="115.75" height="51.549" fill="#b9dbc6"/>
> +                       <g transform="translate(217.98 -18.75)">
> +                               <circle cx="149.97" cy="373.01" r="3.4012"/>
> +                               <circle cx="163.46" cy="373.01" r="3.4012"/>
> +                               <circle cx="176.95" cy="373.01" r="3.4012"/>
> +                       </g>
> +               </g>
> +               <g transform="scale(1.0933 .9147)" stroke-width="32.937" aria-label="{">
> +                       <path d="m275.49 545.57c-35.836-8.432-47.43-24.769-47.957-64.821v-88.536c-0.527-44.795-10.54-57.97-49.538-67.456 38.998-10.013 49.011-23.715 49.538-67.983v-88.536c0.527-40.052 12.121-56.389 47.957-64.821v-5.797c-65.348 0-85.901 17.391-86.955 73.253v93.806c-0.527 36.89-10.013 50.065-44.795 59.551 34.782 10.013 44.268 23.188 44.795 60.078v93.279c1.581 56.389 21.607 73.78 86.955 73.78z"/>
> +               </g>
> +               <g transform="scale(1.1071 .90325)" stroke-width="14.44" aria-label="{">
> +                       <path d="m461.46 443.55c-15.711-3.6967-20.794-10.859-21.025-28.418v-38.815c-0.23104-19.639-4.6209-25.415-21.718-29.574 17.097-4.3898 21.487-10.397 21.718-29.805v-38.815c0.23105-17.559 5.314-24.722 21.025-28.418v-2.5415c-28.649 0-37.66 7.6244-38.122 32.115v41.126c-0.23105 16.173-4.3898 21.949-19.639 26.108 15.249 4.3898 19.408 10.166 19.639 26.339v40.895c0.69313 24.722 9.4728 32.346 38.122 32.346z"/>
> +               </g>
> +               <path d="m449.55 161.84v2.5h49.504v-2.5z" color="#000000" style="-inkscape-stroke:none"/>
> +               <g fill-rule="evenodd">
> +                       <path d="m443.78 163.09 8.65-5v10z" color="#000000" stroke-width="1pt" style="-inkscape-stroke:none"/>
> +                       <path d="m453.1 156.94-10.648 6.1543 0.99804 0.57812 9.6504 5.5781zm-1.334 2.3125v7.6856l-6.6504-3.8438z" color="#000000" style="-inkscape-stroke:none"/>
> +               </g>
> +               <path d="m449.55 461.91v2.5h49.504v-2.5z" color="#000000" style="-inkscape-stroke:none"/>
> +               <g fill-rule="evenodd">
> +                       <path d="m443.78 463.16 8.65-5v10z" color="#000000" stroke-width="1pt" style="-inkscape-stroke:none"/>
> +                       <path d="m453.1 457-10.648 6.1562 0.99804 0.57617 9.6504 5.5781zm-1.334 2.3125v7.6856l-6.6504-3.8438z" color="#000000" style="-inkscape-stroke:none"/>
> +               </g>
> +               <rect x="515.64" y="223.9" width="294.52" height="178.49" fill="#dad4d4" fill-opacity=".91765" stroke="#1a1a1a"/>
> +               <text x="523.33319" y="262.52542" font-family="monospace" font-size="14.667px" style="line-height:1.25" xml:space="preserve"><tspan x="523.33319" y="262.52542"><tspan fill="#008000" font-family="monospace" font-size="14.667px" font-weight="bold">struct</tspan> kunit_suite {</tspan><tspan x="523.33319" y="280.8588"><tspan fill="#008000" font-family="monospace" font-size="14.667px" font-weight="bold">  const char</tspan> name[<tspan fill="#ff00ff" font-size="14.667px">256</tspan>];</tspan><tspan x="523.33319" y="299.19217">  <tspan fill="#008000" font-family="monospace" font-size="14.667px" font-weight="bold">int</tspan> (*init)(<tspan fill="#008000" font-family="monospace" font-size="14.667px" font-weight="bold">struct</tspan> kunit *);</tspan><tspan x="523.33319" y="317.52554">  <tspan fill="#008000" font-family="monospace" font-size="14.667px" font-weight="bold">void</tspan> (*exit)(<tspan fill="#008000" font-family="monospace" font-size="14.667px" font-weight="bold">struct</tspan> kunit *);</tspan><tspan x="523.33319" y="335.85892">  <tspan fill="#008000" font-family="monospace" font-size="14.667px" font-weight="bold">struct</tspan> kunit_case *test_cases;</tspan><tspan x="523.33319" y="354.19229">  ...</tspan><tspan x="523.33319" y="372.52567">};</tspan></text>
> +       </g>
> +</svg>
> diff --git a/Documentation/dev-tools/kunit/start.rst b/Documentation/dev-tools/kunit/start.rst
> index 55f8df1abd40..5dd2c88fa2bd 100644
> --- a/Documentation/dev-tools/kunit/start.rst
> +++ b/Documentation/dev-tools/kunit/start.rst
> @@ -240,6 +240,7 @@ Congrats! You just wrote your first KUnit test.
>  Next Steps
>  ==========
>
> +*   Documentation/dev-tools/kunit/architecture.rst - KUnit architecture.
>  *   Documentation/dev-tools/kunit/usage.rst - KUnit features.
>  *   Documentation/dev-tools/kunit/tips.rst - best practices with
>      examples.
> --
> 2.34.1.173.g76aa8bc2d0-goog
>