RE: [PATCH v2 3/7] Documentation: KUnit: Added KUnit Architecture

From: <Tim.Bird@sony.com>
To: <sharinder@google.com>, <davidgow@google.com>,
	<brendanhiggins@google.com>, <shuah@kernel.org>, <corbet@lwn.net>
Cc: <linux-kselftest@vger.kernel.org>, <kunit-dev@googlegroups.com>,
	<linux-doc@vger.kernel.org>, <linux-kernel@vger.kernel.org>
Subject: RE: [PATCH v2 3/7] Documentation: KUnit: Added KUnit Architecture
Date: Tue, 7 Dec 2021 17:24:49 +0000	[thread overview]
Message-ID: <BYAPR13MB2503D5A5E52D5645C5CA3C3AFD6E9@BYAPR13MB2503.namprd13.prod.outlook.com> (raw)
In-Reply-To: <20211207054019.1455054-4-sharinder@google.com>

> -----Original Message-----
> From: Harinder Singh <sharinder@google.com>
> 
> Describe the components of KUnit and how the kernel mode parts
> interact with kunit_tool.
> 
> Signed-off-by: Harinder Singh <sharinder@google.com>
> ---
>  .../dev-tools/kunit/architecture.rst          | 206 ++++++++++++++++++
>  Documentation/dev-tools/kunit/index.rst       |   2 +
>  .../kunit/kunit_suitememorydiagram.png        | Bin 0 -> 24174 bytes
>  Documentation/dev-tools/kunit/start.rst       |   1 +
>  4 files changed, 209 insertions(+)
>  create mode 100644 Documentation/dev-tools/kunit/architecture.rst
>  create mode 100644 Documentation/dev-tools/kunit/kunit_suitememorydiagram.png
> 
> diff --git a/Documentation/dev-tools/kunit/architecture.rst b/Documentation/dev-tools/kunit/architecture.rst
> new file mode 100644
> index 000000000000..bb0fb3e3ed01
> --- /dev/null
> +++ b/Documentation/dev-tools/kunit/architecture.rst
> @@ -0,0 +1,206 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +==================
> +KUnit Architecture
> +==================
> +
> +The KUnit architecture can be divided into two parts:
> +
> +- Kernel testing library
> +- kunit_tool (Command line test harness)
> +
> +In-Kernel Testing Framework
> +===========================
> +
> +The kernel testing library supports KUnit tests written in C using
> +KUnit. KUnit tests are kernel code. KUnit does several things:
> +
> +- Organizes tests
> +- Reports test results
> +- Provides test utilities
> +
> +Test Cases
> +----------
> +
> +The fundamental unit in KUnit is the test case. The KUnit test cases are
> +grouped into KUnit suites. A KUnit test case is a function with type
> +signature ``void (*)(struct kunit *test)``.
> +These test case functions are wrapped in a struct called
> +``struct kunit_case``. For code, see:
> +https://elixir.bootlin.com/linux/latest/source/include/kunit/test.h#L145
> +
> +It includes:
> +
> +- ``run_case``: the function implementing the actual test case.
> +- ``name``: the test case name.
> +- ``generate_params``: the parameterized tests generator function. This
> +  is optional for non-parameterized tests.
> +
> +Each KUnit test case gets a ``struct kunit`` context
> +object passed to it that tracks a running test. The KUnit assertion
> +macros and other KUnit utilities use the ``struct kunit`` context
> +object. As an exception, there are two fields:
> +
> +- ``->priv``: The setup functions can use it to store arbitrary test
> +  user data.
> +
> +- ``->param_value``: It contains the parameter value which can be
> +  retrieved in the parameterized tests.
> +
> +Test Suites
> +-----------
> +
> +A KUnit suite includes a collection of test cases. The KUnit suites
> +are represented by the ``struct kunit_suite``. For example:
> +
> +.. code-block:: c
> +
> +	static struct kunit_case example_test_cases[] = {
> +		KUNIT_CASE(example_test_foo),
> +		KUNIT_CASE(example_test_bar),
> +		KUNIT_CASE(example_test_baz),
> +		{}
> +	};
> +
> +	static struct kunit_suite example_test_suite = {
> +		.name = "example",
> +		.init = example_test_init,
> +		.exit = example_test_exit,
> +		.test_cases = example_test_cases,
> +	};
> +	kunit_test_suite(example_test_suite);
> +
> +In the above example, the test suite ``example_test_suite``, runs the
> +test cases ``example_test_foo``, ``example_test_bar``, and
> +``example_test_baz``. Before running the test, the ``example_test_init``
> +is called and after running the test, ``example_test_exit`` is called.
> +The ``kunit_test_suite(example_test_suite)`` registers the test suite
> +with the KUnit test framework.
> +
> +Executor
> +--------
> +
> +The KUnit executor can list and run built-in KUnit tests on boot.
> +The Test suites are stored in a linker section
> +called ``.kunit_test_suites``. For code, see:
> +https://elixir.bootlin.com/linux/v5.12/source/include/asm-generic/vmlinux.lds.h#L918.
> +The linker section consists of an array of pointers to
> +``struct kunit_suite``, and is populated by the ``kunit_test_suites()``
> +macro. To run all tests compiled into the kernel, the KUnit executor
> +iterates over the linker section array.
> +
> +.. kernel-figure:: kunit_suitememorydiagram.png
> +	:alt:	KUnit Suite Memory
> +
> +	KUnit Suite Memory Diagram
> +
> +On the kernel boot, the KUnit executor uses the start and end addresses
> +of this section to iterate over and run all tests. For code, see:
> +https://elixir.bootlin.com/linux/latest/source/lib/kunit/executor.c
> +
> +When built as a module, the ``kunit_test_suites()`` macro defines a
> +``module_init()`` function, which runs all the tests in the compilation
> +unit instead of utilizing the executor.
> +
> +In KUnit tests, some error classes do not affect other tests
> +or parts of the kernel, each KUnit case executes in a separate thread
> +context. For code, see:
> +https://elixir.bootlin.com/linux/latest/source/lib/kunit/try-catch.c#L58
> +
> +Assertion Macros
> +----------------
> +
> +KUnit tests verify state using expectations/assertions.
> +All expectations/assertions are formatted as:
> +``KUNIT_{EXPECT|ASSERT}_<op>[_MSG](kunit, property[, message])``
> +
> +- ``{EXPECT|ASSERT}`` determines whether the check is an assertion or an
> +  expectation.
> +
> +	- For an expectation, if the check fails, marks the test as failed
> +	  and logs the failure.
> +
> +	- An assertion, on failure, causes the test case to terminate
> +	  immediately.
> +
> +		- Assertions call function:
> +		  ``void __noreturn kunit_abort(struct kunit *)``.
> +
> +		- ``kunit_abort`` calls function:
> +		  ``void __noreturn kunit_try_catch_throw(struct kunit_try_catch *try_catch)``.
> +
> +		- ``kunit_try_catch_throw`` calls function:
> +		  ``void complete_and_exit(struct completion *, long) __noreturn;``
> +		  and terminates the special thread context.
> +
> +- ``<op>`` denotes a check with options: ``TRUE`` (supplied property
> +  has the boolean value “true”), ``EQ`` (two supplied properties are
> +  equal), ``NOT_ERR_OR_NULL`` (supplied pointer is not null and does not
> +  contain an “err” value).
> +
> +- ``[_MSG]`` prints a custom message on failure.
> +
> +Test Result Reporting
> +---------------------
> +KUnit prints test results in KTAP format. KTAP is based on TAP14, see:
> +https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-14-specification.md.
> +KTAP (yet to be standardized format) works with KUnit and Kselftest.
> +The KUnit executor prints KTAP results to dmesg, and debugfs
> +(if configured).
> +
> +Parameterized Tests
> +-------------------
> +
> +Each KUnit parameterized test is associated with a collection of
> +parameters. The test is invoked multiple times, once for each parameter
> +value and the parameter is stored in the ``param_value`` field.
> +The test case includes a ``KUNIT_CASE_PARAM()`` macro that accepts a
> +generator function.
> +The generator function returns the next parameter given to the

given to the -> given the

> +previous parameter in parameterized tests. It also provides a macro to
> +generate common-case generators based on arrays.
> +
> +For code, see:
> +https://elixir.bootlin.com/linux/v5.12/source/include/kunit/test.h#L1783

The rest looks OK, as far as I can tell.
 -- Tim