From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61AC1C433FE for ; Fri, 17 Dec 2021 10:50:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235133AbhLQKuR (ORCPT ); Fri, 17 Dec 2021 05:50:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47934 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235032AbhLQKuQ (ORCPT ); Fri, 17 Dec 2021 05:50:16 -0500 Received: from mail-ot1-x335.google.com (mail-ot1-x335.google.com [IPv6:2607:f8b0:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E1E17C06173E for ; Fri, 17 Dec 2021 02:50:15 -0800 (PST) Received: by mail-ot1-x335.google.com with SMTP id n104-20020a9d2071000000b005799790cf0bso2371514ota.5 for ; Fri, 17 Dec 2021 02:50:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=ChAQgQ67R/IfTiZcoWSnAH60LCShrosGvHLHIofQtOY=; b=lKx/LadyuNCtf5NKPeaK3K7P5iBR/KAPy+3eeZjSalLRBIwj80hClAAY6lLEWAhved 0JjsHBi/tS9IeSMq+DTKYahwJrauEh28Q/C5K1lsTBPBuXKFTG6qzNbtxdEXZtyaQpbT 8TOyQS/wozWpqfdKz7O2JExRaFoWuUu34Kndj9fV3X5VSC0Bo3BXVPLGc4ZcTKMS+XtZ 1H7s+Iu4bW2L72eQxsVarLcPfbdlKARqrOdQ4qNGJcUiIVug0J2A5BG6XqUAeOKgl2Y3 7Y7lynvjjqM5Bfbyl6nS49utzOF8gPS/gItmtO7srHNhdmYLuoAQIqIqZCrROaIfPFsx 4SIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=ChAQgQ67R/IfTiZcoWSnAH60LCShrosGvHLHIofQtOY=; b=zE2Sb56V7YdTbxF8um5/a9XLOU/q/JJyyfAmUU1V4TMriyViUAKDuSNkdEr1PWkmg4 yPOCfbgOUJmST7vW3GG6doaB0uCd3TAizuUBHGMPuOMx/x30lUej24KSOK1TGYVOM8j9 HnZ63EXnCvF24YZ7IsmDsiqm6otI2sHrOyEGST+MdX1EECq+dJKBOc+5CbwZMBEuaRnh n7l/dLSZfJVm7RZPKfTE9+9oDYSjXmeWHfmCx8feZDyWsOMTjZ+er4CB5GO+W3u2cY7j LxZs/4ciAjBNpv9Z9DcivujP5MkW3bl/TLaxDrWu9erCsdMs54ayFyyaT5gsjvOj0MDU tOKg== X-Gm-Message-State: AOAM530tUM2OfvYVb85n/hfr40FIU9hjkgF38piVJJHz59fBP1/KoOV/ AfrcPGn7tXFutBSDCEEjQdJws0IuidO/rePrIqM1fw== X-Google-Smtp-Source: ABdhPJwRnAonVZjGqTFN2dX7KGtpy2hYHKmh7cmpGhv6LaIh4vcm44N2JaaCFdzONE8hQi/51peHk29GxHVDQms7x1w= X-Received: by 2002:a9d:77d1:: with SMTP id w17mr1674722otl.329.1639738214883; Fri, 17 Dec 2021 02:50:14 -0800 (PST) MIME-Version: 1.0 References: <20211217044911.798817-1-sharinder@google.com> <20211217044911.798817-4-sharinder@google.com> In-Reply-To: <20211217044911.798817-4-sharinder@google.com> From: Marco Elver Date: Fri, 17 Dec 2021 11:50:03 +0100 Message-ID: Subject: Re: [PATCH v6 3/7] Documentation: KUnit: Added KUnit Architecture To: Harinder Singh Cc: davidgow@google.com, brendanhiggins@google.com, shuah@kernel.org, corbet@lwn.net, linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, Tim.Bird@sony.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, 17 Dec 2021 at 05:49, Harinder Singh wrote: > > Describe the components of KUnit and how the kernel mode parts > interact with kunit_tool. > > Signed-off-by: Harinder Singh Acked-by: Marco Elver For the .svg file, I think per https://www.kernel.org/doc/html/latest/process/submitting-patches.html#sign= -your-work-the-developer-s-certificate-of-origin at least my Signed-off-by is required, but probably also my Co-developed-by? In any case my SOB for kunit_suitememorydiagram.svg is hereby given: Signed-off-by: Marco Elver Thanks, -- Marco > --- > .../dev-tools/kunit/architecture.rst | 204 ++++++++++++++++++ > Documentation/dev-tools/kunit/index.rst | 2 + > .../kunit/kunit_suitememorydiagram.svg | 81 +++++++ > Documentation/dev-tools/kunit/start.rst | 1 + > 4 files changed, 288 insertions(+) > create mode 100644 Documentation/dev-tools/kunit/architecture.rst > create mode 100644 Documentation/dev-tools/kunit/kunit_suitememorydiagra= m.svg > > diff --git a/Documentation/dev-tools/kunit/architecture.rst b/Documentati= on/dev-tools/kunit/architecture.rst > new file mode 100644 > index 000000000000..aa2cea821e25 > --- /dev/null > +++ b/Documentation/dev-tools/kunit/architecture.rst > @@ -0,0 +1,204 @@ > +.. SPDX-License-Identifier: GPL-2.0 > + > +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > +KUnit Architecture > +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > + > +The KUnit architecture can be divided into two parts: > + > +- Kernel testing library > +- kunit_tool (Command line test harness) > + > +In-Kernel Testing Framework > +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D > + > +The kernel testing library supports KUnit tests written in C using > +KUnit. KUnit tests are kernel code. KUnit does several things: > + > +- Organizes tests > +- Reports test results > +- Provides test utilities > + > +Test Cases > +---------- > + > +The fundamental unit in KUnit is the test case. The KUnit test cases are > +grouped into KUnit suites. A KUnit test case is a function with type > +signature ``void (*)(struct kunit *test)``. > +These test case functions are wrapped in a struct called > +``struct kunit_case``. For code, see: > + > +.. kernel-doc:: include/kunit/test.h > + :identifiers: kunit_case > + > +.. note: > + ``generate_params`` is optional for non-parameterized tests. > + > +Each KUnit test case gets a ``struct kunit`` context > +object passed to it that tracks a running test. The KUnit assertion > +macros and other KUnit utilities use the ``struct kunit`` context > +object. As an exception, there are two fields: > + > +- ``->priv``: The setup functions can use it to store arbitrary test > + user data. > + > +- ``->param_value``: It contains the parameter value which can be > + retrieved in the parameterized tests. > + > +Test Suites > +----------- > + > +A KUnit suite includes a collection of test cases. The KUnit suites > +are represented by the ``struct kunit_suite``. For example: > + > +.. code-block:: c > + > + static struct kunit_case example_test_cases[] =3D { > + KUNIT_CASE(example_test_foo), > + KUNIT_CASE(example_test_bar), > + KUNIT_CASE(example_test_baz), > + {} > + }; > + > + static struct kunit_suite example_test_suite =3D { > + .name =3D "example", > + .init =3D example_test_init, > + .exit =3D example_test_exit, > + .test_cases =3D example_test_cases, > + }; > + kunit_test_suite(example_test_suite); > + > +In the above example, the test suite ``example_test_suite``, runs the > +test cases ``example_test_foo``, ``example_test_bar``, and > +``example_test_baz``. Before running the test, the ``example_test_init`` > +is called and after running the test, ``example_test_exit`` is called. > +The ``kunit_test_suite(example_test_suite)`` registers the test suite > +with the KUnit test framework. > + > +Executor > +-------- > + > +The KUnit executor can list and run built-in KUnit tests on boot. > +The Test suites are stored in a linker section > +called ``.kunit_test_suites``. For code, see: > +https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/= include/asm-generic/vmlinux.lds.h?h=3Dv5.15#n945. > +The linker section consists of an array of pointers to > +``struct kunit_suite``, and is populated by the ``kunit_test_suites()`` > +macro. To run all tests compiled into the kernel, the KUnit executor > +iterates over the linker section array. > + > +.. kernel-figure:: kunit_suitememorydiagram.svg > + :alt: KUnit Suite Memory > + > + KUnit Suite Memory Diagram > + > +On the kernel boot, the KUnit executor uses the start and end addresses > +of this section to iterate over and run all tests. For code, see: > +https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/= lib/kunit/executor.c > + > +When built as a module, the ``kunit_test_suites()`` macro defines a > +``module_init()`` function, which runs all the tests in the compilation > +unit instead of utilizing the executor. > + > +In KUnit tests, some error classes do not affect other tests > +or parts of the kernel, each KUnit case executes in a separate thread > +context. For code, see: > +https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/= lib/kunit/try-catch.c?h=3Dv5.15#n58 > + > +Assertion Macros > +---------------- > + > +KUnit tests verify state using expectations/assertions. > +All expectations/assertions are formatted as: > +``KUNIT_{EXPECT|ASSERT}_[_MSG](kunit, property[, message])`` > + > +- ``{EXPECT|ASSERT}`` determines whether the check is an assertion or an > + expectation. > + > + - For an expectation, if the check fails, marks the test as faile= d > + and logs the failure. > + > + - An assertion, on failure, causes the test case to terminate > + immediately. > + > + - Assertions call function: > + ``void __noreturn kunit_abort(struct kunit *)``. > + > + - ``kunit_abort`` calls function: > + ``void __noreturn kunit_try_catch_throw(struct kunit_tr= y_catch *try_catch)``. > + > + - ``kunit_try_catch_throw`` calls function: > + ``void complete_and_exit(struct completion *, long) __n= oreturn;`` > + and terminates the special thread context. > + > +- ```` denotes a check with options: ``TRUE`` (supplied property > + has the boolean value =E2=80=9Ctrue=E2=80=9D), ``EQ`` (two supplied pr= operties are > + equal), ``NOT_ERR_OR_NULL`` (supplied pointer is not null and does not > + contain an =E2=80=9Cerr=E2=80=9D value). > + > +- ``[_MSG]`` prints a custom message on failure. > + > +Test Result Reporting > +--------------------- > +KUnit prints test results in KTAP format. KTAP is based on TAP14, see: > +https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-= 14-specification.md. > +KTAP (yet to be standardized format) works with KUnit and Kselftest. > +The KUnit executor prints KTAP results to dmesg, and debugfs > +(if configured). > + > +Parameterized Tests > +------------------- > + > +Each KUnit parameterized test is associated with a collection of > +parameters. The test is invoked multiple times, once for each parameter > +value and the parameter is stored in the ``param_value`` field. > +The test case includes a ``KUNIT_CASE_PARAM()`` macro that accepts a > +generator function. > +The generator function is passed the previous parameter and returns the = next > +parameter. It also provides a macro to generate common-case generators b= ased on > +arrays. > + > +For code, see: > + > +.. kernel-doc:: include/kunit/test.h > + :identifiers: KUNIT_ARRAY_PARAM > + > + > +kunit_tool (Command Line Test Harness) > +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > + > +kunit_tool is a Python script ``(tools/testing/kunit/kunit.py)`` > +that can be used to configure, build, exec, parse and run (runs other > +commands in order) test results. You can either run KUnit tests using > +kunit_tool or can include KUnit in kernel and parse manually. > + > +- ``configure`` command generates the kernel ``.config`` from a > + ``.kunitconfig`` file (and any architecture-specific options). > + For some architectures, additional config options are specified in the > + ``qemu_config`` Python script > + (For example: ``tools/testing/kunit/qemu_configs/powerpc.py``). > + It parses both the existing ``.config`` and the ``.kunitconfig`` files > + and ensures that ``.config`` is a superset of ``.kunitconfig``. > + If this is not the case, it will combine the two and run > + ``make olddefconfig`` to regenerate the ``.config`` file. It then > + verifies that ``.config`` is now a superset. This checks if all > + Kconfig dependencies are correctly specified in ``.kunitconfig``. > + ``kunit_config.py`` includes the parsing Kconfigs code. The code which > + runs ``make olddefconfig`` is a part of ``kunit_kernel.py``. You can > + invoke this command via: ``./tools/testing/kunit/kunit.py config`` and > + generate a ``.config`` file. > +- ``build`` runs ``make`` on the kernel tree with required options > + (depends on the architecture and some options, for example: build_dir) > + and reports any errors. > + To build a KUnit kernel from the current ``.config``, you can use the > + ``build`` argument: ``./tools/testing/kunit/kunit.py build``. > +- ``exec`` command executes kernel results either directly (using > + User-mode Linux configuration), or via an emulator such > + as QEMU. It reads results from the log via standard > + output (stdout), and passes them to ``parse`` to be parsed. > + If you already have built a kernel with built-in KUnit tests, > + you can run the kernel and display the test results with the ``exec`` > + argument: ``./tools/testing/kunit/kunit.py exec``. > +- ``parse`` extracts the KTAP output from a kernel log, parses > + the test results, and prints a summary. For failed tests, any > + diagnostic output will be included. > diff --git a/Documentation/dev-tools/kunit/index.rst b/Documentation/dev-= tools/kunit/index.rst > index 55d2444b0745..50d3ef9359dd 100644 > --- a/Documentation/dev-tools/kunit/index.rst > +++ b/Documentation/dev-tools/kunit/index.rst > @@ -9,6 +9,7 @@ KUnit - Linux Kernel Unit Testing > :caption: Contents: > > start > + architecture > usage > kunit-tool > api/index > @@ -96,6 +97,7 @@ How do I use it? > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > * Documentation/dev-tools/kunit/start.rst - for KUnit new users. > +* Documentation/dev-tools/kunit/architecture.rst - KUnit architecture. > * Documentation/dev-tools/kunit/usage.rst - KUnit features. > * Documentation/dev-tools/kunit/tips.rst - best practices with > examples. > diff --git a/Documentation/dev-tools/kunit/kunit_suitememorydiagram.svg b= /Documentation/dev-tools/kunit/kunit_suitememorydiagram.svg > new file mode 100644 > index 000000000000..cf8fddc27500 > --- /dev/null > +++ b/Documentation/dev-tools/kunit/kunit_suitememorydiagram.svg > @@ -0,0 +1,81 @@ > + > + > + > + > + > + > + > + > + > + > + > + kunit_suite > + > + > + > + kunit_suite > + > + > + > + kunit_suite > + > + > + > + kunit_suite > + > + > + > + kunit_suite > + > + > + > + kunit_suite > + > + > + > + = _kunit_suites_start > + > + > + > + = _kunit_suites_end > + > + > + .init.da= ta > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + > + struct kunit_suite { const char name[256]; int (*init)(struct kunit *= ); void<= /tspan> (*exit)(struct kunit *); struct kunit_ca= se *test_cases; ...= }; > + > + > diff --git a/Documentation/dev-tools/kunit/start.rst b/Documentation/dev-= tools/kunit/start.rst > index 55f8df1abd40..5dd2c88fa2bd 100644 > --- a/Documentation/dev-tools/kunit/start.rst > +++ b/Documentation/dev-tools/kunit/start.rst > @@ -240,6 +240,7 @@ Congrats! You just wrote your first KUnit test. > Next Steps > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > +* Documentation/dev-tools/kunit/architecture.rst - KUnit architecture. > * Documentation/dev-tools/kunit/usage.rst - KUnit features. > * Documentation/dev-tools/kunit/tips.rst - best practices with > examples. > -- > 2.34.1.173.g76aa8bc2d0-goog >