From: Brendan Higgins <brendanhiggins@google.com>
To: David Gow <davidgow@google.com>
Cc: Jeff Dike <jdike@addtoit.com>,
Richard Weinberger <richard@nod.at>,
Anton Ivanov <anton.ivanov@cambridgegreys.com>,
Arnd Bergmann <arnd@arndb.de>, Kees Cook <keescook@chromium.org>,
Shuah Khan <skhan@linuxfoundation.org>,
Alan Maguire <alan.maguire@oracle.com>,
Iurii Zaikin <yzaikin@google.com>,
Andrew Morton <akpm@linux-foundation.org>,
rppt@linux.ibm.com, Frank Rowand <frowand.list@gmail.com>,
catalin.marinas@arm.com, will@kernel.org, monstr@monstr.eu,
Michael Ellerman <mpe@ellerman.id.au>,
Benjamin Herrenschmidt <benh@kernel.crashing.org>,
paulus@samba.org, chris@zankel.net, jcmvbkbc@gmail.com,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
Stephen Boyd <sboyd@kernel.org>,
Logan Gunthorpe <logang@deltatee.com>,
Luis Chamberlain <mcgrof@kernel.org>,
linux-um <linux-um@lists.infradead.org>,
linux-arch@vger.kernel.org,
"open list:KERNEL SELFTEST FRAMEWORK"
<linux-kselftest@vger.kernel.org>,
KUnit Development <kunit-dev@googlegroups.com>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
"open list:DOCUMENTATION" <linux-doc@vger.kernel.org>,
Linux ARM <linux-arm-kernel@lists.infradead.org>,
linuxppc-dev@lists.ozlabs.org, linux-xtensa@linux-xtensa.org
Subject: Re: [PATCH v4 00/11] kunit: create a centralized executor to dispatch all KUnit tests
Date: Fri, 26 Jun 2020 14:10:33 -0700 [thread overview]
Message-ID: <CAFd5g45x04Q+r+1M+kwqkp1DsY9PjJN+Gs0Q+MkOdYTz0SdzKw@mail.gmail.com> (raw)
Message-ID: <20200626211033.7IAUvE_AeyuLKGR0HcVH2Jqvi1FpboiAYNJO8ZNAdNA@z> (raw)
In-Reply-To: <CABVgOSnkYfXZ1YELsXAjA0GzCQT9YWO8x+Tssw_+avkDdBB4aw@mail.gmail.com>
On Wed, Jun 24, 2020 at 6:47 PM David Gow <davidgow@google.com> wrote:
>
> Glad this is back out there: a couple of minor nitpicks below:
>
> On Thu, Jun 25, 2020 at 4:58 AM Brendan Higgins
> <brendanhiggins@google.com> wrote:
> >
> > ## TL;DR
> >
> > This patchset adds a centralized executor to dispatch tests rather than
> > relying on late_initcall to schedule each test suite separately along
> > with a couple of new features that depend on it.
> >
> > Also, sorry for the extreme delay in getting this out. Part of the delay
> > came from finding that there were actually several architectures that
> > the previous revision of this patchset didn't work on, so I went through
> > and attempted to test this patchset on every architecture - more on that
> > later.
> >
> > ## What am I trying to do?
> >
> > Conceptually, I am trying to provide a mechanism by which test suites
> > can be grouped together so that they can be reasoned about collectively.
> > The last two of three patches in this series add features which depend
> > on this:
> >
> > PATCH 8/11 Prints out a test plan[1] right before KUnit tests are run;
> > this is valuable because it makes it possible for a test
> > harness to detect whether the number of tests run matches the
> > number of tests expected to be run, ensuring that no tests
> > silently failed. The test plan includes a count of tests that
> > will run. With the centralized executor, the tests are
> > located in a single data structure and thus can be counted.
> >
>
> This appears to actually be patch 9/11.
>
> > PATCH 9/11 Add a new kernel command-line option which allows the user to
> > specify that the kernel poweroff, halt, or reboot after
> > completing all KUnit tests; this is very handy for running
> > KUnit tests on UML or a VM so that the UML/VM process exits
> > cleanly immediately after running all tests without needing a
> > special initramfs. The centralized executor provides a
> > definitive point when all tests have completed and the
> > poweroff, halt, or reboot could occur.
>
> This seems to have been merged into the above patch (9/11).
Whoops, good catch.
Fixed in v5!
> > In addition, by dispatching tests from a single location, we can
> > guarantee that all KUnit tests run after late_init is complete, which
> > was a concern during the initial KUnit patchset review (this has not
> > been a problem in practice, but resolving with certainty is nevertheless
> > desirable).
> >
> > Other use cases for this exist, but the above features should provide an
> > idea of the value that this could provide.
> >
> > ## Changes since last revision:
> > - On the last revision I got some messages from 0day that showed that
> > this patchset didn't work on several architectures, one issue that
> > this patchset addresses is that we were aligning both memory segments
> > as well as structures in the segments to specific byte boundaries
> > which was incorrect.
> > - The issue mentioned above also caused me to test on additional
> > architectures which revealed that some architectures other than UML
> > do not use the default init linker section macro that most
> > architectures use. There are now several new patches (2, 3, 4, and
> > 6).
> > - Fixed a formatting consistency issue in the kernel params
> > documentation patch (9/9).
> > - Add a brief blurb on how and when the kunit_test_suite macro works.
> >
> > ## Remaining work to be done:
> >
> > The only architecture for which I was able to get a compiler, but was
> > apparently unable to get KUnit into a section that the executor to see
> > was m68k - not sure why.
> >
> > Alan Maguire (1):
> > kunit: test: create a single centralized executor for all tests
> >
> > Brendan Higgins (10):
> > vmlinux.lds.h: add linker section for KUnit test suites
> > arch: arm64: add linker section for KUnit test suites
> > arch: microblaze: add linker section for KUnit test suites
> > arch: powerpc: add linker section for KUnit test suites
> > arch: um: add linker section for KUnit test suites
> > arch: xtensa: add linker section for KUnit test suites
> > init: main: add KUnit to kernel init
> > kunit: test: add test plan to KUnit TAP format
> > Documentation: Add kunit_shutdown to kernel-parameters.txt
> > Documentation: kunit: add a brief blurb about kunit_test_suite
> >
> > .../admin-guide/kernel-parameters.txt | 8 ++
> > Documentation/dev-tools/kunit/usage.rst | 5 ++
> > arch/arm64/kernel/vmlinux.lds.S | 3 +
> > arch/microblaze/kernel/vmlinux.lds.S | 4 +
> > arch/powerpc/kernel/vmlinux.lds.S | 4 +
> > arch/um/include/asm/common.lds.S | 4 +
> > arch/xtensa/kernel/vmlinux.lds.S | 4 +
> > include/asm-generic/vmlinux.lds.h | 8 ++
> > include/kunit/test.h | 73 ++++++++++++-----
> > init/main.c | 4 +
> > lib/kunit/Makefile | 3 +-
> > lib/kunit/executor.c | 63 +++++++++++++++
> > lib/kunit/test.c | 13 +--
> > tools/testing/kunit/kunit_kernel.py | 2 +-
> > tools/testing/kunit/kunit_parser.py | 74 +++++++++++++++---
> > .../test_is_test_passed-all_passed.log | Bin 1562 -> 1567 bytes
> > .../test_data/test_is_test_passed-crash.log | Bin 3016 -> 3021 bytes
> > .../test_data/test_is_test_passed-failure.log | Bin 1700 -> 1705 bytes
> > 18 files changed, 226 insertions(+), 46 deletions(-)
> > create mode 100644 lib/kunit/executor.c
> >
> >
> > base-commit: 4333a9b0b67bb4e8bcd91bdd80da80b0ec151162
> > prerequisite-patch-id: 2d4b5aa9fa8ada9ae04c8584b47c299a822b9455
> > prerequisite-patch-id: 582b6d9d28ce4b71628890ec832df6522ca68de0
> >
> > These patches are available for download with dependencies here:
> >
> > https://kunit-review.googlesource.com/c/linux/+/3829
> >
> > [1] https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-14-specification.md#the-plan
> > [2] https://patchwork.kernel.org/patch/11383635/
> >
> > --
> > 2.27.0.212.ge8ba1cc988-goog
> >
next prev parent reply other threads:[~2020-06-26 21:10 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-06-24 20:55 [PATCH v4 00/11] kunit: create a centralized executor to dispatch all KUnit tests Brendan Higgins
2020-06-24 20:55 ` [PATCH v4 01/11] vmlinux.lds.h: add linker section for KUnit test suites Brendan Higgins
2020-06-24 20:55 ` [PATCH v4 02/11] arch: arm64: " Brendan Higgins
2020-06-24 20:55 ` [PATCH v4 03/11] arch: microblaze: " Brendan Higgins
2020-06-24 20:55 ` Brendan Higgins
2020-06-24 20:55 ` [PATCH v4 04/11] arch: powerpc: " Brendan Higgins
2020-06-24 20:55 ` Brendan Higgins
2020-06-24 20:55 ` [PATCH v4 05/11] arch: um: " Brendan Higgins
2020-06-24 20:55 ` Brendan Higgins
2020-06-24 20:55 ` [PATCH v4 06/11] arch: xtensa: " Brendan Higgins
2020-06-24 20:55 ` [PATCH v4 07/11] kunit: test: create a single centralized executor for all tests Brendan Higgins
2020-06-24 20:55 ` [PATCH v4 08/11] init: main: add KUnit to kernel init Brendan Higgins
2020-06-24 20:55 ` [PATCH v4 09/11] kunit: test: add test plan to KUnit TAP format Brendan Higgins
2020-06-24 20:55 ` Brendan Higgins
2020-06-24 20:55 ` [PATCH v4 10/11] Documentation: Add kunit_shutdown to kernel-parameters.txt Brendan Higgins
2020-06-24 20:55 ` Brendan Higgins
2020-06-24 20:55 ` [PATCH v4 11/11] Documentation: kunit: add a brief blurb about kunit_test_suite Brendan Higgins
2020-06-25 1:47 ` [PATCH v4 00/11] kunit: create a centralized executor to dispatch all KUnit tests David Gow
2020-06-25 1:47 ` David Gow
2020-06-26 21:10 ` Brendan Higgins [this message]
2020-06-26 21:10 ` Brendan Higgins
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAFd5g45x04Q+r+1M+kwqkp1DsY9PjJN+Gs0Q+MkOdYTz0SdzKw@mail.gmail.com \
--to=brendanhiggins@google.com \
--cc=akpm@linux-foundation.org \
--cc=alan.maguire@oracle.com \
--cc=anton.ivanov@cambridgegreys.com \
--cc=arnd@arndb.de \
--cc=benh@kernel.crashing.org \
--cc=catalin.marinas@arm.com \
--cc=chris@zankel.net \
--cc=davidgow@google.com \
--cc=frowand.list@gmail.com \
--cc=gregkh@linuxfoundation.org \
--cc=jcmvbkbc@gmail.com \
--cc=jdike@addtoit.com \
--cc=keescook@chromium.org \
--cc=kunit-dev@googlegroups.com \
--cc=linux-arch@vger.kernel.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-um@lists.infradead.org \
--cc=linux-xtensa@linux-xtensa.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=logang@deltatee.com \
--cc=mcgrof@kernel.org \
--cc=monstr@monstr.eu \
--cc=mpe@ellerman.id.au \
--cc=paulus@samba.org \
--cc=richard@nod.at \
--cc=rppt@linux.ibm.com \
--cc=sboyd@kernel.org \
--cc=skhan@linuxfoundation.org \
--cc=will@kernel.org \
--cc=yzaikin@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).