From: Brendan Higgins <brendanhiggins@google.com> To: jdike@addtoit.com, richard@nod.at, anton.ivanov@cambridgegreys.com, arnd@arndb.de, keescook@chromium.org, skhan@linuxfoundation.org, alan.maguire@oracle.com, yzaikin@google.com, davidgow@google.com, akpm@linux-foundation.org, rppt@linux.ibm.com Cc: gregkh@linuxfoundation.org, sboyd@kernel.org, logang@deltatee.com, mcgrof@kernel.org, knut.omang@oracle.com, linux-um@lists.infradead.org, linux-arch@vger.kernel.org, linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, Brendan Higgins <brendanhiggins@google.com> Subject: [RFC v1 0/6] kunit: create a centralized executor to dispatch all KUnit tests Date: Mon, 16 Dec 2019 14:05:49 -0800 [thread overview] Message-ID: <20191216220555.245089-1-brendanhiggins@google.com> (raw) ## TL;DR This patchset adds a centralized executor to dispatch tests rather than relying on late_initcall to schedule each test suite separately along with a couple of new features that depend on it. ## What am I trying to do? Conceptually, I am trying to provide a mechanism by which test suites can be grouped together so that they can be reasoned about collectively. The last two patches in this series add features which depend on this: RFC 5/6 Prints out a test plan right before KUnit tests are run[1]; this is valuable because it makes it possible for a test harness to detect whether the number of tests run matches the number of tests expected to be run, ensuring that no tests silently failed. RFC 6/6 Add a new kernel command-line option which allows the user to specify that the kernel poweroff, halt, or reboot after completing all KUnit tests; this is very handy for running KUnit tests on UML or a VM so that the UML/VM process exits cleanly immediately after running all tests without needing a special initramfs. In addition, by dispatching tests from a single location, we can guarantee that all KUnit tests run after late_init is complete, which was a concern during the initial KUnit patchset review (this has not been a problem in practice, but resolving with certainty is nevertheless desirable). Other use cases for this exist, but the above features should provide an idea of the value that this could provide. ## What work remains to be done? These patches were based on patches in our non-upstream branch[2], so we have a pretty good idea that they are useable as presented; nevertheless, some of the changes done in this patchset could *definitely* use some review by subsystem experts (linker scripts, init, etc), and will likely change a lot after getting feedback. The biggest thing that I know will require additional attention is integrating this patchset with the KUnit module support patchset[3]. I have not even attempted to build these patches on top of the module support patches as I would like to get people's initial thoughts first (especially Alan's :-) ). I think that making these patches work with module support should be fairly straight forward, nevertheless. Brendan Higgins (5): vmlinux.lds.h: add linker section for KUnit test suites arch: um: add linker section for KUnit test suites kunit: test: create a single centralized executor for all tests init: main: add KUnit to kernel init kunit: test: add test plan to KUnit TAP format David Gow (1): kunit: Add 'kunit_shutdown' option arch/um/include/asm/common.lds.S | 4 + include/asm-generic/vmlinux.lds.h | 8 ++ include/kunit/test.h | 16 ++-- init/main.c | 4 + lib/kunit/Makefile | 3 +- lib/kunit/executor.c | 74 ++++++++++++++++++ lib/kunit/test.c | 11 --- tools/testing/kunit/kunit_kernel.py | 2 +- tools/testing/kunit/kunit_parser.py | 76 +++++++++++++++---- .../test_is_test_passed-all_passed.log | 1 + .../test_data/test_is_test_passed-crash.log | 1 + .../test_data/test_is_test_passed-failure.log | 1 + 12 files changed, 170 insertions(+), 31 deletions(-) create mode 100644 lib/kunit/executor.c [1]: https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-14-specification.md#the-plan [2]: https://kunit-review.googlesource.com/c/linux/+/1037 [3]: https://patchwork.kernel.org/project/linux-kselftest/list/?series=211727 -- 2.24.1.735.g03f4e72817-goog
WARNING: multiple messages have this Message-ID (diff)
From: Brendan Higgins <brendanhiggins@google.com> To: jdike@addtoit.com, richard@nod.at, anton.ivanov@cambridgegreys.com, arnd@arndb.de, keescook@chromium.org, skhan@linuxfoundation.org, alan.maguire@oracle.com, yzaikin@google.com, davidgow@google.com, akpm@linux-foundation.org, rppt@linux.ibm.com Cc: linux-arch@vger.kernel.org, sboyd@kernel.org, gregkh@linuxfoundation.org, knut.omang@oracle.com, linux-kernel@vger.kernel.org, linux-um@lists.infradead.org, mcgrof@kernel.org, linux-kselftest@vger.kernel.org, logang@deltatee.com, Brendan Higgins <brendanhiggins@google.com>, kunit-dev@googlegroups.com Subject: [RFC v1 0/6] kunit: create a centralized executor to dispatch all KUnit tests Date: Mon, 16 Dec 2019 14:05:49 -0800 [thread overview] Message-ID: <20191216220555.245089-1-brendanhiggins@google.com> (raw) ## TL;DR This patchset adds a centralized executor to dispatch tests rather than relying on late_initcall to schedule each test suite separately along with a couple of new features that depend on it. ## What am I trying to do? Conceptually, I am trying to provide a mechanism by which test suites can be grouped together so that they can be reasoned about collectively. The last two patches in this series add features which depend on this: RFC 5/6 Prints out a test plan right before KUnit tests are run[1]; this is valuable because it makes it possible for a test harness to detect whether the number of tests run matches the number of tests expected to be run, ensuring that no tests silently failed. RFC 6/6 Add a new kernel command-line option which allows the user to specify that the kernel poweroff, halt, or reboot after completing all KUnit tests; this is very handy for running KUnit tests on UML or a VM so that the UML/VM process exits cleanly immediately after running all tests without needing a special initramfs. In addition, by dispatching tests from a single location, we can guarantee that all KUnit tests run after late_init is complete, which was a concern during the initial KUnit patchset review (this has not been a problem in practice, but resolving with certainty is nevertheless desirable). Other use cases for this exist, but the above features should provide an idea of the value that this could provide. ## What work remains to be done? These patches were based on patches in our non-upstream branch[2], so we have a pretty good idea that they are useable as presented; nevertheless, some of the changes done in this patchset could *definitely* use some review by subsystem experts (linker scripts, init, etc), and will likely change a lot after getting feedback. The biggest thing that I know will require additional attention is integrating this patchset with the KUnit module support patchset[3]. I have not even attempted to build these patches on top of the module support patches as I would like to get people's initial thoughts first (especially Alan's :-) ). I think that making these patches work with module support should be fairly straight forward, nevertheless. Brendan Higgins (5): vmlinux.lds.h: add linker section for KUnit test suites arch: um: add linker section for KUnit test suites kunit: test: create a single centralized executor for all tests init: main: add KUnit to kernel init kunit: test: add test plan to KUnit TAP format David Gow (1): kunit: Add 'kunit_shutdown' option arch/um/include/asm/common.lds.S | 4 + include/asm-generic/vmlinux.lds.h | 8 ++ include/kunit/test.h | 16 ++-- init/main.c | 4 + lib/kunit/Makefile | 3 +- lib/kunit/executor.c | 74 ++++++++++++++++++ lib/kunit/test.c | 11 --- tools/testing/kunit/kunit_kernel.py | 2 +- tools/testing/kunit/kunit_parser.py | 76 +++++++++++++++---- .../test_is_test_passed-all_passed.log | 1 + .../test_data/test_is_test_passed-crash.log | 1 + .../test_data/test_is_test_passed-failure.log | 1 + 12 files changed, 170 insertions(+), 31 deletions(-) create mode 100644 lib/kunit/executor.c [1]: https://github.com/isaacs/testanything.github.io/blob/tap14/tap-version-14-specification.md#the-plan [2]: https://kunit-review.googlesource.com/c/linux/+/1037 [3]: https://patchwork.kernel.org/project/linux-kselftest/list/?series=211727 -- 2.24.1.735.g03f4e72817-goog _______________________________________________ linux-um mailing list linux-um@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-um
next reply other threads:[~2019-12-16 22:06 UTC|newest] Thread overview: 93+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-12-16 22:05 Brendan Higgins [this message] 2019-12-16 22:05 ` [RFC v1 0/6] kunit: create a centralized executor to dispatch all KUnit tests Brendan Higgins 2019-12-16 22:05 ` [RFC v1 1/6] vmlinux.lds.h: add linker section for KUnit test suites Brendan Higgins 2019-12-16 22:05 ` Brendan Higgins 2019-12-17 8:21 ` Stephen Boyd 2019-12-17 8:21 ` Stephen Boyd 2019-12-17 8:21 ` Stephen Boyd 2019-12-16 22:05 ` [RFC v1 2/6] arch: um: " Brendan Higgins 2019-12-16 22:05 ` Brendan Higgins 2019-12-17 8:21 ` Stephen Boyd 2019-12-17 8:21 ` Stephen Boyd 2019-12-17 8:21 ` Stephen Boyd 2019-12-16 22:05 ` [RFC v1 3/6] kunit: test: create a single centralized executor for all tests Brendan Higgins 2019-12-16 22:05 ` Brendan Higgins 2019-12-17 8:04 ` Stephen Boyd 2019-12-17 8:04 ` Stephen Boyd 2019-12-17 8:04 ` Stephen Boyd 2020-01-23 22:54 ` Brendan Higgins 2020-01-23 22:54 ` Brendan Higgins 2020-01-23 22:54 ` Brendan Higgins 2020-01-23 22:54 ` Brendan Higgins 2019-12-16 22:05 ` [RFC v1 4/6] init: main: add KUnit to kernel init Brendan Higgins 2019-12-16 22:05 ` Brendan Higgins 2019-12-17 7:58 ` Stephen Boyd 2019-12-17 7:58 ` Stephen Boyd 2019-12-17 7:58 ` Stephen Boyd 2020-01-23 22:45 ` Brendan Higgins 2020-01-23 22:45 ` Brendan Higgins 2020-01-23 22:45 ` Brendan Higgins 2020-01-23 22:45 ` Brendan Higgins 2019-12-16 22:05 ` [RFC v1 5/6] kunit: test: add test plan to KUnit TAP format Brendan Higgins 2019-12-16 22:05 ` Brendan Higgins 2019-12-17 8:18 ` Stephen Boyd 2019-12-17 8:18 ` Stephen Boyd 2019-12-17 8:18 ` Stephen Boyd 2019-12-16 22:05 ` [RFC v1 6/6] kunit: Add 'kunit_shutdown' option Brendan Higgins 2019-12-16 22:05 ` Brendan Higgins 2019-12-17 8:06 ` Stephen Boyd 2019-12-17 8:06 ` Stephen Boyd 2019-12-17 8:06 ` Stephen Boyd 2020-01-23 22:56 ` Brendan Higgins 2020-01-23 22:56 ` Brendan Higgins 2020-01-23 22:56 ` Brendan Higgins 2020-01-23 22:56 ` Brendan Higgins 2020-01-06 22:40 ` [RFC v1 0/6] kunit: create a centralized executor to dispatch all KUnit tests Luis Chamberlain 2020-01-06 22:40 ` Luis Chamberlain 2020-01-23 22:40 ` Brendan Higgins 2020-01-23 22:40 ` Brendan Higgins 2020-01-23 22:40 ` Brendan Higgins 2020-01-23 22:40 ` Brendan Higgins 2020-01-27 17:40 ` Frank Rowand 2020-01-27 17:40 ` Frank Rowand 2020-01-27 17:40 ` Frank Rowand 2020-01-27 17:40 ` Frank Rowand 2020-01-28 7:19 ` Brendan Higgins 2020-01-28 7:19 ` Brendan Higgins 2020-01-28 7:19 ` Brendan Higgins 2020-01-28 7:19 ` Brendan Higgins 2020-01-28 18:36 ` Frank Rowand 2020-01-28 18:36 ` Frank Rowand 2020-01-28 18:36 ` Frank Rowand 2020-01-28 18:36 ` Frank Rowand 2020-01-28 19:35 ` Tim.Bird 2020-01-28 19:35 ` Tim.Bird 2020-01-28 19:35 ` Tim.Bird 2020-01-28 19:53 ` Brendan Higgins 2020-01-28 19:53 ` Brendan Higgins 2020-01-28 19:53 ` Brendan Higgins 2020-01-28 19:53 ` Brendan Higgins 2020-01-29 4:24 ` Frank Rowand 2020-01-29 4:24 ` Frank Rowand 2020-01-29 4:24 ` Frank Rowand 2020-01-29 4:24 ` Frank Rowand 2020-01-29 21:18 ` Brendan Higgins 2020-01-29 21:18 ` Brendan Higgins 2020-01-29 21:18 ` Brendan Higgins 2020-01-29 21:18 ` Brendan Higgins 2020-01-29 13:06 ` Alan Maguire 2020-01-29 13:06 ` Alan Maguire 2020-01-29 13:06 ` Alan Maguire 2020-01-29 13:06 ` Alan Maguire 2020-01-29 21:28 ` Brendan Higgins 2020-01-29 21:28 ` Brendan Higgins 2020-01-29 21:28 ` Brendan Higgins 2020-01-29 21:28 ` Brendan Higgins 2020-03-02 19:45 ` Luis Chamberlain 2020-03-02 19:45 ` Luis Chamberlain 2020-03-02 19:45 ` Luis Chamberlain 2020-03-02 19:45 ` Luis Chamberlain 2020-03-02 19:05 ` Luis Chamberlain 2020-03-02 19:05 ` Luis Chamberlain 2020-03-02 19:05 ` Luis Chamberlain 2020-03-02 19:05 ` Luis Chamberlain
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20191216220555.245089-1-brendanhiggins@google.com \ --to=brendanhiggins@google.com \ --cc=akpm@linux-foundation.org \ --cc=alan.maguire@oracle.com \ --cc=anton.ivanov@cambridgegreys.com \ --cc=arnd@arndb.de \ --cc=davidgow@google.com \ --cc=gregkh@linuxfoundation.org \ --cc=jdike@addtoit.com \ --cc=keescook@chromium.org \ --cc=knut.omang@oracle.com \ --cc=kunit-dev@googlegroups.com \ --cc=linux-arch@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-kselftest@vger.kernel.org \ --cc=linux-um@lists.infradead.org \ --cc=logang@deltatee.com \ --cc=mcgrof@kernel.org \ --cc=richard@nod.at \ --cc=rppt@linux.ibm.com \ --cc=sboyd@kernel.org \ --cc=skhan@linuxfoundation.org \ --cc=yzaikin@google.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.