From: frowand.list at gmail.com (Frank Rowand) Subject: [PATCH v2 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework Date: Fri, 10 May 2019 14:52:59 -0700 [thread overview] Message-ID: <2aed675e-0408-c812-3e1a-b90710c528f2@gmail.com> (raw) In-Reply-To: <b09ba170-229b-fde4-3e9a-e50d6ab4c1b5@deltatee.com> On 5/9/19 3:20 PM, Logan Gunthorpe wrote: > > > On 2019-05-09 3:42 p.m., Theodore Ts'o wrote: >> On Thu, May 09, 2019 at 11:12:12AM -0700, Frank Rowand wrote: >>> >>> "My understanding is that the intent of KUnit is to avoid booting a kernel on >>> real hardware or in a virtual machine. That seems to be a matter of semantics >>> to me because isn't invoking a UML Linux just running the Linux kernel in >>> a different form of virtualization? >>> >>> So I do not understand why KUnit is an improvement over kselftest. >>> >>> ... >>> >>> What am I missing?" >> >> One major difference: kselftest requires a userspace environment; >> it starts systemd, requires a root file system from which you can >> load modules, etc. Kunit doesn't require a root file system; >> doesn't require that you start systemd; doesn't allow you to run >> arbitrary perl, python, bash, etc. scripts. As such, it's much >> lighter weight than kselftest, and will have much less overhead >> before you can start running tests. So it's not really the same >> kind of virtualization. I'm back to reply to this subthread, after a delay, as promised. > I largely agree with everything Ted has said in this thread, but I > wonder if we are conflating two different ideas that is causing an > impasse. From what I see, Kunit actually provides two different > things: > 1) An execution environment that can be run very quickly in userspace > on tests in the kernel source. This speeds up the tests and gives a > lot of benefit to developers using those tests because they can get > feedback on their code changes a *lot* quicker. kselftest in-kernel tests provide exactly the same when the tests are configured as "built-in" code instead of as modules. > 2) A framework to write unit tests that provides a lot of the same > facilities as other common unit testing frameworks from userspace > (ie. a runner that runs a list of tests and a bunch of helpers such > as KUNIT_EXPECT_* to simplify test passes and failures). > The first item from Kunit is novel and I see absolutely no overlap > with anything kselftest does. It's also the valuable thing I'd like > to see merged and grow. The first item exists in kselftest. > The second item, arguably, does have significant overlap with > kselftest. Whether you are running short tests in a light weight UML > environment or higher level tests in an heavier VM the two could be > using the same framework for writing or defining in-kernel tests. It > *may* also be valuable for some people to be able to run all the UML > tests in the heavy VM environment along side other higher level > tests. > > Looking at the selftests tree in the repo, we already have similar > items to what Kunit is adding as I described in point (2) above. > kselftest_harness.h contains macros like EXPECT_* and ASSERT_* with > very similar intentions to the new KUNIT_EXECPT_* and KUNIT_ASSERT_* > macros. I might be wrong here because I have not dug deeply enough into the code!!! Does this framework apply to the userspace tests, the in-kernel tests, or both? My "not having dug enough GUESS" is that these are for the user space tests (although if so, they could be extended for in-kernel use also). So I think this one maybe does not have an overlap between KUnit and kselftest. > However, the number of users of this harness appears to be quite > small. Most of the code in the selftests tree seems to be a random > mismash of scripts and userspace code so it's not hard to see it as > something completely different from the new Kunit: > $ git grep --files-with-matches kselftest_harness.h * > Documentation/dev-tools/kselftest.rst > MAINTAINERS > tools/testing/selftests/kselftest_harness.h > tools/testing/selftests/net/tls.c > tools/testing/selftests/rtc/rtctest.c > tools/testing/selftests/seccomp/Makefile > tools/testing/selftests/seccomp/seccomp_bpf.c > tools/testing/selftests/uevent/Makefile > tools/testing/selftests/uevent/uevent_filtering.c > Thus, I can personally see a lot of value in integrating the kunit > test framework with this kselftest harness. There's only a small > number of users of the kselftest harness today, so one way or another > it seems like getting this integrated early would be a good idea. > Letting Kunit and Kselftests progress independently for a few years > will only make this worse and may become something we end up > regretting. Yes, this I agree with. -Frank > > Logan
WARNING: multiple messages have this Message-ID (diff)
From: frowand.list@gmail.com (Frank Rowand) Subject: [PATCH v2 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework Date: Fri, 10 May 2019 14:52:59 -0700 [thread overview] Message-ID: <2aed675e-0408-c812-3e1a-b90710c528f2@gmail.com> (raw) Message-ID: <20190510215259.9O82-uamKam-RoVE95Pam3r-k0zbRpa5tbti1-mMncE@z> (raw) In-Reply-To: <b09ba170-229b-fde4-3e9a-e50d6ab4c1b5@deltatee.com> On 5/9/19 3:20 PM, Logan Gunthorpe wrote: > > > On 2019-05-09 3:42 p.m., Theodore Ts'o wrote: >> On Thu, May 09, 2019@11:12:12AM -0700, Frank Rowand wrote: >>> >>> "My understanding is that the intent of KUnit is to avoid booting a kernel on >>> real hardware or in a virtual machine. That seems to be a matter of semantics >>> to me because isn't invoking a UML Linux just running the Linux kernel in >>> a different form of virtualization? >>> >>> So I do not understand why KUnit is an improvement over kselftest. >>> >>> ... >>> >>> What am I missing?" >> >> One major difference: kselftest requires a userspace environment; >> it starts systemd, requires a root file system from which you can >> load modules, etc. Kunit doesn't require a root file system; >> doesn't require that you start systemd; doesn't allow you to run >> arbitrary perl, python, bash, etc. scripts. As such, it's much >> lighter weight than kselftest, and will have much less overhead >> before you can start running tests. So it's not really the same >> kind of virtualization. I'm back to reply to this subthread, after a delay, as promised. > I largely agree with everything Ted has said in this thread, but I > wonder if we are conflating two different ideas that is causing an > impasse. From what I see, Kunit actually provides two different > things: > 1) An execution environment that can be run very quickly in userspace > on tests in the kernel source. This speeds up the tests and gives a > lot of benefit to developers using those tests because they can get > feedback on their code changes a *lot* quicker. kselftest in-kernel tests provide exactly the same when the tests are configured as "built-in" code instead of as modules. > 2) A framework to write unit tests that provides a lot of the same > facilities as other common unit testing frameworks from userspace > (ie. a runner that runs a list of tests and a bunch of helpers such > as KUNIT_EXPECT_* to simplify test passes and failures). > The first item from Kunit is novel and I see absolutely no overlap > with anything kselftest does. It's also the valuable thing I'd like > to see merged and grow. The first item exists in kselftest. > The second item, arguably, does have significant overlap with > kselftest. Whether you are running short tests in a light weight UML > environment or higher level tests in an heavier VM the two could be > using the same framework for writing or defining in-kernel tests. It > *may* also be valuable for some people to be able to run all the UML > tests in the heavy VM environment along side other higher level > tests. > > Looking at the selftests tree in the repo, we already have similar > items to what Kunit is adding as I described in point (2) above. > kselftest_harness.h contains macros like EXPECT_* and ASSERT_* with > very similar intentions to the new KUNIT_EXECPT_* and KUNIT_ASSERT_* > macros. I might be wrong here because I have not dug deeply enough into the code!!! Does this framework apply to the userspace tests, the in-kernel tests, or both? My "not having dug enough GUESS" is that these are for the user space tests (although if so, they could be extended for in-kernel use also). So I think this one maybe does not have an overlap between KUnit and kselftest. > However, the number of users of this harness appears to be quite > small. Most of the code in the selftests tree seems to be a random > mismash of scripts and userspace code so it's not hard to see it as > something completely different from the new Kunit: > $ git grep --files-with-matches kselftest_harness.h * > Documentation/dev-tools/kselftest.rst > MAINTAINERS > tools/testing/selftests/kselftest_harness.h > tools/testing/selftests/net/tls.c > tools/testing/selftests/rtc/rtctest.c > tools/testing/selftests/seccomp/Makefile > tools/testing/selftests/seccomp/seccomp_bpf.c > tools/testing/selftests/uevent/Makefile > tools/testing/selftests/uevent/uevent_filtering.c > Thus, I can personally see a lot of value in integrating the kunit > test framework with this kselftest harness. There's only a small > number of users of the kselftest harness today, so one way or another > it seems like getting this integrated early would be a good idea. > Letting Kunit and Kselftests progress independently for a few years > will only make this worse and may become something we end up > regretting. Yes, this I agree with. -Frank > > Logan
next prev parent reply other threads:[~2019-05-10 21:52 UTC|newest] Thread overview: 262+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-05-01 23:01 [PATCH v2 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 01/17] kunit: test: add KUnit test runner core brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 02/17] kunit: test: add test resource management API brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 03/17] kunit: test: add string_stream a std::stream like string builder brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-03 1:26 ` shuah 2019-05-03 1:26 ` shuah 2019-05-03 4:37 ` brendanhiggins 2019-05-03 4:37 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 04/17] kunit: test: add kunit_stream a std::stream like logger brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-02 11:00 ` gregkh 2019-05-02 11:00 ` Greg KH 2019-05-02 20:25 ` brendanhiggins 2019-05-02 20:25 ` Brendan Higgins 2019-05-02 21:18 ` frowand.list 2019-05-02 21:18 ` Frank Rowand 2019-05-03 1:50 ` shuah 2019-05-03 1:50 ` shuah 2019-05-03 5:48 ` brendanhiggins 2019-05-03 5:48 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 05/17] kunit: test: add the concept of expectations brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 06/17] kbuild: enable building KUnit brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-10 3:03 ` yamada.masahiro 2019-05-10 3:03 ` Masahiro Yamada 2019-05-10 10:27 ` brendanhiggins 2019-05-10 10:27 ` Brendan Higgins 2019-05-10 10:30 ` yamada.masahiro 2019-05-10 10:30 ` Masahiro Yamada 2019-05-10 10:33 ` brendanhiggins 2019-05-10 10:33 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 07/17] kunit: test: add initial tests brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-02 10:58 ` gregkh 2019-05-02 10:58 ` Greg KH 2019-05-02 20:30 ` brendanhiggins 2019-05-02 20:30 ` Brendan Higgins 2019-05-03 1:27 ` shuah 2019-05-03 1:27 ` shuah 2019-05-03 5:18 ` brendanhiggins 2019-05-03 5:18 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 08/17] kunit: test: add support for test abort brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-03 3:14 ` logang 2019-05-03 3:14 ` Logan Gunthorpe 2019-05-03 6:48 ` brendanhiggins 2019-05-03 6:48 ` Brendan Higgins 2019-05-03 12:33 ` logang 2019-05-03 12:33 ` Logan Gunthorpe 2019-05-06 8:48 ` brendanhiggins 2019-05-06 8:48 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 09/17] kunit: test: add tests for kunit " brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 10/17] kunit: test: add the concept of assertions brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 11/17] kunit: test: add test managed resource tests brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-03 14:34 ` shuah 2019-05-03 14:34 ` shuah 2019-05-06 9:03 ` brendanhiggins 2019-05-06 9:03 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 12/17] kunit: tool: add Python wrappers for running KUnit tests brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-02 11:02 ` gregkh 2019-05-02 11:02 ` Greg KH 2019-05-02 18:07 ` brendanhiggins 2019-05-02 18:07 ` Brendan Higgins 2019-05-02 21:16 ` frowand.list 2019-05-02 21:16 ` Frank Rowand 2019-05-02 23:45 ` brendanhiggins 2019-05-02 23:45 ` Brendan Higgins 2019-05-03 1:45 ` frowand.list 2019-05-03 1:45 ` Frank Rowand 2019-05-03 5:36 ` brendanhiggins 2019-05-03 5:36 ` Brendan Higgins 2019-05-03 18:59 ` frowand.list 2019-05-03 18:59 ` Frank Rowand 2019-05-03 23:14 ` brendanhiggins 2019-05-03 23:14 ` Brendan Higgins 2019-05-04 10:42 ` gregkh 2019-05-04 10:42 ` Greg KH 2019-05-06 0:19 ` frowand.list 2019-05-06 0:19 ` Frank Rowand 2019-05-06 17:43 ` keescook 2019-05-06 17:43 ` Kees Cook 2019-05-06 21:42 ` brendanhiggins 2019-05-06 21:42 ` Brendan Higgins 2019-05-06 21:39 ` brendanhiggins 2019-05-06 21:39 ` Brendan Higgins 2019-05-07 19:13 ` Tim.Bird 2019-05-07 19:13 ` Tim.Bird 2019-05-03 6:41 ` gregkh 2019-05-03 6:41 ` Greg KH 2019-05-01 23:01 ` [PATCH v2 13/17] kunit: defconfig: add defconfigs for building " brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 14/17] Documentation: kunit: add documentation for KUnit brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-09 5:08 ` rdunlap 2019-05-09 5:08 ` Randy Dunlap 2019-05-09 17:38 ` brendanhiggins 2019-05-09 17:38 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 15/17] MAINTAINERS: add entry for KUnit the unit testing framework brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-03 14:38 ` shuah 2019-05-03 14:38 ` shuah 2019-05-06 9:18 ` brendanhiggins 2019-05-06 9:18 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 16/17] kernel/sysctl-test: Add null pointer test for sysctl.c:proc_dointvec() brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-02 11:03 ` gregkh 2019-05-02 11:03 ` Greg KH 2019-05-02 18:14 ` Tim.Bird 2019-05-02 18:14 ` Tim.Bird 2019-05-02 18:45 ` brendanhiggins 2019-05-02 18:45 ` Brendan Higgins 2019-05-03 6:42 ` gregkh 2019-05-03 6:42 ` Greg KH 2019-05-03 23:41 ` brendanhiggins 2019-05-03 23:41 ` Brendan Higgins 2019-05-04 10:40 ` gregkh 2019-05-04 10:40 ` Greg KH 2019-05-01 23:01 ` [PATCH v2 17/17] MAINTAINERS: add proc sysctl KUnit test to PROC SYSCTL section brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-02 10:50 ` [PATCH v2 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework gregkh 2019-05-02 10:50 ` Greg KH 2019-05-02 11:05 ` gregkh 2019-05-02 11:05 ` Greg KH 2019-05-03 0:41 ` brendanhiggins 2019-05-03 0:41 ` Brendan Higgins 2019-05-02 14:04 ` shuah 2019-05-02 14:04 ` shuah 2019-05-03 0:44 ` brendanhiggins 2019-05-03 0:44 ` Brendan Higgins 2019-05-03 3:18 ` logang 2019-05-03 3:18 ` Logan Gunthorpe 2019-05-07 3:14 ` frowand.list 2019-05-07 3:14 ` Frank Rowand 2019-05-07 8:01 ` gregkh 2019-05-07 8:01 ` Greg KH 2019-05-07 15:23 ` shuah 2019-05-07 15:23 ` shuah 2019-05-09 1:01 ` frowand.list 2019-05-09 1:01 ` Frank Rowand 2019-05-07 17:22 ` tytso 2019-05-07 17:22 ` Theodore Ts'o 2019-05-08 19:17 ` brendanhiggins 2019-05-08 19:17 ` Brendan Higgins 2019-05-09 0:58 ` frowand.list 2019-05-09 0:58 ` Frank Rowand 2019-05-09 1:44 ` tytso 2019-05-09 1:44 ` Theodore Ts'o 2019-05-09 2:18 ` frowand.list 2019-05-09 2:18 ` Frank Rowand 2019-05-14 8:22 ` brendanhiggins 2019-05-14 8:22 ` Brendan Higgins 2019-05-09 0:43 ` frowand.list 2019-05-09 0:43 ` Frank Rowand 2019-05-09 1:58 ` tytso 2019-05-09 1:58 ` Theodore Ts'o 2019-05-09 2:13 ` frowand.list 2019-05-09 2:13 ` Frank Rowand 2019-05-09 3:20 ` tytso 2019-05-09 3:20 ` Theodore Ts'o 2019-05-09 11:52 ` knut.omang 2019-05-09 11:52 ` Knut Omang 2019-05-09 13:35 ` tytso 2019-05-09 13:35 ` Theodore Ts'o 2019-05-09 14:48 ` knut.omang 2019-05-09 14:48 ` Knut Omang 2019-05-09 17:00 ` Tim.Bird 2019-05-09 17:00 ` Tim.Bird 2019-05-09 17:42 ` daniel 2019-05-09 17:42 ` Daniel Vetter 2019-05-09 18:12 ` frowand.list 2019-05-09 18:12 ` Frank Rowand 2019-05-09 21:42 ` tytso 2019-05-09 21:42 ` Theodore Ts'o 2019-05-09 22:20 ` logang 2019-05-09 22:20 ` Logan Gunthorpe 2019-05-09 23:30 ` tytso 2019-05-09 23:30 ` Theodore Ts'o 2019-05-09 23:40 ` logang 2019-05-09 23:40 ` Logan Gunthorpe 2019-05-10 4:47 ` tytso 2019-05-10 4:47 ` Theodore Ts'o 2019-05-10 5:18 ` frowand.list 2019-05-10 5:18 ` Frank Rowand 2019-05-10 5:48 ` knut.omang 2019-05-10 5:48 ` Knut Omang 2019-05-10 8:12 ` daniel 2019-05-10 8:12 ` Daniel Vetter 2019-05-10 10:23 ` brendanhiggins 2019-05-10 10:23 ` Brendan Higgins 2019-05-10 12:12 ` knut.omang 2019-05-10 12:12 ` Knut Omang 2019-05-10 20:54 ` brendanhiggins 2019-05-10 20:54 ` Brendan Higgins 2019-05-10 22:18 ` frowand.list 2019-05-10 22:18 ` Frank Rowand 2019-05-11 6:17 ` knut.omang 2019-05-11 6:17 ` Knut Omang 2019-05-14 6:39 ` brendanhiggins 2019-05-14 6:39 ` Brendan Higgins 2019-05-10 21:59 ` frowand.list 2019-05-10 21:59 ` Frank Rowand 2019-05-11 6:43 ` knut.omang 2019-05-11 6:43 ` Knut Omang 2019-05-14 8:00 ` brendanhiggins 2019-05-14 8:00 ` Brendan Higgins 2019-05-10 11:36 ` knut.omang 2019-05-10 11:36 ` Knut Omang 2019-05-10 16:17 ` logang 2019-05-10 16:17 ` Logan Gunthorpe 2019-05-10 22:13 ` frowand.list 2019-05-10 22:13 ` Frank Rowand 2019-05-14 8:38 ` brendanhiggins 2019-05-14 8:38 ` Brendan Higgins 2019-05-15 0:14 ` frowand.list 2019-05-15 0:14 ` Frank Rowand 2019-05-15 0:26 ` logang 2019-05-15 0:26 ` Logan Gunthorpe 2019-05-10 21:52 ` frowand.list [this message] 2019-05-10 21:52 ` Frank Rowand 2019-05-14 20:54 ` brendanhiggins 2019-05-14 20:54 ` Brendan Higgins 2019-05-10 21:12 ` frowand.list 2019-05-10 21:12 ` Frank Rowand 2019-05-11 17:33 ` tytso 2019-05-11 17:33 ` Theodore Ts'o 2019-05-13 14:44 ` daniel 2019-05-13 14:44 ` Daniel Vetter 2019-05-14 6:04 ` brendanhiggins 2019-05-14 6:04 ` Brendan Higgins 2019-05-14 12:05 ` daniel 2019-05-14 12:05 ` Daniel Vetter 2019-05-14 18:36 ` brendanhiggins 2019-05-14 18:36 ` Brendan Higgins 2019-05-15 7:41 ` daniel 2019-05-15 7:41 ` Daniel Vetter 2019-05-22 21:38 ` brendanhiggins 2019-05-22 21:38 ` Brendan Higgins 2019-05-23 8:40 ` daniel 2019-05-23 8:40 ` Daniel Vetter 2019-05-15 0:26 ` frowand.list 2019-05-15 0:26 ` Frank Rowand 2019-05-15 4:28 ` tytso 2019-05-15 4:28 ` Theodore Ts'o 2019-05-10 5:11 ` frowand.list 2019-05-10 5:11 ` Frank Rowand 2019-05-10 10:43 ` tytso 2019-05-10 10:43 ` Theodore Ts'o 2019-05-10 21:05 ` frowand.list 2019-05-10 21:05 ` Frank Rowand 2019-05-09 15:19 ` yamada.masahiro 2019-05-09 15:19 ` Masahiro Yamada 2019-05-10 10:25 ` brendanhiggins 2019-05-10 10:25 ` Brendan Higgins
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=2aed675e-0408-c812-3e1a-b90710c528f2@gmail.com \ --to=linux-kselftest@vger.kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).