From: mcgrof at kernel.org (Luis Chamberlain) Subject: [RFC v3 01/19] kunit: test: add KUnit test runner core Date: Fri, 30 Nov 2018 18:57:35 -0800 [thread overview] Message-ID: <20181201025735.GI28501@garbanzo.do-not-panic.com> (raw) In-Reply-To: <CAFd5g47t94RR78_sZ7Xr7ArDh+hJGYHNwmmG2HJb7AW+i=u3ew@mail.gmail.com> On Fri, Nov 30, 2018 at 05:51:11PM -0800, Brendan Higgins wrote: > On Thu, Nov 29, 2018 at 7:14 PM Luis Chamberlain <mcgrof at kernel.org> wrote: > > > > On Wed, Nov 28, 2018 at 11:36:18AM -0800, Brendan Higgins wrote: > > > +#define module_test(module) \ > > > + static int module_kunit_init##module(void) \ > > > + { \ > > > + return kunit_run_tests(&module); \ > > > + } \ > > > + late_initcall(module_kunit_init##module) > > > > Here in lies an assumption that suffices. I'm inclined to believe we > > need new initcall level here so to ensure we *do* run after all the > > respective kernels iniut calls. Otherwise we're left at the whims of > > link order for kunit. For instance if a kunit test relies on frameworks > > which are also late_initcall() we'd have complete incompatibility with > > anything linked *after* kunit. > > Yep, I have some patches that address this, but I thought this is > sufficient for the initial patchset (I figured that's the type of > thing that people will have opinions about so best to get it out of > the critical path). Do you want me to add those in the next revision? > > > > > > diff --git a/kunit/Kconfig b/kunit/Kconfig > > > new file mode 100644 > > > index 0000000000000..49b44c4f6630a > > > --- /dev/null > > > +++ b/kunit/Kconfig > > > @@ -0,0 +1,17 @@ > > > +# > > > +# KUnit base configuration > > > +# > > > + > > > +menu "KUnit support" > > > + > > > +config KUNIT > > > + bool "Enable support for unit tests (KUnit)" > > > + depends on UML > > > > Consider using: > > > > if UML > > ... > > endif > > > > That allows the depends to be done once. > > If you want to eliminate depends, wouldn't it be best to have KUNIT > depend on whatever it needs, and then do `if KUNIT` below that? That > seems cleaner over the long term. Anyway, Kees actually asked me to > change it to the way it is now; I really don't care either way. Yes, that works better. The idea is to just avoid having to write in depends on over and over again. > > I'm a bit conflicted here. This currently depends on UML but yet you > > noted on RFC v2 that your intention is to liberate kunit from UML and > > ideally allow unit tests to depend only on userspace. I've addressed > > tests using both selftests kernels drivers and also re-written kernel > > APIs to userspace to test there. I think we may need to live with both. > > I am not entirely opposed. The greater isolation we can achieve, the > fewer dependencies, and barriers to setting up test fixtures the > better. I think the best way to do that in most cases is allowing > minimal test binaries to be built that have the absolute minimum > amount of code necessary to test the desired property. That being > said, integration tests are a thing and drawing a line between them > and unit tests is not always possible, so supporting other > architectures might be necessary. Then lets pave the way for it to be done easily. > > Then for the UML stuff, I think if we *really* accept that UML will > > always be a viable option we should probably consider now throwing these > > things under drivers/platform/uml/. This follows the pattern of arch > > specific drivers. Whether or not we end up with a complete userspace > > component independent of UML may implicate having a shared component > > somewhere else. > > Fair enough. What specifically are you suggesting should go in > `drivers/platform/uml`? Just the bits that are completely tied to UML > or a concrete architecture? The bits that are UML specific. As I see it, with the above intention clarified, kunit is a framework for architectures and UML is supported first. The code doesn't currently reflect this. > > Likewise, I realize the goal is to *avoid* using a virtual machine for > > these tests, but would it in any way make sense to share kunit to be > > supported for other architectures to allow easier-to-write tests as > > well? > > You are not the first person to ask for this. > > For the vast majority of tests, I think we can (and consequently > should) make them run without any external dependencies. Doing so > makes it such that someone can run a test without knowing anything > about it, which allows you to do a lot of things. For one, I, as a > developer, don't have to hunt down somebody's QEMU patches, or > whatever. But it also means I, as someone maintaining part of the > kernel, can make nice test runners and build things like presubmit > servers on top of them. > > Nevertheless, I accept that there are things which are just easier to > do with hardware or a VM (for integration tests it is necessary). > Still, I think we need to make sure the vast majority of unit tests do > not depend on real hardware or a VM. When possible, sure. Luis
WARNING: multiple messages have this Message-ID (diff)
From: mcgrof@kernel.org (Luis Chamberlain) Subject: [RFC v3 01/19] kunit: test: add KUnit test runner core Date: Fri, 30 Nov 2018 18:57:35 -0800 [thread overview] Message-ID: <20181201025735.GI28501@garbanzo.do-not-panic.com> (raw) Message-ID: <20181201025735.E70l-3FJki_jNjZbU9FSyBaTmzpdw0bDm4lCUc_lpS4@z> (raw) In-Reply-To: <CAFd5g47t94RR78_sZ7Xr7ArDh+hJGYHNwmmG2HJb7AW+i=u3ew@mail.gmail.com> On Fri, Nov 30, 2018@05:51:11PM -0800, Brendan Higgins wrote: > On Thu, Nov 29, 2018@7:14 PM Luis Chamberlain <mcgrof@kernel.org> wrote: > > > > On Wed, Nov 28, 2018@11:36:18AM -0800, Brendan Higgins wrote: > > > +#define module_test(module) \ > > > + static int module_kunit_init##module(void) \ > > > + { \ > > > + return kunit_run_tests(&module); \ > > > + } \ > > > + late_initcall(module_kunit_init##module) > > > > Here in lies an assumption that suffices. I'm inclined to believe we > > need new initcall level here so to ensure we *do* run after all the > > respective kernels iniut calls. Otherwise we're left at the whims of > > link order for kunit. For instance if a kunit test relies on frameworks > > which are also late_initcall() we'd have complete incompatibility with > > anything linked *after* kunit. > > Yep, I have some patches that address this, but I thought this is > sufficient for the initial patchset (I figured that's the type of > thing that people will have opinions about so best to get it out of > the critical path). Do you want me to add those in the next revision? > > > > > > diff --git a/kunit/Kconfig b/kunit/Kconfig > > > new file mode 100644 > > > index 0000000000000..49b44c4f6630a > > > --- /dev/null > > > +++ b/kunit/Kconfig > > > @@ -0,0 +1,17 @@ > > > +# > > > +# KUnit base configuration > > > +# > > > + > > > +menu "KUnit support" > > > + > > > +config KUNIT > > > + bool "Enable support for unit tests (KUnit)" > > > + depends on UML > > > > Consider using: > > > > if UML > > ... > > endif > > > > That allows the depends to be done once. > > If you want to eliminate depends, wouldn't it be best to have KUNIT > depend on whatever it needs, and then do `if KUNIT` below that? That > seems cleaner over the long term. Anyway, Kees actually asked me to > change it to the way it is now; I really don't care either way. Yes, that works better. The idea is to just avoid having to write in depends on over and over again. > > I'm a bit conflicted here. This currently depends on UML but yet you > > noted on RFC v2 that your intention is to liberate kunit from UML and > > ideally allow unit tests to depend only on userspace. I've addressed > > tests using both selftests kernels drivers and also re-written kernel > > APIs to userspace to test there. I think we may need to live with both. > > I am not entirely opposed. The greater isolation we can achieve, the > fewer dependencies, and barriers to setting up test fixtures the > better. I think the best way to do that in most cases is allowing > minimal test binaries to be built that have the absolute minimum > amount of code necessary to test the desired property. That being > said, integration tests are a thing and drawing a line between them > and unit tests is not always possible, so supporting other > architectures might be necessary. Then lets pave the way for it to be done easily. > > Then for the UML stuff, I think if we *really* accept that UML will > > always be a viable option we should probably consider now throwing these > > things under drivers/platform/uml/. This follows the pattern of arch > > specific drivers. Whether or not we end up with a complete userspace > > component independent of UML may implicate having a shared component > > somewhere else. > > Fair enough. What specifically are you suggesting should go in > `drivers/platform/uml`? Just the bits that are completely tied to UML > or a concrete architecture? The bits that are UML specific. As I see it, with the above intention clarified, kunit is a framework for architectures and UML is supported first. The code doesn't currently reflect this. > > Likewise, I realize the goal is to *avoid* using a virtual machine for > > these tests, but would it in any way make sense to share kunit to be > > supported for other architectures to allow easier-to-write tests as > > well? > > You are not the first person to ask for this. > > For the vast majority of tests, I think we can (and consequently > should) make them run without any external dependencies. Doing so > makes it such that someone can run a test without knowing anything > about it, which allows you to do a lot of things. For one, I, as a > developer, don't have to hunt down somebody's QEMU patches, or > whatever. But it also means I, as someone maintaining part of the > kernel, can make nice test runners and build things like presubmit > servers on top of them. > > Nevertheless, I accept that there are things which are just easier to > do with hardware or a VM (for integration tests it is necessary). > Still, I think we need to make sure the vast majority of unit tests do > not depend on real hardware or a VM. When possible, sure. Luis
next prev parent reply other threads:[~2018-12-01 2:57 UTC|newest] Thread overview: 232+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-11-28 19:36 [RFC v3 00/19] kunit: introduce KUnit, the Linux kernel unit testing framework brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 01/19] kunit: test: add KUnit test runner core brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-30 3:14 ` mcgrof 2018-11-30 3:14 ` Luis Chamberlain 2018-12-01 1:51 ` brendanhiggins 2018-12-01 1:51 ` Brendan Higgins 2018-12-01 2:57 ` mcgrof [this message] 2018-12-01 2:57 ` Luis Chamberlain 2018-12-05 13:15 ` anton.ivanov 2018-12-05 13:15 ` Anton Ivanov 2018-12-05 14:45 ` arnd 2018-12-05 14:45 ` Arnd Bergmann 2018-12-05 14:49 ` anton.ivanov 2018-12-05 14:49 ` Anton Ivanov 2018-11-30 3:28 ` mcgrof 2018-11-30 3:28 ` Luis Chamberlain 2018-12-01 2:08 ` brendanhiggins 2018-12-01 2:08 ` Brendan Higgins 2018-12-01 3:10 ` mcgrof 2018-12-01 3:10 ` Luis Chamberlain 2018-12-03 22:47 ` brendanhiggins 2018-12-03 22:47 ` Brendan Higgins 2018-12-01 3:02 ` mcgrof 2018-12-01 3:02 ` Luis Chamberlain 2018-11-28 19:36 ` [RFC v3 02/19] kunit: test: add test resource management API brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 03/19] kunit: test: add string_stream a std::stream like string builder brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-30 3:29 ` mcgrof 2018-11-30 3:29 ` Luis Chamberlain 2018-12-01 2:14 ` brendanhiggins 2018-12-01 2:14 ` Brendan Higgins 2018-12-01 3:12 ` mcgrof 2018-12-01 3:12 ` Luis Chamberlain 2018-12-03 10:55 ` pmladek 2018-12-03 10:55 ` Petr Mladek 2018-12-04 0:35 ` brendanhiggins 2018-12-04 0:35 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 04/19] kunit: test: add test_stream a std::stream like logger brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 05/19] kunit: test: add the concept of expectations brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 06/19] arch: um: enable running kunit from User Mode Linux brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 21:26 ` robh 2018-11-28 21:26 ` Rob Herring 2018-11-30 3:37 ` mcgrof 2018-11-30 3:37 ` Luis Chamberlain 2018-11-30 14:05 ` robh 2018-11-30 14:05 ` Rob Herring 2018-11-30 18:22 ` mcgrof 2018-11-30 18:22 ` Luis Chamberlain 2018-12-03 23:22 ` brendanhiggins 2018-12-03 23:22 ` Brendan Higgins 2018-11-30 3:30 ` mcgrof 2018-11-30 3:30 ` Luis Chamberlain 2018-11-28 19:36 ` [RFC v3 07/19] kunit: test: add initial tests brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-30 3:40 ` mcgrof 2018-11-30 3:40 ` Luis Chamberlain 2018-12-03 23:26 ` brendanhiggins 2018-12-03 23:26 ` Brendan Higgins 2018-12-03 23:43 ` mcgrof 2018-12-03 23:43 ` Luis Chamberlain 2018-11-28 19:36 ` [RFC v3 08/19] arch: um: add shim to trap to allow installing a fault catcher for tests brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-30 3:34 ` mcgrof 2018-11-30 3:34 ` Luis Chamberlain 2018-12-03 23:34 ` brendanhiggins 2018-12-03 23:34 ` Brendan Higgins 2018-12-03 23:46 ` mcgrof 2018-12-03 23:46 ` Luis Chamberlain 2018-12-04 0:44 ` brendanhiggins 2018-12-04 0:44 ` Brendan Higgins 2018-11-30 3:41 ` mcgrof 2018-11-30 3:41 ` Luis Chamberlain 2018-12-03 23:37 ` brendanhiggins 2018-12-03 23:37 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 09/19] kunit: test: add the concept of assertions brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 10/19] kunit: test: add test managed resource tests brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 11/19] kunit: add Python libraries for handing KUnit config and kernel brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-29 13:54 ` kieran.bingham 2018-11-29 13:54 ` Kieran Bingham 2018-12-03 23:48 ` brendanhiggins 2018-12-03 23:48 ` Brendan Higgins 2018-12-04 20:47 ` mcgrof 2018-12-04 20:47 ` Luis Chamberlain 2018-12-06 12:32 ` kieran.bingham 2018-12-06 12:32 ` Kieran Bingham 2018-12-06 15:37 ` willy 2018-12-06 15:37 ` Matthew Wilcox 2018-12-07 11:30 ` kieran.bingham 2018-12-07 11:30 ` Kieran Bingham 2018-12-11 14:09 ` pmladek 2018-12-11 14:09 ` Petr Mladek 2018-12-11 14:41 ` rostedt 2018-12-11 14:41 ` Steven Rostedt 2018-12-11 17:01 ` anton.ivanov 2018-12-11 17:01 ` Anton Ivanov 2019-02-09 0:40 ` brendanhiggins 2019-02-09 0:40 ` Brendan Higgins 2018-12-07 1:05 ` mcgrof 2018-12-07 1:05 ` Luis Chamberlain 2018-12-07 18:35 ` kent.overstreet 2018-12-07 18:35 ` Kent Overstreet 2018-11-30 3:44 ` mcgrof 2018-11-30 3:44 ` Luis Chamberlain 2018-12-03 23:50 ` brendanhiggins 2018-12-03 23:50 ` Brendan Higgins 2018-12-04 20:48 ` mcgrof 2018-12-04 20:48 ` Luis Chamberlain 2018-11-28 19:36 ` [RFC v3 12/19] kunit: add KUnit wrapper script and simple output parser brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 13/19] kunit: improve output from python wrapper brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 14/19] Documentation: kunit: add documentation for KUnit brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-29 13:56 ` kieran.bingham 2018-11-29 13:56 ` Kieran Bingham 2018-11-30 3:45 ` mcgrof 2018-11-30 3:45 ` Luis Chamberlain 2018-12-03 23:53 ` brendanhiggins 2018-12-03 23:53 ` Brendan Higgins 2018-12-06 12:16 ` kieran.bingham 2018-12-06 12:16 ` Kieran Bingham 2019-02-09 0:56 ` brendanhiggins 2019-02-09 0:56 ` Brendan Higgins 2019-02-11 12:16 ` kieran.bingham 2019-02-11 12:16 ` Kieran Bingham 2019-02-12 22:10 ` brendanhiggins 2019-02-12 22:10 ` Brendan Higgins 2019-02-13 21:55 ` kieran.bingham 2019-02-13 21:55 ` Kieran Bingham 2019-02-14 0:17 ` brendanhiggins 2019-02-14 0:17 ` Brendan Higgins 2019-02-14 17:26 ` mcgrof 2019-02-14 17:26 ` Luis Chamberlain 2019-02-14 22:07 ` brendanhiggins 2019-02-14 22:07 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 15/19] MAINTAINERS: add entry for KUnit the unit testing framework brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 16/19] arch: um: make UML unflatten device tree when testing brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 21:16 ` robh 2018-11-28 21:16 ` Rob Herring 2018-12-04 0:00 ` brendanhiggins 2018-12-04 0:00 ` Brendan Higgins 2018-11-30 3:46 ` mcgrof 2018-11-30 3:46 ` Luis Chamberlain 2018-12-04 0:02 ` brendanhiggins 2018-12-04 0:02 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 17/19] of: unittest: migrate tests to run on KUnit brendanhiggins 2018-11-28 19:36 ` Brendan Higgins [not found] ` <CAL_Jsq+09Kx7yMBC_Jw45QGmk6U_fp4N6HOZDwYrM4tWw+_dOA@mail.gmail.com> 2018-11-30 0:39 ` rdunlap 2018-11-30 0:39 ` Randy Dunlap 2018-12-04 0:13 ` brendanhiggins 2018-12-04 0:13 ` Brendan Higgins 2018-12-04 13:40 ` robh 2018-12-04 13:40 ` Rob Herring 2018-12-05 23:42 ` brendanhiggins 2018-12-05 23:42 ` Brendan Higgins 2018-12-07 0:41 ` robh 2018-12-07 0:41 ` Rob Herring 2018-12-04 0:08 ` brendanhiggins 2018-12-04 0:08 ` Brendan Higgins 2019-02-13 1:44 ` brendanhiggins 2019-02-13 1:44 ` Brendan Higgins 2019-02-14 20:10 ` robh 2019-02-14 20:10 ` Rob Herring 2019-02-14 21:52 ` brendanhiggins 2019-02-14 21:52 ` Brendan Higgins 2019-02-18 22:56 ` frowand.list 2019-02-18 22:56 ` Frank Rowand 2019-02-28 0:29 ` brendanhiggins 2019-02-28 0:29 ` Brendan Higgins 2018-12-04 10:56 ` frowand.list 2018-12-04 10:56 ` Frank Rowand 2018-11-28 19:36 ` [RFC v3 18/19] of: unittest: split out a couple of test cases from unittest brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-12-04 10:58 ` frowand.list 2018-12-04 10:58 ` Frank Rowand 2018-12-05 23:54 ` brendanhiggins 2018-12-05 23:54 ` Brendan Higgins 2019-02-14 23:57 ` frowand.list 2019-02-14 23:57 ` Frank Rowand 2019-02-15 0:56 ` brendanhiggins 2019-02-15 0:56 ` Brendan Higgins 2019-02-15 2:05 ` frowand.list 2019-02-15 2:05 ` Frank Rowand 2019-02-15 10:56 ` brendanhiggins 2019-02-15 10:56 ` Brendan Higgins 2019-02-18 22:25 ` frowand.list 2019-02-18 22:25 ` Frank Rowand 2019-02-20 20:44 ` frowand.list 2019-02-20 20:44 ` Frank Rowand 2019-02-20 20:47 ` frowand.list 2019-02-20 20:47 ` Frank Rowand 2019-02-28 3:52 ` brendanhiggins 2019-02-28 3:52 ` Brendan Higgins 2019-03-22 0:22 ` frowand.list 2019-03-22 0:22 ` Frank Rowand 2019-03-22 1:30 ` brendanhiggins 2019-03-22 1:30 ` Brendan Higgins 2019-03-22 1:47 ` frowand.list 2019-03-22 1:47 ` Frank Rowand 2019-03-25 22:15 ` brendanhiggins 2019-03-25 22:15 ` Brendan Higgins 2019-09-20 16:57 ` Rob Herring 2019-09-21 23:57 ` Frank Rowand 2019-03-22 1:34 ` frowand.list 2019-03-22 1:34 ` Frank Rowand 2019-03-25 22:18 ` brendanhiggins 2019-03-25 22:18 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 19/19] of: unittest: split up some super large test cases brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-12-04 10:52 ` [RFC v3 00/19] kunit: introduce KUnit, the Linux kernel unit testing framework frowand.list 2018-12-04 10:52 ` Frank Rowand 2018-12-04 11:40 ` frowand.list 2018-12-04 11:40 ` Frank Rowand 2018-12-04 13:49 ` robh 2018-12-04 13:49 ` Rob Herring 2018-12-05 23:10 ` brendanhiggins 2018-12-05 23:10 ` Brendan Higgins 2019-03-22 0:27 ` frowand.list 2019-03-22 0:27 ` Frank Rowand 2019-03-25 22:04 ` brendanhiggins 2019-03-25 22:04 ` Brendan Higgins
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20181201025735.GI28501@garbanzo.do-not-panic.com \ --to=linux-kselftest@vger.kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).