From: Matthew Wilcox <willy@infradead.org>
To: Kieran Bingham <kieran.bingham@ideasonboard.com>
Cc: Luis Chamberlain <mcgrof@kernel.org>,
Brendan Higgins <brendanhiggins@google.com>,
Kent Overstreet <kent.overstreet@gmail.com>,
Eryu Guan <guaneryu@gmail.com>,
Eric Sandeen <sandeen@sandeen.net>,
jeffm@suse.com, Sasha Levin <Alexander.Levin@microsoft.com>,
Greg KH <gregkh@linuxfoundation.org>,
Kees Cook <keescook@google.com>,
shuah@kernel.org, Joel Stanley <joel@jms.id.au>,
mpe@ellerman.id.au, joe@perches.com, brakmo@fb.com,
rostedt@goodmis.org, Tim.Bird@sony.com, khilman@baylibre.com,
Julia Lawall <julia.lawall@lip6.fr>,
linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
jdike@addtoit.com, richard@nod.at, linux-um@lists.infradead.org,
Daniel Vetter <daniel@ffwll.ch>,
dri-devel@lists.freedesktop.org, Rob Herring <robh@kernel.org>,
dan.j.williams@intel.com, linux-nvdimm@lists.01.org,
Frank Rowand <frowand.list@gmail.com>,
Knut Omang <knut.omang@oracle.com>,
Felix Guo <felixguoxiuping@gmail.com>,
Petr Mladek <pmladek@suse.com>,
linux-fsdevel@vger.kernel.org
Subject: Re: [RFC v3 11/19] kunit: add Python libraries for handing KUnit config and kernel
Date: Thu, 6 Dec 2018 07:37:18 -0800 [thread overview]
Message-ID: <20181206153718.GD24603@bombadil.infradead.org> (raw)
In-Reply-To: <f8722f4a-44c8-24c2-c433-5178f9f40b82@ideasonboard.com>
On Thu, Dec 06, 2018 at 12:32:47PM +0000, Kieran Bingham wrote:
> On 04/12/2018 20:47, Luis Chamberlain wrote:
> > On Mon, Dec 03, 2018 at 03:48:15PM -0800, Brendan Higgins wrote:
> >> On Thu, Nov 29, 2018 at 5:54 AM Kieran Bingham
> >> <kieran.bingham@ideasonboard.com> wrote:
> >>>
> >>> Hi Brendan,
> >>>
> >>> Thanks again for this series!
> >>>
> >>> On 28/11/2018 19:36, Brendan Higgins wrote:
> >>>> The ultimate goal is to create minimal isolated test binaries; in the
> >>>> meantime we are using UML to provide the infrastructure to run tests, so
> >>>> define an abstract way to configure and run tests that allow us to
> >>>> change the context in which tests are built without affecting the user.
> >>>> This also makes pretty and dynamic error reporting, and a lot of other
> >>>> nice features easier.
> >>>
> >>>
> >>> I wonder if we could somehow generate a shared library object
> >>> 'libkernel' or 'libumlinux' from a UM configured set of headers and
> >>> objects so that we could create binary targets directly ?
> >>
> >> That's an interesting idea. I think it would be difficult to figure
> >> out exactly where to draw the line of what goes in there and what
> >> needs to be built specific to a test a priori. Of course, that leads
> >> into the biggest problem in general, needed to know what I need to
> >> build to test the thing that I want to test.
> >>
> >> Nevertheless, I could definitely imagine that being useful in a lot of cases.
> >
> > Whether or not we can abstract away the kernel into such a mechanism
> > with uml libraries is a good question worth exploring.
> >
> > Developers working upstream do modify their kernels a lot, so we'd have
> > to update such libraries quite a bit, but I think that's fine too. The
> > *real* value I think from the above suggestion would be enterprise /
> > mobile distros or stable kernel maintainers which have a static kernel
> > they need to support for a relatively *long time*, consider a 10 year
> > time frame. Running unit tests without qemu with uml and libraries for
> > respective kernels seems real worthy.
>
> I think any such library might be something generated by the kernel
> build system, so if someone makes substantial changes to a core
> component provided by the library - it can be up to them to build a
> corresponding userspace library as well.
>
> We could also consider to only provide *static* libraries rather than
> dynamic. So any one building some userspace tool / test with this would
> be required to compile against (the version of) the kernel they expect
> perhaps... - much like we expect modules to be compiled currently.
>
> And then the userspace binary would be sufficiently able to live it's
> life on it's own :)
>
> > The overhead for testing a unit test for said targets, *ideally*, would
> > just be to to reboot into the system with such libraries available, a
> > unit test would just look for the respective uname -r library and mimic
> > that kernel, much the same way enterprise distributions today rely on
> > having debugging symbols available to run against crash / gdb. Having
> > debug modules / kernel for crash requires such effort already, so this
> > would just be an extra layer of other prospect tests.
>
> Oh - although, yes - there are some good concepts there - but I'm a bit
> weary of how easy it would be to 'run' the said test against multiple
> kernel version libraries... there would be a lot of possible ABI
> conflicts perhaps.
>
> My main initial idea for a libumlinux is to provide infrastructure such
> as our linked-lists and other kernel formatting so that we can take
> kernel code directly to userspace for test and debug (assuming that
> there are no hardware dependencies or things that we can't mock out)
>
> I think all of this could complement kunit of course - this isn't
> suggesting an alternative implementation :-)
I suspect the reason Luis cc'd me on this is that we already have some
artisinally-crafted userspace kernel-mocking interfaces under tools/.
The tools/testing/radix-tree directory is the source of some of this,
but I've been moving pieces out into tools/ more generally where it
makes sense to.
We have liburcu already, which is good. The main sticking points are:
- No emulation of kernel thread interfaces
- The kernel does not provide the ability to aggressively fail memory
allocations (which is useful when trying to exercise the memory failure
paths).
- printk has started adding a lot of %pX enhancements which printf
obviously doesn't know about.
- No global pseudo-random number generator in the kernel. Probably
we should steal the i915 one.
I know Dan Williams has also done a lot of working mocking kernel
interfaces for libnvdimm.
next prev parent reply other threads:[~2018-12-06 15:37 UTC|newest]
Thread overview: 118+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-11-28 19:36 [RFC v3 00/19] kunit: introduce KUnit, the Linux kernel unit testing framework Brendan Higgins
2018-11-28 19:36 ` [RFC v3 01/19] kunit: test: add KUnit test runner core Brendan Higgins
2018-11-30 3:14 ` Luis Chamberlain
2018-12-01 1:51 ` Brendan Higgins
2018-12-01 2:57 ` Luis Chamberlain
2018-12-05 13:15 ` Anton Ivanov
2018-12-05 14:45 ` Arnd Bergmann
2018-12-05 14:49 ` Anton Ivanov
2018-11-30 3:28 ` Luis Chamberlain
2018-12-01 2:08 ` Brendan Higgins
2018-12-01 3:10 ` Luis Chamberlain
2018-12-03 22:47 ` Brendan Higgins
2018-12-01 3:02 ` Luis Chamberlain
2018-11-28 19:36 ` [RFC v3 02/19] kunit: test: add test resource management API Brendan Higgins
2018-11-28 19:36 ` [RFC v3 03/19] kunit: test: add string_stream a std::stream like string builder Brendan Higgins
2018-11-30 3:29 ` Luis Chamberlain
2018-12-01 2:14 ` Brendan Higgins
2018-12-01 3:12 ` Luis Chamberlain
2018-12-03 10:55 ` Petr Mladek
2018-12-04 0:35 ` Brendan Higgins
2018-11-28 19:36 ` [RFC v3 04/19] kunit: test: add test_stream a std::stream like logger Brendan Higgins
2018-11-28 19:36 ` [RFC v3 05/19] kunit: test: add the concept of expectations Brendan Higgins
2018-11-28 19:36 ` [RFC v3 06/19] arch: um: enable running kunit from User Mode Linux Brendan Higgins
2018-11-28 21:26 ` Rob Herring
2018-11-30 3:37 ` Luis Chamberlain
2018-11-30 14:05 ` Rob Herring
2018-11-30 18:22 ` Luis Chamberlain
2018-12-03 23:22 ` Brendan Higgins
2018-11-30 3:30 ` Luis Chamberlain
2018-11-28 19:36 ` [RFC v3 07/19] kunit: test: add initial tests Brendan Higgins
2018-11-30 3:40 ` Luis Chamberlain
2018-12-03 23:26 ` Brendan Higgins
2018-12-03 23:43 ` Luis Chamberlain
2018-11-28 19:36 ` [RFC v3 08/19] arch: um: add shim to trap to allow installing a fault catcher for tests Brendan Higgins
2018-11-30 3:34 ` Luis Chamberlain
2018-12-03 23:34 ` Brendan Higgins
2018-12-03 23:46 ` Luis Chamberlain
2018-12-04 0:44 ` Brendan Higgins
2018-11-30 3:41 ` Luis Chamberlain
2018-12-03 23:37 ` Brendan Higgins
2018-11-28 19:36 ` [RFC v3 09/19] kunit: test: add the concept of assertions Brendan Higgins
2018-11-28 19:36 ` [RFC v3 10/19] kunit: test: add test managed resource tests Brendan Higgins
2018-11-28 19:36 ` [RFC v3 11/19] kunit: add Python libraries for handing KUnit config and kernel Brendan Higgins
2018-11-29 13:54 ` Kieran Bingham
2018-12-03 23:48 ` Brendan Higgins
2018-12-04 20:47 ` Luis Chamberlain
2018-12-06 12:32 ` Kieran Bingham
2018-12-06 15:37 ` Matthew Wilcox [this message]
2018-12-07 11:30 ` Kieran Bingham
2018-12-11 14:09 ` Petr Mladek
2018-12-11 14:41 ` Steven Rostedt
2018-12-11 17:01 ` Anton Ivanov
2019-02-09 0:40 ` Brendan Higgins
2018-12-07 1:05 ` Luis Chamberlain
2018-12-07 18:35 ` Kent Overstreet
2018-11-30 3:44 ` Luis Chamberlain
2018-12-03 23:50 ` Brendan Higgins
2018-12-04 20:48 ` Luis Chamberlain
2018-11-28 19:36 ` [RFC v3 12/19] kunit: add KUnit wrapper script and simple output parser Brendan Higgins
2018-11-28 19:36 ` [RFC v3 13/19] kunit: improve output from python wrapper Brendan Higgins
2018-11-28 19:36 ` [RFC v3 14/19] Documentation: kunit: add documentation for KUnit Brendan Higgins
2018-11-29 13:56 ` Kieran Bingham
2018-11-30 3:45 ` Luis Chamberlain
2018-12-03 23:53 ` Brendan Higgins
2018-12-06 12:16 ` Kieran Bingham
2019-02-09 0:56 ` Brendan Higgins
2019-02-11 12:16 ` Kieran Bingham
2019-02-12 22:10 ` Brendan Higgins
2019-02-13 21:55 ` Kieran Bingham
2019-02-14 0:17 ` Brendan Higgins
2019-02-14 17:26 ` Luis Chamberlain
2019-02-14 22:07 ` Brendan Higgins
2018-11-28 19:36 ` [RFC v3 15/19] MAINTAINERS: add entry for KUnit the unit testing framework Brendan Higgins
2018-11-28 19:36 ` [RFC v3 16/19] arch: um: make UML unflatten device tree when testing Brendan Higgins
2018-11-28 21:16 ` Rob Herring
2018-12-04 0:00 ` Brendan Higgins
2018-11-30 3:46 ` Luis Chamberlain
2018-12-04 0:02 ` Brendan Higgins
2018-11-28 19:36 ` [RFC v3 17/19] of: unittest: migrate tests to run on KUnit Brendan Higgins
2018-11-28 20:56 ` Rob Herring
2018-11-30 0:39 ` Randy Dunlap
2018-12-04 0:13 ` Brendan Higgins
2018-12-04 13:40 ` Rob Herring
2018-12-05 23:42 ` Brendan Higgins
2018-12-07 0:41 ` Rob Herring
2018-12-04 0:08 ` Brendan Higgins
2019-02-13 1:44 ` Brendan Higgins
2019-02-14 20:10 ` Rob Herring
2019-02-14 21:52 ` Brendan Higgins
2019-02-18 22:56 ` Frank Rowand
2019-02-28 0:29 ` Brendan Higgins
2018-12-04 10:56 ` Frank Rowand
2018-11-28 19:36 ` [RFC v3 18/19] of: unittest: split out a couple of test cases from unittest Brendan Higgins
2018-12-04 10:58 ` Frank Rowand
2018-12-05 23:54 ` Brendan Higgins
2019-02-14 23:57 ` Frank Rowand
2019-02-15 0:56 ` Brendan Higgins
2019-02-15 2:05 ` Frank Rowand
2019-02-15 10:56 ` Brendan Higgins
2019-02-18 22:25 ` Frank Rowand
2019-02-20 20:44 ` Frank Rowand
2019-02-20 20:47 ` Frank Rowand
2019-02-28 3:52 ` Brendan Higgins
2019-03-22 0:22 ` Frank Rowand
2019-03-22 1:30 ` Brendan Higgins
2019-03-22 1:47 ` Frank Rowand
2019-03-25 22:15 ` Brendan Higgins
2019-09-20 16:57 ` Rob Herring
2019-09-21 23:57 ` Frank Rowand
2019-03-22 1:34 ` Frank Rowand
2019-03-25 22:18 ` Brendan Higgins
2018-11-28 19:36 ` [RFC v3 19/19] of: unittest: split up some super large test cases Brendan Higgins
2018-12-04 10:52 ` [RFC v3 00/19] kunit: introduce KUnit, the Linux kernel unit testing framework Frank Rowand
2018-12-04 11:40 ` Frank Rowand
2018-12-04 13:49 ` Rob Herring
2018-12-05 23:10 ` Brendan Higgins
2019-03-22 0:27 ` Frank Rowand
2019-03-25 22:04 ` Brendan Higgins
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20181206153718.GD24603@bombadil.infradead.org \
--to=willy@infradead.org \
--cc=Alexander.Levin@microsoft.com \
--cc=Tim.Bird@sony.com \
--cc=brakmo@fb.com \
--cc=brendanhiggins@google.com \
--cc=dan.j.williams@intel.com \
--cc=daniel@ffwll.ch \
--cc=dri-devel@lists.freedesktop.org \
--cc=felixguoxiuping@gmail.com \
--cc=frowand.list@gmail.com \
--cc=gregkh@linuxfoundation.org \
--cc=guaneryu@gmail.com \
--cc=jdike@addtoit.com \
--cc=jeffm@suse.com \
--cc=joe@perches.com \
--cc=joel@jms.id.au \
--cc=julia.lawall@lip6.fr \
--cc=keescook@google.com \
--cc=kent.overstreet@gmail.com \
--cc=khilman@baylibre.com \
--cc=kieran.bingham@ideasonboard.com \
--cc=knut.omang@oracle.com \
--cc=kunit-dev@googlegroups.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-nvdimm@lists.01.org \
--cc=linux-um@lists.infradead.org \
--cc=mcgrof@kernel.org \
--cc=mpe@ellerman.id.au \
--cc=pmladek@suse.com \
--cc=richard@nod.at \
--cc=robh@kernel.org \
--cc=rostedt@goodmis.org \
--cc=sandeen@sandeen.net \
--cc=shuah@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).