Linux-NVDIMM Archive on lore.kernel.org
 help / color / Atom feed
From: Kieran Bingham <kieran.bingham@ideasonboard.com>
To: Matthew Wilcox <willy@infradead.org>
Cc: brakmo@fb.com, Knut Omang <knut.omang@oracle.com>,
	jeffm@suse.com, Brendan Higgins <brendanhiggins@google.com>,
	dri-devel@lists.freedesktop.org,
	Sasha Levin <Alexander.Levin@microsoft.com>,
	linux-kselftest@vger.kernel.org, shuah@kernel.org,
	Rob Herring <robh@kernel.org>,
	Frank Rowand <frowand.list@gmail.com>,
	linux-nvdimm@lists.01.org, mpe@ellerman.id.au,
	Eric Sandeen <sandeen@sandeen.net>,
	Felix Guo <felixguoxiuping@gmail.com>,
	Joel Stanley <joel@jms.id.au>,
	Tim.Bird@sony.com, Petr Mladek <pmladek@suse.com>,
	jdike@addtoit.com, linux-um@lists.infradead.org,
	rostedt@goodmis.org, Julia Lawall <julia.lawall@lip6.fr>,
	kunit-dev@googlegroups.com, richard@nod.at,
	Greg KH <gregkh@linuxfoundation.org>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	Kent Overstreet <kent.overstreet@gmail.com>,
	Luis Chamberlain <mcgrof@kernel.org>,
	Eryu Guan <guaneryu@gmail.com>, Daniel Vetter <daniel@ffwll.ch>,
	Kees Cook <keescook@google.com>,
	joe@perches.com, linux-fsdevel@vger.kernel.org,
	khilman@baylibre.com
Subject: Re: [RFC v3 11/19] kunit: add Python libraries for handing KUnit config and kernel
Date: Fri, 7 Dec 2018 11:30:30 +0000
Message-ID: <f2a3d7fc-21f2-7bd0-94e2-4122f2073da5@ideasonboard.com> (raw)
In-Reply-To: <20181206153718.GD24603@bombadil.infradead.org>

Hi Matthew,

On 06/12/2018 15:37, Matthew Wilcox wrote:
> On Thu, Dec 06, 2018 at 12:32:47PM +0000, Kieran Bingham wrote:
>> On 04/12/2018 20:47, Luis Chamberlain wrote:
>>> On Mon, Dec 03, 2018 at 03:48:15PM -0800, Brendan Higgins wrote:
>>>> On Thu, Nov 29, 2018 at 5:54 AM Kieran Bingham
>>>> <kieran.bingham@ideasonboard.com> wrote:
>>>>>
>>>>> Hi Brendan,
>>>>>
>>>>> Thanks again for this series!
>>>>>
>>>>> On 28/11/2018 19:36, Brendan Higgins wrote:
>>>>>> The ultimate goal is to create minimal isolated test binaries; in the
>>>>>> meantime we are using UML to provide the infrastructure to run tests, so
>>>>>> define an abstract way to configure and run tests that allow us to
>>>>>> change the context in which tests are built without affecting the user.
>>>>>> This also makes pretty and dynamic error reporting, and a lot of other
>>>>>> nice features easier.
>>>>>
>>>>>
>>>>> I wonder if we could somehow generate a shared library object
>>>>> 'libkernel' or 'libumlinux' from a UM configured set of headers and
>>>>> objects so that we could create binary targets directly ?
>>>>
>>>> That's an interesting idea. I think it would be difficult to figure
>>>> out exactly where to draw the line of what goes in there and what
>>>> needs to be built specific to a test a priori. Of course, that leads
>>>> into the biggest problem in general, needed to know what I need to
>>>> build to test the thing that I want to test.
>>>>
>>>> Nevertheless, I could definitely imagine that being useful in a lot of cases.
>>>
>>> Whether or not we can abstract away the kernel into such a mechanism
>>> with uml libraries is a good question worth exploring.
>>>
>>> Developers working upstream do modify their kernels a lot, so we'd have
>>> to update such libraries quite a bit, but I think that's fine too. The
>>> *real* value I think from the above suggestion would be enterprise /
>>> mobile distros or stable kernel maintainers which have a static kernel
>>> they need to support for a relatively *long time*, consider a 10 year
>>> time frame. Running unit tests without qemu with uml and libraries for
>>> respective kernels seems real worthy.
>>
>> I think any such library might be something generated by the kernel
>> build system, so if someone makes substantial changes to a core
>> component provided by the library - it can be up to them to build a
>> corresponding userspace library as well.
>>
>> We could also consider to only provide *static* libraries rather than
>> dynamic. So any one building some userspace tool / test with this would
>> be required to compile against (the version of) the kernel they expect
>> perhaps... - much like we expect modules to be compiled currently.
>>
>> And then the userspace binary would be sufficiently able to live it's
>> life on it's own :)
>>
>>> The overhead for testing a unit test for said targets, *ideally*, would
>>> just be to to reboot into the system with such libraries available, a
>>> unit test would just look for the respective uname -r library and mimic
>>> that kernel, much the same way enterprise distributions today rely on
>>> having debugging symbols available to run against crash / gdb. Having
>>> debug modules / kernel for crash requires such effort already, so this
>>> would just be an extra layer of other prospect tests.
>>
>> Oh - although, yes - there are some good concepts there - but I'm a bit
>> weary of how easy it would be to 'run' the said test against multiple
>> kernel version libraries... there would be a lot of possible ABI
>> conflicts perhaps.
>>
>> My main initial idea for a libumlinux is to provide infrastructure such
>> as our linked-lists and other kernel formatting so that we can take
>> kernel code directly to userspace for test and debug (assuming that
>> there are no hardware dependencies or things that we can't mock out)
>>
>> I think all of this could complement kunit of course - this isn't
>> suggesting an alternative implementation :-)
> 
> I suspect the reason Luis cc'd me on this is that we already have some
> artisinally-crafted userspace kernel-mocking interfaces under tools/.

Aha - excellent - I had hoped to grab you at Plumbers to talk about
this, after hearing you mention something at your Xarray talk - but
didn't seem to find a suitable time.

> The tools/testing/radix-tree directory is the source of some of this,
> but I've been moving pieces out into tools/ more generally where it
> makes sense to.

Sounds like we already have a starting point then.


> We have liburcu already, which is good.  The main sticking points are:
> 
>  - No emulation of kernel thread interfaces

Scheduling finesse aside, This shouldn't be too hard to emulate/wrap
with pthreads?


>  - The kernel does not provide the ability to aggressively fail memory
>    allocations (which is useful when trying to exercise the memory failure
>    paths).

Fault injection throughout would certainly be a valuable addition to any
unit-testing.

Wrapping tests into a single userspace binary could facilitate further
memory leak checking or other valgrind facilities too.



>  - printk has started adding a lot of %pX enhancements which printf
>    obviously doesn't know about.

Wrapping through User-mode linux essentially provides this already
though. In fact I guess that goes for the thread interfaces topic above too.


>  - No global pseudo-random number generator in the kernel.  Probably
>    we should steal the i915 one.
> 
> I know Dan Williams has also done a lot of working mocking kernel
> interfaces for libnvdimm.


Thanks for the references - more to investigate.

-- 
Regards
--
Kieran
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

  reply index

Thread overview: 118+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-11-28 19:36 [RFC v3 00/19] kunit: introduce KUnit, the Linux kernel unit testing framework Brendan Higgins
2018-11-28 19:36 ` [RFC v3 01/19] kunit: test: add KUnit test runner core Brendan Higgins
2018-11-30  3:14   ` Luis Chamberlain
2018-12-01  1:51     ` Brendan Higgins
2018-12-01  2:57       ` Luis Chamberlain
2018-12-05 13:15     ` Anton Ivanov
2018-12-05 14:45       ` Arnd Bergmann
2018-12-05 14:49         ` Anton Ivanov
2018-11-30  3:28   ` Luis Chamberlain
     [not found]     ` <20181130032802.GG18410-dAjH6bxAqesAS62YNPtMr3dQhYtBYE6JAL8bYrjMMd8@public.gmane.org>
2018-12-01  2:08       ` Brendan Higgins
2018-12-01  3:10         ` Luis Chamberlain
     [not found]           ` <20181201031049.GL28501-dAjH6bxAqesAS62YNPtMr3dQhYtBYE6JAL8bYrjMMd8@public.gmane.org>
2018-12-03 22:47             ` Brendan Higgins
2018-12-01  3:02   ` Luis Chamberlain
2018-11-28 19:36 ` [RFC v3 10/19] kunit: test: add test managed resource tests Brendan Higgins
     [not found] ` <20181128193636.254378-1-brendanhiggins-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
2018-11-28 19:36   ` [RFC v3 02/19] kunit: test: add test resource management API Brendan Higgins
2018-11-28 19:36   ` [RFC v3 03/19] kunit: test: add string_stream a std::stream like string builder Brendan Higgins
2018-11-30  3:29     ` Luis Chamberlain
2018-12-01  2:14       ` Brendan Higgins
2018-12-01  3:12         ` Luis Chamberlain
2018-12-03 10:55       ` Petr Mladek
2018-12-04  0:35         ` Brendan Higgins
2018-11-28 19:36   ` [RFC v3 04/19] kunit: test: add test_stream a std::stream like logger Brendan Higgins
2018-11-28 19:36   ` [RFC v3 05/19] kunit: test: add the concept of expectations Brendan Higgins
2018-11-28 19:36   ` [RFC v3 06/19] arch: um: enable running kunit from User Mode Linux Brendan Higgins
2018-11-28 21:26     ` Rob Herring
2018-11-30  3:37       ` Luis Chamberlain
2018-11-30 14:05         ` Rob Herring
2018-11-30 18:22           ` Luis Chamberlain
     [not found]             ` <20181130182203.GS18410-dAjH6bxAqesAS62YNPtMr3dQhYtBYE6JAL8bYrjMMd8@public.gmane.org>
2018-12-03 23:22               ` Brendan Higgins
2018-11-30  3:30     ` Luis Chamberlain
2018-11-28 19:36   ` [RFC v3 07/19] kunit: test: add initial tests Brendan Higgins
2018-11-30  3:40     ` Luis Chamberlain
2018-12-03 23:26       ` Brendan Higgins
2018-12-03 23:43         ` Luis Chamberlain
2018-11-28 19:36   ` [RFC v3 08/19] arch: um: add shim to trap to allow installing a fault catcher for tests Brendan Higgins
2018-11-30  3:34     ` Luis Chamberlain
     [not found]       ` <20181130033429.GK18410-dAjH6bxAqesAS62YNPtMr3dQhYtBYE6JAL8bYrjMMd8@public.gmane.org>
2018-12-03 23:34         ` Brendan Higgins
     [not found]           ` <CAFd5g45+MAVaSW8HN9x57Y8Um=TV1Oa=-K8yExPBS-4KjLyciQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2018-12-03 23:46             ` Luis Chamberlain
     [not found]               ` <20181203234628.GR28501-dAjH6bxAqesAS62YNPtMr3dQhYtBYE6JAL8bYrjMMd8@public.gmane.org>
2018-12-04  0:44                 ` Brendan Higgins
2018-11-30  3:41     ` Luis Chamberlain
2018-12-03 23:37       ` Brendan Higgins
2018-11-28 19:36   ` [RFC v3 09/19] kunit: test: add the concept of assertions Brendan Higgins
2018-11-28 19:36   ` [RFC v3 11/19] kunit: add Python libraries for handing KUnit config and kernel Brendan Higgins
2018-11-29 13:54     ` Kieran Bingham
     [not found]       ` <841cf4ae-501b-05ae-5863-a51010709b67-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
2018-12-03 23:48         ` Brendan Higgins
2018-12-04 20:47           ` Luis Chamberlain
2018-12-06 12:32             ` Kieran Bingham
2018-12-06 15:37               ` Matthew Wilcox
2018-12-07 11:30                 ` Kieran Bingham [this message]
2018-12-11 14:09                 ` Petr Mladek
2018-12-11 14:41                   ` Steven Rostedt
2018-12-11 17:01                     ` Anton Ivanov
2019-02-09  0:40                       ` Brendan Higgins
2018-12-07  1:05               ` Luis Chamberlain
2018-12-07 18:35               ` Kent Overstreet
2018-11-30  3:44     ` Luis Chamberlain
2018-12-03 23:50       ` Brendan Higgins
2018-12-04 20:48         ` Luis Chamberlain
2018-11-28 19:36   ` [RFC v3 12/19] kunit: add KUnit wrapper script and simple output parser Brendan Higgins
2018-11-28 19:36   ` [RFC v3 13/19] kunit: improve output from python wrapper Brendan Higgins
2018-11-28 19:36   ` [RFC v3 14/19] Documentation: kunit: add documentation for KUnit Brendan Higgins
2018-11-29 13:56     ` Kieran Bingham
2018-11-30  3:45       ` Luis Chamberlain
     [not found]         ` <20181130034525.GP18410-dAjH6bxAqesAS62YNPtMr3dQhYtBYE6JAL8bYrjMMd8@public.gmane.org>
2018-12-03 23:53           ` Brendan Higgins
2018-12-06 12:16             ` Kieran Bingham
2019-02-09  0:56               ` Brendan Higgins
2019-02-11 12:16                 ` Kieran Bingham
2019-02-12 22:10                   ` Brendan Higgins
2019-02-13 21:55                     ` Kieran Bingham
2019-02-14  0:17                       ` Brendan Higgins
2019-02-14 17:26                         ` Luis Chamberlain
2019-02-14 22:07                           ` Brendan Higgins
2018-11-28 19:36   ` [RFC v3 15/19] MAINTAINERS: add entry for KUnit the unit testing framework Brendan Higgins
2018-11-28 19:36   ` [RFC v3 17/19] of: unittest: migrate tests to run on KUnit Brendan Higgins
2018-11-28 20:56     ` Rob Herring
2018-11-30  0:39       ` Randy Dunlap
     [not found]         ` <18814973-8f0a-4647-a097-fcc3dc0b3cd3-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2018-12-04  0:13           ` Brendan Higgins
2018-12-04 13:40             ` Rob Herring
     [not found]               ` <CAL_JsqL_PivQbrJFEusdKAy-2EQtKL3OHbmyYSK9bzuTOQegqA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2018-12-05 23:42                 ` Brendan Higgins
2018-12-07  0:41                   ` Rob Herring
2018-12-04  0:08       ` Brendan Higgins
2019-02-13  1:44       ` Brendan Higgins
2019-02-14 20:10         ` Rob Herring
2019-02-14 21:52           ` Brendan Higgins
2019-02-18 22:56         ` Frank Rowand
2019-02-28  0:29           ` Brendan Higgins
2018-12-04 10:56     ` Frank Rowand
2018-11-28 19:36   ` [RFC v3 18/19] of: unittest: split out a couple of test cases from unittest Brendan Higgins
2018-12-04 10:58     ` Frank Rowand
2018-12-05 23:54       ` Brendan Higgins
2019-02-14 23:57         ` Frank Rowand
2019-02-15  0:56           ` Brendan Higgins
2019-02-15  2:05             ` Frank Rowand
2019-02-15 10:56               ` Brendan Higgins
2019-02-18 22:25                 ` Frank Rowand
2019-02-20 20:44                   ` Frank Rowand
2019-02-20 20:47                     ` Frank Rowand
2019-02-28  3:52                     ` Brendan Higgins
2019-03-22  0:22                       ` Frank Rowand
2019-03-22  1:30                         ` Brendan Higgins
2019-03-22  1:47                           ` Frank Rowand
2019-03-25 22:15                             ` Brendan Higgins
2019-09-20 16:57                           ` Rob Herring
2019-09-21 23:57                             ` Frank Rowand
2019-03-22  1:34                         ` Frank Rowand
2019-03-25 22:18                           ` Brendan Higgins
2018-11-28 19:36   ` [RFC v3 19/19] of: unittest: split up some super large test cases Brendan Higgins
2018-11-28 19:36 ` [RFC v3 16/19] arch: um: make UML unflatten device tree when testing Brendan Higgins
2018-11-28 21:16   ` Rob Herring
     [not found]     ` <CAL_JsqK5cG=QzMBFSZ31_-3ujnxqxv=jj3XYajbRLT7yWYZbfw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2018-12-04  0:00       ` Brendan Higgins
2018-11-30  3:46   ` Luis Chamberlain
2018-12-04  0:02     ` Brendan Higgins
2018-12-04 10:52 ` [RFC v3 00/19] kunit: introduce KUnit, the Linux kernel unit testing framework Frank Rowand
2018-12-04 11:40 ` Frank Rowand
2018-12-04 13:49   ` Rob Herring
2018-12-05 23:10     ` Brendan Higgins
2019-03-22  0:27       ` Frank Rowand
2019-03-25 22:04         ` Brendan Higgins

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=f2a3d7fc-21f2-7bd0-94e2-4122f2073da5@ideasonboard.com \
    --to=kieran.bingham@ideasonboard.com \
    --cc=Alexander.Levin@microsoft.com \
    --cc=Tim.Bird@sony.com \
    --cc=brakmo@fb.com \
    --cc=brendanhiggins@google.com \
    --cc=daniel@ffwll.ch \
    --cc=dri-devel@lists.freedesktop.org \
    --cc=felixguoxiuping@gmail.com \
    --cc=frowand.list@gmail.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=guaneryu@gmail.com \
    --cc=jdike@addtoit.com \
    --cc=jeffm@suse.com \
    --cc=joe@perches.com \
    --cc=joel@jms.id.au \
    --cc=julia.lawall@lip6.fr \
    --cc=keescook@google.com \
    --cc=kent.overstreet@gmail.com \
    --cc=khilman@baylibre.com \
    --cc=knut.omang@oracle.com \
    --cc=kunit-dev@googlegroups.com \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=linux-nvdimm@lists.01.org \
    --cc=linux-um@lists.infradead.org \
    --cc=mcgrof@kernel.org \
    --cc=mpe@ellerman.id.au \
    --cc=pmladek@suse.com \
    --cc=richard@nod.at \
    --cc=robh@kernel.org \
    --cc=rostedt@goodmis.org \
    --cc=sandeen@sandeen.net \
    --cc=shuah@kernel.org \
    --cc=willy@infradead.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-NVDIMM Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-nvdimm/0 linux-nvdimm/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-nvdimm linux-nvdimm/ https://lore.kernel.org/linux-nvdimm \
		linux-nvdimm@lists.01.org
	public-inbox-index linux-nvdimm

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.01.lists.linux-nvdimm


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git