Linux-NVDIMM Archive on lore.kernel.org
 help / color / Atom feed
From: Luis Chamberlain <mcgrof@kernel.org>
To: Brendan Higgins <brendanhiggins@google.com>
Cc: brakmo@fb.com, dri-devel@lists.freedesktop.org,
	Sasha Levin <Alexander.Levin@microsoft.com>,
	linux-kselftest@vger.kernel.org, shuah@kernel.org,
	Rob Herring <robh@kernel.org>,
	Frank Rowand <frowand.list@gmail.com>,
	linux-nvdimm@lists.01.org, richard@nod.at,
	Knut Omang <knut.omang@oracle.com>,
	kieran.bingham@ideasonboard.com, Joel Stanley <joel@jms.id.au>,
	jdike@addtoit.com, Tim.Bird@sony.com,
	Kees Cook <keescook@google.com>,
	linux-um@lists.infradead.org, rostedt@goodmis.org,
	Julia Lawall <julia.lawall@lip6.fr>,
	levinsasha928@gmail.com, kunit-dev@googlegroups.com,
	Greg KH <gregkh@linuxfoundation.org>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	Daniel Vetter <daniel@ffwll.ch>,
	mpe@ellerman.id.au, joe@perches.com, khilman@baylibre.com
Subject: Re: [RFC v3 01/19] kunit: test: add KUnit test runner core
Date: Fri, 30 Nov 2018 18:57:35 -0800
Message-ID: <20181201025735.GI28501@garbanzo.do-not-panic.com> (raw)
In-Reply-To: <CAFd5g47t94RR78_sZ7Xr7ArDh+hJGYHNwmmG2HJb7AW+i=u3ew@mail.gmail.com>

On Fri, Nov 30, 2018 at 05:51:11PM -0800, Brendan Higgins wrote:
> On Thu, Nov 29, 2018 at 7:14 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> >
> > On Wed, Nov 28, 2018 at 11:36:18AM -0800, Brendan Higgins wrote:
> > > +#define module_test(module) \
> > > +             static int module_kunit_init##module(void) \
> > > +             { \
> > > +                     return kunit_run_tests(&module); \
> > > +             } \
> > > +             late_initcall(module_kunit_init##module)
> >
> > Here in lies an assumption that suffices. I'm inclined to believe we
> > need new initcall level here so to ensure we *do* run after all the
> > respective kernels iniut calls. Otherwise we're left at the whims of
> > link order for kunit. For instance if a kunit test relies on frameworks
> > which are also late_initcall() we'd have complete incompatibility with
> > anything linked *after* kunit.
> 
> Yep, I have some patches that address this, but I thought this is
> sufficient for the initial patchset (I figured that's the type of
> thing that people will have opinions about so best to get it out of
> the critical path). Do you want me to add those in the next revision?
> 
> >
> > > diff --git a/kunit/Kconfig b/kunit/Kconfig
> > > new file mode 100644
> > > index 0000000000000..49b44c4f6630a
> > > --- /dev/null
> > > +++ b/kunit/Kconfig
> > > @@ -0,0 +1,17 @@
> > > +#
> > > +# KUnit base configuration
> > > +#
> > > +
> > > +menu "KUnit support"
> > > +
> > > +config KUNIT
> > > +     bool "Enable support for unit tests (KUnit)"
> > > +     depends on UML
> >
> > Consider using:
> >
> > if UML
> >    ...
> > endif
> >
> > That allows the depends to be done once.
> 
> If you want to eliminate depends, wouldn't it be best to have KUNIT
> depend on whatever it needs, and then do `if KUNIT` below that? That
> seems cleaner over the long term. Anyway, Kees actually asked me to
> change it to the way it is now; I really don't care either way.

Yes, that works better. The idea is to just avoid having to write in
depends on over and over again.

> > I'm a bit conflicted here. This currently depends on UML but yet you
> > noted on RFC v2 that your intention is to liberate kunit from UML and
> > ideally allow unit tests to depend only on userspace. I've addressed
> > tests using both selftests kernels drivers and also re-written kernel
> > APIs to userspace to test there. I think we may need to live with both.
> 
> I am not entirely opposed. The greater isolation we can achieve, the
> fewer dependencies, and barriers to setting up test fixtures the
> better. I think the best way to do that in most cases is allowing
> minimal test binaries to be built that have the absolute minimum
> amount of code necessary to test the desired property. That being
> said, integration tests are a thing and drawing a line between them
> and unit tests is not always possible, so supporting other
> architectures might be necessary.

Then lets pave the way for it to be done easily.

> > Then for the UML stuff, I think if we *really* accept that UML will
> > always be a viable option we should probably consider now throwing these
> > things under drivers/platform/uml/. This follows the pattern of arch
> > specific drivers. Whether or not we end up with a complete userspace
> > component independent of UML may implicate having a shared component
> > somewhere else.
> 
> Fair enough. What specifically are you suggesting should go in
> `drivers/platform/uml`? Just the bits that are completely tied to UML
> or a concrete architecture?

The bits that are UML specific. As I see it, with the above intention
clarified, kunit is a framework for architectures and UML is supported
first. The code doesn't currently reflect this.

> > Likewise, I realize the goal is to *avoid* using a virtual machine for
> > these tests, but would it in any way make sense to share kunit to be
> > supported for other architectures to allow easier-to-write tests as
> > well?
> 
> You are not the first person to ask for this.
> 
> For the vast majority of tests, I think we can (and consequently
> should) make them run without any external dependencies. Doing so
> makes it such that someone can run a test without knowing anything
> about it, which allows you to do a lot of things. For one, I, as a
> developer, don't have to hunt down somebody's QEMU patches, or
> whatever. But it also means I, as someone maintaining part of the
> kernel, can make nice test runners and build things like presubmit
> servers on top of them.
> 
> Nevertheless, I accept that there are things which are just easier to
> do with hardware or a VM (for integration tests it is necessary).
> Still, I think we need to make sure the vast majority of unit tests do
> not depend on real hardware or a VM.

When possible, sure.

  Luis
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

  reply index

Thread overview: 118+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-11-28 19:36 [RFC v3 00/19] kunit: introduce KUnit, the Linux kernel unit testing framework Brendan Higgins
2018-11-28 19:36 ` [RFC v3 01/19] kunit: test: add KUnit test runner core Brendan Higgins
2018-11-30  3:14   ` Luis Chamberlain
2018-12-01  1:51     ` Brendan Higgins
2018-12-01  2:57       ` Luis Chamberlain [this message]
2018-12-05 13:15     ` Anton Ivanov
2018-12-05 14:45       ` Arnd Bergmann
2018-12-05 14:49         ` Anton Ivanov
2018-11-30  3:28   ` Luis Chamberlain
     [not found]     ` <20181130032802.GG18410-dAjH6bxAqesAS62YNPtMr3dQhYtBYE6JAL8bYrjMMd8@public.gmane.org>
2018-12-01  2:08       ` Brendan Higgins
2018-12-01  3:10         ` Luis Chamberlain
     [not found]           ` <20181201031049.GL28501-dAjH6bxAqesAS62YNPtMr3dQhYtBYE6JAL8bYrjMMd8@public.gmane.org>
2018-12-03 22:47             ` Brendan Higgins
2018-12-01  3:02   ` Luis Chamberlain
2018-11-28 19:36 ` [RFC v3 10/19] kunit: test: add test managed resource tests Brendan Higgins
     [not found] ` <20181128193636.254378-1-brendanhiggins-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
2018-11-28 19:36   ` [RFC v3 02/19] kunit: test: add test resource management API Brendan Higgins
2018-11-28 19:36   ` [RFC v3 03/19] kunit: test: add string_stream a std::stream like string builder Brendan Higgins
2018-11-30  3:29     ` Luis Chamberlain
2018-12-01  2:14       ` Brendan Higgins
2018-12-01  3:12         ` Luis Chamberlain
2018-12-03 10:55       ` Petr Mladek
2018-12-04  0:35         ` Brendan Higgins
2018-11-28 19:36   ` [RFC v3 04/19] kunit: test: add test_stream a std::stream like logger Brendan Higgins
2018-11-28 19:36   ` [RFC v3 05/19] kunit: test: add the concept of expectations Brendan Higgins
2018-11-28 19:36   ` [RFC v3 06/19] arch: um: enable running kunit from User Mode Linux Brendan Higgins
2018-11-28 21:26     ` Rob Herring
2018-11-30  3:37       ` Luis Chamberlain
2018-11-30 14:05         ` Rob Herring
2018-11-30 18:22           ` Luis Chamberlain
     [not found]             ` <20181130182203.GS18410-dAjH6bxAqesAS62YNPtMr3dQhYtBYE6JAL8bYrjMMd8@public.gmane.org>
2018-12-03 23:22               ` Brendan Higgins
2018-11-30  3:30     ` Luis Chamberlain
2018-11-28 19:36   ` [RFC v3 07/19] kunit: test: add initial tests Brendan Higgins
2018-11-30  3:40     ` Luis Chamberlain
2018-12-03 23:26       ` Brendan Higgins
2018-12-03 23:43         ` Luis Chamberlain
2018-11-28 19:36   ` [RFC v3 08/19] arch: um: add shim to trap to allow installing a fault catcher for tests Brendan Higgins
2018-11-30  3:34     ` Luis Chamberlain
     [not found]       ` <20181130033429.GK18410-dAjH6bxAqesAS62YNPtMr3dQhYtBYE6JAL8bYrjMMd8@public.gmane.org>
2018-12-03 23:34         ` Brendan Higgins
     [not found]           ` <CAFd5g45+MAVaSW8HN9x57Y8Um=TV1Oa=-K8yExPBS-4KjLyciQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2018-12-03 23:46             ` Luis Chamberlain
     [not found]               ` <20181203234628.GR28501-dAjH6bxAqesAS62YNPtMr3dQhYtBYE6JAL8bYrjMMd8@public.gmane.org>
2018-12-04  0:44                 ` Brendan Higgins
2018-11-30  3:41     ` Luis Chamberlain
2018-12-03 23:37       ` Brendan Higgins
2018-11-28 19:36   ` [RFC v3 09/19] kunit: test: add the concept of assertions Brendan Higgins
2018-11-28 19:36   ` [RFC v3 11/19] kunit: add Python libraries for handing KUnit config and kernel Brendan Higgins
2018-11-29 13:54     ` Kieran Bingham
     [not found]       ` <841cf4ae-501b-05ae-5863-a51010709b67-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org>
2018-12-03 23:48         ` Brendan Higgins
2018-12-04 20:47           ` Luis Chamberlain
2018-12-06 12:32             ` Kieran Bingham
2018-12-06 15:37               ` Matthew Wilcox
2018-12-07 11:30                 ` Kieran Bingham
2018-12-11 14:09                 ` Petr Mladek
2018-12-11 14:41                   ` Steven Rostedt
2018-12-11 17:01                     ` Anton Ivanov
2019-02-09  0:40                       ` Brendan Higgins
2018-12-07  1:05               ` Luis Chamberlain
2018-12-07 18:35               ` Kent Overstreet
2018-11-30  3:44     ` Luis Chamberlain
2018-12-03 23:50       ` Brendan Higgins
2018-12-04 20:48         ` Luis Chamberlain
2018-11-28 19:36   ` [RFC v3 12/19] kunit: add KUnit wrapper script and simple output parser Brendan Higgins
2018-11-28 19:36   ` [RFC v3 13/19] kunit: improve output from python wrapper Brendan Higgins
2018-11-28 19:36   ` [RFC v3 14/19] Documentation: kunit: add documentation for KUnit Brendan Higgins
2018-11-29 13:56     ` Kieran Bingham
2018-11-30  3:45       ` Luis Chamberlain
     [not found]         ` <20181130034525.GP18410-dAjH6bxAqesAS62YNPtMr3dQhYtBYE6JAL8bYrjMMd8@public.gmane.org>
2018-12-03 23:53           ` Brendan Higgins
2018-12-06 12:16             ` Kieran Bingham
2019-02-09  0:56               ` Brendan Higgins
2019-02-11 12:16                 ` Kieran Bingham
2019-02-12 22:10                   ` Brendan Higgins
2019-02-13 21:55                     ` Kieran Bingham
2019-02-14  0:17                       ` Brendan Higgins
2019-02-14 17:26                         ` Luis Chamberlain
2019-02-14 22:07                           ` Brendan Higgins
2018-11-28 19:36   ` [RFC v3 15/19] MAINTAINERS: add entry for KUnit the unit testing framework Brendan Higgins
2018-11-28 19:36   ` [RFC v3 17/19] of: unittest: migrate tests to run on KUnit Brendan Higgins
2018-11-28 20:56     ` Rob Herring
2018-11-30  0:39       ` Randy Dunlap
     [not found]         ` <18814973-8f0a-4647-a097-fcc3dc0b3cd3-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org>
2018-12-04  0:13           ` Brendan Higgins
2018-12-04 13:40             ` Rob Herring
     [not found]               ` <CAL_JsqL_PivQbrJFEusdKAy-2EQtKL3OHbmyYSK9bzuTOQegqA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2018-12-05 23:42                 ` Brendan Higgins
2018-12-07  0:41                   ` Rob Herring
2018-12-04  0:08       ` Brendan Higgins
2019-02-13  1:44       ` Brendan Higgins
2019-02-14 20:10         ` Rob Herring
2019-02-14 21:52           ` Brendan Higgins
2019-02-18 22:56         ` Frank Rowand
2019-02-28  0:29           ` Brendan Higgins
2018-12-04 10:56     ` Frank Rowand
2018-11-28 19:36   ` [RFC v3 18/19] of: unittest: split out a couple of test cases from unittest Brendan Higgins
2018-12-04 10:58     ` Frank Rowand
2018-12-05 23:54       ` Brendan Higgins
2019-02-14 23:57         ` Frank Rowand
2019-02-15  0:56           ` Brendan Higgins
2019-02-15  2:05             ` Frank Rowand
2019-02-15 10:56               ` Brendan Higgins
2019-02-18 22:25                 ` Frank Rowand
2019-02-20 20:44                   ` Frank Rowand
2019-02-20 20:47                     ` Frank Rowand
2019-02-28  3:52                     ` Brendan Higgins
2019-03-22  0:22                       ` Frank Rowand
2019-03-22  1:30                         ` Brendan Higgins
2019-03-22  1:47                           ` Frank Rowand
2019-03-25 22:15                             ` Brendan Higgins
2019-09-20 16:57                           ` Rob Herring
2019-09-21 23:57                             ` Frank Rowand
2019-03-22  1:34                         ` Frank Rowand
2019-03-25 22:18                           ` Brendan Higgins
2018-11-28 19:36   ` [RFC v3 19/19] of: unittest: split up some super large test cases Brendan Higgins
2018-11-28 19:36 ` [RFC v3 16/19] arch: um: make UML unflatten device tree when testing Brendan Higgins
2018-11-28 21:16   ` Rob Herring
     [not found]     ` <CAL_JsqK5cG=QzMBFSZ31_-3ujnxqxv=jj3XYajbRLT7yWYZbfw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2018-12-04  0:00       ` Brendan Higgins
2018-11-30  3:46   ` Luis Chamberlain
2018-12-04  0:02     ` Brendan Higgins
2018-12-04 10:52 ` [RFC v3 00/19] kunit: introduce KUnit, the Linux kernel unit testing framework Frank Rowand
2018-12-04 11:40 ` Frank Rowand
2018-12-04 13:49   ` Rob Herring
2018-12-05 23:10     ` Brendan Higgins
2019-03-22  0:27       ` Frank Rowand
2019-03-25 22:04         ` Brendan Higgins

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20181201025735.GI28501@garbanzo.do-not-panic.com \
    --to=mcgrof@kernel.org \
    --cc=Alexander.Levin@microsoft.com \
    --cc=Tim.Bird@sony.com \
    --cc=brakmo@fb.com \
    --cc=brendanhiggins@google.com \
    --cc=daniel@ffwll.ch \
    --cc=dri-devel@lists.freedesktop.org \
    --cc=frowand.list@gmail.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=jdike@addtoit.com \
    --cc=joe@perches.com \
    --cc=joel@jms.id.au \
    --cc=julia.lawall@lip6.fr \
    --cc=keescook@google.com \
    --cc=khilman@baylibre.com \
    --cc=kieran.bingham@ideasonboard.com \
    --cc=knut.omang@oracle.com \
    --cc=kunit-dev@googlegroups.com \
    --cc=levinsasha928@gmail.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=linux-nvdimm@lists.01.org \
    --cc=linux-um@lists.infradead.org \
    --cc=mpe@ellerman.id.au \
    --cc=richard@nod.at \
    --cc=robh@kernel.org \
    --cc=rostedt@goodmis.org \
    --cc=shuah@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-NVDIMM Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-nvdimm/0 linux-nvdimm/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-nvdimm linux-nvdimm/ https://lore.kernel.org/linux-nvdimm \
		linux-nvdimm@lists.01.org
	public-inbox-index linux-nvdimm

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.01.lists.linux-nvdimm


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git