nvdimm.lists.linux.dev archive mirror
 help / color / mirror / Atom feed
From: Brendan Higgins <brendanhiggins@google.com>
To: shuah@kernel.org
Cc: Greg KH <gregkh@linuxfoundation.org>,
	Kees Cook <keescook@google.com>,
	mcgrof@kernel.org, Joel Stanley <joel@jms.id.au>,
	mpe@ellerman.id.au, joe@perches.com, brakmo@fb.com,
	rostedt@goodmis.org, Tim.Bird@sony.com, khilman@baylibre.com,
	Julia Lawall <julia.lawall@lip6.fr>,
	linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	jdike@addtoit.com, richard@nod.at, linux-um@lists.infradead.org,
	Daniel Vetter <daniel@ffwll.ch>,
	dri-devel@lists.freedesktop.org, Rob Herring <robh@kernel.org>,
	dan.j.williams@intel.com, linux-nvdimm@lists.01.org,
	kieran.bingham@ideasonboard.com
Subject: Re: [RFC v2 00/14] kunit: introduce KUnit, the Linux kernel unit testing framework
Date: Tue, 6 Nov 2018 17:17:27 -0800	[thread overview]
Message-ID: <CAFd5g45Tgcx4RisrCF8QHneR7ELezPfpEPKh_mH8KEUi3skE7g@mail.gmail.com> (raw)
In-Reply-To: <f3682d48-6f27-5eef-8274-02f90dd65188@kernel.org>

On Fri, Nov 2, 2018 at 11:23 AM Shuah Khan <shuah@kernel.org> wrote:
>
> Hi Brendan,
<snip>
> Framework looks good. I think it would be helpful to include a real test

Great to hear!

> in the patch series to get a feel for how effective it is.

Alright, will do. Rob suggested converting
https://elixir.bootlin.com/linux/v4.19/source/drivers/of/unittest.c to
KUnit, so that might offer a good comparison.

>
> On one hand, KUnit stands on its own as its own and maybe it should be placed in
> under tools/testing/KUnit, however I am wondering would it be beneficial for the
> framework to under selftests.
>
> I am a bit concerned about the number of test framework we have at the moment and
> are we running the risk of fragmenting the landscape. I am concerned if this would
> lead to developer confusion as to where to add tests.

On one hand separating them makes sense because they are different
things. kselftest seems to do its job, writing end-to-end tests,
pretty well. Trying to associate KUnit with it could also be confusing
since they have different requirements and should be used for
different things. But on the other hand, you are right, it would be
nice to have a coherent experience for developers. In either case, we
will still need developers to understand when they should be writing
unit tests and when they should be writing end-to-end tests. So in the
end, I don't think it will matter too much, we will still need to
address this confusion either way.

The main reason I made it separate was just because I thought people
would find it strange to have essentially normal kernel code depend on
something in `tools/testing/`. You, me, and Greg had talked about this
elsewhere, in summary, I am trying to do in-tree unit testing, so
tests should live side-by-side with the code it tests, and the tests
run at the same level of abstraction as the code that is under test.
The tests do not run as userspace programs on an installed kernel
under test. The goal is to be able to run arbitrary kernel code in
isolation.

Although it would be *possible* to put KUnit under `tools/testing/`,
my understanding is that `tools/testing/` is for programs that compile
independently of the kernel, rather they do not link against anything
built-in to the kernel. (I saw there are some test drivers, but they
seem to be modeled as out of tree drivers that have no option to be
built into the kernel.) KUnit does not fall into this category as
KUnit links directly against arbitrary kernel code, and actually
reuses kernel infrastructure to provide an environment to run kernel
code in.

I do not feel strongly about where KUnit lives, but by my
understanding, putting KUnit in `tools/testing/` might break
assumptions about the relationship between `tools/` and the rest of
the kernel.

I know you previously said that there are unit tests under kselftest
(like maybe this one:
https://elixir.bootlin.com/linux/v4.19/source/lib/locking-selftest.c
?). I agree that something like this example is trying to be a unit
test, but the kselftest infrastructure is built around the idea of
booting kernels and running tests against them; I know that some
kselftests load test modules, but the point is the intention is to run
the kernel somewhere. My end goal with KUnit is that we should not
need to run a kernel anywhere (I know right now I am using UML, but
the goal is to get away from that). So I think that decidedly makes
them two different things.

The number of test frameworks is a problem, I definitely agree with
that. Most of the other tests I have seen for the kernel are
indisputably end-to-end tests; it seems that kselftest is decidedly
the favorite, maybe we should move those under kselftest?

I know there are other things that don't fit. Maybe we could start off
with some suggestions for best practices for what to use and when?
>
> That being said, I don't have a strong opinion one way or the other.

I think it is pretty common on other projects to have unit tests
separated from end-to-end tests in some way or another. You want to be
able to run unit tests quickly and all the time, whereas you usually
only want to run your end-to-end tests when you make a submission or a
release. Regardless of where we put KUnit and what its relationship to
kselftest becomes, we should definitely make it easy to run unit tests
separate from all the end-to-end tests.
>
> btw I started playing with kunit following the instructions and ran into problems:
>
> ./tools/testing/kunit/kunit.py
> usage: kunit.py [-h] {run,new} ...
>
> Helps writing and running KUnit tests.
>
> positional arguments:
>   {run,new}
>     run       Runs KUnit tests.
>     new       Prints out boilerplate for writing new tests.
>
> optional arguments:
>   -h, --help  show this help message and exit
>
> ./tools/testing/kunit/kunit.py run
> Regenerating .config ...
> ERROR:root:Provided Kconfig is not contained in validated .config!

Oh sorry, I need to write some documentation for that.

I take it you checked out
https://kunit.googlesource.com/linux/+/kunit/alpha/master ?

If so, you need a "kunitconfig" in the root of your kernel source with
the following Kconfig options:
CONFIG_TEST=y
CONFIG_TEST_TEST=y
CONFIG_EXAMPLE_TEST=y

I am guessing what happened is that you used the "stable" branch which
we have been using internally, but follow the instructions I posted
for this patchset. Some of the Kconfig options have deviated between
them. Sorry about the confusion.

>
> thanks,
> -- Shuah

Thank you!

  reply	other threads:[~2018-11-07  1:17 UTC|newest]

Thread overview: 30+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-10-23 23:57 Brendan Higgins
     [not found] ` <20181023235750.103146-1-brendanhiggins-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
2018-10-23 23:57   ` [RFC v2 01/14] kunit: test: add KUnit test runner core Brendan Higgins
2018-11-02 18:44     ` Shuah Khan
     [not found]       ` <017b111f-d960-c1ef-46ae-eb0eb639fe5b-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
2018-11-07  1:28         ` Brendan Higgins
2018-11-07 20:02           ` Shuah Khan
2018-10-23 23:57   ` [RFC v2 02/14] kunit: test: add test resource management API Brendan Higgins
2018-10-23 23:57   ` [RFC v2 03/14] kunit: test: add string_stream a std::stream like string builder Brendan Higgins
2018-10-23 23:57   ` [RFC v2 04/14] kunit: test: add test_stream a std::stream like logger Brendan Higgins
2018-10-23 23:57   ` [RFC v2 05/14] kunit: test: add the concept of expectations Brendan Higgins
2018-10-23 23:57   ` [RFC v2 06/14] arch: um: enable running kunit from User Mode Linux Brendan Higgins
2018-10-23 23:57   ` [RFC v2 07/14] kunit: test: add initial tests Brendan Higgins
2018-10-23 23:57   ` [RFC v2 08/14] arch: um: add shim to trap to allow installing a fault catcher for tests Brendan Higgins
2018-10-23 23:57   ` [RFC v2 09/14] kunit: test: add the concept of assertions Brendan Higgins
2018-10-23 23:57   ` [RFC v2 10/14] kunit: add Python libraries for handing KUnit config and kernel Brendan Higgins
2018-10-23 23:57   ` [RFC v2 11/14] kunit: add KUnit wrapper script and simple output parser Brendan Higgins
2018-10-23 23:57   ` [RFC v2 12/14] kunit.py: improve output from python wrapper Brendan Higgins
2018-10-23 23:57   ` [RFC v2 13/14] Documentation: kunit: add documentation for KUnit Brendan Higgins
2018-10-23 23:57   ` [RFC v2 14/14] MAINTAINERS: add entry for KUnit the unit testing framework Brendan Higgins
2018-10-24  9:14 ` [RFC v2 00/14] kunit: introduce KUnit, the Linux kernel " Daniel Vetter
2018-10-25 21:25   ` Brendan Higgins
2018-10-25 17:40 ` Shuah Khan
2018-11-02 18:23 ` Shuah Khan
2018-11-07  1:17   ` Brendan Higgins [this message]
2018-11-07 17:46     ` Frank Rowand
     [not found]       ` <04f677b1-bc44-f004-cf2a-51b47baf0965-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
2018-11-13 10:10         ` Brendan Higgins
2018-11-24  5:15 ` Knut Omang
     [not found]   ` <1543036529.4680.655.camel-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
2018-11-27  1:41     ` Brendan Higgins
2018-11-28 19:54       ` Knut Omang
2018-11-28 20:50         ` shuah
2018-11-30  0:59           ` Luis Chamberlain

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAFd5g45Tgcx4RisrCF8QHneR7ELezPfpEPKh_mH8KEUi3skE7g@mail.gmail.com \
    --to=brendanhiggins@google.com \
    --cc=Tim.Bird@sony.com \
    --cc=brakmo@fb.com \
    --cc=dan.j.williams@intel.com \
    --cc=daniel@ffwll.ch \
    --cc=dri-devel@lists.freedesktop.org \
    --cc=gregkh@linuxfoundation.org \
    --cc=jdike@addtoit.com \
    --cc=joe@perches.com \
    --cc=joel@jms.id.au \
    --cc=julia.lawall@lip6.fr \
    --cc=keescook@google.com \
    --cc=khilman@baylibre.com \
    --cc=kieran.bingham@ideasonboard.com \
    --cc=kunit-dev@googlegroups.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=linux-nvdimm@lists.01.org \
    --cc=linux-um@lists.infradead.org \
    --cc=mcgrof@kernel.org \
    --cc=mpe@ellerman.id.au \
    --cc=richard@nod.at \
    --cc=robh@kernel.org \
    --cc=rostedt@goodmis.org \
    --cc=shuah@kernel.org \
    --subject='Re: [RFC v2 00/14] kunit: introduce KUnit, the Linux kernel unit testing framework' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).