From: shuah <shuah@kernel.org>
To: Greg KH <gregkh@linuxfoundation.org>,
Frank Rowand <frowand.list@gmail.com>
Cc: Brendan Higgins <brendanhiggins@google.com>,
keescook@google.com, kieran.bingham@ideasonboard.com,
mcgrof@kernel.org, robh@kernel.org, sboyd@kernel.org,
devicetree@vger.kernel.org, dri-devel@lists.freedesktop.org,
kunit-dev@googlegroups.com, linux-doc@vger.kernel.org,
linux-fsdevel@vger.kernel.org, linux-kbuild@vger.kernel.org,
linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org,
linux-nvdimm@lists.01.org, linux-um@lists.infradead.org,
Alexander.Levin@microsoft.com, Tim.Bird@sony.com,
amir73il@gmail.com, dan.carpenter@oracle.com,
dan.j.williams@intel.com, daniel@ffwll.ch, jdike@addtoit.com,
joel@jms.id.au, julia.lawall@lip6.fr, khilman@baylibre.com,
knut.omang@oracle.com, logang@deltatee.com, mpe@ellerman.id.au,
pmladek@suse.com, richard@nod.at, rientjes@google.com,
rostedt@goodmis.org, wfg@linux.intel.com,
shuah <shuah@kernel.org>
Subject: Re: [PATCH v2 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework
Date: Tue, 7 May 2019 09:23:31 -0600 [thread overview]
Message-ID: <1b1efa91-0523-21a9-e541-fdc3612bd117@kernel.org> (raw)
In-Reply-To: <20190507080119.GB28121@kroah.com>
On 5/7/19 2:01 AM, Greg KH wrote:
> On Mon, May 06, 2019 at 08:14:12PM -0700, Frank Rowand wrote:
>> On 5/1/19 4:01 PM, Brendan Higgins wrote:
>>> ## TLDR
>>>
>>> I rebased the last patchset on 5.1-rc7 in hopes that we can get this in
>>> 5.2.
>>>
>>> Shuah, I think you, Greg KH, and myself talked off thread, and we agreed
>>> we would merge through your tree when the time came? Am I remembering
>>> correctly?
>>>
>>> ## Background
>>>
>>> This patch set proposes KUnit, a lightweight unit testing and mocking
>>> framework for the Linux kernel.
>>>
>>> Unlike Autotest and kselftest, KUnit is a true unit testing framework;
>>> it does not require installing the kernel on a test machine or in a VM
>>> and does not require tests to be written in userspace running on a host
>>> kernel. Additionally, KUnit is fast: From invocation to completion KUnit
>>> can run several dozen tests in under a second. Currently, the entire
>>> KUnit test suite for KUnit runs in under a second from the initial
>>> invocation (build time excluded).
>>>
>>> KUnit is heavily inspired by JUnit, Python's unittest.mock, and
>>> Googletest/Googlemock for C++. KUnit provides facilities for defining
>>> unit test cases, grouping related test cases into test suites, providing
>>> common infrastructure for running tests, mocking, spying, and much more.
>>
>> As a result of the emails replying to this patch thread, I am now
>> starting to look at kselftest. My level of understanding is based
>> on some slide presentations, an LWN article, https://kselftest.wiki.kernel.org/
>> and a _tiny_ bit of looking at kselftest code.
>>
>> tl;dr; I don't really understand kselftest yet.
>>
>>
>> (1) why KUnit exists
>>
>>> ## What's so special about unit testing?
>>>
>>> A unit test is supposed to test a single unit of code in isolation,
>>> hence the name. There should be no dependencies outside the control of
>>> the test; this means no external dependencies, which makes tests orders
>>> of magnitudes faster. Likewise, since there are no external dependencies,
>>> there are no hoops to jump through to run the tests. Additionally, this
>>> makes unit tests deterministic: a failing unit test always indicates a
>>> problem. Finally, because unit tests necessarily have finer granularity,
>>> they are able to test all code paths easily solving the classic problem
>>> of difficulty in exercising error handling code.
>>
>> (2) KUnit is not meant to replace kselftest
>>
>>> ## Is KUnit trying to replace other testing frameworks for the kernel?
>>>
>>> No. Most existing tests for the Linux kernel are end-to-end tests, which
>>> have their place. A well tested system has lots of unit tests, a
>>> reasonable number of integration tests, and some end-to-end tests. KUnit
>>> is just trying to address the unit test space which is currently not
>>> being addressed.
>>
>> My understanding is that the intent of KUnit is to avoid booting a kernel on
>> real hardware or in a virtual machine. That seems to be a matter of semantics
>> to me because isn't invoking a UML Linux just running the Linux kernel in
>> a different form of virtualization?
>>
>> So I do not understand why KUnit is an improvement over kselftest.
They are in two different categories. Kselftest falls into black box
regression test suite which is a collection of user-space tests with a
few kernel test modules back-ending the tests in some cases.
Kselftest can be used by both kernel developers and users and provides
a good way to regression test releases in test rings.
KUnit is a white box category and is a better fit as unit test framework
for development and provides a in-kernel testing. I wouldn't view them
one replacing the other. They just provide coverage for different areas
of testing.
I wouldn't view KUnit as something that would be easily run in test
rings for example.
Brendan, does that sound about right?
>>
>> It seems to me that KUnit is just another piece of infrastructure that I
>> am going to have to be familiar with as a kernel developer. More overhead,
>> more information to stuff into my tiny little brain.
>>
>> I would guess that some developers will focus on just one of the two test
>> environments (and some will focus on both), splitting the development
>> resources instead of pooling them on a common infrastructure.
>> What am I missing?
>
> kselftest provides no in-kernel framework for testing kernel code
> specifically. That should be what kunit provides, an "easy" way to
> write in-kernel tests for things.
>
> Brendan, did I get it right?
thanks,
-- Shuah
next prev parent reply other threads:[~2019-05-07 15:23 UTC|newest]
Thread overview: 131+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-05-01 23:01 [PATCH v2 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 01/17] kunit: test: add KUnit test runner core Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 02/17] kunit: test: add test resource management API Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 03/17] kunit: test: add string_stream a std::stream like string builder Brendan Higgins
2019-05-03 1:26 ` shuah
2019-05-03 4:37 ` Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 04/17] kunit: test: add kunit_stream a std::stream like logger Brendan Higgins
2019-05-02 11:00 ` Greg KH
2019-05-02 20:25 ` Brendan Higgins
2019-05-02 21:18 ` Frank Rowand
2019-05-03 1:50 ` shuah
2019-05-03 5:48 ` Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 05/17] kunit: test: add the concept of expectations Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 06/17] kbuild: enable building KUnit Brendan Higgins
2019-05-10 3:03 ` Masahiro Yamada
2019-05-10 10:27 ` Brendan Higgins
2019-05-10 10:30 ` Masahiro Yamada
2019-05-10 10:33 ` Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 07/17] kunit: test: add initial tests Brendan Higgins
2019-05-02 10:58 ` Greg KH
2019-05-02 20:30 ` Brendan Higgins
2019-05-03 1:27 ` shuah
2019-05-03 5:18 ` Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 08/17] kunit: test: add support for test abort Brendan Higgins
2019-05-03 3:14 ` Logan Gunthorpe
2019-05-03 6:48 ` Brendan Higgins
2019-05-03 12:33 ` Logan Gunthorpe
2019-05-06 8:48 ` Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 09/17] kunit: test: add tests for kunit " Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 10/17] kunit: test: add the concept of assertions Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 11/17] kunit: test: add test managed resource tests Brendan Higgins
2019-05-03 14:34 ` shuah
2019-05-06 9:03 ` Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 12/17] kunit: tool: add Python wrappers for running KUnit tests Brendan Higgins
2019-05-02 11:02 ` Greg KH
2019-05-02 18:07 ` Brendan Higgins
2019-05-02 21:16 ` Frank Rowand
2019-05-02 23:45 ` Brendan Higgins
2019-05-03 1:45 ` Frank Rowand
2019-05-03 5:36 ` Brendan Higgins
2019-05-03 18:59 ` Frank Rowand
2019-05-03 23:14 ` Brendan Higgins
2019-05-04 10:42 ` Greg KH
2019-05-06 0:19 ` Frank Rowand
2019-05-06 17:43 ` Kees Cook
2019-05-06 21:42 ` Brendan Higgins
2019-05-06 21:39 ` Brendan Higgins
2019-05-07 19:13 ` Tim.Bird
2019-05-03 6:41 ` Greg KH
2019-05-01 23:01 ` [PATCH v2 13/17] kunit: defconfig: add defconfigs for building " Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 14/17] Documentation: kunit: add documentation for KUnit Brendan Higgins
2019-05-09 5:08 ` Randy Dunlap
2019-05-09 17:38 ` Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 15/17] MAINTAINERS: add entry for KUnit the unit testing framework Brendan Higgins
2019-05-03 14:38 ` shuah
2019-05-06 9:18 ` Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 16/17] kernel/sysctl-test: Add null pointer test for sysctl.c:proc_dointvec() Brendan Higgins
2019-05-02 11:03 ` Greg KH
2019-05-02 18:14 ` Tim.Bird
2019-05-02 18:45 ` Brendan Higgins
2019-05-03 6:42 ` Greg KH
2019-05-03 23:41 ` Brendan Higgins
2019-05-04 10:40 ` Greg KH
2019-05-01 23:01 ` [PATCH v2 17/17] MAINTAINERS: add proc sysctl KUnit test to PROC SYSCTL section Brendan Higgins
2019-05-02 10:50 ` [PATCH v2 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework Greg KH
2019-05-02 11:05 ` Greg KH
2019-05-03 0:41 ` Brendan Higgins
2019-05-02 14:04 ` shuah
2019-05-03 0:44 ` Brendan Higgins
2019-05-03 3:18 ` Logan Gunthorpe
2019-05-07 3:14 ` Frank Rowand
2019-05-07 8:01 ` Greg KH
2019-05-07 15:23 ` shuah [this message]
2019-05-09 1:01 ` Frank Rowand
2019-05-07 17:22 ` Theodore Ts'o
2019-05-08 19:17 ` Brendan Higgins
2019-05-09 0:58 ` Frank Rowand
2019-05-09 1:44 ` Theodore Ts'o
2019-05-09 2:18 ` Frank Rowand
2019-05-14 8:22 ` Brendan Higgins
2019-05-09 0:43 ` Frank Rowand
2019-05-09 1:58 ` Theodore Ts'o
2019-05-09 2:13 ` Frank Rowand
2019-05-09 3:20 ` Theodore Ts'o
2019-05-09 11:52 ` Knut Omang
2019-05-09 13:35 ` Theodore Ts'o
2019-05-09 14:48 ` Knut Omang
2019-05-09 17:00 ` Tim.Bird
2019-05-09 17:42 ` Daniel Vetter
2019-05-09 18:12 ` Frank Rowand
2019-05-09 21:42 ` Theodore Ts'o
2019-05-09 22:20 ` Logan Gunthorpe
2019-05-09 23:30 ` Theodore Ts'o
2019-05-09 23:40 ` Logan Gunthorpe
2019-05-10 4:47 ` Theodore Ts'o
2019-05-10 5:18 ` Frank Rowand
2019-05-10 5:48 ` Knut Omang
2019-05-10 8:12 ` Daniel Vetter
2019-05-10 10:23 ` Brendan Higgins
2019-05-10 12:12 ` Knut Omang
2019-05-10 20:54 ` Brendan Higgins
2019-05-10 22:18 ` Frank Rowand
2019-05-11 6:17 ` Knut Omang
2019-05-14 6:39 ` Brendan Higgins
2019-05-10 21:59 ` Frank Rowand
2019-05-11 6:43 ` Knut Omang
2019-05-14 8:00 ` Brendan Higgins
2019-05-10 11:36 ` Knut Omang
2019-05-10 16:17 ` Logan Gunthorpe
2019-05-10 22:13 ` Frank Rowand
2019-05-14 8:38 ` Brendan Higgins
2019-05-15 0:14 ` Frank Rowand
2019-05-15 0:26 ` Logan Gunthorpe
2019-05-10 21:52 ` Frank Rowand
2019-05-14 20:54 ` Brendan Higgins
2019-05-10 21:12 ` Frank Rowand
2019-05-11 17:33 ` Theodore Ts'o
2019-05-13 14:44 ` Daniel Vetter
2019-05-14 6:04 ` Brendan Higgins
2019-05-14 12:05 ` Daniel Vetter
2019-05-14 18:36 ` Brendan Higgins
2019-05-15 7:41 ` Daniel Vetter
2019-05-22 21:38 ` Brendan Higgins
2019-05-23 8:40 ` Daniel Vetter
2019-05-15 0:26 ` Frank Rowand
2019-05-15 4:28 ` Theodore Ts'o
2019-05-10 5:11 ` Frank Rowand
2019-05-10 10:43 ` Theodore Ts'o
2019-05-10 21:05 ` Frank Rowand
2019-05-09 15:19 ` Masahiro Yamada
2019-05-10 10:25 ` Brendan Higgins
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1b1efa91-0523-21a9-e541-fdc3612bd117@kernel.org \
--to=shuah@kernel.org \
--cc=Alexander.Levin@microsoft.com \
--cc=Tim.Bird@sony.com \
--cc=amir73il@gmail.com \
--cc=brendanhiggins@google.com \
--cc=dan.carpenter@oracle.com \
--cc=dan.j.williams@intel.com \
--cc=daniel@ffwll.ch \
--cc=devicetree@vger.kernel.org \
--cc=dri-devel@lists.freedesktop.org \
--cc=frowand.list@gmail.com \
--cc=gregkh@linuxfoundation.org \
--cc=jdike@addtoit.com \
--cc=joel@jms.id.au \
--cc=julia.lawall@lip6.fr \
--cc=keescook@google.com \
--cc=khilman@baylibre.com \
--cc=kieran.bingham@ideasonboard.com \
--cc=knut.omang@oracle.com \
--cc=kunit-dev@googlegroups.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kbuild@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-nvdimm@lists.01.org \
--cc=linux-um@lists.infradead.org \
--cc=logang@deltatee.com \
--cc=mcgrof@kernel.org \
--cc=mpe@ellerman.id.au \
--cc=pmladek@suse.com \
--cc=richard@nod.at \
--cc=rientjes@google.com \
--cc=robh@kernel.org \
--cc=rostedt@goodmis.org \
--cc=sboyd@kernel.org \
--cc=wfg@linux.intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).