From: Brendan Higgins <brendanhiggins@google.com>
To: Frank Rowand <frowand.list@gmail.com>
Cc: Theodore Ts'o <tytso@mit.edu>,
Greg KH <gregkh@linuxfoundation.org>,
keescook@google.com, kieran.bingham@ideasonboard.com,
mcgrof@kernel.org, robh@kernel.org, sboyd@kernel.org,
shuah@kernel.org, devicetree@vger.kernel.org,
dri-devel@lists.freedesktop.org, kunit-dev@googlegroups.com,
linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org,
linux-kbuild@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-kselftest@vger.kernel.org, linux-nvdimm@lists.01.org,
linux-um@lists.infradead.org, Alexander.Levin@microsoft.com,
Tim.Bird@sony.com, amir73il@gmail.com, dan.carpenter@oracle.com,
dan.j.williams@intel.com, daniel@ffwll.ch, jdike@addtoit.com,
joel@jms.id.au, julia.lawall@lip6.fr, khilman@baylibre.com,
knut.omang@oracle.com, logang@deltatee.com, mpe@ellerman.id.au,
pmladek@suse.com, richard@nod.at, rientjes@google.com,
rostedt@goodmis.org, wfg@linux.intel.com
Subject: Re: [PATCH v2 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework
Date: Tue, 14 May 2019 01:22:14 -0700 [thread overview]
Message-ID: <20190514082214.GB230665@google.com> (raw)
In-Reply-To: <4d963cdc-1cbb-35a3-292c-552f865ed1f7@gmail.com>
On Wed, May 08, 2019 at 05:58:49PM -0700, Frank Rowand wrote:
> Hi Ted,
>
> On 5/7/19 10:22 AM, Theodore Ts'o wrote:
> > On Tue, May 07, 2019 at 10:01:19AM +0200, Greg KH wrote:
> Not very helpful to cut the text here, plus not explicitly indicating that
> text was cut (yes, I know the ">>>" will be a clue for the careful reader),
> losing the set up for my question.
>
>
> >>> My understanding is that the intent of KUnit is to avoid booting a kernel on
> >>> real hardware or in a virtual machine. That seems to be a matter of semantics
> >>> to me because isn't invoking a UML Linux just running the Linux kernel in
> >>> a different form of virtualization?
> >>>
> >>> So I do not understand why KUnit is an improvement over kselftest.
> >>>
> >>> It seems to me that KUnit is just another piece of infrastructure that I
> >>> am going to have to be familiar with as a kernel developer. More overhead,
> >>> more information to stuff into my tiny little brain.
> >>>
> >>> I would guess that some developers will focus on just one of the two test
> >>> environments (and some will focus on both), splitting the development
> >>> resources instead of pooling them on a common infrastructure.
> >>>
> >>> What am I missing?
> >>
> >> kselftest provides no in-kernel framework for testing kernel code
> >> specifically. That should be what kunit provides, an "easy" way to
> >> write in-kernel tests for things.
> >>
> >> Brendan, did I get it right?
> >
> > Yes, that's basically right. You don't *have* to use KUnit. It's
>
> If KUnit is added to the kernel, and a subsystem that I am submitting
> code for has chosen to use KUnit instead of kselftest, then yes, I do
> *have* to use KUnit if my submission needs to contain a test for the
> code unless I want to convince the maintainer that somehow my case
> is special and I prefer to use kselftest instead of KUnittest.
>
>
> > supposed to be a simple way to run a large number of small tests that
> > for specific small components in a system.
>
> kselftest also supports running a subset of tests. That subset of tests
> can also be a large number of small tests. There is nothing inherent
> in KUnit vs kselftest in this regard, as far as I am aware.
>
>
> > For example, I currently use xfstests using KVM and GCE to test all of
> > ext4. These tests require using multiple 5 GB and 20GB virtual disks,
> > and it works by mounting ext4 file systems and exercising ext4 through
> > the system call interfaces, using userspace tools such as fsstress,
> > fsx, fio, etc. It requires time overhead to start the VM, create and
> > allocate virtual disks, etc. For example, to run a single 3 seconds
> > xfstest (generic/001), it requires full 10 seconds to run it via
> > kvm-xfstests.
> >
>
>
> > KUnit is something else; it's specifically intended to allow you to
> > create lightweight tests quickly and easily, and by reducing the
> > effort needed to write and run unit tests, hopefully we'll have a lot
> > more of them and thus improve kernel quality.
>
> The same is true of kselftest. You can create lightweight tests in
> kselftest.
>
>
> > As an example, I have a volunteer working on developing KUinit tests
> > for ext4. We're going to start by testing the ext4 extent status
> > tree. The source code is at fs/ext4/extent_status.c; it's
> > approximately 1800 LOC. The Kunit tests for the extent status tree
> > will exercise all of the corner cases for the various extent status
> > tree functions --- e.g., ext4_es_insert_delayed_block(),
> > ext4_es_remove_extent(), ext4_es_cache_extent(), etc. And it will do
> > this in isolation without our needing to create a test file system or
> > using a test block device.
> >
>
> > Next we'll test the ext4 block allocator, again in isolation. To test
> > the block allocator we will have to write "mock functions" which
> > simulate reading allocation bitmaps from disk. Again, this will allow
> > the test writer to explicitly construct corner cases and validate that
> > the block allocator works as expected without having to reverese
> > engineer file system data structures which will force a particular
> > code path to be executed.
>
> This would be a difference, but mock functions do not exist in KUnit.
> The KUnit test will call the real kernel function in the UML kernel.
>
> I think Brendan has indicated a desire to have mock functions in the
> future.
>
> Brendan, do I understand that correctly?
Oh, sorry, I missed this comment from earlier.
Yes, you are correct. Function mocking is a feature I will be
introducing in a follow up patchset (assuming this one gets merged of
course ;-) ).
Cheers!
> -Frank
>
> > So this is why it's largely irrelevant to me that KUinit uses UML. In
> > fact, it's a feature. We're not testing device drivers, or the
> > scheduler, or anything else architecture-specific. UML is not about
> > virtualization. What it's about in this context is allowing us to
> > start running test code as quickly as possible. Booting KVM takes
> > about 3-4 seconds, and this includes initializing virtio_scsi and
> > other device drivers. If by using UML we can hold the amount of
> > unnecessary kernel subsystem initialization down to the absolute
> > minimum, and if it means that we can communicating to the test
> > framework via a userspace "printf" from UML/KUnit code, as opposed to
> > via a virtual serial port to KVM's virtual console, it all makes for
> > lighter weight testing.
> >
> > Why did I go looking for a volunteer to write KUnit tests for ext4?
> > Well, I have a plan to make some changes in restructing how ext4's
> > write path works, in order to support things like copy-on-write, a
> > more efficient delayed allocation system, etc. This will require
> > making changes to the extent status tree, and by having unit tests for
> > the extent status tree, we'll be able to detect any bugs that we might
> > accidentally introduce in the es tree far more quickly than if we
> > didn't have those tests available. Google has long found that having
> > these sorts of unit tests is a real win for developer velocity for any
> > non-trivial code module (or C++ class), even when you take into
> > account the time it takes to create the unit tests.
> >
> > - Ted>
> > P.S. Many thanks to Brendan for finding such a volunteer for me; the
> > person in question is a SRE from Switzerland who is interested in
> > getting involved with kernel testing, and this is going to be their
> > 20% project. :-)
> >
> >
next prev parent reply other threads:[~2019-05-14 8:22 UTC|newest]
Thread overview: 131+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-05-01 23:01 [PATCH v2 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 01/17] kunit: test: add KUnit test runner core Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 02/17] kunit: test: add test resource management API Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 03/17] kunit: test: add string_stream a std::stream like string builder Brendan Higgins
2019-05-03 1:26 ` shuah
2019-05-03 4:37 ` Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 04/17] kunit: test: add kunit_stream a std::stream like logger Brendan Higgins
2019-05-02 11:00 ` Greg KH
2019-05-02 20:25 ` Brendan Higgins
2019-05-02 21:18 ` Frank Rowand
2019-05-03 1:50 ` shuah
2019-05-03 5:48 ` Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 05/17] kunit: test: add the concept of expectations Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 06/17] kbuild: enable building KUnit Brendan Higgins
2019-05-10 3:03 ` Masahiro Yamada
2019-05-10 10:27 ` Brendan Higgins
2019-05-10 10:30 ` Masahiro Yamada
2019-05-10 10:33 ` Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 07/17] kunit: test: add initial tests Brendan Higgins
2019-05-02 10:58 ` Greg KH
2019-05-02 20:30 ` Brendan Higgins
2019-05-03 1:27 ` shuah
2019-05-03 5:18 ` Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 08/17] kunit: test: add support for test abort Brendan Higgins
2019-05-03 3:14 ` Logan Gunthorpe
2019-05-03 6:48 ` Brendan Higgins
2019-05-03 12:33 ` Logan Gunthorpe
2019-05-06 8:48 ` Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 09/17] kunit: test: add tests for kunit " Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 10/17] kunit: test: add the concept of assertions Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 11/17] kunit: test: add test managed resource tests Brendan Higgins
2019-05-03 14:34 ` shuah
2019-05-06 9:03 ` Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 12/17] kunit: tool: add Python wrappers for running KUnit tests Brendan Higgins
2019-05-02 11:02 ` Greg KH
2019-05-02 18:07 ` Brendan Higgins
2019-05-02 21:16 ` Frank Rowand
2019-05-02 23:45 ` Brendan Higgins
2019-05-03 1:45 ` Frank Rowand
2019-05-03 5:36 ` Brendan Higgins
2019-05-03 18:59 ` Frank Rowand
2019-05-03 23:14 ` Brendan Higgins
2019-05-04 10:42 ` Greg KH
2019-05-06 0:19 ` Frank Rowand
2019-05-06 17:43 ` Kees Cook
2019-05-06 21:42 ` Brendan Higgins
2019-05-06 21:39 ` Brendan Higgins
2019-05-07 19:13 ` Tim.Bird
2019-05-03 6:41 ` Greg KH
2019-05-01 23:01 ` [PATCH v2 13/17] kunit: defconfig: add defconfigs for building " Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 14/17] Documentation: kunit: add documentation for KUnit Brendan Higgins
2019-05-09 5:08 ` Randy Dunlap
2019-05-09 17:38 ` Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 15/17] MAINTAINERS: add entry for KUnit the unit testing framework Brendan Higgins
2019-05-03 14:38 ` shuah
2019-05-06 9:18 ` Brendan Higgins
2019-05-01 23:01 ` [PATCH v2 16/17] kernel/sysctl-test: Add null pointer test for sysctl.c:proc_dointvec() Brendan Higgins
2019-05-02 11:03 ` Greg KH
2019-05-02 18:14 ` Tim.Bird
2019-05-02 18:45 ` Brendan Higgins
2019-05-03 6:42 ` Greg KH
2019-05-03 23:41 ` Brendan Higgins
2019-05-04 10:40 ` Greg KH
2019-05-01 23:01 ` [PATCH v2 17/17] MAINTAINERS: add proc sysctl KUnit test to PROC SYSCTL section Brendan Higgins
2019-05-02 10:50 ` [PATCH v2 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework Greg KH
2019-05-02 11:05 ` Greg KH
2019-05-03 0:41 ` Brendan Higgins
2019-05-02 14:04 ` shuah
2019-05-03 0:44 ` Brendan Higgins
2019-05-03 3:18 ` Logan Gunthorpe
2019-05-07 3:14 ` Frank Rowand
2019-05-07 8:01 ` Greg KH
2019-05-07 15:23 ` shuah
2019-05-09 1:01 ` Frank Rowand
2019-05-07 17:22 ` Theodore Ts'o
2019-05-08 19:17 ` Brendan Higgins
2019-05-09 0:58 ` Frank Rowand
2019-05-09 1:44 ` Theodore Ts'o
2019-05-09 2:18 ` Frank Rowand
2019-05-14 8:22 ` Brendan Higgins [this message]
2019-05-09 0:43 ` Frank Rowand
2019-05-09 1:58 ` Theodore Ts'o
2019-05-09 2:13 ` Frank Rowand
2019-05-09 3:20 ` Theodore Ts'o
2019-05-09 11:52 ` Knut Omang
2019-05-09 13:35 ` Theodore Ts'o
2019-05-09 14:48 ` Knut Omang
2019-05-09 17:00 ` Tim.Bird
2019-05-09 17:42 ` Daniel Vetter
2019-05-09 18:12 ` Frank Rowand
2019-05-09 21:42 ` Theodore Ts'o
2019-05-09 22:20 ` Logan Gunthorpe
2019-05-09 23:30 ` Theodore Ts'o
2019-05-09 23:40 ` Logan Gunthorpe
2019-05-10 4:47 ` Theodore Ts'o
2019-05-10 5:18 ` Frank Rowand
2019-05-10 5:48 ` Knut Omang
2019-05-10 8:12 ` Daniel Vetter
2019-05-10 10:23 ` Brendan Higgins
2019-05-10 12:12 ` Knut Omang
2019-05-10 20:54 ` Brendan Higgins
2019-05-10 22:18 ` Frank Rowand
2019-05-11 6:17 ` Knut Omang
2019-05-14 6:39 ` Brendan Higgins
2019-05-10 21:59 ` Frank Rowand
2019-05-11 6:43 ` Knut Omang
2019-05-14 8:00 ` Brendan Higgins
2019-05-10 11:36 ` Knut Omang
2019-05-10 16:17 ` Logan Gunthorpe
2019-05-10 22:13 ` Frank Rowand
2019-05-14 8:38 ` Brendan Higgins
2019-05-15 0:14 ` Frank Rowand
2019-05-15 0:26 ` Logan Gunthorpe
2019-05-10 21:52 ` Frank Rowand
2019-05-14 20:54 ` Brendan Higgins
2019-05-10 21:12 ` Frank Rowand
2019-05-11 17:33 ` Theodore Ts'o
2019-05-13 14:44 ` Daniel Vetter
2019-05-14 6:04 ` Brendan Higgins
2019-05-14 12:05 ` Daniel Vetter
2019-05-14 18:36 ` Brendan Higgins
2019-05-15 7:41 ` Daniel Vetter
2019-05-22 21:38 ` Brendan Higgins
2019-05-23 8:40 ` Daniel Vetter
2019-05-15 0:26 ` Frank Rowand
2019-05-15 4:28 ` Theodore Ts'o
2019-05-10 5:11 ` Frank Rowand
2019-05-10 10:43 ` Theodore Ts'o
2019-05-10 21:05 ` Frank Rowand
2019-05-09 15:19 ` Masahiro Yamada
2019-05-10 10:25 ` Brendan Higgins
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190514082214.GB230665@google.com \
--to=brendanhiggins@google.com \
--cc=Alexander.Levin@microsoft.com \
--cc=Tim.Bird@sony.com \
--cc=amir73il@gmail.com \
--cc=dan.carpenter@oracle.com \
--cc=dan.j.williams@intel.com \
--cc=daniel@ffwll.ch \
--cc=devicetree@vger.kernel.org \
--cc=dri-devel@lists.freedesktop.org \
--cc=frowand.list@gmail.com \
--cc=gregkh@linuxfoundation.org \
--cc=jdike@addtoit.com \
--cc=joel@jms.id.au \
--cc=julia.lawall@lip6.fr \
--cc=keescook@google.com \
--cc=khilman@baylibre.com \
--cc=kieran.bingham@ideasonboard.com \
--cc=knut.omang@oracle.com \
--cc=kunit-dev@googlegroups.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kbuild@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-nvdimm@lists.01.org \
--cc=linux-um@lists.infradead.org \
--cc=logang@deltatee.com \
--cc=mcgrof@kernel.org \
--cc=mpe@ellerman.id.au \
--cc=pmladek@suse.com \
--cc=richard@nod.at \
--cc=rientjes@google.com \
--cc=robh@kernel.org \
--cc=rostedt@goodmis.org \
--cc=sboyd@kernel.org \
--cc=shuah@kernel.org \
--cc=tytso@mit.edu \
--cc=wfg@linux.intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).