From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id DCF5A2125583F for ; Tue, 7 May 2019 10:24:11 -0700 (PDT) Date: Tue, 7 May 2019 13:22:56 -0400 From: "Theodore Ts'o" Subject: Re: [PATCH v2 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework Message-ID: <20190507172256.GB5900@mit.edu> References: <20190501230126.229218-1-brendanhiggins@google.com> <54940124-50df-16ec-1a32-ad794ee05da7@gmail.com> <20190507080119.GB28121@kroah.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20190507080119.GB28121@kroah.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: Greg KH Cc: pmladek@suse.com, linux-doc@vger.kernel.org, amir73il@gmail.com, Brendan Higgins , dri-devel@lists.freedesktop.org, Alexander.Levin@microsoft.com, mpe@ellerman.id.au, linux-kselftest@vger.kernel.org, shuah@kernel.org, robh@kernel.org, linux-nvdimm@lists.01.org, Frank Rowand , knut.omang@oracle.com, kieran.bingham@ideasonboard.com, wfg@linux.intel.com, joel@jms.id.au, rientjes@google.com, jdike@addtoit.com, dan.carpenter@oracle.com, devicetree@vger.kernel.org, linux-kbuild@vger.kernel.org, Tim.Bird@sony.com, linux-um@lists.infradead.org, rostedt@goodmis.org, julia.lawall@lip6.fr, kunit-dev@googlegroups.com, richard@nod.at, sboyd@kernel.org, linux-kernel@vger.kernel.org, mcgrof@kernel.org, daniel@ffwll.ch, keescook@google.com, linux-fsdevel@vger.kernel.org, khilman@baylibre.com List-ID: On Tue, May 07, 2019 at 10:01:19AM +0200, Greg KH wrote: > > My understanding is that the intent of KUnit is to avoid booting a kernel on > > real hardware or in a virtual machine. That seems to be a matter of semantics > > to me because isn't invoking a UML Linux just running the Linux kernel in > > a different form of virtualization? > > > > So I do not understand why KUnit is an improvement over kselftest. > > > > It seems to me that KUnit is just another piece of infrastructure that I > > am going to have to be familiar with as a kernel developer. More overhead, > > more information to stuff into my tiny little brain. > > > > I would guess that some developers will focus on just one of the two test > > environments (and some will focus on both), splitting the development > > resources instead of pooling them on a common infrastructure. > > > > What am I missing? > > kselftest provides no in-kernel framework for testing kernel code > specifically. That should be what kunit provides, an "easy" way to > write in-kernel tests for things. > > Brendan, did I get it right? Yes, that's basically right. You don't *have* to use KUnit. It's supposed to be a simple way to run a large number of small tests that for specific small components in a system. For example, I currently use xfstests using KVM and GCE to test all of ext4. These tests require using multiple 5 GB and 20GB virtual disks, and it works by mounting ext4 file systems and exercising ext4 through the system call interfaces, using userspace tools such as fsstress, fsx, fio, etc. It requires time overhead to start the VM, create and allocate virtual disks, etc. For example, to run a single 3 seconds xfstest (generic/001), it requires full 10 seconds to run it via kvm-xfstests. KUnit is something else; it's specifically intended to allow you to create lightweight tests quickly and easily, and by reducing the effort needed to write and run unit tests, hopefully we'll have a lot more of them and thus improve kernel quality. As an example, I have a volunteer working on developing KUinit tests for ext4. We're going to start by testing the ext4 extent status tree. The source code is at fs/ext4/extent_status.c; it's approximately 1800 LOC. The Kunit tests for the extent status tree will exercise all of the corner cases for the various extent status tree functions --- e.g., ext4_es_insert_delayed_block(), ext4_es_remove_extent(), ext4_es_cache_extent(), etc. And it will do this in isolation without our needing to create a test file system or using a test block device. Next we'll test the ext4 block allocator, again in isolation. To test the block allocator we will have to write "mock functions" which simulate reading allocation bitmaps from disk. Again, this will allow the test writer to explicitly construct corner cases and validate that the block allocator works as expected without having to reverese engineer file system data structures which will force a particular code path to be executed. So this is why it's largely irrelevant to me that KUinit uses UML. In fact, it's a feature. We're not testing device drivers, or the scheduler, or anything else architecture-specific. UML is not about virtualization. What it's about in this context is allowing us to start running test code as quickly as possible. Booting KVM takes about 3-4 seconds, and this includes initializing virtio_scsi and other device drivers. If by using UML we can hold the amount of unnecessary kernel subsystem initialization down to the absolute minimum, and if it means that we can communicating to the test framework via a userspace "printf" from UML/KUnit code, as opposed to via a virtual serial port to KVM's virtual console, it all makes for lighter weight testing. Why did I go looking for a volunteer to write KUnit tests for ext4? Well, I have a plan to make some changes in restructing how ext4's write path works, in order to support things like copy-on-write, a more efficient delayed allocation system, etc. This will require making changes to the extent status tree, and by having unit tests for the extent status tree, we'll be able to detect any bugs that we might accidentally introduce in the es tree far more quickly than if we didn't have those tests available. Google has long found that having these sorts of unit tests is a real win for developer velocity for any non-trivial code module (or C++ class), even when you take into account the time it takes to create the unit tests. - Ted P.S. Many thanks to Brendan for finding such a volunteer for me; the person in question is a SRE from Switzerland who is interested in getting involved with kernel testing, and this is going to be their 20% project. :-) _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.4 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS,USER_AGENT_MUTT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4A61C004C9 for ; Tue, 7 May 2019 17:24:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A8C5E205ED for ; Tue, 7 May 2019 17:24:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727343AbfEGRYp (ORCPT ); Tue, 7 May 2019 13:24:45 -0400 Received: from outgoing-auth-1.mit.edu ([18.9.28.11]:46096 "EHLO outgoing.mit.edu" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726473AbfEGRYp (ORCPT ); Tue, 7 May 2019 13:24:45 -0400 Received: from callcc.thunk.org (guestnat-104-133-0-109.corp.google.com [104.133.0.109] (may be forged)) (authenticated bits=0) (User authenticated as tytso@ATHENA.MIT.EDU) by outgoing.mit.edu (8.14.7/8.12.4) with ESMTP id x47HMurS031460 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 7 May 2019 13:22:59 -0400 Received: by callcc.thunk.org (Postfix, from userid 15806) id 7D6D5420024; Tue, 7 May 2019 13:22:56 -0400 (EDT) Date: Tue, 7 May 2019 13:22:56 -0400 From: "Theodore Ts'o" To: Greg KH Cc: Frank Rowand , Brendan Higgins , keescook@google.com, kieran.bingham@ideasonboard.com, mcgrof@kernel.org, robh@kernel.org, sboyd@kernel.org, shuah@kernel.org, devicetree@vger.kernel.org, dri-devel@lists.freedesktop.org, kunit-dev@googlegroups.com, linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kbuild@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-nvdimm@lists.01.org, linux-um@lists.infradead.org, Alexander.Levin@microsoft.com, Tim.Bird@sony.com, amir73il@gmail.com, dan.carpenter@oracle.com, dan.j.williams@intel.com, daniel@ffwll.ch, jdike@addtoit.com, joel@jms.id.au, julia.lawall@lip6.fr, khilman@baylibre.com, knut.omang@oracle.com, logang@deltatee.com, mpe@ellerman.id.au, pmladek@suse.com, richard@nod.at, rientjes@google.com, rostedt@goodmis.org, wfg@linux.intel.com Subject: Re: [PATCH v2 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework Message-ID: <20190507172256.GB5900@mit.edu> Mail-Followup-To: Theodore Ts'o , Greg KH , Frank Rowand , Brendan Higgins , keescook@google.com, kieran.bingham@ideasonboard.com, mcgrof@kernel.org, robh@kernel.org, sboyd@kernel.org, shuah@kernel.org, devicetree@vger.kernel.org, dri-devel@lists.freedesktop.org, kunit-dev@googlegroups.com, linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kbuild@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-nvdimm@lists.01.org, linux-um@lists.infradead.org, Alexander.Levin@microsoft.com, Tim.Bird@sony.com, amir73il@gmail.com, dan.carpenter@oracle.com, dan.j.williams@intel.com, daniel@ffwll.ch, jdike@addtoit.com, joel@jms.id.au, julia.lawall@lip6.fr, khilman@baylibre.com, knut.omang@oracle.com, logang@deltatee.com, mpe@ellerman.id.au, pmladek@suse.com, richard@nod.at, rientjes@google.com, rostedt@goodmis.org, wfg@linux.intel.com References: <20190501230126.229218-1-brendanhiggins@google.com> <54940124-50df-16ec-1a32-ad794ee05da7@gmail.com> <20190507080119.GB28121@kroah.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20190507080119.GB28121@kroah.com> User-Agent: Mutt/1.10.1 (2018-07-13) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, May 07, 2019 at 10:01:19AM +0200, Greg KH wrote: > > My understanding is that the intent of KUnit is to avoid booting a kernel on > > real hardware or in a virtual machine. That seems to be a matter of semantics > > to me because isn't invoking a UML Linux just running the Linux kernel in > > a different form of virtualization? > > > > So I do not understand why KUnit is an improvement over kselftest. > > > > It seems to me that KUnit is just another piece of infrastructure that I > > am going to have to be familiar with as a kernel developer. More overhead, > > more information to stuff into my tiny little brain. > > > > I would guess that some developers will focus on just one of the two test > > environments (and some will focus on both), splitting the development > > resources instead of pooling them on a common infrastructure. > > > > What am I missing? > > kselftest provides no in-kernel framework for testing kernel code > specifically. That should be what kunit provides, an "easy" way to > write in-kernel tests for things. > > Brendan, did I get it right? Yes, that's basically right. You don't *have* to use KUnit. It's supposed to be a simple way to run a large number of small tests that for specific small components in a system. For example, I currently use xfstests using KVM and GCE to test all of ext4. These tests require using multiple 5 GB and 20GB virtual disks, and it works by mounting ext4 file systems and exercising ext4 through the system call interfaces, using userspace tools such as fsstress, fsx, fio, etc. It requires time overhead to start the VM, create and allocate virtual disks, etc. For example, to run a single 3 seconds xfstest (generic/001), it requires full 10 seconds to run it via kvm-xfstests. KUnit is something else; it's specifically intended to allow you to create lightweight tests quickly and easily, and by reducing the effort needed to write and run unit tests, hopefully we'll have a lot more of them and thus improve kernel quality. As an example, I have a volunteer working on developing KUinit tests for ext4. We're going to start by testing the ext4 extent status tree. The source code is at fs/ext4/extent_status.c; it's approximately 1800 LOC. The Kunit tests for the extent status tree will exercise all of the corner cases for the various extent status tree functions --- e.g., ext4_es_insert_delayed_block(), ext4_es_remove_extent(), ext4_es_cache_extent(), etc. And it will do this in isolation without our needing to create a test file system or using a test block device. Next we'll test the ext4 block allocator, again in isolation. To test the block allocator we will have to write "mock functions" which simulate reading allocation bitmaps from disk. Again, this will allow the test writer to explicitly construct corner cases and validate that the block allocator works as expected without having to reverese engineer file system data structures which will force a particular code path to be executed. So this is why it's largely irrelevant to me that KUinit uses UML. In fact, it's a feature. We're not testing device drivers, or the scheduler, or anything else architecture-specific. UML is not about virtualization. What it's about in this context is allowing us to start running test code as quickly as possible. Booting KVM takes about 3-4 seconds, and this includes initializing virtio_scsi and other device drivers. If by using UML we can hold the amount of unnecessary kernel subsystem initialization down to the absolute minimum, and if it means that we can communicating to the test framework via a userspace "printf" from UML/KUnit code, as opposed to via a virtual serial port to KVM's virtual console, it all makes for lighter weight testing. Why did I go looking for a volunteer to write KUnit tests for ext4? Well, I have a plan to make some changes in restructing how ext4's write path works, in order to support things like copy-on-write, a more efficient delayed allocation system, etc. This will require making changes to the extent status tree, and by having unit tests for the extent status tree, we'll be able to detect any bugs that we might accidentally introduce in the es tree far more quickly than if we didn't have those tests available. Google has long found that having these sorts of unit tests is a real win for developer velocity for any non-trivial code module (or C++ class), even when you take into account the time it takes to create the unit tests. - Ted P.S. Many thanks to Brendan for finding such a volunteer for me; the person in question is a SRE from Switzerland who is interested in getting involved with kernel testing, and this is going to be their 20% project. :-) From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Theodore Ts'o" Subject: Re: [PATCH v2 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework Date: Tue, 7 May 2019 13:22:56 -0400 Message-ID: <20190507172256.GB5900@mit.edu> References: <20190501230126.229218-1-brendanhiggins@google.com> <54940124-50df-16ec-1a32-ad794ee05da7@gmail.com> <20190507080119.GB28121@kroah.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: <20190507080119.GB28121-U8xfFu+wG4EAvxtiuMwx3w@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: Greg KH Cc: pmladek-IBi9RG/b67k@public.gmane.org, linux-doc-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, amir73il-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org, Brendan Higgins , dri-devel-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org, Alexander.Levin-0li6OtcxBFHby3iVrkZq2A@public.gmane.org, mpe-Gsx/Oe8HsFggBc27wqDAHg@public.gmane.org, linux-kselftest-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, shuah-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, robh-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, linux-nvdimm-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org, Frank Rowand , knut.omang-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org, kieran.bingham-ryLnwIuWjnjg/C1BVhZhaw@public.gmane.org, wfg-VuQAYsv1563Yd54FQh9/CA@public.gmane.org, joel-U3u1mxZcP9KHXe+LvDLADg@public.gmane.org, rientjes-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org, jdike-OPE4K8JWMJJBDgjK7y7TUQ@public.gmane.org, dan.carpenter-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org, devicetree-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-kbuild-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Tim.Bird-7U/KSKJipcs@public.gmane.org, linux-um-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org, rostedt-nx8X9YLhiw1AfugRpC6u6w@public.gmane.org, julia.lawall-L2FTfq7BK8M@public.gmane.org, kunit-dev-/JYPxA39Uh5TLH3MbocFFw@public.gmane.org, richard-/L3Ra7n9ekc@public.gmane.org, sboyd-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, mcgrof-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, daniel-/w4YWyX8dFk@public.gmane.org, keescook-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org, linux-fsdevel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, khilman-rdvid1DuHRBWk0Htik3J/w@public.gmane.org List-Id: devicetree@vger.kernel.org On Tue, May 07, 2019 at 10:01:19AM +0200, Greg KH wrote: > > My understanding is that the intent of KUnit is to avoid booting a kernel on > > real hardware or in a virtual machine. That seems to be a matter of semantics > > to me because isn't invoking a UML Linux just running the Linux kernel in > > a different form of virtualization? > > > > So I do not understand why KUnit is an improvement over kselftest. > > > > It seems to me that KUnit is just another piece of infrastructure that I > > am going to have to be familiar with as a kernel developer. More overhead, > > more information to stuff into my tiny little brain. > > > > I would guess that some developers will focus on just one of the two test > > environments (and some will focus on both), splitting the development > > resources instead of pooling them on a common infrastructure. > > > > What am I missing? > > kselftest provides no in-kernel framework for testing kernel code > specifically. That should be what kunit provides, an "easy" way to > write in-kernel tests for things. > > Brendan, did I get it right? Yes, that's basically right. You don't *have* to use KUnit. It's supposed to be a simple way to run a large number of small tests that for specific small components in a system. For example, I currently use xfstests using KVM and GCE to test all of ext4. These tests require using multiple 5 GB and 20GB virtual disks, and it works by mounting ext4 file systems and exercising ext4 through the system call interfaces, using userspace tools such as fsstress, fsx, fio, etc. It requires time overhead to start the VM, create and allocate virtual disks, etc. For example, to run a single 3 seconds xfstest (generic/001), it requires full 10 seconds to run it via kvm-xfstests. KUnit is something else; it's specifically intended to allow you to create lightweight tests quickly and easily, and by reducing the effort needed to write and run unit tests, hopefully we'll have a lot more of them and thus improve kernel quality. As an example, I have a volunteer working on developing KUinit tests for ext4. We're going to start by testing the ext4 extent status tree. The source code is at fs/ext4/extent_status.c; it's approximately 1800 LOC. The Kunit tests for the extent status tree will exercise all of the corner cases for the various extent status tree functions --- e.g., ext4_es_insert_delayed_block(), ext4_es_remove_extent(), ext4_es_cache_extent(), etc. And it will do this in isolation without our needing to create a test file system or using a test block device. Next we'll test the ext4 block allocator, again in isolation. To test the block allocator we will have to write "mock functions" which simulate reading allocation bitmaps from disk. Again, this will allow the test writer to explicitly construct corner cases and validate that the block allocator works as expected without having to reverese engineer file system data structures which will force a particular code path to be executed. So this is why it's largely irrelevant to me that KUinit uses UML. In fact, it's a feature. We're not testing device drivers, or the scheduler, or anything else architecture-specific. UML is not about virtualization. What it's about in this context is allowing us to start running test code as quickly as possible. Booting KVM takes about 3-4 seconds, and this includes initializing virtio_scsi and other device drivers. If by using UML we can hold the amount of unnecessary kernel subsystem initialization down to the absolute minimum, and if it means that we can communicating to the test framework via a userspace "printf" from UML/KUnit code, as opposed to via a virtual serial port to KVM's virtual console, it all makes for lighter weight testing. Why did I go looking for a volunteer to write KUnit tests for ext4? Well, I have a plan to make some changes in restructing how ext4's write path works, in order to support things like copy-on-write, a more efficient delayed allocation system, etc. This will require making changes to the extent status tree, and by having unit tests for the extent status tree, we'll be able to detect any bugs that we might accidentally introduce in the es tree far more quickly than if we didn't have those tests available. Google has long found that having these sorts of unit tests is a real win for developer velocity for any non-trivial code module (or C++ class), even when you take into account the time it takes to create the unit tests. - Ted P.S. Many thanks to Brendan for finding such a volunteer for me; the person in question is a SRE from Switzerland who is interested in getting involved with kernel testing, and this is going to be their 20% project. :-) From mboxrd@z Thu Jan 1 00:00:00 1970 From: tytso at mit.edu (Theodore Ts'o) Date: Tue, 7 May 2019 13:22:56 -0400 Subject: [PATCH v2 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework In-Reply-To: <20190507080119.GB28121@kroah.com> References: <20190501230126.229218-1-brendanhiggins@google.com> <54940124-50df-16ec-1a32-ad794ee05da7@gmail.com> <20190507080119.GB28121@kroah.com> Message-ID: <20190507172256.GB5900@mit.edu> On Tue, May 07, 2019 at 10:01:19AM +0200, Greg KH wrote: > > My understanding is that the intent of KUnit is to avoid booting a kernel on > > real hardware or in a virtual machine. That seems to be a matter of semantics > > to me because isn't invoking a UML Linux just running the Linux kernel in > > a different form of virtualization? > > > > So I do not understand why KUnit is an improvement over kselftest. > > > > It seems to me that KUnit is just another piece of infrastructure that I > > am going to have to be familiar with as a kernel developer. More overhead, > > more information to stuff into my tiny little brain. > > > > I would guess that some developers will focus on just one of the two test > > environments (and some will focus on both), splitting the development > > resources instead of pooling them on a common infrastructure. > > > > What am I missing? > > kselftest provides no in-kernel framework for testing kernel code > specifically. That should be what kunit provides, an "easy" way to > write in-kernel tests for things. > > Brendan, did I get it right? Yes, that's basically right. You don't *have* to use KUnit. It's supposed to be a simple way to run a large number of small tests that for specific small components in a system. For example, I currently use xfstests using KVM and GCE to test all of ext4. These tests require using multiple 5 GB and 20GB virtual disks, and it works by mounting ext4 file systems and exercising ext4 through the system call interfaces, using userspace tools such as fsstress, fsx, fio, etc. It requires time overhead to start the VM, create and allocate virtual disks, etc. For example, to run a single 3 seconds xfstest (generic/001), it requires full 10 seconds to run it via kvm-xfstests. KUnit is something else; it's specifically intended to allow you to create lightweight tests quickly and easily, and by reducing the effort needed to write and run unit tests, hopefully we'll have a lot more of them and thus improve kernel quality. As an example, I have a volunteer working on developing KUinit tests for ext4. We're going to start by testing the ext4 extent status tree. The source code is at fs/ext4/extent_status.c; it's approximately 1800 LOC. The Kunit tests for the extent status tree will exercise all of the corner cases for the various extent status tree functions --- e.g., ext4_es_insert_delayed_block(), ext4_es_remove_extent(), ext4_es_cache_extent(), etc. And it will do this in isolation without our needing to create a test file system or using a test block device. Next we'll test the ext4 block allocator, again in isolation. To test the block allocator we will have to write "mock functions" which simulate reading allocation bitmaps from disk. Again, this will allow the test writer to explicitly construct corner cases and validate that the block allocator works as expected without having to reverese engineer file system data structures which will force a particular code path to be executed. So this is why it's largely irrelevant to me that KUinit uses UML. In fact, it's a feature. We're not testing device drivers, or the scheduler, or anything else architecture-specific. UML is not about virtualization. What it's about in this context is allowing us to start running test code as quickly as possible. Booting KVM takes about 3-4 seconds, and this includes initializing virtio_scsi and other device drivers. If by using UML we can hold the amount of unnecessary kernel subsystem initialization down to the absolute minimum, and if it means that we can communicating to the test framework via a userspace "printf" from UML/KUnit code, as opposed to via a virtual serial port to KVM's virtual console, it all makes for lighter weight testing. Why did I go looking for a volunteer to write KUnit tests for ext4? Well, I have a plan to make some changes in restructing how ext4's write path works, in order to support things like copy-on-write, a more efficient delayed allocation system, etc. This will require making changes to the extent status tree, and by having unit tests for the extent status tree, we'll be able to detect any bugs that we might accidentally introduce in the es tree far more quickly than if we didn't have those tests available. Google has long found that having these sorts of unit tests is a real win for developer velocity for any non-trivial code module (or C++ class), even when you take into account the time it takes to create the unit tests. - Ted P.S. Many thanks to Brendan for finding such a volunteer for me; the person in question is a SRE from Switzerland who is interested in getting involved with kernel testing, and this is going to be their 20% project. :-) From mboxrd@z Thu Jan 1 00:00:00 1970 From: tytso@mit.edu (Theodore Ts'o) Date: Tue, 7 May 2019 13:22:56 -0400 Subject: [PATCH v2 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework In-Reply-To: <20190507080119.GB28121@kroah.com> References: <20190501230126.229218-1-brendanhiggins@google.com> <54940124-50df-16ec-1a32-ad794ee05da7@gmail.com> <20190507080119.GB28121@kroah.com> Message-ID: <20190507172256.GB5900@mit.edu> Content-Type: text/plain; charset="UTF-8" Message-ID: <20190507172256.KZrOK9nd6FvaJjaVW5fkMOKIznZoVhiumXNvGmqMzj0@z> On Tue, May 07, 2019@10:01:19AM +0200, Greg KH wrote: > > My understanding is that the intent of KUnit is to avoid booting a kernel on > > real hardware or in a virtual machine. That seems to be a matter of semantics > > to me because isn't invoking a UML Linux just running the Linux kernel in > > a different form of virtualization? > > > > So I do not understand why KUnit is an improvement over kselftest. > > > > It seems to me that KUnit is just another piece of infrastructure that I > > am going to have to be familiar with as a kernel developer. More overhead, > > more information to stuff into my tiny little brain. > > > > I would guess that some developers will focus on just one of the two test > > environments (and some will focus on both), splitting the development > > resources instead of pooling them on a common infrastructure. > > > > What am I missing? > > kselftest provides no in-kernel framework for testing kernel code > specifically. That should be what kunit provides, an "easy" way to > write in-kernel tests for things. > > Brendan, did I get it right? Yes, that's basically right. You don't *have* to use KUnit. It's supposed to be a simple way to run a large number of small tests that for specific small components in a system. For example, I currently use xfstests using KVM and GCE to test all of ext4. These tests require using multiple 5 GB and 20GB virtual disks, and it works by mounting ext4 file systems and exercising ext4 through the system call interfaces, using userspace tools such as fsstress, fsx, fio, etc. It requires time overhead to start the VM, create and allocate virtual disks, etc. For example, to run a single 3 seconds xfstest (generic/001), it requires full 10 seconds to run it via kvm-xfstests. KUnit is something else; it's specifically intended to allow you to create lightweight tests quickly and easily, and by reducing the effort needed to write and run unit tests, hopefully we'll have a lot more of them and thus improve kernel quality. As an example, I have a volunteer working on developing KUinit tests for ext4. We're going to start by testing the ext4 extent status tree. The source code is at fs/ext4/extent_status.c; it's approximately 1800 LOC. The Kunit tests for the extent status tree will exercise all of the corner cases for the various extent status tree functions --- e.g., ext4_es_insert_delayed_block(), ext4_es_remove_extent(), ext4_es_cache_extent(), etc. And it will do this in isolation without our needing to create a test file system or using a test block device. Next we'll test the ext4 block allocator, again in isolation. To test the block allocator we will have to write "mock functions" which simulate reading allocation bitmaps from disk. Again, this will allow the test writer to explicitly construct corner cases and validate that the block allocator works as expected without having to reverese engineer file system data structures which will force a particular code path to be executed. So this is why it's largely irrelevant to me that KUinit uses UML. In fact, it's a feature. We're not testing device drivers, or the scheduler, or anything else architecture-specific. UML is not about virtualization. What it's about in this context is allowing us to start running test code as quickly as possible. Booting KVM takes about 3-4 seconds, and this includes initializing virtio_scsi and other device drivers. If by using UML we can hold the amount of unnecessary kernel subsystem initialization down to the absolute minimum, and if it means that we can communicating to the test framework via a userspace "printf" from UML/KUnit code, as opposed to via a virtual serial port to KVM's virtual console, it all makes for lighter weight testing. Why did I go looking for a volunteer to write KUnit tests for ext4? Well, I have a plan to make some changes in restructing how ext4's write path works, in order to support things like copy-on-write, a more efficient delayed allocation system, etc. This will require making changes to the extent status tree, and by having unit tests for the extent status tree, we'll be able to detect any bugs that we might accidentally introduce in the es tree far more quickly than if we didn't have those tests available. Google has long found that having these sorts of unit tests is a real win for developer velocity for any non-trivial code module (or C++ class), even when you take into account the time it takes to create the unit tests. - Ted P.S. Many thanks to Brendan for finding such a volunteer for me; the person in question is a SRE from Switzerland who is interested in getting involved with kernel testing, and this is going to be their 20% project. :-) From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from outgoing-auth-1.mit.edu ([18.9.28.11] helo=outgoing.mit.edu) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1hO3pD-00018l-65 for linux-um@lists.infradead.org; Tue, 07 May 2019 17:24:21 +0000 Date: Tue, 7 May 2019 13:22:56 -0400 From: "Theodore Ts'o" Subject: Re: [PATCH v2 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework Message-ID: <20190507172256.GB5900@mit.edu> References: <20190501230126.229218-1-brendanhiggins@google.com> <54940124-50df-16ec-1a32-ad794ee05da7@gmail.com> <20190507080119.GB28121@kroah.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20190507080119.GB28121@kroah.com> List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-um" Errors-To: linux-um-bounces+geert=linux-m68k.org@lists.infradead.org To: Greg KH Cc: pmladek@suse.com, linux-doc@vger.kernel.org, amir73il@gmail.com, Brendan Higgins , dri-devel@lists.freedesktop.org, Alexander.Levin@microsoft.com, mpe@ellerman.id.au, linux-kselftest@vger.kernel.org, shuah@kernel.org, robh@kernel.org, linux-nvdimm@lists.01.org, Frank Rowand , knut.omang@oracle.com, kieran.bingham@ideasonboard.com, wfg@linux.intel.com, joel@jms.id.au, rientjes@google.com, jdike@addtoit.com, dan.carpenter@oracle.com, devicetree@vger.kernel.org, linux-kbuild@vger.kernel.org, Tim.Bird@sony.com, linux-um@lists.infradead.org, rostedt@goodmis.org, julia.lawall@lip6.fr, dan.j.williams@intel.com, kunit-dev@googlegroups.com, richard@nod.at, sboyd@kernel.org, linux-kernel@vger.kernel.org, mcgrof@kernel.org, daniel@ffwll.ch, keescook@google.com, linux-fsdevel@vger.kernel.org, logang@deltatee.com, khilman@baylibre.com On Tue, May 07, 2019 at 10:01:19AM +0200, Greg KH wrote: > > My understanding is that the intent of KUnit is to avoid booting a kernel on > > real hardware or in a virtual machine. That seems to be a matter of semantics > > to me because isn't invoking a UML Linux just running the Linux kernel in > > a different form of virtualization? > > > > So I do not understand why KUnit is an improvement over kselftest. > > > > It seems to me that KUnit is just another piece of infrastructure that I > > am going to have to be familiar with as a kernel developer. More overhead, > > more information to stuff into my tiny little brain. > > > > I would guess that some developers will focus on just one of the two test > > environments (and some will focus on both), splitting the development > > resources instead of pooling them on a common infrastructure. > > > > What am I missing? > > kselftest provides no in-kernel framework for testing kernel code > specifically. That should be what kunit provides, an "easy" way to > write in-kernel tests for things. > > Brendan, did I get it right? Yes, that's basically right. You don't *have* to use KUnit. It's supposed to be a simple way to run a large number of small tests that for specific small components in a system. For example, I currently use xfstests using KVM and GCE to test all of ext4. These tests require using multiple 5 GB and 20GB virtual disks, and it works by mounting ext4 file systems and exercising ext4 through the system call interfaces, using userspace tools such as fsstress, fsx, fio, etc. It requires time overhead to start the VM, create and allocate virtual disks, etc. For example, to run a single 3 seconds xfstest (generic/001), it requires full 10 seconds to run it via kvm-xfstests. KUnit is something else; it's specifically intended to allow you to create lightweight tests quickly and easily, and by reducing the effort needed to write and run unit tests, hopefully we'll have a lot more of them and thus improve kernel quality. As an example, I have a volunteer working on developing KUinit tests for ext4. We're going to start by testing the ext4 extent status tree. The source code is at fs/ext4/extent_status.c; it's approximately 1800 LOC. The Kunit tests for the extent status tree will exercise all of the corner cases for the various extent status tree functions --- e.g., ext4_es_insert_delayed_block(), ext4_es_remove_extent(), ext4_es_cache_extent(), etc. And it will do this in isolation without our needing to create a test file system or using a test block device. Next we'll test the ext4 block allocator, again in isolation. To test the block allocator we will have to write "mock functions" which simulate reading allocation bitmaps from disk. Again, this will allow the test writer to explicitly construct corner cases and validate that the block allocator works as expected without having to reverese engineer file system data structures which will force a particular code path to be executed. So this is why it's largely irrelevant to me that KUinit uses UML. In fact, it's a feature. We're not testing device drivers, or the scheduler, or anything else architecture-specific. UML is not about virtualization. What it's about in this context is allowing us to start running test code as quickly as possible. Booting KVM takes about 3-4 seconds, and this includes initializing virtio_scsi and other device drivers. If by using UML we can hold the amount of unnecessary kernel subsystem initialization down to the absolute minimum, and if it means that we can communicating to the test framework via a userspace "printf" from UML/KUnit code, as opposed to via a virtual serial port to KVM's virtual console, it all makes for lighter weight testing. Why did I go looking for a volunteer to write KUnit tests for ext4? Well, I have a plan to make some changes in restructing how ext4's write path works, in order to support things like copy-on-write, a more efficient delayed allocation system, etc. This will require making changes to the extent status tree, and by having unit tests for the extent status tree, we'll be able to detect any bugs that we might accidentally introduce in the es tree far more quickly than if we didn't have those tests available. Google has long found that having these sorts of unit tests is a real win for developer velocity for any non-trivial code module (or C++ class), even when you take into account the time it takes to create the unit tests. - Ted P.S. Many thanks to Brendan for finding such a volunteer for me; the person in question is a SRE from Switzerland who is interested in getting involved with kernel testing, and this is going to be their 20% project. :-) _______________________________________________ linux-um mailing list linux-um@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-um