From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 891C321290DEE for ; Fri, 21 Jun 2019 07:59:51 -0700 (PDT) Subject: Re: [PATCH v5 00/18] kunit: introduce KUnit, the Linux kernel unit testing framework References: <20190617082613.109131-1-brendanhiggins@google.com> <10feac3e-7621-65e5-fbf0-9c63fcbe09c9@gmail.com> From: shuah Message-ID: <69809117-dcda-160a-ee0a-d1d3b4c5cd8a@kernel.org> Date: Fri, 21 Jun 2019 08:59:48 -0600 MIME-Version: 1.0 In-Reply-To: <10feac3e-7621-65e5-fbf0-9c63fcbe09c9@gmail.com> Content-Language: en-US List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: Frank Rowand , Brendan Higgins , gregkh@linuxfoundation.org, jpoimboe@redhat.com, keescook@google.com, kieran.bingham@ideasonboard.com, mcgrof@kernel.org, peterz@infradead.org, robh@kernel.org, sboyd@kernel.org, tytso@mit.edu, yamada.masahiro@socionext.com Cc: pmladek@suse.com, linux-doc@vger.kernel.org, amir73il@gmail.com, dri-devel@lists.freedesktop.org, Alexander.Levin@microsoft.com, linux-kselftest@vger.kernel.org, shuah , linux-nvdimm@lists.01.org, khilman@baylibre.com, knut.omang@oracle.com, wfg@linux.intel.com, joel@jms.id.au, rientjes@google.com, jdike@addtoit.com, dan.carpenter@oracle.com, devicetree@vger.kernel.org, linux-kbuild@vger.kernel.org, Tim.Bird@sony.com, linux-um@lists.infradead.org, rostedt@goodmis.org, julia.lawall@lip6.fr, kunit-dev@googlegroups.com, richard@nod.at, rdunlap@infradead.org, linux-kernel@vger.kernel.org, daniel@ffwll.ch, mpe@ellerman.id.au, linux-fsdevel@vger.kernel.org List-ID: Hi Brendan, On 6/19/19 7:17 PM, Frank Rowand wrote: > Hi Brendan, > > I am only responding to this because you asked me to in the v4 thread. > > Thank you for evaluating my comments in the v4 thread and asking me to > comment on v5 > > On 6/17/19 1:25 AM, Brendan Higgins wrote: >> ## TL;DR >> >> A not so quick follow-up to Stephen's suggestions on PATCH v4. Nothing >> that really changes any functionality or usage with the minor exception >> of a couple public functions that Stephen asked me to rename. >> Nevertheless, a good deal of clean up and fixes. See changes below. >> >> As for our current status, right now we got Reviewed-bys on all patches >> except: >> >> - [PATCH v5 08/18] objtool: add kunit_try_catch_throw to the noreturn >> list >> >> However, it would probably be good to get reviews/acks from the >> subsystem maintainers on: >> >> - [PATCH v5 06/18] kbuild: enable building KUnit >> - [PATCH v5 08/18] objtool: add kunit_try_catch_throw to the noreturn >> list >> - [PATCH v5 15/18] Documentation: kunit: add documentation for KUnit >> - [PATCH v5 17/18] kernel/sysctl-test: Add null pointer test for >> sysctl.c:proc_dointvec() >> - [PATCH v5 18/18] MAINTAINERS: add proc sysctl KUnit test to PROC >> SYSCTL section >> >> Other than that, I think we should be good to go. >> >> One last thing, I updated the background to include my thoughts on KUnit >> vs. in kernel testing with kselftest in the background sections as >> suggested by Frank in the discussion on PATCH v2. >> >> ## Background >> >> This patch set proposes KUnit, a lightweight unit testing and mocking >> framework for the Linux kernel. >> >> Unlike Autotest and kselftest, KUnit is a true unit testing framework; >> it does not require installing the kernel on a test machine or in a VM >> (however, KUnit still allows you to run tests on test machines or in VMs >> if you want[1]) and does not require tests to be written in userspace >> running on a host kernel. Additionally, KUnit is fast: From invocation >> to completion KUnit can run several dozen tests in under a second. >> Currently, the entire KUnit test suite for KUnit runs in under a second >> from the initial invocation (build time excluded). >> >> KUnit is heavily inspired by JUnit, Python's unittest.mock, and >> Googletest/Googlemock for C++. KUnit provides facilities for defining >> unit test cases, grouping related test cases into test suites, providing >> common infrastructure for running tests, mocking, spying, and much more. >> > > I looked only at this section, as was specifically requested: > >> ### But wait! Doesn't kselftest support in kernel testing?! >> >> In a previous version of this patchset Frank pointed out that kselftest >> already supports writing a test that resides in the kernel using the >> test module feature[2]. LWN did a really great summary on this >> discussion here[3]. >> >> Kselftest has a feature that allows a test module to be loaded into a >> kernel using the kselftest framework; this does allow someone to write >> tests against kernel code not directly exposed to userland; however, it >> does not provide much of a framework around how to structure the tests. >> The kselftest test module feature just provides a header which has a >> standardized way of reporting test failures, > > >> and then provides >> infrastructure to load and run the tests using the kselftest test >> harness. > > The in-kernel tests can also be invoked at boot time if they are > configured (Kconfig) as in-kernel instead of as modules. I did not > check how many of the tests have tri-state configuration to allow > this, but the few that I looked at did. > >> >> The kselftest test module does not seem to be opinionated at all in >> regards to how tests are structured, how they check for failures, how >> tests are organized. Even in the method it provides for reporting >> failures is pretty simple; it doesn't have any more advanced failure >> reporting or logging features. Given what's there, I think it is fair to >> say that it is not actually a framework, but a feature that makes it >> possible for someone to do some checks in kernel space. > > I would call that description a little dismissive. The set of in-kernel > tests that I looked like followed a common pattern and reported results > in a uniform manner. > >> I think I commented on this before. I agree with the statement that there is no overlap between Kselftest and KUnit. I would like see this removed. Kselftest module support supports use-cases KUnit won't be able to. I can build an kernel with Kselftest test modules and use it in the filed to load and run tests if I need to debug a problem and get data from a system. I can't do that with KUnit. In my mind, I am not viewing this as which is better. Kselftest and KUnit both have their place in the kernel development process. It isn't productive and/or necessary to comparing Kselftest and KUnit without a good understanding of the problem spaces for each of these. I would strongly recommend not making reference to Kselftest and talk about what KUnit offers. >> Furthermore, kselftest test module has very few users. I checked for all >> the tests that use it using the following grep command: >> >> grep -Hrn -e 'kselftest_module\.h' >> >> and only got three results: lib/test_strscpy.c, lib/test_printf.c, and >> lib/test_bitmap.c. > Again, unnecessary. KUnit can't replace Kselftest module in the way Kselftest module support can be used for debugging and gathering information on system that might be in active use and not dedicated to test and development alone. I wouldn't hesitate loading a Kselftest test module on my laptop and running tests, but I wouldn't use KUnit the same way. Again, this is not a competition between which is better. Kselftest and KUnit serve different needs and problem spaces. Please redo this documentation to reflect that. thanks, -- Shuah _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A125C43613 for ; Fri, 21 Jun 2019 14:59:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D9651215EA for ; Fri, 21 Jun 2019 14:59:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1561129195; bh=jQOqVXNcfZkUcGV1wjafezc/ZNY8PMgE5h6rHbcBqkg=; h=Subject:To:Cc:References:From:Date:In-Reply-To:List-ID:From; b=1cJiMXAX7r7QU7iui+e9howx4VnjUeIDOwxx10Rgnn2sWbn0rJqvtUfl/dv6K3uJZ olP8wTmtL0XMUbkxpO74AH9kZm7hHyQDWyryukwRyy7F/XFzbKjUSQWDafDlvXCPn7 EXC5eWgG/a/cxTlbzM/dvYIAv6TVbLOMX/Ld/Vyg= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726293AbfFUO7x (ORCPT ); Fri, 21 Jun 2019 10:59:53 -0400 Received: from mail.kernel.org ([198.145.29.99]:44056 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725985AbfFUO7x (ORCPT ); Fri, 21 Jun 2019 10:59:53 -0400 Received: from [192.168.1.112] (c-24-9-64-241.hsd1.co.comcast.net [24.9.64.241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 2FA3F2070B; Fri, 21 Jun 2019 14:59:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1561129191; bh=jQOqVXNcfZkUcGV1wjafezc/ZNY8PMgE5h6rHbcBqkg=; h=Subject:To:Cc:References:From:Date:In-Reply-To:From; b=fVF0cWJ0mpMA6KTd3g5GTFKTqGb07hlbAKRaXsHiU5gZicbhsP8nzhP5lBm7wdR3Q 86P67oj3InItTJvZZf4vHPGbEwHeuhFRn7muUTOHYoK/hhOwBhjYJrlR4tOL/ZkWh0 lyQWIOYhLwZoPcg3Yd91RlXYf2pF/02sm7Qm2pBI= Subject: Re: [PATCH v5 00/18] kunit: introduce KUnit, the Linux kernel unit testing framework To: Frank Rowand , Brendan Higgins , gregkh@linuxfoundation.org, jpoimboe@redhat.com, keescook@google.com, kieran.bingham@ideasonboard.com, mcgrof@kernel.org, peterz@infradead.org, robh@kernel.org, sboyd@kernel.org, tytso@mit.edu, yamada.masahiro@socionext.com Cc: devicetree@vger.kernel.org, dri-devel@lists.freedesktop.org, kunit-dev@googlegroups.com, linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kbuild@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-nvdimm@lists.01.org, linux-um@lists.infradead.org, Alexander.Levin@microsoft.com, Tim.Bird@sony.com, amir73il@gmail.com, dan.carpenter@oracle.com, daniel@ffwll.ch, jdike@addtoit.com, joel@jms.id.au, julia.lawall@lip6.fr, khilman@baylibre.com, knut.omang@oracle.com, logang@deltatee.com, mpe@ellerman.id.au, pmladek@suse.com, rdunlap@infradead.org, richard@nod.at, rientjes@google.com, rostedt@goodmis.org, wfg@linux.intel.com, shuah References: <20190617082613.109131-1-brendanhiggins@google.com> <10feac3e-7621-65e5-fbf0-9c63fcbe09c9@gmail.com> From: shuah Message-ID: <69809117-dcda-160a-ee0a-d1d3b4c5cd8a@kernel.org> Date: Fri, 21 Jun 2019 08:59:48 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.7.0 MIME-Version: 1.0 In-Reply-To: <10feac3e-7621-65e5-fbf0-9c63fcbe09c9@gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Brendan, On 6/19/19 7:17 PM, Frank Rowand wrote: > Hi Brendan, > > I am only responding to this because you asked me to in the v4 thread. > > Thank you for evaluating my comments in the v4 thread and asking me to > comment on v5 > > On 6/17/19 1:25 AM, Brendan Higgins wrote: >> ## TL;DR >> >> A not so quick follow-up to Stephen's suggestions on PATCH v4. Nothing >> that really changes any functionality or usage with the minor exception >> of a couple public functions that Stephen asked me to rename. >> Nevertheless, a good deal of clean up and fixes. See changes below. >> >> As for our current status, right now we got Reviewed-bys on all patches >> except: >> >> - [PATCH v5 08/18] objtool: add kunit_try_catch_throw to the noreturn >> list >> >> However, it would probably be good to get reviews/acks from the >> subsystem maintainers on: >> >> - [PATCH v5 06/18] kbuild: enable building KUnit >> - [PATCH v5 08/18] objtool: add kunit_try_catch_throw to the noreturn >> list >> - [PATCH v5 15/18] Documentation: kunit: add documentation for KUnit >> - [PATCH v5 17/18] kernel/sysctl-test: Add null pointer test for >> sysctl.c:proc_dointvec() >> - [PATCH v5 18/18] MAINTAINERS: add proc sysctl KUnit test to PROC >> SYSCTL section >> >> Other than that, I think we should be good to go. >> >> One last thing, I updated the background to include my thoughts on KUnit >> vs. in kernel testing with kselftest in the background sections as >> suggested by Frank in the discussion on PATCH v2. >> >> ## Background >> >> This patch set proposes KUnit, a lightweight unit testing and mocking >> framework for the Linux kernel. >> >> Unlike Autotest and kselftest, KUnit is a true unit testing framework; >> it does not require installing the kernel on a test machine or in a VM >> (however, KUnit still allows you to run tests on test machines or in VMs >> if you want[1]) and does not require tests to be written in userspace >> running on a host kernel. Additionally, KUnit is fast: From invocation >> to completion KUnit can run several dozen tests in under a second. >> Currently, the entire KUnit test suite for KUnit runs in under a second >> from the initial invocation (build time excluded). >> >> KUnit is heavily inspired by JUnit, Python's unittest.mock, and >> Googletest/Googlemock for C++. KUnit provides facilities for defining >> unit test cases, grouping related test cases into test suites, providing >> common infrastructure for running tests, mocking, spying, and much more. >> > > I looked only at this section, as was specifically requested: > >> ### But wait! Doesn't kselftest support in kernel testing?! >> >> In a previous version of this patchset Frank pointed out that kselftest >> already supports writing a test that resides in the kernel using the >> test module feature[2]. LWN did a really great summary on this >> discussion here[3]. >> >> Kselftest has a feature that allows a test module to be loaded into a >> kernel using the kselftest framework; this does allow someone to write >> tests against kernel code not directly exposed to userland; however, it >> does not provide much of a framework around how to structure the tests. >> The kselftest test module feature just provides a header which has a >> standardized way of reporting test failures, > > >> and then provides >> infrastructure to load and run the tests using the kselftest test >> harness. > > The in-kernel tests can also be invoked at boot time if they are > configured (Kconfig) as in-kernel instead of as modules. I did not > check how many of the tests have tri-state configuration to allow > this, but the few that I looked at did. > >> >> The kselftest test module does not seem to be opinionated at all in >> regards to how tests are structured, how they check for failures, how >> tests are organized. Even in the method it provides for reporting >> failures is pretty simple; it doesn't have any more advanced failure >> reporting or logging features. Given what's there, I think it is fair to >> say that it is not actually a framework, but a feature that makes it >> possible for someone to do some checks in kernel space. > > I would call that description a little dismissive. The set of in-kernel > tests that I looked like followed a common pattern and reported results > in a uniform manner. > >> I think I commented on this before. I agree with the statement that there is no overlap between Kselftest and KUnit. I would like see this removed. Kselftest module support supports use-cases KUnit won't be able to. I can build an kernel with Kselftest test modules and use it in the filed to load and run tests if I need to debug a problem and get data from a system. I can't do that with KUnit. In my mind, I am not viewing this as which is better. Kselftest and KUnit both have their place in the kernel development process. It isn't productive and/or necessary to comparing Kselftest and KUnit without a good understanding of the problem spaces for each of these. I would strongly recommend not making reference to Kselftest and talk about what KUnit offers. >> Furthermore, kselftest test module has very few users. I checked for all >> the tests that use it using the following grep command: >> >> grep -Hrn -e 'kselftest_module\.h' >> >> and only got three results: lib/test_strscpy.c, lib/test_printf.c, and >> lib/test_bitmap.c. > Again, unnecessary. KUnit can't replace Kselftest module in the way Kselftest module support can be used for debugging and gathering information on system that might be in active use and not dedicated to test and development alone. I wouldn't hesitate loading a Kselftest test module on my laptop and running tests, but I wouldn't use KUnit the same way. Again, this is not a competition between which is better. Kselftest and KUnit serve different needs and problem spaces. Please redo this documentation to reflect that. thanks, -- Shuah From mboxrd@z Thu Jan 1 00:00:00 1970 From: shuah Subject: Re: [PATCH v5 00/18] kunit: introduce KUnit, the Linux kernel unit testing framework Date: Fri, 21 Jun 2019 08:59:48 -0600 Message-ID: <69809117-dcda-160a-ee0a-d1d3b4c5cd8a@kernel.org> References: <20190617082613.109131-1-brendanhiggins@google.com> <10feac3e-7621-65e5-fbf0-9c63fcbe09c9@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8"; Format="flowed" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <10feac3e-7621-65e5-fbf0-9c63fcbe09c9@gmail.com> Content-Language: en-US List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: Frank Rowand , Brendan Higgins , gregkh@linuxfoundation.org, jpoimboe@redhat.com, keescook@google.com, kieran.bingham@ideasonboard.com, mcgrof@kernel.org, peterz@infradead.org, robh@kernel.org, sboyd@kernel.org, tytso@mit.edu, yamada.masahiro@socionext.com Cc: pmladek@suse.com, linux-doc@vger.kernel.org, amir73il@gmail.com, dri-devel@lists.freedesktop.org, Alexander.Levin@microsoft.com, linux-kselftest@vger.kernel.org, shuah , linux-nvdimm@lists.01.org, khilman@baylibre.com, knut.omang@oracle.com, wfg@linux.intel.com, joel@jms.id.au, rientjes@google.com, jdike@addtoit.com, dan.carpenter@oracle.com, devicetree@vger.kernel.org, linux-kbuild@vger.kernel.org, Tim.Bird@sony.com, linux-um@lists.infradead.org, rostedt@goodmis.org, julia.lawall@lip6.fr, kunit-dev@googlegroups.com, richard@nod.at, rdunlap@infradead.org, linux-kernel@vger.kernel.org, mpe@ellerman.id.au, linux-fsdevel@vger.kernel.org, logang@deltatee.com List-Id: devicetree@vger.kernel.org SGkgQnJlbmRhbiwKCk9uIDYvMTkvMTkgNzoxNyBQTSwgRnJhbmsgUm93YW5kIHdyb3RlOgo+IEhp IEJyZW5kYW4sCj4gCj4gSSBhbSBvbmx5IHJlc3BvbmRpbmcgdG8gdGhpcyBiZWNhdXNlIHlvdSBh c2tlZCBtZSB0byBpbiB0aGUgdjQgdGhyZWFkLgo+IAo+IFRoYW5rIHlvdSBmb3IgZXZhbHVhdGlu ZyBteSBjb21tZW50cyBpbiB0aGUgdjQgdGhyZWFkIGFuZCBhc2tpbmcgbWUgdG8KPiBjb21tZW50 IG9uIHY1Cj4gCj4gT24gNi8xNy8xOSAxOjI1IEFNLCBCcmVuZGFuIEhpZ2dpbnMgd3JvdGU6Cj4+ ICMjIFRMO0RSCj4+Cj4+IEEgbm90IHNvIHF1aWNrIGZvbGxvdy11cCB0byBTdGVwaGVuJ3Mgc3Vn Z2VzdGlvbnMgb24gUEFUQ0ggdjQuIE5vdGhpbmcKPj4gdGhhdCByZWFsbHkgY2hhbmdlcyBhbnkg ZnVuY3Rpb25hbGl0eSBvciB1c2FnZSB3aXRoIHRoZSBtaW5vciBleGNlcHRpb24KPj4gb2YgYSBj b3VwbGUgcHVibGljIGZ1bmN0aW9ucyB0aGF0IFN0ZXBoZW4gYXNrZWQgbWUgdG8gcmVuYW1lLgo+ PiBOZXZlcnRoZWxlc3MsIGEgZ29vZCBkZWFsIG9mIGNsZWFuIHVwIGFuZCBmaXhlcy4gU2VlIGNo YW5nZXMgYmVsb3cuCj4+Cj4+IEFzIGZvciBvdXIgY3VycmVudCBzdGF0dXMsIHJpZ2h0IG5vdyB3 ZSBnb3QgUmV2aWV3ZWQtYnlzIG9uIGFsbCBwYXRjaGVzCj4+IGV4Y2VwdDoKPj4KPj4gLSBbUEFU Q0ggdjUgMDgvMThdIG9ianRvb2w6IGFkZCBrdW5pdF90cnlfY2F0Y2hfdGhyb3cgdG8gdGhlIG5v cmV0dXJuCj4+ICAgIGxpc3QKPj4KPj4gSG93ZXZlciwgaXQgd291bGQgcHJvYmFibHkgYmUgZ29v ZCB0byBnZXQgcmV2aWV3cy9hY2tzIGZyb20gdGhlCj4+IHN1YnN5c3RlbSBtYWludGFpbmVycyBv bjoKPj4KPj4gLSBbUEFUQ0ggdjUgMDYvMThdIGtidWlsZDogZW5hYmxlIGJ1aWxkaW5nIEtVbml0 Cj4+IC0gW1BBVENIIHY1IDA4LzE4XSBvYmp0b29sOiBhZGQga3VuaXRfdHJ5X2NhdGNoX3Rocm93 IHRvIHRoZSBub3JldHVybgo+PiAgICBsaXN0Cj4+IC0gW1BBVENIIHY1IDE1LzE4XSBEb2N1bWVu dGF0aW9uOiBrdW5pdDogYWRkIGRvY3VtZW50YXRpb24gZm9yIEtVbml0Cj4+IC0gW1BBVENIIHY1 IDE3LzE4XSBrZXJuZWwvc3lzY3RsLXRlc3Q6IEFkZCBudWxsIHBvaW50ZXIgdGVzdCBmb3IKPj4g ICAgc3lzY3RsLmM6cHJvY19kb2ludHZlYygpCj4+IC0gW1BBVENIIHY1IDE4LzE4XSBNQUlOVEFJ TkVSUzogYWRkIHByb2Mgc3lzY3RsIEtVbml0IHRlc3QgdG8gUFJPQwo+PiAgICBTWVNDVEwgc2Vj dGlvbgo+Pgo+PiBPdGhlciB0aGFuIHRoYXQsIEkgdGhpbmsgd2Ugc2hvdWxkIGJlIGdvb2QgdG8g Z28uCj4+Cj4+IE9uZSBsYXN0IHRoaW5nLCBJIHVwZGF0ZWQgdGhlIGJhY2tncm91bmQgdG8gaW5j bHVkZSBteSB0aG91Z2h0cyBvbiBLVW5pdAo+PiB2cy4gaW4ga2VybmVsIHRlc3Rpbmcgd2l0aCBr c2VsZnRlc3QgaW4gdGhlIGJhY2tncm91bmQgc2VjdGlvbnMgYXMKPj4gc3VnZ2VzdGVkIGJ5IEZy YW5rIGluIHRoZSBkaXNjdXNzaW9uIG9uIFBBVENIIHYyLgo+Pgo+PiAjIyBCYWNrZ3JvdW5kCj4+ Cj4+IFRoaXMgcGF0Y2ggc2V0IHByb3Bvc2VzIEtVbml0LCBhIGxpZ2h0d2VpZ2h0IHVuaXQgdGVz dGluZyBhbmQgbW9ja2luZwo+PiBmcmFtZXdvcmsgZm9yIHRoZSBMaW51eCBrZXJuZWwuCj4+Cj4+ IFVubGlrZSBBdXRvdGVzdCBhbmQga3NlbGZ0ZXN0LCBLVW5pdCBpcyBhIHRydWUgdW5pdCB0ZXN0 aW5nIGZyYW1ld29yazsKPj4gaXQgZG9lcyBub3QgcmVxdWlyZSBpbnN0YWxsaW5nIHRoZSBrZXJu ZWwgb24gYSB0ZXN0IG1hY2hpbmUgb3IgaW4gYSBWTQo+PiAoaG93ZXZlciwgS1VuaXQgc3RpbGwg YWxsb3dzIHlvdSB0byBydW4gdGVzdHMgb24gdGVzdCBtYWNoaW5lcyBvciBpbiBWTXMKPj4gaWYg eW91IHdhbnRbMV0pIGFuZCBkb2VzIG5vdCByZXF1aXJlIHRlc3RzIHRvIGJlIHdyaXR0ZW4gaW4g dXNlcnNwYWNlCj4+IHJ1bm5pbmcgb24gYSBob3N0IGtlcm5lbC4gQWRkaXRpb25hbGx5LCBLVW5p dCBpcyBmYXN0OiBGcm9tIGludm9jYXRpb24KPj4gdG8gY29tcGxldGlvbiBLVW5pdCBjYW4gcnVu IHNldmVyYWwgZG96ZW4gdGVzdHMgaW4gdW5kZXIgYSBzZWNvbmQuCj4+IEN1cnJlbnRseSwgdGhl IGVudGlyZSBLVW5pdCB0ZXN0IHN1aXRlIGZvciBLVW5pdCBydW5zIGluIHVuZGVyIGEgc2Vjb25k Cj4+IGZyb20gdGhlIGluaXRpYWwgaW52b2NhdGlvbiAoYnVpbGQgdGltZSBleGNsdWRlZCkuCj4+ Cj4+IEtVbml0IGlzIGhlYXZpbHkgaW5zcGlyZWQgYnkgSlVuaXQsIFB5dGhvbidzIHVuaXR0ZXN0 Lm1vY2ssIGFuZAo+PiBHb29nbGV0ZXN0L0dvb2dsZW1vY2sgZm9yIEMrKy4gS1VuaXQgcHJvdmlk ZXMgZmFjaWxpdGllcyBmb3IgZGVmaW5pbmcKPj4gdW5pdCB0ZXN0IGNhc2VzLCBncm91cGluZyBy ZWxhdGVkIHRlc3QgY2FzZXMgaW50byB0ZXN0IHN1aXRlcywgcHJvdmlkaW5nCj4+IGNvbW1vbiBp bmZyYXN0cnVjdHVyZSBmb3IgcnVubmluZyB0ZXN0cywgbW9ja2luZywgc3B5aW5nLCBhbmQgbXVj aCBtb3JlLgo+Pgo+IAo+IEkgbG9va2VkIG9ubHkgYXQgdGhpcyBzZWN0aW9uLCBhcyB3YXMgc3Bl Y2lmaWNhbGx5IHJlcXVlc3RlZDoKPiAKPj4gIyMjIEJ1dCB3YWl0ISBEb2Vzbid0IGtzZWxmdGVz dCBzdXBwb3J0IGluIGtlcm5lbCB0ZXN0aW5nPyEKPj4KPj4gSW4gYSBwcmV2aW91cyB2ZXJzaW9u IG9mIHRoaXMgcGF0Y2hzZXQgRnJhbmsgcG9pbnRlZCBvdXQgdGhhdCBrc2VsZnRlc3QKPj4gYWxy ZWFkeSBzdXBwb3J0cyB3cml0aW5nIGEgdGVzdCB0aGF0IHJlc2lkZXMgaW4gdGhlIGtlcm5lbCB1 c2luZyB0aGUKPj4gdGVzdCBtb2R1bGUgZmVhdHVyZVsyXS4gTFdOIGRpZCBhIHJlYWxseSBncmVh dCBzdW1tYXJ5IG9uIHRoaXMKPj4gZGlzY3Vzc2lvbiBoZXJlWzNdLgo+Pgo+PiBLc2VsZnRlc3Qg aGFzIGEgZmVhdHVyZSB0aGF0IGFsbG93cyBhIHRlc3QgbW9kdWxlIHRvIGJlIGxvYWRlZCBpbnRv IGEKPj4ga2VybmVsIHVzaW5nIHRoZSBrc2VsZnRlc3QgZnJhbWV3b3JrOyB0aGlzIGRvZXMgYWxs b3cgc29tZW9uZSB0byB3cml0ZQo+PiB0ZXN0cyBhZ2FpbnN0IGtlcm5lbCBjb2RlIG5vdCBkaXJl Y3RseSBleHBvc2VkIHRvIHVzZXJsYW5kOyBob3dldmVyLCBpdAo+PiBkb2VzIG5vdCBwcm92aWRl IG11Y2ggb2YgYSBmcmFtZXdvcmsgYXJvdW5kIGhvdyB0byBzdHJ1Y3R1cmUgdGhlIHRlc3RzLgo+ PiBUaGUga3NlbGZ0ZXN0IHRlc3QgbW9kdWxlIGZlYXR1cmUganVzdCBwcm92aWRlcyBhIGhlYWRl ciB3aGljaCBoYXMgYQo+PiBzdGFuZGFyZGl6ZWQgd2F5IG9mIHJlcG9ydGluZyB0ZXN0IGZhaWx1 cmVzLAo+IAo+IAo+PiBhbmQgdGhlbiBwcm92aWRlcwo+PiBpbmZyYXN0cnVjdHVyZSB0byBsb2Fk IGFuZCBydW4gdGhlIHRlc3RzIHVzaW5nIHRoZSBrc2VsZnRlc3QgdGVzdAo+PiBoYXJuZXNzLgo+ IAo+IFRoZSBpbi1rZXJuZWwgdGVzdHMgY2FuIGFsc28gYmUgaW52b2tlZCBhdCBib290IHRpbWUg aWYgdGhleSBhcmUKPiBjb25maWd1cmVkIChLY29uZmlnKSBhcyBpbi1rZXJuZWwgaW5zdGVhZCBv ZiBhcyBtb2R1bGVzLiAgSSBkaWQgbm90Cj4gY2hlY2sgaG93IG1hbnkgb2YgdGhlIHRlc3RzIGhh dmUgdHJpLXN0YXRlIGNvbmZpZ3VyYXRpb24gdG8gYWxsb3cKPiB0aGlzLCBidXQgdGhlIGZldyB0 aGF0IEkgbG9va2VkIGF0IGRpZC4KPiAKPj4KPj4gVGhlIGtzZWxmdGVzdCB0ZXN0IG1vZHVsZSBk b2VzIG5vdCBzZWVtIHRvIGJlIG9waW5pb25hdGVkIGF0IGFsbCBpbgo+PiByZWdhcmRzIHRvIGhv dyB0ZXN0cyBhcmUgc3RydWN0dXJlZCwgaG93IHRoZXkgY2hlY2sgZm9yIGZhaWx1cmVzLCBob3cK Pj4gdGVzdHMgYXJlIG9yZ2FuaXplZC4gRXZlbiBpbiB0aGUgbWV0aG9kIGl0IHByb3ZpZGVzIGZv ciByZXBvcnRpbmcKPj4gZmFpbHVyZXMgaXMgcHJldHR5IHNpbXBsZTsgaXQgZG9lc24ndCBoYXZl IGFueSBtb3JlIGFkdmFuY2VkIGZhaWx1cmUKPj4gcmVwb3J0aW5nIG9yIGxvZ2dpbmcgZmVhdHVy ZXMuIEdpdmVuIHdoYXQncyB0aGVyZSwgSSB0aGluayBpdCBpcyBmYWlyIHRvCj4+IHNheSB0aGF0 IGl0IGlzIG5vdCBhY3R1YWxseSBhIGZyYW1ld29yaywgYnV0IGEgZmVhdHVyZSB0aGF0IG1ha2Vz IGl0Cj4+IHBvc3NpYmxlIGZvciBzb21lb25lIHRvIGRvIHNvbWUgY2hlY2tzIGluIGtlcm5lbCBz cGFjZS4KPiAKPiBJIHdvdWxkIGNhbGwgdGhhdCBkZXNjcmlwdGlvbiBhIGxpdHRsZSBkaXNtaXNz aXZlLiAgVGhlIHNldCBvZiBpbi1rZXJuZWwKPiB0ZXN0cyB0aGF0IEkgbG9va2VkIGxpa2UgZm9s bG93ZWQgYSBjb21tb24gcGF0dGVybiBhbmQgcmVwb3J0ZWQgcmVzdWx0cwo+IGluIGEgdW5pZm9y bSBtYW5uZXIuCj4gCj4+CgpJIHRoaW5rIEkgY29tbWVudGVkIG9uIHRoaXMgYmVmb3JlLiBJIGFn cmVlIHdpdGggdGhlIHN0YXRlbWVudCB0aGF0CnRoZXJlIGlzIG5vIG92ZXJsYXAgYmV0d2VlbiBL c2VsZnRlc3QgYW5kIEtVbml0LiBJIHdvdWxkIGxpa2Ugc2VlIHRoaXMKcmVtb3ZlZC4gS3NlbGZ0 ZXN0IG1vZHVsZSBzdXBwb3J0IHN1cHBvcnRzIHVzZS1jYXNlcyBLVW5pdCB3b24ndCBiZSBhYmxl CnRvLiBJIGNhbiBidWlsZCBhbiBrZXJuZWwgd2l0aCBLc2VsZnRlc3QgdGVzdCBtb2R1bGVzIGFu ZCB1c2UgaXQgaW4gdGhlCmZpbGVkIHRvIGxvYWQgYW5kIHJ1biB0ZXN0cyBpZiBJIG5lZWQgdG8g ZGVidWcgYSBwcm9ibGVtIGFuZCBnZXQgZGF0YQpmcm9tIGEgc3lzdGVtLiBJIGNhbid0IGRvIHRo YXQgd2l0aCBLVW5pdC4KCkluIG15IG1pbmQsIEkgYW0gbm90IHZpZXdpbmcgdGhpcyBhcyB3aGlj aCBpcyBiZXR0ZXIuIEtzZWxmdGVzdCBhbmQKS1VuaXQgYm90aCBoYXZlIHRoZWlyIHBsYWNlIGlu IHRoZSBrZXJuZWwgZGV2ZWxvcG1lbnQgcHJvY2Vzcy4gSXQgaXNuJ3QKcHJvZHVjdGl2ZSBhbmQv b3IgbmVjZXNzYXJ5IHRvIGNvbXBhcmluZyBLc2VsZnRlc3QgYW5kIEtVbml0IHdpdGhvdXQgYQpn b29kIHVuZGVyc3RhbmRpbmcgb2YgdGhlIHByb2JsZW0gc3BhY2VzIGZvciBlYWNoIG9mIHRoZXNl LgoKSSB3b3VsZCBzdHJvbmdseSByZWNvbW1lbmQgbm90IG1ha2luZyByZWZlcmVuY2UgdG8gS3Nl bGZ0ZXN0IGFuZCB0YWxrCmFib3V0IHdoYXQgS1VuaXQgb2ZmZXJzLgoKPj4gRnVydGhlcm1vcmUs IGtzZWxmdGVzdCB0ZXN0IG1vZHVsZSBoYXMgdmVyeSBmZXcgdXNlcnMuIEkgY2hlY2tlZCBmb3Ig YWxsCj4+IHRoZSB0ZXN0cyB0aGF0IHVzZSBpdCB1c2luZyB0aGUgZm9sbG93aW5nIGdyZXAgY29t bWFuZDoKPj4KPj4gZ3JlcCAtSHJuIC1lICdrc2VsZnRlc3RfbW9kdWxlXC5oJwo+Pgo+PiBhbmQg b25seSBnb3QgdGhyZWUgcmVzdWx0czogbGliL3Rlc3Rfc3Ryc2NweS5jLCBsaWIvdGVzdF9wcmlu dGYuYywgYW5kCj4+IGxpYi90ZXN0X2JpdG1hcC5jLgo+IAoKQWdhaW4sIHVubmVjZXNzYXJ5LiBL VW5pdCBjYW4ndCByZXBsYWNlIEtzZWxmdGVzdCBtb2R1bGUgaW4gdGhlIHdheQpLc2VsZnRlc3Qg bW9kdWxlIHN1cHBvcnQgY2FuIGJlIHVzZWQgZm9yIGRlYnVnZ2luZyBhbmQgZ2F0aGVyaW5nCmlu Zm9ybWF0aW9uIG9uIHN5c3RlbSB0aGF0IG1pZ2h0IGJlIGluIGFjdGl2ZSB1c2UgYW5kIG5vdCBk ZWRpY2F0ZWQKdG8gdGVzdCBhbmQgZGV2ZWxvcG1lbnQgYWxvbmUuIEkgd291bGRuJ3QgaGVzaXRh dGUgbG9hZGluZyBhIEtzZWxmdGVzdAp0ZXN0IG1vZHVsZSBvbiBteSBsYXB0b3AgYW5kIHJ1bm5p bmcgdGVzdHMsIGJ1dCBJIHdvdWxkbid0IHVzZSBLVW5pdAp0aGUgc2FtZSB3YXkuCgpBZ2Fpbiwg dGhpcyBpcyBub3QgYSBjb21wZXRpdGlvbiBiZXR3ZWVuIHdoaWNoIGlzIGJldHRlci4gS3NlbGZ0 ZXN0CmFuZCBLVW5pdCBzZXJ2ZSBkaWZmZXJlbnQgbmVlZHMgYW5kIHByb2JsZW0gc3BhY2VzLgoK UGxlYXNlIHJlZG8gdGhpcyBkb2N1bWVudGF0aW9uIHRvIHJlZmxlY3QgdGhhdC4KCnRoYW5rcywK LS0gU2h1YWgKCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f XwpkcmktZGV2ZWwgbWFpbGluZyBsaXN0CmRyaS1kZXZlbEBsaXN0cy5mcmVlZGVza3RvcC5vcmcK aHR0cHM6Ly9saXN0cy5mcmVlZGVza3RvcC5vcmcvbWFpbG1hbi9saXN0aW5mby9kcmktZGV2ZWw= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.92 #3 (Red Hat Linux)) id 1heL16-0002UO-Ag for linux-um@lists.infradead.org; Fri, 21 Jun 2019 14:59:53 +0000 Subject: Re: [PATCH v5 00/18] kunit: introduce KUnit, the Linux kernel unit testing framework References: <20190617082613.109131-1-brendanhiggins@google.com> <10feac3e-7621-65e5-fbf0-9c63fcbe09c9@gmail.com> From: shuah Message-ID: <69809117-dcda-160a-ee0a-d1d3b4c5cd8a@kernel.org> Date: Fri, 21 Jun 2019 08:59:48 -0600 MIME-Version: 1.0 In-Reply-To: <10feac3e-7621-65e5-fbf0-9c63fcbe09c9@gmail.com> Content-Language: en-US List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-um" Errors-To: linux-um-bounces+geert=linux-m68k.org@lists.infradead.org To: Frank Rowand , Brendan Higgins , gregkh@linuxfoundation.org, jpoimboe@redhat.com, keescook@google.com, kieran.bingham@ideasonboard.com, mcgrof@kernel.org, peterz@infradead.org, robh@kernel.org, sboyd@kernel.org, tytso@mit.edu, yamada.masahiro@socionext.com Cc: pmladek@suse.com, linux-doc@vger.kernel.org, amir73il@gmail.com, dri-devel@lists.freedesktop.org, Alexander.Levin@microsoft.com, linux-kselftest@vger.kernel.org, shuah , linux-nvdimm@lists.01.org, khilman@baylibre.com, knut.omang@oracle.com, wfg@linux.intel.com, joel@jms.id.au, rientjes@google.com, jdike@addtoit.com, dan.carpenter@oracle.com, devicetree@vger.kernel.org, linux-kbuild@vger.kernel.org, Tim.Bird@sony.com, linux-um@lists.infradead.org, rostedt@goodmis.org, julia.lawall@lip6.fr, kunit-dev@googlegroups.com, richard@nod.at, rdunlap@infradead.org, linux-kernel@vger.kernel.org, daniel@ffwll.ch, mpe@ellerman.id.au, linux-fsdevel@vger.kernel.org, logang@deltatee.com Hi Brendan, On 6/19/19 7:17 PM, Frank Rowand wrote: > Hi Brendan, > > I am only responding to this because you asked me to in the v4 thread. > > Thank you for evaluating my comments in the v4 thread and asking me to > comment on v5 > > On 6/17/19 1:25 AM, Brendan Higgins wrote: >> ## TL;DR >> >> A not so quick follow-up to Stephen's suggestions on PATCH v4. Nothing >> that really changes any functionality or usage with the minor exception >> of a couple public functions that Stephen asked me to rename. >> Nevertheless, a good deal of clean up and fixes. See changes below. >> >> As for our current status, right now we got Reviewed-bys on all patches >> except: >> >> - [PATCH v5 08/18] objtool: add kunit_try_catch_throw to the noreturn >> list >> >> However, it would probably be good to get reviews/acks from the >> subsystem maintainers on: >> >> - [PATCH v5 06/18] kbuild: enable building KUnit >> - [PATCH v5 08/18] objtool: add kunit_try_catch_throw to the noreturn >> list >> - [PATCH v5 15/18] Documentation: kunit: add documentation for KUnit >> - [PATCH v5 17/18] kernel/sysctl-test: Add null pointer test for >> sysctl.c:proc_dointvec() >> - [PATCH v5 18/18] MAINTAINERS: add proc sysctl KUnit test to PROC >> SYSCTL section >> >> Other than that, I think we should be good to go. >> >> One last thing, I updated the background to include my thoughts on KUnit >> vs. in kernel testing with kselftest in the background sections as >> suggested by Frank in the discussion on PATCH v2. >> >> ## Background >> >> This patch set proposes KUnit, a lightweight unit testing and mocking >> framework for the Linux kernel. >> >> Unlike Autotest and kselftest, KUnit is a true unit testing framework; >> it does not require installing the kernel on a test machine or in a VM >> (however, KUnit still allows you to run tests on test machines or in VMs >> if you want[1]) and does not require tests to be written in userspace >> running on a host kernel. Additionally, KUnit is fast: From invocation >> to completion KUnit can run several dozen tests in under a second. >> Currently, the entire KUnit test suite for KUnit runs in under a second >> from the initial invocation (build time excluded). >> >> KUnit is heavily inspired by JUnit, Python's unittest.mock, and >> Googletest/Googlemock for C++. KUnit provides facilities for defining >> unit test cases, grouping related test cases into test suites, providing >> common infrastructure for running tests, mocking, spying, and much more. >> > > I looked only at this section, as was specifically requested: > >> ### But wait! Doesn't kselftest support in kernel testing?! >> >> In a previous version of this patchset Frank pointed out that kselftest >> already supports writing a test that resides in the kernel using the >> test module feature[2]. LWN did a really great summary on this >> discussion here[3]. >> >> Kselftest has a feature that allows a test module to be loaded into a >> kernel using the kselftest framework; this does allow someone to write >> tests against kernel code not directly exposed to userland; however, it >> does not provide much of a framework around how to structure the tests. >> The kselftest test module feature just provides a header which has a >> standardized way of reporting test failures, > > >> and then provides >> infrastructure to load and run the tests using the kselftest test >> harness. > > The in-kernel tests can also be invoked at boot time if they are > configured (Kconfig) as in-kernel instead of as modules. I did not > check how many of the tests have tri-state configuration to allow > this, but the few that I looked at did. > >> >> The kselftest test module does not seem to be opinionated at all in >> regards to how tests are structured, how they check for failures, how >> tests are organized. Even in the method it provides for reporting >> failures is pretty simple; it doesn't have any more advanced failure >> reporting or logging features. Given what's there, I think it is fair to >> say that it is not actually a framework, but a feature that makes it >> possible for someone to do some checks in kernel space. > > I would call that description a little dismissive. The set of in-kernel > tests that I looked like followed a common pattern and reported results > in a uniform manner. > >> I think I commented on this before. I agree with the statement that there is no overlap between Kselftest and KUnit. I would like see this removed. Kselftest module support supports use-cases KUnit won't be able to. I can build an kernel with Kselftest test modules and use it in the filed to load and run tests if I need to debug a problem and get data from a system. I can't do that with KUnit. In my mind, I am not viewing this as which is better. Kselftest and KUnit both have their place in the kernel development process. It isn't productive and/or necessary to comparing Kselftest and KUnit without a good understanding of the problem spaces for each of these. I would strongly recommend not making reference to Kselftest and talk about what KUnit offers. >> Furthermore, kselftest test module has very few users. I checked for all >> the tests that use it using the following grep command: >> >> grep -Hrn -e 'kselftest_module\.h' >> >> and only got three results: lib/test_strscpy.c, lib/test_printf.c, and >> lib/test_bitmap.c. > Again, unnecessary. KUnit can't replace Kselftest module in the way Kselftest module support can be used for debugging and gathering information on system that might be in active use and not dedicated to test and development alone. I wouldn't hesitate loading a Kselftest test module on my laptop and running tests, but I wouldn't use KUnit the same way. Again, this is not a competition between which is better. Kselftest and KUnit serve different needs and problem spaces. Please redo this documentation to reflect that. thanks, -- Shuah _______________________________________________ linux-um mailing list linux-um@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-um