From mboxrd@z Thu Jan 1 00:00:00 1970 From: Brendan Higgins Subject: Re: [PATCH v5 00/18] kunit: introduce KUnit, the Linux kernel unit testing framework Date: Fri, 21 Jun 2019 17:54:31 -0700 Message-ID: References: <20190617082613.109131-1-brendanhiggins@google.com> <10feac3e-7621-65e5-fbf0-9c63fcbe09c9@gmail.com> <69809117-dcda-160a-ee0a-d1d3b4c5cd8a@kernel.org> <20190621181342.GA17166@mit.edu> <6f3f5184-d14e-1b46-17f1-391ee67e699c@kernel.org> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <6f3f5184-d14e-1b46-17f1-391ee67e699c-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linux-nvdimm-bounces-hn68Rpc1hR1g9hUCZPvPmw@public.gmane.org Sender: "Linux-nvdimm" To: shuah Cc: Petr Mladek , "open list:DOCUMENTATION" , Peter Zijlstra , Amir Goldstein , dri-devel , Sasha Levin , Masahiro Yamada , Michael Ellerman , "open list:KERNEL SELFTEST FRAMEWORK" , Frank Rowand , Rob Herring , linux-nvdimm , Kevin Hilman , Knut Omang , Kieran Bingham , wfg-VuQAYsv1563Yd54FQh9/CA@public.gmane.org, Joel Stanley , David Rientjes , Jeff Dike , Dan Carpenter , devicetree , linux-kbuild List-Id: linux-nvdimm@lists.01.org On Fri, Jun 21, 2019 at 12:21 PM shuah wrote: > > On 6/21/19 12:13 PM, Theodore Ts'o wrote: > > On Fri, Jun 21, 2019 at 08:59:48AM -0600, shuah wrote: > >>>> ### But wait! Doesn't kselftest support in kernel testing?! > >>>> > >>>> .... > >> > >> I think I commented on this before. I agree with the statement that > >> there is no overlap between Kselftest and KUnit. I would like see this > >> removed. Kselftest module support supports use-cases KUnit won't be able > >> to. I can build an kernel with Kselftest test modules and use it in the > >> filed to load and run tests if I need to debug a problem and get data > >> from a system. I can't do that with KUnit. Sure, I think this point has been brought up a number of times before. Maybe I didn't write this section well because, like Frank said, it comes across as being critical of the Kselftest module support; that wasn't my intention. I was speaking from the perspective that Kselftest module support is just a feature of Kselftest, and not a full framework like KUnit is (obviously Kselftest itself *is* a framework, but a very small part of it is not). It was obvious to me what Kselftest module support was intended for, and it is not intended to cover the use case that KUnit is targeting. > >> In my mind, I am not viewing this as which is better. Kselftest and > >> KUnit both have their place in the kernel development process. It isn't > >> productive and/or necessary to comparing Kselftest and KUnit without a > >> good understanding of the problem spaces for each of these. Again, I didn't mean to draw a comparison of which is better than the other. I was just trying to point out that Kselftest module support doesn't make sense as a stand alone unit testing framework, or really a framework of any kind, despite how it might actually be used. > >> I would strongly recommend not making reference to Kselftest and talk > >> about what KUnit offers. I can see your point. It seems that both you and Frank seem to think that I drew a comparison between Kselftest and KUnit, which was unintended. I probably should have spent more time editing this section, but I can see the point of drawing the comparison itself might invite this confusion. > > Shuah, > > > > Just to recall the history, this section of the FAQ was added to rebut > > the ***very*** strong statements that Frank made that there was > > overlap between Kselftest and Kunit, and that having too many ways for > > kernel developers to do the identical thing was harmful (he said it > > was too much of a burden on a kernel developer) --- and this was an > > argument for not including Kunit in the upstream kernel. I don't think he was actually advocating that we don't include KUnit, maybe playing devil's advocate; nevertheless, at the end, Frank seemed to agree that there were valuable things that KUnit offered. I thought he just wanted to make the point that I hadn't made the distinction sufficiently clear in the cover letter, and other reviewers might get confused in the future as well. Additionally, it does look like people were trying to use Kselftest module support to cover some things which really were trying to be unit tests. I know this isn't really intended - everything looks like a nail when you only have a hammer, which I think Frank was pointing out furthers the above confusion. In anycase, it sounds like I have, if anything, only made the discussion even more confusing by adding this section; sorry about that. > > If we're past that objection, then perhaps this section can be > > dropped, but there's a very good reason why it was there. I wouldn't > > Brendan to be accused of ignoring feedback from those who reviewed his > > patches. :-) > > > > Agreed. I understand that this FAQ probably was needed at one time and > Brendan added it to address the concerns. I don't want to speak for Frank, but I don't think it was an objection to KUnit itself, but rather an objection to not sufficiently addressing the point about how they differ. > I think at some point we do need to have a document that outlines when > to KUnit and when to use Kselftest modules. I think one concern people > have is that if KUnit is perceived as a replacement for Ksefltest > module, Kselftest module will be ignored leaving users without the > ability to build and run with Kselftest modules and load them on a need > basis to gather data on a systems that aren't dedicated strictly for > testing. I absolutely agree! I posed a suggestion here[1], which after I just now searched for a link, I realize for some reason it didn't seem like it reached a number of the mailing lists that I sent it to, so I should probably resend it. Anyway, a summary of what I suggested: We should start off by better organizing Documentation/dev-tools/ and create a landing page that groups the dev-tools by function according to what person is likely to use them and for what. Eventually and specifically for Kselftest and KUnit, I would like to have a testing guide for the kernel that explains what testing procedure should look like and what to use and when. > I am trying to move the conversation forward from KUnit vs. Kselftest > modules discussion to which problem areas each one addresses keeping > in mind that it is not about which is better. Kselftest and KUnit both > have their place in the kernel development process. We just have to be > clear on usage as we write tests for each. I think that is the right long term approach. I think a good place to start, like I suggested above, is cleaning up Documentation/dev-tools/, but I think that belongs in a (probably several) follow-up patchset. Frank, I believe your objection was mostly related to how KUnit is presented specifically in the cover letter, and doesn't necessarily deal with the intended use case. So I don't think that doing this, especially doing this later, really addresses your concern. I don't want to belabor the issue, but I would also rather not put words in your mouth, what are your thoughts on the above? I think my main concern moving forward on this point is that I am not sure that I can address the debate that this section covers in a way that is both sufficiently concise for a cover letter, but also doesn't invite more potential confusion. My inclination at this point is to drop it since I think the set of reviewers for this patchset has at this point become fixed, and it seems that it will likely cause more confusion rather than reduce it; also, I don't really think this will be an issue for end users, especially once we have proper documentation in place. Alternatively, I guess I could maybe address the point elsewhere and refer to it in the cover letter? Or maybe just put it at the end of the cover letter? [1] https://www.mail-archive.com/kgdb-bugreport-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org/msg05059.html From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.4 required=3.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13D12C43613 for ; Sat, 22 Jun 2019 01:00:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C72952075E for ; Sat, 22 Jun 2019 01:00:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="r0d0D49h" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726395AbfFVBAz (ORCPT ); Fri, 21 Jun 2019 21:00:55 -0400 Received: from mail-pf1-f193.google.com ([209.85.210.193]:40855 "EHLO mail-pf1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726052AbfFVBAx (ORCPT ); Fri, 21 Jun 2019 21:00:53 -0400 Received: by mail-pf1-f193.google.com with SMTP id p184so4404123pfp.7 for ; Fri, 21 Jun 2019 18:00:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=pn4/IgcE/a3A9HRsf8YysVQmDOFkLMRirlv4eJOCAEY=; b=r0d0D49hNJck2jmBOZ7hKq1EyIQm5VIsrCLXNfWA/zqw7JUB/PFT9/TEg3f9m8Exka 4OB4CLnAA+t7ZyRfMqNWActJqwB4eElCzw9OwOGul12RrF6Xah0/adQB3rDPYCiKwGSb 1cQfwLiJupuJHVE6kDeqU7eCLKZrrozRyUdaq1BAJ5w/IyIufGa6roBqLtZxSQ3wQLl8 cBD+npmSvsH7CB96WJOcXSzHxVdpszLog3BtkM+9yhp75ZfrhE2bcktgSWTAXvuCSrTY kUvb5BfS0a52N4OmjgywAeDaICRhfTcMxYZ0cxZGqUP9umwcIynczzP5GVJDJnjw5aRm 2oGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=pn4/IgcE/a3A9HRsf8YysVQmDOFkLMRirlv4eJOCAEY=; b=QMGMQ3gG7PY+yUCNVTJE0X5gf5KHyPXuyyk8F9KYhdzOo3fmf3BI3Idu+f6cOxymOb 08pYrOKZogt6bZV6aPALK6KvfUE8FoX1U62ZuASQcGDpf68QZiay8k5R1KX5G+Q3J9G3 FULbEhJ/XYe5jgFjoYZpOYx1ZX9XKQXvWRtfejueQAqCSg491R3Gt6dPP2sprR5peuXm cv1YIwBn1KI0/i5dZaWc+Utr9rEllHHCxg3sLq+xot6K2ZiCdkRt4D4NNbf6kTsj56wE /ZT4EYV1oj0zfVRBfVxdxPxNf8cOF60zhUzQZBpiBImhTO8r1GcD/yr+p7vTyiqZHaTw M3gw== X-Gm-Message-State: APjAAAX2aEQSsQD7kUs1k8q0ausXDJp60TcYvhCV9PQ4UINqQEFDrGcn XzJG+V+n/eDYs5tuHgCoomuIVYsjyJmMJP8/3Eco+w== X-Google-Smtp-Source: APXvYqwOxize9xmrZ2ApT7No74w3onQmjRsjgozaiDJFMKpG33Vqht+PxeXD1e0v4vbFE+q+E/bORwyHwGlVO6g99RY= X-Received: by 2002:a17:90a:9382:: with SMTP id q2mr10190000pjo.131.1561164883594; Fri, 21 Jun 2019 17:54:43 -0700 (PDT) MIME-Version: 1.0 References: <20190617082613.109131-1-brendanhiggins@google.com> <10feac3e-7621-65e5-fbf0-9c63fcbe09c9@gmail.com> <69809117-dcda-160a-ee0a-d1d3b4c5cd8a@kernel.org> <20190621181342.GA17166@mit.edu> <6f3f5184-d14e-1b46-17f1-391ee67e699c@kernel.org> In-Reply-To: <6f3f5184-d14e-1b46-17f1-391ee67e699c@kernel.org> From: Brendan Higgins Date: Fri, 21 Jun 2019 17:54:31 -0700 Message-ID: Subject: Re: [PATCH v5 00/18] kunit: introduce KUnit, the Linux kernel unit testing framework To: shuah Cc: "Theodore Ts'o" , Frank Rowand , Greg KH , Josh Poimboeuf , Kees Cook , Kieran Bingham , Luis Chamberlain , Peter Zijlstra , Rob Herring , Stephen Boyd , Masahiro Yamada , devicetree , dri-devel , kunit-dev@googlegroups.com, "open list:DOCUMENTATION" , linux-fsdevel@vger.kernel.org, linux-kbuild , Linux Kernel Mailing List , "open list:KERNEL SELFTEST FRAMEWORK" , linux-nvdimm , linux-um@lists.infradead.org, Sasha Levin , "Bird, Timothy" , Amir Goldstein , Dan Carpenter , Daniel Vetter , Jeff Dike , Joel Stanley , Julia Lawall , Kevin Hilman , Knut Omang , Logan Gunthorpe , Michael Ellerman , Petr Mladek , Randy Dunlap , Richard Weinberger , David Rientjes , Steven Rostedt , wfg@linux.intel.com Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Jun 21, 2019 at 12:21 PM shuah wrote: > > On 6/21/19 12:13 PM, Theodore Ts'o wrote: > > On Fri, Jun 21, 2019 at 08:59:48AM -0600, shuah wrote: > >>>> ### But wait! Doesn't kselftest support in kernel testing?! > >>>> > >>>> .... > >> > >> I think I commented on this before. I agree with the statement that > >> there is no overlap between Kselftest and KUnit. I would like see this > >> removed. Kselftest module support supports use-cases KUnit won't be able > >> to. I can build an kernel with Kselftest test modules and use it in the > >> filed to load and run tests if I need to debug a problem and get data > >> from a system. I can't do that with KUnit. Sure, I think this point has been brought up a number of times before. Maybe I didn't write this section well because, like Frank said, it comes across as being critical of the Kselftest module support; that wasn't my intention. I was speaking from the perspective that Kselftest module support is just a feature of Kselftest, and not a full framework like KUnit is (obviously Kselftest itself *is* a framework, but a very small part of it is not). It was obvious to me what Kselftest module support was intended for, and it is not intended to cover the use case that KUnit is targeting. > >> In my mind, I am not viewing this as which is better. Kselftest and > >> KUnit both have their place in the kernel development process. It isn't > >> productive and/or necessary to comparing Kselftest and KUnit without a > >> good understanding of the problem spaces for each of these. Again, I didn't mean to draw a comparison of which is better than the other. I was just trying to point out that Kselftest module support doesn't make sense as a stand alone unit testing framework, or really a framework of any kind, despite how it might actually be used. > >> I would strongly recommend not making reference to Kselftest and talk > >> about what KUnit offers. I can see your point. It seems that both you and Frank seem to think that I drew a comparison between Kselftest and KUnit, which was unintended. I probably should have spent more time editing this section, but I can see the point of drawing the comparison itself might invite this confusion. > > Shuah, > > > > Just to recall the history, this section of the FAQ was added to rebut > > the ***very*** strong statements that Frank made that there was > > overlap between Kselftest and Kunit, and that having too many ways for > > kernel developers to do the identical thing was harmful (he said it > > was too much of a burden on a kernel developer) --- and this was an > > argument for not including Kunit in the upstream kernel. I don't think he was actually advocating that we don't include KUnit, maybe playing devil's advocate; nevertheless, at the end, Frank seemed to agree that there were valuable things that KUnit offered. I thought he just wanted to make the point that I hadn't made the distinction sufficiently clear in the cover letter, and other reviewers might get confused in the future as well. Additionally, it does look like people were trying to use Kselftest module support to cover some things which really were trying to be unit tests. I know this isn't really intended - everything looks like a nail when you only have a hammer, which I think Frank was pointing out furthers the above confusion. In anycase, it sounds like I have, if anything, only made the discussion even more confusing by adding this section; sorry about that. > > If we're past that objection, then perhaps this section can be > > dropped, but there's a very good reason why it was there. I wouldn't > > Brendan to be accused of ignoring feedback from those who reviewed his > > patches. :-) > > > > Agreed. I understand that this FAQ probably was needed at one time and > Brendan added it to address the concerns. I don't want to speak for Frank, but I don't think it was an objection to KUnit itself, but rather an objection to not sufficiently addressing the point about how they differ. > I think at some point we do need to have a document that outlines when > to KUnit and when to use Kselftest modules. I think one concern people > have is that if KUnit is perceived as a replacement for Ksefltest > module, Kselftest module will be ignored leaving users without the > ability to build and run with Kselftest modules and load them on a need > basis to gather data on a systems that aren't dedicated strictly for > testing. I absolutely agree! I posed a suggestion here[1], which after I just now searched for a link, I realize for some reason it didn't seem like it reached a number of the mailing lists that I sent it to, so I should probably resend it. Anyway, a summary of what I suggested: We should start off by better organizing Documentation/dev-tools/ and create a landing page that groups the dev-tools by function according to what person is likely to use them and for what. Eventually and specifically for Kselftest and KUnit, I would like to have a testing guide for the kernel that explains what testing procedure should look like and what to use and when. > I am trying to move the conversation forward from KUnit vs. Kselftest > modules discussion to which problem areas each one addresses keeping > in mind that it is not about which is better. Kselftest and KUnit both > have their place in the kernel development process. We just have to be > clear on usage as we write tests for each. I think that is the right long term approach. I think a good place to start, like I suggested above, is cleaning up Documentation/dev-tools/, but I think that belongs in a (probably several) follow-up patchset. Frank, I believe your objection was mostly related to how KUnit is presented specifically in the cover letter, and doesn't necessarily deal with the intended use case. So I don't think that doing this, especially doing this later, really addresses your concern. I don't want to belabor the issue, but I would also rather not put words in your mouth, what are your thoughts on the above? I think my main concern moving forward on this point is that I am not sure that I can address the debate that this section covers in a way that is both sufficiently concise for a cover letter, but also doesn't invite more potential confusion. My inclination at this point is to drop it since I think the set of reviewers for this patchset has at this point become fixed, and it seems that it will likely cause more confusion rather than reduce it; also, I don't really think this will be an issue for end users, especially once we have proper documentation in place. Alternatively, I guess I could maybe address the point elsewhere and refer to it in the cover letter? Or maybe just put it at the end of the cover letter? [1] https://www.mail-archive.com/kgdb-bugreport@lists.sourceforge.net/msg05059.html From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf1-f195.google.com ([209.85.210.195]:38630 "EHLO mail-pf1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726052AbfFVBC5 (ORCPT ); Fri, 21 Jun 2019 21:02:57 -0400 Received: by mail-pf1-f195.google.com with SMTP id a186so4409384pfa.5 for ; Fri, 21 Jun 2019 18:02:56 -0700 (PDT) MIME-Version: 1.0 References: <20190617082613.109131-1-brendanhiggins@google.com> <10feac3e-7621-65e5-fbf0-9c63fcbe09c9@gmail.com> <69809117-dcda-160a-ee0a-d1d3b4c5cd8a@kernel.org> <20190621181342.GA17166@mit.edu> <6f3f5184-d14e-1b46-17f1-391ee67e699c@kernel.org> In-Reply-To: <6f3f5184-d14e-1b46-17f1-391ee67e699c@kernel.org> From: Brendan Higgins Date: Fri, 21 Jun 2019 17:54:31 -0700 Message-ID: Subject: Re: [PATCH v5 00/18] kunit: introduce KUnit, the Linux kernel unit testing framework Content-Type: text/plain; charset="UTF-8" Sender: linux-kbuild-owner@vger.kernel.org List-ID: To: shuah Cc: Theodore Ts'o , Frank Rowand , Greg KH , Josh Poimboeuf , Kees Cook , Kieran Bingham , Luis Chamberlain , Peter Zijlstra , Rob Herring , Stephen Boyd , Masahiro Yamada , devicetree , dri-devel , kunit-dev@googlegroups.com, "open list:DOCUMENTATION" , linux-fsdevel@vger.kernel.org, linux-kbuild , Linux Kernel Mailing List , "open list:KERNEL SELFTEST FRAMEWORK" , linux-nvdimm , linux-um@lists.infradead.org, Sasha Levin , "Bird, Timothy" , Amir Goldstein , Dan Carpenter , Daniel Vetter , Jeff Dike , Joel Stanley , Julia Lawall , Kevin Hilman , Knut Omang , Logan Gunthorpe , Michael Ellerman , Petr Mladek , Randy Dunlap , Richard Weinberger , David Rientjes , Steven Rostedt , wfg@linux.intel.com On Fri, Jun 21, 2019 at 12:21 PM shuah wrote: > > On 6/21/19 12:13 PM, Theodore Ts'o wrote: > > On Fri, Jun 21, 2019 at 08:59:48AM -0600, shuah wrote: > >>>> ### But wait! Doesn't kselftest support in kernel testing?! > >>>> > >>>> .... > >> > >> I think I commented on this before. I agree with the statement that > >> there is no overlap between Kselftest and KUnit. I would like see this > >> removed. Kselftest module support supports use-cases KUnit won't be able > >> to. I can build an kernel with Kselftest test modules and use it in the > >> filed to load and run tests if I need to debug a problem and get data > >> from a system. I can't do that with KUnit. Sure, I think this point has been brought up a number of times before. Maybe I didn't write this section well because, like Frank said, it comes across as being critical of the Kselftest module support; that wasn't my intention. I was speaking from the perspective that Kselftest module support is just a feature of Kselftest, and not a full framework like KUnit is (obviously Kselftest itself *is* a framework, but a very small part of it is not). It was obvious to me what Kselftest module support was intended for, and it is not intended to cover the use case that KUnit is targeting. > >> In my mind, I am not viewing this as which is better. Kselftest and > >> KUnit both have their place in the kernel development process. It isn't > >> productive and/or necessary to comparing Kselftest and KUnit without a > >> good understanding of the problem spaces for each of these. Again, I didn't mean to draw a comparison of which is better than the other. I was just trying to point out that Kselftest module support doesn't make sense as a stand alone unit testing framework, or really a framework of any kind, despite how it might actually be used. > >> I would strongly recommend not making reference to Kselftest and talk > >> about what KUnit offers. I can see your point. It seems that both you and Frank seem to think that I drew a comparison between Kselftest and KUnit, which was unintended. I probably should have spent more time editing this section, but I can see the point of drawing the comparison itself might invite this confusion. > > Shuah, > > > > Just to recall the history, this section of the FAQ was added to rebut > > the ***very*** strong statements that Frank made that there was > > overlap between Kselftest and Kunit, and that having too many ways for > > kernel developers to do the identical thing was harmful (he said it > > was too much of a burden on a kernel developer) --- and this was an > > argument for not including Kunit in the upstream kernel. I don't think he was actually advocating that we don't include KUnit, maybe playing devil's advocate; nevertheless, at the end, Frank seemed to agree that there were valuable things that KUnit offered. I thought he just wanted to make the point that I hadn't made the distinction sufficiently clear in the cover letter, and other reviewers might get confused in the future as well. Additionally, it does look like people were trying to use Kselftest module support to cover some things which really were trying to be unit tests. I know this isn't really intended - everything looks like a nail when you only have a hammer, which I think Frank was pointing out furthers the above confusion. In anycase, it sounds like I have, if anything, only made the discussion even more confusing by adding this section; sorry about that. > > If we're past that objection, then perhaps this section can be > > dropped, but there's a very good reason why it was there. I wouldn't > > Brendan to be accused of ignoring feedback from those who reviewed his > > patches. :-) > > > > Agreed. I understand that this FAQ probably was needed at one time and > Brendan added it to address the concerns. I don't want to speak for Frank, but I don't think it was an objection to KUnit itself, but rather an objection to not sufficiently addressing the point about how they differ. > I think at some point we do need to have a document that outlines when > to KUnit and when to use Kselftest modules. I think one concern people > have is that if KUnit is perceived as a replacement for Ksefltest > module, Kselftest module will be ignored leaving users without the > ability to build and run with Kselftest modules and load them on a need > basis to gather data on a systems that aren't dedicated strictly for > testing. I absolutely agree! I posed a suggestion here[1], which after I just now searched for a link, I realize for some reason it didn't seem like it reached a number of the mailing lists that I sent it to, so I should probably resend it. Anyway, a summary of what I suggested: We should start off by better organizing Documentation/dev-tools/ and create a landing page that groups the dev-tools by function according to what person is likely to use them and for what. Eventually and specifically for Kselftest and KUnit, I would like to have a testing guide for the kernel that explains what testing procedure should look like and what to use and when. > I am trying to move the conversation forward from KUnit vs. Kselftest > modules discussion to which problem areas each one addresses keeping > in mind that it is not about which is better. Kselftest and KUnit both > have their place in the kernel development process. We just have to be > clear on usage as we write tests for each. I think that is the right long term approach. I think a good place to start, like I suggested above, is cleaning up Documentation/dev-tools/, but I think that belongs in a (probably several) follow-up patchset. Frank, I believe your objection was mostly related to how KUnit is presented specifically in the cover letter, and doesn't necessarily deal with the intended use case. So I don't think that doing this, especially doing this later, really addresses your concern. I don't want to belabor the issue, but I would also rather not put words in your mouth, what are your thoughts on the above? I think my main concern moving forward on this point is that I am not sure that I can address the debate that this section covers in a way that is both sufficiently concise for a cover letter, but also doesn't invite more potential confusion. My inclination at this point is to drop it since I think the set of reviewers for this patchset has at this point become fixed, and it seems that it will likely cause more confusion rather than reduce it; also, I don't really think this will be an issue for end users, especially once we have proper documentation in place. Alternatively, I guess I could maybe address the point elsewhere and refer to it in the cover letter? Or maybe just put it at the end of the cover letter? [1] https://www.mail-archive.com/kgdb-bugreport@lists.sourceforge.net/msg05059.html From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pg1-x543.google.com ([2607:f8b0:4864:20::543]) by bombadil.infradead.org with esmtps (Exim 4.92 #3 (Red Hat Linux)) id 1heUIn-0002NV-5q for linux-um@lists.infradead.org; Sat, 22 Jun 2019 00:54:47 +0000 Received: by mail-pg1-x543.google.com with SMTP id p10so4152892pgn.1 for ; Fri, 21 Jun 2019 17:54:44 -0700 (PDT) MIME-Version: 1.0 References: <20190617082613.109131-1-brendanhiggins@google.com> <10feac3e-7621-65e5-fbf0-9c63fcbe09c9@gmail.com> <69809117-dcda-160a-ee0a-d1d3b4c5cd8a@kernel.org> <20190621181342.GA17166@mit.edu> <6f3f5184-d14e-1b46-17f1-391ee67e699c@kernel.org> In-Reply-To: <6f3f5184-d14e-1b46-17f1-391ee67e699c@kernel.org> From: Brendan Higgins Date: Fri, 21 Jun 2019 17:54:31 -0700 Message-ID: Subject: Re: [PATCH v5 00/18] kunit: introduce KUnit, the Linux kernel unit testing framework List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-um" Errors-To: linux-um-bounces+geert=linux-m68k.org@lists.infradead.org To: shuah Cc: Petr Mladek , "open list:DOCUMENTATION" , Peter Zijlstra , Amir Goldstein , dri-devel , Sasha Levin , Masahiro Yamada , Michael Ellerman , "open list:KERNEL SELFTEST FRAMEWORK" , Frank Rowand , Rob Herring , linux-nvdimm , Kevin Hilman , Knut Omang , Kieran Bingham , wfg@linux.intel.com, Joel Stanley , David Rientjes , Jeff Dike , Dan Carpenter , devicetree , linux-kbuild , "Bird, Timothy , linux-um@lists.infradead.org, Steven Rostedt" , Julia Lawall , Josh Poimboeuf , kunit-dev@googlegroups.com, Theodore Ts'o , Richard Weinberger , Stephen Boyd , Greg KH , Randy Dunlap , Linux Kernel Mailing List , Luis Chamberlain , Daniel Vetter , Kees Cook , linux-fsdevel@vger.kernel.org, Logan Gunthorpe On Fri, Jun 21, 2019 at 12:21 PM shuah wrote: > > On 6/21/19 12:13 PM, Theodore Ts'o wrote: > > On Fri, Jun 21, 2019 at 08:59:48AM -0600, shuah wrote: > >>>> ### But wait! Doesn't kselftest support in kernel testing?! > >>>> > >>>> .... > >> > >> I think I commented on this before. I agree with the statement that > >> there is no overlap between Kselftest and KUnit. I would like see this > >> removed. Kselftest module support supports use-cases KUnit won't be able > >> to. I can build an kernel with Kselftest test modules and use it in the > >> filed to load and run tests if I need to debug a problem and get data > >> from a system. I can't do that with KUnit. Sure, I think this point has been brought up a number of times before. Maybe I didn't write this section well because, like Frank said, it comes across as being critical of the Kselftest module support; that wasn't my intention. I was speaking from the perspective that Kselftest module support is just a feature of Kselftest, and not a full framework like KUnit is (obviously Kselftest itself *is* a framework, but a very small part of it is not). It was obvious to me what Kselftest module support was intended for, and it is not intended to cover the use case that KUnit is targeting. > >> In my mind, I am not viewing this as which is better. Kselftest and > >> KUnit both have their place in the kernel development process. It isn't > >> productive and/or necessary to comparing Kselftest and KUnit without a > >> good understanding of the problem spaces for each of these. Again, I didn't mean to draw a comparison of which is better than the other. I was just trying to point out that Kselftest module support doesn't make sense as a stand alone unit testing framework, or really a framework of any kind, despite how it might actually be used. > >> I would strongly recommend not making reference to Kselftest and talk > >> about what KUnit offers. I can see your point. It seems that both you and Frank seem to think that I drew a comparison between Kselftest and KUnit, which was unintended. I probably should have spent more time editing this section, but I can see the point of drawing the comparison itself might invite this confusion. > > Shuah, > > > > Just to recall the history, this section of the FAQ was added to rebut > > the ***very*** strong statements that Frank made that there was > > overlap between Kselftest and Kunit, and that having too many ways for > > kernel developers to do the identical thing was harmful (he said it > > was too much of a burden on a kernel developer) --- and this was an > > argument for not including Kunit in the upstream kernel. I don't think he was actually advocating that we don't include KUnit, maybe playing devil's advocate; nevertheless, at the end, Frank seemed to agree that there were valuable things that KUnit offered. I thought he just wanted to make the point that I hadn't made the distinction sufficiently clear in the cover letter, and other reviewers might get confused in the future as well. Additionally, it does look like people were trying to use Kselftest module support to cover some things which really were trying to be unit tests. I know this isn't really intended - everything looks like a nail when you only have a hammer, which I think Frank was pointing out furthers the above confusion. In anycase, it sounds like I have, if anything, only made the discussion even more confusing by adding this section; sorry about that. > > If we're past that objection, then perhaps this section can be > > dropped, but there's a very good reason why it was there. I wouldn't > > Brendan to be accused of ignoring feedback from those who reviewed his > > patches. :-) > > > > Agreed. I understand that this FAQ probably was needed at one time and > Brendan added it to address the concerns. I don't want to speak for Frank, but I don't think it was an objection to KUnit itself, but rather an objection to not sufficiently addressing the point about how they differ. > I think at some point we do need to have a document that outlines when > to KUnit and when to use Kselftest modules. I think one concern people > have is that if KUnit is perceived as a replacement for Ksefltest > module, Kselftest module will be ignored leaving users without the > ability to build and run with Kselftest modules and load them on a need > basis to gather data on a systems that aren't dedicated strictly for > testing. I absolutely agree! I posed a suggestion here[1], which after I just now searched for a link, I realize for some reason it didn't seem like it reached a number of the mailing lists that I sent it to, so I should probably resend it. Anyway, a summary of what I suggested: We should start off by better organizing Documentation/dev-tools/ and create a landing page that groups the dev-tools by function according to what person is likely to use them and for what. Eventually and specifically for Kselftest and KUnit, I would like to have a testing guide for the kernel that explains what testing procedure should look like and what to use and when. > I am trying to move the conversation forward from KUnit vs. Kselftest > modules discussion to which problem areas each one addresses keeping > in mind that it is not about which is better. Kselftest and KUnit both > have their place in the kernel development process. We just have to be > clear on usage as we write tests for each. I think that is the right long term approach. I think a good place to start, like I suggested above, is cleaning up Documentation/dev-tools/, but I think that belongs in a (probably several) follow-up patchset. Frank, I believe your objection was mostly related to how KUnit is presented specifically in the cover letter, and doesn't necessarily deal with the intended use case. So I don't think that doing this, especially doing this later, really addresses your concern. I don't want to belabor the issue, but I would also rather not put words in your mouth, what are your thoughts on the above? I think my main concern moving forward on this point is that I am not sure that I can address the debate that this section covers in a way that is both sufficiently concise for a cover letter, but also doesn't invite more potential confusion. My inclination at this point is to drop it since I think the set of reviewers for this patchset has at this point become fixed, and it seems that it will likely cause more confusion rather than reduce it; also, I don't really think this will be an issue for end users, especially once we have proper documentation in place. Alternatively, I guess I could maybe address the point elsewhere and refer to it in the cover letter? Or maybe just put it at the end of the cover letter? [1] https://www.mail-archive.com/kgdb-bugreport@lists.sourceforge.net/msg05059.html _______________________________________________ linux-um mailing list linux-um@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-um