From: frowand.list at gmail.com (Frank Rowand) Subject: [RFC v3 00/19] kunit: introduce KUnit, the Linux kernel unit testing framework Date: Tue, 4 Dec 2018 03:40:37 -0800 [thread overview] Message-ID: <f34ee937-83d6-3ff6-12b3-c6cd9acbfc38@gmail.com> (raw) In-Reply-To: <20181128193636.254378-1-brendanhiggins@google.com> Hi Brendan, Rob, On 11/28/18 11:36 AM, Brendan Higgins wrote: > This patch set proposes KUnit, a lightweight unit testing and mocking > framework for the Linux kernel. > > Unlike Autotest and kselftest, KUnit is a true unit testing framework; > it does not require installing the kernel on a test machine or in a VM > and does not require tests to be written in userspace running on a host > kernel. Additionally, KUnit is fast: From invocation to completion KUnit > can run several dozen tests in under a second. Currently, the entire > KUnit test suite for KUnit runs in under a second from the initial > invocation (build time excluded). > > KUnit is heavily inspired by JUnit, Python's unittest.mock, and > Googletest/Googlemock for C++. KUnit provides facilities for defining > unit test cases, grouping related test cases into test suites, providing > common infrastructure for running tests, mocking, spying, and much more. > > ## What's so special about unit testing? > > A unit test is supposed to test a single unit of code in isolation, > hence the name. There should be no dependencies outside the control of > the test; this means no external dependencies, which makes tests orders > of magnitudes faster. Likewise, since there are no external dependencies, > there are no hoops to jump through to run the tests. Additionally, this > makes unit tests deterministic: a failing unit test always indicates a > problem. Finally, because unit tests necessarily have finer granularity, > they are able to test all code paths easily solving the classic problem > of difficulty in exercising error handling code. > > ## Is KUnit trying to replace other testing frameworks for the kernel? > > No. Most existing tests for the Linux kernel are end-to-end tests, which > have their place. A well tested system has lots of unit tests, a > reasonable number of integration tests, and some end-to-end tests. KUnit > is just trying to address the unit test space which is currently not > being addressed. > > ## More information on KUnit > > There is a bunch of documentation near the end of this patch set that > describes how to use KUnit and best practices for writing unit tests. > For convenience I am hosting the compiled docs here: > https://google.github.io/kunit-docs/third_party/kernel/docs/ > Additionally for convenience, I have applied these patches to a branch: > https://kunit.googlesource.com/linux/+/kunit/rfc/4.19/v3 > The repo may be cloned with: > git clone https://kunit.googlesource.com/linux > This patchset is on the kunit/rfc/4.19/v3 branch. > > ## Changes Since Last Version > > - Changed namespace prefix from `test_*` to `kunit_*` as requested by > Shuah. > - Started converting/cleaning up the device tree unittest to use KUnit. > - Started adding KUnit expectations with custom messages. > Sorry I missed your reply to me in the v1 patch thread. I've been traveling a lot the last few weeks. I'm starting to read messages that occurred late in the v1 patch thread and the v2 patch thread, so I'm just coming up to speed on this. My comments below are motivated by adding the devicetree unittest to this version of the patch series. Pulling a comment from way back in the v1 patch thread: On 10/17/18 3:22 PM, Brendan Higgins wrote: > On Wed, Oct 17, 2018 at 10:49 AM <Tim.Bird at sony.com> wrote: < snip > > The test and the code under test are linked together in the same > binary and are compiled under Kbuild. Right now I am linking > everything into a UML kernel, but I would ultimately like to make > tests compile into completely independent test binaries. So each test > file would get compiled into its own test binary and would link > against only the code needed to run the test, but we are a bit of a > ways off from that. I have never used UML, so you should expect naive questions from me, exhibiting my lack of understanding. Does this mean that I have to build a UML architecture kernel to run the KUnit tests? *** Rob, if the answer is yes, then it seems like for my workflow, which is to build for real ARM hardware, my work is doubled (or worse), because for every patch/commit that I apply, I not only have to build the ARM kernel and boot on the real hardware to test, I also have to build the UML kernel and boot in UML. If that is correct then I see this as a major problem for me. Brenden, in the above quote you said that in the future you would like to make the "tests compile into completely independent test binaries". I am assuming those are intended to run as standalone user space programs instead of inside UML. Is that correct? If so, how will KUnit tests be able to test code that uses locking mechanisms that require instructions that are not available to user space execution? (I _think_ that such instructions may be present, depending on which locking mechanism, but I might be mistaken.) Another possible concern that I have for removing the devicetree unit tests from my normal kernel build process is that I think that the ability to use sparse to analyze the source in the unit tests is removed. Please correct me if I misunderstand that. Another issue is that the devicetree unit tests will no longer be cross compiled with my ARM compiler, so I lose a small amount of testing for compiler related issues. Overall, I'm still trying to learn enough to determine whether the gains from moving to KUnit outweigh the losses. -Frank
WARNING: multiple messages have this Message-ID (diff)
From: frowand.list@gmail.com (Frank Rowand) Subject: [RFC v3 00/19] kunit: introduce KUnit, the Linux kernel unit testing framework Date: Tue, 4 Dec 2018 03:40:37 -0800 [thread overview] Message-ID: <f34ee937-83d6-3ff6-12b3-c6cd9acbfc38@gmail.com> (raw) Message-ID: <20181204114037.Jd9vifcBXuxRoSPxsvsrMlOHwB1X-ziqaPXwdErh6s0@z> (raw) In-Reply-To: <20181128193636.254378-1-brendanhiggins@google.com> Hi Brendan, Rob, On 11/28/18 11:36 AM, Brendan Higgins wrote: > This patch set proposes KUnit, a lightweight unit testing and mocking > framework for the Linux kernel. > > Unlike Autotest and kselftest, KUnit is a true unit testing framework; > it does not require installing the kernel on a test machine or in a VM > and does not require tests to be written in userspace running on a host > kernel. Additionally, KUnit is fast: From invocation to completion KUnit > can run several dozen tests in under a second. Currently, the entire > KUnit test suite for KUnit runs in under a second from the initial > invocation (build time excluded). > > KUnit is heavily inspired by JUnit, Python's unittest.mock, and > Googletest/Googlemock for C++. KUnit provides facilities for defining > unit test cases, grouping related test cases into test suites, providing > common infrastructure for running tests, mocking, spying, and much more. > > ## What's so special about unit testing? > > A unit test is supposed to test a single unit of code in isolation, > hence the name. There should be no dependencies outside the control of > the test; this means no external dependencies, which makes tests orders > of magnitudes faster. Likewise, since there are no external dependencies, > there are no hoops to jump through to run the tests. Additionally, this > makes unit tests deterministic: a failing unit test always indicates a > problem. Finally, because unit tests necessarily have finer granularity, > they are able to test all code paths easily solving the classic problem > of difficulty in exercising error handling code. > > ## Is KUnit trying to replace other testing frameworks for the kernel? > > No. Most existing tests for the Linux kernel are end-to-end tests, which > have their place. A well tested system has lots of unit tests, a > reasonable number of integration tests, and some end-to-end tests. KUnit > is just trying to address the unit test space which is currently not > being addressed. > > ## More information on KUnit > > There is a bunch of documentation near the end of this patch set that > describes how to use KUnit and best practices for writing unit tests. > For convenience I am hosting the compiled docs here: > https://google.github.io/kunit-docs/third_party/kernel/docs/ > Additionally for convenience, I have applied these patches to a branch: > https://kunit.googlesource.com/linux/+/kunit/rfc/4.19/v3 > The repo may be cloned with: > git clone https://kunit.googlesource.com/linux > This patchset is on the kunit/rfc/4.19/v3 branch. > > ## Changes Since Last Version > > - Changed namespace prefix from `test_*` to `kunit_*` as requested by > Shuah. > - Started converting/cleaning up the device tree unittest to use KUnit. > - Started adding KUnit expectations with custom messages. > Sorry I missed your reply to me in the v1 patch thread. I've been traveling a lot the last few weeks. I'm starting to read messages that occurred late in the v1 patch thread and the v2 patch thread, so I'm just coming up to speed on this. My comments below are motivated by adding the devicetree unittest to this version of the patch series. Pulling a comment from way back in the v1 patch thread: On 10/17/18 3:22 PM, Brendan Higgins wrote: > On Wed, Oct 17, 2018@10:49 AM <Tim.Bird@sony.com> wrote: < snip > > The test and the code under test are linked together in the same > binary and are compiled under Kbuild. Right now I am linking > everything into a UML kernel, but I would ultimately like to make > tests compile into completely independent test binaries. So each test > file would get compiled into its own test binary and would link > against only the code needed to run the test, but we are a bit of a > ways off from that. I have never used UML, so you should expect naive questions from me, exhibiting my lack of understanding. Does this mean that I have to build a UML architecture kernel to run the KUnit tests? *** Rob, if the answer is yes, then it seems like for my workflow, which is to build for real ARM hardware, my work is doubled (or worse), because for every patch/commit that I apply, I not only have to build the ARM kernel and boot on the real hardware to test, I also have to build the UML kernel and boot in UML. If that is correct then I see this as a major problem for me. Brenden, in the above quote you said that in the future you would like to make the "tests compile into completely independent test binaries". I am assuming those are intended to run as standalone user space programs instead of inside UML. Is that correct? If so, how will KUnit tests be able to test code that uses locking mechanisms that require instructions that are not available to user space execution? (I _think_ that such instructions may be present, depending on which locking mechanism, but I might be mistaken.) Another possible concern that I have for removing the devicetree unit tests from my normal kernel build process is that I think that the ability to use sparse to analyze the source in the unit tests is removed. Please correct me if I misunderstand that. Another issue is that the devicetree unit tests will no longer be cross compiled with my ARM compiler, so I lose a small amount of testing for compiler related issues. Overall, I'm still trying to learn enough to determine whether the gains from moving to KUnit outweigh the losses. -Frank
next prev parent reply other threads:[~2018-12-04 11:40 UTC|newest] Thread overview: 232+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-11-28 19:36 [RFC v3 00/19] kunit: introduce KUnit, the Linux kernel unit testing framework brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 01/19] kunit: test: add KUnit test runner core brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-30 3:14 ` mcgrof 2018-11-30 3:14 ` Luis Chamberlain 2018-12-01 1:51 ` brendanhiggins 2018-12-01 1:51 ` Brendan Higgins 2018-12-01 2:57 ` mcgrof 2018-12-01 2:57 ` Luis Chamberlain 2018-12-05 13:15 ` anton.ivanov 2018-12-05 13:15 ` Anton Ivanov 2018-12-05 14:45 ` arnd 2018-12-05 14:45 ` Arnd Bergmann 2018-12-05 14:49 ` anton.ivanov 2018-12-05 14:49 ` Anton Ivanov 2018-11-30 3:28 ` mcgrof 2018-11-30 3:28 ` Luis Chamberlain 2018-12-01 2:08 ` brendanhiggins 2018-12-01 2:08 ` Brendan Higgins 2018-12-01 3:10 ` mcgrof 2018-12-01 3:10 ` Luis Chamberlain 2018-12-03 22:47 ` brendanhiggins 2018-12-03 22:47 ` Brendan Higgins 2018-12-01 3:02 ` mcgrof 2018-12-01 3:02 ` Luis Chamberlain 2018-11-28 19:36 ` [RFC v3 02/19] kunit: test: add test resource management API brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 03/19] kunit: test: add string_stream a std::stream like string builder brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-30 3:29 ` mcgrof 2018-11-30 3:29 ` Luis Chamberlain 2018-12-01 2:14 ` brendanhiggins 2018-12-01 2:14 ` Brendan Higgins 2018-12-01 3:12 ` mcgrof 2018-12-01 3:12 ` Luis Chamberlain 2018-12-03 10:55 ` pmladek 2018-12-03 10:55 ` Petr Mladek 2018-12-04 0:35 ` brendanhiggins 2018-12-04 0:35 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 04/19] kunit: test: add test_stream a std::stream like logger brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 05/19] kunit: test: add the concept of expectations brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 06/19] arch: um: enable running kunit from User Mode Linux brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 21:26 ` robh 2018-11-28 21:26 ` Rob Herring 2018-11-30 3:37 ` mcgrof 2018-11-30 3:37 ` Luis Chamberlain 2018-11-30 14:05 ` robh 2018-11-30 14:05 ` Rob Herring 2018-11-30 18:22 ` mcgrof 2018-11-30 18:22 ` Luis Chamberlain 2018-12-03 23:22 ` brendanhiggins 2018-12-03 23:22 ` Brendan Higgins 2018-11-30 3:30 ` mcgrof 2018-11-30 3:30 ` Luis Chamberlain 2018-11-28 19:36 ` [RFC v3 07/19] kunit: test: add initial tests brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-30 3:40 ` mcgrof 2018-11-30 3:40 ` Luis Chamberlain 2018-12-03 23:26 ` brendanhiggins 2018-12-03 23:26 ` Brendan Higgins 2018-12-03 23:43 ` mcgrof 2018-12-03 23:43 ` Luis Chamberlain 2018-11-28 19:36 ` [RFC v3 08/19] arch: um: add shim to trap to allow installing a fault catcher for tests brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-30 3:34 ` mcgrof 2018-11-30 3:34 ` Luis Chamberlain 2018-12-03 23:34 ` brendanhiggins 2018-12-03 23:34 ` Brendan Higgins 2018-12-03 23:46 ` mcgrof 2018-12-03 23:46 ` Luis Chamberlain 2018-12-04 0:44 ` brendanhiggins 2018-12-04 0:44 ` Brendan Higgins 2018-11-30 3:41 ` mcgrof 2018-11-30 3:41 ` Luis Chamberlain 2018-12-03 23:37 ` brendanhiggins 2018-12-03 23:37 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 09/19] kunit: test: add the concept of assertions brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 10/19] kunit: test: add test managed resource tests brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 11/19] kunit: add Python libraries for handing KUnit config and kernel brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-29 13:54 ` kieran.bingham 2018-11-29 13:54 ` Kieran Bingham 2018-12-03 23:48 ` brendanhiggins 2018-12-03 23:48 ` Brendan Higgins 2018-12-04 20:47 ` mcgrof 2018-12-04 20:47 ` Luis Chamberlain 2018-12-06 12:32 ` kieran.bingham 2018-12-06 12:32 ` Kieran Bingham 2018-12-06 15:37 ` willy 2018-12-06 15:37 ` Matthew Wilcox 2018-12-07 11:30 ` kieran.bingham 2018-12-07 11:30 ` Kieran Bingham 2018-12-11 14:09 ` pmladek 2018-12-11 14:09 ` Petr Mladek 2018-12-11 14:41 ` rostedt 2018-12-11 14:41 ` Steven Rostedt 2018-12-11 17:01 ` anton.ivanov 2018-12-11 17:01 ` Anton Ivanov 2019-02-09 0:40 ` brendanhiggins 2019-02-09 0:40 ` Brendan Higgins 2018-12-07 1:05 ` mcgrof 2018-12-07 1:05 ` Luis Chamberlain 2018-12-07 18:35 ` kent.overstreet 2018-12-07 18:35 ` Kent Overstreet 2018-11-30 3:44 ` mcgrof 2018-11-30 3:44 ` Luis Chamberlain 2018-12-03 23:50 ` brendanhiggins 2018-12-03 23:50 ` Brendan Higgins 2018-12-04 20:48 ` mcgrof 2018-12-04 20:48 ` Luis Chamberlain 2018-11-28 19:36 ` [RFC v3 12/19] kunit: add KUnit wrapper script and simple output parser brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 13/19] kunit: improve output from python wrapper brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 14/19] Documentation: kunit: add documentation for KUnit brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-29 13:56 ` kieran.bingham 2018-11-29 13:56 ` Kieran Bingham 2018-11-30 3:45 ` mcgrof 2018-11-30 3:45 ` Luis Chamberlain 2018-12-03 23:53 ` brendanhiggins 2018-12-03 23:53 ` Brendan Higgins 2018-12-06 12:16 ` kieran.bingham 2018-12-06 12:16 ` Kieran Bingham 2019-02-09 0:56 ` brendanhiggins 2019-02-09 0:56 ` Brendan Higgins 2019-02-11 12:16 ` kieran.bingham 2019-02-11 12:16 ` Kieran Bingham 2019-02-12 22:10 ` brendanhiggins 2019-02-12 22:10 ` Brendan Higgins 2019-02-13 21:55 ` kieran.bingham 2019-02-13 21:55 ` Kieran Bingham 2019-02-14 0:17 ` brendanhiggins 2019-02-14 0:17 ` Brendan Higgins 2019-02-14 17:26 ` mcgrof 2019-02-14 17:26 ` Luis Chamberlain 2019-02-14 22:07 ` brendanhiggins 2019-02-14 22:07 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 15/19] MAINTAINERS: add entry for KUnit the unit testing framework brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 16/19] arch: um: make UML unflatten device tree when testing brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 21:16 ` robh 2018-11-28 21:16 ` Rob Herring 2018-12-04 0:00 ` brendanhiggins 2018-12-04 0:00 ` Brendan Higgins 2018-11-30 3:46 ` mcgrof 2018-11-30 3:46 ` Luis Chamberlain 2018-12-04 0:02 ` brendanhiggins 2018-12-04 0:02 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 17/19] of: unittest: migrate tests to run on KUnit brendanhiggins 2018-11-28 19:36 ` Brendan Higgins [not found] ` <CAL_Jsq+09Kx7yMBC_Jw45QGmk6U_fp4N6HOZDwYrM4tWw+_dOA@mail.gmail.com> 2018-11-30 0:39 ` rdunlap 2018-11-30 0:39 ` Randy Dunlap 2018-12-04 0:13 ` brendanhiggins 2018-12-04 0:13 ` Brendan Higgins 2018-12-04 13:40 ` robh 2018-12-04 13:40 ` Rob Herring 2018-12-05 23:42 ` brendanhiggins 2018-12-05 23:42 ` Brendan Higgins 2018-12-07 0:41 ` robh 2018-12-07 0:41 ` Rob Herring 2018-12-04 0:08 ` brendanhiggins 2018-12-04 0:08 ` Brendan Higgins 2019-02-13 1:44 ` brendanhiggins 2019-02-13 1:44 ` Brendan Higgins 2019-02-14 20:10 ` robh 2019-02-14 20:10 ` Rob Herring 2019-02-14 21:52 ` brendanhiggins 2019-02-14 21:52 ` Brendan Higgins 2019-02-18 22:56 ` frowand.list 2019-02-18 22:56 ` Frank Rowand 2019-02-28 0:29 ` brendanhiggins 2019-02-28 0:29 ` Brendan Higgins 2018-12-04 10:56 ` frowand.list 2018-12-04 10:56 ` Frank Rowand 2018-11-28 19:36 ` [RFC v3 18/19] of: unittest: split out a couple of test cases from unittest brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-12-04 10:58 ` frowand.list 2018-12-04 10:58 ` Frank Rowand 2018-12-05 23:54 ` brendanhiggins 2018-12-05 23:54 ` Brendan Higgins 2019-02-14 23:57 ` frowand.list 2019-02-14 23:57 ` Frank Rowand 2019-02-15 0:56 ` brendanhiggins 2019-02-15 0:56 ` Brendan Higgins 2019-02-15 2:05 ` frowand.list 2019-02-15 2:05 ` Frank Rowand 2019-02-15 10:56 ` brendanhiggins 2019-02-15 10:56 ` Brendan Higgins 2019-02-18 22:25 ` frowand.list 2019-02-18 22:25 ` Frank Rowand 2019-02-20 20:44 ` frowand.list 2019-02-20 20:44 ` Frank Rowand 2019-02-20 20:47 ` frowand.list 2019-02-20 20:47 ` Frank Rowand 2019-02-28 3:52 ` brendanhiggins 2019-02-28 3:52 ` Brendan Higgins 2019-03-22 0:22 ` frowand.list 2019-03-22 0:22 ` Frank Rowand 2019-03-22 1:30 ` brendanhiggins 2019-03-22 1:30 ` Brendan Higgins 2019-03-22 1:47 ` frowand.list 2019-03-22 1:47 ` Frank Rowand 2019-03-25 22:15 ` brendanhiggins 2019-03-25 22:15 ` Brendan Higgins 2019-09-20 16:57 ` Rob Herring 2019-09-21 23:57 ` Frank Rowand 2019-03-22 1:34 ` frowand.list 2019-03-22 1:34 ` Frank Rowand 2019-03-25 22:18 ` brendanhiggins 2019-03-25 22:18 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 19/19] of: unittest: split up some super large test cases brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-12-04 10:52 ` [RFC v3 00/19] kunit: introduce KUnit, the Linux kernel unit testing framework frowand.list 2018-12-04 10:52 ` Frank Rowand 2018-12-04 11:40 ` frowand.list [this message] 2018-12-04 11:40 ` Frank Rowand 2018-12-04 13:49 ` robh 2018-12-04 13:49 ` Rob Herring 2018-12-05 23:10 ` brendanhiggins 2018-12-05 23:10 ` Brendan Higgins 2019-03-22 0:27 ` frowand.list 2019-03-22 0:27 ` Frank Rowand 2019-03-25 22:04 ` brendanhiggins 2019-03-25 22:04 ` Brendan Higgins
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=f34ee937-83d6-3ff6-12b3-c6cd9acbfc38@gmail.com \ --to=linux-kselftest@vger.kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).