From: frowand.list at gmail.com (Frank Rowand) Subject: [PATCH v2 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework Date: Wed, 8 May 2019 17:43:35 -0700 [thread overview] Message-ID: <a09a7e0e-9894-8c1a-34eb-fc482b1759d0@gmail.com> (raw) In-Reply-To: <20190507080119.GB28121@kroah.com> On 5/7/19 1:01 AM, Greg KH wrote: > On Mon, May 06, 2019 at 08:14:12PM -0700, Frank Rowand wrote: >> On 5/1/19 4:01 PM, Brendan Higgins wrote: >>> ## TLDR >>> >>> I rebased the last patchset on 5.1-rc7 in hopes that we can get this in >>> 5.2. >>> >>> Shuah, I think you, Greg KH, and myself talked off thread, and we agreed >>> we would merge through your tree when the time came? Am I remembering >>> correctly? >>> >>> ## Background >>> >>> This patch set proposes KUnit, a lightweight unit testing and mocking >>> framework for the Linux kernel. >>> >>> Unlike Autotest and kselftest, KUnit is a true unit testing framework; >>> it does not require installing the kernel on a test machine or in a VM >>> and does not require tests to be written in userspace running on a host >>> kernel. Additionally, KUnit is fast: From invocation to completion KUnit >>> can run several dozen tests in under a second. Currently, the entire >>> KUnit test suite for KUnit runs in under a second from the initial >>> invocation (build time excluded). >>> >>> KUnit is heavily inspired by JUnit, Python's unittest.mock, and >>> Googletest/Googlemock for C++. KUnit provides facilities for defining >>> unit test cases, grouping related test cases into test suites, providing >>> common infrastructure for running tests, mocking, spying, and much more. >> >> As a result of the emails replying to this patch thread, I am now >> starting to look at kselftest. My level of understanding is based >> on some slide presentations, an LWN article, https://kselftest.wiki.kernel.org/ >> and a _tiny_ bit of looking at kselftest code. >> >> tl;dr; I don't really understand kselftest yet. >> >> >> (1) why KUnit exists >> >>> ## What's so special about unit testing? >>> >>> A unit test is supposed to test a single unit of code in isolation, >>> hence the name. There should be no dependencies outside the control of >>> the test; this means no external dependencies, which makes tests orders >>> of magnitudes faster. Likewise, since there are no external dependencies, >>> there are no hoops to jump through to run the tests. Additionally, this >>> makes unit tests deterministic: a failing unit test always indicates a >>> problem. Finally, because unit tests necessarily have finer granularity, >>> they are able to test all code paths easily solving the classic problem >>> of difficulty in exercising error handling code. >> >> (2) KUnit is not meant to replace kselftest >> >>> ## Is KUnit trying to replace other testing frameworks for the kernel? >>> >>> No. Most existing tests for the Linux kernel are end-to-end tests, which >>> have their place. A well tested system has lots of unit tests, a >>> reasonable number of integration tests, and some end-to-end tests. KUnit >>> is just trying to address the unit test space which is currently not >>> being addressed. >> >> My understanding is that the intent of KUnit is to avoid booting a kernel on >> real hardware or in a virtual machine. That seems to be a matter of semantics >> to me because isn't invoking a UML Linux just running the Linux kernel in >> a different form of virtualization? >> >> So I do not understand why KUnit is an improvement over kselftest. >> >> It seems to me that KUnit is just another piece of infrastructure that I >> am going to have to be familiar with as a kernel developer. More overhead, >> more information to stuff into my tiny little brain. >> >> I would guess that some developers will focus on just one of the two test >> environments (and some will focus on both), splitting the development >> resources instead of pooling them on a common infrastructure. >> >> What am I missing? > > kselftest provides no in-kernel framework for testing kernel code > specifically. That should be what kunit provides, an "easy" way to > write in-kernel tests for things. kselftest provides a mechanism for in-kernel tests via modules. For example, see: tools/testing/selftests/vm/run_vmtests invokes: tools/testing/selftests/vm/test_vmalloc.sh loads module: test_vmalloc (which is built from lib/test_vmalloc.c if CONFIG_TEST_VMALLOC) A very quick and dirty search (likely to miss some tests) finds modules: test_bitmap test_bpf test_firmware test_printf test_static_key_base test_static_keys test_user_copy test_vmalloc -Frank > > Brendan, did I get it right? > > thanks, > > greg k-h > . >
WARNING: multiple messages have this Message-ID (diff)
From: frowand.list@gmail.com (Frank Rowand) Subject: [PATCH v2 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework Date: Wed, 8 May 2019 17:43:35 -0700 [thread overview] Message-ID: <a09a7e0e-9894-8c1a-34eb-fc482b1759d0@gmail.com> (raw) Message-ID: <20190509004335.a9ZQ_0To6lFBrWH7_AEfgf4-ZOBt7ttyPmsteCIz5fg@z> (raw) In-Reply-To: <20190507080119.GB28121@kroah.com> On 5/7/19 1:01 AM, Greg KH wrote: > On Mon, May 06, 2019@08:14:12PM -0700, Frank Rowand wrote: >> On 5/1/19 4:01 PM, Brendan Higgins wrote: >>> ## TLDR >>> >>> I rebased the last patchset on 5.1-rc7 in hopes that we can get this in >>> 5.2. >>> >>> Shuah, I think you, Greg KH, and myself talked off thread, and we agreed >>> we would merge through your tree when the time came? Am I remembering >>> correctly? >>> >>> ## Background >>> >>> This patch set proposes KUnit, a lightweight unit testing and mocking >>> framework for the Linux kernel. >>> >>> Unlike Autotest and kselftest, KUnit is a true unit testing framework; >>> it does not require installing the kernel on a test machine or in a VM >>> and does not require tests to be written in userspace running on a host >>> kernel. Additionally, KUnit is fast: From invocation to completion KUnit >>> can run several dozen tests in under a second. Currently, the entire >>> KUnit test suite for KUnit runs in under a second from the initial >>> invocation (build time excluded). >>> >>> KUnit is heavily inspired by JUnit, Python's unittest.mock, and >>> Googletest/Googlemock for C++. KUnit provides facilities for defining >>> unit test cases, grouping related test cases into test suites, providing >>> common infrastructure for running tests, mocking, spying, and much more. >> >> As a result of the emails replying to this patch thread, I am now >> starting to look at kselftest. My level of understanding is based >> on some slide presentations, an LWN article, https://kselftest.wiki.kernel.org/ >> and a _tiny_ bit of looking at kselftest code. >> >> tl;dr; I don't really understand kselftest yet. >> >> >> (1) why KUnit exists >> >>> ## What's so special about unit testing? >>> >>> A unit test is supposed to test a single unit of code in isolation, >>> hence the name. There should be no dependencies outside the control of >>> the test; this means no external dependencies, which makes tests orders >>> of magnitudes faster. Likewise, since there are no external dependencies, >>> there are no hoops to jump through to run the tests. Additionally, this >>> makes unit tests deterministic: a failing unit test always indicates a >>> problem. Finally, because unit tests necessarily have finer granularity, >>> they are able to test all code paths easily solving the classic problem >>> of difficulty in exercising error handling code. >> >> (2) KUnit is not meant to replace kselftest >> >>> ## Is KUnit trying to replace other testing frameworks for the kernel? >>> >>> No. Most existing tests for the Linux kernel are end-to-end tests, which >>> have their place. A well tested system has lots of unit tests, a >>> reasonable number of integration tests, and some end-to-end tests. KUnit >>> is just trying to address the unit test space which is currently not >>> being addressed. >> >> My understanding is that the intent of KUnit is to avoid booting a kernel on >> real hardware or in a virtual machine. That seems to be a matter of semantics >> to me because isn't invoking a UML Linux just running the Linux kernel in >> a different form of virtualization? >> >> So I do not understand why KUnit is an improvement over kselftest. >> >> It seems to me that KUnit is just another piece of infrastructure that I >> am going to have to be familiar with as a kernel developer. More overhead, >> more information to stuff into my tiny little brain. >> >> I would guess that some developers will focus on just one of the two test >> environments (and some will focus on both), splitting the development >> resources instead of pooling them on a common infrastructure. >> >> What am I missing? > > kselftest provides no in-kernel framework for testing kernel code > specifically. That should be what kunit provides, an "easy" way to > write in-kernel tests for things. kselftest provides a mechanism for in-kernel tests via modules. For example, see: tools/testing/selftests/vm/run_vmtests invokes: tools/testing/selftests/vm/test_vmalloc.sh loads module: test_vmalloc (which is built from lib/test_vmalloc.c if CONFIG_TEST_VMALLOC) A very quick and dirty search (likely to miss some tests) finds modules: test_bitmap test_bpf test_firmware test_printf test_static_key_base test_static_keys test_user_copy test_vmalloc -Frank > > Brendan, did I get it right? > > thanks, > > greg k-h > . >
next prev parent reply other threads:[~2019-05-09 0:43 UTC|newest] Thread overview: 262+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-05-01 23:01 [PATCH v2 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 01/17] kunit: test: add KUnit test runner core brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 02/17] kunit: test: add test resource management API brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 03/17] kunit: test: add string_stream a std::stream like string builder brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-03 1:26 ` shuah 2019-05-03 1:26 ` shuah 2019-05-03 4:37 ` brendanhiggins 2019-05-03 4:37 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 04/17] kunit: test: add kunit_stream a std::stream like logger brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-02 11:00 ` gregkh 2019-05-02 11:00 ` Greg KH 2019-05-02 20:25 ` brendanhiggins 2019-05-02 20:25 ` Brendan Higgins 2019-05-02 21:18 ` frowand.list 2019-05-02 21:18 ` Frank Rowand 2019-05-03 1:50 ` shuah 2019-05-03 1:50 ` shuah 2019-05-03 5:48 ` brendanhiggins 2019-05-03 5:48 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 05/17] kunit: test: add the concept of expectations brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 06/17] kbuild: enable building KUnit brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-10 3:03 ` yamada.masahiro 2019-05-10 3:03 ` Masahiro Yamada 2019-05-10 10:27 ` brendanhiggins 2019-05-10 10:27 ` Brendan Higgins 2019-05-10 10:30 ` yamada.masahiro 2019-05-10 10:30 ` Masahiro Yamada 2019-05-10 10:33 ` brendanhiggins 2019-05-10 10:33 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 07/17] kunit: test: add initial tests brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-02 10:58 ` gregkh 2019-05-02 10:58 ` Greg KH 2019-05-02 20:30 ` brendanhiggins 2019-05-02 20:30 ` Brendan Higgins 2019-05-03 1:27 ` shuah 2019-05-03 1:27 ` shuah 2019-05-03 5:18 ` brendanhiggins 2019-05-03 5:18 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 08/17] kunit: test: add support for test abort brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-03 3:14 ` logang 2019-05-03 3:14 ` Logan Gunthorpe 2019-05-03 6:48 ` brendanhiggins 2019-05-03 6:48 ` Brendan Higgins 2019-05-03 12:33 ` logang 2019-05-03 12:33 ` Logan Gunthorpe 2019-05-06 8:48 ` brendanhiggins 2019-05-06 8:48 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 09/17] kunit: test: add tests for kunit " brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 10/17] kunit: test: add the concept of assertions brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 11/17] kunit: test: add test managed resource tests brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-03 14:34 ` shuah 2019-05-03 14:34 ` shuah 2019-05-06 9:03 ` brendanhiggins 2019-05-06 9:03 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 12/17] kunit: tool: add Python wrappers for running KUnit tests brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-02 11:02 ` gregkh 2019-05-02 11:02 ` Greg KH 2019-05-02 18:07 ` brendanhiggins 2019-05-02 18:07 ` Brendan Higgins 2019-05-02 21:16 ` frowand.list 2019-05-02 21:16 ` Frank Rowand 2019-05-02 23:45 ` brendanhiggins 2019-05-02 23:45 ` Brendan Higgins 2019-05-03 1:45 ` frowand.list 2019-05-03 1:45 ` Frank Rowand 2019-05-03 5:36 ` brendanhiggins 2019-05-03 5:36 ` Brendan Higgins 2019-05-03 18:59 ` frowand.list 2019-05-03 18:59 ` Frank Rowand 2019-05-03 23:14 ` brendanhiggins 2019-05-03 23:14 ` Brendan Higgins 2019-05-04 10:42 ` gregkh 2019-05-04 10:42 ` Greg KH 2019-05-06 0:19 ` frowand.list 2019-05-06 0:19 ` Frank Rowand 2019-05-06 17:43 ` keescook 2019-05-06 17:43 ` Kees Cook 2019-05-06 21:42 ` brendanhiggins 2019-05-06 21:42 ` Brendan Higgins 2019-05-06 21:39 ` brendanhiggins 2019-05-06 21:39 ` Brendan Higgins 2019-05-07 19:13 ` Tim.Bird 2019-05-07 19:13 ` Tim.Bird 2019-05-03 6:41 ` gregkh 2019-05-03 6:41 ` Greg KH 2019-05-01 23:01 ` [PATCH v2 13/17] kunit: defconfig: add defconfigs for building " brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 14/17] Documentation: kunit: add documentation for KUnit brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-09 5:08 ` rdunlap 2019-05-09 5:08 ` Randy Dunlap 2019-05-09 17:38 ` brendanhiggins 2019-05-09 17:38 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 15/17] MAINTAINERS: add entry for KUnit the unit testing framework brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-03 14:38 ` shuah 2019-05-03 14:38 ` shuah 2019-05-06 9:18 ` brendanhiggins 2019-05-06 9:18 ` Brendan Higgins 2019-05-01 23:01 ` [PATCH v2 16/17] kernel/sysctl-test: Add null pointer test for sysctl.c:proc_dointvec() brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-02 11:03 ` gregkh 2019-05-02 11:03 ` Greg KH 2019-05-02 18:14 ` Tim.Bird 2019-05-02 18:14 ` Tim.Bird 2019-05-02 18:45 ` brendanhiggins 2019-05-02 18:45 ` Brendan Higgins 2019-05-03 6:42 ` gregkh 2019-05-03 6:42 ` Greg KH 2019-05-03 23:41 ` brendanhiggins 2019-05-03 23:41 ` Brendan Higgins 2019-05-04 10:40 ` gregkh 2019-05-04 10:40 ` Greg KH 2019-05-01 23:01 ` [PATCH v2 17/17] MAINTAINERS: add proc sysctl KUnit test to PROC SYSCTL section brendanhiggins 2019-05-01 23:01 ` Brendan Higgins 2019-05-02 10:50 ` [PATCH v2 00/17] kunit: introduce KUnit, the Linux kernel unit testing framework gregkh 2019-05-02 10:50 ` Greg KH 2019-05-02 11:05 ` gregkh 2019-05-02 11:05 ` Greg KH 2019-05-03 0:41 ` brendanhiggins 2019-05-03 0:41 ` Brendan Higgins 2019-05-02 14:04 ` shuah 2019-05-02 14:04 ` shuah 2019-05-03 0:44 ` brendanhiggins 2019-05-03 0:44 ` Brendan Higgins 2019-05-03 3:18 ` logang 2019-05-03 3:18 ` Logan Gunthorpe 2019-05-07 3:14 ` frowand.list 2019-05-07 3:14 ` Frank Rowand 2019-05-07 8:01 ` gregkh 2019-05-07 8:01 ` Greg KH 2019-05-07 15:23 ` shuah 2019-05-07 15:23 ` shuah 2019-05-09 1:01 ` frowand.list 2019-05-09 1:01 ` Frank Rowand 2019-05-07 17:22 ` tytso 2019-05-07 17:22 ` Theodore Ts'o 2019-05-08 19:17 ` brendanhiggins 2019-05-08 19:17 ` Brendan Higgins 2019-05-09 0:58 ` frowand.list 2019-05-09 0:58 ` Frank Rowand 2019-05-09 1:44 ` tytso 2019-05-09 1:44 ` Theodore Ts'o 2019-05-09 2:18 ` frowand.list 2019-05-09 2:18 ` Frank Rowand 2019-05-14 8:22 ` brendanhiggins 2019-05-14 8:22 ` Brendan Higgins 2019-05-09 0:43 ` frowand.list [this message] 2019-05-09 0:43 ` Frank Rowand 2019-05-09 1:58 ` tytso 2019-05-09 1:58 ` Theodore Ts'o 2019-05-09 2:13 ` frowand.list 2019-05-09 2:13 ` Frank Rowand 2019-05-09 3:20 ` tytso 2019-05-09 3:20 ` Theodore Ts'o 2019-05-09 11:52 ` knut.omang 2019-05-09 11:52 ` Knut Omang 2019-05-09 13:35 ` tytso 2019-05-09 13:35 ` Theodore Ts'o 2019-05-09 14:48 ` knut.omang 2019-05-09 14:48 ` Knut Omang 2019-05-09 17:00 ` Tim.Bird 2019-05-09 17:00 ` Tim.Bird 2019-05-09 17:42 ` daniel 2019-05-09 17:42 ` Daniel Vetter 2019-05-09 18:12 ` frowand.list 2019-05-09 18:12 ` Frank Rowand 2019-05-09 21:42 ` tytso 2019-05-09 21:42 ` Theodore Ts'o 2019-05-09 22:20 ` logang 2019-05-09 22:20 ` Logan Gunthorpe 2019-05-09 23:30 ` tytso 2019-05-09 23:30 ` Theodore Ts'o 2019-05-09 23:40 ` logang 2019-05-09 23:40 ` Logan Gunthorpe 2019-05-10 4:47 ` tytso 2019-05-10 4:47 ` Theodore Ts'o 2019-05-10 5:18 ` frowand.list 2019-05-10 5:18 ` Frank Rowand 2019-05-10 5:48 ` knut.omang 2019-05-10 5:48 ` Knut Omang 2019-05-10 8:12 ` daniel 2019-05-10 8:12 ` Daniel Vetter 2019-05-10 10:23 ` brendanhiggins 2019-05-10 10:23 ` Brendan Higgins 2019-05-10 12:12 ` knut.omang 2019-05-10 12:12 ` Knut Omang 2019-05-10 20:54 ` brendanhiggins 2019-05-10 20:54 ` Brendan Higgins 2019-05-10 22:18 ` frowand.list 2019-05-10 22:18 ` Frank Rowand 2019-05-11 6:17 ` knut.omang 2019-05-11 6:17 ` Knut Omang 2019-05-14 6:39 ` brendanhiggins 2019-05-14 6:39 ` Brendan Higgins 2019-05-10 21:59 ` frowand.list 2019-05-10 21:59 ` Frank Rowand 2019-05-11 6:43 ` knut.omang 2019-05-11 6:43 ` Knut Omang 2019-05-14 8:00 ` brendanhiggins 2019-05-14 8:00 ` Brendan Higgins 2019-05-10 11:36 ` knut.omang 2019-05-10 11:36 ` Knut Omang 2019-05-10 16:17 ` logang 2019-05-10 16:17 ` Logan Gunthorpe 2019-05-10 22:13 ` frowand.list 2019-05-10 22:13 ` Frank Rowand 2019-05-14 8:38 ` brendanhiggins 2019-05-14 8:38 ` Brendan Higgins 2019-05-15 0:14 ` frowand.list 2019-05-15 0:14 ` Frank Rowand 2019-05-15 0:26 ` logang 2019-05-15 0:26 ` Logan Gunthorpe 2019-05-10 21:52 ` frowand.list 2019-05-10 21:52 ` Frank Rowand 2019-05-14 20:54 ` brendanhiggins 2019-05-14 20:54 ` Brendan Higgins 2019-05-10 21:12 ` frowand.list 2019-05-10 21:12 ` Frank Rowand 2019-05-11 17:33 ` tytso 2019-05-11 17:33 ` Theodore Ts'o 2019-05-13 14:44 ` daniel 2019-05-13 14:44 ` Daniel Vetter 2019-05-14 6:04 ` brendanhiggins 2019-05-14 6:04 ` Brendan Higgins 2019-05-14 12:05 ` daniel 2019-05-14 12:05 ` Daniel Vetter 2019-05-14 18:36 ` brendanhiggins 2019-05-14 18:36 ` Brendan Higgins 2019-05-15 7:41 ` daniel 2019-05-15 7:41 ` Daniel Vetter 2019-05-22 21:38 ` brendanhiggins 2019-05-22 21:38 ` Brendan Higgins 2019-05-23 8:40 ` daniel 2019-05-23 8:40 ` Daniel Vetter 2019-05-15 0:26 ` frowand.list 2019-05-15 0:26 ` Frank Rowand 2019-05-15 4:28 ` tytso 2019-05-15 4:28 ` Theodore Ts'o 2019-05-10 5:11 ` frowand.list 2019-05-10 5:11 ` Frank Rowand 2019-05-10 10:43 ` tytso 2019-05-10 10:43 ` Theodore Ts'o 2019-05-10 21:05 ` frowand.list 2019-05-10 21:05 ` Frank Rowand 2019-05-09 15:19 ` yamada.masahiro 2019-05-09 15:19 ` Masahiro Yamada 2019-05-10 10:25 ` brendanhiggins 2019-05-10 10:25 ` Brendan Higgins
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=a09a7e0e-9894-8c1a-34eb-fc482b1759d0@gmail.com \ --to=linux-kselftest@vger.kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).