From: kieran.bingham at ideasonboard.com (Kieran Bingham) Subject: [RFC v3 11/19] kunit: add Python libraries for handing KUnit config and kernel Date: Thu, 6 Dec 2018 12:32:47 +0000 [thread overview] Message-ID: <f8722f4a-44c8-24c2-c433-5178f9f40b82@ideasonboard.com> (raw) In-Reply-To: <20181204204701.GT28501@garbanzo.do-not-panic.com> Hi Luis, On 04/12/2018 20:47, Luis Chamberlain wrote: > On Mon, Dec 03, 2018 at 03:48:15PM -0800, Brendan Higgins wrote: >> On Thu, Nov 29, 2018 at 5:54 AM Kieran Bingham >> <kieran.bingham at ideasonboard.com> wrote: >>> >>> Hi Brendan, >>> >>> Thanks again for this series! >>> >>> On 28/11/2018 19:36, Brendan Higgins wrote: >>>> The ultimate goal is to create minimal isolated test binaries; in the >>>> meantime we are using UML to provide the infrastructure to run tests, so >>>> define an abstract way to configure and run tests that allow us to >>>> change the context in which tests are built without affecting the user. >>>> This also makes pretty and dynamic error reporting, and a lot of other >>>> nice features easier. >>> >>> >>> I wonder if we could somehow generate a shared library object >>> 'libkernel' or 'libumlinux' from a UM configured set of headers and >>> objects so that we could create binary targets directly ? >> >> That's an interesting idea. I think it would be difficult to figure >> out exactly where to draw the line of what goes in there and what >> needs to be built specific to a test a priori. Of course, that leads >> into the biggest problem in general, needed to know what I need to >> build to test the thing that I want to test. >> >> Nevertheless, I could definitely imagine that being useful in a lot of cases. > > Whether or not we can abstract away the kernel into such a mechanism > with uml libraries is a good question worth exploring. > > Developers working upstream do modify their kernels a lot, so we'd have > to update such libraries quite a bit, but I think that's fine too. The > *real* value I think from the above suggestion would be enterprise / > mobile distros or stable kernel maintainers which have a static kernel > they need to support for a relatively *long time*, consider a 10 year > time frame. Running unit tests without qemu with uml and libraries for > respective kernels seems real worthy. I think any such library might be something generated by the kernel build system, so if someone makes substantial changes to a core component provided by the library - it can be up to them to build a corresponding userspace library as well. We could also consider to only provide *static* libraries rather than dynamic. So any one building some userspace tool / test with this would be required to compile against (the version of) the kernel they expect perhaps... - much like we expect modules to be compiled currently. And then the userspace binary would be sufficiently able to live it's life on it's own :) > The overhead for testing a unit test for said targets, *ideally*, would > just be to to reboot into the system with such libraries available, a > unit test would just look for the respective uname -r library and mimic > that kernel, much the same way enterprise distributions today rely on > having debugging symbols available to run against crash / gdb. Having > debug modules / kernel for crash requires such effort already, so this > would just be an extra layer of other prospect tests. Oh - although, yes - there are some good concepts there - but I'm a bit weary of how easy it would be to 'run' the said test against multiple kernel version libraries... there would be a lot of possible ABI conflicts perhaps. My main initial idea for a libumlinux is to provide infrastructure such as our linked-lists and other kernel formatting so that we can take kernel code directly to userspace for test and debug (assuming that there are no hardware dependencies or things that we can't mock out) I think all of this could complement kunit of course - this isn't suggesting an alternative implementation :-) > All ideaware for now, but the roadmap seems to be paving itself. I guess all great ideas start as ideaware somehow ... Now we just have to start the race to see who can tweak the kernel build system to produce an output library first :) (I won't be upset if I don't win the race) -- Regards -- Kieran
WARNING: multiple messages have this Message-ID (diff)
From: kieran.bingham@ideasonboard.com (Kieran Bingham) Subject: [RFC v3 11/19] kunit: add Python libraries for handing KUnit config and kernel Date: Thu, 6 Dec 2018 12:32:47 +0000 [thread overview] Message-ID: <f8722f4a-44c8-24c2-c433-5178f9f40b82@ideasonboard.com> (raw) Message-ID: <20181206123247.1s6iyriR_BXKPD1IgNwuWJLIZ58T7UfB38sQZXSeR8g@z> (raw) In-Reply-To: <20181204204701.GT28501@garbanzo.do-not-panic.com> Hi Luis, On 04/12/2018 20:47, Luis Chamberlain wrote: > On Mon, Dec 03, 2018@03:48:15PM -0800, Brendan Higgins wrote: >> On Thu, Nov 29, 2018 at 5:54 AM Kieran Bingham >> <kieran.bingham@ideasonboard.com> wrote: >>> >>> Hi Brendan, >>> >>> Thanks again for this series! >>> >>> On 28/11/2018 19:36, Brendan Higgins wrote: >>>> The ultimate goal is to create minimal isolated test binaries; in the >>>> meantime we are using UML to provide the infrastructure to run tests, so >>>> define an abstract way to configure and run tests that allow us to >>>> change the context in which tests are built without affecting the user. >>>> This also makes pretty and dynamic error reporting, and a lot of other >>>> nice features easier. >>> >>> >>> I wonder if we could somehow generate a shared library object >>> 'libkernel' or 'libumlinux' from a UM configured set of headers and >>> objects so that we could create binary targets directly ? >> >> That's an interesting idea. I think it would be difficult to figure >> out exactly where to draw the line of what goes in there and what >> needs to be built specific to a test a priori. Of course, that leads >> into the biggest problem in general, needed to know what I need to >> build to test the thing that I want to test. >> >> Nevertheless, I could definitely imagine that being useful in a lot of cases. > > Whether or not we can abstract away the kernel into such a mechanism > with uml libraries is a good question worth exploring. > > Developers working upstream do modify their kernels a lot, so we'd have > to update such libraries quite a bit, but I think that's fine too. The > *real* value I think from the above suggestion would be enterprise / > mobile distros or stable kernel maintainers which have a static kernel > they need to support for a relatively *long time*, consider a 10 year > time frame. Running unit tests without qemu with uml and libraries for > respective kernels seems real worthy. I think any such library might be something generated by the kernel build system, so if someone makes substantial changes to a core component provided by the library - it can be up to them to build a corresponding userspace library as well. We could also consider to only provide *static* libraries rather than dynamic. So any one building some userspace tool / test with this would be required to compile against (the version of) the kernel they expect perhaps... - much like we expect modules to be compiled currently. And then the userspace binary would be sufficiently able to live it's life on it's own :) > The overhead for testing a unit test for said targets, *ideally*, would > just be to to reboot into the system with such libraries available, a > unit test would just look for the respective uname -r library and mimic > that kernel, much the same way enterprise distributions today rely on > having debugging symbols available to run against crash / gdb. Having > debug modules / kernel for crash requires such effort already, so this > would just be an extra layer of other prospect tests. Oh - although, yes - there are some good concepts there - but I'm a bit weary of how easy it would be to 'run' the said test against multiple kernel version libraries... there would be a lot of possible ABI conflicts perhaps. My main initial idea for a libumlinux is to provide infrastructure such as our linked-lists and other kernel formatting so that we can take kernel code directly to userspace for test and debug (assuming that there are no hardware dependencies or things that we can't mock out) I think all of this could complement kunit of course - this isn't suggesting an alternative implementation :-) > All ideaware for now, but the roadmap seems to be paving itself. I guess all great ideas start as ideaware somehow ... Now we just have to start the race to see who can tweak the kernel build system to produce an output library first :) (I won't be upset if I don't win the race) -- Regards -- Kieran
next prev parent reply other threads:[~2018-12-06 12:32 UTC|newest] Thread overview: 232+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-11-28 19:36 [RFC v3 00/19] kunit: introduce KUnit, the Linux kernel unit testing framework brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 01/19] kunit: test: add KUnit test runner core brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-30 3:14 ` mcgrof 2018-11-30 3:14 ` Luis Chamberlain 2018-12-01 1:51 ` brendanhiggins 2018-12-01 1:51 ` Brendan Higgins 2018-12-01 2:57 ` mcgrof 2018-12-01 2:57 ` Luis Chamberlain 2018-12-05 13:15 ` anton.ivanov 2018-12-05 13:15 ` Anton Ivanov 2018-12-05 14:45 ` arnd 2018-12-05 14:45 ` Arnd Bergmann 2018-12-05 14:49 ` anton.ivanov 2018-12-05 14:49 ` Anton Ivanov 2018-11-30 3:28 ` mcgrof 2018-11-30 3:28 ` Luis Chamberlain 2018-12-01 2:08 ` brendanhiggins 2018-12-01 2:08 ` Brendan Higgins 2018-12-01 3:10 ` mcgrof 2018-12-01 3:10 ` Luis Chamberlain 2018-12-03 22:47 ` brendanhiggins 2018-12-03 22:47 ` Brendan Higgins 2018-12-01 3:02 ` mcgrof 2018-12-01 3:02 ` Luis Chamberlain 2018-11-28 19:36 ` [RFC v3 02/19] kunit: test: add test resource management API brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 03/19] kunit: test: add string_stream a std::stream like string builder brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-30 3:29 ` mcgrof 2018-11-30 3:29 ` Luis Chamberlain 2018-12-01 2:14 ` brendanhiggins 2018-12-01 2:14 ` Brendan Higgins 2018-12-01 3:12 ` mcgrof 2018-12-01 3:12 ` Luis Chamberlain 2018-12-03 10:55 ` pmladek 2018-12-03 10:55 ` Petr Mladek 2018-12-04 0:35 ` brendanhiggins 2018-12-04 0:35 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 04/19] kunit: test: add test_stream a std::stream like logger brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 05/19] kunit: test: add the concept of expectations brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 06/19] arch: um: enable running kunit from User Mode Linux brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 21:26 ` robh 2018-11-28 21:26 ` Rob Herring 2018-11-30 3:37 ` mcgrof 2018-11-30 3:37 ` Luis Chamberlain 2018-11-30 14:05 ` robh 2018-11-30 14:05 ` Rob Herring 2018-11-30 18:22 ` mcgrof 2018-11-30 18:22 ` Luis Chamberlain 2018-12-03 23:22 ` brendanhiggins 2018-12-03 23:22 ` Brendan Higgins 2018-11-30 3:30 ` mcgrof 2018-11-30 3:30 ` Luis Chamberlain 2018-11-28 19:36 ` [RFC v3 07/19] kunit: test: add initial tests brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-30 3:40 ` mcgrof 2018-11-30 3:40 ` Luis Chamberlain 2018-12-03 23:26 ` brendanhiggins 2018-12-03 23:26 ` Brendan Higgins 2018-12-03 23:43 ` mcgrof 2018-12-03 23:43 ` Luis Chamberlain 2018-11-28 19:36 ` [RFC v3 08/19] arch: um: add shim to trap to allow installing a fault catcher for tests brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-30 3:34 ` mcgrof 2018-11-30 3:34 ` Luis Chamberlain 2018-12-03 23:34 ` brendanhiggins 2018-12-03 23:34 ` Brendan Higgins 2018-12-03 23:46 ` mcgrof 2018-12-03 23:46 ` Luis Chamberlain 2018-12-04 0:44 ` brendanhiggins 2018-12-04 0:44 ` Brendan Higgins 2018-11-30 3:41 ` mcgrof 2018-11-30 3:41 ` Luis Chamberlain 2018-12-03 23:37 ` brendanhiggins 2018-12-03 23:37 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 09/19] kunit: test: add the concept of assertions brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 10/19] kunit: test: add test managed resource tests brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 11/19] kunit: add Python libraries for handing KUnit config and kernel brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-29 13:54 ` kieran.bingham 2018-11-29 13:54 ` Kieran Bingham 2018-12-03 23:48 ` brendanhiggins 2018-12-03 23:48 ` Brendan Higgins 2018-12-04 20:47 ` mcgrof 2018-12-04 20:47 ` Luis Chamberlain 2018-12-06 12:32 ` kieran.bingham [this message] 2018-12-06 12:32 ` Kieran Bingham 2018-12-06 15:37 ` willy 2018-12-06 15:37 ` Matthew Wilcox 2018-12-07 11:30 ` kieran.bingham 2018-12-07 11:30 ` Kieran Bingham 2018-12-11 14:09 ` pmladek 2018-12-11 14:09 ` Petr Mladek 2018-12-11 14:41 ` rostedt 2018-12-11 14:41 ` Steven Rostedt 2018-12-11 17:01 ` anton.ivanov 2018-12-11 17:01 ` Anton Ivanov 2019-02-09 0:40 ` brendanhiggins 2019-02-09 0:40 ` Brendan Higgins 2018-12-07 1:05 ` mcgrof 2018-12-07 1:05 ` Luis Chamberlain 2018-12-07 18:35 ` kent.overstreet 2018-12-07 18:35 ` Kent Overstreet 2018-11-30 3:44 ` mcgrof 2018-11-30 3:44 ` Luis Chamberlain 2018-12-03 23:50 ` brendanhiggins 2018-12-03 23:50 ` Brendan Higgins 2018-12-04 20:48 ` mcgrof 2018-12-04 20:48 ` Luis Chamberlain 2018-11-28 19:36 ` [RFC v3 12/19] kunit: add KUnit wrapper script and simple output parser brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 13/19] kunit: improve output from python wrapper brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 14/19] Documentation: kunit: add documentation for KUnit brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-29 13:56 ` kieran.bingham 2018-11-29 13:56 ` Kieran Bingham 2018-11-30 3:45 ` mcgrof 2018-11-30 3:45 ` Luis Chamberlain 2018-12-03 23:53 ` brendanhiggins 2018-12-03 23:53 ` Brendan Higgins 2018-12-06 12:16 ` kieran.bingham 2018-12-06 12:16 ` Kieran Bingham 2019-02-09 0:56 ` brendanhiggins 2019-02-09 0:56 ` Brendan Higgins 2019-02-11 12:16 ` kieran.bingham 2019-02-11 12:16 ` Kieran Bingham 2019-02-12 22:10 ` brendanhiggins 2019-02-12 22:10 ` Brendan Higgins 2019-02-13 21:55 ` kieran.bingham 2019-02-13 21:55 ` Kieran Bingham 2019-02-14 0:17 ` brendanhiggins 2019-02-14 0:17 ` Brendan Higgins 2019-02-14 17:26 ` mcgrof 2019-02-14 17:26 ` Luis Chamberlain 2019-02-14 22:07 ` brendanhiggins 2019-02-14 22:07 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 15/19] MAINTAINERS: add entry for KUnit the unit testing framework brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 16/19] arch: um: make UML unflatten device tree when testing brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 21:16 ` robh 2018-11-28 21:16 ` Rob Herring 2018-12-04 0:00 ` brendanhiggins 2018-12-04 0:00 ` Brendan Higgins 2018-11-30 3:46 ` mcgrof 2018-11-30 3:46 ` Luis Chamberlain 2018-12-04 0:02 ` brendanhiggins 2018-12-04 0:02 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 17/19] of: unittest: migrate tests to run on KUnit brendanhiggins 2018-11-28 19:36 ` Brendan Higgins [not found] ` <CAL_Jsq+09Kx7yMBC_Jw45QGmk6U_fp4N6HOZDwYrM4tWw+_dOA@mail.gmail.com> 2018-11-30 0:39 ` rdunlap 2018-11-30 0:39 ` Randy Dunlap 2018-12-04 0:13 ` brendanhiggins 2018-12-04 0:13 ` Brendan Higgins 2018-12-04 13:40 ` robh 2018-12-04 13:40 ` Rob Herring 2018-12-05 23:42 ` brendanhiggins 2018-12-05 23:42 ` Brendan Higgins 2018-12-07 0:41 ` robh 2018-12-07 0:41 ` Rob Herring 2018-12-04 0:08 ` brendanhiggins 2018-12-04 0:08 ` Brendan Higgins 2019-02-13 1:44 ` brendanhiggins 2019-02-13 1:44 ` Brendan Higgins 2019-02-14 20:10 ` robh 2019-02-14 20:10 ` Rob Herring 2019-02-14 21:52 ` brendanhiggins 2019-02-14 21:52 ` Brendan Higgins 2019-02-18 22:56 ` frowand.list 2019-02-18 22:56 ` Frank Rowand 2019-02-28 0:29 ` brendanhiggins 2019-02-28 0:29 ` Brendan Higgins 2018-12-04 10:56 ` frowand.list 2018-12-04 10:56 ` Frank Rowand 2018-11-28 19:36 ` [RFC v3 18/19] of: unittest: split out a couple of test cases from unittest brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-12-04 10:58 ` frowand.list 2018-12-04 10:58 ` Frank Rowand 2018-12-05 23:54 ` brendanhiggins 2018-12-05 23:54 ` Brendan Higgins 2019-02-14 23:57 ` frowand.list 2019-02-14 23:57 ` Frank Rowand 2019-02-15 0:56 ` brendanhiggins 2019-02-15 0:56 ` Brendan Higgins 2019-02-15 2:05 ` frowand.list 2019-02-15 2:05 ` Frank Rowand 2019-02-15 10:56 ` brendanhiggins 2019-02-15 10:56 ` Brendan Higgins 2019-02-18 22:25 ` frowand.list 2019-02-18 22:25 ` Frank Rowand 2019-02-20 20:44 ` frowand.list 2019-02-20 20:44 ` Frank Rowand 2019-02-20 20:47 ` frowand.list 2019-02-20 20:47 ` Frank Rowand 2019-02-28 3:52 ` brendanhiggins 2019-02-28 3:52 ` Brendan Higgins 2019-03-22 0:22 ` frowand.list 2019-03-22 0:22 ` Frank Rowand 2019-03-22 1:30 ` brendanhiggins 2019-03-22 1:30 ` Brendan Higgins 2019-03-22 1:47 ` frowand.list 2019-03-22 1:47 ` Frank Rowand 2019-03-25 22:15 ` brendanhiggins 2019-03-25 22:15 ` Brendan Higgins 2019-09-20 16:57 ` Rob Herring 2019-09-21 23:57 ` Frank Rowand 2019-03-22 1:34 ` frowand.list 2019-03-22 1:34 ` Frank Rowand 2019-03-25 22:18 ` brendanhiggins 2019-03-25 22:18 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 19/19] of: unittest: split up some super large test cases brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-12-04 10:52 ` [RFC v3 00/19] kunit: introduce KUnit, the Linux kernel unit testing framework frowand.list 2018-12-04 10:52 ` Frank Rowand 2018-12-04 11:40 ` frowand.list 2018-12-04 11:40 ` Frank Rowand 2018-12-04 13:49 ` robh 2018-12-04 13:49 ` Rob Herring 2018-12-05 23:10 ` brendanhiggins 2018-12-05 23:10 ` Brendan Higgins 2019-03-22 0:27 ` frowand.list 2019-03-22 0:27 ` Frank Rowand 2019-03-25 22:04 ` brendanhiggins 2019-03-25 22:04 ` Brendan Higgins
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=f8722f4a-44c8-24c2-c433-5178f9f40b82@ideasonboard.com \ --to=linux-kselftest@vger.kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).