From: kieran.bingham at ideasonboard.com (Kieran Bingham) Subject: [RFC v3 11/19] kunit: add Python libraries for handing KUnit config and kernel Date: Fri, 7 Dec 2018 11:30:30 +0000 [thread overview] Message-ID: <f2a3d7fc-21f2-7bd0-94e2-4122f2073da5@ideasonboard.com> (raw) In-Reply-To: <20181206153718.GD24603@bombadil.infradead.org> Hi Matthew, On 06/12/2018 15:37, Matthew Wilcox wrote: > On Thu, Dec 06, 2018 at 12:32:47PM +0000, Kieran Bingham wrote: >> On 04/12/2018 20:47, Luis Chamberlain wrote: >>> On Mon, Dec 03, 2018 at 03:48:15PM -0800, Brendan Higgins wrote: >>>> On Thu, Nov 29, 2018 at 5:54 AM Kieran Bingham >>>> <kieran.bingham at ideasonboard.com> wrote: >>>>> >>>>> Hi Brendan, >>>>> >>>>> Thanks again for this series! >>>>> >>>>> On 28/11/2018 19:36, Brendan Higgins wrote: >>>>>> The ultimate goal is to create minimal isolated test binaries; in the >>>>>> meantime we are using UML to provide the infrastructure to run tests, so >>>>>> define an abstract way to configure and run tests that allow us to >>>>>> change the context in which tests are built without affecting the user. >>>>>> This also makes pretty and dynamic error reporting, and a lot of other >>>>>> nice features easier. >>>>> >>>>> >>>>> I wonder if we could somehow generate a shared library object >>>>> 'libkernel' or 'libumlinux' from a UM configured set of headers and >>>>> objects so that we could create binary targets directly ? >>>> >>>> That's an interesting idea. I think it would be difficult to figure >>>> out exactly where to draw the line of what goes in there and what >>>> needs to be built specific to a test a priori. Of course, that leads >>>> into the biggest problem in general, needed to know what I need to >>>> build to test the thing that I want to test. >>>> >>>> Nevertheless, I could definitely imagine that being useful in a lot of cases. >>> >>> Whether or not we can abstract away the kernel into such a mechanism >>> with uml libraries is a good question worth exploring. >>> >>> Developers working upstream do modify their kernels a lot, so we'd have >>> to update such libraries quite a bit, but I think that's fine too. The >>> *real* value I think from the above suggestion would be enterprise / >>> mobile distros or stable kernel maintainers which have a static kernel >>> they need to support for a relatively *long time*, consider a 10 year >>> time frame. Running unit tests without qemu with uml and libraries for >>> respective kernels seems real worthy. >> >> I think any such library might be something generated by the kernel >> build system, so if someone makes substantial changes to a core >> component provided by the library - it can be up to them to build a >> corresponding userspace library as well. >> >> We could also consider to only provide *static* libraries rather than >> dynamic. So any one building some userspace tool / test with this would >> be required to compile against (the version of) the kernel they expect >> perhaps... - much like we expect modules to be compiled currently. >> >> And then the userspace binary would be sufficiently able to live it's >> life on it's own :) >> >>> The overhead for testing a unit test for said targets, *ideally*, would >>> just be to to reboot into the system with such libraries available, a >>> unit test would just look for the respective uname -r library and mimic >>> that kernel, much the same way enterprise distributions today rely on >>> having debugging symbols available to run against crash / gdb. Having >>> debug modules / kernel for crash requires such effort already, so this >>> would just be an extra layer of other prospect tests. >> >> Oh - although, yes - there are some good concepts there - but I'm a bit >> weary of how easy it would be to 'run' the said test against multiple >> kernel version libraries... there would be a lot of possible ABI >> conflicts perhaps. >> >> My main initial idea for a libumlinux is to provide infrastructure such >> as our linked-lists and other kernel formatting so that we can take >> kernel code directly to userspace for test and debug (assuming that >> there are no hardware dependencies or things that we can't mock out) >> >> I think all of this could complement kunit of course - this isn't >> suggesting an alternative implementation :-) > > I suspect the reason Luis cc'd me on this is that we already have some > artisinally-crafted userspace kernel-mocking interfaces under tools/. Aha - excellent - I had hoped to grab you at Plumbers to talk about this, after hearing you mention something at your Xarray talk - but didn't seem to find a suitable time. > The tools/testing/radix-tree directory is the source of some of this, > but I've been moving pieces out into tools/ more generally where it > makes sense to. Sounds like we already have a starting point then. > We have liburcu already, which is good. The main sticking points are: > > - No emulation of kernel thread interfaces Scheduling finesse aside, This shouldn't be too hard to emulate/wrap with pthreads? > - The kernel does not provide the ability to aggressively fail memory > allocations (which is useful when trying to exercise the memory failure > paths). Fault injection throughout would certainly be a valuable addition to any unit-testing. Wrapping tests into a single userspace binary could facilitate further memory leak checking or other valgrind facilities too. > - printk has started adding a lot of %pX enhancements which printf > obviously doesn't know about. Wrapping through User-mode linux essentially provides this already though. In fact I guess that goes for the thread interfaces topic above too. > - No global pseudo-random number generator in the kernel. Probably > we should steal the i915 one. > > I know Dan Williams has also done a lot of working mocking kernel > interfaces for libnvdimm. Thanks for the references - more to investigate. -- Regards -- Kieran
WARNING: multiple messages have this Message-ID (diff)
From: kieran.bingham@ideasonboard.com (Kieran Bingham) Subject: [RFC v3 11/19] kunit: add Python libraries for handing KUnit config and kernel Date: Fri, 7 Dec 2018 11:30:30 +0000 [thread overview] Message-ID: <f2a3d7fc-21f2-7bd0-94e2-4122f2073da5@ideasonboard.com> (raw) Message-ID: <20181207113030.XrHM85gme2owxpfk6yFe35bnLX2iDm4FkKN5EK5vS0I@z> (raw) In-Reply-To: <20181206153718.GD24603@bombadil.infradead.org> Hi Matthew, On 06/12/2018 15:37, Matthew Wilcox wrote: > On Thu, Dec 06, 2018@12:32:47PM +0000, Kieran Bingham wrote: >> On 04/12/2018 20:47, Luis Chamberlain wrote: >>> On Mon, Dec 03, 2018@03:48:15PM -0800, Brendan Higgins wrote: >>>> On Thu, Nov 29, 2018 at 5:54 AM Kieran Bingham >>>> <kieran.bingham@ideasonboard.com> wrote: >>>>> >>>>> Hi Brendan, >>>>> >>>>> Thanks again for this series! >>>>> >>>>> On 28/11/2018 19:36, Brendan Higgins wrote: >>>>>> The ultimate goal is to create minimal isolated test binaries; in the >>>>>> meantime we are using UML to provide the infrastructure to run tests, so >>>>>> define an abstract way to configure and run tests that allow us to >>>>>> change the context in which tests are built without affecting the user. >>>>>> This also makes pretty and dynamic error reporting, and a lot of other >>>>>> nice features easier. >>>>> >>>>> >>>>> I wonder if we could somehow generate a shared library object >>>>> 'libkernel' or 'libumlinux' from a UM configured set of headers and >>>>> objects so that we could create binary targets directly ? >>>> >>>> That's an interesting idea. I think it would be difficult to figure >>>> out exactly where to draw the line of what goes in there and what >>>> needs to be built specific to a test a priori. Of course, that leads >>>> into the biggest problem in general, needed to know what I need to >>>> build to test the thing that I want to test. >>>> >>>> Nevertheless, I could definitely imagine that being useful in a lot of cases. >>> >>> Whether or not we can abstract away the kernel into such a mechanism >>> with uml libraries is a good question worth exploring. >>> >>> Developers working upstream do modify their kernels a lot, so we'd have >>> to update such libraries quite a bit, but I think that's fine too. The >>> *real* value I think from the above suggestion would be enterprise / >>> mobile distros or stable kernel maintainers which have a static kernel >>> they need to support for a relatively *long time*, consider a 10 year >>> time frame. Running unit tests without qemu with uml and libraries for >>> respective kernels seems real worthy. >> >> I think any such library might be something generated by the kernel >> build system, so if someone makes substantial changes to a core >> component provided by the library - it can be up to them to build a >> corresponding userspace library as well. >> >> We could also consider to only provide *static* libraries rather than >> dynamic. So any one building some userspace tool / test with this would >> be required to compile against (the version of) the kernel they expect >> perhaps... - much like we expect modules to be compiled currently. >> >> And then the userspace binary would be sufficiently able to live it's >> life on it's own :) >> >>> The overhead for testing a unit test for said targets, *ideally*, would >>> just be to to reboot into the system with such libraries available, a >>> unit test would just look for the respective uname -r library and mimic >>> that kernel, much the same way enterprise distributions today rely on >>> having debugging symbols available to run against crash / gdb. Having >>> debug modules / kernel for crash requires such effort already, so this >>> would just be an extra layer of other prospect tests. >> >> Oh - although, yes - there are some good concepts there - but I'm a bit >> weary of how easy it would be to 'run' the said test against multiple >> kernel version libraries... there would be a lot of possible ABI >> conflicts perhaps. >> >> My main initial idea for a libumlinux is to provide infrastructure such >> as our linked-lists and other kernel formatting so that we can take >> kernel code directly to userspace for test and debug (assuming that >> there are no hardware dependencies or things that we can't mock out) >> >> I think all of this could complement kunit of course - this isn't >> suggesting an alternative implementation :-) > > I suspect the reason Luis cc'd me on this is that we already have some > artisinally-crafted userspace kernel-mocking interfaces under tools/. Aha - excellent - I had hoped to grab you at Plumbers to talk about this, after hearing you mention something at your Xarray talk - but didn't seem to find a suitable time. > The tools/testing/radix-tree directory is the source of some of this, > but I've been moving pieces out into tools/ more generally where it > makes sense to. Sounds like we already have a starting point then. > We have liburcu already, which is good. The main sticking points are: > > - No emulation of kernel thread interfaces Scheduling finesse aside, This shouldn't be too hard to emulate/wrap with pthreads? > - The kernel does not provide the ability to aggressively fail memory > allocations (which is useful when trying to exercise the memory failure > paths). Fault injection throughout would certainly be a valuable addition to any unit-testing. Wrapping tests into a single userspace binary could facilitate further memory leak checking or other valgrind facilities too. > - printk has started adding a lot of %pX enhancements which printf > obviously doesn't know about. Wrapping through User-mode linux essentially provides this already though. In fact I guess that goes for the thread interfaces topic above too. > - No global pseudo-random number generator in the kernel. Probably > we should steal the i915 one. > > I know Dan Williams has also done a lot of working mocking kernel > interfaces for libnvdimm. Thanks for the references - more to investigate. -- Regards -- Kieran
next prev parent reply other threads:[~2018-12-07 11:30 UTC|newest] Thread overview: 232+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-11-28 19:36 [RFC v3 00/19] kunit: introduce KUnit, the Linux kernel unit testing framework brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 01/19] kunit: test: add KUnit test runner core brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-30 3:14 ` mcgrof 2018-11-30 3:14 ` Luis Chamberlain 2018-12-01 1:51 ` brendanhiggins 2018-12-01 1:51 ` Brendan Higgins 2018-12-01 2:57 ` mcgrof 2018-12-01 2:57 ` Luis Chamberlain 2018-12-05 13:15 ` anton.ivanov 2018-12-05 13:15 ` Anton Ivanov 2018-12-05 14:45 ` arnd 2018-12-05 14:45 ` Arnd Bergmann 2018-12-05 14:49 ` anton.ivanov 2018-12-05 14:49 ` Anton Ivanov 2018-11-30 3:28 ` mcgrof 2018-11-30 3:28 ` Luis Chamberlain 2018-12-01 2:08 ` brendanhiggins 2018-12-01 2:08 ` Brendan Higgins 2018-12-01 3:10 ` mcgrof 2018-12-01 3:10 ` Luis Chamberlain 2018-12-03 22:47 ` brendanhiggins 2018-12-03 22:47 ` Brendan Higgins 2018-12-01 3:02 ` mcgrof 2018-12-01 3:02 ` Luis Chamberlain 2018-11-28 19:36 ` [RFC v3 02/19] kunit: test: add test resource management API brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 03/19] kunit: test: add string_stream a std::stream like string builder brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-30 3:29 ` mcgrof 2018-11-30 3:29 ` Luis Chamberlain 2018-12-01 2:14 ` brendanhiggins 2018-12-01 2:14 ` Brendan Higgins 2018-12-01 3:12 ` mcgrof 2018-12-01 3:12 ` Luis Chamberlain 2018-12-03 10:55 ` pmladek 2018-12-03 10:55 ` Petr Mladek 2018-12-04 0:35 ` brendanhiggins 2018-12-04 0:35 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 04/19] kunit: test: add test_stream a std::stream like logger brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 05/19] kunit: test: add the concept of expectations brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 06/19] arch: um: enable running kunit from User Mode Linux brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 21:26 ` robh 2018-11-28 21:26 ` Rob Herring 2018-11-30 3:37 ` mcgrof 2018-11-30 3:37 ` Luis Chamberlain 2018-11-30 14:05 ` robh 2018-11-30 14:05 ` Rob Herring 2018-11-30 18:22 ` mcgrof 2018-11-30 18:22 ` Luis Chamberlain 2018-12-03 23:22 ` brendanhiggins 2018-12-03 23:22 ` Brendan Higgins 2018-11-30 3:30 ` mcgrof 2018-11-30 3:30 ` Luis Chamberlain 2018-11-28 19:36 ` [RFC v3 07/19] kunit: test: add initial tests brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-30 3:40 ` mcgrof 2018-11-30 3:40 ` Luis Chamberlain 2018-12-03 23:26 ` brendanhiggins 2018-12-03 23:26 ` Brendan Higgins 2018-12-03 23:43 ` mcgrof 2018-12-03 23:43 ` Luis Chamberlain 2018-11-28 19:36 ` [RFC v3 08/19] arch: um: add shim to trap to allow installing a fault catcher for tests brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-30 3:34 ` mcgrof 2018-11-30 3:34 ` Luis Chamberlain 2018-12-03 23:34 ` brendanhiggins 2018-12-03 23:34 ` Brendan Higgins 2018-12-03 23:46 ` mcgrof 2018-12-03 23:46 ` Luis Chamberlain 2018-12-04 0:44 ` brendanhiggins 2018-12-04 0:44 ` Brendan Higgins 2018-11-30 3:41 ` mcgrof 2018-11-30 3:41 ` Luis Chamberlain 2018-12-03 23:37 ` brendanhiggins 2018-12-03 23:37 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 09/19] kunit: test: add the concept of assertions brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 10/19] kunit: test: add test managed resource tests brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 11/19] kunit: add Python libraries for handing KUnit config and kernel brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-29 13:54 ` kieran.bingham 2018-11-29 13:54 ` Kieran Bingham 2018-12-03 23:48 ` brendanhiggins 2018-12-03 23:48 ` Brendan Higgins 2018-12-04 20:47 ` mcgrof 2018-12-04 20:47 ` Luis Chamberlain 2018-12-06 12:32 ` kieran.bingham 2018-12-06 12:32 ` Kieran Bingham 2018-12-06 15:37 ` willy 2018-12-06 15:37 ` Matthew Wilcox 2018-12-07 11:30 ` kieran.bingham [this message] 2018-12-07 11:30 ` Kieran Bingham 2018-12-11 14:09 ` pmladek 2018-12-11 14:09 ` Petr Mladek 2018-12-11 14:41 ` rostedt 2018-12-11 14:41 ` Steven Rostedt 2018-12-11 17:01 ` anton.ivanov 2018-12-11 17:01 ` Anton Ivanov 2019-02-09 0:40 ` brendanhiggins 2019-02-09 0:40 ` Brendan Higgins 2018-12-07 1:05 ` mcgrof 2018-12-07 1:05 ` Luis Chamberlain 2018-12-07 18:35 ` kent.overstreet 2018-12-07 18:35 ` Kent Overstreet 2018-11-30 3:44 ` mcgrof 2018-11-30 3:44 ` Luis Chamberlain 2018-12-03 23:50 ` brendanhiggins 2018-12-03 23:50 ` Brendan Higgins 2018-12-04 20:48 ` mcgrof 2018-12-04 20:48 ` Luis Chamberlain 2018-11-28 19:36 ` [RFC v3 12/19] kunit: add KUnit wrapper script and simple output parser brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 13/19] kunit: improve output from python wrapper brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 14/19] Documentation: kunit: add documentation for KUnit brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-29 13:56 ` kieran.bingham 2018-11-29 13:56 ` Kieran Bingham 2018-11-30 3:45 ` mcgrof 2018-11-30 3:45 ` Luis Chamberlain 2018-12-03 23:53 ` brendanhiggins 2018-12-03 23:53 ` Brendan Higgins 2018-12-06 12:16 ` kieran.bingham 2018-12-06 12:16 ` Kieran Bingham 2019-02-09 0:56 ` brendanhiggins 2019-02-09 0:56 ` Brendan Higgins 2019-02-11 12:16 ` kieran.bingham 2019-02-11 12:16 ` Kieran Bingham 2019-02-12 22:10 ` brendanhiggins 2019-02-12 22:10 ` Brendan Higgins 2019-02-13 21:55 ` kieran.bingham 2019-02-13 21:55 ` Kieran Bingham 2019-02-14 0:17 ` brendanhiggins 2019-02-14 0:17 ` Brendan Higgins 2019-02-14 17:26 ` mcgrof 2019-02-14 17:26 ` Luis Chamberlain 2019-02-14 22:07 ` brendanhiggins 2019-02-14 22:07 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 15/19] MAINTAINERS: add entry for KUnit the unit testing framework brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 16/19] arch: um: make UML unflatten device tree when testing brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-11-28 21:16 ` robh 2018-11-28 21:16 ` Rob Herring 2018-12-04 0:00 ` brendanhiggins 2018-12-04 0:00 ` Brendan Higgins 2018-11-30 3:46 ` mcgrof 2018-11-30 3:46 ` Luis Chamberlain 2018-12-04 0:02 ` brendanhiggins 2018-12-04 0:02 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 17/19] of: unittest: migrate tests to run on KUnit brendanhiggins 2018-11-28 19:36 ` Brendan Higgins [not found] ` <CAL_Jsq+09Kx7yMBC_Jw45QGmk6U_fp4N6HOZDwYrM4tWw+_dOA@mail.gmail.com> 2018-11-30 0:39 ` rdunlap 2018-11-30 0:39 ` Randy Dunlap 2018-12-04 0:13 ` brendanhiggins 2018-12-04 0:13 ` Brendan Higgins 2018-12-04 13:40 ` robh 2018-12-04 13:40 ` Rob Herring 2018-12-05 23:42 ` brendanhiggins 2018-12-05 23:42 ` Brendan Higgins 2018-12-07 0:41 ` robh 2018-12-07 0:41 ` Rob Herring 2018-12-04 0:08 ` brendanhiggins 2018-12-04 0:08 ` Brendan Higgins 2019-02-13 1:44 ` brendanhiggins 2019-02-13 1:44 ` Brendan Higgins 2019-02-14 20:10 ` robh 2019-02-14 20:10 ` Rob Herring 2019-02-14 21:52 ` brendanhiggins 2019-02-14 21:52 ` Brendan Higgins 2019-02-18 22:56 ` frowand.list 2019-02-18 22:56 ` Frank Rowand 2019-02-28 0:29 ` brendanhiggins 2019-02-28 0:29 ` Brendan Higgins 2018-12-04 10:56 ` frowand.list 2018-12-04 10:56 ` Frank Rowand 2018-11-28 19:36 ` [RFC v3 18/19] of: unittest: split out a couple of test cases from unittest brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-12-04 10:58 ` frowand.list 2018-12-04 10:58 ` Frank Rowand 2018-12-05 23:54 ` brendanhiggins 2018-12-05 23:54 ` Brendan Higgins 2019-02-14 23:57 ` frowand.list 2019-02-14 23:57 ` Frank Rowand 2019-02-15 0:56 ` brendanhiggins 2019-02-15 0:56 ` Brendan Higgins 2019-02-15 2:05 ` frowand.list 2019-02-15 2:05 ` Frank Rowand 2019-02-15 10:56 ` brendanhiggins 2019-02-15 10:56 ` Brendan Higgins 2019-02-18 22:25 ` frowand.list 2019-02-18 22:25 ` Frank Rowand 2019-02-20 20:44 ` frowand.list 2019-02-20 20:44 ` Frank Rowand 2019-02-20 20:47 ` frowand.list 2019-02-20 20:47 ` Frank Rowand 2019-02-28 3:52 ` brendanhiggins 2019-02-28 3:52 ` Brendan Higgins 2019-03-22 0:22 ` frowand.list 2019-03-22 0:22 ` Frank Rowand 2019-03-22 1:30 ` brendanhiggins 2019-03-22 1:30 ` Brendan Higgins 2019-03-22 1:47 ` frowand.list 2019-03-22 1:47 ` Frank Rowand 2019-03-25 22:15 ` brendanhiggins 2019-03-25 22:15 ` Brendan Higgins 2019-09-20 16:57 ` Rob Herring 2019-09-21 23:57 ` Frank Rowand 2019-03-22 1:34 ` frowand.list 2019-03-22 1:34 ` Frank Rowand 2019-03-25 22:18 ` brendanhiggins 2019-03-25 22:18 ` Brendan Higgins 2018-11-28 19:36 ` [RFC v3 19/19] of: unittest: split up some super large test cases brendanhiggins 2018-11-28 19:36 ` Brendan Higgins 2018-12-04 10:52 ` [RFC v3 00/19] kunit: introduce KUnit, the Linux kernel unit testing framework frowand.list 2018-12-04 10:52 ` Frank Rowand 2018-12-04 11:40 ` frowand.list 2018-12-04 11:40 ` Frank Rowand 2018-12-04 13:49 ` robh 2018-12-04 13:49 ` Rob Herring 2018-12-05 23:10 ` brendanhiggins 2018-12-05 23:10 ` Brendan Higgins 2019-03-22 0:27 ` frowand.list 2019-03-22 0:27 ` Frank Rowand 2019-03-25 22:04 ` brendanhiggins 2019-03-25 22:04 ` Brendan Higgins
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=f2a3d7fc-21f2-7bd0-94e2-4122f2073da5@ideasonboard.com \ --to=linux-kselftest@vger.kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).