Re: Plan for hybrid testing

[Knut Omang <knut.omang@oracle.com>]

On Mon, 2019-10-14 at 10:42 +0000, Luis Chamberlain wrote:
> On Fri, Sep 13, 2019 at 02:02:47PM -0700, Brendan Higgins wrote:
> > Hey Knut and Shuah,
> > 
> > Following up on our offline discussion on Wednesday night:
> > 
> > We decided that it would make sense for Knut to try to implement Hybrid
> > Testing (testing that crosses the kernel userspace boundary) that he
> > introduced here[1] on top of the existing KUnit infrastructure.
> > 
> > We discussed several possible things in the kernel that Knut could test
> > with the new Hybrid Testing feature as an initial example. Those were
> > (in reverse order of expected difficulty):
> > 
> > 1. RDS (Reliable Datagram Sockets) - We decided that, although this was
> >    one of the more complicated subsystems to work with, it was probably
> >    the best candidate for Knut to start with because it was in desperate
> >    need of better testing, much of the testing would require crossing
> >    the kernel userspace boundary to be effective, and Knut has access to
> >    RDS (since he works at Oracle).
> > 
> > 2. KMOD - Probably much simpler than RDS, and the maintainer, Luis
> >    Chamberlain (CC'ed) would like to see better testing here, but
> >    probably still not as good as RDS because it is in less dire need of
> >    testing, collaboration on this would be more difficult, and Luis is
> >    currently on an extended vacation. Luis and I had already been
> >    discussing testing KMOD here[2].
> 
> I'm back!
> 
> I'm also happy and thrilled to help review the infrastructure in great
> detail given I have lofty future objectives with testing in the kernel.
> Also, kmod is a bit more complex to test, if Knut wants a simpler *easy*
> target I think test_sysctl.c would be a good target. I think the goal
> there would be to add probes for a few of the sysctl callers, and then
> test them through userspace somehow, for instance?

That sounds like a good case for the hybrid tests.
The challenge in a kunit setting would be that it relies on a significant part of KTF
to work as we have used it so far:

- module support - Alan has been working on this 
- netlink approach from KTF (to allow user space execution of kernel 
  part of test, and gathering reporting in one place)
- probe infrastructure 

> The complexities with testing kmod is the threading aspect. So that is
> more of a challenge for a test infrastructure as a whole. However kmod
> also already has a pretty sound kthread solution which could be used
> as basis for any sound kernel multithread test solution.
> 
> Curious, what was decided with the regards to the generic netlink approach?

I think in some way functionality similar to the netlink support is needed 
for the features in KTF that we discussed, so I get it is a "yes" to add 
support for it?

Knut

>   Luis