linux-kselftest.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: shuah <shuah@kernel.org>
To: Luis Chamberlain <mcgrof@kernel.org>, Knut Omang <knut.omang@oracle.com>
Cc: Brendan Higgins <brendanhiggins@google.com>,
	kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org,
	Alan Maguire <alan.maguire@oracle.com>, shuah <shuah@kernel.org>
Subject: Re: Plan for hybrid testing
Date: Fri, 18 Oct 2019 15:42:46 -0600	[thread overview]
Message-ID: <b4f4fc02-46c3-337a-474b-aac7521d139d@kernel.org> (raw)
In-Reply-To: <20191016130857.GJ16384@42.do-not-panic.com>

On 10/16/19 7:08 AM, Luis Chamberlain wrote:
> On Wed, Oct 16, 2019 at 12:52:12PM +0200, Knut Omang wrote:
>> On Mon, 2019-10-14 at 13:01 -0600, shuah wrote:
>>> On 10/14/19 12:38 PM, Knut Omang wrote:
>>>> On Mon, 2019-10-14 at 10:42 +0000, Luis Chamberlain wrote:
>>>>> On Fri, Sep 13, 2019 at 02:02:47PM -0700, Brendan Higgins wrote:
>>>>>> Hey Knut and Shuah,
>>>>>>
>>>>>> Following up on our offline discussion on Wednesday night:
>>>>>>
>>>>>> We decided that it would make sense for Knut to try to implement Hybrid
>>>>>> Testing (testing that crosses the kernel userspace boundary) that he
>>>>>> introduced here[1] on top of the existing KUnit infrastructure.
>>>>>>
>>>>>> We discussed several possible things in the kernel that Knut could test
>>>>>> with the new Hybrid Testing feature as an initial example. Those were
>>>>>> (in reverse order of expected difficulty):
>>>>>>
>>>>>> 1. RDS (Reliable Datagram Sockets) - We decided that, although this was
>>>>>>      one of the more complicated subsystems to work with, it was probably
>>>>>>      the best candidate for Knut to start with because it was in desperate
>>>>>>      need of better testing, much of the testing would require crossing
>>>>>>      the kernel userspace boundary to be effective, and Knut has access to
>>>>>>      RDS (since he works at Oracle).
>>>>>>
>>>
>>> Any update on if you are able to explore this work.
>>
>> I am working on this, but it's going to take some time, as this ties in
>> with internal projects at Oracle. Basing work on RDS or RDS related tests
>> (such as generic socket etc) is the best option for us, since that allows
>> progress on our internal deliverables as well ;-)
>>
>>>>>> 2. KMOD - Probably much simpler than RDS, and the maintainer, Luis
>>>>>>      Chamberlain (CC'ed) would like to see better testing here, but
>>>>>>      probably still not as good as RDS because it is in less dire need of
>>>>>>      testing, collaboration on this would be more difficult, and Luis is
>>>>>>      currently on an extended vacation. Luis and I had already been
>>>>>>      discussing testing KMOD here[2].
>>>>>
>>>>> I'm back!
>>>>>
>>>>> I'm also happy and thrilled to help review the infrastructure in great
>>>>> detail given I have lofty future objectives with testing in the kernel.
>>>>> Also, kmod is a bit more complex to test, if Knut wants a simpler *easy*
>>>>> target I think test_sysctl.c would be a good target. I think the goal
>>>>> there would be to add probes for a few of the sysctl callers, and then
>>>>> test them through userspace somehow, for instance?
>>>>
>>>> That sounds like a good case for the hybrid tests.
>>>> The challenge in a kunit setting would be that it relies on a significant part of KTF
>>>> to work as we have used it so far:
>>>>
>>>> - module support - Alan has been working on this
>>>
>>> I see the patches. Thanks for working on this.
>>>
>>>> - netlink approach from KTF (to allow user space execution of kernel
>>>>     part of test, and gathering reporting in one place)
>>>> - probe infrastructure
>>>>
>>>>> The complexities with testing kmod is the threading aspect. So that is
>>>>> more of a challenge for a test infrastructure as a whole. However kmod
>>>>> also already has a pretty sound kthread solution which could be used
>>>>> as basis for any sound kernel multithread test solution.
>>>>>
>>>>> Curious, what was decided with the regards to the generic netlink approach?
>>>
>>> Can this work be be done without netlink approach? At least some of it.
>>> I would like to see some patches and would like to get a better feel
>>> for the dependency on generic netlink.
>>
>> A flexible out-of-band communication channel is needed for several of the
>> features, and definitely for hybrid tests. It does not need to be netlink
>> in principle, but that has served the purpose well so far in KTF, and
>> reimplementing something will be at the cost of the core task of getting more
>> and better tests, which after all is the goal of this effort.
> 
> I don't think you did justice to *why* netlink would be good, but in
> principle I suspect its the right thing long term if we want a nice
> interface to decide what to test and how.
> 
> So kselftest today implicates you know what you want to test. Or rather
> this is defined through what you enable and then you run *all* enabled
> kselftests.
> 

Right. Kselftests are static in nature and that is intended. There is
some level of flexibility to select tests using TARGETS and a newly
SKIP_TARGETS allows flexibility to skip. It is inline with it being
the developer tests. The rest of the nice wrapper stuff is left for
users and CIs to figure out.

> Similar logic follwos for kunit.
> 
> Yes, you can have scripts and your own tests infrastructure that knows
> what to test or avoid, but this is not nice for autonegotiation.
> Consider also the complexity of dealing with testing new tests on older
> kenrels. Today we address this on kselftests by striving to make sure
> the old scripts / tests work or yield for old kernel who don't have
> a feature. What if we want to really figure out what is there or not
> concretely? A generic netlink interface could easily allow for these
> sorts of things to grow and be auto negotiated.
> 
> Then, collection of results: we have each kselftest desiging its own
> scatter/gather set of way to go and do what it can to test what it can
> and expose what should be tested, or in what order, or to allow knobs.
> A generic netlink interface allows a standard interface to be sketched
> up.
>  > I don't think the generic netlink interface implemented in the KTF
> patches accomplished any of this, but it at least moved the needle
> forward IMHO in terms of what we should consider long term.
> 
>> It would be really good if more people had a closer look at the KTF patch
>> set before we embark on significant work of porting it to Kunit.
>>
>> For reference, the netlink code in KTF:
>> https://lkml.org/lkml/2019/8/13/92
> 
> I read it, and it seems like a lot of stuff implemented but without
> a lot of proper justification and also without a lot of community
> feedback. Kind of like: too much done in house without proper feedback.
> Then it became bloated and once kunit was almost about to be merged
> you guys knee-jerk reacted and wanted to merge your stuff too...
> 
> And the quality of code. Gosh, much to be said about that...
> 
> So I think that asking to consolidate with kunit is the right thing
> at this point. Because we need to grow it in *community*. kunit itself
> has been heavily modifed to adjust to early feedback, for example, so
> its an example of evolution needed.
> 
>> Note that unlike kunit, in KTF no tests are executed by default,
>> instead KTF provides an API to query about, set up and trigger execution of
>> tests and test parts in the kernel, and leave the actual initiation
>> to user space tools.
> 
> This is all excellent. However baby steps. Let's demo it with a few
> simple tests, rather than trying to ensure it works with *all* the
> stuff you guys probably already have in house. That will probably have
> to be phased out in the future with whatever we grow *together*.
> 

I am in favor of adding features that would make it easier for testing
the kernel. As I said in another response to this thread, it would be
nice to start simple with one or two tests and go from there.

thanks,
-- Shuah

      parent reply	other threads:[~2019-10-18 21:42 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-13 21:02 Plan for hybrid testing Brendan Higgins
2019-09-16 16:20 ` shuah
2019-10-14 10:42 ` Luis Chamberlain
2019-10-14 18:38   ` Knut Omang
2019-10-14 19:01     ` shuah
2019-10-16 10:52       ` Knut Omang
2019-10-16 13:08         ` Luis Chamberlain
2019-10-17 17:46           ` Knut Omang
2019-10-17 19:11             ` shuah
2019-10-18  9:47             ` Luis Chamberlain
2019-10-18 18:35               ` Brendan Higgins
2019-10-18 19:22                 ` Luis Chamberlain
2019-10-18 19:58                   ` Brendan Higgins
2019-10-19 18:44                     ` Luis Chamberlain
2019-10-18 21:42           ` shuah [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=b4f4fc02-46c3-337a-474b-aac7521d139d@kernel.org \
    --to=shuah@kernel.org \
    --cc=alan.maguire@oracle.com \
    --cc=brendanhiggins@google.com \
    --cc=knut.omang@oracle.com \
    --cc=kunit-dev@googlegroups.com \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=mcgrof@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).