Using kernelci.org for testing Linux NFSD

Using kernelci.org for testing Linux NFSD

[Chuck Lever III]

Hi-

As co-maintainers of NFSD, we're interested in bringing up a
CI platform for automating a lot of the tests we do before
requesting a pull into the Linux kernel. The kernelci.org
platform caught my eye last week during the talk at LPC.

We have some questions about what can be supported on
kernelci.org. For example, the tests we need to run involve
at least two hosts (an NFS client and an NFS server). How
easy is it to set up that kind of test?

If we want to test features that involve Kerberos, is there
a KDC available, or would we need to provide a KDC image
that can be used for our tests?

Thanks for any guidance!


--
Chuck Lever

Re: Using kernelci.org for testing Linux NFSD

[Gustavo Padovan]

Hi Chuck,

Thanks for your message and interest in KernelCI.

On 10/1/21 16:01, Chuck Lever III wrote:
> Hi-
>
> As co-maintainers of NFSD, we're interested in bringing up a
> CI platform for automating a lot of the tests we do before
> requesting a pull into the Linux kernel. The kernelci.org
> platform caught my eye last week during the talk at LPC.

Nice!

>
> We have some questions about what can be supported on
> kernelci.org. For example, the tests we need to run involve
> at least two hosts (an NFS client and an NFS server). How
> easy is it to set up that kind of test?

We don't have support right now for testing across two different hosts. 
Can you share more the details of what you want to achieve? Do you have 
some scripts or tests code somewhere that you can share with us? eg. are 
both the client and server subject of the tests at the same time or do 
you test changes one side against a fixed version of the other side 
(when it is possible).

Also, is there any kind of simple/smoke tests that we can run with only 
one host (eg as part of kselftests or LTP)? That helps us enables the 
most basic stuff more easily, as the above will need some design and 
discussion between the KernelCI team.

>
> If we want to test features that involve Kerberos, is there
> a KDC available, or would we need to provide a KDC image
> that can be used for our tests?

We don't have any provision for that yet (because no one ever needed 
it!). It might be easier if you just provide an image that serves you 
the best. I'd need to check with the team the details to make it 
available to your tests.

Best Regards,

Gustavo

Re: Using kernelci.org for testing Linux NFSD

[Bruce Fields]

On Tue, Oct 5, 2021 at 9:18 AM Gustavo Padovan
<gustavo.padovan@collabora.com> wrote:
> On 10/1/21 16:01, Chuck Lever III wrote:
> > We have some questions about what can be supported on
> > kernelci.org. For example, the tests we need to run involve
> > at least two hosts (an NFS client and an NFS server). How
> > easy is it to set up that kind of test?
>
> We don't have support right now for testing across two different hosts.
> Can you share more the details of what you want to achieve?

Personally for basic regression testing I use two VMs, one configured
as an NFS client and one as an NFS server, and I have bash scripts
that ssh into the client and do things like mount the server with
various options and run xfstests.

> Do you have
> some scripts or tests code somewhere that you can share with us?

I don't know if my thicket of test scripts will make much sense to
anyone but me, but for what it's worth, I keep them all here:

  http://git.linux-nfs.org/?p=bfields/testd.git;a=tree;hb=HEAD

The thing I usually run before submitting is in bin/do-all.

> eg. are
> both the client and server subject of the tests at the same time or do
> you test changes one side against a fixed version of the other side
> (when it is possible).

I normally boot both client and server to the kernel under test.
However it's useful to have the option to vary the kernel on just one
side or the other, as that occasionally helps pin down which side is
at fault for a regression.

> Also, is there any kind of simple/smoke tests that we can run with only
> one host (eg as part of kselftests or LTP)?

I like using two hosts:  It's a more realistic setup.  When there's a
crash or other problem, it makes it more obvious which side was at
fault.  Etc.

But, sure, as a basic smoke test, we could do something like (untested
for now, off the top of my head):

  git clone git://git.linux-nfs.org/projects/steved/cthon04.git
  cd cthon04
  make
  mkdir /export
  chmod a+rwx /export
  echo "/export *(lo)" >/etc/exports
  systemctl start nfs-server
  mount localhost:/export /mnt/
  NFSTESTDIR=/mnt/TMP ./runtests -a -f

That cthon04 test hardly ever changes, so it really only needs to be
downloaded once. Better would probably be to put /export on a separate
filesystem.  And more tests would be better, but cthon04 is something
that will basically always pass unless something is extremely wrong.

> That helps us enables the
> most basic stuff more easily, as the above will need some design and
> discussion between the KernelCI team.

Understood.

Thanks for your interest.

--b.

> > If we want to test features that involve Kerberos, is there
> > a KDC available, or would we need to provide a KDC image
> > that can be used for our tests?
>
> We don't have any provision for that yet (because no one ever needed
> it!). It might be easier if you just provide an image that serves you
> the best. I'd need to check with the team the details to make it
> available to your tests.

Re: Using kernelci.org for testing Linux NFSD

[Chuck Lever III]

Hello Gustavo, good to meet you. More inline.


> On Oct 5, 2021, at 9:47 AM, Bruce Fields <bfields@redhat.com> wrote:
> 
> On Tue, Oct 5, 2021 at 9:18 AM Gustavo Padovan
> <gustavo.padovan@collabora.com> wrote:
>> On 10/1/21 16:01, Chuck Lever III wrote:
>>> We have some questions about what can be supported on
>>> kernelci.org. For example, the tests we need to run involve
>>> at least two hosts (an NFS client and an NFS server). How
>>> easy is it to set up that kind of test?
>> 
>> We don't have support right now for testing across two different hosts.

Other network filesystems and block storage subsystems might
also be interested in a two-host arrangement, fwiw.


>> Can you share more the details of what you want to achieve?
> 
> Personally for basic regression testing I use two VMs, one configured
> as an NFS client and one as an NFS server, and I have bash scripts
> that ssh into the client and do things like mount the server with
> various options and run xfstests.
> 
>> Do you have
>> some scripts or tests code somewhere that you can share with us?
> 
> I don't know if my thicket of test scripts will make much sense to
> anyone but me, but for what it's worth, I keep them all here:
> 
>  http://git.linux-nfs.org/?p=bfields/testd.git;a=tree;hb=HEAD
> 
> The thing I usually run before submitting is in bin/do-all.
> 
>> eg. are
>> both the client and server subject of the tests at the same time or do
>> you test changes one side against a fixed version of the other side
>> (when it is possible).
> 
> I normally boot both client and server to the kernel under test.
> However it's useful to have the option to vary the kernel on just one
> side or the other, as that occasionally helps pin down which side is
> at fault for a regression.

I agree, two hosts is best. A loopback NFS mount could be used,
but that is generally a fraught and uncommon configuration. For
one thing, the memory needs of the client and server can often
interact, and sometimes that results in system deadlock.


>> Also, is there any kind of simple/smoke tests that we can run with only
>> one host (eg as part of kselftests or LTP)?
> 
> I like using two hosts:  It's a more realistic setup.  When there's a
> crash or other problem, it makes it more obvious which side was at
> fault.  Etc.
> 
> But, sure, as a basic smoke test, we could do something like (untested
> for now, off the top of my head):
> 
>  git clone git://git.linux-nfs.org/projects/steved/cthon04.git
>  cd cthon04
>  make
>  mkdir /export
>  chmod a+rwx /export
>  echo "/export *(lo)" >/etc/exports
>  systemctl start nfs-server
>  mount localhost:/export /mnt/
>  NFSTESTDIR=/mnt/TMP ./runtests -a -f
> 
> That cthon04 test hardly ever changes, so it really only needs to be
> downloaded once. Better would probably be to put /export on a separate
> filesystem.  And more tests would be better, but cthon04 is something
> that will basically always pass unless something is extremely wrong.

cthon04 is simple to set up, and hits enough of the basic NFS
functionality that it is a commonly used smoke test.

Typically you want to exercise all versions of NFS (2, 3, 4.0,
4.1, and 4.2), which requires a separate cthon run for each
NFS version. 

We'd like to get some code coverage analysis of any tests that
are run, and that should be simple for you to enable from the
start?

Single-host testing is not useful for us outside of the usual
source code analysis tools (does it build on all platforms, are
there new warnings or build problems, and so on). But that stuff
is already covered by 0-day bots.


>> That helps us enables the
>> most basic stuff more easily, as the above will need some design and
>> discussion between the KernelCI team.
> 
> Understood.
> 
> Thanks for your interest.
> 
> --b.
> 
>>> If we want to test features that involve Kerberos, is there
>>> a KDC available, or would we need to provide a KDC image
>>> that can be used for our tests?
>> 
>> We don't have any provision for that yet (because no one ever needed
>> it!). It might be easier if you just provide an image that serves you
>> the best. I'd need to check with the team the details to make it
>> available to your tests.

Kerberos can be a stretch goal. Certainly not something we
require on day one.

--
Chuck Lever

Re: Using kernelci.org for testing Linux NFSD

[Bjorn Andersson]

On Fri 01 Oct 12:01 PDT 2021, Chuck Lever III wrote:

> Hi-
> 

Hi Chuck,

> As co-maintainers of NFSD, we're interested in bringing up a
> CI platform for automating a lot of the tests we do before
> requesting a pull into the Linux kernel. The kernelci.org
> platform caught my eye last week during the talk at LPC.
> 
> We have some questions about what can be supported on
> kernelci.org. For example, the tests we need to run involve
> at least two hosts (an NFS client and an NFS server). How
> easy is it to set up that kind of test?
> 
> If we want to test features that involve Kerberos, is there
> a KDC available, or would we need to provide a KDC image
> that can be used for our tests?
> 
> Thanks for any guidance!
> 

In my view, kernelci.org provides two things:

1) A platform that provide build artifacts (i.e kernel images) of
various trees and collects test results from labs booting these builds,
running tests and reporting back results.

2) A set of labs, with pretty much standardized way of running these
tests on artifacts coming from the builder and reporting the results
back.


Given that your testing setup and test cases do differ quite a bit from
what's today seem to happen in the existing labs ("boot a kernel on one
board, run common tests, gather results") my suggestion is that you
automate the process done by these labs on your end and use the common
infrastructure.

I.e. build something that takes a kernel image, boots this on your
machines and runs your test and generate test results in a structured
form.

Then you can plug this in with kernelci (#1 above) using the available
REST API, to get the kernel images and to report the results.


This is what I do for the Qualcomm boards in my lab, where I'm not able
to run the standard lab setup (not sure if that even was a thing when I
started) and it allows me to run more board specific tests - with all
the results reported back to a single place.

Regards,
Bjorn

Re: Using kernelci.org for testing Linux NFSD

[Chuck Lever III]

> On Oct 5, 2021, at 10:33 AM, Chuck Lever III <chuck.lever@oracle.com> wrote:
> 
> Hello Gustavo, good to meet you. More inline.
> 
> 
>> On Oct 5, 2021, at 9:47 AM, Bruce Fields <bfields@redhat.com> wrote:
>> 
>> On Tue, Oct 5, 2021 at 9:18 AM Gustavo Padovan
>> <gustavo.padovan@collabora.com> wrote:
>>> On 10/1/21 16:01, Chuck Lever III wrote:
>>>> We have some questions about what can be supported on
>>>> kernelci.org. For example, the tests we need to run involve
>>>> at least two hosts (an NFS client and an NFS server). How
>>>> easy is it to set up that kind of test?
>>> 
>>> We don't have support right now for testing across two different hosts.
> 
> Other network filesystems and block storage subsystems might
> also be interested in a two-host arrangement, fwiw.
> 
> 
>>> Can you share more the details of what you want to achieve?
>> 
>> Personally for basic regression testing I use two VMs, one configured
>> as an NFS client and one as an NFS server, and I have bash scripts
>> that ssh into the client and do things like mount the server with
>> various options and run xfstests.
>> 
>>> Do you have
>>> some scripts or tests code somewhere that you can share with us?
>> 
>> I don't know if my thicket of test scripts will make much sense to
>> anyone but me, but for what it's worth, I keep them all here:
>> 
>> http://git.linux-nfs.org/?p=bfields/testd.git;a=tree;hb=HEAD
>> 
>> The thing I usually run before submitting is in bin/do-all.
>> 
>>> eg. are
>>> both the client and server subject of the tests at the same time or do
>>> you test changes one side against a fixed version of the other side
>>> (when it is possible).
>> 
>> I normally boot both client and server to the kernel under test.
>> However it's useful to have the option to vary the kernel on just one
>> side or the other, as that occasionally helps pin down which side is
>> at fault for a regression.
> 
> I agree, two hosts is best. A loopback NFS mount could be used,
> but that is generally a fraught and uncommon configuration. For
> one thing, the memory needs of the client and server can often
> interact, and sometimes that results in system deadlock.
> 
> 
>>> Also, is there any kind of simple/smoke tests that we can run with only
>>> one host (eg as part of kselftests or LTP)?
>> 
>> I like using two hosts:  It's a more realistic setup.  When there's a
>> crash or other problem, it makes it more obvious which side was at
>> fault.  Etc.
>> 
>> But, sure, as a basic smoke test, we could do something like (untested
>> for now, off the top of my head):
>> 
>> git clone git://git.linux-nfs.org/projects/steved/cthon04.git
>> cd cthon04
>> make
>> mkdir /export
>> chmod a+rwx /export
>> echo "/export *(lo)" >/etc/exports
>> systemctl start nfs-server
>> mount localhost:/export /mnt/
>> NFSTESTDIR=/mnt/TMP ./runtests -a -f
>> 
>> That cthon04 test hardly ever changes, so it really only needs to be
>> downloaded once. Better would probably be to put /export on a separate
>> filesystem.  And more tests would be better, but cthon04 is something
>> that will basically always pass unless something is extremely wrong.
> 
> cthon04 is simple to set up, and hits enough of the basic NFS
> functionality that it is a commonly used smoke test.
> 
> Typically you want to exercise all versions of NFS (2, 3, 4.0,
> 4.1, and 4.2), which requires a separate cthon run for each
> NFS version. 
> 
> We'd like to get some code coverage analysis of any tests that
> are run, and that should be simple for you to enable from the
> start?
> 
> Single-host testing is not useful for us outside of the usual
> source code analysis tools (does it build on all platforms, are
> there new warnings or build problems, and so on). But that stuff
> is already covered by 0-day bots.

A member of the Linux NFS community built some Jenkins infrastructure
several years ago to provide continuous integration testing.

http://git.linux-nfs.org/?p=anna/nfs-ordeal.git;a=summary

The README describes what's in the repository. It includes the
aforementioned cthon04 and also xfstests.

It looks like one host is booted, and a pair of guests on that
host act as the systems under test. The host drives the testing
by ssh'ing into the client guest.

I hope this is helpful to explain what we're looking for.


>>> That helps us enables the
>>> most basic stuff more easily, as the above will need some design and
>>> discussion between the KernelCI team.
>> 
>> Understood.
>> 
>> Thanks for your interest.
>> 
>> --b.
>> 
>>>> If we want to test features that involve Kerberos, is there
>>>> a KDC available, or would we need to provide a KDC image
>>>> that can be used for our tests?
>>> 
>>> We don't have any provision for that yet (because no one ever needed
>>> it!). It might be easier if you just provide an image that serves you
>>> the best. I'd need to check with the team the details to make it
>>> available to your tests.
> 
> Kerberos can be a stretch goal. Certainly not something we
> require on day one.
> 
> --
> Chuck Lever

--
Chuck Lever