the Linux kernel, testsuites, and maybe *you*

the Linux kernel, testsuites, and maybe *you*

[Mike Frysinger]

is there any sort of standard for testing and integration into
mainline ?  in the Blackfin world, we've been developing little
external kernel modules and adding them to our own testsuite, but
often times these things are not Blackfin specific.  case in point,
we're integrating a string testsuite to make sure all of the fun str*
and mem* functions are sane and operate as they expected, but rather
than having just Blackfin benefit here, i'd like to see this pushed
upstream ...

i'm fully aware of LTP (as i work on it), but i feel that serves a
great purpose for exercising the API/ABI exposed to userspace either
directly through the kernel or indirectly through the system libc ...
it isnt a very good tool for testing kernel internals, especially as
the kernel changes and evolves.

is there a framework already in place i'm not aware of ?  should there
be ?  should this all live in LTP ?  i wouldnt mind an option under
kernel hacking "Enable testsuites" ... as far as i can tell, the
testing process is really based extensively on feedback from people,
nothing really automated.
-mike

Re: the Linux kernel, testsuites, and maybe *you*

[Robin Getz]

On Fri 31 Aug 2007 17:22, Mike Frysinger pondered:
> is there any sort of standard for testing and integration into
> mainline ?  in the Blackfin world, we've been developing little
> external kernel modules and adding them to our own testsuite, but
> often times these things are not Blackfin specific.  case in point,
> we're integrating a string testsuite to make sure all of the fun str*
> and mem* functions are sane and operate as they expected, but rather
> than having just Blackfin benefit here, i'd like to see this pushed
> upstream ...

I know there have been some discussions at past OLS about some testing that 
Martin was working on.

http://www.linuxsymposium.org/2006/view_abstract.php?content_key=13

But I think this was more functional tests, less unit tests that what you are 
talking/asking about.

http://test.kernel.org/functional/index.html

Also - If I remember - most of the existing tests were for a self hosted 
environment - and might not be well suited to embedded (like Blackfin) which 
requires cross compile, and pretty thin runtime environment (uClibc + 
busybox's msh as shell).

Martin?

-Robin

Re: the Linux kernel, testsuites, and maybe *you*

[Andi Kleen]

"Mike Frysinger" <vapier.adi@gmail.com> writes:

> is there any sort of standard for testing and integration into
> mainline ?  

Everybody does their own.

> in the Blackfin world, we've been developing little
> external kernel modules and adding them to our own testsuite, but
> often times these things are not Blackfin specific.  case in point,
> we're integrating a string testsuite to make sure all of the fun str*
> and mem* functions are sane and operate as they expected, but rather
> than having just Blackfin benefit here, i'd like to see this pushed
> upstream ...

I would like to see this too. I wrote a couple of unit tests
during x86-64 development too and they would be handy
to have in a central place.

The SUSE kernel also has a crasher module that exercises the
allocators and is pretty useful to stress code in general.

Disadvantage is that test code tends to be hackish and ugly
and very specialized and will probably not pass standard review procedures.

Also the rt people seem to have pushed a couple of tests in already, 
but I always hated it that they're in the standard directories.
RCU also has, in fact they added a "eat all my CPU in tests"
CONFIG option. Just making that dependent on a CONFIG_UNIT_TESTS
would be a good change in itself. And then there are the slab
failure inducers of course.

BTW string functions are best tested in user space. That's
a relatively bad example.

-Andi

Re: the Linux kernel, testsuites, and maybe *you*

[Mike Frysinger]

On 02 Sep 2007 00:08:57 +0200, Andi Kleen <andi@firstfloor.org> wrote:
> BTW string functions are best tested in user space. That's
> a relatively bad example.

in theory, maybe ... in reality, i really dont think so

the string implementations are spread out over the kernel ... there's
implementations in lib/, include/asm-*/, and arch/*/lib/ ... so any
test code that would use these sources is going to be an ugly hack to
make sure it grabs all the right pieces from all the right places.

then there's the issue of API ... compiling code that is designed to
be in the kernel and nowhere else is going to require a mighty bit of
crappy glue logic in order for it to compile properly in userspace

but let's cheat and just use the kernel build system to produce .o's
for us ... then there's the issue of ABI.  there is no guarantee the
ABI the kernel uses internally is the same as the ABI userspace uses
(and in fact, on some architectures it is simply not possible).  so
you're left with a solution that may work for some, but not for
others.

or we put one common piece of C test code right into the kernel and be
done.  no worrying about the tested API/ABI/environment being
different from the actual runtime the kernel itself has.
-mike

Re: the Linux kernel, testsuites, and maybe *you*

[Mike Frysinger]

On 9/1/07, Robin Getz <rgetz@blackfin.uclinux.org> wrote:
> On Fri 31 Aug 2007 17:22, Mike Frysinger pondered:
> > is there any sort of standard for testing and integration into
> > mainline ?  in the Blackfin world, we've been developing little
> > external kernel modules and adding them to our own testsuite, but
> > often times these things are not Blackfin specific.  case in point,
> > we're integrating a string testsuite to make sure all of the fun str*
> > and mem* functions are sane and operate as they expected, but rather
> > than having just Blackfin benefit here, i'd like to see this pushed
> > upstream ...
>
> I know there have been some discussions at past OLS about some testing that
> Martin was working on.
>
> http://www.linuxsymposium.org/2006/view_abstract.php?content_key=13
>
> But I think this was more functional tests, less unit tests that what you are
> talking/asking about.
>
> http://test.kernel.org/functional/index.html
>
> Also - If I remember - most of the existing tests were for a self hosted
> environment - and might not be well suited to embedded (like Blackfin) which
> requires cross compile, and pretty thin runtime environment (uClibc +
> busybox's msh as shell).

i flipped through the autotest code a bit but i dont think this can do
what i'm looking for.  it appears to be a mix of LTP provides and what
we have for our Blackfin test infrastructure -- iow, all userspace
stuff and no real way of exercising kernel internals.
-mike

Re: the Linux kernel, testsuites, and maybe *you*

[Bill Davidsen]

Mike Frysinger wrote:
> is there any sort of standard for testing and integration into
> mainline ?  in the Blackfin world, we've been developing little
> external kernel modules and adding them to our own testsuite, but
> often times these things are not Blackfin specific.  case in point,
> we're integrating a string testsuite to make sure all of the fun str*
> and mem* functions are sane and operate as they expected, but rather
> than having just Blackfin benefit here, i'd like to see this pushed
> upstream ...
> 
> i'm fully aware of LTP (as i work on it), but i feel that serves a
> great purpose for exercising the API/ABI exposed to userspace either
> directly through the kernel or indirectly through the system libc ...
> it isnt a very good tool for testing kernel internals, especially as
> the kernel changes and evolves.
> 
> is there a framework already in place i'm not aware of ?  should there
> be ?  should this all live in LTP ?  i wouldnt mind an option under
> kernel hacking "Enable testsuites" ... as far as i can tell, the
> testing process is really based extensively on feedback from people,
> nothing really automated.
>
If you want to test that stuff and run it on the current code in the 
kernel, how about a kernel module? You could "modprobe sanitytest" or 
something and report to syslog at module load time. And maybe have a 
parameter which does something drastic if something core is so hosed 
that filesystem damage is likely. Or just optional init code run by a 
kernel option, perhaps sanity testing after boot is not a great idea.

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: the Linux kernel, testsuites, and maybe *you*

[Mike Frysinger]

On 9/1/07, Bill Davidsen <davidsen@tmr.com> wrote:
> If you want to test that stuff and run it on the current code in the
> kernel, how about a kernel module? You could "modprobe sanitytest" or
> something and report to syslog at module load time. And maybe have a
> parameter which does something drastic if something core is so hosed
> that filesystem damage is likely. Or just optional init code run by a
> kernel option, perhaps sanity testing after boot is not a great idea.

i'm fully capable of doing my own testing right now; that isnt the
point of this thread ... the point is for everyone to benefit, not
just the Blackfin architecture
-mike

Re: the Linux kernel, testsuites, and maybe *you*

[Andi Kleen]

On Sat, Sep 01, 2007 at 06:50:30PM -0400, Mike Frysinger wrote:
> On 02 Sep 2007 00:08:57 +0200, Andi Kleen <andi@firstfloor.org> wrote:
> > BTW string functions are best tested in user space. That's
> > a relatively bad example.
> 
> in theory, maybe ... in reality, i really dont think so
> 
> the string implementations are spread out over the kernel ... there's
> implementations in lib/, include/asm-*/, and arch/*/lib/ ... so any
> test code that would use these sources is going to be an ugly hack to
> make sure it grabs all the right pieces from all the right places.

string functions tend to be self contained.

The other issue to test some of them properly you need unmapped pages
etc. That gets much easier to do in user space. There are some other
issues.

-Andi

Re: the Linux kernel, testsuites, and maybe *you*

[Mike Frysinger]

On 9/2/07, Andi Kleen <andi@firstfloor.org> wrote:
> On Sat, Sep 01, 2007 at 06:50:30PM -0400, Mike Frysinger wrote:
> > On 02 Sep 2007 00:08:57 +0200, Andi Kleen <andi@firstfloor.org> wrote:
> > > BTW string functions are best tested in user space. That's
> > > a relatively bad example.
> >
> > in theory, maybe ... in reality, i really dont think so
> >
> > the string implementations are spread out over the kernel ... there's
> > implementations in lib/, include/asm-*/, and arch/*/lib/ ... so any
> > test code that would use these sources is going to be an ugly hack to
> > make sure it grabs all the right pieces from all the right places.
>
> string functions tend to be self contained.

"tend to be" does not lend itself to being cleanly tested ... and i
imagine i'd get some pretty heavy resistance if i proposed
re-organzing code just so that i could compile it outside of the
kernel

there is still the ABI issue ... code written in kernel space in pure
asm cannot always be compiled in userspace and work properly/the same

> The other issue to test some of them properly you need unmapped pages
> etc. That gets much easier to do in user space. There are some other
> issues.

you mean testing boundary overflows ?  can be handled with canaries
rather than segfaults i imagine ...
-mike

Re: the Linux kernel, testsuites, and maybe *you*

[Andi Kleen]

On Sun, Sep 02, 2007 at 11:15:23AM -0400, Mike Frysinger wrote:
> there is still the ABI issue ... code written in kernel space in pure
> asm cannot always be compiled in userspace and work properly/the same

Is that a blackfin weirdness? 

> 
> > The other issue to test some of them properly you need unmapped pages
> > etc. That gets much easier to do in user space. There are some other
> > issues.
> 
> you mean testing boundary overflows ?  can be handled with canaries
> rather than segfaults i imagine ...

Not for reads, no.

-Andi

Re: the Linux kernel, testsuites, and maybe *you*

[Håvard Skinnemoen]

On 9/2/07, Andi Kleen <andi@firstfloor.org> wrote:
> The other issue to test some of them properly you need unmapped pages
> etc. That gets much easier to do in user space. There are some other
> issues.

vmalloc, vmap, etc. always put a guard page after the allocation. So
if you test string operations on vmalloc()'ed memory, you have
unmapped pages in both ends.

I like the idea of a test suite for kernel internals, but I also think
it would be best to run it in kernel mode since things like atomics,
etc. may work differently in user space on some architectures.

Haavard

Re: the Linux kernel, testsuites, and maybe *you*

[Bill Davidsen]

Mike Frysinger wrote:
> On 9/1/07, Bill Davidsen <davidsen@tmr.com> wrote:
>   
>> If you want to test that stuff and run it on the current code in the
>> kernel, how about a kernel module? You could "modprobe sanitytest" or
>> something and report to syslog at module load time. And maybe have a
>> parameter which does something drastic if something core is so hosed
>> that filesystem damage is likely. Or just optional init code run by a
>> kernel option, perhaps sanity testing after boot is not a great idea.
>>     
>
> i'm fully capable of doing my own testing right now; that isnt the
> point of this thread ... the point is for everyone to benefit, not
> just the Blackfin architecture
>   

That's exactly the point I was making, if your tests and any others 
people feel are useful were in a module and/or could be run as init code 
only when wanted, everyone could benefit from the tests. Anything 
related to build sanity could be in one place, to guard against 
surprises like new versions of gcc, typos in patches, etc.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: the Linux kernel, testsuites, and maybe *you*

[Robin Getz]

On Sat 1 Sep 2007 18:08, Andi Kleen pondered:
> "Mike Frysinger" <vapier.adi@gmail.com> writes:
> 
> > is there any sort of standard for testing and integration into
> > mainline ?  
> 
> Everybody does their own.

That kind of stinks - and seems to be a potential duplication of effort all 
over the place.

> > in the Blackfin world, we've been developing little
> > external kernel modules and adding them to our own testsuite, but
> > often times these things are not Blackfin specific.  case in point,
> > we're integrating a string testsuite to make sure all of the fun str*
> > and mem* functions are sane and operate as they expected, but rather
> > than having just Blackfin benefit here, i'd like to see this pushed
> > upstream ...
> 
> I would like to see this too. I wrote a couple of unit tests
> during x86-64 development too and they would be handy
> to have in a central place.
> 
> The SUSE kernel also has a crasher module that exercises the
> allocators and is pretty useful to stress code in general.
> 
> Disadvantage is that test code tends to be hackish and ugly
> and very specialized and will probably not pass standard review
> procedures.

Then this makes perfect sense to actually have it in the proper places, and 
have it go through the standard review process. I have run into problems 
recently, where ugly hackish tests seem never really catch actual faults. 
(Since they were written by the same person who wrote the driver - Design a 
fault in, code the fault, test the fault is there).

With many eyes - and all that stuff - hopefully we would catch things.

> Also the rt people seem to have pushed a couple of tests in already, 
> but I always hated it that they're in the standard directories.
> RCU also has, in fact they added a "eat all my CPU in tests"
> CONFIG option. Just making that dependent on a CONFIG_UNIT_TESTS
> would be a good change in itself. And then there are the slab
> failure inducers of course.

Putting them close to the standard directories might be OK - keeping the tests 
close to the thing that is being tested may help with maintainers keeping an 
eye on things that might be missed otherwise.

Maybe these types of tests should go into a separate directory - power on self 
tests (post) directories in the respective places? ./kernel/post 
and ./drivers/xxxx/post?

So Mike's proposed string tests would go into ./lib/post/strings.c

When things pass - you get a single indication (all tests for unit X passed) 
or a pass for each test?

-Robin

Re: the Linux kernel, testsuites, and maybe *you*

[Mike Frysinger]

On 9/2/07, Andi Kleen <andi@firstfloor.org> wrote:
> On Sun, Sep 02, 2007 at 11:15:23AM -0400, Mike Frysinger wrote:
> > there is still the ABI issue ... code written in kernel space in pure
> > asm cannot always be compiled in userspace and work properly/the same
>
> Is that a blackfin weirdness?

yes, Blackfin is weird in this respect due to being a no-mmu and
wanting ELF, but i was being sufficiently general to bolster my claims
and to leave the door open for any other port out there i'm not
familiar with and any port that comes along

> > > The other issue to test some of them properly you need unmapped pages
> > > etc. That gets much easier to do in user space. There are some other
> > > issues.
> >
> > you mean testing boundary overflows ?  can be handled with canaries
> > rather than segfaults i imagine ...
>
> Not for reads, no.

true, but generally the string functions arent searching for crashes
in the boundary conditions ... they just happen to sometimes find them
:)
-mike

Re: the Linux kernel, testsuites, and maybe *you*

[Mike Frysinger]

On 9/4/07, Mike Frysinger <vapier.adi@gmail.com> wrote:
> On 9/2/07, Andi Kleen <andi@firstfloor.org> wrote:
> > On Sun, Sep 02, 2007 at 11:15:23AM -0400, Mike Frysinger wrote:
> > > there is still the ABI issue ... code written in kernel space in pure
> > > asm cannot always be compiled in userspace and work properly/the same
> >
> > Is that a blackfin weirdness?
>
> yes, Blackfin is weird in this respect due to being a no-mmu and
> wanting ELF, but i was being sufficiently general to bolster my claims
> and to leave the door open for any other port out there i'm not
> familiar with and any port that comes along

actually, there was another case i was thinking of ... it isnt
uncommon for architectures to use compiler flags which affect the ABI
and thus make userland not a 1-to-1 test example for the kernel ...
consider the regparam stuff we use on i386
-mike