[Ksummit-discuss] [CORE TOPIC] Testing

[Ksummit-discuss] [CORE TOPIC] Testing

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1456 bytes --]

One thing we typically cover at Kernel Summit is some of the activity
that's going on around testing upstream.  I think it'd be useful to have
some more of those discussions, both in terms of making people aware of
what's available and in terms of helping the people doing testing figure
out what would be useful.  A lot of this is probably well suited to a
workshop session between the interested people but I do think some
element in the core day beyond just a readout will be useful.

The main things I'm aware of that are happening at the minute are
kselftest development, the 0day tester, plus kernelci.org and the other
build and boot/test bots that are running against various trees.

In terms of discussion topics some of the issues I'm seeing are:

 - Can we pool resources to share the workload of running things and
   interpreting results, ideally also providing some central way for
   people to discover what results are out there for them to look at
   for a given kernel in the different systems?

 - Should we start carrying config fragments upstream designed to
   support testing, things like the distro config fragments that keep
   getting discussed are one example here but there's other things like
   collections of debug options we could be looking at.  Should we be
   more generally slimming defconfigs and moving things into fragments?

and there's always the the perennial ones about what people would like
to see testing for.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Alexey Dobriyan]

On Tue, Jul 7, 2015 at 12:24 PM, Mark Brown <broonie@kernel.org> wrote:

>  - Should we start carrying config fragments upstream designed to
>    support testing, things like the distro config fragments that keep
>    getting discussed are one example here but there's other things like
>    collections of debug options we could be looking at.

This will gravitate everyone to running the same config which is the opposite
of what you want.

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 887 bytes --]

On Tue, Jul 07, 2015 at 04:02:13PM +0300, Alexey Dobriyan wrote:
> On Tue, Jul 7, 2015 at 12:24 PM, Mark Brown <broonie@kernel.org> wrote:

> >  - Should we start carrying config fragments upstream designed to
> >    support testing, things like the distro config fragments that keep
> >    getting discussed are one example here but there's other things like
> >    collections of debug options we could be looking at.

> This will gravitate everyone to running the same config which is the opposite
> of what you want.

Perhaps, perhaps not - it's not an unequivocal thing either way.  The
more barriers there are to enabling things the more likely it is that
people just won't bother in the first place (or that they'll run into
somme problem and give up before they get things working) and it's not
clear that having to figure these things out is always a good use of
people's time.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Guenter Roeck]

On 07/07/2015 02:24 AM, Mark Brown wrote:
> One thing we typically cover at Kernel Summit is some of the activity
> that's going on around testing upstream.  I think it'd be useful to have
> some more of those discussions, both in terms of making people aware of
> what's available and in terms of helping the people doing testing figure
> out what would be useful.  A lot of this is probably well suited to a
> workshop session between the interested people but I do think some
> element in the core day beyond just a readout will be useful.
>
> The main things I'm aware of that are happening at the minute are
> kselftest development, the 0day tester, plus kernelci.org and the other
> build and boot/test bots that are running against various trees.
>
Maybe list all known ones as a start ?

> In terms of discussion topics some of the issues I'm seeing are:
>
>   - Can we pool resources to share the workload of running things and
>     interpreting results, ideally also providing some central way for
>     people to discover what results are out there for them to look at
>     for a given kernel in the different systems?
>
That might be quite useful. However, I have seen that it doesn't really
help to just provide the test results. kissb test results have been
available for ages, and people just don't look at it. Even the regular
"Build regression" e-mails sent out by Geert seem to be widely ignored.

What I really found to help is to bisect new problems and send an e-mail
to the responsible maintainer and to the submitter of the patch which
introduced it. I'd like to automate that with my test system, but
unfortunately I just don't have the time to do it.

>   - Should we start carrying config fragments upstream designed to
>     support testing, things like the distro config fragments that keep
>     getting discussed are one example here but there's other things like
>     collections of debug options we could be looking at.  Should we be
>     more generally slimming defconfigs and moving things into fragments?
>
> and there's always the the perennial ones about what people would like
> to see testing for.
>

Sharing as many test bot configuration scripts and relevant configurations
as possible would be quite helpful. For example, I am building various
configurations for all architectures, but I don't really know if they
are relevant. Also, I would like to run more qemu configurations,
but it is really hard to find working ones.

Guenter

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 3283 bytes --]

On Tue, Jul 07, 2015 at 08:25:21AM -0700, Guenter Roeck wrote:
> On 07/07/2015 02:24 AM, Mark Brown wrote:

> >The main things I'm aware of that are happening at the minute are
> >kselftest development, the 0day tester, plus kernelci.org and the other
> >build and boot/test bots that are running against various trees.

> Maybe list all known ones as a start ?

Off the top of my head the automated ones I'm aware of are Olof's build
& boot test, Dan running smatch and I think some other static analysis
stuff, someone (not sure who?) running some coccinelle stuff, Coverity
and I've got a builder too.

> >In terms of discussion topics some of the issues I'm seeing are:

> >  - Can we pool resources to share the workload of running things and
> >    interpreting results, ideally also providing some central way for
> >    people to discover what results are out there for them to look at
> >    for a given kernel in the different systems?

> That might be quite useful. However, I have seen that it doesn't really
> help to just provide the test results. kissb test results have been
> available for ages, and people just don't look at it. Even the regular
> "Build regression" e-mails sent out by Geert seem to be widely ignored.

> What I really found to help is to bisect new problems and send an e-mail
> to the responsible maintainer and to the submitter of the patch which
> introduced it. I'd like to automate that with my test system, but
> unfortunately I just don't have the time to do it.

Yes, that's the "and interpreting" bit in the above - this only really
works with people actively pushing.  You do start to get people checking
themselves once things are perceived as something people care about but
it does take active work to establish and maintain that.  

It also really helps if things are delivered promptly, and against trees
people are actively developing for.  But even with clear reports and
sometimes patches not everyone shows an interest.  As we get more and
more actual testing running that's going to start to become more
serious, breaking the build or boot will also mean that automated tests
don't get to run.

This is one of the things 0day gets really right, when it kicks in it'll
e-mail people directly and promptly.

> >  - Should we start carrying config fragments upstream designed to
> >    support testing, things like the distro config fragments that keep
> >    getting discussed are one example here but there's other things like
> >    collections of debug options we could be looking at.  Should we be
> >    more generally slimming defconfigs and moving things into fragments?

> >and there's always the the perennial ones about what people would like
> >to see testing for.

> Sharing as many test bot configuration scripts and relevant configurations
> as possible would be quite helpful. For example, I am building various
> configurations for all architectures, but I don't really know if they
> are relevant. Also, I would like to run more qemu configurations,
> but it is really hard to find working ones.

Grant (just CCed) was working intermittently on the qemu bit.  I think
the last plan was to enhance the scripts Kevin has for driving his build
farm.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Julia Lawall]

On Tue, 7 Jul 2015, Mark Brown wrote:

> On Tue, Jul 07, 2015 at 08:25:21AM -0700, Guenter Roeck wrote:
> > On 07/07/2015 02:24 AM, Mark Brown wrote:
>
> > >The main things I'm aware of that are happening at the minute are
> > >kselftest development, the 0day tester, plus kernelci.org and the other
> > >build and boot/test bots that are running against various trees.
>
> > Maybe list all known ones as a start ?
>
> Off the top of my head the automated ones I'm aware of are Olof's build
> & boot test, Dan running smatch and I think some other static analysis
> stuff, someone (not sure who?) running some coccinelle stuff, Coverity
> and I've got a builder too.

The 0day service runs Coccinelle.

Coccinelle does not need the build to succeed.

julia

> > >In terms of discussion topics some of the issues I'm seeing are:
>
> > >  - Can we pool resources to share the workload of running things and
> > >    interpreting results, ideally also providing some central way for
> > >    people to discover what results are out there for them to look at
> > >    for a given kernel in the different systems?
>
> > That might be quite useful. However, I have seen that it doesn't really
> > help to just provide the test results. kissb test results have been
> > available for ages, and people just don't look at it. Even the regular
> > "Build regression" e-mails sent out by Geert seem to be widely ignored.
>
> > What I really found to help is to bisect new problems and send an e-mail
> > to the responsible maintainer and to the submitter of the patch which
> > introduced it. I'd like to automate that with my test system, but
> > unfortunately I just don't have the time to do it.
>
> Yes, that's the "and interpreting" bit in the above - this only really
> works with people actively pushing.  You do start to get people checking
> themselves once things are perceived as something people care about but
> it does take active work to establish and maintain that.
>
> It also really helps if things are delivered promptly, and against trees
> people are actively developing for.  But even with clear reports and
> sometimes patches not everyone shows an interest.  As we get more and
> more actual testing running that's going to start to become more
> serious, breaking the build or boot will also mean that automated tests
> don't get to run.
>
> This is one of the things 0day gets really right, when it kicks in it'll
> e-mail people directly and promptly.
>
> > >  - Should we start carrying config fragments upstream designed to
> > >    support testing, things like the distro config fragments that keep
> > >    getting discussed are one example here but there's other things like
> > >    collections of debug options we could be looking at.  Should we be
> > >    more generally slimming defconfigs and moving things into fragments?
>
> > >and there's always the the perennial ones about what people would like
> > >to see testing for.
>
> > Sharing as many test bot configuration scripts and relevant configurations
> > as possible would be quite helpful. For example, I am building various
> > configurations for all architectures, but I don't really know if they
> > are relevant. Also, I would like to run more qemu configurations,
> > but it is really hard to find working ones.
>
> Grant (just CCed) was working intermittently on the qemu bit.  I think
> the last plan was to enhance the scripts Kevin has for driving his build
> farm.
>

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Shuah Khan]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/07/2015 11:18 AM, Mark Brown wrote:
> On Tue, Jul 07, 2015 at 08:25:21AM -0700, Guenter Roeck wrote:
>> On 07/07/2015 02:24 AM, Mark Brown wrote:
> 
>>> The main things I'm aware of that are happening at the minute
>>> are kselftest development, the 0day tester, plus kernelci.org
>>> and the other build and boot/test bots that are running against
>>> various trees.
> 
>> Maybe list all known ones as a start ?
> 
> Off the top of my head the automated ones I'm aware of are Olof's
> build & boot test, Dan running smatch and I think some other static
> analysis stuff, someone (not sure who?) running some coccinelle
> stuff, Coverity and I've got a builder too.
> 
>>> In terms of discussion topics some of the issues I'm seeing
>>> are:
> 
>>> - Can we pool resources to share the workload of running things
>>> and interpreting results, ideally also providing some central
>>> way for people to discover what results are out there for them
>>> to look at for a given kernel in the different systems?
> 
>> That might be quite useful. However, I have seen that it doesn't
>> really help to just provide the test results. kissb test results
>> have been available for ages, and people just don't look at it.
>> Even the regular "Build regression" e-mails sent out by Geert
>> seem to be widely ignored.
> 
>> What I really found to help is to bisect new problems and send an
>> e-mail to the responsible maintainer and to the submitter of the
>> patch which introduced it. I'd like to automate that with my test
>> system, but unfortunately I just don't have the time to do it.
> 
> Yes, that's the "and interpreting" bit in the above - this only
> really works with people actively pushing.  You do start to get
> people checking themselves once things are perceived as something
> people care about but it does take active work to establish and
> maintain that.
> 
> It also really helps if things are delivered promptly, and against
> trees people are actively developing for.  But even with clear
> reports and sometimes patches not everyone shows an interest.  As
> we get more and more actual testing running that's going to start
> to become more serious, breaking the build or boot will also mean
> that automated tests don't get to run.
> 
> This is one of the things 0day gets really right, when it kicks in
> it'll e-mail people directly and promptly.
> 
>>> - Should we start carrying config fragments upstream designed
>>> to support testing, things like the distro config fragments
>>> that keep getting discussed are one example here but there's
>>> other things like collections of debug options we could be
>>> looking at.  Should we be more generally slimming defconfigs
>>> and moving things into fragments?
> 
>>> and there's always the the perennial ones about what people
>>> would like to see testing for.
> 
>> Sharing as many test bot configuration scripts and relevant
>> configurations as possible would be quite helpful. For example, I
>> am building various configurations for all architectures, but I
>> don't really know if they are relevant. Also, I would like to run
>> more qemu configurations, but it is really hard to find working
>> ones.
> 
> Grant (just CCed) was working intermittently on the qemu bit.  I
> think the last plan was to enhance the scripts Kevin has for
> driving his build farm.
> 

Thanks for starting this discussion. Now that Kselftest install is
in place and several cross-compile problems are fixed, I would like
to gauge interest in being able to include kselftest in qemu boot
tests. I added quicktest option in 4.2 to meet qemu environment time
constraints.

thanks,
- -- Shuah

- -- 
Shuah Khan
Sr. Linux Kernel Developer
Open Source Innovation Group
Samsung Research America (Silicon Valley)
shuahkh@osg.samsung.com | (970) 217-8978
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBAgAGBQJVnAtBAAoJEAsCRMQNDUMca5wP/jrsBO++J4IsOnaJYwChWRp1
XHNEtrVwnvuR9zGo295lQUZL0cihug8FiTkfHX3P6M+9kEJQaUKlMyJymh9y4dzU
cmDLhh8kXwdtPq1qM8jzaTqBb6sqXvzXxRoMi3X7AYPd2yA4uLMluAgj+dg8n1x/
W55fIi4ghypNQvtLcC92zyp5MKtBd5zr75HUZPxsqPwo7AHtqy0KoV+1fx+qQ/Lt
GnZxIlWQMmA0yubiAGZHl+nECfiwcx8iS302y30d3+8/YxAztUy9l/7VfgBBgyOp
9Ouw6xC8Xmyb8GfFUUBy4MIAvS0QrBcYQp/z9Qlmj5MioOBydPaS3+Y4AIKLEaeI
Vu+/OVqZO9At+HML6TeDBvM44PWcjfC0plm8n8PLvFtruVnLEnxKeeaXCUrj9AG7
8GUTscjkOA5p8r3qqtKny3OftKvUINXZUL+zbEnV7kUlhlDuOBSN5SDZk6VkpTSP
lTJOTew+At0JH0x8YEA5k6L94JNT4zfksS/4N7z6bGcIdBIITrn3MGYkoi1fCqh2
GLHqKV8fLMv5aEK4tt9phDgmJSxITsMbfwuwoPYK0KRY4hUfri73uuBB1Igg9noI
d3H7yVYpL7Td3b98g7VuzIfOEdMbv9jFGgPJb8A0MiGYeu87iu56jJEXaj2JIAEo
SxNLh6SDkNkuH+6l2FAJ
=jD4c
-----END PGP SIGNATURE-----

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Guenter Roeck]

On 07/07/2015 10:24 AM, Shuah Khan wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
[ ... ]
>
> Thanks for starting this discussion. Now that Kselftest install is
> in place and several cross-compile problems are fixed, I would like
> to gauge interest in being able to include kselftest in qemu boot
> tests. I added quicktest option in 4.2 to meet qemu environment time
> constraints.
>

Me too, and very much so. Only reason for not doing it yet is lack
of time.

Guenter

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Guenter Roeck]

On 07/07/2015 10:18 AM, Mark Brown wrote:
> On Tue, Jul 07, 2015 at 08:25:21AM -0700, Guenter Roeck wrote:
>> On 07/07/2015 02:24 AM, Mark Brown wrote:
>
>>> The main things I'm aware of that are happening at the minute are
>>> kselftest development, the 0day tester, plus kernelci.org and the other
>>> build and boot/test bots that are running against various trees.
>
>> Maybe list all known ones as a start ?
>
> Off the top of my head the automated ones I'm aware of are Olof's build
> & boot test, Dan running smatch and I think some other static analysis
> stuff, someone (not sure who?) running some coccinelle stuff, Coverity
> and I've got a builder too.
>
Plus mine, of course. Only part missing is automated bisect and e-mail
if something starts failing.

Which reminds me - do you use buildbot ? I think you are sending automated
e-mail on failures. It would help me a lot if someone had automated bisect
and the ability to e-mail results using buildbot to get me started.

>>> In terms of discussion topics some of the issues I'm seeing are:
>
>>>   - Can we pool resources to share the workload of running things and
>>>     interpreting results, ideally also providing some central way for
>>>     people to discover what results are out there for them to look at
>>>     for a given kernel in the different systems?
>
>> That might be quite useful. However, I have seen that it doesn't really
>> help to just provide the test results. kissb test results have been
>> available for ages, and people just don't look at it. Even the regular
>> "Build regression" e-mails sent out by Geert seem to be widely ignored.
>
>> What I really found to help is to bisect new problems and send an e-mail
>> to the responsible maintainer and to the submitter of the patch which
>> introduced it. I'd like to automate that with my test system, but
>> unfortunately I just don't have the time to do it.
>
> Yes, that's the "and interpreting" bit in the above - this only really
> works with people actively pushing.  You do start to get people checking
> themselves once things are perceived as something people care about but
> it does take active work to establish and maintain that.
>
> It also really helps if things are delivered promptly, and against trees
> people are actively developing for.  But even with clear reports and
> sometimes patches not everyone shows an interest.  As we get more and
> more actual testing running that's going to start to become more
> serious, breaking the build or boot will also mean that automated tests
> don't get to run.
>
Yes, I have seen that too. Especially 4.1 was pretty bad in this regard.
4.2 seems to be a bit better, though, so I hope that 4.1 was an exception.

Not really sure what to do about it. What turned out to help in the last
two companies I worked for was automatic revert of broken patches. That
sounds radical and I dislike it myself, but it helped.

> This is one of the things 0day gets really right, when it kicks in it'll
> e-mail people directly and promptly.
>
Agreed.

>>>   - Should we start carrying config fragments upstream designed to
>>>     support testing, things like the distro config fragments that keep
>>>     getting discussed are one example here but there's other things like
>>>     collections of debug options we could be looking at.  Should we be
>>>     more generally slimming defconfigs and moving things into fragments?
>
>>> and there's always the the perennial ones about what people would like
>>> to see testing for.
>
>> Sharing as many test bot configuration scripts and relevant configurations
>> as possible would be quite helpful. For example, I am building various
>> configurations for all architectures, but I don't really know if they
>> are relevant. Also, I would like to run more qemu configurations,
>> but it is really hard to find working ones.
>
> Grant (just CCed) was working intermittently on the qemu bit.  I think
> the last plan was to enhance the scripts Kevin has for driving his build
> farm.
>
Also of interest here (at least for me) would be to explore means to get
more hardware (both architectures and platforms) supported in qemu, but
I guess that may be a bit off topic.

Thanks,
Guenter

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1341 bytes --]

On Tue, Jul 07, 2015 at 10:52:04AM -0700, Guenter Roeck wrote:
> On 07/07/2015 10:18 AM, Mark Brown wrote:

> >Off the top of my head the automated ones I'm aware of are Olof's build
> >& boot test, Dan running smatch and I think some other static analysis
> >stuff, someone (not sure who?) running some coccinelle stuff, Coverity
> >and I've got a builder too.

> Plus mine, of course. Only part missing is automated bisect and e-mail
> if something starts failing.

> Which reminds me - do you use buildbot ? I think you are sending automated
> e-mail on failures. It would help me a lot if someone had automated bisect
> and the ability to e-mail results using buildbot to get me started.

No, not me - all my failure reports are lovingly hand crafted using
traditional artisan techniques.  Kevin, Tyler and Fengguang have things
but apart from 0day I think everything is still manually triggered.

> Not really sure what to do about it. What turned out to help in the last
> two companies I worked for was automatic revert of broken patches. That
> sounds radical and I dislike it myself, but it helped.

Perhaps that's something we should be discussing?  It may be something
that we just evolve a solution for as we proceed though - right now it's
largely theoretical.  For -next Stephen will often carry extra patches
that make sense.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Steven Rostedt]

On Tue, 7 Jul 2015 14:14:11 +0100
Mark Brown <broonie@sirena.org.uk> wrote:

> On Tue, Jul 07, 2015 at 04:02:13PM +0300, Alexey Dobriyan wrote:
> > On Tue, Jul 7, 2015 at 12:24 PM, Mark Brown <broonie@kernel.org> wrote:
> 
> > >  - Should we start carrying config fragments upstream designed to
> > >    support testing, things like the distro config fragments that keep
> > >    getting discussed are one example here but there's other things like
> > >    collections of debug options we could be looking at.
> 
> > This will gravitate everyone to running the same config which is the opposite
> > of what you want.
> 
> Perhaps, perhaps not - it's not an unequivocal thing either way.  The
> more barriers there are to enabling things the more likely it is that
> people just won't bother in the first place (or that they'll run into
> somme problem and give up before they get things working) and it's not
> clear that having to figure these things out is always a good use of
> people's time.

The testing/selftests tests should have three results: PASS, FAIL,
UNSUPPORTED. The UNSUPPORTED is what should be returned if the kernel
configuration doesn't have the needed features configured. For example,
if you run the ftrace selftests without function tracing enabled, all
the tests that test the function tracer return UNSUPPORTED.

Perhaps we should have a central location that each test needs to add
the required configuration for it to be properly tested. Then if users
want to test various subsystems, they would look in this location for
the proper configs (be it a directory that has files of the tests they
represent, and contain the configs needed). Then there should be no
real barrier for people to run these tests.

Of course if the test requires certain hardware, or a file system, then
that should be properly documented.

-- Steve

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Geert Uytterhoeven]

On Tue, Jul 7, 2015 at 5:25 PM, Guenter Roeck <linux@roeck-us.net> wrote:
>> In terms of discussion topics some of the issues I'm seeing are:
>>
>>   - Can we pool resources to share the workload of running things and
>>     interpreting results, ideally also providing some central way for
>>     people to discover what results are out there for them to look at
>>     for a given kernel in the different systems?
>>
> That might be quite useful. However, I have seen that it doesn't really
> help to just provide the test results. kissb test results have been
> available for ages, and people just don't look at it. Even the regular
> "Build regression" e-mails sent out by Geert seem to be widely ignored.

There's still a manual step involved, so I only download the logs and
generate the regression emails for every rc release.
My original plan was to automate it more, so I could run it fully automated
on the linux-next builds, too.

> What I really found to help is to bisect new problems and send an e-mail
> to the responsible maintainer and to the submitter of the patch which
> introduced it. I'd like to automate that with my test system, but
> unfortunately I just don't have the time to do it.

IIRC, kisskb used to bisect failures, but stopped doing that when it started
consuming too much time. Perhaps it can be restarted, given the system
got an upgrade a few months ago and became much much faster?
Stiil, there's the "analyzing" part.

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Kees Cook]

On Tue, Jul 7, 2015 at 11:47 AM, Steven Rostedt <rostedt@goodmis.org> wrote:
> On Tue, 7 Jul 2015 14:14:11 +0100
> Mark Brown <broonie@sirena.org.uk> wrote:
>
>> On Tue, Jul 07, 2015 at 04:02:13PM +0300, Alexey Dobriyan wrote:
>> > On Tue, Jul 7, 2015 at 12:24 PM, Mark Brown <broonie@kernel.org> wrote:
>>
>> > >  - Should we start carrying config fragments upstream designed to
>> > >    support testing, things like the distro config fragments that keep
>> > >    getting discussed are one example here but there's other things like
>> > >    collections of debug options we could be looking at.
>>
>> > This will gravitate everyone to running the same config which is the opposite
>> > of what you want.
>>
>> Perhaps, perhaps not - it's not an unequivocal thing either way.  The
>> more barriers there are to enabling things the more likely it is that
>> people just won't bother in the first place (or that they'll run into
>> somme problem and give up before they get things working) and it's not
>> clear that having to figure these things out is always a good use of
>> people's time.
>
> The testing/selftests tests should have three results: PASS, FAIL,
> UNSUPPORTED. The UNSUPPORTED is what should be returned if the kernel
> configuration doesn't have the needed features configured. For example,
> if you run the ftrace selftests without function tracing enabled, all
> the tests that test the function tracer return UNSUPPORTED.
>
> Perhaps we should have a central location that each test needs to add
> the required configuration for it to be properly tested. Then if users
> want to test various subsystems, they would look in this location for
> the proper configs (be it a directory that has files of the tests they
> represent, and contain the configs needed). Then there should be no
> real barrier for people to run these tests.
>
> Of course if the test requires certain hardware, or a file system, then
> that should be properly documented.

There's also sysctl settings, and we also want privilege level as part
of that configuration. (i.e. some tests should run as root, some show
not, etc). It'd be nice to have the tests report these in some
machine-readable form so that a test harness could check, prepare the
environment, and then run the tests.

-Kees

-- 
Kees Cook
Chrome OS Security

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Andy Lutomirski]

On Tue, Jul 7, 2015 at 1:46 PM, Kees Cook <keescook@chromium.org> wrote:
> There's also sysctl settings, and we also want privilege level as part
> of that configuration. (i.e. some tests should run as root, some show
> not, etc). It'd be nice to have the tests report these in some
> machine-readable form so that a test harness could check, prepare the
> environment, and then run the tests.
>

There was a fair amount of discussion about machine-readable output at
the last KS.  I don't know whether it ever got implemented.

--Andy

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Peter Hüwe]

Hi,

I definitely think discussing the next steps in terms of automated regression 
testing is important - so thanks for raising this issue!!

> >> Maybe list all known ones as a start ?
> > 
> > Off the top of my head the automated ones I'm aware of are Olof's build
> > & boot test, Dan running smatch and I think some other static analysis
> > stuff, someone (not sure who?) running some coccinelle stuff, Coverity
> > and I've got a builder too.

For the TPM subsystem I have several things in place:
- applying patches automatically triggers a build on travis-ci.org
-- it triggers some basic style checkers
-- the build runs a qemu (with the tpm simulator) running another qemu (which 
now sees a real /dev/tpm0) with the new kernel.
--- the qemu-qemu-linux runs my tpm driver testsuite.

The reason behind this is that I can hook up besides the TPM1.2 simulator also 
the binary only windows based TPM2.0 simulator (with wine) without modifying 
qemu source -- but still the qemu-qemu-kernel sees 'real' hardware.

- some scripts for deploying new kernels to real hw machines and running tests 
with real hw tpms

- a mocked i2c tpm which I can hook up via the I2C_Slave interface, which also 
passes my driver testsuite -- so I can test the drivers within UML :)

- and of course wolfram's ninja-check scripts for style checkers

 
> Plus mine, of course. Only part missing is automated bisect and e-mail
> if something starts failing.


Yeah - getting the reports reasonably fast is probably the most important 
stuff -- the 0day testing really does a great job here.


Thanks,
Peter

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Dan Carpenter]

Doesn't the 0day system obsolete the "Build regression" emails?  It
should be sending emails to everyone introducing new build warnings.
Are people ignoring them?

regards,
dan carpenter

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Geert Uytterhoeven]

Hi Dan,

On Wed, Jul 8, 2015 at 9:54 AM, Dan Carpenter <dan.carpenter@oracle.com> wrote:
> Doesn't the 0day system obsolete the "Build regression" emails?  It

I think kisskb builds more/different configs.

Is there are list of configs and trees covered by the 0day system available?

> should be sending emails to everyone introducing new build warnings.
> Are people ignoring them?

Of course people ignore them ;-)

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Michael Ellerman]

On Tue, 2015-07-07 at 08:25 -0700, Guenter Roeck wrote:
> On 07/07/2015 02:24 AM, Mark Brown wrote:
> > One thing we typically cover at Kernel Summit is some of the activity
> > that's going on around testing upstream.  I think it'd be useful to have
> > some more of those discussions, both in terms of making people aware of
> > what's available and in terms of helping the people doing testing figure
> > out what would be useful.  A lot of this is probably well suited to a
> > workshop session between the interested people but I do think some
> > element in the core day beyond just a readout will be useful.
> >
> > In terms of discussion topics some of the issues I'm seeing are:
> >
> >   - Can we pool resources to share the workload of running things and
> >     interpreting results, ideally also providing some central way for
> >     people to discover what results are out there for them to look at
> >     for a given kernel in the different systems?
>
> That might be quite useful. However, I have seen that it doesn't really
> help to just provide the test results. kissb test results have been
> available for ages, and people just don't look at it.

My concern with kisskb sending emails was always that I didn't want it to
become a spam bot. So it can send emails, but it's opt-in.

The 0-day bot takes the opposite approach, ie. mails everyone without asking,
and in hindsight that is clearly the better option in terms of getting people
to act on the results.


> Sharing as many test bot configuration scripts and relevant configurations
> as possible would be quite helpful. For example, I am building various
> configurations for all architectures, but I don't really know if they
> are relevant.

Agreed. Your buildbot is epic. I'd love to see the config for that. My local
buildbot is running only ~40 builders, which I thought was a lot until I saw
yours :)

The kernelci.org stuff is also really interesting, that's the closest thing
anyone has at the moment to a "proper" kernel CI setup AFAIK.

cheers

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 655 bytes --]

On Wed, Jul 08, 2015 at 10:54:09AM +0300, Dan Carpenter wrote:

> Doesn't the 0day system obsolete the "Build regression" emails?  It
> should be sending emails to everyone introducing new build warnings.

0day only covers some configurations and doesn't always manage to kick
in (I know I've had stuff come in via other channels sometimes, it seems
to depend on loading).

> Are people ignoring them?

They're not reliably followed through on, no, and one of the things with
0day is that it just generates a one time report so if things don't get
followed up on then that's that.  A regular "these are all the issues"
mail helps chase down those issues.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1489 bytes --]

On Tue, Jul 07, 2015 at 02:47:25PM -0400, Steven Rostedt wrote:
> Mark Brown <broonie@sirena.org.uk> wrote:

> > Perhaps, perhaps not - it's not an unequivocal thing either way.  The
> > more barriers there are to enabling things the more likely it is that
> > people just won't bother in the first place (or that they'll run into
> > somme problem and give up before they get things working) and it's not
> > clear that having to figure these things out is always a good use of
> > people's time.

> The testing/selftests tests should have three results: PASS, FAIL,
> UNSUPPORTED. The UNSUPPORTED is what should be returned if the kernel
> configuration doesn't have the needed features configured. For example,
> if you run the ftrace selftests without function tracing enabled, all
> the tests that test the function tracer return UNSUPPORTED.

That's roughly what they're supposed to be doing now (I'd need to go
check exactly what happens in the unsupported case).

> Perhaps we should have a central location that each test needs to add
> the required configuration for it to be properly tested. Then if users
> want to test various subsystems, they would look in this location for
> the proper configs (be it a directory that has files of the tests they
> represent, and contain the configs needed). Then there should be no
> real barrier for people to run these tests.

Right, this is what I'm suggesting roughly - make the configurations
required to run tests easier to pick up.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Jiri Kosina]

On Wed, 8 Jul 2015, Geert Uytterhoeven wrote:

> > should be sending emails to everyone introducing new build warnings.
> > Are people ignoring them?
> 
> Of course people ignore them ;-)

I think the biggest problems with these are:

- they are all squashed together into one report, totally unrelated things 
  together at one place. Noone is ever going to be actively looking into 
  it to see whether something he's responsible for hasn't popped up

- they are not addressed to anybody explicitly. Sending them just to LKML 
  is a direct ticket to the "be ignored" land

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Josh Boyer]

On Wed, Jul 8, 2015 at 8:10 AM, Jiri Kosina <jkosina@suse.com> wrote:
> On Wed, 8 Jul 2015, Geert Uytterhoeven wrote:
>
>> > should be sending emails to everyone introducing new build warnings.
>> > Are people ignoring them?
>>
>> Of course people ignore them ;-)
>
> I think the biggest problems with these are:
>
> - they are all squashed together into one report, totally unrelated things
>   together at one place. Noone is ever going to be actively looking into
>   it to see whether something he's responsible for hasn't popped up
>
> - they are not addressed to anybody explicitly. Sending them just to LKML
>   is a direct ticket to the "be ignored" land

Yep.  The same goes for bug reports.  LKML is a great archiving tool
for digging through history, but it is useless as a form of general
communication.

josh

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Guenter Roeck]

Hi Michael,

On 07/08/2015 02:27 AM, Michael Ellerman wrote:
> On Tue, 2015-07-07 at 08:25 -0700, Guenter Roeck wrote:
>> On 07/07/2015 02:24 AM, Mark Brown wrote:
>>> One thing we typically cover at Kernel Summit is some of the activity
>>> that's going on around testing upstream.  I think it'd be useful to have
>>> some more of those discussions, both in terms of making people aware of
>>> what's available and in terms of helping the people doing testing figure
>>> out what would be useful.  A lot of this is probably well suited to a
>>> workshop session between the interested people but I do think some
>>> element in the core day beyond just a readout will be useful.
>>>
>>> In terms of discussion topics some of the issues I'm seeing are:
>>>
>>>    - Can we pool resources to share the workload of running things and
>>>      interpreting results, ideally also providing some central way for
>>>      people to discover what results are out there for them to look at
>>>      for a given kernel in the different systems?
>>
>> That might be quite useful. However, I have seen that it doesn't really
>> help to just provide the test results. kissb test results have been
>> available for ages, and people just don't look at it.
>
> My concern with kisskb sending emails was always that I didn't want it to
> become a spam bot. So it can send emails, but it's opt-in.
>
> The 0-day bot takes the opposite approach, ie. mails everyone without asking,
> and in hindsight that is clearly the better option in terms of getting people
> to act on the results.
>
>
>> Sharing as many test bot configuration scripts and relevant configurations
>> as possible would be quite helpful. For example, I am building various
>> configurations for all architectures, but I don't really know if they
>> are relevant.
>
> Agreed. Your buildbot is epic. I'd love to see the config for that. My local
> buildbot is running only ~40 builders, which I thought was a lot until I saw
> yours :)
>
It is on github: https://github.com/groeck/linux-build-test.

If I count correctly, it runs more than 900 builders. Major hiccup is with
caching - it collects around 10GB of logging data per month, and under
some circumstances keeps it all in memory, so after about two months it
consumes the entire 32GB of RAM on the server it is running on, and I have
to do some manual cleanup. Other than that, it runs surprisingly well.

> The kernelci.org stuff is also really interesting, that's the closest thing
> anyone has at the moment to a "proper" kernel CI setup AFAIK.
>
Agreed.

Guenter

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Kevin Hilman]

Guenter Roeck <linux@roeck-us.net> writes:

> Hi Michael,
>
> On 07/08/2015 02:27 AM, Michael Ellerman wrote:
>> On Tue, 2015-07-07 at 08:25 -0700, Guenter Roeck wrote:
>>> On 07/07/2015 02:24 AM, Mark Brown wrote:
>>>> One thing we typically cover at Kernel Summit is some of the activity
>>>> that's going on around testing upstream.  I think it'd be useful to have
>>>> some more of those discussions, both in terms of making people aware of
>>>> what's available and in terms of helping the people doing testing figure
>>>> out what would be useful.  A lot of this is probably well suited to a
>>>> workshop session between the interested people but I do think some
>>>> element in the core day beyond just a readout will be useful.
>>>>
>>>> In terms of discussion topics some of the issues I'm seeing are:
>>>>
>>>>    - Can we pool resources to share the workload of running things and
>>>>      interpreting results, ideally also providing some central way for
>>>>      people to discover what results are out there for them to look at
>>>>      for a given kernel in the different systems?
>>>
>>> That might be quite useful. However, I have seen that it doesn't really
>>> help to just provide the test results. kissb test results have been
>>> available for ages, and people just don't look at it.
>>
>> My concern with kisskb sending emails was always that I didn't want it to
>> become a spam bot. So it can send emails, but it's opt-in.
>>
>> The 0-day bot takes the opposite approach, ie. mails everyone without asking,
>> and in hindsight that is clearly the better option in terms of getting people
>> to act on the results.
>>
>>
>>> Sharing as many test bot configuration scripts and relevant configurations
>>> as possible would be quite helpful. For example, I am building various
>>> configurations for all architectures, but I don't really know if they
>>> are relevant.
>>
>> Agreed. Your buildbot is epic. I'd love to see the config for that. My local
>> buildbot is running only ~40 builders, which I thought was a lot until I saw
>> yours :)
>>
> It is on github: https://github.com/groeck/linux-build-test.
>
> If I count correctly, it runs more than 900 builders. Major hiccup is with
> caching - it collects around 10GB of logging data per month, and under
> some circumstances keeps it all in memory, so after about two months it
> consumes the entire 32GB of RAM on the server it is running on, and I have
> to do some manual cleanup. Other than that, it runs surprisingly well.

How soon after a branch is pushed are the build results available?

Are the build artfiacts ([bz]Image, System.map, modules, etc.) made
available someplace?

>> The kernelci.org stuff is also really interesting, that's the closest thing
>> anyone has at the moment to a "proper" kernel CI setup AFAIK.
>
> Agreed.

Glad you find it useful.

Speaking for kernelci.org... with limited time/resources, we'd like to
focus less on building (others are doing this faster/better) and more on
boot testing across a wide variety of hardware (e.g for the latest -next
build, we did ~430 boots on 88 unique boards covering 23 different
SoCs[1]).  We're also in the process of automating kselftest runs on our
boards.

If we could consume the output of other builders, we'd happily do that
instead of doing our own builds.  Ideally, the builders should produce
some sort of machine readable data with the build artifacts..  Currently
our builders produce a JSON file[2] which can be submitted to
kernelci.org using a RESTful API[3].

AFAICT, 0day doesn't have publically available build artfiacts, and
I haven't had a chance to look closely at Gunter's stuff.

Kevin

[1] http://kernelci.org/boot/all/job/next/kernel/next-20150708/
[2] example for an ARM multi_v7_defconfib build:
    http://storage.kernelci.org/next/next-20150708/arm-multi_v7_defconfig/build.json
[3] http://api.kernelci.org/

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Guenter Roeck]

On 07/08/2015 09:40 AM, Kevin Hilman wrote:
> Guenter Roeck <linux@roeck-us.net> writes:
>
>> Hi Michael,
>>
>> On 07/08/2015 02:27 AM, Michael Ellerman wrote:
>>> On Tue, 2015-07-07 at 08:25 -0700, Guenter Roeck wrote:
>>>> On 07/07/2015 02:24 AM, Mark Brown wrote:
>>>>> One thing we typically cover at Kernel Summit is some of the activity
>>>>> that's going on around testing upstream.  I think it'd be useful to have
>>>>> some more of those discussions, both in terms of making people aware of
>>>>> what's available and in terms of helping the people doing testing figure
>>>>> out what would be useful.  A lot of this is probably well suited to a
>>>>> workshop session between the interested people but I do think some
>>>>> element in the core day beyond just a readout will be useful.
>>>>>
>>>>> In terms of discussion topics some of the issues I'm seeing are:
>>>>>
>>>>>     - Can we pool resources to share the workload of running things and
>>>>>       interpreting results, ideally also providing some central way for
>>>>>       people to discover what results are out there for them to look at
>>>>>       for a given kernel in the different systems?
>>>>
>>>> That might be quite useful. However, I have seen that it doesn't really
>>>> help to just provide the test results. kissb test results have been
>>>> available for ages, and people just don't look at it.
>>>
>>> My concern with kisskb sending emails was always that I didn't want it to
>>> become a spam bot. So it can send emails, but it's opt-in.
>>>
>>> The 0-day bot takes the opposite approach, ie. mails everyone without asking,
>>> and in hindsight that is clearly the better option in terms of getting people
>>> to act on the results.
>>>
>>>
>>>> Sharing as many test bot configuration scripts and relevant configurations
>>>> as possible would be quite helpful. For example, I am building various
>>>> configurations for all architectures, but I don't really know if they
>>>> are relevant.
>>>
>>> Agreed. Your buildbot is epic. I'd love to see the config for that. My local
>>> buildbot is running only ~40 builders, which I thought was a lot until I saw
>>> yours :)
>>>
>> It is on github: https://github.com/groeck/linux-build-test.
>>
>> If I count correctly, it runs more than 900 builders. Major hiccup is with
>> caching - it collects around 10GB of logging data per month, and under
>> some circumstances keeps it all in memory, so after about two months it
>> consumes the entire 32GB of RAM on the server it is running on, and I have
>> to do some manual cleanup. Other than that, it runs surprisingly well.
>
> How soon after a branch is pushed are the build results available?
>

Depends on the branch and the load on the system. Building a single branch takes
maybe - I am guessing here - 90 minutes. If several branches are changed at the
same time (like when Greg pushes changes into the -stable branches he manages),
the overall build time is multiplied with the number of branches to build.

In addition to that, each branch has a different set of rules. I only pull mainline
a couple of times per week, because it changes too much and by itself would keep the
system busy all the time. -next is pulled once per day, as is -mmotm. Plus, I have
a number of different methods on how to handle changes. For example, Greg tends
to push many changes into -stable queues within a short period of time, resulting
in a lot of build churn if I would start builds immediately after a change is detected.
So if a change on a -stable branch maintained by Greg is detected, the system waits
for a couple of hours before actually starting a build to reduce churn. Other branches
see less churn; -mmotm and -next are built almost immediately after a change is detected,
for example.

In a nutshell, it can take anytime between about two hours and about three days
for results to be available, depending on the branch and the load on the system.

> Are the build artfiacts ([bz]Image, System.map, modules, etc.) made
> available someplace?
>
Not currently. I could make it available if there is interest, though I'd
probably have to find some place out in the web to keep the data (we are talking
16 branches times ~150 builds per branch times the number of builds kept in the log).
If I assume that five builds are kept per branch, and each build takes only 10MB
of space, this already adds up to more than 100 GB of space needed, plus of course
all the associated net bandwidth. There are a couple of sites pulling the complete
build history from my servers on a daily basis (happily ignoring robots.txt),
so that might end up consuming a lot of bandwidth.

>>> The kernelci.org stuff is also really interesting, that's the closest thing
>>> anyone has at the moment to a "proper" kernel CI setup AFAIK.
>>
>> Agreed.
>
> Glad you find it useful.
>
> Speaking for kernelci.org... with limited time/resources, we'd like to
> focus less on building (others are doing this faster/better) and more on
> boot testing across a wide variety of hardware (e.g for the latest -next
> build, we did ~430 boots on 88 unique boards covering 23 different
> SoCs[1]).  We're also in the process of automating kselftest runs on our
> boards.
>
> If we could consume the output of other builders, we'd happily do that
> instead of doing our own builds.  Ideally, the builders should produce
> some sort of machine readable data with the build artifacts..  Currently
> our builders produce a JSON file[2] which can be submitted to
> kernelci.org using a RESTful API[3].
>
Unfortunately, I had to disable JSON on my buildbot installation.
Reason is again that there are sites pulling the entire build history
on a regular basis, and my server ended up getting stuck doing nothing
but serving JSON requests (those take a lot of CPU).

Guenter

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 834 bytes --]

On Wed, Jul 08, 2015 at 02:10:36PM +0200, Jiri Kosina wrote:
> On Wed, 8 Jul 2015, Geert Uytterhoeven wrote:

[0day]
> > > should be sending emails to everyone introducing new build warnings.
> > > Are people ignoring them?

> > Of course people ignore them ;-)

> I think the biggest problems with these are:

> - they are all squashed together into one report, totally unrelated things 
>   together at one place. Noone is ever going to be actively looking into 
>   it to see whether something he's responsible for hasn't popped up

> - they are not addressed to anybody explicitly. Sending them just to LKML 
>   is a direct ticket to the "be ignored" land

That's not the case for the 0day warning reports, they're sent directly
to the people mentioned in the commit on a per-commit basis but as a one
shot.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 349 bytes --]

On Tue, Jul 07, 2015 at 03:02:11PM -0700, Andy Lutomirski wrote:

> There was a fair amount of discussion about machine-readable output at
> the last KS.  I don't know whether it ever got implemented.

Yes, but that's very much at the most basic stage of reporting pass and
fail - there's probably room for looking at if we're ready to extend
that.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Kevin Hilman]

Guenter Roeck <linux@roeck-us.net> writes:

>> Are the build artfiacts ([bz]Image, System.map, modules, etc.) made
>> available someplace?
>>
> Not currently. I could make it available if there is interest, though I'd
> probably have to find some place out in the web to keep the data (we are talking
> 16 branches times ~150 builds per branch times the number of builds kept in the log).
> If I assume that five builds are kept per branch, and each build takes only 10MB
> of space, this already adds up to more than 100 GB of space needed, 

Understood. storage.kernelci.org is currently using ~400G for 45 days
worth of builds from the 55+ branches x ~150 builds/branch we do, and
includes boot logs/results too.

Kevin

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Michael Ellerman]

On Wed, 2015-07-08 at 09:40 -0700, Kevin Hilman wrote:
>
> Speaking for kernelci.org... with limited time/resources, we'd like to
> focus less on building (others are doing this faster/better) and more on
> boot testing across a wide variety of hardware (e.g for the latest -next
> build, we did ~430 boots on 88 unique boards covering 23 different
> SoCs[1]).  We're also in the process of automating kselftest runs on our
> boards.
> 
> If we could consume the output of other builders, we'd happily do that
> instead of doing our own builds.  Ideally, the builders should produce
> some sort of machine readable data with the build artifacts.. 

Unfortunately providing build artifacts means you're distributing binaries, and
so life just got complicated. At least for folks like me who work at a company
with too many lawyers.

cheers

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Masami Hiramatsu]

On 2015/07/08 3:47, Steven Rostedt wrote:
> On Tue, 7 Jul 2015 14:14:11 +0100
> Mark Brown <broonie@sirena.org.uk> wrote:
> 
>> On Tue, Jul 07, 2015 at 04:02:13PM +0300, Alexey Dobriyan wrote:
>>> On Tue, Jul 7, 2015 at 12:24 PM, Mark Brown <broonie@kernel.org> wrote:
>>
>>>>  - Should we start carrying config fragments upstream designed to
>>>>    support testing, things like the distro config fragments that keep
>>>>    getting discussed are one example here but there's other things like
>>>>    collections of debug options we could be looking at.
>>
>>> This will gravitate everyone to running the same config which is the opposite
>>> of what you want.
>>
>> Perhaps, perhaps not - it's not an unequivocal thing either way.  The
>> more barriers there are to enabling things the more likely it is that
>> people just won't bother in the first place (or that they'll run into
>> somme problem and give up before they get things working) and it's not
>> clear that having to figure these things out is always a good use of
>> people's time.
> 
> The testing/selftests tests should have three results: PASS, FAIL,
> UNSUPPORTED. The UNSUPPORTED is what should be returned if the kernel
> configuration doesn't have the needed features configured. For example,
> if you run the ftrace selftests without function tracing enabled, all
> the tests that test the function tracer return UNSUPPORTED.

This may be an off-topic, but I'd like to ask the selftest for tools.
Currently tools/testing/selftests tests the kernel itself, but
there are many tools under tools/, like perf too.

Those are not configured by the kconfig, but selftests are also needed
for tools. I have a runtests script which is just a bit modified
ftracetest for perf-probe. I'd like to integrate it to selftests
but I'm not sure that is a scope of kselftests.

> Perhaps we should have a central location that each test needs to add
> the required configuration for it to be properly tested. Then if users
> want to test various subsystems, they would look in this location for
> the proper configs (be it a directory that has files of the tests they
> represent, and contain the configs needed). Then there should be no
> real barrier for people to run these tests.

/proc/kconfig[.gz]? I think we can add a list of required kconfigs
for each testcase and indicate it. Moreover, we can include it as
a part of kconfig and introduce CONFIG_KSELFTEST to enable those
configs :)

Thank you,

> 
> Of course if the test requires certain hardware, or a file system, then
> that should be properly documented.
> 
> -- Steve
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
> 


-- 
Masami HIRAMATSU
Linux Technology Research Center, System Productivity Research Dept.
Center for Technology Innovation - Systems Engineering
Hitachi, Ltd., Research & Development Group
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Steven Rostedt]

On Thu, 09 Jul 2015 19:24:55 +0900
Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:

> /proc/kconfig[.gz]? I think we can add a list of required kconfigs
> for each testcase and indicate it. Moreover, we can include it as
> a part of kconfig and introduce CONFIG_KSELFTEST to enable those
> configs :)
> 
> 

I like the idea of having each test a separate config. We could have a
config called "ALL TESTS" and selects all of them, for those that want
to test all tests. But for each test, it would select the needed config
options to test everything for that specific test.

-- Steve

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Guenter Roeck]

On Thu, Jul 09, 2015 at 02:23:53PM +1000, Michael Ellerman wrote:
> On Wed, 2015-07-08 at 09:40 -0700, Kevin Hilman wrote:
> >
> > Speaking for kernelci.org... with limited time/resources, we'd like to
> > focus less on building (others are doing this faster/better) and more on
> > boot testing across a wide variety of hardware (e.g for the latest -next
> > build, we did ~430 boots on 88 unique boards covering 23 different
> > SoCs[1]).  We're also in the process of automating kselftest runs on our
> > boards.
> > 
> > If we could consume the output of other builders, we'd happily do that
> > instead of doing our own builds.  Ideally, the builders should produce
> > some sort of machine readable data with the build artifacts.. 
> 
> Unfortunately providing build artifacts means you're distributing binaries, and
> so life just got complicated. At least for folks like me who work at a company
> with too many lawyers.
> 
Never thought about that aspect, but you are right, there would be some hoops
to go through if the test infrastructure runs in a corporate environment.
Should not be too difficult, though, at least if the lawyers are reasonable.
And if not, maybe they just need a different set of guildelines.

Guenter

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Alexey Dobriyan]

On Thu, Jul 9, 2015 at 1:24 PM, Masami Hiramatsu
<masami.hiramatsu.pt@hitachi.com> wrote:

> This may be an off-topic, but I'd like to ask the selftest for tools.
> Currently tools/testing/selftests tests the kernel itself, but
> there are many tools under tools/, like perf too.
>
> Those are not configured by the kconfig, but selftests are also needed
> for tools. I have a runtests script which is just a bit modified
> ftracetest for perf-probe. I'd like to integrate it to selftests
> but I'm not sure that is a scope of kselftests.

This confusion is partially created by peculiar place where people
who wrote testsuite put it.

Gentlemen,
testsuite should be first class citizen in toplevel test/ directory,
command to run it should be "make test" not "make kselftest".
Only placing it in very visible place and using names which are intuitive
and familiar from userspace (git's t/ directory, glibc "make test") will give
hope that other developers will notice it and start using and improving it.
Excuse me, but tools/testing/selftests is hopeless.

>> Perhaps we should have a central location that each test needs to add
>> the required configuration for it to be properly tested. Then if users
>> want to test various subsystems, they would look in this location for
>> the proper configs (be it a directory that has files of the tests they
>> represent, and contain the configs needed). Then there should be no
>> real barrier for people to run these tests.
>
> /proc/kconfig[.gz]? I think we can add a list of required kconfigs
> for each testcase and indicate it. Moreover, we can include it as
> a part of kconfig and introduce CONFIG_KSELFTEST to enable those
> configs :)

I think primary use case is this:
* user builds and reboots into kernel with his custom config,
* user runs "make test" from fresh build directory,
* test harness runs everything runnable and maybe reports necessary
  config options to run more

/proc/kconfig.gz should be kept strictly for runtime config.

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Shuah Khan]

On 07/10/2015 04:39 AM, Alexey Dobriyan wrote:
> On Thu, Jul 9, 2015 at 1:24 PM, Masami Hiramatsu
> <masami.hiramatsu.pt@hitachi.com> wrote:
> 
>> This may be an off-topic, but I'd like to ask the selftest for tools.
>> Currently tools/testing/selftests tests the kernel itself, but
>> there are many tools under tools/, like perf too.
>>
>> Those are not configured by the kconfig, but selftests are also needed
>> for tools. I have a runtests script which is just a bit modified
>> ftracetest for perf-probe. I'd like to integrate it to selftests
>> but I'm not sure that is a scope of kselftests.
> 
> This confusion is partially created by peculiar place where people
> who wrote testsuite put it.
> 
> Gentlemen,
> testsuite should be first class citizen in toplevel test/ directory,
> command to run it should be "make test" not "make kselftest".
> Only placing it in very visible place and using names which are intuitive
> and familiar from userspace (git's t/ directory, glibc "make test") will give
> hope that other developers will notice it and start using and improving it.
> Excuse me, but tools/testing/selftests is hopeless.

selftests are intended for kernel developers primarily. If developers
and users don't want to figure out what are the ways to test, then it
doesn't matter what the option is named. I would like to hear some
concrete data on why naming test would make it lot more usable.

> 
>>> Perhaps we should have a central location that each test needs to add
>>> the required configuration for it to be properly tested. Then if users
>>> want to test various subsystems, they would look in this location for
>>> the proper configs (be it a directory that has files of the tests they
>>> represent, and contain the configs needed). Then there should be no
>>> real barrier for people to run these tests.
>>
>> /proc/kconfig[.gz]? I think we can add a list of required kconfigs
>> for each testcase and indicate it. Moreover, we can include it as
>> a part of kconfig and introduce CONFIG_KSELFTEST to enable those
>> configs :)

I don't believe adding yet another kernel config option especially
for KSELFTEST is a good idea.

> 
> I think primary use case is this:
> * user builds and reboots into kernel with his custom config,
> * user runs "make test" from fresh build directory,
> * test harness runs everything runnable and maybe reports necessary
>   config options to run more
> 
> /proc/kconfig.gz should be kept strictly for runtime config.
> 

We do have ktest for that. Maybe ktest could include kselftest
run in its default boot test.

thanks,
-- Shuah

-- 
Shuah Khan
Sr. Linux Kernel Developer
Open Source Innovation Group
Samsung Research America (Silicon Valley)
shuahkh@osg.samsung.com | (970) 217-8978

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Alexey Dobriyan]

On Fri, Jul 10, 2015 at 5:02 PM, Shuah Khan <shuahkh@osg.samsung.com> wrote:
> On 07/10/2015 04:39 AM, Alexey Dobriyan wrote:
>> On Thu, Jul 9, 2015 at 1:24 PM, Masami Hiramatsu
>> <masami.hiramatsu.pt@hitachi.com> wrote:
>>
>>> This may be an off-topic, but I'd like to ask the selftest for tools.
>>> Currently tools/testing/selftests tests the kernel itself, but
>>> there are many tools under tools/, like perf too.
>>>
>>> Those are not configured by the kconfig, but selftests are also needed
>>> for tools. I have a runtests script which is just a bit modified
>>> ftracetest for perf-probe. I'd like to integrate it to selftests
>>> but I'm not sure that is a scope of kselftests.
>>
>> This confusion is partially created by peculiar place where people
>> who wrote testsuite put it.
>>
>> Gentlemen,
>> testsuite should be first class citizen in toplevel test/ directory,
>> command to run it should be "make test" not "make kselftest".
>> Only placing it in very visible place and using names which are intuitive
>> and familiar from userspace (git's t/ directory, glibc "make test") will give
>> hope that other developers will notice it and start using and improving it.
>> Excuse me, but tools/testing/selftests is hopeless.
>
> selftests are intended for kernel developers primarily. If developers
> and users don't want to figure out what are the ways to test, then it
> doesn't matter what the option is named. I would like to hear some
> concrete data on why naming test would make it lot more usable.

Not usable, but visible. Top level implies "important", 3 directories deep
implies some obscure driver you didn't even hear about.

And it's easier to type.

git gets it right:

test: all
        $(MAKE) -C t/ all

    Alexey

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Steven Rostedt]

On Fri, 10 Jul 2015 08:02:14 -0600
Shuah Khan <shuahkh@osg.samsung.com> wrote:

> > 
> > I think primary use case is this:
> > * user builds and reboots into kernel with his custom config,
> > * user runs "make test" from fresh build directory,
> > * test harness runs everything runnable and maybe reports necessary
> >   config options to run more
> > 
> > /proc/kconfig.gz should be kept strictly for runtime config.
> > 
> 
> We do have ktest for that. Maybe ktest could include kselftest
> run in its default boot test.

I could have an option to have ktest run the kselftests, but the user
would still need to be able to update configs to tell ktest where the
tests are. Ktest doesn't assume to be run from the repo (I never do
that).

-- Steve

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Shuah Khan]

On 07/10/2015 09:05 AM, Steven Rostedt wrote:
> On Fri, 10 Jul 2015 08:02:14 -0600
> Shuah Khan <shuahkh@osg.samsung.com> wrote:
> 
>>>
>>> I think primary use case is this:
>>> * user builds and reboots into kernel with his custom config,
>>> * user runs "make test" from fresh build directory,
>>> * test harness runs everything runnable and maybe reports necessary
>>>   config options to run more
>>>
>>> /proc/kconfig.gz should be kept strictly for runtime config.
>>>
>>
>> We do have ktest for that. Maybe ktest could include kselftest
>> run in its default boot test.
> 
> I could have an option to have ktest run the kselftests, but the user
> would still need to be able to update configs to tell ktest where the
> tests are. Ktest doesn't assume to be run from the repo (I never do
> that).
> 

Right. Thanks for clarifying it. We have a small number of tests that
depend on a specific kernel config option. Those tests are supposed to
exit gracefully and let the rest of the tests run when the config option
they depend on, isn't enabled. User needs to make a conscious choice on
which tests they want to run. In other words, it is users responsibility
to ensure that the kernel is built correctly to run a desired test. Just
like ktest, the responsibility to pick the right config falls outside
the kselftest framework. Kselftest run will test the parts of the kernel
it can test based on the current configuration.

thanks,
-- Shuah

-- 
Shuah Khan
Sr. Linux Kernel Developer
Open Source Innovation Group
Samsung Research America (Silicon Valley)
shuahkh@osg.samsung.com | (970) 217-8978

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Fengguang Wu]

Hi Geert,

On Wed, Jul 08, 2015 at 10:37:50AM +0200, Geert Uytterhoeven wrote:
> Hi Dan,
> 
> On Wed, Jul 8, 2015 at 9:54 AM, Dan Carpenter <dan.carpenter@oracle.com> wrote:
> > Doesn't the 0day system obsolete the "Build regression" emails?  It
> 
> I think kisskb builds more/different configs.
> 
> Is there are list of configs and trees covered by the 0day system available?

Apart from randconfig and all(yes|no|def|mod) configs, the
comprehensive list of configs can be found in arch/*/configs

        $ find arch/*/configs -type f | wc -l
        461

0-day tests them all. Since there are so many of them, only 10% are
run immediately after git pushes aiming for 1-hour respond time. The
others will be tested in machine idle time and see longer delay.

The code coverage is now 543 git trees. Here is the comprehensive list:

https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/tree/repo/linux

$ cat repo/linux/ext4
url: git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4.git

I'm always seeking to add more trees. Patches to add new URLs and
tests are highly appreciated!

Thanks,
Fengguang

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Fengguang Wu]

On Wed, Jul 08, 2015 at 10:52:07AM +0100, Mark Brown wrote:
> On Wed, Jul 08, 2015 at 10:54:09AM +0300, Dan Carpenter wrote:
> 
> > Doesn't the 0day system obsolete the "Build regression" emails?  It
> > should be sending emails to everyone introducing new build warnings.
> 
> 0day only covers some configurations

Last year it has extended kconfig coverage to all arch/*/configs/*.
Adding defconfigs there will auto add them to 0day. :)

> and doesn't always manage to kick
> in (I know I've had stuff come in via other channels sometimes, it seems
> to depend on loading).

If so, it should be considered a bug. Feel free to report such cases
to me.

If 0day failed to catch the error, it might be due to

- regressions in 0day itself -- it's still in active development,
  in the past year I did 1k patches to 0day build/boot scripts and
  2k patches to the follow up LKP (Linux Kernel Performance) tests.

- the "report once" logic, it's tricky and can possibly hide issues

- failed bisect, rare but possible

- machine hang, network/disk fails, etc. maintenance incidents

"Loading" may add latency, however it's not the cause to miss errors.

> > Are people ignoring them?
> 
> They're not reliably followed through on, no, and one of the things with
> 0day is that it just generates a one time report so if things don't get
> followed up on then that's that.  A regular "these are all the issues"
> mail helps chase down those issues.

0day has such report type. It will be sent after each git push (unless
you push too quickly) and it looks like this. Just drop me a note and
list the git trees/branches you wish to receive such notice emails.

 Subject: [amirv:for-upstream] b132dcd12e0ab2a49ae5b02b5549cb65408a96ef BUILD DONE

git://flatbed.openfabrics.org/~amirv/linux.git  for-upstream
b132dcd12e0ab2a49ae5b02b5549cb65408a96ef  IB/mlx4: Use correct device for XRC [ib-next:issue_511091]
                                                                                                       
drivers/infiniband/core/cache.c:558:9: sparse: too many arguments for function __builtin_expect
drivers/infiniband/core/device.c:176:9: sparse: too many arguments for function __builtin_expect
drivers/infiniband/core/device.c:767:4-21: code aligned with following code on line 768

Error ids grouped by kconfigs:

recent_errors
├── i386-allmodconfig
│   └── drivers-infiniband-core-device.c:code-aligned-with-following-code-on-line
└── x86_64-allmodconfig
    ├── drivers-infiniband-core-cache.c:sparse:too-many-arguments-for-function-__builtin_expect
    └── drivers-infiniband-core-device.c:sparse:too-many-arguments-for-function-__builtin_expect

elapsed time: 53m

configs tested: 83

powerpc                     tqm8548_defconfig
powerpc                     tqm8555_defconfig
um                               alldefconfig
i386                     randconfig-a0-201527
x86_64                            allnoconfig
sh                            titan_defconfig
sh                          rsk7269_defconfig
sh                  sh7785lcr_32bit_defconfig
sh                                allnoconfig
i386                             allyesconfig
arm                                    sa1100
arm                         at91_dt_defconfig
arm                               allnoconfig
arm                                   samsung
...

Thanks,
Fengguang

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1685 bytes --]

On Sun, Jul 12, 2015 at 07:15:47PM +0800, Fengguang Wu wrote:

> If 0day failed to catch the error, it might be due to

> - regressions in 0day itself -- it's still in active development,
>   in the past year I did 1k patches to 0day build/boot scripts and
>   2k patches to the follow up LKP (Linux Kernel Performance) tests.

> - the "report once" logic, it's tricky and can possibly hide issues

> - failed bisect, rare but possible

I think some of the issues have been due to bisection getting confused
by issues appearing in merge commits but ICBW.

> - machine hang, network/disk fails, etc. maintenance incidents

> "Loading" may add latency, however it's not the cause to miss errors.

Latency seems like it might be an issue here, when I say I'm not seeing
things that's issues coming up in the -next build before they're
reported by 0day so they might well get fixed before 0day catches up.

> > > Are people ignoring them?

> > They're not reliably followed through on, no, and one of the things with
> > 0day is that it just generates a one time report so if things don't get
> > followed up on then that's that.  A regular "these are all the issues"
> > mail helps chase down those issues.

> 0day has such report type. It will be sent after each git push (unless
> you push too quickly) and it looks like this. Just drop me a note and
> list the git trees/branches you wish to receive such notice emails.

For me personally it'd be more interesting to be able to get them on
demand (eg, from a web page) than e-mailed, or e-mailed by a human
(possibly with fixes!).  The kernelci.org reporting does a lot of this
but doesn't cover anything except raw compiler warnings.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Fengguang Wu]

On Mon, Jul 13, 2015 at 07:34:41PM +0100, Mark Brown wrote:
> On Sun, Jul 12, 2015 at 07:15:47PM +0800, Fengguang Wu wrote:
> 
> > If 0day failed to catch the error, it might be due to
> 
> > - regressions in 0day itself -- it's still in active development,
> >   in the past year I did 1k patches to 0day build/boot scripts and
> >   2k patches to the follow up LKP (Linux Kernel Performance) tests.
> 
> > - the "report once" logic, it's tricky and can possibly hide issues
> 
> > - failed bisect, rare but possible
> 
> I think some of the issues have been due to bisection getting confused
> by issues appearing in merge commits but ICBW.
> 
> > - machine hang, network/disk fails, etc. maintenance incidents
> 
> > "Loading" may add latency, however it's not the cause to miss errors.
> 
> Latency seems like it might be an issue here, when I say I'm not seeing
> things that's issues coming up in the -next build before they're
> reported by 0day so they might well get fixed before 0day catches up.

Ah linux-next happen to have lower priority and can take many hours to
finish. I'll increase its priority due to its importance.

> > > > Are people ignoring them?
> 
> > > They're not reliably followed through on, no, and one of the things with
> > > 0day is that it just generates a one time report so if things don't get

The real policy is in fact a bit more smart than "report-once".

Build errors will be auto re-reported if it's still not fixed
after 10 days when the branch that introduced the error is updated.

Warnings won't be auto re-reported.

> > > followed up on then that's that.  A regular "these are all the issues"
> > > mail helps chase down those issues.
> 
> > 0day has such report type. It will be sent after each git push (unless
> > you push too quickly) and it looks like this. Just drop me a note and
> > list the git trees/branches you wish to receive such notice emails.
> 
> For me personally it'd be more interesting to be able to get them on
> demand (eg, from a web page) than e-mailed, or e-mailed by a human
> (possibly with fixes!).  The kernelci.org reporting does a lot of this
> but doesn't cover anything except raw compiler warnings.

It should be mostly equivalent if you direct such emails to a local
mbox and check it on demand. :)

For example, setup .procmailrc rule like this:

:0:
* ^Subject: \[.*\] ........................................ BUILD
build-complete-notification

Thanks,
Fengguang

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1090 bytes --]

On Tue, Jul 14, 2015 at 10:22:41PM +0800, Fengguang Wu wrote:
> On Mon, Jul 13, 2015 at 07:34:41PM +0100, Mark Brown wrote:

> > Latency seems like it might be an issue here, when I say I'm not seeing
> > things that's issues coming up in the -next build before they're
> > reported by 0day so they might well get fixed before 0day catches up.

> Ah linux-next happen to have lower priority and can take many hours to
> finish. I'll increase its priority due to its importance.

This was actually things in my trees that get merged into -next (or
which Stephen tries to merge into -next and finds problems with) prior
to being reported by 0day.

> > For me personally it'd be more interesting to be able to get them on
> > demand (eg, from a web page) than e-mailed, or e-mailed by a human
> > (possibly with fixes!).  The kernelci.org reporting does a lot of this
> > but doesn't cover anything except raw compiler warnings.

> It should be mostly equivalent if you direct such emails to a local
> mbox and check it on demand. :)

Well, they don't go away automatically when fixed then...

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Fengguang Wu]

On Tue, Jul 14, 2015 at 04:38:31PM +0100, Mark Brown wrote:
> On Tue, Jul 14, 2015 at 10:22:41PM +0800, Fengguang Wu wrote:
> > On Mon, Jul 13, 2015 at 07:34:41PM +0100, Mark Brown wrote:
> 
> > > Latency seems like it might be an issue here, when I say I'm not seeing
> > > things that's issues coming up in the -next build before they're
> > > reported by 0day so they might well get fixed before 0day catches up.
> 
> > Ah linux-next happen to have lower priority and can take many hours to
> > finish. I'll increase its priority due to its importance.
> 
> This was actually things in my trees that get merged into -next (or
> which Stephen tries to merge into -next and finds problems with) prior
> to being reported by 0day.

I feel sorry about that and would like to improve. When it happen in
future, feel free to forward me of the missed/delayed error.  Then I
can try to root cause case-by-case.

> > > For me personally it'd be more interesting to be able to get them on
> > > demand (eg, from a web page) than e-mailed, or e-mailed by a human
> > > (possibly with fixes!).  The kernelci.org reporting does a lot of this
> > > but doesn't cover anything except raw compiler warnings.
> 
> > It should be mostly equivalent if you direct such emails to a local
> > mbox and check it on demand. :)
> 
> Well, they don't go away automatically when fixed then...

You may just check the latest build compete notification email.
It'll contain the latest information.

Thanks,
Fengguang

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Mel Gorman]

On Tue, Jul 07, 2015 at 06:18:20PM +0100, Mark Brown wrote:
> On Tue, Jul 07, 2015 at 08:25:21AM -0700, Guenter Roeck wrote:
> > On 07/07/2015 02:24 AM, Mark Brown wrote:
> 
> > >The main things I'm aware of that are happening at the minute are
> > >kselftest development, the 0day tester, plus kernelci.org and the other
> > >build and boot/test bots that are running against various trees.
> 
> > Maybe list all known ones as a start ?
> 
> Off the top of my head the automated ones I'm aware of are Olof's build
> & boot test, Dan running smatch and I think some other static analysis
> stuff, someone (not sure who?) running some coccinelle stuff, Coverity
> and I've got a builder too.
> 

There is also Marvin which has existed in some shape or form since November
2013. It checks once a month for new releases and executes a battery of
tests on them across a range of machines. The primary purpose of it is
performance verification and the tests are typically more complex than
what I'd expect to see in kselftest. There are small amounts of overlap
with 0-day but generally it's expected that Marvin runs tests that are more
long-lived. Unlike 0-day, it also does not automatically notify people about
regressions as some verification work is often required and I did not want
it generating noise in any inbox.  Technically, it does support automatic
bisection but it's something I trigger manually when I confirm a problem
is a performance regression and cannot quickly identify the root cause.

It actually has been publishing reports for several months
now but I never mentioned it on the lists. I wrote up
some details after reading this thread and posted it at
http://www.csn.ul.ie/~mel/blog/index.php?/archives/23-Continual-testing-of-mainline-kernels.html

If there was a workshop on testing then I'd be interested in attending and
discussing what Marvin does if there was interest. Right now, performance
tends to my area of interest so I'd be interesting is discussing if there
are areas we are continually getting worse at that are slipping through
the cracks. Chris proposed a topic in this general area that I think would
be useful. I've only started looking at mainline kernel performance again
recently and right now, I'm not aware of a single area where we are getting
consistently worse. More commonly I see cases where we create problems
and then later cover them up by fixing something else in the general
area. Any time I find problems, it's a simple matter of programming and
time to fix them but it'd be useful to hear what other peoples recent
experiences have been.

-- 
Mel Gorman
SUSE Labs

Re: [Ksummit-discuss] [CORE TOPIC] Testing

[Shuah Khan]

On 07/20/2015 09:53 AM, Mel Gorman wrote:
> On Tue, Jul 07, 2015 at 06:18:20PM +0100, Mark Brown wrote:
>> On Tue, Jul 07, 2015 at 08:25:21AM -0700, Guenter Roeck wrote:
>>> On 07/07/2015 02:24 AM, Mark Brown wrote:
>>
>>>> The main things I'm aware of that are happening at the minute are
>>>> kselftest development, the 0day tester, plus kernelci.org and the other
>>>> build and boot/test bots that are running against various trees.
>>
>>> Maybe list all known ones as a start ?
>>
>> Off the top of my head the automated ones I'm aware of are Olof's build
>> & boot test, Dan running smatch and I think some other static analysis
>> stuff, someone (not sure who?) running some coccinelle stuff, Coverity
>> and I've got a builder too.
>>
> 
> There is also Marvin which has existed in some shape or form since November
> 2013. It checks once a month for new releases and executes a battery of
> tests on them across a range of machines. The primary purpose of it is
> performance verification and the tests are typically more complex than
> what I'd expect to see in kselftest. There are small amounts of overlap
> with 0-day but generally it's expected that Marvin runs tests that are more
> long-lived. Unlike 0-day, it also does not automatically notify people about
> regressions as some verification work is often required and I did not want
> it generating noise in any inbox.  Technically, it does support automatic
> bisection but it's something I trigger manually when I confirm a problem
> is a performance regression and cannot quickly identify the root cause.
> 
> It actually has been publishing reports for several months
> now but I never mentioned it on the lists. I wrote up
> some details after reading this thread and posted it at
> http://www.csn.ul.ie/~mel/blog/index.php?/archives/23-Continual-testing-of-mainline-kernels.html
> 
> If there was a workshop on testing then I'd be interested in attending and
> discussing what Marvin does if there was interest. Right now, performance
> tends to my area of interest so I'd be interesting is discussing if there
> are areas we are continually getting worse at that are slipping through
> the cracks. Chris proposed a topic in this general area that I think would
> be useful. I've only started looking at mainline kernel performance again
> recently and right now, I'm not aware of a single area where we are getting
> consistently worse. More commonly I see cases where we create problems
> and then later cover them up by fixing something else in the general
> area. Any time I find problems, it's a simple matter of programming and
> time to fix them but it'd be useful to hear what other peoples recent
> experiences have been.
> 

If there is enough interest, I can pull together a session on testing
at the kernel summit. We can discuss kselftest future plans as well as
adding performance tests into the kselftest, and discuss kselftest use
in auotomated test environments.

thanks,
-- Shuahs

-- 
Shuah Khan
Sr. Linux Kernel Developer
Open Source Innovation Group
Samsung Research America (Silicon Valley)
shuahkh@osg.samsung.com | (970) 217-8978