Wiki for automatic reports / fixes

Wiki for automatic reports / fixes

[Luis R. Rodriguez]

Fengguang, Dan, Mel,

Valentin has some open source tools that uses some SAT solver the
backend to hunt for dead, the undertaker tool [0] inspects Kconfig,
and uses SAT solver to do this work. A while ago it was determined
that this code could not be merged upstream for a variety of reasons,
but mostly due to the fact that it was R&D work and it was under GPLv3
[1]. I know a few of you have other testing tools which helps find
other sorts of issues. To my knowledge automated reports for these may
be expected readily these days by developers from 0-day bot, and for
smatch issues, but its unclear what we should expect from the other
ones. There is also some pending ideas over a few of these other
testing frameworks, for instance a recent discussion was started to
evaluate the potential for integration of Xen into the 0-day bot
testing scheme. While discussing expectations and information about
reports over these with Valentin it occurred to me information about
all these may be scattered separately and some developers may be
surprised when they first get reports / fixes from these sorts of
testing systems and that perhaps it may be useful if we had a single
wiki entry point where we could refer folks to the different ongoing
testing infrastructures out there working upstream.

If we could piggy back off of an already existing wiki then great, but
if not I was thinking something off of wiki.kernel.org might be good.
How about tests.wiki.kernel.org ? If such projects don't have a wiki
they could perhaps use pages off of tests.wiki.kernel.org to elaborate
and set expectations straight. Thoughts?

[0] http://vamos.informatik.uni-erlangen.de/trac/undertake
[1] https://lwn.net/Articles/617383/

 Luis

Re: Wiki for automatic reports / fixes

[Ian Jackson]

Luis R. Rodriguez writes ("Wiki for automatic reports / fixes"):
[...]
>  While discussing expectations and information about
> reports over these with Valentin it occurred to me information about
> all these may be scattered separately and some developers may be
> surprised when they first get reports / fixes from these sorts of
> testing systems and that perhaps it may be useful if we had a single
> wiki entry point where we could refer folks to the different ongoing
> testing infrastructures out there working upstream.
> 
> If we could piggy back off of an already existing wiki then great, but
> if not I was thinking something off of wiki.kernel.org might be good.
> How about tests.wiki.kernel.org ? If such projects don't have a wiki
> they could perhaps use pages off of tests.wiki.kernel.org to elaborate
> and set expectations straight. Thoughts?

To clarify what I think you are suggesting, is to create a new wiki or
wiki page which gives information about automatic tests that are
performed on upstream (or going-upstream) Linux branches ?

I think this is a good idea.  I'm not sure how much information we
need for each tester, but a page for each would be about right.

Ian.

Re: Wiki for automatic reports / fixes

[Luis R. Rodriguez]

On Mon, Oct 5, 2015 at 9:56 AM, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> Luis R. Rodriguez writes ("Wiki for automatic reports / fixes"):
> [...]
>>  While discussing expectations and information about
>> reports over these with Valentin it occurred to me information about
>> all these may be scattered separately and some developers may be
>> surprised when they first get reports / fixes from these sorts of
>> testing systems and that perhaps it may be useful if we had a single
>> wiki entry point where we could refer folks to the different ongoing
>> testing infrastructures out there working upstream.
>>
>> If we could piggy back off of an already existing wiki then great, but
>> if not I was thinking something off of wiki.kernel.org might be good.
>> How about tests.wiki.kernel.org ? If such projects don't have a wiki
>> they could perhaps use pages off of tests.wiki.kernel.org to elaborate
>> and set expectations straight. Thoughts?
>
> To clarify what I think you are suggesting, is to create a new wiki or
> wiki page which gives information about automatic tests that are
> performed on upstream (or going-upstream) Linux branches ?

That's right, as it stands we have a slew of folks doing a series of
battery of tests on either linux-next or other branches, and
developers / maintainers get e-mails about this. Typically one becomes
aware of these tests through experience and in dealing with reports
but other times one may not even be aware of ongoing effort on this
front, such was the case of Valentin's dead code analysis with
undertaker. Knowing what existing work is being done can and could be
used can also save people from re-inventing the wheel, but also and
most importantly collaborate.

> I think this is a good idea.  I'm not sure how much information we
> need for each tester, but a page for each would be about right.

Sure, I figure if each tester framework has its own dedicated page we
can at least refer to it, but a basic page which describes general
coverage / mailing lists / contact info / and what to expect might be
useful. As it stands most of this is just tribal knowledge.

 Luis

Re: Wiki for automatic reports / fixes

[Luis R. Rodriguez]

On Mon, Oct 5, 2015 at 10:03 AM, Luis R. Rodriguez
<mcgrof@do-not-panic.com> wrote:
> On Mon, Oct 5, 2015 at 9:56 AM, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
>> Luis R. Rodriguez writes ("Wiki for automatic reports / fixes"):
>> [...]
>>>  While discussing expectations and information about
>>> reports over these with Valentin it occurred to me information about
>>> all these may be scattered separately and some developers may be
>>> surprised when they first get reports / fixes from these sorts of
>>> testing systems and that perhaps it may be useful if we had a single
>>> wiki entry point where we could refer folks to the different ongoing
>>> testing infrastructures out there working upstream.
>>>
>>> If we could piggy back off of an already existing wiki then great, but
>>> if not I was thinking something off of wiki.kernel.org might be good.
>>> How about tests.wiki.kernel.org ? If such projects don't have a wiki
>>> they could perhaps use pages off of tests.wiki.kernel.org to elaborate
>>> and set expectations straight. Thoughts?
>>
>> To clarify what I think you are suggesting, is to create a new wiki or
>> wiki page which gives information about automatic tests that are
>> performed on upstream (or going-upstream) Linux branches ?
>
> That's right, as it stands we have a slew of folks doing a series of
> battery of tests on either linux-next or other branches, and
> developers / maintainers get e-mails about this. Typically one becomes
> aware of these tests through experience and in dealing with reports
> but other times one may not even be aware of ongoing effort on this
> front, such was the case of Valentin's dead code analysis with
> undertaker. Knowing what existing work is being done can and could be
> used can also save people from re-inventing the wheel, but also and
> most importantly collaborate.
>
>> I think this is a good idea.  I'm not sure how much information we
>> need for each tester, but a page for each would be about right.
>
> Sure, I figure if each tester framework has its own dedicated page we
> can at least refer to it, but a basic page which describes general
> coverage / mailing lists / contact info / and what to expect might be
> useful. As it stands most of this is just tribal knowledge.

OK I'll poke and see if we can get this created.

 Luis

Re: Wiki for automatic reports / fixes

[Luis R. Rodriguez]

On Mon, Oct 12, 2015 at 12:55 PM, Luis R. Rodriguez
<mcgrof@do-not-panic.com> wrote:
> On Mon, Oct 5, 2015 at 10:03 AM, Luis R. Rodriguez
> <mcgrof@do-not-panic.com> wrote:
>> On Mon, Oct 5, 2015 at 9:56 AM, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
>>> Luis R. Rodriguez writes ("Wiki for automatic reports / fixes"):
>>> [...]
>>>>  While discussing expectations and information about
>>>> reports over these with Valentin it occurred to me information about
>>>> all these may be scattered separately and some developers may be
>>>> surprised when they first get reports / fixes from these sorts of
>>>> testing systems and that perhaps it may be useful if we had a single
>>>> wiki entry point where we could refer folks to the different ongoing
>>>> testing infrastructures out there working upstream.
>>>>
>>>> If we could piggy back off of an already existing wiki then great, but
>>>> if not I was thinking something off of wiki.kernel.org might be good.
>>>> How about tests.wiki.kernel.org ? If such projects don't have a wiki
>>>> they could perhaps use pages off of tests.wiki.kernel.org to elaborate
>>>> and set expectations straight. Thoughts?
>>>
>>> To clarify what I think you are suggesting, is to create a new wiki or
>>> wiki page which gives information about automatic tests that are
>>> performed on upstream (or going-upstream) Linux branches ?
>>
>> That's right, as it stands we have a slew of folks doing a series of
>> battery of tests on either linux-next or other branches, and
>> developers / maintainers get e-mails about this. Typically one becomes
>> aware of these tests through experience and in dealing with reports
>> but other times one may not even be aware of ongoing effort on this
>> front, such was the case of Valentin's dead code analysis with
>> undertaker. Knowing what existing work is being done can and could be
>> used can also save people from re-inventing the wheel, but also and
>> most importantly collaborate.
>>
>>> I think this is a good idea.  I'm not sure how much information we
>>> need for each tester, but a page for each would be about right.
>>
>> Sure, I figure if each tester framework has its own dedicated page we
>> can at least refer to it, but a basic page which describes general
>> coverage / mailing lists / contact info / and what to expect might be
>> useful. As it stands most of this is just tribal knowledge.
>
> OK I'll poke and see if we can get this created.

OK we have a page up now to help track and document (if no
documentation exists) or refer to existing testing efforts for Linux:

https://bottest.wiki.kernel.org/

I went ahead and provided references for those projects that I was
aware of that had a page, otherwise I added some boiler plate
documentation page for them with some initial documentation I just
wrote. Please feel free to use this for your own documentation of your
test suite or refer to your own project if you have one from the top
level page.

 Luis

[Cocci] Wiki for automatic reports / fixes

[Luis R. Rodriguez]

On Mon, Oct 12, 2015 at 12:55 PM, Luis R. Rodriguez
<mcgrof@do-not-panic.com> wrote:
> On Mon, Oct 5, 2015 at 10:03 AM, Luis R. Rodriguez
> <mcgrof@do-not-panic.com> wrote:
>> On Mon, Oct 5, 2015 at 9:56 AM, Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
>>> Luis R. Rodriguez writes ("Wiki for automatic reports / fixes"):
>>> [...]
>>>>  While discussing expectations and information about
>>>> reports over these with Valentin it occurred to me information about
>>>> all these may be scattered separately and some developers may be
>>>> surprised when they first get reports / fixes from these sorts of
>>>> testing systems and that perhaps it may be useful if we had a single
>>>> wiki entry point where we could refer folks to the different ongoing
>>>> testing infrastructures out there working upstream.
>>>>
>>>> If we could piggy back off of an already existing wiki then great, but
>>>> if not I was thinking something off of wiki.kernel.org might be good.
>>>> How about tests.wiki.kernel.org ? If such projects don't have a wiki
>>>> they could perhaps use pages off of tests.wiki.kernel.org to elaborate
>>>> and set expectations straight. Thoughts?
>>>
>>> To clarify what I think you are suggesting, is to create a new wiki or
>>> wiki page which gives information about automatic tests that are
>>> performed on upstream (or going-upstream) Linux branches ?
>>
>> That's right, as it stands we have a slew of folks doing a series of
>> battery of tests on either linux-next or other branches, and
>> developers / maintainers get e-mails about this. Typically one becomes
>> aware of these tests through experience and in dealing with reports
>> but other times one may not even be aware of ongoing effort on this
>> front, such was the case of Valentin's dead code analysis with
>> undertaker. Knowing what existing work is being done can and could be
>> used can also save people from re-inventing the wheel, but also and
>> most importantly collaborate.
>>
>>> I think this is a good idea.  I'm not sure how much information we
>>> need for each tester, but a page for each would be about right.
>>
>> Sure, I figure if each tester framework has its own dedicated page we
>> can at least refer to it, but a basic page which describes general
>> coverage / mailing lists / contact info / and what to expect might be
>> useful. As it stands most of this is just tribal knowledge.
>
> OK I'll poke and see if we can get this created.

OK we have a page up now to help track and document (if no
documentation exists) or refer to existing testing efforts for Linux:

https://bottest.wiki.kernel.org/

I went ahead and provided references for those projects that I was
aware of that had a page, otherwise I added some boiler plate
documentation page for them with some initial documentation I just
wrote. Please feel free to use this for your own documentation of your
test suite or refer to your own project if you have one from the top
level page.

 Luis