[Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Shuah Khan]

I would like to propose a topic to discuss ideas to keep up with Syzbot
bugs on an ongoing basis. Good news is, as of this writing, we have 1276
fixed, 86 in moderation, and 62 fix pending. However, there are 523 open
bugs. A good number of them have been open for longer than 300 days.

The oldest one has been open for 537 days. I plan to take a closer at
the open bugs to identify areas that might need more help.

I have been sending fixes to some of them as I find time like many other
developers. In addition, I included syzbot bug analysis as a required
contribution in the Linux Kernel Mentorship Program application process.
This is for learning debugging skills as well as getting help towards
fixing bugs. It has been successful in getting a few fixes in. My goal
is  to make these efforts a bit more structured to focus on areas that
need help.

There are some challenges, the obvious ones being finding time to
analyze, reproduce, and fix. Reproducers are available in many cases,
and these bugs can be reproduced. Identifying the right fix and fixing
is the overwhelming part with the number of outstanding problems.

My objective for this topic is to explore and identify areas that might
need help in analyzing and fixing bugs in general and especially the
ones that have been open for a while. It would help me channel and focus
the Mentorship Program efforts and my efforts to help the areas in need.

thanks,
-- Shuah

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Laura Abbott]

On 5/30/19 7:30 PM, Shuah Khan wrote:
> I would like to propose a topic to discuss ideas to keep up with Syzbot
> bugs on an ongoing basis. Good news is, as of this writing, we have 1276
> fixed, 86 in moderation, and 62 fix pending. However, there are 523 open
> bugs. A good number of them have been open for longer than 300 days.
> 
> The oldest one has been open for 537 days. I plan to take a closer at
> the open bugs to identify areas that might need more help.
> 
> I have been sending fixes to some of them as I find time like many other
> developers. In addition, I included syzbot bug analysis as a required
> contribution in the Linux Kernel Mentorship Program application process.
> This is for learning debugging skills as well as getting help towards
> fixing bugs. It has been successful in getting a few fixes in. My goal
> is  to make these efforts a bit more structured to focus on areas that
> need help.
> 
> There are some challenges, the obvious ones being finding time to
> analyze, reproduce, and fix. Reproducers are available in many cases,
> and these bugs can be reproduced. Identifying the right fix and fixing
> is the overwhelming part with the number of outstanding problems.
> 
> My objective for this topic is to explore and identify areas that might
> need help in analyzing and fixing bugs in general and especially the
> ones that have been open for a while. It would help me channel and focus
> the Mentorship Program efforts and my efforts to help the areas in need.
> 
> thanks,
> -- Shuah

I'm interested in this topic for not just syzbot but other bug trackers
as well. Fedora gets a steady flow of bugs filed and the three official
maintainers do their best to review but sometimes things slip through,
especially if we hit a time when several of us are out or traveling.
The kernel.org bugzilla also gets a number of bug reports that sometimes
get lost. I'd love to see if there's a process that could work for syzbot
and other high volume kernel trackers.

Thanks,
Laura

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Shuah Khan]

On 5/31/19 6:01 AM, Laura Abbott wrote:
> On 5/30/19 7:30 PM, Shuah Khan wrote:
>> I would like to propose a topic to discuss ideas to keep up with Syzbot
>> bugs on an ongoing basis. Good news is, as of this writing, we have 1276
>> fixed, 86 in moderation, and 62 fix pending. However, there are 523 open
>> bugs. A good number of them have been open for longer than 300 days.
>>
>> The oldest one has been open for 537 days. I plan to take a closer at
>> the open bugs to identify areas that might need more help.
>>
>> I have been sending fixes to some of them as I find time like many other
>> developers. In addition, I included syzbot bug analysis as a required
>> contribution in the Linux Kernel Mentorship Program application process.
>> This is for learning debugging skills as well as getting help towards
>> fixing bugs. It has been successful in getting a few fixes in. My goal
>> is  to make these efforts a bit more structured to focus on areas that
>> need help.
>>
>> There are some challenges, the obvious ones being finding time to
>> analyze, reproduce, and fix. Reproducers are available in many cases,
>> and these bugs can be reproduced. Identifying the right fix and fixing
>> is the overwhelming part with the number of outstanding problems.
>>
>> My objective for this topic is to explore and identify areas that might
>> need help in analyzing and fixing bugs in general and especially the
>> ones that have been open for a while. It would help me channel and focus
>> the Mentorship Program efforts and my efforts to help the areas in need.
>>
>> thanks,
>> -- Shuah
> 
> I'm interested in this topic for not just syzbot but other bug trackers
> as well. Fedora gets a steady flow of bugs filed and the three official
> maintainers do their best to review but sometimes things slip through,
> especially if we hit a time when several of us are out or traveling.
> The kernel.org bugzilla also gets a number of bug reports that sometimes
> get lost. I'd love to see if there's a process that could work for syzbot
> and other high volume kernel trackers.
> 

I am hoping we can look at all bugs as well. Syzbot happens to be just a
good starting for me to start this discussion.

thanks,
-- Shuah

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Leon Romanovsky]

On Fri, May 31, 2019 at 09:56:32AM -0600, Shuah Khan wrote:
> On 5/31/19 6:01 AM, Laura Abbott wrote:
> > On 5/30/19 7:30 PM, Shuah Khan wrote:
> > > I would like to propose a topic to discuss ideas to keep up with Syzbot
> > > bugs on an ongoing basis. Good news is, as of this writing, we have 1276
> > > fixed, 86 in moderation, and 62 fix pending. However, there are 523 open
> > > bugs. A good number of them have been open for longer than 300 days.
> > >
> > > The oldest one has been open for 537 days. I plan to take a closer at
> > > the open bugs to identify areas that might need more help.
> > >
> > > I have been sending fixes to some of them as I find time like many other
> > > developers. In addition, I included syzbot bug analysis as a required
> > > contribution in the Linux Kernel Mentorship Program application process.
> > > This is for learning debugging skills as well as getting help towards
> > > fixing bugs. It has been successful in getting a few fixes in. My goal
> > > is  to make these efforts a bit more structured to focus on areas that
> > > need help.
> > >
> > > There are some challenges, the obvious ones being finding time to
> > > analyze, reproduce, and fix. Reproducers are available in many cases,
> > > and these bugs can be reproduced. Identifying the right fix and fixing
> > > is the overwhelming part with the number of outstanding problems.
> > >
> > > My objective for this topic is to explore and identify areas that might
> > > need help in analyzing and fixing bugs in general and especially the
> > > ones that have been open for a while. It would help me channel and focus
> > > the Mentorship Program efforts and my efforts to help the areas in need.
> > >
> > > thanks,
> > > -- Shuah
> >
> > I'm interested in this topic for not just syzbot but other bug trackers
> > as well. Fedora gets a steady flow of bugs filed and the three official
> > maintainers do their best to review but sometimes things slip through,
> > especially if we hit a time when several of us are out or traveling.
> > The kernel.org bugzilla also gets a number of bug reports that sometimes
> > get lost. I'd love to see if there's a process that could work for syzbot
> > and other high volume kernel trackers.
> >
>
> I am hoping we can look at all bugs as well. Syzbot happens to be just a
> good starting for me to start this discussion.

I would say that single place to see/report/manage all bugs (it can be
per-subsystem or for whole kernel) and ability to set priority will do
the trick.

Maybe, it is worth to integrate syzbot dashboard into kernel bugzilla,
at least for priority.

Thanks

>
> thanks,
> -- Shuah
>
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Kees Cook]

On Fri, May 31, 2019 at 5:02 AM Laura Abbott <labbott@redhat.com> wrote:
>
> On 5/30/19 7:30 PM, Shuah Khan wrote:
> > I would like to propose a topic to discuss ideas to keep up with Syzbot
> > bugs on an ongoing basis. Good news is, as of this writing, we have 1276
> > fixed, 86 in moderation, and 62 fix pending. However, there are 523 open
> > bugs. A good number of them have been open for longer than 300 days.
> >
> > The oldest one has been open for 537 days. I plan to take a closer at
> > the open bugs to identify areas that might need more help.

Many seem to get discussed for a while with people all mostly agreeing
on what needs to be done, but then nothing actually materializes.

Now, part of this, I think, is a lack of an "objective" sense of
priority for the various bugs on the dashboard. Their frequency of
appearance (currently the only thing that really turns red), does not
reflect anything about their severity nor reachability. Severity is
pretty straight forward: stack smashing, use-after-free, etc are way
worse (generally speaking) than crash/hang or WARN(). For
reachability, if we could narrow the scope of things a bit more ("this
is reachable only by init namespace root", or "only reachable via
CAP_NET_ADMIN", or "only with this CONFIG that doesn't appear in any
modern distro .config") that would be great.

> > I have been sending fixes to some of them as I find time like many other
> > developers. In addition, I included syzbot bug analysis as a required
> > contribution in the Linux Kernel Mentorship Program application process.
> > This is for learning debugging skills as well as getting help towards
> > fixing bugs. It has been successful in getting a few fixes in. My goal
> > is  to make these efforts a bit more structured to focus on areas that
> > need help.
> >
> > There are some challenges, the obvious ones being finding time to
> > analyze, reproduce, and fix. Reproducers are available in many cases,
> > and these bugs can be reproduced. Identifying the right fix and fixing
> > is the overwhelming part with the number of outstanding problems.
> >
> > My objective for this topic is to explore and identify areas that might
> > need help in analyzing and fixing bugs in general and especially the
> > ones that have been open for a while. It would help me channel and focus
> > the Mentorship Program efforts and my efforts to help the areas in need.

What I want to find is a way to fund on-going work in this area. It
just does not appear to be working to find funding within various
companies to do this kind of sustained bug-hunting. It's a bit of a
resource conflict: if a company has kernel developers on staff they
tend to want to have them focused on "new things".

> I'm interested in this topic for not just syzbot but other bug trackers
> as well. Fedora gets a steady flow of bugs filed and the three official
> maintainers do their best to review but sometimes things slip through,
> especially if we hit a time when several of us are out or traveling.
> The kernel.org bugzilla also gets a number of bug reports that sometimes
> get lost. I'd love to see if there's a process that could work for syzbot
> and other high volume kernel trackers.

I'd especially love to see deduplication and cross-referencing working
a little more automatically between trackers. "Oh, I've seen that
before over here *link*" by a machine would be quite helpful.

-- 
Kees Cook

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Jan Kara]

On Fri 31-05-19 15:15:55, Kees Cook wrote:
> On Fri, May 31, 2019 at 5:02 AM Laura Abbott <labbott@redhat.com> wrote:
> >
> > On 5/30/19 7:30 PM, Shuah Khan wrote:
> > > I would like to propose a topic to discuss ideas to keep up with Syzbot
> > > bugs on an ongoing basis. Good news is, as of this writing, we have 1276
> > > fixed, 86 in moderation, and 62 fix pending. However, there are 523 open
> > > bugs. A good number of them have been open for longer than 300 days.
> > >
> > > The oldest one has been open for 537 days. I plan to take a closer at
> > > the open bugs to identify areas that might need more help.
> 
> Many seem to get discussed for a while with people all mostly agreeing
> on what needs to be done, but then nothing actually materializes.
> 
> Now, part of this, I think, is a lack of an "objective" sense of
> priority for the various bugs on the dashboard. Their frequency of
> appearance (currently the only thing that really turns red), does not
> reflect anything about their severity nor reachability. Severity is
> pretty straight forward: stack smashing, use-after-free, etc are way
> worse (generally speaking) than crash/hang or WARN(). For
> reachability, if we could narrow the scope of things a bit more ("this
> is reachable only by init namespace root", or "only reachable via
> CAP_NET_ADMIN", or "only with this CONFIG that doesn't appear in any
> modern distro .config") that would be great.

Yes, I think this would be really useful.

> > > I have been sending fixes to some of them as I find time like many other
> > > developers. In addition, I included syzbot bug analysis as a required
> > > contribution in the Linux Kernel Mentorship Program application process.
> > > This is for learning debugging skills as well as getting help towards
> > > fixing bugs. It has been successful in getting a few fixes in. My goal
> > > is  to make these efforts a bit more structured to focus on areas that
> > > need help.
> > >
> > > There are some challenges, the obvious ones being finding time to
> > > analyze, reproduce, and fix. Reproducers are available in many cases,
> > > and these bugs can be reproduced. Identifying the right fix and fixing
> > > is the overwhelming part with the number of outstanding problems.
> > >
> > > My objective for this topic is to explore and identify areas that might
> > > need help in analyzing and fixing bugs in general and especially the
> > > ones that have been open for a while. It would help me channel and focus
> > > the Mentorship Program efforts and my efforts to help the areas in need.
> 
> What I want to find is a way to fund on-going work in this area. It
> just does not appear to be working to find funding within various
> companies to do this kind of sustained bug-hunting. It's a bit of a
> resource conflict: if a company has kernel developers on staff they
> tend to want to have them focused on "new things".

Well, not necessarily only "new things". Distros have kernel people that do
bugfixing but the most focus naturally goes towards bugs customers are
likely to get exposed to. Triaging through syzbot reports when you have
enough bugs your customers actually hit thus does not look very appealing
:) But this goes back to your idea of automated way of classifying bug
priority. Unpriviledged user triggerable kernel oops in MM is going to get
more attention than init namespace root triggerable one in some random
old driver (which is likely to bitrot for long and that's just a reality of
limited resources)...

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Laura Abbott]

On 6/3/19 6:42 AM, Jan Kara wrote:
>> What I want to find is a way to fund on-going work in this area. It
>> just does not appear to be working to find funding within various
>> companies to do this kind of sustained bug-hunting. It's a bit of a
>> resource conflict: if a company has kernel developers on staff they
>> tend to want to have them focused on "new things".
> Well, not necessarily only "new things". Distros have kernel people that do
> bugfixing but the most focus naturally goes towards bugs customers are
> likely to get exposed to. Triaging through syzbot reports when you have
> enough bugs your customers actually hit thus does not look very appealing
> :)  But this goes back to your idea of automated way of classifying bug
> priority. Unpriviledged user triggerable kernel oops in MM is going to get
> more attention than init namespace root triggerable one in some random
> old driver (which is likely to bitrot for long and that's just a reality of
> limited resources)...

Part of it is also a matter of how much time you want to spend. There are
some bugs I can spend 30 minutes looking at and know I can either come up with
the proper fix or write up a report that's likely to get fixed by a maintainer.
Then there are others which look like they could be solvable with more time
that I don't have. The long tail of low priority bugs are the ones that
would be really good candidates for interested community kernel developers
to spend some time on.

I'll also just state straight out that just doing bug triage gets very tedious
after a while. It's valuable work but after a while you aren't actually growing
your skills very much. This might be something that would be best as a yearly
contract position. Of course, people who aren't me might also have more
patience for it long term :)

Thanks,
Laura

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Shuah Khan]

On 5/31/19 4:15 PM, Kees Cook wrote:
> On Fri, May 31, 2019 at 5:02 AM Laura Abbott <labbott@redhat.com> wrote:
>>
>> On 5/30/19 7:30 PM, Shuah Khan wrote:
>>> I would like to propose a topic to discuss ideas to keep up with Syzbot
>>> bugs on an ongoing basis. Good news is, as of this writing, we have 1276
>>> fixed, 86 in moderation, and 62 fix pending. However, there are 523 open
>>> bugs. A good number of them have been open for longer than 300 days.
>>>
>>> The oldest one has been open for 537 days. I plan to take a closer at
>>> the open bugs to identify areas that might need more help.
> 
> Many seem to get discussed for a while with people all mostly agreeing
> on what needs to be done, but then nothing actually materializes.
> 

Right. My experience with a fix I sent is that there is no agreement
on how to fix between maintainers. Hence, it is in limbo.

> Now, part of this, I think, is a lack of an "objective" sense of
> priority for the various bugs on the dashboard. Their frequency of
> appearance (currently the only thing that really turns red), does not
> reflect anything about their severity nor reachability. Severity is
> pretty straight forward: stack smashing, use-after-free, etc are way
> worse (generally speaking) than crash/hang or WARN(). For
> reachability, if we could narrow the scope of things a bit more ("this
> is reachable only by init namespace root", or "only reachable via
> CAP_NET_ADMIN", or "only with this CONFIG that doesn't appear in any
> modern distro .config") that would be great.
> 

Agree. It is very hard to parse the information to determine priority.
I have been looking at how often it can be reproduced. In some cases,
it isn't clear how likely for this bug to be triggered in regular cases.
This is where syzbot adds value that it stresses error cases.

This one bug I am looking at is in an error leg, which is nice. However,
syzbot reports are hard to manage. I found 3 instances of the same bug,
with fix pulled into the mainline associated with one of the 3.

I spent time on Friday to find a quick way to reply to one of them with
list of IDs. It will be great to have a ID associated with these bugs,
so I can go say, hey this fix applies to these IDs.

>>> I have been sending fixes to some of them as I find time like many other
>>> developers. In addition, I included syzbot bug analysis as a required
>>> contribution in the Linux Kernel Mentorship Program application process.
>>> This is for learning debugging skills as well as getting help towards
>>> fixing bugs. It has been successful in getting a few fixes in. My goal
>>> is  to make these efforts a bit more structured to focus on areas that
>>> need help.
>>>
>>> There are some challenges, the obvious ones being finding time to
>>> analyze, reproduce, and fix. Reproducers are available in many cases,
>>> and these bugs can be reproduced. Identifying the right fix and fixing
>>> is the overwhelming part with the number of outstanding problems.
>>>
>>> My objective for this topic is to explore and identify areas that might
>>> need help in analyzing and fixing bugs in general and especially the
>>> ones that have been open for a while. It would help me channel and focus
>>> the Mentorship Program efforts and my efforts to help the areas in need.
> 
> What I want to find is a way to fund on-going work in this area. It
> just does not appear to be working to find funding within various
> companies to do this kind of sustained bug-hunting. It's a bit of a
> resource conflict: if a company has kernel developers on staff they
> tend to want to have them focused on "new things".
> 

Yes. This is my concern as well. This is why I am trying to see if we
can find resources. I would like to start with trying to channel the
resources from the mentorship program for this. I do know that this
adds overhead as well. Maybe some low hanging fruit and easy fix type
bugs can be taken care of using this program and then we can focus our
expert energies on harder bugs.

>> I'm interested in this topic for not just syzbot but other bug trackers
>> as well. Fedora gets a steady flow of bugs filed and the three official
>> maintainers do their best to review but sometimes things slip through,
>> especially if we hit a time when several of us are out or traveling.
>> The kernel.org bugzilla also gets a number of bug reports that sometimes
>> get lost. I'd love to see if there's a process that could work for syzbot
>> and other high volume kernel trackers.
> 

Yes. This is what I am finding. Syzbot or any other bug tracking system
to be effective, it has to be easy to manage and mark bugs as duplicate,
closed, fixed. etc.

thanks,
-- Shuah

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Sasha Levin]

On Fri, May 31, 2019 at 08:01:15AM -0400, Laura Abbott wrote:
>On 5/30/19 7:30 PM, Shuah Khan wrote:
>>I would like to propose a topic to discuss ideas to keep up with Syzbot
>>bugs on an ongoing basis. Good news is, as of this writing, we have 1276
>>fixed, 86 in moderation, and 62 fix pending. However, there are 523 open
>>bugs. A good number of them have been open for longer than 300 days.
>>
>>The oldest one has been open for 537 days. I plan to take a closer at
>>the open bugs to identify areas that might need more help.
>>
>>I have been sending fixes to some of them as I find time like many other
>>developers. In addition, I included syzbot bug analysis as a required
>>contribution in the Linux Kernel Mentorship Program application process.
>>This is for learning debugging skills as well as getting help towards
>>fixing bugs. It has been successful in getting a few fixes in. My goal
>>is  to make these efforts a bit more structured to focus on areas that
>>need help.
>>
>>There are some challenges, the obvious ones being finding time to
>>analyze, reproduce, and fix. Reproducers are available in many cases,
>>and these bugs can be reproduced. Identifying the right fix and fixing
>>is the overwhelming part with the number of outstanding problems.
>>
>>My objective for this topic is to explore and identify areas that might
>>need help in analyzing and fixing bugs in general and especially the
>>ones that have been open for a while. It would help me channel and focus
>>the Mentorship Program efforts and my efforts to help the areas in need.
>>
>>thanks,
>>-- Shuah
>
>I'm interested in this topic for not just syzbot but other bug trackers
>as well. Fedora gets a steady flow of bugs filed and the three official
>maintainers do their best to review but sometimes things slip through,
>especially if we hit a time when several of us are out or traveling.
>The kernel.org bugzilla also gets a number of bug reports that sometimes
>get lost. I'd love to see if there's a process that could work for syzbot
>and other high volume kernel trackers.

Maybe the solution here is to standardize bug reporting bots? A
bugs@kernel.org mailing list which our various bots could report to with
a standard format that'll allow users/maintainers to easily filter it to
get only bugs they care about?

I'd also love to get a concrete solution we could actually go ahead and
try for a year. It doesn't have to be pretty or perfect, but just
something we could experiment with and possibly build upon later? We
keep kicking this can down the road for way too long.

--
Thanks,
Sasha

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Shuah Khan]

On 6/2/19 12:09 PM, Sasha Levin wrote:
> On Fri, May 31, 2019 at 08:01:15AM -0400, Laura Abbott wrote:
>> On 5/30/19 7:30 PM, Shuah Khan wrote:
>>> I would like to propose a topic to discuss ideas to keep up with Syzbot
>>> bugs on an ongoing basis. Good news is, as of this writing, we have 1276
>>> fixed, 86 in moderation, and 62 fix pending. However, there are 523 open
>>> bugs. A good number of them have been open for longer than 300 days.
>>>
>>> The oldest one has been open for 537 days. I plan to take a closer at
>>> the open bugs to identify areas that might need more help.
>>>
>>> I have been sending fixes to some of them as I find time like many other
>>> developers. In addition, I included syzbot bug analysis as a required
>>> contribution in the Linux Kernel Mentorship Program application process.
>>> This is for learning debugging skills as well as getting help towards
>>> fixing bugs. It has been successful in getting a few fixes in. My goal
>>> is  to make these efforts a bit more structured to focus on areas that
>>> need help.
>>>
>>> There are some challenges, the obvious ones being finding time to
>>> analyze, reproduce, and fix. Reproducers are available in many cases,
>>> and these bugs can be reproduced. Identifying the right fix and fixing
>>> is the overwhelming part with the number of outstanding problems.
>>>
>>> My objective for this topic is to explore and identify areas that might
>>> need help in analyzing and fixing bugs in general and especially the
>>> ones that have been open for a while. It would help me channel and focus
>>> the Mentorship Program efforts and my efforts to help the areas in need.
>>>
>>> thanks,
>>> -- Shuah
>>
>> I'm interested in this topic for not just syzbot but other bug trackers
>> as well. Fedora gets a steady flow of bugs filed and the three official
>> maintainers do their best to review but sometimes things slip through,
>> especially if we hit a time when several of us are out or traveling.
>> The kernel.org bugzilla also gets a number of bug reports that sometimes
>> get lost. I'd love to see if there's a process that could work for syzbot
>> and other high volume kernel trackers.
> 
> Maybe the solution here is to standardize bug reporting bots? A
> bugs@kernel.org mailing list which our various bots could report to with
> a standard format that'll allow users/maintainers to easily filter it to
> get only bugs they care about?
> 

Easy to filter and manage is important for usability. I mentioned in my
response Kees that I am finding it hard to update syzbot bug status to
mark them as duplicate and/or fixed. The email interface is nice for
individual bugs, but not very easy when it comes to other actions.

> I'd also love to get a concrete solution we could actually go ahead and
> try for a year. It doesn't have to be pretty or perfect, but just
> something we could experiment with and possibly build upon later? We
> keep kicking this can down the road for way too long.
> 

I am hoping we will come up something concrete.

thanks,
-- Shuah

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Konstantin Ryabitsev]

On Mon, Jun 03, 2019 at 11:25:51AM -0600, Shuah Khan wrote:
>>>I'm interested in this topic for not just syzbot but other bug 
>>>trackers
>>>as well. Fedora gets a steady flow of bugs filed and the three official
>>>maintainers do their best to review but sometimes things slip through,
>>>especially if we hit a time when several of us are out or traveling.
>>>The kernel.org bugzilla also gets a number of bug reports that sometimes
>>>get lost. I'd love to see if there's a process that could work for syzbot
>>>and other high volume kernel trackers.
>>
>>Maybe the solution here is to standardize bug reporting bots? A
>>bugs@kernel.org mailing list which our various bots could report to with
>>a standard format that'll allow users/maintainers to easily filter it to
>>get only bugs they care about?
>>
>
>Easy to filter and manage is important for usability. I mentioned in my
>response Kees that I am finding it hard to update syzbot bug status to
>mark them as duplicate and/or fixed. The email interface is nice for
>individual bugs, but not very easy when it comes to other actions.

May I recommend using a project like git-bug [1] instead of a mailing 
list? I am increasingly worried that with mail services being 
increasingly centralized across 4-5 major providers we'll soon find it 
quite difficult to continue using mailing lists and have them reliably 
deliver mail to all participants. Heck, I've been battling with Gmail 
over the weekend due to it arbitrarily deciding that Robert Richter is 
receiving way too much mail.

Git-bug, on the other hand, is a great internal bug tracker for 
automatically generated bug reports -- it already provides the necessary 
structured platform, is decentralized, easily clonable and offline-able, 
and does not suffer from some mail provider arbitrarily deciding 
throttle its traffic for no obvious reason.

.. [1] https://github.com/MichaelMure/git-bug

-K

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Shuah Khan]

On 6/3/19 12:09 PM, Konstantin Ryabitsev wrote:
> On Mon, Jun 03, 2019 at 11:25:51AM -0600, Shuah Khan wrote:
>>>> I'm interested in this topic for not just syzbot but other bug trackers
>>>> as well. Fedora gets a steady flow of bugs filed and the three official
>>>> maintainers do their best to review but sometimes things slip through,
>>>> especially if we hit a time when several of us are out or traveling.
>>>> The kernel.org bugzilla also gets a number of bug reports that 
>>>> sometimes
>>>> get lost. I'd love to see if there's a process that could work for 
>>>> syzbot
>>>> and other high volume kernel trackers.
>>>
>>> Maybe the solution here is to standardize bug reporting bots? A
>>> bugs@kernel.org mailing list which our various bots could report to with
>>> a standard format that'll allow users/maintainers to easily filter it to
>>> get only bugs they care about?
>>>
>>
>> Easy to filter and manage is important for usability. I mentioned in my
>> response Kees that I am finding it hard to update syzbot bug status to
>> mark them as duplicate and/or fixed. The email interface is nice for
>> individual bugs, but not very easy when it comes to other actions.
> 
> May I recommend using a project like git-bug [1] instead of a mailing 
> list? I am increasingly worried that with mail services being 
> increasingly centralized across 4-5 major providers we'll soon find it 
> quite difficult to continue using mailing lists and have them reliably 
> deliver mail to all participants. Heck, I've been battling with Gmail 
> over the weekend due to it arbitrarily deciding that Robert Richter is 
> receiving way too much mail.
> 
> Git-bug, on the other hand, is a great internal bug tracker for 
> automatically generated bug reports -- it already provides the necessary 
> structured platform, is decentralized, easily clonable and offline-able, 
> and does not suffer from some mail provider arbitrarily deciding 
> throttle its traffic for no obvious reason.
> 
> .. [1] https://github.com/MichaelMure/git-bug
> 

Thanks for the pointer. I will play with this and see how this works.

-- Shuah

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Linus Torvalds]

On Mon, Jun 3, 2019 at 11:10 AM Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
>
> May I recommend using a project like git-bug [1] instead of a mailing
> list?

I'm not a huge fan of mailing lists only for bug handling, but on the
other hand, I do think that a bug handling thing fundamentally needs

 (a) automatic aging out of bug reports

 (b) targeted push notifications (ie ractically speaking - email)

and that the email part is important. And it can't just be a "hey,
something changed". It needs to contain enough information in the
email to give a developer sufficient knowledge that the developer
knows whether it's even worth it looking deeper at the bug, and
looking at the database (in fact, it should contain enough actual
information that for simple issues, the developer should go "D'oh!"
and just fix the bug).

We went through this with syzbot already, where the notification was
too heavy-handed and not useful. It has improved immensely. There's a
back-end with lots more data, but the emails have gone from "illegible
data" to "pretty informational", and I think it improved peoples
reaction to them a lot.

Apparently syzbot doesn't do well on the (a) side though. Honestly,
any system that just keeps things open for 300+ days has something
wrong with it. At some point it needs to be automatically either
closed, or re-notified. The "just keep it around" is simply not an
option.

                  Linus

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Sasha Levin]

On Mon, Jun 03, 2019 at 01:48:22PM -0700, Linus Torvalds wrote:
>On Mon, Jun 3, 2019 at 11:10 AM Konstantin Ryabitsev
><konstantin@linuxfoundation.org> wrote:
>>
>> May I recommend using a project like git-bug [1] instead of a mailing
>> list?
>
>I'm not a huge fan of mailing lists only for bug handling, but on the
>other hand, I do think that a bug handling thing fundamentally needs
>
> (a) automatic aging out of bug reports
>
> (b) targeted push notifications (ie ractically speaking - email)
>
>and that the email part is important. And it can't just be a "hey,
>something changed". It needs to contain enough information in the
>email to give a developer sufficient knowledge that the developer
>knows whether it's even worth it looking deeper at the bug, and
>looking at the database (in fact, it should contain enough actual
>information that for simple issues, the developer should go "D'oh!"
>and just fix the bug).

Would it make sense to do some bikeshedding around how an ideal email
bug report should look like, and then get syzbot and maybe kernelci to
switch to using this new format as an experiment?

A bunch the syzbot/kernelci/etc folks will be at LPC as well, and we'll
have the testing & fuzzing MC going on where this can be discussed and
agreed on once there is a concrete proposal.

>We went through this with syzbot already, where the notification was
>too heavy-handed and not useful. It has improved immensely. There's a
>back-end with lots more data, but the emails have gone from "illegible
>data" to "pretty informational", and I think it improved peoples
>reaction to them a lot.
>
>Apparently syzbot doesn't do well on the (a) side though. Honestly,
>any system that just keeps things open for 300+ days has something
>wrong with it. At some point it needs to be automatically either
>closed, or re-notified. The "just keep it around" is simply not an
>option.

My understanding is that syzbot closes a bug once the bot sees an
annotated commit indicating that it fixes a certain bug. These
annotations get lost/forgotten/etc so bugs remain open even after they
were fixed.

Maybe we could sync up our bugtrackers with the kernel's release cycle?
close all bugs older than a certain kernel version once the
corresponding stable kernel goes EoL?

--
Thanks,
Sasha

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Jiri Kosina]

On Mon, 3 Jun 2019, Sasha Levin wrote:

> > Apparently syzbot doesn't do well on the (a) side though. Honestly, 
> > any system that just keeps things open for 300+ days has something 
> > wrong with it. At some point it needs to be automatically either 
> > closed, or re-notified. The "just keep it around" is simply not an 
> > option.
> 
> My understanding is that syzbot closes a bug once the bot sees an
> annotated commit indicating that it fixes a certain bug. These
> annotations get lost/forgotten/etc so bugs remain open even after they
> were fixed.

A lot of such bugs get fixed (directly or totally indirectly) without 
authors of the fixes being even remotely aware of the syzbot report.

So being able to perform "ok, this is not an issue any more for whatever 
reason, I am closing myself as RESOLVED" operation would be really nice to 
have feature, which I think is currently missing.

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Linus Torvalds]

On Mon, Jun 3, 2019 at 2:15 PM Jiri Kosina <jikos@kernel.org> wrote:
>
> So being able to perform "ok, this is not an issue any more for whatever
> reason, I am closing myself as RESOLVED" operation would be really nice to
> have feature, which I think is currently missing.

Honestly, syzbot - or some related infrastructure - should just do
that automatically.

If the syzbot bug is old, and no longer reproduces with modern
kernels, then just close it. No human should waste any time on that.

As you say, syzbot can find things that others find independently, so
it's not even a "syzbot didn't get the credit, and didn't close things
as a result" issue. It can just as well be "bug was fixed without
syzbot being directly involved at all, and the syzbot report is just
stale old data".

                     Linus

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Jiri Kosina]

On Mon, 3 Jun 2019, Linus Torvalds wrote:

> > So being able to perform "ok, this is not an issue any more for whatever
> > reason, I am closing myself as RESOLVED" operation would be really nice to
> > have feature, which I think is currently missing.
> 
> Honestly, syzbot - or some related infrastructure - should just do
> that automatically.
> 
> If the syzbot bug is old, and no longer reproduces with modern
> kernels, then just close it. No human should waste any time on that.
> 
> As you say, syzbot can find things that others find independently, so
> it's not even a "syzbot didn't get the credit, and didn't close things
> as a result" issue. It can just as well be "bug was fixed without
> syzbot being directly involved at all, and the syzbot report is just
> stale old data".

Agreed.

At the same time, I definitely don't want this to sound as if syzbot is 
just reporting random things that people will be fixing anyway. That's 
definitely not the case in general.

But the volume is simply so overwhelming, that syzbot needs to be able to 
cope with the fact that bugs are also fixed independently.

That'll help the state / usefulness of the reported bugs a lot IMO.

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1382 bytes --]

On Mon, Jun 03, 2019 at 05:10:04PM -0400, Sasha Levin wrote:

> Would it make sense to do some bikeshedding around how an ideal email
> bug report should look like, and then get syzbot and maybe kernelci to
> switch to using this new format as an experiment?

> A bunch the syzbot/kernelci/etc folks will be at LPC as well, and we'll
> have the testing & fuzzing MC going on where this can be discussed and
> agreed on once there is a concrete proposal.

I think we'll find that there's a lot of variation in what's useful to
report directly in the e-mail and how comes from the particular tests
that are being done, what's critical information about the environment
will vary quite a bit and there's going to be taste differences as well.
A lot of this is a combination of common sense and the reporters being
able to be responsive to feedback from their users, there's not much
commonality in the formatting of the existing reports from static
analysis but they're pretty much all useful (at least the ones I tend to
get are useful to me).

There is the question of tooling to track the bugs between multiple bots
which would benefit if they were doing standard things but if people are
working on that rather than trying to serve both humans and computers at
once we might be better off with some sort of API and providing a URL in
the emails for the machines to go and look at.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Shuah Khan]

On 6/3/19 4:11 PM, Mark Brown wrote:
> On Mon, Jun 03, 2019 at 05:10:04PM -0400, Sasha Levin wrote:
> 
>> Would it make sense to do some bikeshedding around how an ideal email
>> bug report should look like, and then get syzbot and maybe kernelci to
>> switch to using this new format as an experiment?
> 
>> A bunch the syzbot/kernelci/etc folks will be at LPC as well, and we'll
>> have the testing & fuzzing MC going on where this can be discussed and
>> agreed on once there is a concrete proposal.
> 
> I think we'll find that there's a lot of variation in what's useful to
> report directly in the e-mail and how comes from the particular tests
> that are being done, what's critical information about the environment
> will vary quite a bit and there's going to be taste differences as well.
> A lot of this is a combination of common sense and the reporters being
> able to be responsive to feedback from their users, there's not much
> commonality in the formatting of the existing reports from static
> analysis but they're pretty much all useful (at least the ones I tend to
> get are useful to me).
> 
> There is the question of tooling to track the bugs between multiple bots
> which would benefit if they were doing standard things but if people are
> working on that rather than trying to serve both humans and computers at
> once we might be better off with some sort of API and providing a URL in
> the emails for the machines to go and look at.
> 

I am going to bounce a proposal and brace for impact :)

What do people think about managing bugs the way we handle our
development workflow?

- Create a directory under our source tree for bugs. This way bugs
   stay with the source and we manage them the same way we handle
   non-runtime objects: Documenation, scripts etc. This can be flat
   directory or have a sub-dir structure underneath.

   ../bugs similar to ../Documentation

- Associate a mailing list - linux-bugs
- Define a format in which bugs should be sent to this list and what
   should be included in the report.

   e.g: [BUG REPORT] sub-system title
   Failure mode, release, stack trace (if applicable) etc.

- A group of maintainers/developers (volunteers) take turns to watch
   this list as a long term plan. Yes, I am proposing a mailing list,
   consistent with our development model.
- History of the bug including the fix commit ID will be preserved.
   We will have to figure out what it means to close a bug and general
   lifetime. There is a big difference between these objects and other
   source objects. These objects have short lifetime.

I haven't thought through everything yet.

My motivation is that, this way bug reporting and managing will be part
of our workflow and becomes part of our process. It helps centralize bug
reports.

I will volunteer to start a sandbox repo and take on the role of
defining format etc. and maintain it for a release or two, to test the
idea. If we like it after the trial, we can adapt it and switch to long
term maintainer plan I mentioned above.

thanks,
-- Shuah

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 476 bytes --]

On Tue, Jun 04, 2019 at 11:16:17AM -0600, Shuah Khan wrote:

> - Create a directory under our source tree for bugs. This way bugs
>   stay with the source and we manage them the same way we handle
>   non-runtime objects: Documenation, scripts etc. This can be flat
>   directory or have a sub-dir structure underneath.
> 
>   ../bugs similar to ../Documentation

If bugs are tracked in the source tree doesn't that make it difficult
for people to update the bugs?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Geert Uytterhoeven]

Hi Mark,

On Wed, Jun 5, 2019 at 11:27 AM Mark Brown <broonie@kernel.org> wrote:
> On Tue, Jun 04, 2019 at 11:16:17AM -0600, Shuah Khan wrote:
> > - Create a directory under our source tree for bugs. This way bugs
> >   stay with the source and we manage them the same way we handle
> >   non-runtime objects: Documenation, scripts etc. This can be flat
> >   directory or have a sub-dir structure underneath.
> >
> >   ../bugs similar to ../Documentation
>
> If bugs are tracked in the source tree doesn't that make it difficult
> for people to update the bugs?

Indeed. While storing the bug state with the sources sounds like a
great idea, it's far from trivial in distributed development:
  - Where are bugs reported (added to git)?
    In upstream, or in maintainer repos?
    At least the tree where a fix is applied should carry the bug report.
  - Unless downstream trees merge from upstream on a regular basis,
    they won't have all reports for bugs that apply to their trees.

Closing bugs does follow the expected path, though: both fix and bug
report update move upstream through a pull request.

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Shuah Khan]

Hi Mark and Geert,

On 6/5/19 5:48 AM, Geert Uytterhoeven wrote:
> Hi Mark,
> 
> On Wed, Jun 5, 2019 at 11:27 AM Mark Brown <broonie@kernel.org> wrote:
>> On Tue, Jun 04, 2019 at 11:16:17AM -0600, Shuah Khan wrote:
>>> - Create a directory under our source tree for bugs. This way bugs
>>>    stay with the source and we manage them the same way we handle
>>>    non-runtime objects: Documenation, scripts etc. This can be flat
>>>    directory or have a sub-dir structure underneath.
>>>
>>>    ../bugs similar to ../Documentation
>>
>> If bugs are tracked in the source tree doesn't that make it difficult
>> for people to update the bugs?
> 

Right. It would be harder for users to report bugs. Unless we com up 
with a way to automate the process of taking user reported generating 
reports that can be committed.

As I was playing with git-bug, I started to think maybe we can leverage 
our process to keep bugs in the same repo.

> Indeed. While storing the bug state with the sources sounds like a
> great idea, it's far from trivial in distributed development:
>    - Where are bugs reported (added to git)?
>      In upstream, or in maintainer repos?
>      At least the tree where a fix is applied should carry the bug report.
>    - Unless downstream trees merge from upstream on a regular basis,
>      they won't have all reports for bugs that apply to their trees.
> 

My thinking is that we follow the same process of keeping the repos 
consistent.

> Closing bugs does follow the expected path, though: both fix and bug
> report update move upstream through a pull request.
> 

This is the value add for me as I proposed the idea. There are lot of 
logistics to consider as you both pointed out.

thanks,
-- Shuah

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Sasha Levin]

On Tue, Jun 04, 2019 at 11:16:17AM -0600, Shuah Khan wrote:
>On 6/3/19 4:11 PM, Mark Brown wrote:
>>On Mon, Jun 03, 2019 at 05:10:04PM -0400, Sasha Levin wrote:
>>
>>>Would it make sense to do some bikeshedding around how an ideal email
>>>bug report should look like, and then get syzbot and maybe kernelci to
>>>switch to using this new format as an experiment?
>>
>>>A bunch the syzbot/kernelci/etc folks will be at LPC as well, and we'll
>>>have the testing & fuzzing MC going on where this can be discussed and
>>>agreed on once there is a concrete proposal.
>>
>>I think we'll find that there's a lot of variation in what's useful to
>>report directly in the e-mail and how comes from the particular tests
>>that are being done, what's critical information about the environment
>>will vary quite a bit and there's going to be taste differences as well.
>>A lot of this is a combination of common sense and the reporters being
>>able to be responsive to feedback from their users, there's not much
>>commonality in the formatting of the existing reports from static
>>analysis but they're pretty much all useful (at least the ones I tend to
>>get are useful to me).
>>
>>There is the question of tooling to track the bugs between multiple bots
>>which would benefit if they were doing standard things but if people are
>>working on that rather than trying to serve both humans and computers at
>>once we might be better off with some sort of API and providing a URL in
>>the emails for the machines to go and look at.
>>
>
>I am going to bounce a proposal and brace for impact :)
>
>What do people think about managing bugs the way we handle our
>development workflow?
>
>- Create a directory under our source tree for bugs. This way bugs
>  stay with the source and we manage them the same way we handle
>  non-runtime objects: Documenation, scripts etc. This can be flat
>  directory or have a sub-dir structure underneath.
>
>  ../bugs similar to ../Documentation

I think that having them in the same tree is going to complicate things
more than it will help.

It would probably be simpler to have this in a separate tree (or as git
objects inside Linus's tree).

>- Associate a mailing list - linux-bugs
>- Define a format in which bugs should be sent to this list and what
>  should be included in the report.
>
>  e.g: [BUG REPORT] sub-system title
>  Failure mode, release, stack trace (if applicable) etc.

Yes! Hopefully something that is both easily machine readable and human
readable.

>- A group of maintainers/developers (volunteers) take turns to watch
>  this list as a long term plan. Yes, I am proposing a mailing list,
>  consistent with our development model.

I'd be happy to help here.

>- History of the bug including the fix commit ID will be preserved.
>  We will have to figure out what it means to close a bug and general
>  lifetime. There is a big difference between these objects and other
>  source objects. These objects have short lifetime.
>
>I haven't thought through everything yet.
>
>My motivation is that, this way bug reporting and managing will be part
>of our workflow and becomes part of our process. It helps centralize bug
>reports.

Having bugs work the same way like patches is a great approach IMHO.
Being able to git-send-email bugs, or to use our various autmation tools
for git commits on bugs is really great.

We have a lot of infrastructure written around our existing git/email
process, so why not leverage it more as you suggest?

--
Thanks,
Sasha

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Shuah Khan]

On 6/5/19 7:19 AM, Sasha Levin wrote:
> On Tue, Jun 04, 2019 at 11:16:17AM -0600, Shuah Khan wrote:
>> On 6/3/19 4:11 PM, Mark Brown wrote:
>>> On Mon, Jun 03, 2019 at 05:10:04PM -0400, Sasha Levin wrote:
>>>
>>>> Would it make sense to do some bikeshedding around how an ideal email
>>>> bug report should look like, and then get syzbot and maybe kernelci to
>>>> switch to using this new format as an experiment?
>>>
>>>> A bunch the syzbot/kernelci/etc folks will be at LPC as well, and we'll
>>>> have the testing & fuzzing MC going on where this can be discussed and
>>>> agreed on once there is a concrete proposal.
>>>
>>> I think we'll find that there's a lot of variation in what's useful to
>>> report directly in the e-mail and how comes from the particular tests
>>> that are being done, what's critical information about the environment
>>> will vary quite a bit and there's going to be taste differences as well.
>>> A lot of this is a combination of common sense and the reporters being
>>> able to be responsive to feedback from their users, there's not much
>>> commonality in the formatting of the existing reports from static
>>> analysis but they're pretty much all useful (at least the ones I tend to
>>> get are useful to me).
>>>
>>> There is the question of tooling to track the bugs between multiple bots
>>> which would benefit if they were doing standard things but if people are
>>> working on that rather than trying to serve both humans and computers at
>>> once we might be better off with some sort of API and providing a URL in
>>> the emails for the machines to go and look at.
>>>
>>
>> I am going to bounce a proposal and brace for impact :)
>>
>> What do people think about managing bugs the way we handle our
>> development workflow?
>>
>> - Create a directory under our source tree for bugs. This way bugs
>>  stay with the source and we manage them the same way we handle
>>  non-runtime objects: Documenation, scripts etc. This can be flat
>>  directory or have a sub-dir structure underneath.
>>
>>  ../bugs similar to ../Documentation
> 
> I think that having them in the same tree is going to complicate things
> more than it will help.
> 
> It would probably be simpler to have this in a separate tree (or as git
> objects inside Linus's tree).
> 
>> - Associate a mailing list - linux-bugs
>> - Define a format in which bugs should be sent to this list and what
>>  should be included in the report.
>>
>>  e.g: [BUG REPORT] sub-system title
>>  Failure mode, release, stack trace (if applicable) etc.
> 
> Yes! Hopefully something that is both easily machine readable and human
> readable.
> 

Can the AUTO SEL logic be leveraged for updates to bugs?

>> - A group of maintainers/developers (volunteers) take turns to watch
>>  this list as a long term plan. Yes, I am proposing a mailing list,
>>  consistent with our development model.
> 
> I'd be happy to help here.
> 

Awesome.

>> - History of the bug including the fix commit ID will be preserved.
>>  We will have to figure out what it means to close a bug and general
>>  lifetime. There is a big difference between these objects and other
>>  source objects. These objects have short lifetime.
>>
>> I haven't thought through everything yet.
>>
>> My motivation is that, this way bug reporting and managing will be part
>> of our workflow and becomes part of our process. It helps centralize bug
>> reports.
> 
> Having bugs work the same way like patches is a great approach IMHO.
> Being able to git-send-email bugs, or to use our various autmation tools
> for git commits on bugs is really great.
> 
> We have a lot of infrastructure written around our existing git/email
> process, so why not leverage it more as you suggest?
> 

Right. I am hoping we can leverage a lot of our existing tools and 
processes if we go with the patch process. Maybe be AUTO SEL logic can 
also be leveraged and extended to generate bugs reports and/or generate 
updates for bugs.

thanks,
-- Shuah

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Laura Abbott]

On 6/3/19 5:10 PM, Sasha Levin wrote:
> Maybe we could sync up our bugtrackers with the kernel's release cycle?
> close all bugs older than a certain kernel version once the
> corresponding stable kernel goes EoL?

We do something like this for Fedora. After the new kernel gets rebased,
we ask the reporters if the bug is still reproducible. This helps
prune down some of the one off cases nobody ever saw again as well as
issues which were actually fixed. Humans sometimes get annoyed at this as
it looks like we're not paying attention but bots wouldn't care if we
closed their own issues.

Laura

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Sasha Levin]

On Tue, Jun 04, 2019 at 02:09:19PM -0400, Laura Abbott wrote:
>On 6/3/19 5:10 PM, Sasha Levin wrote:
>>Maybe we could sync up our bugtrackers with the kernel's release cycle?
>>close all bugs older than a certain kernel version once the
>>corresponding stable kernel goes EoL?
>
>We do something like this for Fedora. After the new kernel gets rebased,
>we ask the reporters if the bug is still reproducible. This helps
>prune down some of the one off cases nobody ever saw again as well as
>issues which were actually fixed. Humans sometimes get annoyed at this as
>it looks like we're not paying attention but bots wouldn't care if we
>closed their own issues.

Sadly that's reality, we can't fix *all* bugs, so sometimes real bugs
would get closed but then hopefully be re-opened again if bots/users are
still hitting them.

Maybe they should be closed with a "-ENOREPRO" which would hopefully
encourage those users/bots to try and repro the issue on a newer version
of the kernel.

--
Thanks,
Sasha

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Konstantin Ryabitsev]

On Mon, Jun 03, 2019 at 01:48:22PM -0700, Linus Torvalds wrote:
>On Mon, Jun 3, 2019 at 11:10 AM Konstantin Ryabitsev
><konstantin@linuxfoundation.org> wrote:
>>
>> May I recommend using a project like git-bug [1] instead of a mailing
>> list?
>
>I'm not a huge fan of mailing lists only for bug handling, but on the
>other hand, I do think that a bug handling thing fundamentally needs
>
> (a) automatic aging out of bug reports
>
> (b) targeted push notifications (ie ractically speaking - email)
>
>and that the email part is important. And it can't just be a "hey,
>something changed". It needs to contain enough information in the
>email to give a developer sufficient knowledge that the developer
>knows whether it's even worth it looking deeper at the bug, and
>looking at the database (in fact, it should contain enough actual
>information that for simple issues, the developer should go "D'oh!"
>and just fix the bug).

Yes, I was envisioning a mechanism with two different moving parts -- 
one the actual bot that would find and create bug entries in a 
distributed system like git-bug, and another an alert and collaboration 
tool that would monitor and notify developers when a bug is created 
matching the parameters they have defined. For example, Greg can define 
a query matching "all bugs mentioning usb" and attach the action of 
"send bug summary to the mailing list." We can run the matching and 
notification daemon at kernel.org, or anyone can set it up to run it on 
their own system to fire as a post-update hook on when they pull the 
git-bug repo.

These are the upsides for a distributed system like this:

1. Doesn't require a stable internet connection when running on the 
laptop of a developer who's out in a remote corner of the world (except 
for the actual pull to fetch new data).
2. Provides full bug history for the project that doesn't depend on 
kernel.org not falling off the face of the earth.
3. Offers ability to easily analyze the data, including historical (the 
same way public-inbox on lore.kernel.org/lkml allows anyone to crunch 
full kernel development history).
4. Can hook into other bugtracking systems like bugzilla, Gitlab, or 
Github via one-way or two-way bridges. E.g. instead of "send an email to 
the list" for USB bugs, the action can be "create a bug on 
bugzilla.kernel.org and sync back comments and bug status changes." The 
bridges for launchpad and GitHub already exist for git-bug.
5. Bots can check for duplicates before creating a new bug (I'm not 100% 
sure how well this would work, but theoretically this is possible). This 
includes bugs created by other bots, as long as they all feed the same 
distributed tracker.
6. Bots don't have to develop notification tooling and can concentrate 
on just creating and pushing new bugs. This lowers the entry barrier if 
new bots are developed.

It's entirely possible that git-bug is the wrong tool for this -- and, 
in fact, it's easy to argue that git is the wrong tool for this and what 
git-bug is doing to twist git to suit its purposes is wrong and 
horrible. Some would argue that blockchains were envisioned for exactly 
this purpose, except that blockchain as a technology is currently in the 
"sardonically cringe whenever someone mentions it" stage of adoption 
(especially every time bitcoin drops a few thousand points).

In general, I believe that we should be aiming to develop tools that are 
git-like in the sense that they do not depend on a centralized entity.  
For the longest time, the distributed nature of email was what made it 
possible for Linux development to remain truly decentralized, but email 
in 2019 is radically different from email in 2009. Running an 
independent email server that reliably sends and receives email is 
becoming more and more difficult now that most of email traffic goes 
through 5-6 major companies -- you must do SPF, DKIM, ARC/DMARC, TLS, 
and who knows what next, just to be reliably accepted by Gmail (maybe).  
And even if someone uses one of those major email providers doesn't mean 
the patches won't end up in someone's spam folder, show up a week late, 
or arrive mangled.

Public-inbox makes this situation a bit better, but it still relies on 
email remaining generally functional. My hope it that we'll be able to 
take this a step further and create truly distributed alternatives to 
platforms like Github/Gitlab that wouldn't require using centralized 
resources for bug tracking and CI. We've done this for source code with 
git, and we should aim to do the same for the rest.

-K

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Shuah Khan]

On 6/4/19 3:43 PM, Konstantin Ryabitsev wrote:
> On Mon, Jun 03, 2019 at 01:48:22PM -0700, Linus Torvalds wrote:
>> On Mon, Jun 3, 2019 at 11:10 AM Konstantin Ryabitsev
>> <konstantin@linuxfoundation.org> wrote:
>>>
>>> May I recommend using a project like git-bug [1] instead of a mailing
>>> list?
>>

I experimented with git-bug a bit. Doesn't look like it is mature enough
for our use. It depends on golang-go

I would like to have the option to build from sources, the instructions
are rather poor and couldn't build following those.

Is there another git-bug like option we can look into?

thanks,
-- Shuah

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Konstantin Ryabitsev]

On Tue, Jun 04, 2019 at 04:02:21PM -0600, Shuah Khan wrote:
>>>>May I recommend using a project like git-bug [1] instead of a 
>>>>mailing
>>>>list?
>>>
>
>I experimented with git-bug a bit. Doesn't look like it is mature enough
>for our use. It depends on golang-go
>
>I would like to have the option to build from sources, the instructions
>are rather poor and couldn't build following those.
>
>Is there another git-bug like option we can look into?

If there exists one, I'm unaware of it, unfortunately. The right person 
to talk to about developing something like this would be the author of 
Public-Inbox (Eric Wong), who digs distributed systems and would 
probably love working on something like this.

However, I have been unsuccessful in getting LF to fund such development 
projects, and he's certainly not in a position to work on something like 
this for free.

Hey, you work at LF now. Maybe you'll have more luck convincing The 
Right People that this is important work that would greatly benefit the 
Linux community? :)

-K

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Shuah Khan]

On 6/4/19 4:22 PM, Konstantin Ryabitsev wrote:
> On Tue, Jun 04, 2019 at 04:02:21PM -0600, Shuah Khan wrote:
>>>>> May I recommend using a project like git-bug [1] instead of a mailing
>>>>> list?
>>>>
>>
>> I experimented with git-bug a bit. Doesn't look like it is mature enough
>> for our use. It depends on golang-go
>>
>> I would like to have the option to build from sources, the instructions
>> are rather poor and couldn't build following those.
>>

I was able to compile with "go" under my home directory finally and run 
a few commands on git-bug repo. There is the next step of trying to see 
if it works for other repos.

I still have my concerns about dependency on "go" and I would like to 
get this to work under a non-home directory structure.

>> Is there another git-bug like option we can look into?
> 
> If there exists one, I'm unaware of it, unfortunately. The right person 
> to talk to about developing something like this would be the author of 
> Public-Inbox (Eric Wong), who digs distributed systems and would 
> probably love working on something like this.
> 

Okay.

> However, I have been unsuccessful in getting LF to fund such development 
> projects, and he's certainly not in a position to work on something like 
> this for free.
> 
> Hey, you work at LF now. Maybe you'll have more luck convincing The 
> Right People that this is important work that would greatly benefit the 
> Linux community? :)
> 

:)

My thoughts on this are: "we needed a solution yesterday" and we can't 
start building a shelter for rain while its raining. We need some 
interim solution, even if we choose to explore a long term project 
development idea.

thanks,
-- Shuah

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Sasha Levin]

On Mon, Jun 03, 2019 at 11:25:51AM -0600, Shuah Khan wrote:
>On 6/2/19 12:09 PM, Sasha Levin wrote:
>>On Fri, May 31, 2019 at 08:01:15AM -0400, Laura Abbott wrote:
>>>On 5/30/19 7:30 PM, Shuah Khan wrote:
>>>>I would like to propose a topic to discuss ideas to keep up with Syzbot
>>>>bugs on an ongoing basis. Good news is, as of this writing, we have 1276
>>>>fixed, 86 in moderation, and 62 fix pending. However, there are 523 open
>>>>bugs. A good number of them have been open for longer than 300 days.
>>>>
>>>>The oldest one has been open for 537 days. I plan to take a closer at
>>>>the open bugs to identify areas that might need more help.
>>>>
>>>>I have been sending fixes to some of them as I find time like many other
>>>>developers. In addition, I included syzbot bug analysis as a required
>>>>contribution in the Linux Kernel Mentorship Program application process.
>>>>This is for learning debugging skills as well as getting help towards
>>>>fixing bugs. It has been successful in getting a few fixes in. My goal
>>>>is  to make these efforts a bit more structured to focus on areas that
>>>>need help.
>>>>
>>>>There are some challenges, the obvious ones being finding time to
>>>>analyze, reproduce, and fix. Reproducers are available in many cases,
>>>>and these bugs can be reproduced. Identifying the right fix and fixing
>>>>is the overwhelming part with the number of outstanding problems.
>>>>
>>>>My objective for this topic is to explore and identify areas that might
>>>>need help in analyzing and fixing bugs in general and especially the
>>>>ones that have been open for a while. It would help me channel and focus
>>>>the Mentorship Program efforts and my efforts to help the areas in need.
>>>>
>>>>thanks,
>>>>-- Shuah
>>>
>>>I'm interested in this topic for not just syzbot but other bug trackers
>>>as well. Fedora gets a steady flow of bugs filed and the three official
>>>maintainers do their best to review but sometimes things slip through,
>>>especially if we hit a time when several of us are out or traveling.
>>>The kernel.org bugzilla also gets a number of bug reports that sometimes
>>>get lost. I'd love to see if there's a process that could work for syzbot
>>>and other high volume kernel trackers.
>>
>>Maybe the solution here is to standardize bug reporting bots? A
>>bugs@kernel.org mailing list which our various bots could report to with
>>a standard format that'll allow users/maintainers to easily filter it to
>>get only bugs they care about?
>>
>
>Easy to filter and manage is important for usability. I mentioned in my
>response Kees that I am finding it hard to update syzbot bug status to
>mark them as duplicate and/or fixed. The email interface is nice for
>individual bugs, but not very easy when it comes to other actions.

Sure, there are quite a few drawbacks to the mailing-list-as-bugtracker
approach, but I feel that we need to start with something and either
build on top of that or replace it in the future with something better.

I don't think that there's a bugtracker in existence today that would be
perfect for our "weird" usecase.

--
Thanks,
Sasha

[Ksummit-discuss] [MAINTAINERS SUMMIT] Squashing bugs!

[Shuah Khan]

Squashing bugs!

I would like to propose a topic to discuss ideas to get a handle on
Syzbot bugs. I have been sending fixes to some of them. Clearly that
is very inefficient. In addition, I included syzbot bug analysis as a
required contribution in the Linux kernel Mentorship program application
process. This is for learning debugging skills as well as getting help
towards fixing bugs. It has been successful in getting a few fixes in.

My objective for this topic is to explore ideas and get some traction in
analyzing and fixing these bugs.

thanks,
-- Shuah