[Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[Laura Abbott]

Fedora tends to follow the most recent stable kernel very closely
(e.g. 4.11.6 is currently pending for Fedora 24, 25, and 26).
This works well enough, but there still seem to be some
disconnects in the bug reporting process. Examples I can think of:

- When users report bugs on the Fedora tracker that look like
actual upstream bugs, what's the best way to have those reported?
I typically end up having to summarize from the Fedora bugzilla
and send an e-mail which ends up being tedious. Can we make this
bug reporting easier for non-kernel developers?
- There's still a gap between when bugs hit Linus' tree and when
stable releases come out. This is not a knock against stable or
a request for stable to go faster :). I try
and monitor stable@ for fixes but there still seems to be a
large time gap for identifying fixes that have been fixes in
master that are relevant to a recent stable release.
- Are pictures of kernel panic still the most reliable method
of getting information for early crashes? I keep getting pictures
in 'creative' formats. 


Thanks,
Laura

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[Jiri Kosina]

On Wed, 21 Jun 2017, Laura Abbott wrote:

> Fedora tends to follow the most recent stable kernel very closely
> (e.g. 4.11.6 is currently pending for Fedora 24, 25, and 26).
> This works well enough, but there still seem to be some
> disconnects in the bug reporting process. Examples I can think of:
> 
> - When users report bugs on the Fedora tracker that look like
> actual upstream bugs, what's the best way to have those reported?
> I typically end up having to summarize from the Fedora bugzilla
> and send an e-mail which ends up being tedious. Can we make this
> bug reporting easier for non-kernel developers?

Just as a data point -- we do a "Kernel of the day" build of a branch that 
follows Linus' tree (with a few SUSE specific patches floating on top of 
it) and provide it in an optional package repository.

That allows the reporter to easily check whether the issue has been fixed 
in latest upstream without needing to have the skills required to compile 
own kernel.

If the issue is confirmed to be present in latest upstream as well, our 
internal person / maintainer responsible for that particular area usually 
takes over (there are cases when the reporter prefers to report the bug 
upstream by himself though).

I am not sure if there is a way how to improve this process even further 
... do you have any particular ideas?

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[Takashi Iwai]

On Thu, 22 Jun 2017 00:34:11 +0200,
Laura Abbott wrote:
> 
> Fedora tends to follow the most recent stable kernel very closely
> (e.g. 4.11.6 is currently pending for Fedora 24, 25, and 26).
> This works well enough, but there still seem to be some
> disconnects in the bug reporting process. Examples I can think of:
> 
> - When users report bugs on the Fedora tracker that look like
> actual upstream bugs, what's the best way to have those reported?
> I typically end up having to summarize from the Fedora bugzilla
> and send an e-mail which ends up being tedious. Can we make this
> bug reporting easier for non-kernel developers?

The topic is what I wanted to brought up, so thanks for heading up,
Laura.

Below are other related points I noticed while working on openSUSE /
SUSE kernel bugzilla triage:

- Very little information about the bug reporting:
  MAINTAINERS file lists only a few.  (Oh, my area also missing...)
  For some areas, it's extremely hard to report a bug.  There is
  always a fallback to LKML, but who reads it?

- The inconsistent bug tracking over the whole kernel areas:
  which way to report purely depends on the subsystem.  Even inside a
  subsystem, some prefer bugzilla while some don't.

- No information about debugging:
  each subsystem tends to have a dedicated script or tool to gather
  the debug information, but it remains secret like grandmother's
  recipe.

- Not well tied with regression tracking:
  for the already released kernels, we have no way to check what's
  still broken and what's been fixed.  Even a simple check list would
  be helpful...

- Inactive bug handling:
  many bug reports on bugzilla are just ignored and rotten.
  In most cases, maintainers play a role as "bug master" on bugzilla.
  But they are often too overloaded or don't pay much attention on old
  bugs.  Later on, when the bug list becomes a pile, they loose the
  gut to solve such bugs any longer.  (The model is the man standing
  on the mirror.)
  

The bug handling really need a big resource.  It takes time to
analyze, fix it, and it needs more efforts for communication, as the
bug reporter tend to have less development experiences.

It would be great to have someone assigned helping for bug  tracking
in both upstream (kernel subsystem) side, and in distributor side.
We have a better coordination regarding the security bugs, and it
should be extended for larger areas.


About your other points:
> - There's still a gap between when bugs hit Linus' tree and when
> stable releases come out. This is not a knock against stable or
> a request for stable to go faster :). I try
> and monitor stable@ for fixes but there still seems to be a
> large time gap for identifying fixes that have been fixes in
> master that are relevant to a recent stable release.

IMO, the stable upstream development is fast enough.  A fast delivery
of a particular fix can be done in the distro side, after all.


> - Are pictures of kernel panic still the most reliable method
> of getting information for early crashes? I keep getting pictures
> in 'creative' formats. 

There have been discussions about this, sending QR code, or whatever,
but I don't see any development since then, unfortunately.
But as a "reliable" method, taking the picture always works, it's not
too bad in the end.


thanks,

Takashi

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[Jiri Kosina]

On Thu, 22 Jun 2017, Takashi Iwai wrote:

> - The inconsistent bug tracking over the whole kernel areas:
>   which way to report purely depends on the subsystem.  Even inside a
>   subsystem, some prefer bugzilla while some don't.

Agreed that this might be annoying for the reporters, but I don't think we 
want to go towards pushing maintainers to use one unified solution. That'd 
be counter-productive.

> It would be great to have someone assigned helping for bug tracking in 
> both upstream (kernel subsystem) side, and in distributor side. We have 
> a better coordination regarding the security bugs, and it should be 
> extended for larger areas.

Completely agreed. But given the problems we've had finding someone who'd 
be tracking regressions (which is a small subset of the whole "bug 
handling"), I'm a bit skeptical.

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[Takashi Iwai]

On Thu, 22 Jun 2017 16:12:09 +0200,
Jiri Kosina wrote:
> 
> On Thu, 22 Jun 2017, Takashi Iwai wrote:
> 
> > - The inconsistent bug tracking over the whole kernel areas:
> >   which way to report purely depends on the subsystem.  Even inside a
> >   subsystem, some prefer bugzilla while some don't.
> 
> Agreed that this might be annoying for the reporters, but I don't think we 
> want to go towards pushing maintainers to use one unified solution. That'd 
> be counter-productive.

I don't pursue that, either.  It'd be great, though, if we can reduce
the too much differences.  The variety is the strength of open source,
but in this case...

> > It would be great to have someone assigned helping for bug tracking in 
> > both upstream (kernel subsystem) side, and in distributor side. We have 
> > a better coordination regarding the security bugs, and it should be 
> > extended for larger areas.
> 
> Completely agreed. But given the problems we've had finding someone who'd 
> be tracking regressions (which is a small subset of the whole "bug 
> handling"), I'm a bit skeptical.

Yeah, I'm also not optimistic about this.  But we need to change our
mind set at first, taking the bug handling more seriously, trying to
put more resources there.


thanks,

Takashi

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[James Bottomley]

On Wed, 2017-06-21 at 15:34 -0700, Laura Abbott wrote:
> Fedora tends to follow the most recent stable kernel very closely
> (e.g. 4.11.6 is currently pending for Fedora 24, 25, and 26).
> This works well enough, but there still seem to be some
> disconnects in the bug reporting process. Examples I can think of:
> 
> - When users report bugs on the Fedora tracker that look like
> actual upstream bugs, what's the best way to have those reported?
> I typically end up having to summarize from the Fedora bugzilla
> and send an e-mail which ends up being tedious. Can we make this
> bug reporting easier for non-kernel developers?

The way we handle bugzilla.kernel.org in SCSI is that it's vectored on
to the linux-scsi mailing list and bugzilla then captures the email
 interactions.  Can this be done with distro bugzillas as well?  You'd
still have to write the initial summary email and send it to the list
and add the cc to the bugzilla, but thereafter your bugzilla should
capture the interaction.  In our system, the reporter can use bugzilla
to interact with the list, because every bug report is vectored there
based on subsystem.  In your bugzillas, I think you can add the list as
a watcher of the bug, so if the original reporter adds something it
will get sent to the list.

James

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[Greg KH]

On Wed, Jun 21, 2017 at 03:34:11PM -0700, Laura Abbott wrote:
> - There's still a gap between when bugs hit Linus' tree and when
> stable releases come out. This is not a knock against stable or
> a request for stable to go faster :). I try
> and monitor stable@ for fixes but there still seems to be a
> large time gap for identifying fixes that have been fixes in
> master that are relevant to a recent stable release.

Ok, I'll bite, what exactly do you mean by "large time"?  During the
-rc1 merge window, it can take a few weeks for me to catch up with the
large onslaught of patches marked for stable, but almost all of those
were determined by the develoers to not be all-so-serious, as they were
delayed in getting to Linus for -rc1, and not before.

You can always email and ask for specific patches to be queued up now,
as I go through the list in a semi-random manner.

After the big -rc1 chunk of patches are merged, it's usually only a week
at most before the patches hit a stable release, people usually complain
it goes too fast, not that it's too slow these days.

thanks,

greg k-h

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[Jiri Kosina]

On Fri, 23 Jun 2017, Greg KH wrote:

> Ok, I'll bite, what exactly do you mean by "large time"?  During the 
> -rc1 merge window, it can take a few weeks for me to catch up with the 
> large onslaught of patches marked for stable, but almost all of those 
> were determined by the develoers to not be all-so-serious, as they were 
> delayed in getting to Linus for -rc1, and not before.
> 
> You can always email and ask for specific patches to be queued up now,
> as I go through the list in a semi-random manner.
> 
> After the big -rc1 chunk of patches are merged, it's usually only a week
> at most before the patches hit a stable release, people usually complain
> it goes too fast, not that it's too slow these days.

I agree, we don't see the pace at which patches make it into -stable as a 
big issue for us (*); in case we need anything cherry-picked from Linus' 
tree earlier than it appears in -stable, we just do that, and drop it from 
our tree once we merge new -stable.

(*) sometimes it's actually quite the contrary :)

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[Laura Abbott]

On 06/23/2017 07:52 AM, Greg KH wrote:
> On Wed, Jun 21, 2017 at 03:34:11PM -0700, Laura Abbott wrote:
>> - There's still a gap between when bugs hit Linus' tree and when
>> stable releases come out. This is not a knock against stable or
>> a request for stable to go faster :). I try
>> and monitor stable@ for fixes but there still seems to be a
>> large time gap for identifying fixes that have been fixes in
>> master that are relevant to a recent stable release.
> 
> Ok, I'll bite, what exactly do you mean by "large time"?  During the
> -rc1 merge window, it can take a few weeks for me to catch up with the
> large onslaught of patches marked for stable, but almost all of those
> were determined by the develoers to not be all-so-serious, as they were
> delayed in getting to Linus for -rc1, and not before.>

'large' is my completely biased perception :)

Fedora generally does not rebase until the .2 or .3 stable release
which usually takes care of most of the -rc1 issue. The problem
tends to come from issues which are not found until later -rcs
and for whatever reason haven't made it to stable yet.

> You can always email and ask for specific patches to be queued up now,
> as I go through the list in a semi-random manner.
> 

If I can find the patch, I have no problem bringing it into Fedora
or requesting it for stable. Finding a patch is the tricky part
which I'd like to improve. My current technique relies on a combination
of good guessing where a fix might come from and random inspiration.
Maybe this is just the curse of the maintainer and there isn't a
better method but having something a bit more systematic to check
would be useful, especially for training other maintainers.

> After the big -rc1 chunk of patches are merged, it's usually only a week
> at most before the patches hit a stable release, people usually complain
> it goes too fast, not that it's too slow these days.
> > thanks,
> 
> greg k-h
> 


Thanks,
Laura

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[Luis R. Rodriguez]

On Thu, Jun 22, 2017 at 02:36:13PM +0200, Jiri Kosina wrote:
> On Wed, 21 Jun 2017, Laura Abbott wrote:
> 
> > Fedora tends to follow the most recent stable kernel very closely
> > (e.g. 4.11.6 is currently pending for Fedora 24, 25, and 26).
> > This works well enough, but there still seem to be some
> > disconnects in the bug reporting process. Examples I can think of:
> > 
> > - When users report bugs on the Fedora tracker that look like
> > actual upstream bugs, what's the best way to have those reported?
> > I typically end up having to summarize from the Fedora bugzilla
> > and send an e-mail which ends up being tedious. Can we make this
> > bug reporting easier for non-kernel developers?
> 
> Just as a data point -- we do a "Kernel of the day" build of a branch that 
> follows Linus' tree (with a few SUSE specific patches floating on top of 
> it) and provide it in an optional package repository.
> 
> That allows the reporter to easily check whether the issue has been fixed 
> in latest upstream without needing to have the skills required to compile 
> own kernel.
> 
> If the issue is confirmed to be present in latest upstream as well, our 
> internal person / maintainer responsible for that particular area usually 
> takes over (there are cases when the reporter prefers to report the bug 
> upstream by himself though).
> 
> I am not sure if there is a way how to improve this process even further 
> ... do you have any particular ideas?

The Kernel Of The Day (KOTD) helps *a lot*. On the XFS front I can say that 90%
of the time so far most bugs can simply be reverse bisected by testing an issue
with KOTD and if it works then doing a reverse bisect. So much so that I
actually *yearn* for the day we get an actual real valid upstream bug. The
other 10% BTW consist of "bad backports" so far.

But one day it comes that KOTD is not sufficient, and there is that pesky delta
on linux-next which *might* also have a fix for you. Problem is booting
linux-next can often fail. Based on personal experience with testing linux-next
more regularly on more machines over the years I can say we are getting much
better with this these days, but every now and then its just poop. That said,
we have a not-so-well known daily linux-next KOTD rpm type of tree as well.
So I recommend that as a next step.

Due to the possible failures possible with linux-next, or random regressions
with other subsystems you often only want to test *one* subsystem. To help
with this there are two options I'm aware of:

  o Subsystem maintiners also backport their -next tree for vanilla, in the
    the like of wireless-testing, which only carries 802.11 on Linus' tree.
    Not sure if other subsystems have similar type of trees, if not I encourage
    it.

  o Backports: backporting to random kernels can be a pain in the ass, but
    backporting to the KOTD should not take much effort if you have the
    right framework [0]. For instance I just created an XFS backport from
    linux-next to KOTD in one day's effort, I can use this to generate
    a tarball for modules for folks to try on top of KOTD. If this would 
    actually be maintained upstream then the amount of work needed is even less,
    and you can have daily snapshots generated. Although sometimes backports
    can be buggy, to my surprise using Coccinelle actually has improved
    correctness of backports, this is only visible once you replace a series
    of patches with the output form an SmPL grammar patch. Given Coccinelle
    is also used, once you backport one subsystem driver, adding more is
    drivers from the same subsystem becomes relatively easier.

HTH,

[0] https://backports.wiki.kernel.org/index.php/Documentation/backports/hacking#Adding_new_driver

  Luis

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[Laurent Pinchart]

Hi Luis,

On Tuesday 27 Jun 2017 19:53:21 Luis R. Rodriguez wrote:
> On Thu, Jun 22, 2017 at 02:36:13PM +0200, Jiri Kosina wrote:
> > On Wed, 21 Jun 2017, Laura Abbott wrote:
> > > Fedora tends to follow the most recent stable kernel very closely
> > > (e.g. 4.11.6 is currently pending for Fedora 24, 25, and 26).
> > > This works well enough, but there still seem to be some
> > > disconnects in the bug reporting process. Examples I can think of:
> > > 
> > > - When users report bugs on the Fedora tracker that look like
> > > actual upstream bugs, what's the best way to have those reported?
> > > I typically end up having to summarize from the Fedora bugzilla
> > > and send an e-mail which ends up being tedious. Can we make this
> > > bug reporting easier for non-kernel developers?
> > 
> > Just as a data point -- we do a "Kernel of the day" build of a branch that
> > follows Linus' tree (with a few SUSE specific patches floating on top of
> > it) and provide it in an optional package repository.
> > 
> > That allows the reporter to easily check whether the issue has been fixed
> > in latest upstream without needing to have the skills required to compile
> > own kernel.
> > 
> > If the issue is confirmed to be present in latest upstream as well, our
> > internal person / maintainer responsible for that particular area usually
> > takes over (there are cases when the reporter prefers to report the bug
> > upstream by himself though).
> > 
> > I am not sure if there is a way how to improve this process even further
> > ... do you have any particular ideas?
> 
> The Kernel Of The Day (KOTD) helps *a lot*. On the XFS front I can say that
> 90% of the time so far most bugs can simply be reverse bisected by testing
> an issue with KOTD and if it works then doing a reverse bisect. So much so
> that I actually *yearn* for the day we get an actual real valid upstream
> bug. The other 10% BTW consist of "bad backports" so far.
> 
> But one day it comes that KOTD is not sufficient, and there is that pesky
> delta on linux-next which *might* also have a fix for you. Problem is
> booting linux-next can often fail. Based on personal experience with
> testing linux-next more regularly on more machines over the years I can say
> we are getting much better with this these days, but every now and then its
> just poop. That said, we have a not-so-well known daily linux-next KOTD rpm
> type of tree as well. So I recommend that as a next step.
> 
> Due to the possible failures possible with linux-next, or random regressions
> with other subsystems you often only want to test *one* subsystem. To help
> with this there are two options I'm aware of:
> 
>   o Subsystem maintiners also backport their -next tree for vanilla, in the
>     the like of wireless-testing, which only carries 802.11 on Linus' tree.
>     Not sure if other subsystems have similar type of trees, if not I
>     encourage it.

The linux-media subsystem has a set of build scripts and backport patches 
(https://git.linuxtv.org/media_build.git) that can be used to compile the 
latest version of the subsystem on an older kernel. This is very useful for 
testing purpose, even if some drivers are blacklisted for too old kernel 
versions as they depend on features not easily backportable.

>   o Backports: backporting to random kernels can be a pain in the ass, but
>     backporting to the KOTD should not take much effort if you have the
>     right framework [0]. For instance I just created an XFS backport from
>     linux-next to KOTD in one day's effort, I can use this to generate
>     a tarball for modules for folks to try on top of KOTD. If this would
>     actually be maintained upstream then the amount of work needed is even
>     less, and you can have daily snapshots generated. Although sometimes
>     backports can be buggy, to my surprise using Coccinelle actually has
>     improved correctness of backports, this is only visible once you replace
>     a series of patches with the output form an SmPL grammar patch. Given
>     Coccinelle is also used, once you backport one subsystem driver, adding
>     more is drivers from the same subsystem becomes relatively easier.
> 
> HTH,
> 
> [0]
> https://backports.wiki.kernel.org/index.php/Documentation/backports/hacking
> #Adding_new_driver

-- 
Regards,

Laurent Pinchart

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[James Bottomley]

On Tue, 2017-06-27 at 19:53 +0200, Luis R. Rodriguez wrote:
> On Thu, Jun 22, 2017 at 02:36:13PM +0200, Jiri Kosina wrote:
> > 
> > On Wed, 21 Jun 2017, Laura Abbott wrote:
> > 
> > > 
> > > Fedora tends to follow the most recent stable kernel very closely
> > > (e.g. 4.11.6 is currently pending for Fedora 24, 25, and 26).
> > > This works well enough, but there still seem to be some
> > > disconnects in the bug reporting process. Examples I can think
> > > of:
> > > 
> > > - When users report bugs on the Fedora tracker that look like
> > > actual upstream bugs, what's the best way to have those reported?
> > > I typically end up having to summarize from the Fedora bugzilla
> > > and send an e-mail which ends up being tedious. Can we make this
> > > bug reporting easier for non-kernel developers?
> > 
> > Just as a data point -- we do a "Kernel of the day" build of a
> > branch that follows Linus' tree (with a few SUSE specific patches
> > floating on top of it) and provide it in an optional package
> > repository.
> > 
> > That allows the reporter to easily check whether the issue has been
> > fixed in latest upstream without needing to have the skills
> > required to compile own kernel.
> > 
> > If the issue is confirmed to be present in latest upstream as well,
> > our internal person / maintainer responsible for that particular
> > area usually takes over (there are cases when the reporter prefers
> > to report the bug upstream by himself though).
> > 
> > I am not sure if there is a way how to improve this process even
> > further ... do you have any particular ideas?
> 
> The Kernel Of The Day (KOTD) helps *a lot*. On the XFS front I can
> say that 90% of the time so far most bugs can simply be reverse
> bisected by testing an issue with KOTD and if it works then doing a
> reverse bisect. So much so that I actually *yearn* for the day we get
> an actual real valid upstream bug. The other 10% BTW consist of "bad
> backports" so far.
> 
> But one day it comes that KOTD is not sufficient, and there is that
> pesky delta on linux-next which *might* also have a fix for you.
> Problem is booting linux-next can often fail.

Just a minute: I'd like to question that assumption. -next is supposed
to be all the upstream trees targetting the merge window.  Boot failure
regressions in those trees are very rare.  Fine, not non-existent so
that's why we run testing and inspection on them, but "often fail" is a
mischaracterisation.  I think the correct characterisation would be
"rarely fail", but I can compromise on "sometimes fail".

>  Based on personal experience with testing linux-next more regularly
> on more machines over the years I can say we are getting much better
> with this these days, but every now and then its just poop.

Really?  Even assuming it to be true for the sake of argument, next
stop for that "poop" is mainline via the merge window, so perhaps
detecting we have a problem before it hits would be a valuable service.

>  That said, we have a not-so-well known daily linux-next KOTD rpm
> type of tree as well. So I recommend that as a next step.
> 
> Due to the possible failures possible with linux-next, or random
> regressions with other subsystems you often only want to test *one*
> subsystem. To help with this there are two options I'm aware of:

I still don't see what's wrong with booting -next?  Fine, there will
occasionally be the rare boot failure regressions, in which case you
can move on to all your other stuff (which is very time intensive), but
if -next boots fine, whether the bug is present or not tells you
something and if it's not present, it saves you a lot of time which is
very valuable because we shouldn't be wasting the time of our most
valuable test group (those which are close to mainline).

So I think it would be a useful service for distro's to provide a
release of -next that users can try.  Perhaps it doesn't have to be
daily, but at least weekly would be enormously helpful.

James

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[Takashi Iwai]

On Tue, 27 Jun 2017 19:53:21 +0200,
Luis R. Rodriguez wrote:
> 
> On Thu, Jun 22, 2017 at 02:36:13PM +0200, Jiri Kosina wrote:
> > On Wed, 21 Jun 2017, Laura Abbott wrote:
> > 
> > > Fedora tends to follow the most recent stable kernel very closely
> > > (e.g. 4.11.6 is currently pending for Fedora 24, 25, and 26).
> > > This works well enough, but there still seem to be some
> > > disconnects in the bug reporting process. Examples I can think of:
> > > 
> > > - When users report bugs on the Fedora tracker that look like
> > > actual upstream bugs, what's the best way to have those reported?
> > > I typically end up having to summarize from the Fedora bugzilla
> > > and send an e-mail which ends up being tedious. Can we make this
> > > bug reporting easier for non-kernel developers?
> > 
> > Just as a data point -- we do a "Kernel of the day" build of a branch that 
> > follows Linus' tree (with a few SUSE specific patches floating on top of 
> > it) and provide it in an optional package repository.
> > 
> > That allows the reporter to easily check whether the issue has been fixed 
> > in latest upstream without needing to have the skills required to compile 
> > own kernel.
> > 
> > If the issue is confirmed to be present in latest upstream as well, our 
> > internal person / maintainer responsible for that particular area usually 
> > takes over (there are cases when the reporter prefers to report the bug 
> > upstream by himself though).
> > 
> > I am not sure if there is a way how to improve this process even further 
> > ... do you have any particular ideas?
> 
> The Kernel Of The Day (KOTD) helps *a lot*. On the XFS front I can say that 90%
> of the time so far most bugs can simply be reverse bisected by testing an issue
> with KOTD and if it works then doing a reverse bisect. So much so that I
> actually *yearn* for the day we get an actual real valid upstream bug. The
> other 10% BTW consist of "bad backports" so far.
> 
> But one day it comes that KOTD is not sufficient, and there is that pesky delta
> on linux-next which *might* also have a fix for you. Problem is booting
> linux-next can often fail. Based on personal experience with testing linux-next
> more regularly on more machines over the years I can say we are getting much
> better with this these days, but every now and then its just poop. That said,
> we have a not-so-well known daily linux-next KOTD rpm type of tree as well.
> So I recommend that as a next step.
> 
> Due to the possible failures possible with linux-next, or random regressions
> with other subsystems you often only want to test *one* subsystem. To help
> with this there are two options I'm aware of:
> 
>   o Subsystem maintiners also backport their -next tree for vanilla, in the
>     the like of wireless-testing, which only carries 802.11 on Linus' tree.
>     Not sure if other subsystems have similar type of trees, if not I encourage
>     it.
> 
>   o Backports: backporting to random kernels can be a pain in the ass, but
>     backporting to the KOTD should not take much effort if you have the
>     right framework [0]. For instance I just created an XFS backport from
>     linux-next to KOTD in one day's effort, I can use this to generate
>     a tarball for modules for folks to try on top of KOTD. If this would 
>     actually be maintained upstream then the amount of work needed is even less,
>     and you can have daily snapshots generated. Although sometimes backports
>     can be buggy, to my surprise using Coccinelle actually has improved
>     correctness of backports, this is only visible once you replace a series
>     of patches with the output form an SmPL grammar patch. Given Coccinelle
>     is also used, once you backport one subsystem driver, adding more is
>     drivers from the same subsystem becomes relatively easier.

I guess it shouldn't be too difficult to build a kernel package based
on subsystem for-next branch regularly like we do for KOTD and
linux-next kernel packages.  You'd need to set up some cron job to git
pull, repackage tarball and adjust config file somehow automatically,
then feed it to osc (in the case of openSUSE/SUSE OBS).


thanks,

Takashi

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[Daniel Vetter]

On Tue, Jun 27, 2017 at 8:30 PM, James Bottomley
<James.Bottomley@hansenpartnership.com> wrote:
>> But one day it comes that KOTD is not sufficient, and there is that
>> pesky delta on linux-next which *might* also have a fix for you.
>> Problem is booting linux-next can often fail.
>
> Just a minute: I'd like to question that assumption. -next is supposed
> to be all the upstream trees targetting the merge window.  Boot failure
> regressions in those trees are very rare.  Fine, not non-existent so
> that's why we run testing and inspection on them, but "often fail" is a
> mischaracterisation.  I think the correct characterisation would be
> "rarely fail", but I can compromise on "sometimes fail".

The merge window takes out about 30% of our CI machines when it lands
(not boot, also stuff like suspend/resume so a bit more than your
criteria here, but all because of issues outside of drm, we tend to
catch our own crap on our own machines). We probably should test
linux-next to catch this stuff earlier, but atm we just don't have the
time - just getting the -rc1 fallout back under control takes a lot of
time (e.g. we still have a e1000e regression fix in our CI branches
since the patch doesn't seem to go anywhere, despite nagging). I
wound't call this "rarely fails" when you can easily see the spacing
of merge windows in our CI stats. But maybe no one tests on random
piles of recent and semi-recent intel desktops and laptops, dunno.

Not sure what the solution would be since I'm pretty sure drm/i915
isn't innocent in taking out other systems, except much more testing
of linux-next (for which we simply don't have the machine time nor
people to triage the fallout right now).
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[Luis R. Rodriguez]

On Tue, Jun 27, 2017 at 11:30:34AM -0700, James Bottomley wrote:
> On Tue, 2017-06-27 at 19:53 +0200, Luis R. Rodriguez wrote:
> > The Kernel Of The Day (KOTD) helps *a lot*. On the XFS front I can
> > say that 90% of the time so far most bugs can simply be reverse
> > bisected by testing an issue with KOTD and if it works then doing a
> > reverse bisect. So much so that I actually *yearn* for the day we get
> > an actual real valid upstream bug. The other 10% BTW consist of "bad
> > backports" so far.
> > 
> > But one day it comes that KOTD is not sufficient, and there is that
> > pesky delta on linux-next which *might* also have a fix for you.
> > Problem is booting linux-next can often fail.
> 
> Just a minute: I'd like to question that assumption. -next is supposed
> to be all the upstream trees targetting the merge window.  Boot failure
> regressions in those trees are very rare.

Like I said, we've gotten better. For my day to day development systems it
is true that linux-next can be groovy. When it comes to actually booting it
on a real system being evaluated though, your luck varies. My luck recently
was not so great.

Guenter Rock maintains a map of both kernel compile and qemu run time
testign of linux-next accross a different set of architectures, he can
perhaps tell you better how things look these days on his map. This is
outside of the scope of the architectures that Stephen tests AFAICT.

> Fine, not non-existent so
> that's why we run testing and inspection on them, but "often fail" is a
> mischaracterisation.  I think the correct characterisation would be
> "rarely fail", but I can compromise on "sometimes fail".

That's fair, how about this: for my development system linux-next rarely fails
now. For production test system, linux-next sometimes fails :D

> >  Based on personal experience with testing linux-next more regularly
> > on more machines over the years I can say we are getting much better
> > with this these days, but every now and then its just poop.
> 
> Really?  Even assuming it to be true for the sake of argument, next
> stop for that "poop" is mainline via the merge window, so perhaps
> detecting we have a problem before it hits would be a valuable service.

Of course. Its why everyone and their uncles should be giving linux-next
a go often.

> >  That said, we have a not-so-well known daily linux-next KOTD rpm
> > type of tree as well. So I recommend that as a next step.
> > 
> > Due to the possible failures possible with linux-next, or random
> > regressions with other subsystems you often only want to test *one*
> > subsystem. To help with this there are two options I'm aware of:
> 
> I still don't see what's wrong with booting -next?  Fine, there will
> occasionally be the rare boot failure regressions, in which case you
> can move on to all your other stuff (which is very time intensive), but
> if -next boots fine, whether the bug is present or not tells you
> something and if it's not present, it saves you a lot of time which is
> very valuable because we shouldn't be wasting the time of our most
> valuable test group (those which are close to mainline).

Note how Laura mentioned they actually skip rc1. Even though my own
experience these days is linux-next is *more stable* than rc1, its
can't be a surprise linux-next can have issues.

We're talking about *all* development ramp up. One commit is bound
to have a pesky stupid thing merged.

> So I think it would be a useful service for distro's to provide a
> release of -next that users can try.  Perhaps it doesn't have to be
> daily, but at least weekly would be enormously helpful.

Daily. No questions asked.

A) KOTD --> B) linux-next --> mailing list

  Luis

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[Luis R. Rodriguez]

On Tue, Jun 27, 2017 at 08:31:17PM +0200, Takashi Iwai wrote:
> On Tue, 27 Jun 2017 19:53:21 +0200,
> Luis R. Rodriguez wrote:
> > 
> > On Thu, Jun 22, 2017 at 02:36:13PM +0200, Jiri Kosina wrote:
> > > On Wed, 21 Jun 2017, Laura Abbott wrote:
> > > 
> > > > Fedora tends to follow the most recent stable kernel very closely
> > > > (e.g. 4.11.6 is currently pending for Fedora 24, 25, and 26).
> > > > This works well enough, but there still seem to be some
> > > > disconnects in the bug reporting process. Examples I can think of:
> > > > 
> > > > - When users report bugs on the Fedora tracker that look like
> > > > actual upstream bugs, what's the best way to have those reported?
> > > > I typically end up having to summarize from the Fedora bugzilla
> > > > and send an e-mail which ends up being tedious. Can we make this
> > > > bug reporting easier for non-kernel developers?
> > > 
> > > Just as a data point -- we do a "Kernel of the day" build of a branch that 
> > > follows Linus' tree (with a few SUSE specific patches floating on top of 
> > > it) and provide it in an optional package repository.
> > > 
> > > That allows the reporter to easily check whether the issue has been fixed 
> > > in latest upstream without needing to have the skills required to compile 
> > > own kernel.
> > > 
> > > If the issue is confirmed to be present in latest upstream as well, our 
> > > internal person / maintainer responsible for that particular area usually 
> > > takes over (there are cases when the reporter prefers to report the bug 
> > > upstream by himself though).
> > > 
> > > I am not sure if there is a way how to improve this process even further 
> > > ... do you have any particular ideas?
> > 
> > The Kernel Of The Day (KOTD) helps *a lot*. On the XFS front I can say that 90%
> > of the time so far most bugs can simply be reverse bisected by testing an issue
> > with KOTD and if it works then doing a reverse bisect. So much so that I
> > actually *yearn* for the day we get an actual real valid upstream bug. The
> > other 10% BTW consist of "bad backports" so far.
> > 
> > But one day it comes that KOTD is not sufficient, and there is that pesky delta
> > on linux-next which *might* also have a fix for you. Problem is booting
> > linux-next can often fail. Based on personal experience with testing linux-next
> > more regularly on more machines over the years I can say we are getting much
> > better with this these days, but every now and then its just poop. That said,
> > we have a not-so-well known daily linux-next KOTD rpm type of tree as well.
> > So I recommend that as a next step.
> > 
> > Due to the possible failures possible with linux-next, or random regressions
> > with other subsystems you often only want to test *one* subsystem. To help
> > with this there are two options I'm aware of:
> > 
> >   o Subsystem maintiners also backport their -next tree for vanilla, in the
> >     the like of wireless-testing, which only carries 802.11 on Linus' tree.
> >     Not sure if other subsystems have similar type of trees, if not I encourage
> >     it.
> > 
> >   o Backports: backporting to random kernels can be a pain in the ass, but
> >     backporting to the KOTD should not take much effort if you have the
> >     right framework [0]. For instance I just created an XFS backport from
> >     linux-next to KOTD in one day's effort, I can use this to generate
> >     a tarball for modules for folks to try on top of KOTD. If this would 
> >     actually be maintained upstream then the amount of work needed is even less,
> >     and you can have daily snapshots generated. Although sometimes backports
> >     can be buggy, to my surprise using Coccinelle actually has improved
> >     correctness of backports, this is only visible once you replace a series
> >     of patches with the output form an SmPL grammar patch. Given Coccinelle
> >     is also used, once you backport one subsystem driver, adding more is
> >     drivers from the same subsystem becomes relatively easier.
> 
> I guess it shouldn't be too difficult to build a kernel package based
> on subsystem for-next branch regularly like we do for KOTD and
> linux-next kernel packages.  You'd need to set up some cron job to git
> pull, repackage tarball and adjust config file somehow automatically,
> then feed it to osc (in the case of openSUSE/SUSE OBS).

That'd be great. It sounds like we have trees like this for media, and
wireless. Not sure of others.

  Luis

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[Guenter Roeck]

On Tue, Jun 27, 2017 at 09:02:02PM +0200, Luis R. Rodriguez wrote:
> On Tue, Jun 27, 2017 at 11:30:34AM -0700, James Bottomley wrote:
> > On Tue, 2017-06-27 at 19:53 +0200, Luis R. Rodriguez wrote:
> > > The Kernel Of The Day (KOTD) helps *a lot*. On the XFS front I can
> > > say that 90% of the time so far most bugs can simply be reverse
> > > bisected by testing an issue with KOTD and if it works then doing a
> > > reverse bisect. So much so that I actually *yearn* for the day we get
> > > an actual real valid upstream bug. The other 10% BTW consist of "bad
> > > backports" so far.
> > > 
> > > But one day it comes that KOTD is not sufficient, and there is that
> > > pesky delta on linux-next which *might* also have a fix for you.
> > > Problem is booting linux-next can often fail.
> > 
> > Just a minute: I'd like to question that assumption. -next is supposed
> > to be all the upstream trees targetting the merge window.  Boot failure
> > regressions in those trees are very rare.
> 
> Like I said, we've gotten better. For my day to day development systems it
> is true that linux-next can be groovy. When it comes to actually booting it
> on a real system being evaluated though, your luck varies. My luck recently
> was not so great.
> 
> Guenter Rock maintains a map of both kernel compile and qemu run time
> testign of linux-next accross a different set of architectures, he can
> perhaps tell you better how things look these days on his map. This is
> outside of the scope of the architectures that Stephen tests AFAICT.
> 

There are almost always some build and/or runtime failures in -next. Right now
we are lucky (the current qemu boot failures are really build failures).
For reference, from next-20170627:

Build results:
	total: 145 pass: 140 fail: 5
Failed builds:
	arm:allmodconfig
	arm64:allmodconfig
	hexagon:defconfig
	hexagon:allnoconfig
	parisc:generic-64bit_defconfig

Qemu test results:
	total: 122 pass: 120 fail: 2
Failed tests:
	arm:versatilepb-scsi:versatile_defconfig:versatile-pb
	sh:rts7751r2dplus_defconfig

I'd say maybe once or twice in a relase cycle I see half or more of
the qemu tests failing.

> > Fine, not non-existent so
> > that's why we run testing and inspection on them, but "often fail" is a
> > mischaracterisation.  I think the correct characterisation would be
> > "rarely fail", but I can compromise on "sometimes fail".
> 
> That's fair, how about this: for my development system linux-next rarely fails
> now. For production test system, linux-next sometimes fails :D
> 
Depends on the scope. If "fail" refers to x86_64:defconfig, I would agree to
"rarely" (say, maybe once or twice a month on average ). If "fail" refers to
"one or more of my qemu boot tests fail", "almost always" would be more
accurate. If "fail" refers to "one or more of my build or qemu boot
tests fail", it would be "pretty much always".

> > >  Based on personal experience with testing linux-next more regularly
> > > on more machines over the years I can say we are getting much better
> > > with this these days, but every now and then its just poop.
> > 
> > Really?  Even assuming it to be true for the sake of argument, next
> > stop for that "poop" is mainline via the merge window, so perhaps
> > detecting we have a problem before it hits would be a valuable service.
> 
> Of course. Its why everyone and their uncles should be giving linux-next
> a go often.
> 
> > >  That said, we have a not-so-well known daily linux-next KOTD rpm
> > > type of tree as well. So I recommend that as a next step.
> > > 
> > > Due to the possible failures possible with linux-next, or random
> > > regressions with other subsystems you often only want to test *one*
> > > subsystem. To help with this there are two options I'm aware of:
> > 
> > I still don't see what's wrong with booting -next?  Fine, there will
> > occasionally be the rare boot failure regressions, in which case you
> > can move on to all your other stuff (which is very time intensive), but
> > if -next boots fine, whether the bug is present or not tells you
> > something and if it's not present, it saves you a lot of time which is
> > very valuable because we shouldn't be wasting the time of our most
> > valuable test group (those which are close to mainline).
> 
> Note how Laura mentioned they actually skip rc1. Even though my own
> experience these days is linux-next is *more stable* than rc1, its
> can't be a surprise linux-next can have issues.
> 
I tend to agree, as long as you refer to -next a few days before the commit
window opens. Sometimes there is a flurry of commits showing up in the last
-next before the commit window opens (or in the first -next after it
opened). Those commits tend to make it into mainline almost immediately,
and my non-scientific impression is that they cause a high percentage of
failures.

Guenter

> We're talking about *all* development ramp up. One commit is bound
> to have a pesky stupid thing merged.
> 
> > So I think it would be a useful service for distro's to provide a
> > release of -next that users can try.  Perhaps it doesn't have to be
> > daily, but at least weekly would be enormously helpful.
> 
> Daily. No questions asked.
> 
> A) KOTD --> B) linux-next --> mailing list
> 
>   Luis

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[Jiri Kosina]

On Tue, 27 Jun 2017, James Bottomley wrote:

> So I think it would be a useful service for distro's to provide a 
> release of -next that users can try.  Perhaps it doesn't have to be 
> daily, but at least weekly would be enormously helpful.

We do:

	https://build.opensuse.org/repositories/Kernel:linux-next

But we are getting close to zero bugreports against it. Which either means 
it's very rarely used (and therefore we should perhaps be more vocal about 
it), or people who are educated enough to use it are actually educated 
enough to report any issues directly upstream.

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[Takashi Iwai]

On Wed, 28 Jun 2017 00:35:40 +0200,
Jiri Kosina wrote:
> 
> On Tue, 27 Jun 2017, James Bottomley wrote:
> 
> > So I think it would be a useful service for distro's to provide a 
> > release of -next that users can try.  Perhaps it doesn't have to be 
> > daily, but at least weekly would be enormously helpful.
> 
> We do:
> 
> 	https://build.opensuse.org/repositories/Kernel:linux-next
> 
> But we are getting close to zero bugreports against it. Which either means 
> it's very rarely used (and therefore we should perhaps be more vocal about 
> it), or people who are educated enough to use it are actually educated 
> enough to report any issues directly upstream.

IMO, we should consider deploying more CI tests (either openQA or
simplified ones) for KOTD and linux-next.  I discussed with out QA
guys once, but it's not really happening yet.


thanks,

Takashi

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[Daniel Vetter]

On Tue, Jun 27, 2017 at 9:04 PM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
>> I guess it shouldn't be too difficult to build a kernel package based
>> on subsystem for-next branch regularly like we do for KOTD and
>> linux-next kernel packages.  You'd need to set up some cron job to git
>> pull, repackage tarball and adjust config file somehow automatically,
>> then feed it to osc (in the case of openSUSE/SUSE OBS).
>
> That'd be great. It sounds like we have trees like this for media, and
> wireless. Not sure of others.

In case anyone cares, I'm trying to make drm-tip _the_ integration
tree for graphics. Doesn't contain everything yet, but generally a
good place for build-bots, and iirc at least ubuntu has a ppa for it:

https://cgit.freedesktop.org/drm-tip

And the integration manifest:

https://cgit.freedesktop.org/drm-tip/tree/integration-manifest

Probably the neatest thing is that there's no single person
maintaining it, but conflicts get resolved and stored when they
happen, by the person who pushes the patches creating the conflict.
See

https://cgit.freedesktop.org/drm-tip/log/?h=rerere-cache

So yeah we've managed to scale maintaining an integration tree to a
team of 20+ people, everyone with commit rights in the various trees
essentially :-)

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 524 bytes --]

On Tue, Jun 27, 2017 at 12:46:22PM -0700, Guenter Roeck wrote:

> There are almost always some build and/or runtime failures in -next. Right now
> we are lucky (the current qemu boot failures are really build failures).
> For reference, from next-20170627:

Right, the important thing is that we sit on the results and no
individual failure tends to last for terribly long and hopefully by the
time things hit Linus' tree we're in good shape - the failure rate in
mainline from the build and boot tests is very much better.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[Jani Nikula]

On Thu, 22 Jun 2017, Takashi Iwai <tiwai@suse.de> wrote:
> On Thu, 22 Jun 2017 16:12:09 +0200,
> Jiri Kosina wrote:
>> 
>> On Thu, 22 Jun 2017, Takashi Iwai wrote:
>> 
>> > - The inconsistent bug tracking over the whole kernel areas:
>> >   which way to report purely depends on the subsystem.  Even inside a
>> >   subsystem, some prefer bugzilla while some don't.
>> 
>> Agreed that this might be annoying for the reporters, but I don't think we 
>> want to go towards pushing maintainers to use one unified solution. That'd 
>> be counter-productive.
>
> I don't pursue that, either.  It'd be great, though, if we can reduce
> the too much differences.  The variety is the strength of open source,
> but in this case...

The least we can do is use the MAINTAINERS "B:" tag to document the
preferences of each subsystem or driver.


BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop

[Takashi Iwai]

On Wed, 28 Jun 2017 15:12:29 +0200,
Jani Nikula wrote:
> 
> On Thu, 22 Jun 2017, Takashi Iwai <tiwai@suse.de> wrote:
> > On Thu, 22 Jun 2017 16:12:09 +0200,
> > Jiri Kosina wrote:
> >> 
> >> On Thu, 22 Jun 2017, Takashi Iwai wrote:
> >> 
> >> > - The inconsistent bug tracking over the whole kernel areas:
> >> >   which way to report purely depends on the subsystem.  Even inside a
> >> >   subsystem, some prefer bugzilla while some don't.
> >> 
> >> Agreed that this might be annoying for the reporters, but I don't think we 
> >> want to go towards pushing maintainers to use one unified solution. That'd 
> >> be counter-productive.
> >
> > I don't pursue that, either.  It'd be great, though, if we can reduce
> > the too much differences.  The variety is the strength of open source,
> > but in this case...
> 
> The least we can do is use the MAINTAINERS "B:" tag to document the
> preferences of each subsystem or driver.

Yes.  There are way too few entries for "B:" right now.


Takashi