* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop
2017-06-27 17:53 ` Luis R. Rodriguez
@ 2017-06-27 18:26 ` Laurent Pinchart
2017-06-27 18:30 ` James Bottomley
2017-06-27 18:31 ` Takashi Iwai
2 siblings, 0 replies; 23+ messages in thread
From: Laurent Pinchart @ 2017-06-27 18:26 UTC (permalink / raw)
To: ksummit-discuss
Hi Luis,
On Tuesday 27 Jun 2017 19:53:21 Luis R. Rodriguez wrote:
> On Thu, Jun 22, 2017 at 02:36:13PM +0200, Jiri Kosina wrote:
> > On Wed, 21 Jun 2017, Laura Abbott wrote:
> > > Fedora tends to follow the most recent stable kernel very closely
> > > (e.g. 4.11.6 is currently pending for Fedora 24, 25, and 26).
> > > This works well enough, but there still seem to be some
> > > disconnects in the bug reporting process. Examples I can think of:
> > >
> > > - When users report bugs on the Fedora tracker that look like
> > > actual upstream bugs, what's the best way to have those reported?
> > > I typically end up having to summarize from the Fedora bugzilla
> > > and send an e-mail which ends up being tedious. Can we make this
> > > bug reporting easier for non-kernel developers?
> >
> > Just as a data point -- we do a "Kernel of the day" build of a branch that
> > follows Linus' tree (with a few SUSE specific patches floating on top of
> > it) and provide it in an optional package repository.
> >
> > That allows the reporter to easily check whether the issue has been fixed
> > in latest upstream without needing to have the skills required to compile
> > own kernel.
> >
> > If the issue is confirmed to be present in latest upstream as well, our
> > internal person / maintainer responsible for that particular area usually
> > takes over (there are cases when the reporter prefers to report the bug
> > upstream by himself though).
> >
> > I am not sure if there is a way how to improve this process even further
> > ... do you have any particular ideas?
>
> The Kernel Of The Day (KOTD) helps *a lot*. On the XFS front I can say that
> 90% of the time so far most bugs can simply be reverse bisected by testing
> an issue with KOTD and if it works then doing a reverse bisect. So much so
> that I actually *yearn* for the day we get an actual real valid upstream
> bug. The other 10% BTW consist of "bad backports" so far.
>
> But one day it comes that KOTD is not sufficient, and there is that pesky
> delta on linux-next which *might* also have a fix for you. Problem is
> booting linux-next can often fail. Based on personal experience with
> testing linux-next more regularly on more machines over the years I can say
> we are getting much better with this these days, but every now and then its
> just poop. That said, we have a not-so-well known daily linux-next KOTD rpm
> type of tree as well. So I recommend that as a next step.
>
> Due to the possible failures possible with linux-next, or random regressions
> with other subsystems you often only want to test *one* subsystem. To help
> with this there are two options I'm aware of:
>
> o Subsystem maintiners also backport their -next tree for vanilla, in the
> the like of wireless-testing, which only carries 802.11 on Linus' tree.
> Not sure if other subsystems have similar type of trees, if not I
> encourage it.
The linux-media subsystem has a set of build scripts and backport patches
(https://git.linuxtv.org/media_build.git) that can be used to compile the
latest version of the subsystem on an older kernel. This is very useful for
testing purpose, even if some drivers are blacklisted for too old kernel
versions as they depend on features not easily backportable.
> o Backports: backporting to random kernels can be a pain in the ass, but
> backporting to the KOTD should not take much effort if you have the
> right framework [0]. For instance I just created an XFS backport from
> linux-next to KOTD in one day's effort, I can use this to generate
> a tarball for modules for folks to try on top of KOTD. If this would
> actually be maintained upstream then the amount of work needed is even
> less, and you can have daily snapshots generated. Although sometimes
> backports can be buggy, to my surprise using Coccinelle actually has
> improved correctness of backports, this is only visible once you replace
> a series of patches with the output form an SmPL grammar patch. Given
> Coccinelle is also used, once you backport one subsystem driver, adding
> more is drivers from the same subsystem becomes relatively easier.
>
> HTH,
>
> [0]
> https://backports.wiki.kernel.org/index.php/Documentation/backports/hacking
> #Adding_new_driver
--
Regards,
Laurent Pinchart
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop
2017-06-27 17:53 ` Luis R. Rodriguez
2017-06-27 18:26 ` Laurent Pinchart
@ 2017-06-27 18:30 ` James Bottomley
2017-06-27 18:41 ` Daniel Vetter
` (2 more replies)
2017-06-27 18:31 ` Takashi Iwai
2 siblings, 3 replies; 23+ messages in thread
From: James Bottomley @ 2017-06-27 18:30 UTC (permalink / raw)
To: Luis R. Rodriguez, Jiri Kosina; +Cc: ksummit
On Tue, 2017-06-27 at 19:53 +0200, Luis R. Rodriguez wrote:
> On Thu, Jun 22, 2017 at 02:36:13PM +0200, Jiri Kosina wrote:
> >
> > On Wed, 21 Jun 2017, Laura Abbott wrote:
> >
> > >
> > > Fedora tends to follow the most recent stable kernel very closely
> > > (e.g. 4.11.6 is currently pending for Fedora 24, 25, and 26).
> > > This works well enough, but there still seem to be some
> > > disconnects in the bug reporting process. Examples I can think
> > > of:
> > >
> > > - When users report bugs on the Fedora tracker that look like
> > > actual upstream bugs, what's the best way to have those reported?
> > > I typically end up having to summarize from the Fedora bugzilla
> > > and send an e-mail which ends up being tedious. Can we make this
> > > bug reporting easier for non-kernel developers?
> >
> > Just as a data point -- we do a "Kernel of the day" build of a
> > branch that follows Linus' tree (with a few SUSE specific patches
> > floating on top of it) and provide it in an optional package
> > repository.
> >
> > That allows the reporter to easily check whether the issue has been
> > fixed in latest upstream without needing to have the skills
> > required to compile own kernel.
> >
> > If the issue is confirmed to be present in latest upstream as well,
> > our internal person / maintainer responsible for that particular
> > area usually takes over (there are cases when the reporter prefers
> > to report the bug upstream by himself though).
> >
> > I am not sure if there is a way how to improve this process even
> > further ... do you have any particular ideas?
>
> The Kernel Of The Day (KOTD) helps *a lot*. On the XFS front I can
> say that 90% of the time so far most bugs can simply be reverse
> bisected by testing an issue with KOTD and if it works then doing a
> reverse bisect. So much so that I actually *yearn* for the day we get
> an actual real valid upstream bug. The other 10% BTW consist of "bad
> backports" so far.
>
> But one day it comes that KOTD is not sufficient, and there is that
> pesky delta on linux-next which *might* also have a fix for you.
> Problem is booting linux-next can often fail.
Just a minute: I'd like to question that assumption. -next is supposed
to be all the upstream trees targetting the merge window. Boot failure
regressions in those trees are very rare. Fine, not non-existent so
that's why we run testing and inspection on them, but "often fail" is a
mischaracterisation. I think the correct characterisation would be
"rarely fail", but I can compromise on "sometimes fail".
> Based on personal experience with testing linux-next more regularly
> on more machines over the years I can say we are getting much better
> with this these days, but every now and then its just poop.
Really? Even assuming it to be true for the sake of argument, next
stop for that "poop" is mainline via the merge window, so perhaps
detecting we have a problem before it hits would be a valuable service.
> That said, we have a not-so-well known daily linux-next KOTD rpm
> type of tree as well. So I recommend that as a next step.
>
> Due to the possible failures possible with linux-next, or random
> regressions with other subsystems you often only want to test *one*
> subsystem. To help with this there are two options I'm aware of:
I still don't see what's wrong with booting -next? Fine, there will
occasionally be the rare boot failure regressions, in which case you
can move on to all your other stuff (which is very time intensive), but
if -next boots fine, whether the bug is present or not tells you
something and if it's not present, it saves you a lot of time which is
very valuable because we shouldn't be wasting the time of our most
valuable test group (those which are close to mainline).
So I think it would be a useful service for distro's to provide a
release of -next that users can try. Perhaps it doesn't have to be
daily, but at least weekly would be enormously helpful.
James
^ permalink raw reply [flat|nested] 23+ messages in thread* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop
2017-06-27 18:30 ` James Bottomley
@ 2017-06-27 18:41 ` Daniel Vetter
2017-06-27 19:02 ` Luis R. Rodriguez
2017-06-27 22:35 ` Jiri Kosina
2 siblings, 0 replies; 23+ messages in thread
From: Daniel Vetter @ 2017-06-27 18:41 UTC (permalink / raw)
To: James Bottomley; +Cc: ksummit
On Tue, Jun 27, 2017 at 8:30 PM, James Bottomley
<James.Bottomley@hansenpartnership.com> wrote:
>> But one day it comes that KOTD is not sufficient, and there is that
>> pesky delta on linux-next which *might* also have a fix for you.
>> Problem is booting linux-next can often fail.
>
> Just a minute: I'd like to question that assumption. -next is supposed
> to be all the upstream trees targetting the merge window. Boot failure
> regressions in those trees are very rare. Fine, not non-existent so
> that's why we run testing and inspection on them, but "often fail" is a
> mischaracterisation. I think the correct characterisation would be
> "rarely fail", but I can compromise on "sometimes fail".
The merge window takes out about 30% of our CI machines when it lands
(not boot, also stuff like suspend/resume so a bit more than your
criteria here, but all because of issues outside of drm, we tend to
catch our own crap on our own machines). We probably should test
linux-next to catch this stuff earlier, but atm we just don't have the
time - just getting the -rc1 fallout back under control takes a lot of
time (e.g. we still have a e1000e regression fix in our CI branches
since the patch doesn't seem to go anywhere, despite nagging). I
wound't call this "rarely fails" when you can easily see the spacing
of merge windows in our CI stats. But maybe no one tests on random
piles of recent and semi-recent intel desktops and laptops, dunno.
Not sure what the solution would be since I'm pretty sure drm/i915
isn't innocent in taking out other systems, except much more testing
of linux-next (for which we simply don't have the machine time nor
people to triage the fallout right now).
-Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop
2017-06-27 18:30 ` James Bottomley
2017-06-27 18:41 ` Daniel Vetter
@ 2017-06-27 19:02 ` Luis R. Rodriguez
2017-06-27 19:46 ` Guenter Roeck
2017-06-27 22:35 ` Jiri Kosina
2 siblings, 1 reply; 23+ messages in thread
From: Luis R. Rodriguez @ 2017-06-27 19:02 UTC (permalink / raw)
To: James Bottomley, Guenter Roeck; +Cc: ksummit
On Tue, Jun 27, 2017 at 11:30:34AM -0700, James Bottomley wrote:
> On Tue, 2017-06-27 at 19:53 +0200, Luis R. Rodriguez wrote:
> > The Kernel Of The Day (KOTD) helps *a lot*. On the XFS front I can
> > say that 90% of the time so far most bugs can simply be reverse
> > bisected by testing an issue with KOTD and if it works then doing a
> > reverse bisect. So much so that I actually *yearn* for the day we get
> > an actual real valid upstream bug. The other 10% BTW consist of "bad
> > backports" so far.
> >
> > But one day it comes that KOTD is not sufficient, and there is that
> > pesky delta on linux-next which *might* also have a fix for you.
> > Problem is booting linux-next can often fail.
>
> Just a minute: I'd like to question that assumption. -next is supposed
> to be all the upstream trees targetting the merge window. Boot failure
> regressions in those trees are very rare.
Like I said, we've gotten better. For my day to day development systems it
is true that linux-next can be groovy. When it comes to actually booting it
on a real system being evaluated though, your luck varies. My luck recently
was not so great.
Guenter Rock maintains a map of both kernel compile and qemu run time
testign of linux-next accross a different set of architectures, he can
perhaps tell you better how things look these days on his map. This is
outside of the scope of the architectures that Stephen tests AFAICT.
> Fine, not non-existent so
> that's why we run testing and inspection on them, but "often fail" is a
> mischaracterisation. I think the correct characterisation would be
> "rarely fail", but I can compromise on "sometimes fail".
That's fair, how about this: for my development system linux-next rarely fails
now. For production test system, linux-next sometimes fails :D
> > Based on personal experience with testing linux-next more regularly
> > on more machines over the years I can say we are getting much better
> > with this these days, but every now and then its just poop.
>
> Really? Even assuming it to be true for the sake of argument, next
> stop for that "poop" is mainline via the merge window, so perhaps
> detecting we have a problem before it hits would be a valuable service.
Of course. Its why everyone and their uncles should be giving linux-next
a go often.
> > That said, we have a not-so-well known daily linux-next KOTD rpm
> > type of tree as well. So I recommend that as a next step.
> >
> > Due to the possible failures possible with linux-next, or random
> > regressions with other subsystems you often only want to test *one*
> > subsystem. To help with this there are two options I'm aware of:
>
> I still don't see what's wrong with booting -next? Fine, there will
> occasionally be the rare boot failure regressions, in which case you
> can move on to all your other stuff (which is very time intensive), but
> if -next boots fine, whether the bug is present or not tells you
> something and if it's not present, it saves you a lot of time which is
> very valuable because we shouldn't be wasting the time of our most
> valuable test group (those which are close to mainline).
Note how Laura mentioned they actually skip rc1. Even though my own
experience these days is linux-next is *more stable* than rc1, its
can't be a surprise linux-next can have issues.
We're talking about *all* development ramp up. One commit is bound
to have a pesky stupid thing merged.
> So I think it would be a useful service for distro's to provide a
> release of -next that users can try. Perhaps it doesn't have to be
> daily, but at least weekly would be enormously helpful.
Daily. No questions asked.
A) KOTD --> B) linux-next --> mailing list
Luis
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop
2017-06-27 19:02 ` Luis R. Rodriguez
@ 2017-06-27 19:46 ` Guenter Roeck
2017-06-28 10:19 ` Mark Brown
0 siblings, 1 reply; 23+ messages in thread
From: Guenter Roeck @ 2017-06-27 19:46 UTC (permalink / raw)
To: Luis R. Rodriguez; +Cc: James Bottomley, ksummit
On Tue, Jun 27, 2017 at 09:02:02PM +0200, Luis R. Rodriguez wrote:
> On Tue, Jun 27, 2017 at 11:30:34AM -0700, James Bottomley wrote:
> > On Tue, 2017-06-27 at 19:53 +0200, Luis R. Rodriguez wrote:
> > > The Kernel Of The Day (KOTD) helps *a lot*. On the XFS front I can
> > > say that 90% of the time so far most bugs can simply be reverse
> > > bisected by testing an issue with KOTD and if it works then doing a
> > > reverse bisect. So much so that I actually *yearn* for the day we get
> > > an actual real valid upstream bug. The other 10% BTW consist of "bad
> > > backports" so far.
> > >
> > > But one day it comes that KOTD is not sufficient, and there is that
> > > pesky delta on linux-next which *might* also have a fix for you.
> > > Problem is booting linux-next can often fail.
> >
> > Just a minute: I'd like to question that assumption. -next is supposed
> > to be all the upstream trees targetting the merge window. Boot failure
> > regressions in those trees are very rare.
>
> Like I said, we've gotten better. For my day to day development systems it
> is true that linux-next can be groovy. When it comes to actually booting it
> on a real system being evaluated though, your luck varies. My luck recently
> was not so great.
>
> Guenter Rock maintains a map of both kernel compile and qemu run time
> testign of linux-next accross a different set of architectures, he can
> perhaps tell you better how things look these days on his map. This is
> outside of the scope of the architectures that Stephen tests AFAICT.
>
There are almost always some build and/or runtime failures in -next. Right now
we are lucky (the current qemu boot failures are really build failures).
For reference, from next-20170627:
Build results:
total: 145 pass: 140 fail: 5
Failed builds:
arm:allmodconfig
arm64:allmodconfig
hexagon:defconfig
hexagon:allnoconfig
parisc:generic-64bit_defconfig
Qemu test results:
total: 122 pass: 120 fail: 2
Failed tests:
arm:versatilepb-scsi:versatile_defconfig:versatile-pb
sh:rts7751r2dplus_defconfig
I'd say maybe once or twice in a relase cycle I see half or more of
the qemu tests failing.
> > Fine, not non-existent so
> > that's why we run testing and inspection on them, but "often fail" is a
> > mischaracterisation. I think the correct characterisation would be
> > "rarely fail", but I can compromise on "sometimes fail".
>
> That's fair, how about this: for my development system linux-next rarely fails
> now. For production test system, linux-next sometimes fails :D
>
Depends on the scope. If "fail" refers to x86_64:defconfig, I would agree to
"rarely" (say, maybe once or twice a month on average ). If "fail" refers to
"one or more of my qemu boot tests fail", "almost always" would be more
accurate. If "fail" refers to "one or more of my build or qemu boot
tests fail", it would be "pretty much always".
> > > Based on personal experience with testing linux-next more regularly
> > > on more machines over the years I can say we are getting much better
> > > with this these days, but every now and then its just poop.
> >
> > Really? Even assuming it to be true for the sake of argument, next
> > stop for that "poop" is mainline via the merge window, so perhaps
> > detecting we have a problem before it hits would be a valuable service.
>
> Of course. Its why everyone and their uncles should be giving linux-next
> a go often.
>
> > > That said, we have a not-so-well known daily linux-next KOTD rpm
> > > type of tree as well. So I recommend that as a next step.
> > >
> > > Due to the possible failures possible with linux-next, or random
> > > regressions with other subsystems you often only want to test *one*
> > > subsystem. To help with this there are two options I'm aware of:
> >
> > I still don't see what's wrong with booting -next? Fine, there will
> > occasionally be the rare boot failure regressions, in which case you
> > can move on to all your other stuff (which is very time intensive), but
> > if -next boots fine, whether the bug is present or not tells you
> > something and if it's not present, it saves you a lot of time which is
> > very valuable because we shouldn't be wasting the time of our most
> > valuable test group (those which are close to mainline).
>
> Note how Laura mentioned they actually skip rc1. Even though my own
> experience these days is linux-next is *more stable* than rc1, its
> can't be a surprise linux-next can have issues.
>
I tend to agree, as long as you refer to -next a few days before the commit
window opens. Sometimes there is a flurry of commits showing up in the last
-next before the commit window opens (or in the first -next after it
opened). Those commits tend to make it into mainline almost immediately,
and my non-scientific impression is that they cause a high percentage of
failures.
Guenter
> We're talking about *all* development ramp up. One commit is bound
> to have a pesky stupid thing merged.
>
> > So I think it would be a useful service for distro's to provide a
> > release of -next that users can try. Perhaps it doesn't have to be
> > daily, but at least weekly would be enormously helpful.
>
> Daily. No questions asked.
>
> A) KOTD --> B) linux-next --> mailing list
>
> Luis
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop
2017-06-27 18:30 ` James Bottomley
2017-06-27 18:41 ` Daniel Vetter
2017-06-27 19:02 ` Luis R. Rodriguez
@ 2017-06-27 22:35 ` Jiri Kosina
2017-06-28 6:59 ` Takashi Iwai
2 siblings, 1 reply; 23+ messages in thread
From: Jiri Kosina @ 2017-06-27 22:35 UTC (permalink / raw)
To: James Bottomley; +Cc: ksummit
On Tue, 27 Jun 2017, James Bottomley wrote:
> So I think it would be a useful service for distro's to provide a
> release of -next that users can try. Perhaps it doesn't have to be
> daily, but at least weekly would be enormously helpful.
We do:
https://build.opensuse.org/repositories/Kernel:linux-next
But we are getting close to zero bugreports against it. Which either means
it's very rarely used (and therefore we should perhaps be more vocal about
it), or people who are educated enough to use it are actually educated
enough to report any issues directly upstream.
--
Jiri Kosina
SUSE Labs
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop
2017-06-27 22:35 ` Jiri Kosina
@ 2017-06-28 6:59 ` Takashi Iwai
0 siblings, 0 replies; 23+ messages in thread
From: Takashi Iwai @ 2017-06-28 6:59 UTC (permalink / raw)
To: Jiri Kosina; +Cc: James Bottomley, ksummit
On Wed, 28 Jun 2017 00:35:40 +0200,
Jiri Kosina wrote:
>
> On Tue, 27 Jun 2017, James Bottomley wrote:
>
> > So I think it would be a useful service for distro's to provide a
> > release of -next that users can try. Perhaps it doesn't have to be
> > daily, but at least weekly would be enormously helpful.
>
> We do:
>
> https://build.opensuse.org/repositories/Kernel:linux-next
>
> But we are getting close to zero bugreports against it. Which either means
> it's very rarely used (and therefore we should perhaps be more vocal about
> it), or people who are educated enough to use it are actually educated
> enough to report any issues directly upstream.
IMO, we should consider deploying more CI tests (either openQA or
simplified ones) for KOTD and linux-next. I discussed with out QA
guys once, but it's not really happening yet.
thanks,
Takashi
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop
2017-06-27 17:53 ` Luis R. Rodriguez
2017-06-27 18:26 ` Laurent Pinchart
2017-06-27 18:30 ` James Bottomley
@ 2017-06-27 18:31 ` Takashi Iwai
2017-06-27 19:04 ` Luis R. Rodriguez
2 siblings, 1 reply; 23+ messages in thread
From: Takashi Iwai @ 2017-06-27 18:31 UTC (permalink / raw)
To: Luis R. Rodriguez; +Cc: ksummit
On Tue, 27 Jun 2017 19:53:21 +0200,
Luis R. Rodriguez wrote:
>
> On Thu, Jun 22, 2017 at 02:36:13PM +0200, Jiri Kosina wrote:
> > On Wed, 21 Jun 2017, Laura Abbott wrote:
> >
> > > Fedora tends to follow the most recent stable kernel very closely
> > > (e.g. 4.11.6 is currently pending for Fedora 24, 25, and 26).
> > > This works well enough, but there still seem to be some
> > > disconnects in the bug reporting process. Examples I can think of:
> > >
> > > - When users report bugs on the Fedora tracker that look like
> > > actual upstream bugs, what's the best way to have those reported?
> > > I typically end up having to summarize from the Fedora bugzilla
> > > and send an e-mail which ends up being tedious. Can we make this
> > > bug reporting easier for non-kernel developers?
> >
> > Just as a data point -- we do a "Kernel of the day" build of a branch that
> > follows Linus' tree (with a few SUSE specific patches floating on top of
> > it) and provide it in an optional package repository.
> >
> > That allows the reporter to easily check whether the issue has been fixed
> > in latest upstream without needing to have the skills required to compile
> > own kernel.
> >
> > If the issue is confirmed to be present in latest upstream as well, our
> > internal person / maintainer responsible for that particular area usually
> > takes over (there are cases when the reporter prefers to report the bug
> > upstream by himself though).
> >
> > I am not sure if there is a way how to improve this process even further
> > ... do you have any particular ideas?
>
> The Kernel Of The Day (KOTD) helps *a lot*. On the XFS front I can say that 90%
> of the time so far most bugs can simply be reverse bisected by testing an issue
> with KOTD and if it works then doing a reverse bisect. So much so that I
> actually *yearn* for the day we get an actual real valid upstream bug. The
> other 10% BTW consist of "bad backports" so far.
>
> But one day it comes that KOTD is not sufficient, and there is that pesky delta
> on linux-next which *might* also have a fix for you. Problem is booting
> linux-next can often fail. Based on personal experience with testing linux-next
> more regularly on more machines over the years I can say we are getting much
> better with this these days, but every now and then its just poop. That said,
> we have a not-so-well known daily linux-next KOTD rpm type of tree as well.
> So I recommend that as a next step.
>
> Due to the possible failures possible with linux-next, or random regressions
> with other subsystems you often only want to test *one* subsystem. To help
> with this there are two options I'm aware of:
>
> o Subsystem maintiners also backport their -next tree for vanilla, in the
> the like of wireless-testing, which only carries 802.11 on Linus' tree.
> Not sure if other subsystems have similar type of trees, if not I encourage
> it.
>
> o Backports: backporting to random kernels can be a pain in the ass, but
> backporting to the KOTD should not take much effort if you have the
> right framework [0]. For instance I just created an XFS backport from
> linux-next to KOTD in one day's effort, I can use this to generate
> a tarball for modules for folks to try on top of KOTD. If this would
> actually be maintained upstream then the amount of work needed is even less,
> and you can have daily snapshots generated. Although sometimes backports
> can be buggy, to my surprise using Coccinelle actually has improved
> correctness of backports, this is only visible once you replace a series
> of patches with the output form an SmPL grammar patch. Given Coccinelle
> is also used, once you backport one subsystem driver, adding more is
> drivers from the same subsystem becomes relatively easier.
I guess it shouldn't be too difficult to build a kernel package based
on subsystem for-next branch regularly like we do for KOTD and
linux-next kernel packages. You'd need to set up some cron job to git
pull, repackage tarball and adjust config file somehow automatically,
then feed it to osc (in the case of openSUSE/SUSE OBS).
thanks,
Takashi
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop
2017-06-27 18:31 ` Takashi Iwai
@ 2017-06-27 19:04 ` Luis R. Rodriguez
2017-06-28 8:04 ` Daniel Vetter
0 siblings, 1 reply; 23+ messages in thread
From: Luis R. Rodriguez @ 2017-06-27 19:04 UTC (permalink / raw)
To: Takashi Iwai; +Cc: ksummit
On Tue, Jun 27, 2017 at 08:31:17PM +0200, Takashi Iwai wrote:
> On Tue, 27 Jun 2017 19:53:21 +0200,
> Luis R. Rodriguez wrote:
> >
> > On Thu, Jun 22, 2017 at 02:36:13PM +0200, Jiri Kosina wrote:
> > > On Wed, 21 Jun 2017, Laura Abbott wrote:
> > >
> > > > Fedora tends to follow the most recent stable kernel very closely
> > > > (e.g. 4.11.6 is currently pending for Fedora 24, 25, and 26).
> > > > This works well enough, but there still seem to be some
> > > > disconnects in the bug reporting process. Examples I can think of:
> > > >
> > > > - When users report bugs on the Fedora tracker that look like
> > > > actual upstream bugs, what's the best way to have those reported?
> > > > I typically end up having to summarize from the Fedora bugzilla
> > > > and send an e-mail which ends up being tedious. Can we make this
> > > > bug reporting easier for non-kernel developers?
> > >
> > > Just as a data point -- we do a "Kernel of the day" build of a branch that
> > > follows Linus' tree (with a few SUSE specific patches floating on top of
> > > it) and provide it in an optional package repository.
> > >
> > > That allows the reporter to easily check whether the issue has been fixed
> > > in latest upstream without needing to have the skills required to compile
> > > own kernel.
> > >
> > > If the issue is confirmed to be present in latest upstream as well, our
> > > internal person / maintainer responsible for that particular area usually
> > > takes over (there are cases when the reporter prefers to report the bug
> > > upstream by himself though).
> > >
> > > I am not sure if there is a way how to improve this process even further
> > > ... do you have any particular ideas?
> >
> > The Kernel Of The Day (KOTD) helps *a lot*. On the XFS front I can say that 90%
> > of the time so far most bugs can simply be reverse bisected by testing an issue
> > with KOTD and if it works then doing a reverse bisect. So much so that I
> > actually *yearn* for the day we get an actual real valid upstream bug. The
> > other 10% BTW consist of "bad backports" so far.
> >
> > But one day it comes that KOTD is not sufficient, and there is that pesky delta
> > on linux-next which *might* also have a fix for you. Problem is booting
> > linux-next can often fail. Based on personal experience with testing linux-next
> > more regularly on more machines over the years I can say we are getting much
> > better with this these days, but every now and then its just poop. That said,
> > we have a not-so-well known daily linux-next KOTD rpm type of tree as well.
> > So I recommend that as a next step.
> >
> > Due to the possible failures possible with linux-next, or random regressions
> > with other subsystems you often only want to test *one* subsystem. To help
> > with this there are two options I'm aware of:
> >
> > o Subsystem maintiners also backport their -next tree for vanilla, in the
> > the like of wireless-testing, which only carries 802.11 on Linus' tree.
> > Not sure if other subsystems have similar type of trees, if not I encourage
> > it.
> >
> > o Backports: backporting to random kernels can be a pain in the ass, but
> > backporting to the KOTD should not take much effort if you have the
> > right framework [0]. For instance I just created an XFS backport from
> > linux-next to KOTD in one day's effort, I can use this to generate
> > a tarball for modules for folks to try on top of KOTD. If this would
> > actually be maintained upstream then the amount of work needed is even less,
> > and you can have daily snapshots generated. Although sometimes backports
> > can be buggy, to my surprise using Coccinelle actually has improved
> > correctness of backports, this is only visible once you replace a series
> > of patches with the output form an SmPL grammar patch. Given Coccinelle
> > is also used, once you backport one subsystem driver, adding more is
> > drivers from the same subsystem becomes relatively easier.
>
> I guess it shouldn't be too difficult to build a kernel package based
> on subsystem for-next branch regularly like we do for KOTD and
> linux-next kernel packages. You'd need to set up some cron job to git
> pull, repackage tarball and adjust config file somehow automatically,
> then feed it to osc (in the case of openSUSE/SUSE OBS).
That'd be great. It sounds like we have trees like this for media, and
wireless. Not sure of others.
Luis
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug reporting feedback loop
2017-06-27 19:04 ` Luis R. Rodriguez
@ 2017-06-28 8:04 ` Daniel Vetter
0 siblings, 0 replies; 23+ messages in thread
From: Daniel Vetter @ 2017-06-28 8:04 UTC (permalink / raw)
To: Luis R. Rodriguez; +Cc: ksummit
On Tue, Jun 27, 2017 at 9:04 PM, Luis R. Rodriguez <mcgrof@kernel.org> wrote:
>> I guess it shouldn't be too difficult to build a kernel package based
>> on subsystem for-next branch regularly like we do for KOTD and
>> linux-next kernel packages. You'd need to set up some cron job to git
>> pull, repackage tarball and adjust config file somehow automatically,
>> then feed it to osc (in the case of openSUSE/SUSE OBS).
>
> That'd be great. It sounds like we have trees like this for media, and
> wireless. Not sure of others.
In case anyone cares, I'm trying to make drm-tip _the_ integration
tree for graphics. Doesn't contain everything yet, but generally a
good place for build-bots, and iirc at least ubuntu has a ppa for it:
https://cgit.freedesktop.org/drm-tip
And the integration manifest:
https://cgit.freedesktop.org/drm-tip/tree/integration-manifest
Probably the neatest thing is that there's no single person
maintaining it, but conflicts get resolved and stored when they
happen, by the person who pushes the patches creating the conflict.
See
https://cgit.freedesktop.org/drm-tip/log/?h=rerere-cache
So yeah we've managed to scale maintaining an integration tree to a
team of 20+ people, everyone with commit rights in the various trees
essentially :-)
Cheers, Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
^ permalink raw reply [flat|nested] 23+ messages in thread