[Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[James Bottomley]

I'm seeing a lot of wasted effort by our customers on kernel bugs and
trying to engage the distribution to fix them.  As a caveat, I'm
working in the cloud, so the distributions in question are usually
community ones not enterprise ones.  However, we do have a fair few
customers on LTS kernels from Distributions.

Mostly they find a cloud performance regression, they try to engage the
distro, spend ages working on it or submitting bugs and usually end up
with an unsatisfactory result.  By the time they call my team in, we've
likely only got a week to fix the issue.  However, step one is always
confirming whether upstream works (95% of the time it does) and then
finding the fix by bisection (usually assisted by knowledge of where
the bug is).  To do the bisection we usually have to build a kernel
package with our guesses and get them to try it, so it can be a bit
slow.  Once we have the backport, we send it to stable and notify the
distribution to include it in their next kernel release.

Here's the rub: community distributions (even LTS ones) don't have the
resources even to triage cloud bugs in environments they likely can't
reproduce, so we really need to develop assistive tools for customers
to perform bisections to identify what caused the bug or (in the 95%
case) what fixed it.  Having a bugzilla and using it as first line of
support implies a service expectation (usually coming from Enterprise)
that simply isn't met, so distributions need to fix this at the point
of interaction: bugzilla.

The first suggestion is that kernel builds are pretty much automated
and we try to make every commit buildable, so could we automate the
machinery that allows a customer to do bisection simply by installing a
kernel package? (we here, obviously means the distro, but going from
git bisect to kernel package would be the useful link).

Second suggestion is that the bugzillas need to say much more strongly
that the reporter really needs to confirm the fix in upstream and do
the bisection themselves (and ideally request the backport to stable
themselves).

James

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 799 bytes --]

On Wed, Sep 05, 2018 at 11:13:52AM +0100, James Bottomley wrote:

> The first suggestion is that kernel builds are pretty much automated
> and we try to make every commit buildable, so could we automate the
> machinery that allows a customer to do bisection simply by installing a
> kernel package? (we here, obviously means the distro, but going from
> git bisect to kernel package would be the useful link).

Improving bisectability would obviously help with other testing efforts
too - we have existing users, Guillaume Tucker implemented automated
bisection support in KernelCI which is incredibly useful providing one
can actually bisect.  Right now it works pretty well a lot of the time
but there are cases where it gets messy, especially when you add boot
issues onto the buildability ones.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Paul E. McKenney]

On Wed, Sep 05, 2018 at 12:37:15PM +0100, Mark Brown wrote:
> On Wed, Sep 05, 2018 at 11:13:52AM +0100, James Bottomley wrote:
> 
> > The first suggestion is that kernel builds are pretty much automated
> > and we try to make every commit buildable, so could we automate the
> > machinery that allows a customer to do bisection simply by installing a
> > kernel package? (we here, obviously means the distro, but going from
> > git bisect to kernel package would be the useful link).
> 
> Improving bisectability would obviously help with other testing efforts
> too - we have existing users, Guillaume Tucker implemented automated
> bisection support in KernelCI which is incredibly useful providing one
> can actually bisect.  Right now it works pretty well a lot of the time
> but there are cases where it gets messy, especially when you add boot
> issues onto the buildability ones.

I am one of those strange people who rebase in order to improve
bisectability.  But one reason I can do that is that I have relatively
few patches, and it gets harder the more patches I am carrying.  I suppose
that someone (not me!) could rebase -stable to make it more bisectable,
but that sounds difficult, painful, and error-prone.  Could added tooling
make bisection work better?  Sounds valuable, but non-trivial.

In some of my bisection efforts, I have had to apply fix patches to
fix various unrelated bugs.  I suppose that this could be automated,
for example by tracking which fix-patches are needed at which potential
bisection points, though this sounds like a large effort.  Of course,
automated backporting of patches would make it easier, and much else
easier as well.  ;-)

							Thanx, Paul

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Steven Rostedt]

On Wed, 5 Sep 2018 08:03:15 -0700
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:

> I am one of those strange people who rebase in order to improve
> bisectability.  But one reason I can do that is that I have relatively
> few patches, and it gets harder the more patches I am carrying.  I suppose
> that someone (not me!) could rebase -stable to make it more bisectable,

How would rebasing it make stable more bisectable? Once you rebase, you
don't have a tree that use to work? Although I guess you may find the
commit that caused the problem better. But rebasing creates a lot of
other issues, I would not recommend rebasing stable, as that would
totally break the RT stable tree work flow.

-- Steve


> but that sounds difficult, painful, and error-prone.  Could added tooling
> make bisection work better?  Sounds valuable, but non-trivial.

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Paul E. McKenney]

On Wed, Sep 05, 2018 at 11:50:08AM -0400, Steven Rostedt wrote:
> On Wed, 5 Sep 2018 08:03:15 -0700
> "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> 
> > I am one of those strange people who rebase in order to improve
> > bisectability.  But one reason I can do that is that I have relatively
> > few patches, and it gets harder the more patches I am carrying.  I suppose
> > that someone (not me!) could rebase -stable to make it more bisectable,
> 
> How would rebasing it make stable more bisectable? Once you rebase, you
> don't have a tree that use to work? Although I guess you may find the
> commit that caused the problem better. But rebasing creates a lot of
> other issues, I would not recommend rebasing stable, as that would
> totally break the RT stable tree work flow.

Instead of leaving the buggy commit and the span where the bug exists,
you rebase the fix into the original buggy fix.

And I bet that rebasing -stable would cause no end of broken glass in
a great many projects.  ;-)

							Thanx, Paul

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[James Bottomley]

On Wed, 2018-09-05 at 09:20 -0700, Paul E. McKenney wrote:
> On Wed, Sep 05, 2018 at 11:50:08AM -0400, Steven Rostedt wrote:
> > On Wed, 5 Sep 2018 08:03:15 -0700
> > "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> > 
> > > I am one of those strange people who rebase in order to improve
> > > bisectability.  But one reason I can do that is that I have
> > > relatively
> > > few patches, and it gets harder the more patches I am
> > > carrying.  I suppose
> > > that someone (not me!) could rebase -stable to make it more
> > > bisectable,
> > 
> > How would rebasing it make stable more bisectable? Once you rebase,
> > you don't have a tree that use to work? Although I guess you may
> > find the commit that caused the problem better. But rebasing
> > creates a lot of other issues, I would not recommend rebasing
> > stable, as that would totally break the RT stable tree work flow.
> 
> Instead of leaving the buggy commit and the span where the bug
> exists, you rebase the fix into the original buggy fix.

We do this in SCSI as well, but only if the tree hasn't yet been
submitted to Linus.  The technical term is folding.  It's obviously
better to fix buggy commits that haven't gone upstream because it
improves bisectability.

> And I bet that rebasing -stable would cause no end of broken glass in
> a great many  projects.  ;-)

If others rely on your tree, rebasing is harder and must be done more
carefully and with co-ordination, but it's not impossible assuming you
have a problem big enough.  Again, it's an expediency based trade off.

James

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Paul E. McKenney]

On Wed, Sep 05, 2018 at 05:45:14PM +0100, James Bottomley wrote:
> On Wed, 2018-09-05 at 09:20 -0700, Paul E. McKenney wrote:
> > On Wed, Sep 05, 2018 at 11:50:08AM -0400, Steven Rostedt wrote:
> > > On Wed, 5 Sep 2018 08:03:15 -0700
> > > "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote:
> > > 
> > > > I am one of those strange people who rebase in order to improve
> > > > bisectability.  But one reason I can do that is that I have
> > > > relatively
> > > > few patches, and it gets harder the more patches I am
> > > > carrying.  I suppose
> > > > that someone (not me!) could rebase -stable to make it more
> > > > bisectable,
> > > 
> > > How would rebasing it make stable more bisectable? Once you rebase,
> > > you don't have a tree that use to work? Although I guess you may
> > > find the commit that caused the problem better. But rebasing
> > > creates a lot of other issues, I would not recommend rebasing
> > > stable, as that would totally break the RT stable tree work flow.
> > 
> > Instead of leaving the buggy commit and the span where the bug
> > exists, you rebase the fix into the original buggy fix.
> 
> We do this in SCSI as well, but only if the tree hasn't yet been
> submitted to Linus.  The technical term is folding.  It's obviously
> better to fix buggy commits that haven't gone upstream because it
> improves bisectability.

So I am not the only one doing this.  ;-)

> > And I bet that rebasing -stable would cause no end of broken glass in
> > a great many  projects.  ;-)
> 
> If others rely on your tree, rebasing is harder and must be done more
> carefully and with co-ordination, but it's not impossible assuming you
> have a problem big enough.  Again, it's an expediency based trade off.

I use date-coded branches to keep the old commits around.  I delete
them periodically, but keep them around for at least six months, on the
theory that if you are developing against a two-year old -rcu branch,
you have bigger problems.  And you should have instead developed against
a formal release anyway.

							Thanx, Paul

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Jiri Kosina]

On Wed, 5 Sep 2018, James Bottomley wrote:

> We do this in SCSI as well, but only if the tree hasn't yet been 
> submitted to Linus.  The technical term is folding.  It's obviously 
> better to fix buggy commits that haven't gone upstream because it 
> improves bisectability.

We are drifting away a bit here, but now that you have mentioned it, let 
me add a datapoint to this -- it's actually causing issues to our 
workflow, as we have scsi.git as one of the upstreams [1], and when you 
rebase, it blows up our git workflow and we have to fixup things manually.

So if you are aware of your tree having downstreams, and care about not 
breaking them and want to be nice to them, you shouldn't rebase that tree 
[2].

[1] there are some funny technical details, but in basic principle it's 
    exactly like that

[2] https://yarchive.net/comp/linux/git_rebase.html

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[James Bottomley]

On Wed, 2018-09-05 at 21:25 +0200, Jiri Kosina wrote:
> On Wed, 5 Sep 2018, James Bottomley wrote:
> 
> > We do this in SCSI as well, but only if the tree hasn't yet been 
> > submitted to Linus.  The technical term is folding.  It's
> > obviously better to fix buggy commits that haven't gone upstream
> > because it improves bisectability.
> 
> We are drifting away a bit here, but now that you have mentioned it,
> let me add a datapoint to this -- it's actually causing issues to
> our workflow, as we have scsi.git as one of the upstreams [1], and
> when you rebase, it blows up our git workflow and we have to fixup
> things manually.

Describe the issues and we'll try to come up with a fix, but on the
whole you should regard the scsi trees somewhat similarly to linux-
next: it's our proposal for a patch set but we may update it.

> So if you are aware of your tree having downstreams, and care about
> not breaking them and want to be nice to them, you shouldn't rebase
> that tree [2].

Well, I wasn't aware of this one, but I'm sure we can come up with a
functional workflow once I understand what's happening with the
downstream tree.

James


> [1] there are some funny technical details, but in basic principle
> it's exactly like that
> 
> [2] https://yarchive.net/comp/linux/git_rebase.html
> 
> Thanks,
> 

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Jiri Kosina]

On Wed, 5 Sep 2018, James Bottomley wrote:

> > We are drifting away a bit here, but now that you have mentioned it,
> > let me add a datapoint to this -- it's actually causing issues to
> > our workflow, as we have scsi.git as one of the upstreams [1], and
> > when you rebase, it blows up our git workflow and we have to fixup
> > things manually.
> 
> Describe the issues and we'll try to come up with a fix, but on the
> whole you should regard the scsi trees somewhat similarly to linux-
> next: it's our proposal for a patch set but we may update it.

We order patches in our trees in the same git-topological-ordering as they 
are upstream. It has a lot of benefits, most importantly: it doesn't 
introduce artificial conflicts that don't exist in reality.

In order to achieve that, we of course need 1:1 mapping between our 
patches and upstream commits.
Rebases destroy that mapping.

And in some areas (scsi is one, but not the only one), we basically had no 
other choice than considering maintainer's tree to be already "upstream 
enough", without waiting for Linus' tree merge.

> > So if you are aware of your tree having downstreams, and care about
> > not breaking them and want to be nice to them, you shouldn't rebase
> > that tree [2].
> 
> Well, I wasn't aware of this one, but I'm sure we can come up with a
> functional workflow once I understand what's happening with the
> downstream tree.

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Martin K. Petersen]

Hi Jiri,

Sorry I'm a little late to the game here. Been out on vacation.

> We order patches in our trees in the same git-topological-ordering as they 
> are upstream. It has a lot of benefits, most importantly: it doesn't 
> introduce artificial conflicts that don't exist in reality.
>
> In order to achieve that, we of course need 1:1 mapping between our
> patches and upstream commits.  Rebases destroy that mapping.
>
> And in some areas (scsi is one, but not the only one), we basically had no 
> other choice than considering maintainer's tree to be already "upstream 
> enough", without waiting for Linus' tree merge.

When I discussed this with Johannes a little while ago, I suggested you
guys used git patch-id to track patches instead of commit ids. That's
how we track patches applied across many different trees internally.
Works much better than using the upstream sha.

I would like to understand your "upstream enough" requirement. Why do
you need a tree that's stable before Linus pulls the changes?

Note that I am generally only rebasing as a last resort and typically
only very early in the rc cycle. It usually happens when I need to drop
a patch series that turned out to be unfixable in its current state.

And before everyone screams because I'm not supposed to be pushing stuff
that breaks, please realize that it is impossible to test all the
different types of hardware I have to merge drivers for. There is no
regression test suite or lab setup with anything resembling
comprehensive coverage. I test changes to the SCSI core code and do some
rudimentary testing on SAS and FC on x86_64. But that's really the best
I can do.

Even though most patches posted to linux-scsi get picked up by 0day,
more often than not they only get x86_64 build coverage. Whereas 0day
build failures on arm, mips, sparc32, whatever typically only get
reported after patches have been simmering in linux-next for a
while. Depends how busy 0day is.

Also, actual driver failures on platforms not officially supported and
tested by the controller vendor are only found after the fact. And most
of the time it's not a matter of reverting a single patch but
effectively dropping all of the patches in the series until they can be
reworked. Sometimes a workaround takes a week or two to deliver, and
people don't appreciate not being able to boot their systems in the
meantime. So that's why I generally drop the series instead.

I would love for every patch sent to linux-scsi to be bug free and
instantly build tested by 0day on every architecture.  And I would love
for hardware vendors to be more cognizant about architectures they don't
commercially support.  But reality is that things break frequently when
I merge big, complex driver update patch series.

As a result, the preference has been to have the flexibility to amend or
drop patches early in every cycle. It hasn't really been a problem
because there have been no downstream users of SCSI at all. I only
recently found out about your use case.

I'm pretty flexible about how to address this, there are a couple of
ways to go about it.

1. I could just always revert instead of dropping the patches. The
   downside is that we end up with a pretty messy history because, as I
   pointed out above, it's usually a matter of dropping tens of patches
   at a time and not reverting a single offending commit. In addition,
   having a messy history makes it harder on distro kernel people to
   track driver updates.

2. The other option is that I set up a scsi-staging tree where stuff can
   simmer for a bit and hopefully get some 0day coverage before getting
   shuffled over to scsi-queue. However, I do question how much actual
   real hardware coverage we'll get by having a SCSI tree that people
   would explicitly have to pull to test. As opposed to linux-next which
   at least gets some coverage by test farms and users.

3. The third option is that we pick a number like rc4 and I will promise
   not to rebase after that. We can see how that works out.

I'm open to ideas. The most important thing to me is that 0day and
linux-next are indispensable tools in my workflow given that I have no
way to personally test most of the code I merge. So it is imperative
that I have the ability to push code early because that's where I get
the bulk of my (non-SCSI-core) build and test coverage.

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Geert Uytterhoeven]

On Tue, Sep 18, 2018 at 3:43 PM Martin K. Petersen
<martin.petersen@oracle.com> wrote:
> > We order patches in our trees in the same git-topological-ordering as they
> > are upstream. It has a lot of benefits, most importantly: it doesn't
> > introduce artificial conflicts that don't exist in reality.
> >
> > In order to achieve that, we of course need 1:1 mapping between our
> > patches and upstream commits.  Rebases destroy that mapping.
> >
> > And in some areas (scsi is one, but not the only one), we basically had no
> > other choice than considering maintainer's tree to be already "upstream
> > enough", without waiting for Linus' tree merge.
>
> When I discussed this with Johannes a little while ago, I suggested you
> guys used git patch-id to track patches instead of commit ids. That's
> how we track patches applied across many different trees internally.
> Works much better than using the upstream sha.
>
> I would like to understand your "upstream enough" requirement. Why do
> you need a tree that's stable before Linus pulls the changes?
>
> Note that I am generally only rebasing as a last resort and typically
> only very early in the rc cycle. It usually happens when I need to drop
> a patch series that turned out to be unfixable in its current state.
>
> And before everyone screams because I'm not supposed to be pushing stuff
> that breaks, please realize that it is impossible to test all the
> different types of hardware I have to merge drivers for. There is no
> regression test suite or lab setup with anything resembling
> comprehensive coverage. I test changes to the SCSI core code and do some
> rudimentary testing on SAS and FC on x86_64. But that's really the best
> I can do.
>
> Even though most patches posted to linux-scsi get picked up by 0day,
> more often than not they only get x86_64 build coverage. Whereas 0day
> build failures on arm, mips, sparc32, whatever typically only get
> reported after patches have been simmering in linux-next for a
> while. Depends how busy 0day is.
>
> Also, actual driver failures on platforms not officially supported and
> tested by the controller vendor are only found after the fact. And most
> of the time it's not a matter of reverting a single patch but
> effectively dropping all of the patches in the series until they can be
> reworked. Sometimes a workaround takes a week or two to deliver, and
> people don't appreciate not being able to boot their systems in the
> meantime. So that's why I generally drop the series instead.
>
> I would love for every patch sent to linux-scsi to be bug free and
> instantly build tested by 0day on every architecture.  And I would love
> for hardware vendors to be more cognizant about architectures they don't
> commercially support.  But reality is that things break frequently when
> I merge big, complex driver update patch series.
>
> As a result, the preference has been to have the flexibility to amend or
> drop patches early in every cycle. It hasn't really been a problem
> because there have been no downstream users of SCSI at all. I only
> recently found out about your use case.
>
> I'm pretty flexible about how to address this, there are a couple of
> ways to go about it.
>
> 1. I could just always revert instead of dropping the patches. The
>    downside is that we end up with a pretty messy history because, as I
>    pointed out above, it's usually a matter of dropping tens of patches
>    at a time and not reverting a single offending commit. In addition,
>    having a messy history makes it harder on distro kernel people to
>    track driver updates.
>
> 2. The other option is that I set up a scsi-staging tree where stuff can
>    simmer for a bit and hopefully get some 0day coverage before getting
>    shuffled over to scsi-queue. However, I do question how much actual
>    real hardware coverage we'll get by having a SCSI tree that people
>    would explicitly have to pull to test. As opposed to linux-next which
>    at least gets some coverage by test farms and users.
>
> 3. The third option is that we pick a number like rc4 and I will promise
>    not to rebase after that. We can see how that works out.
>
> I'm open to ideas. The most important thing to me is that 0day and
> linux-next are indispensable tools in my workflow given that I have no
> way to personally test most of the code I merge. So it is imperative
> that I have the ability to push code early because that's where I get
> the bulk of my (non-SCSI-core) build and test coverage.

You could start using topic branches (for core, and per driver/vendor),
and recreate your for-next branch as a merge of all topic branches on a
daily basis. If one patch series turns out to be bad, at least the commit
IDs in the other topic branches will remain stable.

I believe Mark Brown used to have such a system (except for dropping
patches, he never rebases his topic branches), unless he got bashed by
Linus for his cephalopodic merges having more branches than an octopus(?).

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Martin K. Petersen]

Geert,

> You could start using topic branches (for core, and per driver/vendor),
> and recreate your for-next branch as a merge of all topic branches on a
> daily basis. If one patch series turns out to be bad, at least the commit
> IDs in the other topic branches will remain stable.

Yeah. I thought about doing what Jens does, splitting core and drivers
into separate branches. However, compared to block I have so few SCSI
core updates that I am not convinced it's worth it. Core changes only
amount to a couple percent of the total number of commits every release.

For topic branches to be truly effective I'd need a separate branch per
driver. Not impossible, but it seems like a pretty big hammer...

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Christoph Hellwig]

On Tue, Sep 18, 2018 at 11:01:50AM -0400, Martin K. Petersen wrote:
> Yeah. I thought about doing what Jens does, splitting core and drivers
> into separate branches. However, compared to block I have so few SCSI
> core updates that I am not convinced it's worth it. Core changes only
> amount to a couple percent of the total number of commits every release.

I did that for the SCSI tree a long time ago, and it did not work out
at all as the few core changes are usually required by driver changes.

Note that I've not really seen the split recently in the block tree
either, probably for the same reason.

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Jens Axboe]

On 9/18/18 9:27 AM, Christoph Hellwig wrote:
> On Tue, Sep 18, 2018 at 11:01:50AM -0400, Martin K. Petersen wrote:
>> Yeah. I thought about doing what Jens does, splitting core and drivers
>> into separate branches. However, compared to block I have so few SCSI
>> core updates that I am not convinced it's worth it. Core changes only
>> amount to a couple percent of the total number of commits every release.
> 
> I did that for the SCSI tree a long time ago, and it did not work out
> at all as the few core changes are usually required by driver changes.
> 
> Note that I've not really seen the split recently in the block tree
> either, probably for the same reason.

Yeah exactly, I basically stopped doing that as the most active driver
was NVMe, and for quite a while that also helped drive core changes.
Since they went hand in hand, there wasn't much point in doing the split
branches anymore, in fact it was harmful.

I'd love to start doing it again, and I may do that, since we've now got
less feature additions that end up depending on core changes.

-- 
Jens Axboe

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 658 bytes --]

On Tue, Sep 18, 2018 at 08:27:11AM -0700, Christoph Hellwig wrote:

> I did that for the SCSI tree a long time ago, and it did not work out
> at all as the few core changes are usually required by driver changes.

> Note that I've not really seen the split recently in the block tree
> either, probably for the same reason.

Most of my topic branches end up being per-driver ones, with embedded
stuff that often winds up being useful for cross tree merges that
weren't anticipated.  The core changes that affect or are used by
drivers either get the relevant driver changes stacked onto the same
branch or merged into the driver branches depending on taste.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 781 bytes --]

On Tue, Sep 18, 2018 at 09:43:08AM -0400, Martin K. Petersen wrote:

> 1. I could just always revert instead of dropping the patches. The
>    downside is that we end up with a pretty messy history because, as I
>    pointed out above, it's usually a matter of dropping tens of patches
>    at a time and not reverting a single offending commit. In addition,
>    having a messy history makes it harder on distro kernel people to
>    track driver updates.

I used to deal with this by using topic branches heavily and making my
-next be an automated merge of those branches, if something went badly I
could just throw away the branch.  I stopped for a while because Linus
didn't like the number of branches I was creating, though it wasn't a
problem with the approach in general.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Takashi Iwai]

On Tue, 18 Sep 2018 18:12:01 +0200,
Mark Brown wrote:
> 
> On Tue, Sep 18, 2018 at 09:43:08AM -0400, Martin K. Petersen wrote:
> 
> > 1. I could just always revert instead of dropping the patches. The
> >    downside is that we end up with a pretty messy history because, as I
> >    pointed out above, it's usually a matter of dropping tens of patches
> >    at a time and not reverting a single offending commit. In addition,
> >    having a messy history makes it harder on distro kernel people to
> >    track driver updates.
> 
> I used to deal with this by using topic branches heavily and making my
> -next be an automated merge of those branches, if something went badly I
> could just throw away the branch.  I stopped for a while because Linus
> didn't like the number of branches I was creating, though it wasn't a
> problem with the approach in general.

I think that the topic branch approach would work well if you merge
topic branches back to the main branch more often.  That is, each
topic branch lives only for a relatively short time (e.g. a few
weeks), and not merging the whole branches in a shot at the end of the
development cycle.

Basically the merge to the main branch "fixates" the developments of
the given topic, hence other people can start working on the commits
safely.  Meanwhile the not-yet-merged branches can be still thrown
away if they are really bad.


Takashi

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1301 bytes --]

On Tue, Sep 18, 2018 at 10:20:17PM +0200, Takashi Iwai wrote:
> Mark Brown wrote:

> > I used to deal with this by using topic branches heavily and making my
> > -next be an automated merge of those branches, if something went badly I
> > could just throw away the branch.  I stopped for a while because Linus
> > didn't like the number of branches I was creating, though it wasn't a
> > problem with the approach in general.

> I think that the topic branch approach would work well if you merge
> topic branches back to the main branch more often.  That is, each
> topic branch lives only for a relatively short time (e.g. a few
> weeks), and not merging the whole branches in a shot at the end of the
> development cycle.

That approach tends to lead to more merges (as you end up with more of
the small topic branches) and if you don't have any commits actually on
the trunk then the long series of merges is also seen as an antipattern
by some.

> Basically the merge to the main branch "fixates" the developments of
> the given topic, hence other people can start working on the commits
> safely.  Meanwhile the not-yet-merged branches can be still thrown
> away if they are really bad.

For less widely used stuff it can be unfortunately difficult to get the
confidence that things are done :(

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Takashi Iwai]

On Tue, 18 Sep 2018 15:43:08 +0200,
Martin K. Petersen wrote:
> 
> 
> Hi Jiri,
> 
> Sorry I'm a little late to the game here. Been out on vacation.
> 
> > We order patches in our trees in the same git-topological-ordering as they 
> > are upstream. It has a lot of benefits, most importantly: it doesn't 
> > introduce artificial conflicts that don't exist in reality.
> >
> > In order to achieve that, we of course need 1:1 mapping between our
> > patches and upstream commits.  Rebases destroy that mapping.
> >
> > And in some areas (scsi is one, but not the only one), we basically had no 
> > other choice than considering maintainer's tree to be already "upstream 
> > enough", without waiting for Linus' tree merge.
> 
> When I discussed this with Johannes a little while ago, I suggested you
> guys used git patch-id to track patches instead of commit ids. That's
> how we track patches applied across many different trees internally.
> Works much better than using the upstream sha.

What is git patch id?  Is there any internal patch tag?

> I would like to understand your "upstream enough" requirement. Why do
> you need a tree that's stable before Linus pulls the changes?

A few reasons:
- We'd like to sort also the multiple would-be-upstream patches in the
  git topo order, which allows us to keep the patches closer to the
  original shape.

- Once when such patches are merged to upstream, they are moved to the
  upstream section, again ordered.  For moving them, of course it'd be
  easy if the commit ids are kept.

And here the worst case is that the old commit ID disappears after the
rebase.  In that sense, at least, it'd be helpful if you can preserve
the old branches via tag or another named branch in the repo; then the
commits are not pruned, and we can still track the order of patches.

> Note that I am generally only rebasing as a last resort and typically
> only very early in the rc cycle. It usually happens when I need to drop
> a patch series that turned out to be unfixable in its current state.

... and we shouldn't take such a patch series always as
"would-be-upstream" thing, either.

So it sounds like a sort of lack of communication to me; you might be
interested in that a certain patch series are already pushed to distro
kernels, and we are definitely interested in whether the patches are
merged or dropped.  Hmm...


Takashi

> And before everyone screams because I'm not supposed to be pushing stuff
> that breaks, please realize that it is impossible to test all the
> different types of hardware I have to merge drivers for. There is no
> regression test suite or lab setup with anything resembling
> comprehensive coverage. I test changes to the SCSI core code and do some
> rudimentary testing on SAS and FC on x86_64. But that's really the best
> I can do.
> 
> Even though most patches posted to linux-scsi get picked up by 0day,
> more often than not they only get x86_64 build coverage. Whereas 0day
> build failures on arm, mips, sparc32, whatever typically only get
> reported after patches have been simmering in linux-next for a
> while. Depends how busy 0day is.
> 
> Also, actual driver failures on platforms not officially supported and
> tested by the controller vendor are only found after the fact. And most
> of the time it's not a matter of reverting a single patch but
> effectively dropping all of the patches in the series until they can be
> reworked. Sometimes a workaround takes a week or two to deliver, and
> people don't appreciate not being able to boot their systems in the
> meantime. So that's why I generally drop the series instead.
> 
> I would love for every patch sent to linux-scsi to be bug free and
> instantly build tested by 0day on every architecture.  And I would love
> for hardware vendors to be more cognizant about architectures they don't
> commercially support.  But reality is that things break frequently when
> I merge big, complex driver update patch series.
> 
> As a result, the preference has been to have the flexibility to amend or
> drop patches early in every cycle. It hasn't really been a problem
> because there have been no downstream users of SCSI at all. I only
> recently found out about your use case.
> 
> I'm pretty flexible about how to address this, there are a couple of
> ways to go about it.
> 
> 1. I could just always revert instead of dropping the patches. The
>    downside is that we end up with a pretty messy history because, as I
>    pointed out above, it's usually a matter of dropping tens of patches
>    at a time and not reverting a single offending commit. In addition,
>    having a messy history makes it harder on distro kernel people to
>    track driver updates.
> 
> 2. The other option is that I set up a scsi-staging tree where stuff can
>    simmer for a bit and hopefully get some 0day coverage before getting
>    shuffled over to scsi-queue. However, I do question how much actual
>    real hardware coverage we'll get by having a SCSI tree that people
>    would explicitly have to pull to test. As opposed to linux-next which
>    at least gets some coverage by test farms and users.
> 
> 3. The third option is that we pick a number like rc4 and I will promise
>    not to rebase after that. We can see how that works out.
> 
> I'm open to ideas. The most important thing to me is that 0day and
> linux-next are indispensable tools in my workflow given that I have no
> way to personally test most of the code I merge. So it is imperative
> that I have the ability to push code early because that's where I get
> the bulk of my (non-SCSI-core) build and test coverage.
> 
> -- 
> Martin K. Petersen	Oracle Linux Engineering
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
> 

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Geert Uytterhoeven]

Hi Iwai-san,

On Tue, Sep 18, 2018 at 10:37 PM Takashi Iwai <tiwai@suse.de> wrote:
> On Tue, 18 Sep 2018 15:43:08 +0200,
> Martin K. Petersen wrote:
> > Sorry I'm a little late to the game here. Been out on vacation.
> >
> > > We order patches in our trees in the same git-topological-ordering as they
> > > are upstream. It has a lot of benefits, most importantly: it doesn't
> > > introduce artificial conflicts that don't exist in reality.
> > >
> > > In order to achieve that, we of course need 1:1 mapping between our
> > > patches and upstream commits.  Rebases destroy that mapping.
> > >
> > > And in some areas (scsi is one, but not the only one), we basically had no
> > > other choice than considering maintainer's tree to be already "upstream
> > > enough", without waiting for Linus' tree merge.
> >
> > When I discussed this with Johannes a little while ago, I suggested you
> > guys used git patch-id to track patches instead of commit ids. That's
> > how we track patches applied across many different trees internally.
> > Works much better than using the upstream sha.
>
> What is git patch id?  Is there any internal patch tag?

It's the output of "git patch-id", which is a hash of the patch, without
including its dependencies.  This allows to match identical commits in
different branches, and is what "git cherry" and "git rebase" use to detect
already applied patches.

Example (from v4.18.y):

$ git show -s e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
commit e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
Author: Robert Munteanu <rombert@apache.org>
Date:   Tue Jun 19 11:20:40 2018 +0300

    HID: redragon: fix num lock and caps lock LEDs

    commit dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e upstream.

    The redragon asura keyboard registers two input devices. The initial commit
    85455dd906d5 ("HID: redragon: Fix modifier keys for Redragon Asura
Keyboard")
    considered this an error and prevented one of the devices from registering.
    However, once this is done the num lock and caps lock leds no
longer toggle on
    and off, although the key functionality is not affected.

    This commit removes the code that prevents the input device
    registration and restores the num lock and caps lock LEDs.

    Fixes: 85455dd906d5 ("HID: redragon: Fix modifier keys for
Redragon Asura Keyboard")
    Signed-off-by: Robert Munteanu <rombert@apache.org>
    Signed-off-by: Jiri Kosina <jkosina@suse.cz>
    Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

$ git show e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 | git patch-id
9d4546a92b459de7249748c0a38b50c989f13bd1
e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
$ git show dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e | git patch-id
9d4546a92b459de7249748c0a38b50c989f13bd1
dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e

So both commits are identical, as they have the same patch ID.

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Takashi Iwai]

On Wed, 19 Sep 2018 08:16:17 +0200,
Geert Uytterhoeven wrote:
> 
> Hi Iwai-san,
> 
> On Tue, Sep 18, 2018 at 10:37 PM Takashi Iwai <tiwai@suse.de> wrote:
> > On Tue, 18 Sep 2018 15:43:08 +0200,
> > Martin K. Petersen wrote:
> > > Sorry I'm a little late to the game here. Been out on vacation.
> > >
> > > > We order patches in our trees in the same git-topological-ordering as they
> > > > are upstream. It has a lot of benefits, most importantly: it doesn't
> > > > introduce artificial conflicts that don't exist in reality.
> > > >
> > > > In order to achieve that, we of course need 1:1 mapping between our
> > > > patches and upstream commits.  Rebases destroy that mapping.
> > > >
> > > > And in some areas (scsi is one, but not the only one), we basically had no
> > > > other choice than considering maintainer's tree to be already "upstream
> > > > enough", without waiting for Linus' tree merge.
> > >
> > > When I discussed this with Johannes a little while ago, I suggested you
> > > guys used git patch-id to track patches instead of commit ids. That's
> > > how we track patches applied across many different trees internally.
> > > Works much better than using the upstream sha.
> >
> > What is git patch id?  Is there any internal patch tag?
> 
> It's the output of "git patch-id", which is a hash of the patch, without
> including its dependencies.  This allows to match identical commits in
> different branches, and is what "git cherry" and "git rebase" use to detect
> already applied patches.
> 
> Example (from v4.18.y):
> 
> $ git show -s e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
> commit e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
> Author: Robert Munteanu <rombert@apache.org>
> Date:   Tue Jun 19 11:20:40 2018 +0300
> 
>     HID: redragon: fix num lock and caps lock LEDs
> 
>     commit dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e upstream.
> 
>     The redragon asura keyboard registers two input devices. The initial commit
>     85455dd906d5 ("HID: redragon: Fix modifier keys for Redragon Asura
> Keyboard")
>     considered this an error and prevented one of the devices from registering.
>     However, once this is done the num lock and caps lock leds no
> longer toggle on
>     and off, although the key functionality is not affected.
> 
>     This commit removes the code that prevents the input device
>     registration and restores the num lock and caps lock LEDs.
> 
>     Fixes: 85455dd906d5 ("HID: redragon: Fix modifier keys for
> Redragon Asura Keyboard")
>     Signed-off-by: Robert Munteanu <rombert@apache.org>
>     Signed-off-by: Jiri Kosina <jkosina@suse.cz>
>     Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> 
> $ git show e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 | git patch-id
> 9d4546a92b459de7249748c0a38b50c989f13bd1
> e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
> $ git show dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e | git patch-id
> 9d4546a92b459de7249748c0a38b50c989f13bd1
> dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e
> 
> So both commits are identical, as they have the same patch ID.

Thanks for the explanation!

This would help for identifying the rebased patch, yes.  But it still
doesn't help for sorting as is.  You cannot get the topology
information from the patch id, so you don't know the patch order,
either.  (You can deduce the actual commit from the patch id, but it's
too heavy if you have 20,000 patches :)


Takashi

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Jan Kara]

On Wed 19-09-18 08:31:04, Takashi Iwai wrote:
> On Wed, 19 Sep 2018 08:16:17 +0200,
> Geert Uytterhoeven wrote:
> > 
> > Hi Iwai-san,
> > 
> > On Tue, Sep 18, 2018 at 10:37 PM Takashi Iwai <tiwai@suse.de> wrote:
> > > On Tue, 18 Sep 2018 15:43:08 +0200,
> > > Martin K. Petersen wrote:
> > > > Sorry I'm a little late to the game here. Been out on vacation.
> > > >
> > > > > We order patches in our trees in the same git-topological-ordering as they
> > > > > are upstream. It has a lot of benefits, most importantly: it doesn't
> > > > > introduce artificial conflicts that don't exist in reality.
> > > > >
> > > > > In order to achieve that, we of course need 1:1 mapping between our
> > > > > patches and upstream commits.  Rebases destroy that mapping.
> > > > >
> > > > > And in some areas (scsi is one, but not the only one), we basically had no
> > > > > other choice than considering maintainer's tree to be already "upstream
> > > > > enough", without waiting for Linus' tree merge.
> > > >
> > > > When I discussed this with Johannes a little while ago, I suggested you
> > > > guys used git patch-id to track patches instead of commit ids. That's
> > > > how we track patches applied across many different trees internally.
> > > > Works much better than using the upstream sha.
> > >
> > > What is git patch id?  Is there any internal patch tag?
> > 
> > It's the output of "git patch-id", which is a hash of the patch, without
> > including its dependencies.  This allows to match identical commits in
> > different branches, and is what "git cherry" and "git rebase" use to detect
> > already applied patches.
> > 
> > Example (from v4.18.y):
> > 
> > $ git show -s e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
> > commit e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
> > Author: Robert Munteanu <rombert@apache.org>
> > Date:   Tue Jun 19 11:20:40 2018 +0300
> > 
> >     HID: redragon: fix num lock and caps lock LEDs
> > 
> >     commit dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e upstream.
> > 
> >     The redragon asura keyboard registers two input devices. The initial commit
> >     85455dd906d5 ("HID: redragon: Fix modifier keys for Redragon Asura
> > Keyboard")
> >     considered this an error and prevented one of the devices from registering.
> >     However, once this is done the num lock and caps lock leds no
> > longer toggle on
> >     and off, although the key functionality is not affected.
> > 
> >     This commit removes the code that prevents the input device
> >     registration and restores the num lock and caps lock LEDs.
> > 
> >     Fixes: 85455dd906d5 ("HID: redragon: Fix modifier keys for
> > Redragon Asura Keyboard")
> >     Signed-off-by: Robert Munteanu <rombert@apache.org>
> >     Signed-off-by: Jiri Kosina <jkosina@suse.cz>
> >     Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> > 
> > $ git show e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 | git patch-id
> > 9d4546a92b459de7249748c0a38b50c989f13bd1
> > e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
> > $ git show dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e | git patch-id
> > 9d4546a92b459de7249748c0a38b50c989f13bd1
> > dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e
> > 
> > So both commits are identical, as they have the same patch ID.
> 
> Thanks for the explanation!
> 
> This would help for identifying the rebased patch, yes.  But it still
> doesn't help for sorting as is.  You cannot get the topology
> information from the patch id, so you don't know the patch order,
> either.  (You can deduce the actual commit from the patch id, but it's
> too heavy if you have 20,000 patches :)

Actually we could use the patch id to identify new commit id of a rebased
patch and infer proper ordering once we have the new commit id. So I think
using patch id (additionally to commit id) would really help us and solve
95% of rebases of e.g. SCSI tree by our automation without human
interaction.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Takashi Iwai]

On Wed, 19 Sep 2018 11:23:43 +0200,
Jan Kara wrote:
> 
> On Wed 19-09-18 08:31:04, Takashi Iwai wrote:
> > On Wed, 19 Sep 2018 08:16:17 +0200,
> > Geert Uytterhoeven wrote:
> > > 
> > > Hi Iwai-san,
> > > 
> > > On Tue, Sep 18, 2018 at 10:37 PM Takashi Iwai <tiwai@suse.de> wrote:
> > > > On Tue, 18 Sep 2018 15:43:08 +0200,
> > > > Martin K. Petersen wrote:
> > > > > Sorry I'm a little late to the game here. Been out on vacation.
> > > > >
> > > > > > We order patches in our trees in the same git-topological-ordering as they
> > > > > > are upstream. It has a lot of benefits, most importantly: it doesn't
> > > > > > introduce artificial conflicts that don't exist in reality.
> > > > > >
> > > > > > In order to achieve that, we of course need 1:1 mapping between our
> > > > > > patches and upstream commits.  Rebases destroy that mapping.
> > > > > >
> > > > > > And in some areas (scsi is one, but not the only one), we basically had no
> > > > > > other choice than considering maintainer's tree to be already "upstream
> > > > > > enough", without waiting for Linus' tree merge.
> > > > >
> > > > > When I discussed this with Johannes a little while ago, I suggested you
> > > > > guys used git patch-id to track patches instead of commit ids. That's
> > > > > how we track patches applied across many different trees internally.
> > > > > Works much better than using the upstream sha.
> > > >
> > > > What is git patch id?  Is there any internal patch tag?
> > > 
> > > It's the output of "git patch-id", which is a hash of the patch, without
> > > including its dependencies.  This allows to match identical commits in
> > > different branches, and is what "git cherry" and "git rebase" use to detect
> > > already applied patches.
> > > 
> > > Example (from v4.18.y):
> > > 
> > > $ git show -s e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
> > > commit e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
> > > Author: Robert Munteanu <rombert@apache.org>
> > > Date:   Tue Jun 19 11:20:40 2018 +0300
> > > 
> > >     HID: redragon: fix num lock and caps lock LEDs
> > > 
> > >     commit dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e upstream.
> > > 
> > >     The redragon asura keyboard registers two input devices. The initial commit
> > >     85455dd906d5 ("HID: redragon: Fix modifier keys for Redragon Asura
> > > Keyboard")
> > >     considered this an error and prevented one of the devices from registering.
> > >     However, once this is done the num lock and caps lock leds no
> > > longer toggle on
> > >     and off, although the key functionality is not affected.
> > > 
> > >     This commit removes the code that prevents the input device
> > >     registration and restores the num lock and caps lock LEDs.
> > > 
> > >     Fixes: 85455dd906d5 ("HID: redragon: Fix modifier keys for
> > > Redragon Asura Keyboard")
> > >     Signed-off-by: Robert Munteanu <rombert@apache.org>
> > >     Signed-off-by: Jiri Kosina <jkosina@suse.cz>
> > >     Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> > > 
> > > $ git show e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9 | git patch-id
> > > 9d4546a92b459de7249748c0a38b50c989f13bd1
> > > e496c9a4c17d4a97ed8d222f368f3b8ad8133fb9
> > > $ git show dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e | git patch-id
> > > 9d4546a92b459de7249748c0a38b50c989f13bd1
> > > dc9b8e85ed95cbe7e3ad0eabb5b48d617bbc365e
> > > 
> > > So both commits are identical, as they have the same patch ID.
> > 
> > Thanks for the explanation!
> > 
> > This would help for identifying the rebased patch, yes.  But it still
> > doesn't help for sorting as is.  You cannot get the topology
> > information from the patch id, so you don't know the patch order,
> > either.  (You can deduce the actual commit from the patch id, but it's
> > too heavy if you have 20,000 patches :)
> 
> Actually we could use the patch id to identify new commit id of a rebased
> patch and infer proper ordering once we have the new commit id. So I think
> using patch id (additionally to commit id) would really help us and solve
> 95% of rebases of e.g. SCSI tree by our automation without human
> interaction.

Yes, we can use the patch id in addition to the commit id as
complementary information for tracking the upstream.  I chatted with
people here now, and it seems feasible.

Thanks for the nice suggestion!


Takashi

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Takashi Iwai]

On Wed, 05 Sep 2018 12:13:52 +0200,
James Bottomley wrote:
> 
> The first suggestion is that kernel builds are pretty much automated
> and we try to make every commit buildable, so could we automate the
> machinery that allows a customer to do bisection simply by installing a
> kernel package? (we here, obviously means the distro, but going from
> git bisect to kernel package would be the useful link).

Oh yeah, that's a thing I've been dreaming for years! :)

A part of problems is the build power: it still takes an hour or
longer (depending on arch) to build a full distro kernel package, for
example, on OBS.  That's too painful for bisection.

So some semi-automatic way to reduce the config would be required for
usable bisection.  I guess automatic make localmodconfig, prep the
repo and kick off.  Also some garbage collections for stale repos,
etc.

> Second suggestion is that the bugzillas need to say much more strongly
> that the reporter really needs to confirm the fix in upstream and do
> the bisection themselves (and ideally request the backport to stable
> themselves).

OK, distros definitely need to try hard not to annoy upstream devs.

In the case of SUSE Kernel, we usually ask testing the latest
(more-or-less) vanilla kernel at first.  If it's an upstream problem,
then it's often tossed to the upstream.  If it's already addressed in
the upstream kernel, we take the responsibility for backports.  Asking
bisection by reporter is usually the last resort.

It'd be helpful if we get any suggestion to improve the process.


thanks,

Takashi

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Jiri Kosina]

On Wed, 5 Sep 2018, Takashi Iwai wrote:

> Asking bisection by reporter is usually the last resort.
> 
> It'd be helpful if we get any suggestion to improve the process.

IIRC, years ago someone started on "web-based bisection tool", but that 
was never finished.

The idea was that the user would just click "Good" and "Bad" buttons, and 
would be getting kernel RPMs to test.

If this would have been picked up and finished (GSoC perhaps?), it might 
help a lot with this aspect.

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Konstantin Ryabitsev]

On Wed, Sep 05, 2018 at 03:16:59PM +0200, Takashi Iwai wrote:
> > Second suggestion is that the bugzillas need to say much more strongly
> > that the reporter really needs to confirm the fix in upstream and do
> > the bisection themselves (and ideally request the backport to stable
> > themselves).
> 
> OK, distros definitely need to try hard not to annoy upstream devs.
> 
> In the case of SUSE Kernel, we usually ask testing the latest
> (more-or-less) vanilla kernel at first.  If it's an upstream problem,
> then it's often tossed to the upstream.  If it's already addressed in
> the upstream kernel, we take the responsibility for backports.  Asking
> bisection by reporter is usually the last resort.
> 
> It'd be helpful if we get any suggestion to improve the process.

It would be awesome to have a "bisect@home" type of thing with a similar
idea like seti@home and folding@home. Have a central queue where
developers can submit upstream commits and testcases, and a swarm of
volunteer drones would grab and bisect-build them until the
bug-introducing commit is identified and reported back.

I'll totally host the hell out of this.

-K

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Sasha Levin]

On Wed, Sep 05, 2018 at 09:39:16AM -0400, Konstantin Ryabitsev wrote:
>On Wed, Sep 05, 2018 at 03:16:59PM +0200, Takashi Iwai wrote:
>> > Second suggestion is that the bugzillas need to say much more strongly
>> > that the reporter really needs to confirm the fix in upstream and do
>> > the bisection themselves (and ideally request the backport to stable
>> > themselves).
>>
>> OK, distros definitely need to try hard not to annoy upstream devs.
>>
>> In the case of SUSE Kernel, we usually ask testing the latest
>> (more-or-less) vanilla kernel at first.  If it's an upstream problem,
>> then it's often tossed to the upstream.  If it's already addressed in
>> the upstream kernel, we take the responsibility for backports.  Asking
>> bisection by reporter is usually the last resort.
>>
>> It'd be helpful if we get any suggestion to improve the process.
>
>It would be awesome to have a "bisect@home" type of thing with a similar
>idea like seti@home and folding@home. Have a central queue where
>developers can submit upstream commits and testcases, and a swarm of
>volunteer drones would grab and bisect-build them until the
>bug-introducing commit is identified and reported back.
>
>I'll totally host the hell out of this.

I don't think KernelCI are too far on this front. They're already
working on auto bisection, so if there's an ability to add (simple)
testcases you'll be all set.

--
Thanks,
Sasha

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Laura Abbott]

On 09/05/2018 06:39 AM, Konstantin Ryabitsev wrote:
> On Wed, Sep 05, 2018 at 03:16:59PM +0200, Takashi Iwai wrote:
>>> Second suggestion is that the bugzillas need to say much more strongly
>>> that the reporter really needs to confirm the fix in upstream and do
>>> the bisection themselves (and ideally request the backport to stable
>>> themselves).
>>
>> OK, distros definitely need to try hard not to annoy upstream devs.
>>
>> In the case of SUSE Kernel, we usually ask testing the latest
>> (more-or-less) vanilla kernel at first.  If it's an upstream problem,
>> then it's often tossed to the upstream.  If it's already addressed in
>> the upstream kernel, we take the responsibility for backports.  Asking
>> bisection by reporter is usually the last resort.
>>
>> It'd be helpful if we get any suggestion to improve the process.
> 
> It would be awesome to have a "bisect@home" type of thing with a similar
> idea like seti@home and folding@home. Have a central queue where
> developers can submit upstream commits and testcases, and a swarm of
> volunteer drones would grab and bisect-build them until the
> bug-introducing commit is identified and reported back.
> 
> I'll totally host the hell out of this.
> 

I played around with bisect scripts for Fedora a while ago but mostly
abandoned them due to lack of interest and fragility. I was attempting
to use the full Fedora patch set and spec file but making that work
for arbitrary kernel commits was a pain.

Developers usually have no problem building and bisecting kernels,
it's non-kernel developers who often struggle with bisection.
One idea that I haven't followed up on was to extend the existing
targets for building distro packages to just build the source
side of things and then take advantage of existing environments
(e.g. COPR) to build the package binaries. I'd love a web interface
that would handle some of this automatically but, again, lack of
resources and knowledge of web frameworks.

Thanks,
Laura

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Konstantin Ryabitsev]

On Wed, Sep 05, 2018 at 09:44:23AM -0700, Laura Abbott wrote:
>>It would be awesome to have a "bisect@home" type of thing with a 
>>similar
>>idea like seti@home and folding@home. Have a central queue where
>>developers can submit upstream commits and testcases, and a swarm of
>>volunteer drones would grab and bisect-build them until the
>>bug-introducing commit is identified and reported back.
>>
>>I'll totally host the hell out of this.
>>
>Developers usually have no problem building and bisecting kernels,
>it's non-kernel developers who often struggle with bisection.
>One idea that I haven't followed up on was to extend the existing
>targets for building distro packages to just build the source
>side of things and then take advantage of existing environments
>(e.g. COPR) to build the package binaries. I'd love a web interface
>that would handle some of this automatically but, again, lack of
>resources and knowledge of web frameworks.

I'm excited that kernelci.org is coming on board as a full-fledged Linux 
Foundation project, since I'm hoping that their charter would include 
funding this kind of development. [1] I was already chatting with Kevin 
about some of the cool things we could do to make various 
CI/fuzzing/bug-reporting tools more streamlined, so I'll add "bisecting 
as a service" to my list of suggestions for the glorious kernel CI tool 
of the future. :)

-K

[1] I don't bring this up too often, but kernel.org's official 
organizational purpose is "to distribute the Linux kernel and other Open 
Source software to the public without charge" -- so drifting too far 
away from such activity could potentially jeopardize its legal standing 
as far as non-profit filing paperwork is concerned. This is why we can 
organize funding for things like public-inbox and lore.kernel.org, but 
not other things that wouldn't be overall related to "distributing the 
Linux kernel free of charge."

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Takashi Iwai]

On Wed, 05 Sep 2018 22:15:33 +0200,
Konstantin Ryabitsev wrote:
> 
> On Wed, Sep 05, 2018 at 09:44:23AM -0700, Laura Abbott wrote:
> >> It would be awesome to have a "bisect@home" type of thing with a
> >> similar
> >>idea like seti@home and folding@home. Have a central queue where
> >>developers can submit upstream commits and testcases, and a swarm of
> >>volunteer drones would grab and bisect-build them until the
> >>bug-introducing commit is identified and reported back.
> >>
> >>I'll totally host the hell out of this.
> >>
> >Developers usually have no problem building and bisecting kernels,
> >it's non-kernel developers who often struggle with bisection.
> >One idea that I haven't followed up on was to extend the existing
> >targets for building distro packages to just build the source
> >side of things and then take advantage of existing environments
> >(e.g. COPR) to build the package binaries. I'd love a web interface
> >that would handle some of this automatically but, again, lack of
> >resources and knowledge of web frameworks.
> 
> I'm excited that kernelci.org is coming on board as a full-fledged
> Linux Foundation project, since I'm hoping that their charter would
> include funding this kind of development. [1] I was already chatting
> with Kevin about some of the cool things we could do to make various
> CI/fuzzing/bug-reporting tools more streamlined, so I'll add
> "bisecting as a service" to my list of suggestions for the glorious
> kernel CI tool of the future. :)

That sounds promising, thanks!


Takashi

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Mauro Carvalho Chehab]

Em Wed, 5 Sep 2018 16:15:33 -0400
Konstantin Ryabitsev <konstantin@linuxfoundation.org> escreveu:

> I'm excited that kernelci.org is coming on board as a full-fledged Linux 
> Foundation project, since I'm hoping that their charter would include 
> funding this kind of development. [1] I was already chatting with Kevin 
> about some of the cool things we could do to make various 
> CI/fuzzing/bug-reporting tools more streamlined, so I'll add "bisecting 
> as a service" to my list of suggestions for the glorious kernel CI tool 
> of the future. :)

Kernel-ci seems interesting. Yet, 1/3 of the Kernel patches are drivers.
At least for media stuff, IMO, CI is too far away from our needs, as one
would need some real hardware to do some testing stuff (like for example, 
some webcams and some ways to check if the webcam support is actually
working). To make it worse, there are a wide variety of different hardware
with different features and their quirk needs. I suspect that similar
troubles would affect sound devices as well.

It could be worth to have some discussions with CI people with regards
to how it can better cope with driver's subsystem needs.

Thanks,
Mauro

Re: [Ksummit-discuss] [MAINTAINER SUMMIT] Distribution kernel bugzillas considered harmful

[Laura Abbott]

On 09/05/2018 03:13 AM, James Bottomley wrote:
> I'm seeing a lot of wasted effort by our customers on kernel bugs and
> trying to engage the distribution to fix them.  As a caveat, I'm
> working in the cloud, so the distributions in question are usually
> community ones not enterprise ones.  However, we do have a fair few
> customers on LTS kernels from Distributions.
> 
> Mostly they find a cloud performance regression, they try to engage the
> distro, spend ages working on it or submitting bugs and usually end up
> with an unsatisfactory result.  By the time they call my team in, we've
> likely only got a week to fix the issue.  However, step one is always
> confirming whether upstream works (95% of the time it does) and then
> finding the fix by bisection (usually assisted by knowledge of where
> the bug is).  To do the bisection we usually have to build a kernel
> package with our guesses and get them to try it, so it can be a bit
> slow.  Once we have the backport, we send it to stable and notify the
> distribution to include it in their next kernel release.
> 
> Here's the rub: community distributions (even LTS ones) don't have the
> resources even to triage cloud bugs in environments they likely can't
> reproduce, so we really need to develop assistive tools for customers
> to perform bisections to identify what caused the bug or (in the 95%
> case) what fixed it.  Having a bugzilla and using it as first line of
> support implies a service expectation (usually coming from Enterprise)
> that simply isn't met, so distributions need to fix this at the point
> of interaction: bugzilla.
> 
> The first suggestion is that kernel builds are pretty much automated
> and we try to make every commit buildable, so could we automate the
> machinery that allows a customer to do bisection simply by installing a
> kernel package? (we here, obviously means the distro, but going from
> git bisect to kernel package would be the useful link).
> 

As mentioned further down, I did try having scripts to do this but
it was ultimately fairly fragile. I think a better approach is
leveraging the existing in tree support for building a distro
package and using that with regular git bisect.

> Second suggestion is that the bugzillas need to say much more strongly
> that the reporter really needs to confirm the fix in upstream and do
> the bisection themselves (and ideally request the backport to stable
> themselves).
> 

At least in Fedora this is something we hit fairly frequently. We
do strongly encourage people to report bugs and bisect. This runs
into a number of problems:

- Bisecting on local machines is slow and people often don't want
to give up their machine resources.
- If people give me a test case I can reproduce, I'm usually
okay to run a bisect myself but it's pretty rare to get a
test case.
- People are hesitant to run bisections and build kernels.
There's a lot of steps involved. We try and point people to wiki
pages with instructions but many times we end up having to
go back and forth explaining how to do the setup.
- People are hesitant to report bugs to the upstream. I end up
having to explain where to report bugs or run get_maintainer.pl
for people otherwise they just file it against kernel.org
bugzilla or just e-mail LKML. I started a skeleton of a web
project to make a web interface for get_maintainer.pl but it
never got very far.

Tooling can certainly help with a lot of this but some of
this may also just be more documentation and needing to
guide people.

Thanks,
Laura