bug-introducing patches

bug-introducing patches

[Sasha Levin]

Working on AUTOSEL, it became even more obvious to me how difficult it is for a
patch to get a proper review. Maintainers found it difficult to keep up with
the upstream work for their subsystem, and reviewing additional -stable patches
put even more load on them which some suggested would be more than what they
can handle.

While AUTOSEL tries to understand if a patch fixes a bug, this was a bit late:
the bug was already introduced, folks already have to deal with it, and the
kernel is broken. I was wondering if I can do a similar process to AUTOSEL, but
teach the AI about bug-introducing patches.

When someone fixes a bug, he would describe the patch differently than he would
if he was writing a new feature. This lets AUTOSEL build on different commit
message constructs, among various inputs, to recognize bug fixes. However,
people are unaware that they introduce a bug, so the commit message for bug
introducing patches is essentially the same as for commits that don't introduce
a bug. This meant that I had to try and source data out of different sources.

Few of the parameters I ended up using are:
 - -next data (days spent in -next, changes in the patch between -next trees)
 - Mailing list data (was this patch ever sent to a ML? How long before it was
   merged? How many replies did it get? ...)
 - Author/commiter/maintainer chain data. Just like sports, some folks are more
   likely to produce better results than others. This goes beyond just "skill",
but also looks at things such as whether the author patches a subsystem he's
"familiar with" (== subsystem where most of his patches usually go), or is he
modifying a subsystem he never sent a patch for.
 - Patch complexity metrics - various code metrics to indicate how "complex" a
   patch is. Think 100 lines of whitespace fixes vs 100 lines that
significantly changes a subsystem.
 - Kernel process correctness - I tried using "violations" of the kernel
   process (patch formatting, correctness of the mailing to lkml, etc) as an
indicator of how familiar the author is with the kernel, with the presumption
that folks who are newer to kernel development are more likely to introduce
bugs

Running an initial iteration on a set of commits made two things very obvious
to me:

1. -rc releases suck. seriously suck. The quality of commits that went in -rc
cycles was much worse that merge window commit:
 - All commits had the same chance of introducing a bug whether they came in a
   merge window or an -rc cycle. This means that -rc commits mostly end up
replacing obvious bugs with less obvious ones.
 - While the average merge window commit changes, on average, 3x more lines
   than an -rc commit, the chances of a bug introduced per patch is the same,
which means that bugs-per-line metric of code is much higher with -rc patches.
 - A merge window commit spent 50% more days, on average, in -next than a -rc
   commit.
 - The number of -rc commits that never saw any mailing list or has never been
   replied to on a mailing list was **way** higher than merge window commits.
 - For some reason, the odds of a -rc commit to be targetted for -stable is
   over 20%, while for merge window commits it's about 3%. I can't quite
explain why that happens, but this would suggest that -rc commits end up
hurting -stable pretty badly.

2. Maintainers need to stop writing patches, commiting them, and pushing them
in without reviews.  In -rc cycles there is quite a large number of commits
that were either written by maintainers, commited, and merged upstream the same
day. These patches are very likely to introduce a new bug.


I don't really have a proposal beyond "tighten up -rc cycles", but I think it's
a discussion worth having. We have enough data to show what parts of kernel
development work, and what parts are just hurting us.

I'd be happy to gather more data if someone has an idea he wants to look
into. The data used for this work is based on:

 - v4.4..v4.16 (just becuase it's as far as linux-next-history goes).
 - "bugs" are commits that were mentioned in a Fixes: tag of a later
   commit.
 - "stable commits" are commits that made it to a -stable tree.

Re: bug-introducing patches

[Theodore Y. Ts'o]

On Tue, May 01, 2018 at 04:38:21PM +0000, Sasha Levin wrote:
>  - A merge window commit spent 50% more days, on average, in -next than a -rc
>    commit.

So it *used* to be the case that after the merge window, I would queue
up bug fixes for the next merge window.  Greg K-H pushed for me to
send them to Linus sooner, instead of waiting for the next merge
window.  TBH, it's actually easier for me to just wait until the next
merge window, but please understand that there are multiple pressures
on maintainers going on here, and the latest efforts to try to use
AUTOSEL is just the most recent pressure placed on maintainers.

The other thing is that when there is a regression users who are
testing linux-next want it fixed *fast*.  That's considered more
important to them than waiting for one, perfect patch, just to keep
AUTOSEL happy.

So please understand that when you say that maintainers *need* to do X
or Y, that there you are not the only one standing in line putting
pressures on maintainers saying they *need* to do something.  And
quite frankly, I consider keeping people who are nice enough to test
linux-next happy to be **far** more important than AUTOSEL.

Sorry.

						- Ted

Re: bug-introducing patches

[Sasha Levin]

On Tue, May 01, 2018 at 03:44:50PM -0400, Theodore Y. Ts'o wrote:
>On Tue, May 01, 2018 at 04:38:21PM +0000, Sasha Levin wrote:
>>  - A merge window commit spent 50% more days, on average, in -next than a -rc
>>    commit.
>
>So it *used* to be the case that after the merge window, I would queue
>up bug fixes for the next merge window.  Greg K-H pushed for me to
>send them to Linus sooner, instead of waiting for the next merge
>window.  TBH, it's actually easier for me to just wait until the next
>merge window, but please understand that there are multiple pressures
>on maintainers going on here, and the latest efforts to try to use
>AUTOSEL is just the most recent pressure placed on maintainers.
>
>The other thing is that when there is a regression users who are
>testing linux-next want it fixed *fast*.  That's considered more
>important to them than waiting for one, perfect patch, just to keep
>AUTOSEL happy.
>
>So please understand that when you say that maintainers *need* to do X
>or Y, that there you are not the only one standing in line putting
>pressures on maintainers saying they *need* to do something.  And
>quite frankly, I consider keeping people who are nice enough to test
>linux-next happy to be **far** more important than AUTOSEL.

Ted,

I'm not at all asking to wait before adding the patches to your tree,
or to -next. I'm asking to hold on to them a bit longer before you
push them to Linus because I can show that patches that don't spend
enough time in -next are more likely to introduce bugs.

Yes, linux-next users want it fixed *now* and I completely agree it
should be done that way, but the fix should not be immediately pushed to
Linus as well.

I've just finished reading an interesting article on LWN about the
PostgreSQL fsync issues (https://lwn.net/Articles/752952/). If you
look at Willy's commit, he wrote the final version of it about 5 days
ago, Jeff merged it in 3 days ago, and Linus merged it in the tree
today. Did it spend any time getting -next testing? nope.

What's worse is that that commit is tagged for stable, which means
that (given Greg's schedule) it may find it's way to -stable users
even before some -next users/bots had a chance to test it out.

This is less about AUTOSEL, and more about asking maintainers to
improve the testing commits get before they are sent to Linus.
Linus would rant about commits during merge window that didn't go
through -next, but for -rc commits this rule is somehow forgiven,
which is what I'm trying to change.

Re: bug-introducing patches

[Willy Tarreau]

On Tue, May 01, 2018 at 08:00:21PM +0000, Sasha Levin wrote:
> What's worse is that that commit is tagged for stable, which means
> that (given Greg's schedule) it may find it's way to -stable users
> even before some -next users/bots had a chance to test it out.

But it's a difficult trade-off. I think that -next is mostly used by
developers and that as such the audience remains limited. On the
opposite, -stable is used by many users. So how many days of -next
does it take to get the equivalent coverage of one day of -stable,
I don't know but it's probably a lot. Also server workloads are
almost exclusively on -stable. So a bug affecting only server users
will not benefit from -next exposition.

In the end it's all about responding to users' expectations to see
the bugs fixed. In -stable we regularly see users asking to backport
certain fixes. Sometimes they have to wait one or two extra versions
before they get their fix, and they are really not happy at all. If
the fix is rushed too fast and doesn't work, they won't be happy
either. Making them happy means backporting the right fix the quickest
possible. Too little test can result in a wrong fix, but too much test
results in a slow backport.

Again, I really don't find the -stable situation bad nowadays, quite
the opposite. I often suggest to people who don't follow too closely
to stick to latest LTS minus 1 or 2 releases. This way they don't get
the very latest fixes and have a chance that if something breaks very
badly, it gets fixed quickly. It works pretty well apparently.

I suspect that some of the issues that really need to be improved are
the fixes to recently merged code. That's never easy by definition
because if the code is young, it's not yet very well known even by
its author.

What *could* possibly be done (though I'm not fond of this) would be
to state a rule that past a certain number of stacked fixes for a
recently merged code, an extra review delay will be enforced on the
subsystem or on patches coming from the submitter. But I really doubt
it would significantly improve the situation.

Willy

Re: bug-introducing patches

[Sasha Levin]

On Tue, May 01, 2018 at 10:33:25PM +0200, Willy Tarreau wrote:
>On Tue, May 01, 2018 at 08:00:21PM +0000, Sasha Levin wrote:
>> What's worse is that that commit is tagged for stable, which means
>> that (given Greg's schedule) it may find it's way to -stable users
>> even before some -next users/bots had a chance to test it out.
>
>But it's a difficult trade-off. I think that -next is mostly used by
>developers and that as such the audience remains limited. On the
>opposite, -stable is used by many users. So how many days of -next
>does it take to get the equivalent coverage of one day of -stable,
>I don't know but it's probably a lot. Also server workloads are
>almost exclusively on -stable. So a bug affecting only server users
>will not benefit from -next exposition.
>
>In the end it's all about responding to users' expectations to see
>the bugs fixed. In -stable we regularly see users asking to backport
>certain fixes. Sometimes they have to wait one or two extra versions
>before they get their fix, and they are really not happy at all. If
>the fix is rushed too fast and doesn't work, they won't be happy
>either. Making them happy means backporting the right fix the quickest
>possible. Too little test can result in a wrong fix, but too much test
>results in a slow backport.
>
>Again, I really don't find the -stable situation bad nowadays, quite
>the opposite. I often suggest to people who don't follow too closely
>to stick to latest LTS minus 1 or 2 releases. This way they don't get
>the very latest fixes and have a chance that if something breaks very
>badly, it gets fixed quickly. It works pretty well apparently.
>
>I suspect that some of the issues that really need to be improved are
>the fixes to recently merged code. That's never easy by definition
>because if the code is young, it's not yet very well known even by
>its author.
>
>What *could* possibly be done (though I'm not fond of this) would be
>to state a rule that past a certain number of stacked fixes for a
>recently merged code, an extra review delay will be enforced on the
>subsystem or on patches coming from the submitter. But I really doubt
>it would significantly improve the situation.

I think that this discussion has shifted to -stable issues, which is not
what I was aiming for.

I tried to present a statistic from the recent kernel commits showing
that per changed line of code, an -rc commit has more than 3 times the
likelyhood to introduce a bug rather than a merge window one.

Is this something the community sees as an issue, or do we expect a
significantly higher odds of introducing bugs in -rc commits?

Feed free to ignore any proposals I've made. If you see this as an
issue, what could we do to address it?

Let's leave -stable out of this for now.

Re: [Ksummit-discuss] bug-introducing patches

[Theodore Y. Ts'o]

On Tue, May 01, 2018 at 08:00:21PM +0000, Sasha Levin wrote:
> 
> Yes, linux-next users want it fixed *now* and I completely agree it
> should be done that way, but the fix should not be immediately pushed to
> Linus as well.

I should have linux-head/linux-rc said testers, sorry.  The fact that
we have very few live users testing linux-next is a separate problem,
which I accidentally conflated.  But if a user who is testing -rc2
finds a problem, it is highly desirable to send a fix for -rc3,
instead of making that user wait to -rc4 or -rc5.  And *that* is more
important than AUTOSEL.

> I've just finished reading an interesting article on LWN about the
> PostgreSQL fsync issues (https://lwn.net/Articles/752952/). If you
> look at Willy's commit, he wrote the final version of it about 5 days
> ago, Jeff merged it in 3 days ago, and Linus merged it in the tree
> today. Did it spend any time getting -next testing? nope.

I agree that having the errseq patch go straight into Linus's tree is
certainly unfortunate.  The justification was this was a regression
fix, which I don't think it qualifies, since errseq_t went in some 9+
months ago.

It might be a good thing to quantify whether the patches you are
talking about are new features, bug fixes, or fixing a bug that was
introduced during the merge window or subsequently (e.g., a
regression).

> What's worse is that that commit is tagged for stable, which means
> that (given Greg's schedule) it may find it's way to -stable users
> even before some -next users/bots had a chance to test it out.

Well, it used to be that things tagged for stable most-merge window
are *supposed* to marinate for at least a week or before they would
get pulled into a stable release.  Part of the whole problem is that
people are wanting to be a lot more aggressive (both in time and
volume) in shovelling things into stable.

> This is less about AUTOSEL, and more about asking maintainers to
> improve the testing commits get before they are sent to Linus.
> Linus would rant about commits during merge window that didn't go
> through -next, but for -rc commits this rule is somehow forgiven,
> which is what I'm trying to change.

I do think it's about AUTOSEL, because when I'm dealing with a
regression, I want to get it fixed fast.  Because the alternative is
the merge-window commit getting reverted.  AUTOSEL seems wants perfect
patches that it can cherry pick once, as opposed to a case where if the
user confirms that it fixes the regression, I want to send it to Linus
quickly.  I do *not* want it to marinate in linux-next for 1-2 weeks.
I would much rather that *stable* hold off on picking up the patch for
1-2 weeks, but get it fixed in Linux HEAD sooner.  If that means that
the regression fix needs a further clean up, so be it.

Post -rc3 or -rc4, in my opinion bug fixes should wait until the next
merge window before they get merged at all.  (And certainly features
bugs should be Right Out.)  And sure, bug fixes should certainly get
more testing.  So I guess my main objection is your making a blanket
statement about all fixes, instead of breaking out regression fixes
versus bug fixes.  Since in my opinion they are very different animals...

       	   	   	       	       	    - Ted

_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Mark Brown]

[-- Attachment #1.1: Type: text/plain, Size: 1147 bytes --]

On Tue, May 01, 2018 at 04:54:48PM -0400, Theodore Y. Ts'o wrote:

> I do think it's about AUTOSEL, because when I'm dealing with a
> regression, I want to get it fixed fast.  Because the alternative is
> the merge-window commit getting reverted.  AUTOSEL seems wants perfect
> patches that it can cherry pick once, as opposed to a case where if the
> user confirms that it fixes the regression, I want to send it to Linus
> quickly.  I do *not* want it to marinate in linux-next for 1-2 weeks.
> I would much rather that *stable* hold off on picking up the patch for
> 1-2 weeks, but get it fixed in Linux HEAD sooner.  If that means that
> the regression fix needs a further clean up, so be it.

We've had issues with the automated testing systems in the past where a
maintainer has had a policy of letting things percoltate for a week
before sending to Linus and there's been a bug that caused a substantial
set of tests to fail to run (generally it's something that had unnoticed
dependencies in -next so wasn't caught there) - we essentially end up
not getting the affected bits of test coverage for that period of time
which is not helpful.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 186 bytes --]

_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Daniel Vetter]

On Tue, May 1, 2018 at 11:15 PM, Mark Brown <broonie@kernel.org> wrote:
> On Tue, May 01, 2018 at 04:54:48PM -0400, Theodore Y. Ts'o wrote:
>> I do think it's about AUTOSEL, because when I'm dealing with a
>> regression, I want to get it fixed fast.  Because the alternative is
>> the merge-window commit getting reverted.  AUTOSEL seems wants perfect
>> patches that it can cherry pick once, as opposed to a case where if the
>> user confirms that it fixes the regression, I want to send it to Linus
>> quickly.  I do *not* want it to marinate in linux-next for 1-2 weeks.
>> I would much rather that *stable* hold off on picking up the patch for
>> 1-2 weeks, but get it fixed in Linux HEAD sooner.  If that means that
>> the regression fix needs a further clean up, so be it.
>
> We've had issues with the automated testing systems in the past where a
> maintainer has had a policy of letting things percoltate for a week
> before sending to Linus and there's been a bug that caused a substantial
> set of tests to fail to run (generally it's something that had unnoticed
> dependencies in -next so wasn't caught there) - we essentially end up
> not getting the affected bits of test coverage for that period of time
> which is not helpful.

So much agreed. For our CI we carry a constantly rolling set of fixup
patches to keep it working, because regression fixes sometimes take
too long. And too long here for our needs is measured in days/hours -
developers start screaming pretty much immediately when our CI is down
:-)

Ofc I prefer if all subsystems ramp up pre-merge testing as much as
possible (and with xfstests and stuff like that I think filesystems
are leading here, if not consistently). But given the huge scope of
the kernel we'll never reach 100%, and oddball regressions will be
inevitable. Once a regression has crept through it imo really should
get fixed asap, with no unecessary soaking times - get a better
CI/kerneltests in place if you feel like you need to soak stuff.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin via Ksummit-discuss]

On Wed, May 02, 2018 at 10:11:14AM +0200, Daniel Vetter wrote:
>On Tue, May 1, 2018 at 11:15 PM, Mark Brown <broonie@kernel.org> wrote:
>> On Tue, May 01, 2018 at 04:54:48PM -0400, Theodore Y. Ts'o wrote:
>>> I do think it's about AUTOSEL, because when I'm dealing with a
>>> regression, I want to get it fixed fast.  Because the alternative is
>>> the merge-window commit getting reverted.  AUTOSEL seems wants perfect
>>> patches that it can cherry pick once, as opposed to a case where if the
>>> user confirms that it fixes the regression, I want to send it to Linus
>>> quickly.  I do *not* want it to marinate in linux-next for 1-2 weeks.
>>> I would much rather that *stable* hold off on picking up the patch for
>>> 1-2 weeks, but get it fixed in Linux HEAD sooner.  If that means that
>>> the regression fix needs a further clean up, so be it.
>>
>> We've had issues with the automated testing systems in the past where a
>> maintainer has had a policy of letting things percoltate for a week
>> before sending to Linus and there's been a bug that caused a substantial
>> set of tests to fail to run (generally it's something that had unnoticed
>> dependencies in -next so wasn't caught there) - we essentially end up
>> not getting the affected bits of test coverage for that period of time
>> which is not helpful.
>
>So much agreed. For our CI we carry a constantly rolling set of fixup
>patches to keep it working, because regression fixes sometimes take
>too long. And too long here for our needs is measured in days/hours -
>developers start screaming pretty much immediately when our CI is down
>:-)
>
>Ofc I prefer if all subsystems ramp up pre-merge testing as much as
>possible (and with xfstests and stuff like that I think filesystems
>are leading here, if not consistently). But given the huge scope of
>the kernel we'll never reach 100%, and oddball regressions will be
>inevitable. Once a regression has crept through it imo really should
>get fixed asap, with no unecessary soaking times - get a better
>CI/kerneltests in place if you feel like you need to soak stuff.

Oh I agree with what you're saying, if you have a good testing setup
this is (usually) much better than just throwing stuff in -next, so I
didn't mean to force soaking every fix in -next for a few weeks.

As you said, the regression should be fixed "asap", not "immediately".
It should go through some sort of review and testing the maintainers are
happy with, but unfourtenately it doesn't happen now.
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Mark Brown via Ksummit-discuss]

[-- Attachment #1.1: Type: text/plain, Size: 621 bytes --]

On Wed, May 02, 2018 at 07:46:34PM +0000, Sasha Levin wrote:

> As you said, the regression should be fixed "asap", not "immediately".
> It should go through some sort of review and testing the maintainers are
> happy with, but unfourtenately it doesn't happen now.

Doesn't happen some of the time.  It's not like this is a universal
problem.

Especially for driver specific things there's at times no realistic
prospect of getting useful independent review of fixes, the hardware
isn't always widely available and if the fix isn't a pure software thing
at some point you just have to trust the judgement of the vendor.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 186 bytes --]

_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Theodore Y. Ts'o]

On Thu, May 03, 2018 at 11:05:50AM +0900, Mark Brown wrote:
> On Wed, May 02, 2018 at 07:46:34PM +0000, Sasha Levin wrote:
> 
> > As you said, the regression should be fixed "asap", not "immediately".
> > It should go through some sort of review and testing the maintainers are
> > happy with, but unfourtenately it doesn't happen now.
> 
> Doesn't happen some of the time.  It's not like this is a universal
> problem.
> 
> Especially for driver specific things there's at times no realistic
> prospect of getting useful independent review of fixes, the hardware
> isn't always widely available and if the fix isn't a pure software thing
> at some point you just have to trust the judgement of the vendor.

And sometimes the Demon Murphy will cause a regression fix for user A,
to cause breakage for slightly different hardware belonging to user B.  :-(

   	 	      	       		 - Ted


_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Guenter Roeck]

On 05/02/2018 08:10 PM, Theodore Y. Ts'o wrote:
> On Thu, May 03, 2018 at 11:05:50AM +0900, Mark Brown wrote:
>> On Wed, May 02, 2018 at 07:46:34PM +0000, Sasha Levin wrote:
>>
>>> As you said, the regression should be fixed "asap", not "immediately".
>>> It should go through some sort of review and testing the maintainers are
>>> happy with, but unfourtenately it doesn't happen now.
>>
>> Doesn't happen some of the time.  It's not like this is a universal
>> problem.
>>
>> Especially for driver specific things there's at times no realistic
>> prospect of getting useful independent review of fixes, the hardware
>> isn't always widely available and if the fix isn't a pure software thing
>> at some point you just have to trust the judgement of the vendor.
> 
> And sometimes the Demon Murphy will cause a regression fix for user A,
> to cause breakage for slightly different hardware belonging to user B.  :-(
> 

Believe me, I get my share of those. 7dac4a1726a9 ("ext4: add validity checks
for bitmap block numbers") and its fix 22be37acce25 (" ext4: fix bitmap
position validation") are pretty good examples. Yet, at the same time I had
to deal with three additional CVEs in the ext4 code. Even though the initial
fix for one of the four was buggy, I am glad that I got the other three through
stable releases.

As for -next, me and others stopped reporting bugs in it, because when we do
we tend to get flamed for the "noise". Is anyone aware (or cares) that mips
and nds32 images don't build ? Soaking clothes in an empty bathtub won't make
them wet, and bugs in code which no one builds, much less tests or uses, won't
be found.

I can only repeat - what we need is more sophisticated testing, not a more
restrictive process.

Guenter
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Greg KH]

On Wed, May 02, 2018 at 08:52:29PM -0700, Guenter Roeck wrote:
> On 05/02/2018 08:10 PM, Theodore Y. Ts'o wrote:
> > On Thu, May 03, 2018 at 11:05:50AM +0900, Mark Brown wrote:
> > > On Wed, May 02, 2018 at 07:46:34PM +0000, Sasha Levin wrote:
> > > 
> > > > As you said, the regression should be fixed "asap", not "immediately".
> > > > It should go through some sort of review and testing the maintainers are
> > > > happy with, but unfourtenately it doesn't happen now.
> > > 
> > > Doesn't happen some of the time.  It's not like this is a universal
> > > problem.
> > > 
> > > Especially for driver specific things there's at times no realistic
> > > prospect of getting useful independent review of fixes, the hardware
> > > isn't always widely available and if the fix isn't a pure software thing
> > > at some point you just have to trust the judgement of the vendor.
> > 
> > And sometimes the Demon Murphy will cause a regression fix for user A,
> > to cause breakage for slightly different hardware belonging to user B.  :-(
> > 
> 
> Believe me, I get my share of those. 7dac4a1726a9 ("ext4: add validity checks
> for bitmap block numbers") and its fix 22be37acce25 (" ext4: fix bitmap
> position validation") are pretty good examples. Yet, at the same time I had
> to deal with three additional CVEs in the ext4 code. Even though the initial
> fix for one of the four was buggy, I am glad that I got the other three through
> stable releases.
> 
> As for -next, me and others stopped reporting bugs in it, because when we do
> we tend to get flamed for the "noise". Is anyone aware (or cares) that mips
> and nds32 images don't build ? Soaking clothes in an empty bathtub won't make
> them wet, and bugs in code which no one builds, much less tests or uses, won't
> be found.
> 
> I can only repeat - what we need is more sophisticated testing, not a more
> restrictive process.

I agree, and people are working on this.  But we can always use more!

thanks,

greg k-h
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 783 bytes --]

On Wed, May 02, 2018 at 08:52:29PM -0700, Guenter Roeck wrote:

> As for -next, me and others stopped reporting bugs in it, because when we do
> we tend to get flamed for the "noise". Is anyone aware (or cares) that mips
> and nds32 images don't build ? Soaking clothes in an empty bathtub won't make
> them wet, and bugs in code which no one builds, much less tests or uses, won't
> be found.

You've been flamed for testing -next?  That's not been my experience and
frankly it's pretty horrifying that it's happening.  Testing is pretty
much the whole point of -next existing in the first place so you have to
wonder why people are putting their trees there if they don't want
testing.  I have seen a few issues with people reporting bugs on old
versions of -next but otherwise...

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Ksummit-discuss] bug-introducing patches

[Tony Lindgren]

* Mark Brown <broonie@kernel.org> [180503 22:44]:
> On Wed, May 02, 2018 at 08:52:29PM -0700, Guenter Roeck wrote:
> 
> > As for -next, me and others stopped reporting bugs in it, because when we do
> > we tend to get flamed for the "noise". Is anyone aware (or cares) that mips
> > and nds32 images don't build ? Soaking clothes in an empty bathtub won't make
> > them wet, and bugs in code which no one builds, much less tests or uses, won't
> > be found.
> 
> You've been flamed for testing -next?  That's not been my experience and
> frankly it's pretty horrifying that it's happening.  Testing is pretty
> much the whole point of -next existing in the first place so you have to
> wonder why people are putting their trees there if they don't want
> testing.  I have seen a few issues with people reporting bugs on old
> versions of -next but otherwise...

Yes I agree testing Linux next is very important. That's the best way for
maintainers to ensure a usable -rc1 after a merge window. And then for
the -rc cycle, there not much of need for chasing bugs to get things working.

Bugs reported for Linux next often seem to get fixed or reverted faster
compared to the -rc cycle too. I think that's because people realize that
their code will not get merged until it's been fixed.

So some daily testing of Linux next can save a lot scrambling after the
merge window :)

Users don't usually upgrade kernels until after later -rc releases or only
after major releases so that probably explains some of the -rc cycle fixes.

Regards,

Tony

Re: [Ksummit-discuss] bug-introducing patches

[Ulf Hansson]

Tony, Sasha, Mark

On 4 May 2018 at 01:09, Tony Lindgren <tony@atomide.com> wrote:
> * Mark Brown <broonie@kernel.org> [180503 22:44]:
>> On Wed, May 02, 2018 at 08:52:29PM -0700, Guenter Roeck wrote:
>>
>> > As for -next, me and others stopped reporting bugs in it, because when we do
>> > we tend to get flamed for the "noise". Is anyone aware (or cares) that mips
>> > and nds32 images don't build ? Soaking clothes in an empty bathtub won't make
>> > them wet, and bugs in code which no one builds, much less tests or uses, won't
>> > be found.
>>
>> You've been flamed for testing -next?  That's not been my experience and
>> frankly it's pretty horrifying that it's happening.  Testing is pretty
>> much the whole point of -next existing in the first place so you have to
>> wonder why people are putting their trees there if they don't want
>> testing.  I have seen a few issues with people reporting bugs on old
>> versions of -next but otherwise...
>
> Yes I agree testing Linux next is very important. That's the best way for
> maintainers to ensure a usable -rc1 after a merge window. And then for
> the -rc cycle, there not much of need for chasing bugs to get things working.
>
> Bugs reported for Linux next often seem to get fixed or reverted faster
> compared to the -rc cycle too. I think that's because people realize that
> their code will not get merged until it's been fixed.
>
> So some daily testing of Linux next can save a lot scrambling after the
> merge window :)
>
> Users don't usually upgrade kernels until after later -rc releases or only
> after major releases so that probably explains some of the -rc cycle fixes.

I fully agree with above, linux-next works very nicely as a
pre-integration tree, however for *both* fixes and new material.

I guess the concern here is about getting more confidence about pure
fixes, before they hit Linus' tree or any other stable tree. Although
without having to introduce delays or additional work for maintainers
etc.

Then, why don't we have a pre-integration tree for fixes? That would
at least simply automated testing of fixes separately from new
material.

Perhaps this has already been discussed, and concluded and it's not
worth it, then apologize for my ignorance.

Kind regards
Uffe
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Mark Brown]

[-- Attachment #1.1: Type: text/plain, Size: 548 bytes --]

On Fri, May 04, 2018 at 04:21:26PM +0200, Ulf Hansson wrote:

> Then, why don't we have a pre-integration tree for fixes? That would
> at least simply automated testing of fixes separately from new
> material.

> Perhaps this has already been discussed, and concluded and it's not
> worth it, then apologize for my ignorance.

I think this is an excellent idea, copying in Stephen for his input.
I'm currently on holiday but unless someone convinces me it's a terrible
idea I'm willing to at least give it a go on a trial basis once I'm back
home.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 186 bytes --]

_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Daniel Vetter]

On Wed, May 9, 2018 at 10:44 AM, Mark Brown <broonie@kernel.org> wrote:
> On Fri, May 04, 2018 at 04:21:26PM +0200, Ulf Hansson wrote:
>
>> Then, why don't we have a pre-integration tree for fixes? That would
>> at least simply automated testing of fixes separately from new
>> material.
>
>> Perhaps this has already been discussed, and concluded and it's not
>> worth it, then apologize for my ignorance.
>
> I think this is an excellent idea, copying in Stephen for his input.
> I'm currently on holiday but unless someone convinces me it's a terrible
> idea I'm willing to at least give it a go on a trial basis once I'm back
> home.

Since Stephen merges all -fixes branches first, before merging all the
-next branches, he already generates that as part of linux-next. All
he'd need to do is push that intermediate state out to some
linux-fixes branch for consumption by test bots.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Geert Uytterhoeven]

On Wed, May 9, 2018 at 10:47 AM, Daniel Vetter <daniel.vetter@ffwll.ch> wrote:
> On Wed, May 9, 2018 at 10:44 AM, Mark Brown <broonie@kernel.org> wrote:
>> On Fri, May 04, 2018 at 04:21:26PM +0200, Ulf Hansson wrote:
>>> Then, why don't we have a pre-integration tree for fixes? That would
>>> at least simply automated testing of fixes separately from new
>>> material.
>>
>>> Perhaps this has already been discussed, and concluded and it's not
>>> worth it, then apologize for my ignorance.
>>
>> I think this is an excellent idea, copying in Stephen for his input.
>> I'm currently on holiday but unless someone convinces me it's a terrible
>> idea I'm willing to at least give it a go on a trial basis once I'm back
>> home.
>
> Since Stephen merges all -fixes branches first, before merging all the
> -next branches, he already generates that as part of linux-next. All
> he'd need to do is push that intermediate state out to some
> linux-fixes branch for consumption by test bots.

+1

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Mark Brown]

[-- Attachment #1.1: Type: text/plain, Size: 970 bytes --]

On Wed, May 09, 2018 at 10:47:57AM +0200, Daniel Vetter wrote:
> On Wed, May 9, 2018 at 10:44 AM, Mark Brown <broonie@kernel.org> wrote:

> > I think this is an excellent idea, copying in Stephen for his input.
> > I'm currently on holiday but unless someone convinces me it's a terrible
> > idea I'm willing to at least give it a go on a trial basis once I'm back
> > home.

> Since Stephen merges all -fixes branches first, before merging all the
> -next branches, he already generates that as part of linux-next. All
> he'd need to do is push that intermediate state out to some
> linux-fixes branch for consumption by test bots.

True.  It's currently only those -fixes branches that people have asked
him to merge separately which isn't as big a proportion of trees as have
them (perhaps fortunately given people's enthusiasm for fixes branches
that don't merge cleanly with their development branches) so we'd also
need to encourage people to add them separately.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 186 bytes --]

_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Stephen Rothwell]

[-- Attachment #1.1: Type: text/plain, Size: 1241 bytes --]

On Wed, 9 May 2018 18:03:46 +0900 Mark Brown <broonie@kernel.org> wrote:
>
> On Wed, May 09, 2018 at 10:47:57AM +0200, Daniel Vetter wrote:
> > On Wed, May 9, 2018 at 10:44 AM, Mark Brown <broonie@kernel.org> wrote:  
> 
> > > I think this is an excellent idea, copying in Stephen for his input.
> > > I'm currently on holiday but unless someone convinces me it's a terrible
> > > idea I'm willing to at least give it a go on a trial basis once I'm back
> > > home.  
> 
> > Since Stephen merges all -fixes branches first, before merging all the
> > -next branches, he already generates that as part of linux-next. All
> > he'd need to do is push that intermediate state out to some
> > linux-fixes branch for consumption by test bots.

Good idea ... I will see what I can do.
  
> True.  It's currently only those -fixes branches that people have asked
> him to merge separately which isn't as big a proportion of trees as have
> them (perhaps fortunately given people's enthusiasm for fixes branches
> that don't merge cleanly with their development branches) so we'd also
> need to encourage people to add them separately.

I currently have 44 such fixes branches.  More welcome!

-- 
Cheers,
Stephen Rothwell

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 186 bytes --]

_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Vinod Koul]

On 09-05-18, 20:47, Stephen Rothwell wrote:
> On Wed, 9 May 2018 18:03:46 +0900 Mark Brown <broonie@kernel.org> wrote:
> >
> > On Wed, May 09, 2018 at 10:47:57AM +0200, Daniel Vetter wrote:
> > > On Wed, May 9, 2018 at 10:44 AM, Mark Brown <broonie@kernel.org> wrote:  
> > 
> > > > I think this is an excellent idea, copying in Stephen for his input.
> > > > I'm currently on holiday but unless someone convinces me it's a terrible
> > > > idea I'm willing to at least give it a go on a trial basis once I'm back
> > > > home.  
> > 
> > > Since Stephen merges all -fixes branches first, before merging all the
> > > -next branches, he already generates that as part of linux-next. All
> > > he'd need to do is push that intermediate state out to some
> > > linux-fixes branch for consumption by test bots.
> 
> Good idea ... I will see what I can do.
>   
> > True.  It's currently only those -fixes branches that people have asked
> > him to merge separately which isn't as big a proportion of trees as have
> > them (perhaps fortunately given people's enthusiasm for fixes branches
> > that don't merge cleanly with their development branches) so we'd also
> > need to encourage people to add them separately.
> 
> I currently have 44 such fixes branches.  More welcome!

Great so do you want us to send fixes branch or scan the existing trees and add
them.

In case of former please do add slave-dma/fixes as well

~Vinod

Re: [Ksummit-discuss] bug-introducing patches

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 459 bytes --]

Hi Vinod,

On Wed, 9 May 2018 16:25:34 +0530 Vinod Koul <vinod.koul@linaro.org> wrote:
> > 
> > I currently have 44 such fixes branches.  More welcome!  
> 
> Great so do you want us to send fixes branch or scan the existing trees and add
> them.

The former.

> In case of former please do add slave-dma/fixes as well

Done.  Should I switch your contact address to your kernel.org one
(from your Intel one)?

-- 
Cheers,
Stephen Rothwell

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Ksummit-discuss] bug-introducing patches

[Vinod Koul]

On 09-05-18, 22:43, Stephen Rothwell wrote:
> Hi Vinod,
> 
> On Wed, 9 May 2018 16:25:34 +0530 Vinod Koul <vinod.koul@linaro.org> wrote:
> > > 
> > > I currently have 44 such fixes branches.  More welcome!  
> > 
> > Great so do you want us to send fixes branch or scan the existing trees and add
> > them.
> 
> The former.
> 
> > In case of former please do add slave-dma/fixes as well
> 
> Done.  Should I switch your contact address to your kernel.org one
> (from your Intel one)?

Yes please, that is no longer valid.

-- 
~Vinod

Re: [Ksummit-discuss] bug-introducing patches

[Krzysztof Kozlowski]

On Wed, May 9, 2018 at 2:43 PM, Stephen Rothwell <sfr@canb.auug.org.au> wrote:
> Hi Vinod,
>
> On Wed, 9 May 2018 16:25:34 +0530 Vinod Koul <vinod.koul@linaro.org> wrote:
>> >
>> > I currently have 44 such fixes branches.  More welcome!
>>
>> Great so do you want us to send fixes branch or scan the existing trees and add
>> them.
>
> The former.

Please merge following fixes branches from my trees:
Tree: samsung-krzk
git://git.kernel.org/pub/scm/linux/kernel/git/krzk/linux.git
branch: fixes

Tree: pinctrl-samsung
git://git.kernel.org/pub/scm/linux/kernel/git/pinctrl/samsung.git
branch: pinctrl-fixes

Although both are currently empty... but I guess you are collecting
this also for future.

Best regards,
Krzysztof

Re: [Ksummit-discuss] bug-introducing patches

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 1388 bytes --]

Hi Krzysztof,

On Tue, 15 May 2018 12:42:49 +0200 Krzysztof Kozlowski <krzk@kernel.org> wrote:
>
> Please merge following fixes branches from my trees:
> Tree: samsung-krzk
> git://git.kernel.org/pub/scm/linux/kernel/git/krzk/linux.git
> branch: fixes
> 
> Tree: pinctrl-samsung
> git://git.kernel.org/pub/scm/linux/kernel/git/pinctrl/samsung.git
> branch: pinctrl-fixes
> 
> Although both are currently empty... but I guess you are collecting
> this also for future.

Yes, indeed.

Added from tomorrow.

Thanks for adding your subsystem tree as a participant of linux-next.  As
you may know, this is not a judgement of your code.  The purpose of
linux-next is for integration testing and to lower the impact of
conflicts between subsystems in the next merge window. 

You will need to ensure that the patches/commits in your tree/series have
been:
     * submitted under GPL v2 (or later) and include the Contributor's
        Signed-off-by,
     * posted to the relevant mailing list,
     * reviewed by you (or another maintainer of your subsystem tree),
     * successfully unit tested, and 
     * destined for the current or next Linux merge window.

Basically, this should be just what you would send to Linus (or ask him
to fetch).  It is allowed to be rebased if you deem it necessary.

-- 
Cheers,
Stephen Rothwell 
sfr@canb.auug.org.au

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Ksummit-discuss] bug-introducing patches

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 807 bytes --]

On Wed, May 09, 2018 at 08:47:27PM +1000, Stephen Rothwell wrote:
> On Wed, 9 May 2018 18:03:46 +0900 Mark Brown <broonie@kernel.org> wrote:
> > On Wed, May 09, 2018 at 10:47:57AM +0200, Daniel Vetter wrote:
> > > On Wed, May 9, 2018 at 10:44 AM, Mark Brown <broonie@kernel.org> wrote:  

> > True.  It's currently only those -fixes branches that people have asked
> > him to merge separately which isn't as big a proportion of trees as have
> > them (perhaps fortunately given people's enthusiasm for fixes branches
> > that don't merge cleanly with their development branches) so we'd also
> > need to encourage people to add them separately.

> I currently have 44 such fixes branches.  More welcome!

Well, all my trees have a for-linus branch to go with the for-next
branch for a start.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Ksummit-discuss] bug-introducing patches

[Stephen Rothwell]

[-- Attachment #1.1: Type: text/plain, Size: 1169 bytes --]

Hi Mark,

On Wed, 9 May 2018 23:05:32 +0900 Mark Brown <broonie@kernel.org> wrote:
>
> Well, all my trees have a for-linus branch to go with the for-next
> branch for a start.

The regmap and regulator trees have no for-linus branch (currently).
Added sound-asoc-fixes and spi-fixes from today.

Thanks for adding your subsystem tree as a participant of linux-next.  As
you may know, this is not a judgement of your code.  The purpose of
linux-next is for integration testing and to lower the impact of
conflicts between subsystems in the next merge window. 

You will need to ensure that the patches/commits in your tree/series have
been:
     * submitted under GPL v2 (or later) and include the Contributor's
        Signed-off-by,
     * posted to the relevant mailing list,
     * reviewed by you (or another maintainer of your subsystem tree),
     * successfully unit tested, and 
     * destined for the current or next Linux merge window.

Basically, this should be just what you would send to Linus (or ask him
to fetch).  It is allowed to be rebased if you deem it necessary.

-- 
Cheers,
Stephen Rothwell 
sfr@canb.auug.org.au

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 186 bytes --]

_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 494 bytes --]

On Thu, May 10, 2018 at 08:09:09AM +1000, Stephen Rothwell wrote:
> On Wed, 9 May 2018 23:05:32 +0900 Mark Brown <broonie@kernel.org> wrote:

> > Well, all my trees have a for-linus branch to go with the for-next
> > branch for a start.

> The regmap and regulator trees have no for-linus branch (currently).
> Added sound-asoc-fixes and spi-fixes from today.

That's not what git claims when I try to push...  no idea what's going
on there, I just deleted and repushed - hopefully that helps.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Ksummit-discuss] bug-introducing patches

[Stephen Rothwell]

[-- Attachment #1.1: Type: text/plain, Size: 711 bytes --]

Hi Mark,

On Thu, 10 May 2018 22:36:28 +0900 Mark Brown <broonie@kernel.org> wrote:
>
> On Thu, May 10, 2018 at 08:09:09AM +1000, Stephen Rothwell wrote:
> > On Wed, 9 May 2018 23:05:32 +0900 Mark Brown <broonie@kernel.org> wrote:  
> 
> > > Well, all my trees have a for-linus branch to go with the for-next
> > > branch for a start.  
> 
> > The regmap and regulator trees have no for-linus branch (currently).
> > Added sound-asoc-fixes and spi-fixes from today.  
> 
> That's not what git claims when I try to push...  no idea what's going
> on there, I just deleted and repushed - hopefully that helps.

I have added regmap-fixes and regulator-fixes now.

-- 
Cheers,
Stephen Rothwell

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 186 bytes --]

_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Guenter Roeck]

On Wed, May 09, 2018 at 08:47:27PM +1000, Stephen Rothwell wrote:
> On Wed, 9 May 2018 18:03:46 +0900 Mark Brown <broonie@kernel.org> wrote:
> >
> > On Wed, May 09, 2018 at 10:47:57AM +0200, Daniel Vetter wrote:
> > > On Wed, May 9, 2018 at 10:44 AM, Mark Brown <broonie@kernel.org> wrote:  
> > 
> > > > I think this is an excellent idea, copying in Stephen for his input.
> > > > I'm currently on holiday but unless someone convinces me it's a terrible
> > > > idea I'm willing to at least give it a go on a trial basis once I'm back
> > > > home.  
> > 
> > > Since Stephen merges all -fixes branches first, before merging all the
> > > -next branches, he already generates that as part of linux-next. All
> > > he'd need to do is push that intermediate state out to some
> > > linux-fixes branch for consumption by test bots.
> 
> Good idea ... I will see what I can do.
>   
> > True.  It's currently only those -fixes branches that people have asked
> > him to merge separately which isn't as big a proportion of trees as have
> > them (perhaps fortunately given people's enthusiasm for fixes branches
> > that don't merge cleanly with their development branches) so we'd also
> > need to encourage people to add them separately.
> 
> I currently have 44 such fixes branches.  More welcome!
> 
Please add

git://git.kernel.org/pub/scm/linux/kernel/git/groeck/linux-staging.git hwmon

as fixes branch.

Thanks,
Guenter

> -- 
> Cheers,
> Stephen Rothwell



> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Stephen Rothwell]

[-- Attachment #1.1: Type: text/plain, Size: 1103 bytes --]

Hi Guenter,

On Wed, 9 May 2018 08:57:33 -0700 Guenter Roeck <linux@roeck-us.net> wrote:
>
> Please add
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/groeck/linux-staging.git hwmon
> 
> as fixes branch.

Added from today.

Thanks for adding your subsystem tree as a participant of linux-next.  As
you may know, this is not a judgement of your code.  The purpose of
linux-next is for integration testing and to lower the impact of
conflicts between subsystems in the next merge window. 

You will need to ensure that the patches/commits in your tree/series have
been:
     * submitted under GPL v2 (or later) and include the Contributor's
        Signed-off-by,
     * posted to the relevant mailing list,
     * reviewed by you (or another maintainer of your subsystem tree),
     * successfully unit tested, and 
     * destined for the current or next Linux merge window.

Basically, this should be just what you would send to Linus (or ask him
to fetch).  It is allowed to be rebased if you deem it necessary.

-- 
Cheers,
Stephen Rothwell 
sfr@canb.auug.org.au

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 186 bytes --]

_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Dan Williams]

On Wed, May 9, 2018 at 3:47 AM, Stephen Rothwell <sfr@canb.auug.org.au> wrote:
> On Wed, 9 May 2018 18:03:46 +0900 Mark Brown <broonie@kernel.org> wrote:
>>
>> On Wed, May 09, 2018 at 10:47:57AM +0200, Daniel Vetter wrote:
>> > On Wed, May 9, 2018 at 10:44 AM, Mark Brown <broonie@kernel.org> wrote:
>>
>> > > I think this is an excellent idea, copying in Stephen for his input.
>> > > I'm currently on holiday but unless someone convinces me it's a terrible
>> > > idea I'm willing to at least give it a go on a trial basis once I'm back
>> > > home.
>>
>> > Since Stephen merges all -fixes branches first, before merging all the
>> > -next branches, he already generates that as part of linux-next. All
>> > he'd need to do is push that intermediate state out to some
>> > linux-fixes branch for consumption by test bots.
>
> Good idea ... I will see what I can do.
>
>> True.  It's currently only those -fixes branches that people have asked
>> him to merge separately which isn't as big a proportion of trees as have
>> them (perhaps fortunately given people's enthusiasm for fixes branches
>> that don't merge cleanly with their development branches) so we'd also
>> need to encourage people to add them separately.
>
> I currently have 44 such fixes branches.  More welcome!

Please add:

    git://git.kernel.org/pub/scm/linux/kernel/git/nvdimm/nvdimm.git
libnvdimm-fixes

We currently merge this into libnvdimm-for-next for -next coverage,
and resolve any conflicts vs new development. Do you want to see those
conflicts? Otherwise I would recommend only pulling libnvdimm-for-next
for -next and libnvdimm-fixes for this new -next-fixes effort.

Re: [Ksummit-discuss] bug-introducing patches

[Stephen Rothwell]

[-- Attachment #1.1: Type: text/plain, Size: 1629 bytes --]

Hi Dan,

On Wed, 9 May 2018 09:04:31 -0700 Dan Williams <dan.j.williams@intel.com> wrote:
>
> Please add:
> 
>     git://git.kernel.org/pub/scm/linux/kernel/git/nvdimm/nvdimm.git
> libnvdimm-fixes
> 
> We currently merge this into libnvdimm-for-next for -next coverage,
> and resolve any conflicts vs new development. Do you want to see those
> conflicts? Otherwise I would recommend only pulling libnvdimm-for-next
> for -next and libnvdimm-fixes for this new -next-fixes effort.

The conflicts are usually fine (but if you do the merges, I won't see
them - which is even better :-)).  I like to have the fixes branches in
linux-next so that noone has to worry about probelsm in Linus' tree
that have pending fixes already.

Added from today.

Thanks for adding your subsystem tree as a participant of linux-next.  As
you may know, this is not a judgement of your code.  The purpose of
linux-next is for integration testing and to lower the impact of
conflicts between subsystems in the next merge window. 

You will need to ensure that the patches/commits in your tree/series have
been:
     * submitted under GPL v2 (or later) and include the Contributor's
        Signed-off-by,
     * posted to the relevant mailing list,
     * reviewed by you (or another maintainer of your subsystem tree),
     * successfully unit tested, and 
     * destined for the current or next Linux merge window.

Basically, this should be just what you would send to Linus (or ask him
to fetch).  It is allowed to be rebased if you deem it necessary.

-- 
Cheers,
Stephen Rothwell 
sfr@canb.auug.org.au

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 186 bytes --]

_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Boris Brezillon]

Hi Stephen,

On Wed, 9 May 2018 20:47:27 +1000
Stephen Rothwell <sfr@canb.auug.org.au> wrote:

> On Wed, 9 May 2018 18:03:46 +0900 Mark Brown <broonie@kernel.org> wrote:
> >
> > On Wed, May 09, 2018 at 10:47:57AM +0200, Daniel Vetter wrote:  
> > > On Wed, May 9, 2018 at 10:44 AM, Mark Brown <broonie@kernel.org> wrote:    
> >   
> > > > I think this is an excellent idea, copying in Stephen for his input.
> > > > I'm currently on holiday but unless someone convinces me it's a terrible
> > > > idea I'm willing to at least give it a go on a trial basis once I'm back
> > > > home.    
> >   
> > > Since Stephen merges all -fixes branches first, before merging all the
> > > -next branches, he already generates that as part of linux-next. All
> > > he'd need to do is push that intermediate state out to some
> > > linux-fixes branch for consumption by test bots.  
> 
> Good idea ... I will see what I can do.
>   
> > True.  It's currently only those -fixes branches that people have asked
> > him to merge separately which isn't as big a proportion of trees as have
> > them (perhaps fortunately given people's enthusiasm for fixes branches
> > that don't merge cleanly with their development branches) so we'd also
> > need to encourage people to add them separately.  
> 
> I currently have 44 such fixes branches.  More welcome!

I see that the nand/fixes and spi-nor/fixes branch are already there [1].
You can add:

mtd-fixes	git	git://git.infradead.org/linux-mtd.git#master

You can also remove the mtd entry [2], since mtd-2.6.git is just a sym
link to linux-mtd.git, so it will just be a duplicate of the mtd-fixes
entry. You can also rename the l2-mtd entry [3] into mtd.

Regards,

Boris

[1]https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next/+/43cd1f4979998ba0ef1c0b8e1c5d23d2de5ab172/Next/Trees#41
[2]https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next/+/43cd1f4979998ba0ef1c0b8e1c5d23d2de5ab172/Next/Trees#155
[3]https://kernel.googlesource.com/pub/scm/linux/kernel/git/next/linux-next/+/43cd1f4979998ba0ef1c0b8e1c5d23d2de5ab172/Next/Trees#156

Re: [Ksummit-discuss] bug-introducing patches

[Stephen Rothwell]

[-- Attachment #1.1: Type: text/plain, Size: 1373 bytes --]

Hi Boris,

On Wed, 9 May 2018 21:35:28 +0200 Boris Brezillon <boris.brezillon@bootlin.com> wrote:
>
> I see that the nand/fixes and spi-nor/fixes branch are already there [1].
> You can add:
> 
> mtd-fixes	git	git://git.infradead.org/linux-mtd.git#master

Added from today.

> You can also remove the mtd entry [2], since mtd-2.6.git is just a sym
> link to linux-mtd.git, so it will just be a duplicate of the mtd-fixes
> entry. You can also rename the l2-mtd entry [3] into mtd.

All done.

Thanks for adding your subsystem tree as a participant of linux-next.  As
you may know, this is not a judgement of your code.  The purpose of
linux-next is for integration testing and to lower the impact of
conflicts between subsystems in the next merge window. 

You will need to ensure that the patches/commits in your tree/series have
been:
     * submitted under GPL v2 (or later) and include the Contributor's
        Signed-off-by,
     * posted to the relevant mailing list,
     * reviewed by you (or another maintainer of your subsystem tree),
     * successfully unit tested, and 
     * destined for the current or next Linux merge window.

Basically, this should be just what you would send to Linus (or ask him
to fetch).  It is allowed to be rebased if you deem it necessary.

-- 
Cheers,
Stephen Rothwell 
sfr@canb.auug.org.au

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 186 bytes --]

_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin via Ksummit-discuss]

On Wed, May 09, 2018 at 08:47:27PM +1000, Stephen Rothwell wrote:
>On Wed, 9 May 2018 18:03:46 +0900 Mark Brown <broonie@kernel.org> wrote:
>>
>> On Wed, May 09, 2018 at 10:47:57AM +0200, Daniel Vetter wrote:
>> > On Wed, May 9, 2018 at 10:44 AM, Mark Brown <broonie@kernel.org> wrote:
>>
>> > > I think this is an excellent idea, copying in Stephen for his input.
>> > > I'm currently on holiday but unless someone convinces me it's a terrible
>> > > idea I'm willing to at least give it a go on a trial basis once I'm back
>> > > home.
>>
>> > Since Stephen merges all -fixes branches first, before merging all the
>> > -next branches, he already generates that as part of linux-next. All
>> > he'd need to do is push that intermediate state out to some
>> > linux-fixes branch for consumption by test bots.
>
>Good idea ... I will see what I can do.

This is very interesting! I'm curious how the statistics will look when
we'll compare patches that didn't go through this tree, patches that
spent minimal time in this tree, and patches that has spent some time
in the tree before being merged in Linus's tree.

Would this be something we would want to point actual users to, rather
than just bots? If every commit in next-fixes is essentially queued up
for Linus at some point, users might as well test out next-fixes instead
of -rc.

>> True.  It's currently only those -fixes branches that people have asked
>> him to merge separately which isn't as big a proportion of trees as have
>> them (perhaps fortunately given people's enthusiasm for fixes branches
>> that don't merge cleanly with their development branches) so we'd also
>> need to encourage people to add them separately.
>
>I currently have 44 such fixes branches.  More welcome!

I've tried looking at git for bigger subsystems, and a few branches that
would fit this description, and are not in -next are:

git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git perf/urgent
git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git sched/urgent
git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86/urgent
git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git timers/urgent
git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git irq/urgent
git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git core/urgent
git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git efi/urgent
git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git locking/urgent
git.kernel.dk/linux-block.git for-linus
git.kernel.org/pub/scm/linux/kernel/git/s390/linux.git fixes
git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse.git for-linux
git.kernel.org/pub/scm/linux/kernel/git/iwlwifi/iwlwifi-fixes.git master
git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi.git fixes
git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git queue-rc
git.kernel.org/pub/scm/linux/kernel/git/jikos/hid.git for-linus
git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git ftrace/urgent

Would it make sense to ping the respective maintainers to ack the
inclusion of these branches?

There are a few other trees where the fixes branch has a name that
depends on the release, we can ask them to also create a simple fixes
branch so that next-fixes could merge it in?

git.kernel.org/pub/scm/fs/xfs/xfs-linux.git xfs-4.17-fixes
git.kernel.org/pub/scm/linux/kernel/git/rw/uml.git for-linux-4.17-rc1
git.kernel.org/pub/scm/linux/kernel/git/khilman/linux-amlogic.git v4.17/fixes
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Tony Lindgren]

[-- Attachment #1: Type: text/plain, Size: 239 bytes --]

* Stephen Rothwell <sfr@canb.auug.org.au> [180509 10:49]:
> I currently have 44 such fixes branches.  More welcome!

Can you please also add mine:

git://git.kernel.org/pub/scm/linux/kernel/git/tmlind/linux-omap.git fixes

Thanks,

Tony



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Ksummit-discuss] bug-introducing patches

[Stephen Rothwell]

[-- Attachment #1.1: Type: text/plain, Size: 1219 bytes --]

Hi Tony,

On Thu, 10 May 2018 08:57:55 -0700 Tony Lindgren <tony@atomide.com> wrote:
>
> * Stephen Rothwell <sfr@canb.auug.org.au> [180509 10:49]:
> > I currently have 44 such fixes branches.  More welcome!  
> 
> Can you please also add mine:
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/tmlind/linux-omap.git fixes

Added from today.

Thanks for adding your subsystem tree as a participant of linux-next.  As
you may know, this is not a judgement of your code.  The purpose of
linux-next is for integration testing and to lower the impact of
conflicts between subsystems in the next merge window. 

You will need to ensure that the patches/commits in your tree/series have
been:
     * submitted under GPL v2 (or later) and include the Contributor's
        Signed-off-by,
     * posted to the relevant mailing list,
     * reviewed by you (or another maintainer of your subsystem tree),
     * successfully unit tested, and 
     * destined for the current or next Linux merge window.

Basically, this should be just what you would send to Linus (or ask him
to fetch).  It is allowed to be rebased if you deem it necessary.

-- 
Cheers,
Stephen Rothwell 
sfr@canb.auug.org.au

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 186 bytes --]

_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[David Sterba]

On Wed, May 09, 2018 at 08:47:27PM +1000, Stephen Rothwell wrote:
> > True.  It's currently only those -fixes branches that people have asked
> > him to merge separately which isn't as big a proportion of trees as have
> > them (perhaps fortunately given people's enthusiasm for fixes branches
> > that don't merge cleanly with their development branches) so we'd also
> > need to encourage people to add them separately.
> 
> I currently have 44 such fixes branches.  More welcome!

Please add

  git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git next-fixes

Thanks.

Re: [Ksummit-discuss] bug-introducing patches

[Stephen Rothwell]

[-- Attachment #1.1: Type: text/plain, Size: 1090 bytes --]

Hi David,

On Fri, 11 May 2018 10:47:01 +0200 David Sterba <dsterba@suse.cz> wrote:
>
> Please add
> 
>   git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git next-fixes

Added from Monday (as btrfs-fixes).

Thanks for adding your subsystem tree as a participant of linux-next.  As
you may know, this is not a judgement of your code.  The purpose of
linux-next is for integration testing and to lower the impact of
conflicts between subsystems in the next merge window. 

You will need to ensure that the patches/commits in your tree/series have
been:
     * submitted under GPL v2 (or later) and include the Contributor's
        Signed-off-by,
     * posted to the relevant mailing list,
     * reviewed by you (or another maintainer of your subsystem tree),
     * successfully unit tested, and 
     * destined for the current or next Linux merge window.

Basically, this should be just what you would send to Linus (or ask him
to fetch).  It is allowed to be rebased if you deem it necessary.

-- 
Cheers,
Stephen Rothwell 
sfr@canb.auug.org.au

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 186 bytes --]

_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 1133 bytes --]

Hi all,

On Wed, 9 May 2018 20:47:27 +1000 Stephen Rothwell <sfr@canb.auug.org.au> wrote:
>
> On Wed, 9 May 2018 18:03:46 +0900 Mark Brown <broonie@kernel.org> wrote:
> >
> > On Wed, May 09, 2018 at 10:47:57AM +0200, Daniel Vetter wrote:  
> > > On Wed, May 9, 2018 at 10:44 AM, Mark Brown <broonie@kernel.org> wrote:    
> >   
> > > > I think this is an excellent idea, copying in Stephen for his input.
> > > > I'm currently on holiday but unless someone convinces me it's a terrible
> > > > idea I'm willing to at least give it a go on a trial basis once I'm back
> > > > home.    
> >   
> > > Since Stephen merges all -fixes branches first, before merging all the
> > > -next branches, he already generates that as part of linux-next. All
> > > he'd need to do is push that intermediate state out to some
> > > linux-fixes branch for consumption by test bots.  
> 
> Good idea ... I will see what I can do.

See my announcement of a pending-fixes branch in linux-next (on LKML
and others)

> I currently have 44 such fixes branches.  More welcome!

We are up to 55.

-- 
Cheers,
Stephen Rothwell

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Ksummit-discuss] bug-introducing patches

[Guenter Roeck]

On 05/11/2018 09:38 PM, Stephen Rothwell wrote:
> Hi all,
> 
> On Wed, 9 May 2018 20:47:27 +1000 Stephen Rothwell <sfr@canb.auug.org.au> wrote:
>>
>> On Wed, 9 May 2018 18:03:46 +0900 Mark Brown <broonie@kernel.org> wrote:
>>>
>>> On Wed, May 09, 2018 at 10:47:57AM +0200, Daniel Vetter wrote:
>>>> On Wed, May 9, 2018 at 10:44 AM, Mark Brown <broonie@kernel.org> wrote:
>>>    
>>>>> I think this is an excellent idea, copying in Stephen for his input.
>>>>> I'm currently on holiday but unless someone convinces me it's a terrible
>>>>> idea I'm willing to at least give it a go on a trial basis once I'm back
>>>>> home.
>>>    
>>>> Since Stephen merges all -fixes branches first, before merging all the
>>>> -next branches, he already generates that as part of linux-next. All
>>>> he'd need to do is push that intermediate state out to some
>>>> linux-fixes branch for consumption by test bots.
>>
>> Good idea ... I will see what I can do.
> 
> See my announcement of a pending-fixes branch in linux-next (on LKML
> and others)

Excellent.

Build/test results match mainline.

For v4.17-rc4-241-ga1b6c55:

Build results:
	total: 132 pass: 130 fail: 2
Failed builds:
	m68k:allmodconfig
	xtensa:allmodconfig
Qemu test results:
	total: 138 pass: 138 fail: 0

Guenter

Re: [Ksummit-discuss] bug-introducing patches

[Andy Shevchenko]

On Sat, May 12, 2018 at 7:38 AM, Stephen Rothwell <sfr@canb.auug.org.au> wrote:
> Hi all,
>
> On Wed, 9 May 2018 20:47:27 +1000 Stephen Rothwell <sfr@canb.auug.org.au> wrote:
>>
>> On Wed, 9 May 2018 18:03:46 +0900 Mark Brown <broonie@kernel.org> wrote:
>> >
>> > On Wed, May 09, 2018 at 10:47:57AM +0200, Daniel Vetter wrote:
>> > > On Wed, May 9, 2018 at 10:44 AM, Mark Brown <broonie@kernel.org> wrote:
>> >
>> > > > I think this is an excellent idea, copying in Stephen for his input.
>> > > > I'm currently on holiday but unless someone convinces me it's a terrible
>> > > > idea I'm willing to at least give it a go on a trial basis once I'm back
>> > > > home.
>> >
>> > > Since Stephen merges all -fixes branches first, before merging all the
>> > > -next branches, he already generates that as part of linux-next. All
>> > > he'd need to do is push that intermediate state out to some
>> > > linux-fixes branch for consumption by test bots.
>>
>> Good idea ... I will see what I can do.
>
> See my announcement of a pending-fixes branch in linux-next (on LKML
> and others)
>
>> I currently have 44 such fixes branches.  More welcome!
>
> We are up to 55.

For PDx86 we have

git://git.infradead.org/linux-platform-drivers-x86.git fixes

branch

-- 
With Best Regards,
Andy Shevchenko

Re: [Ksummit-discuss] bug-introducing patches

[Ulf Hansson]

On 9 May 2018 at 12:47, Stephen Rothwell <sfr@canb.auug.org.au> wrote:
> On Wed, 9 May 2018 18:03:46 +0900 Mark Brown <broonie@kernel.org> wrote:
>>
>> On Wed, May 09, 2018 at 10:47:57AM +0200, Daniel Vetter wrote:
>> > On Wed, May 9, 2018 at 10:44 AM, Mark Brown <broonie@kernel.org> wrote:
>>
>> > > I think this is an excellent idea, copying in Stephen for his input.
>> > > I'm currently on holiday but unless someone convinces me it's a terrible
>> > > idea I'm willing to at least give it a go on a trial basis once I'm back
>> > > home.
>>
>> > Since Stephen merges all -fixes branches first, before merging all the
>> > -next branches, he already generates that as part of linux-next. All
>> > he'd need to do is push that intermediate state out to some
>> > linux-fixes branch for consumption by test bots.
>
> Good idea ... I will see what I can do.
>
>> True.  It's currently only those -fixes branches that people have asked
>> him to merge separately which isn't as big a proportion of trees as have
>> them (perhaps fortunately given people's enthusiasm for fixes branches
>> that don't merge cleanly with their development branches) so we'd also
>> need to encourage people to add them separately.
>
> I currently have 44 such fixes branches.  More welcome!

Great!

Stephen, thanks for picking up the idea so quickly!

I will ping the kernelci folkz to request them to include your new
fixes branch for daily builds.

For mmc, please add my fixes branch according to below.

git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc.git fixes

Kind regards
Uffe
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Stephen Rothwell]

[-- Attachment #1.1: Type: text/plain, Size: 1239 bytes --]

Hi Ulf,

On Mon, 14 May 2018 10:36:04 +0200 Ulf Hansson <ulf.hansson@linaro.org> wrote:
>
> I will ping the kernelci folkz to request them to include your new
> fixes branch for daily builds.

Excellent, thanks.

> For mmc, please add my fixes branch according to below.
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc.git fixes

Added from today.

Thanks for adding your subsystem tree as a participant of linux-next.  As
you may know, this is not a judgement of your code.  The purpose of
linux-next is for integration testing and to lower the impact of
conflicts between subsystems in the next merge window. 

You will need to ensure that the patches/commits in your tree/series have
been:
     * submitted under GPL v2 (or later) and include the Contributor's
        Signed-off-by,
     * posted to the relevant mailing list,
     * reviewed by you (or another maintainer of your subsystem tree),
     * successfully unit tested, and 
     * destined for the current or next Linux merge window.

Basically, this should be just what you would send to Linus (or ask him
to fetch).  It is allowed to be rebased if you deem it necessary.

-- 
Cheers,
Stephen Rothwell 
sfr@canb.auug.org.au

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 186 bytes --]

_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 194 bytes --]

On Mon, May 14, 2018 at 10:36:04AM +0200, Ulf Hansson wrote:

> I will ping the kernelci folkz to request them to include your new
> fixes branch for daily builds.

No need, I already added it.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Ksummit-discuss] bug-introducing patches

[Jiri Kosina]

On Wed, 9 May 2018, Daniel Vetter wrote:

> >> Then, why don't we have a pre-integration tree for fixes? That would
> >> at least simply automated testing of fixes separately from new
> >> material.
> >
> >> Perhaps this has already been discussed, and concluded and it's not
> >> worth it, then apologize for my ignorance.
> >
> > I think this is an excellent idea, copying in Stephen for his input.
> > I'm currently on holiday but unless someone convinces me it's a terrible
> > idea I'm willing to at least give it a go on a trial basis once I'm back
> > home.
> 
> Since Stephen merges all -fixes branches first, before merging all the
> -next branches, he already generates that as part of linux-next. All
> he'd need to do is push that intermediate state out to some
> linux-fixes branch for consumption by test bots.

What I do for my trees is that I actually merge the '-fixes' branch (that 
is scheduled to go to Linus in the 'current' cycle) into my for-next 
branch as well.

This has the advantage of (a) getting all the coverage linux-next does (b) 
seeing any potential merge conflicts early

Is this not feasible for other trees?

-- 
Jiri Kosina
SUSE Labs

_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin via Ksummit-discuss]

On Thu, May 10, 2018 at 06:03:22PM +0200, Jiri Kosina wrote:
>On Wed, 9 May 2018, Daniel Vetter wrote:
>
>> >> Then, why don't we have a pre-integration tree for fixes? That would
>> >> at least simply automated testing of fixes separately from new
>> >> material.
>> >
>> >> Perhaps this has already been discussed, and concluded and it's not
>> >> worth it, then apologize for my ignorance.
>> >
>> > I think this is an excellent idea, copying in Stephen for his input.
>> > I'm currently on holiday but unless someone convinces me it's a terrible
>> > idea I'm willing to at least give it a go on a trial basis once I'm back
>> > home.
>>
>> Since Stephen merges all -fixes branches first, before merging all the
>> -next branches, he already generates that as part of linux-next. All
>> he'd need to do is push that intermediate state out to some
>> linux-fixes branch for consumption by test bots.
>
>What I do for my trees is that I actually merge the '-fixes' branch (that
>is scheduled to go to Linus in the 'current' cycle) into my for-next
>branch as well.
>
>This has the advantage of (a) getting all the coverage linux-next does (b)
>seeing any potential merge conflicts early
>
>Is this not feasible for other trees?

When Linus tags -rc1, -next will start filling up with commits destined
for the next merge window. The resulting -next tree becomes very
unstable, and very difficult to test.

The idea behind next-fixes is to provide a tree that will contain fixes
for the current merge window, which will generate a much more stable
tree that users/bots could actually run and validate the fixes that will
be merged in the upcoming weeks.

Right now, with the method you've described, there is no easy way to
test your '-fixes' branch even though the commits in there will be
pulled in by Linus much sooner than your 'for-next' branch.

You'll still get the same coverage from -next, but if you provide your
-fixes branch seperately you'll also get more coverage for the fixes
you're about to send to Linus.

_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Geert Uytterhoeven]

On Thu, May 10, 2018 at 6:47 PM, Sasha Levin via Ksummit-discuss
<ksummit-discuss@lists.linuxfoundation.org> wrote:
> On Thu, May 10, 2018 at 06:03:22PM +0200, Jiri Kosina wrote:
>>On Wed, 9 May 2018, Daniel Vetter wrote:
>>> >> Then, why don't we have a pre-integration tree for fixes? That would
>>> >> at least simply automated testing of fixes separately from new
>>> >> material.
>>> >
>>> >> Perhaps this has already been discussed, and concluded and it's not
>>> >> worth it, then apologize for my ignorance.
>>> >
>>> > I think this is an excellent idea, copying in Stephen for his input.
>>> > I'm currently on holiday but unless someone convinces me it's a terrible
>>> > idea I'm willing to at least give it a go on a trial basis once I'm back
>>> > home.
>>>
>>> Since Stephen merges all -fixes branches first, before merging all the
>>> -next branches, he already generates that as part of linux-next. All
>>> he'd need to do is push that intermediate state out to some
>>> linux-fixes branch for consumption by test bots.
>>
>>What I do for my trees is that I actually merge the '-fixes' branch (that
>>is scheduled to go to Linus in the 'current' cycle) into my for-next
>>branch as well.
>>
>>This has the advantage of (a) getting all the coverage linux-next does (b)
>>seeing any potential merge conflicts early
>>
>>Is this not feasible for other trees?
>
> When Linus tags -rc1, -next will start filling up with commits destined
> for the next merge window. The resulting -next tree becomes very
> unstable, and very difficult to test.
>
> The idea behind next-fixes is to provide a tree that will contain fixes
> for the current merge window, which will generate a much more stable
> tree that users/bots could actually run and validate the fixes that will
> be merged in the upcoming weeks.
>
> Right now, with the method you've described, there is no easy way to
> test your '-fixes' branch even though the commits in there will be
> pulled in by Linus much sooner than your 'for-next' branch.
>
> You'll still get the same coverage from -next, but if you provide your
> -fixes branch seperately you'll also get more coverage for the fixes
> you're about to send to Linus.

I think you missed the "as well" in Jiri's response.

When I create the bi-weekly renesas-drivers release (see e.g.
https://www.spinics.net/lists/linux-renesas-soc/msg27350.html), there are
some subsystems that manage to have several conflicts between their
for-next branch and their fixes in Linus' tree almost every single release.
Hence I strongly support merging your own fixes branches into your own
for-next branch, and resolve the conflicts yourself, to keep your for-next
branch conflict free.

(Note that the last release linked above was very atypical: it was one of
the very few (first one ever?) that didn't have any conflicts).

Thanks!

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Geert Uytterhoeven]

On Tue, May 1, 2018 at 10:00 PM, Sasha Levin
<Alexander.Levin@microsoft.com> wrote:
> On Tue, May 01, 2018 at 03:44:50PM -0400, Theodore Y. Ts'o wrote:
>>On Tue, May 01, 2018 at 04:38:21PM +0000, Sasha Levin wrote:
>>>  - A merge window commit spent 50% more days, on average, in -next than a -rc
>>>    commit.
>>
>>So it *used* to be the case that after the merge window, I would queue
>>up bug fixes for the next merge window.  Greg K-H pushed for me to
>>send them to Linus sooner, instead of waiting for the next merge
>>window.  TBH, it's actually easier for me to just wait until the next
>>merge window, but please understand that there are multiple pressures
>>on maintainers going on here, and the latest efforts to try to use
>>AUTOSEL is just the most recent pressure placed on maintainers.
>>
>>The other thing is that when there is a regression users who are
>>testing linux-next want it fixed *fast*.  That's considered more
>>important to them than waiting for one, perfect patch, just to keep
>>AUTOSEL happy.
>>
>>So please understand that when you say that maintainers *need* to do X
>>or Y, that there you are not the only one standing in line putting
>>pressures on maintainers saying they *need* to do something.  And
>>quite frankly, I consider keeping people who are nice enough to test
>>linux-next happy to be **far** more important than AUTOSEL.
>
> Ted,
>
> I'm not at all asking to wait before adding the patches to your tree,
> or to -next. I'm asking to hold on to them a bit longer before you
> push them to Linus because I can show that patches that don't spend
> enough time in -next are more likely to introduce bugs.
>
> Yes, linux-next users want it fixed *now* and I completely agree it
> should be done that way, but the fix should not be immediately pushed to
> Linus as well.
>
> I've just finished reading an interesting article on LWN about the
> PostgreSQL fsync issues (https://lwn.net/Articles/752952/). If you
> look at Willy's commit, he wrote the final version of it about 5 days
> ago, Jeff merged it in 3 days ago, and Linus merged it in the tree
> today. Did it spend any time getting -next testing? nope.
>
> What's worse is that that commit is tagged for stable, which means
> that (given Greg's schedule) it may find it's way to -stable users
> even before some -next users/bots had a chance to test it out.

I just noticed a case where a commit was picked up for stable, while a
bot had flagged it as a build regression 18 hours earlier (with a CC to
lkml).

So it looks like the script for backporting commits should be enhanced to
check for this (searching for the commit ID in my email archive found the
bot report).

Thanks!

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Boris Brezillon]

On Mon, 14 May 2018 10:00:30 +0200
Geert Uytterhoeven <geert@linux-m68k.org> wrote:

> On Tue, May 1, 2018 at 10:00 PM, Sasha Levin
> <Alexander.Levin@microsoft.com> wrote:
> > On Tue, May 01, 2018 at 03:44:50PM -0400, Theodore Y. Ts'o wrote:  
> >>On Tue, May 01, 2018 at 04:38:21PM +0000, Sasha Levin wrote:  
> >>>  - A merge window commit spent 50% more days, on average, in -next than a -rc
> >>>    commit.  
> >>
> >>So it *used* to be the case that after the merge window, I would queue
> >>up bug fixes for the next merge window.  Greg K-H pushed for me to
> >>send them to Linus sooner, instead of waiting for the next merge
> >>window.  TBH, it's actually easier for me to just wait until the next
> >>merge window, but please understand that there are multiple pressures
> >>on maintainers going on here, and the latest efforts to try to use
> >>AUTOSEL is just the most recent pressure placed on maintainers.
> >>
> >>The other thing is that when there is a regression users who are
> >>testing linux-next want it fixed *fast*.  That's considered more
> >>important to them than waiting for one, perfect patch, just to keep
> >>AUTOSEL happy.
> >>
> >>So please understand that when you say that maintainers *need* to do X
> >>or Y, that there you are not the only one standing in line putting
> >>pressures on maintainers saying they *need* to do something.  And
> >>quite frankly, I consider keeping people who are nice enough to test
> >>linux-next happy to be **far** more important than AUTOSEL.  
> >
> > Ted,
> >
> > I'm not at all asking to wait before adding the patches to your tree,
> > or to -next. I'm asking to hold on to them a bit longer before you
> > push them to Linus because I can show that patches that don't spend
> > enough time in -next are more likely to introduce bugs.
> >
> > Yes, linux-next users want it fixed *now* and I completely agree it
> > should be done that way, but the fix should not be immediately pushed to
> > Linus as well.
> >
> > I've just finished reading an interesting article on LWN about the
> > PostgreSQL fsync issues (https://lwn.net/Articles/752952/). If you
> > look at Willy's commit, he wrote the final version of it about 5 days
> > ago, Jeff merged it in 3 days ago, and Linus merged it in the tree
> > today. Did it spend any time getting -next testing? nope.
> >
> > What's worse is that that commit is tagged for stable, which means
> > that (given Greg's schedule) it may find it's way to -stable users
> > even before some -next users/bots had a chance to test it out.  
> 
> I just noticed a case where a commit was picked up for stable, while a
> bot had flagged it as a build regression 18 hours earlier (with a CC to
> lkml).

Also, this patch has been on a tree that I know is tested by Fengguang's
robots for more than a week (and in linux-next for 2 days, which, I
agree, is probably not enough), and still, I only received the bug
report when the patch reached mainline. Are there tests that are only
run on Linus' tree?

> 
> So it looks like the script for backporting commits should be enhanced to
> check for this (searching for the commit ID in my email archive found the
> bot report).
> 
> Thanks!
> 
> Gr{oetje,eeting}s,
> 
>                         Geert
> 

_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Geert Uytterhoeven]

Hi Boris,

On Mon, May 14, 2018 at 10:12 AM, Boris Brezillon
<boris.brezillon@bootlin.com> wrote:
> On Mon, 14 May 2018 10:00:30 +0200
> Geert Uytterhoeven <geert@linux-m68k.org> wrote:
>> On Tue, May 1, 2018 at 10:00 PM, Sasha Levin
>> <Alexander.Levin@microsoft.com> wrote:
>> > On Tue, May 01, 2018 at 03:44:50PM -0400, Theodore Y. Ts'o wrote:
>> >>On Tue, May 01, 2018 at 04:38:21PM +0000, Sasha Levin wrote:
>> > What's worse is that that commit is tagged for stable, which means
>> > that (given Greg's schedule) it may find it's way to -stable users
>> > even before some -next users/bots had a chance to test it out.
>>
>> I just noticed a case where a commit was picked up for stable, while a
>> bot had flagged it as a build regression 18 hours earlier (with a CC to
>> lkml).
>
> Also, this patch has been on a tree that I know is tested by Fengguang's
> robots for more than a week (and in linux-next for 2 days, which, I
> agree, is probably not enough), and still, I only received the bug
> report when the patch reached mainline. Are there tests that are only
> run on Linus' tree?

Have your received a success report from Fengguang's bot, listing all
configs tested (the broken one should be included; it is included in the
configs tested on my branches)?

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Boris Brezillon]

On Mon, 14 May 2018 10:29:04 +0200
Geert Uytterhoeven <geert@linux-m68k.org> wrote:

> Hi Boris,
> 
> On Mon, May 14, 2018 at 10:12 AM, Boris Brezillon
> <boris.brezillon@bootlin.com> wrote:
> > On Mon, 14 May 2018 10:00:30 +0200
> > Geert Uytterhoeven <geert@linux-m68k.org> wrote:  
> >> On Tue, May 1, 2018 at 10:00 PM, Sasha Levin
> >> <Alexander.Levin@microsoft.com> wrote:  
> >> > On Tue, May 01, 2018 at 03:44:50PM -0400, Theodore Y. Ts'o wrote:  
> >> >>On Tue, May 01, 2018 at 04:38:21PM +0000, Sasha Levin wrote:  
> >> > What's worse is that that commit is tagged for stable, which means
> >> > that (given Greg's schedule) it may find it's way to -stable users
> >> > even before some -next users/bots had a chance to test it out.  
> >>
> >> I just noticed a case where a commit was picked up for stable, while a
> >> bot had flagged it as a build regression 18 hours earlier (with a CC to
> >> lkml).  
> >
> > Also, this patch has been on a tree that I know is tested by Fengguang's
> > robots for more than a week (and in linux-next for 2 days, which, I
> > agree, is probably not enough), and still, I only received the bug
> > report when the patch reached mainline. Are there tests that are only
> > run on Linus' tree?  
> 
> Have your received a success report from Fengguang's bot, listing all
> configs tested (the broken one should be included; it is included in the
> configs tested on my branches)?

Yes I did (see below).

-->8--
From: kbuild test robot <lkp@intel.com>
To: Boris Brezillon <boris.brezillon@bootlin.com>
Subject: [bbrezillon-0day:mtd/fixes] BUILD SUCCESS  fc3a9e15b492eef707afd56b7478001fdecfe53f
Date: Mon, 07 May 2018 20:05:52 +0800
User-Agent: Heirloom mailx 12.5 6/20/10

tree/branch: https://github.com/bbrezillon/linux-0day  mtd/fixes
branch HEAD: fc3a9e15b492eef707afd56b7478001fdecfe53f  mtd: rawnand: Make sure we wait tWB before polling the STATUS reg

elapsed time: 49m

configs tested: 142

The following configs have been built successfully.
More configs may be tested in the coming days.

powerpc                     skiroot_defconfig
sh                          kfr2r09_defconfig
x86_64                             acpi-redef
x86_64                           allyesdebian
x86_64                                nfsroot
m68k                       bvme6000_defconfig
powerpc                     ppa8548_defconfig
sh                                allnoconfig
sh                          rsk7269_defconfig
sh                  sh7785lcr_32bit_defconfig
sh                            titan_defconfig
i386                   randconfig-c0-05071338
i386                               tinyconfig
i386                     randconfig-n0-201818
x86_64                 randconfig-x002-201818
x86_64                 randconfig-x006-201818
x86_64                 randconfig-x005-201818
x86_64                 randconfig-x001-201818
x86_64                 randconfig-x009-201818
x86_64                 randconfig-x004-201818
x86_64                 randconfig-x003-201818
x86_64                 randconfig-x007-201818
x86_64                 randconfig-x000-201818
x86_64                 randconfig-x008-201818
i386                     randconfig-i1-201818
i386                     randconfig-i0-201818
alpha                               defconfig
parisc                            allnoconfig
parisc                         b180_defconfig
parisc                        c3000_defconfig
parisc                              defconfig
ia64                                defconfig
mips                                defconfig
powerpc                           allnoconfig
powerpc                             defconfig
powerpc                       ppc64_defconfig
s390                        default_defconfig
x86_64                 randconfig-g0-05071702
openrisc                    or1ksim_defconfig
um                             i386_defconfig
um                           x86_64_defconfig
i386                             allmodconfig
ia64                             alldefconfig
ia64                              allnoconfig
i386                     randconfig-a0-201818
i386                     randconfig-a1-201818
x86_64                 randconfig-s0-05071833
x86_64                 randconfig-s1-05071833
x86_64                 randconfig-s2-05071833
x86_64                 randconfig-s0-05071933
x86_64                 randconfig-s1-05071933
x86_64                 randconfig-s2-05071933
c6x                        evmc6678_defconfig
h8300                    h8300h-sim_defconfig
nios2                         10m50_defconfig
xtensa                       common_defconfig
xtensa                          iss_defconfig
i386                             alldefconfig
i386                              allnoconfig
i386                                defconfig
x86_64                              federa-25
x86_64                                   rhel
x86_64                               rhel-7.2
m68k                       m5475evb_defconfig
m68k                          multi_defconfig
m68k                           sun3_defconfig
i386                     randconfig-s0-201818
i386                     randconfig-s1-201818
x86_64                 randconfig-s3-05071918
x86_64                 randconfig-s4-05071918
x86_64                 randconfig-s5-05071918
mips                           32r2_defconfig
mips                         64r6el_defconfig
mips                              allnoconfig
mips                      fuloong2e_defconfig
mips                                   jz4740
mips                      malta_kvm_defconfig
mips                                     txx9
x86_64                   randconfig-i0-201818
i386                   randconfig-b0-05071353
sparc                               defconfig
sparc64                           allnoconfig
sparc64                             defconfig
mips                      maltaaprp_defconfig
mips                           mtx1_defconfig
microblaze                      mmu_defconfig
microblaze                    nommu_defconfig
i386                   randconfig-h0-05070458
i386                   randconfig-h1-05070458
x86_64                randconfig-ne0-05071525
x86_64                 randconfig-h0-05070841
i386                   randconfig-x008-201818
i386                   randconfig-x006-201818
i386                   randconfig-x000-201818
i386                   randconfig-x004-201818
i386                   randconfig-x009-201818
i386                   randconfig-x003-201818
i386                   randconfig-x001-201818
i386                   randconfig-x002-201818
i386                   randconfig-x005-201818
i386                   randconfig-x007-201818
x86_64                 randconfig-x010-201818
x86_64                 randconfig-x014-201818
x86_64                 randconfig-x013-201818
x86_64                 randconfig-x011-201818
x86_64                 randconfig-x015-201818
x86_64                 randconfig-x019-201818
x86_64                 randconfig-x016-201818
x86_64                 randconfig-x018-201818
x86_64                 randconfig-x012-201818
x86_64                 randconfig-x017-201818
i386                   randconfig-x013-201818
i386                   randconfig-x014-201818
i386                   randconfig-x019-201818
i386                   randconfig-x016-201818
i386                   randconfig-x015-201818
i386                   randconfig-x012-201818
i386                   randconfig-x018-201818
i386                   randconfig-x010-201818
i386                   randconfig-x011-201818
i386                   randconfig-x017-201818
i386                   randconfig-x073-201818
i386                   randconfig-x075-201818
i386                   randconfig-x079-201818
i386                   randconfig-x071-201818
i386                   randconfig-x076-201818
i386                   randconfig-x074-201818
i386                   randconfig-x078-201818
i386                   randconfig-x077-201818
i386                   randconfig-x070-201818
i386                   randconfig-x072-201818
arm                               allnoconfig
arm                         at91_dt_defconfig
arm                           efm32_defconfig
arm                          exynos_defconfig
arm                        multi_v5_defconfig
arm                        multi_v7_defconfig
arm                        shmobile_defconfig
arm                           sunxi_defconfig
arm64                             allnoconfig
arm64                               defconfig
i386                   randconfig-x0-05071422

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Geert Uytterhoeven]

Hi Boris,

On Mon, May 14, 2018 at 10:34 AM, Boris Brezillon
<boris.brezillon@bootlin.com> wrote:
> On Mon, 14 May 2018 10:29:04 +0200
> Geert Uytterhoeven <geert@linux-m68k.org> wrote:
>> On Mon, May 14, 2018 at 10:12 AM, Boris Brezillon
>> <boris.brezillon@bootlin.com> wrote:
>> > On Mon, 14 May 2018 10:00:30 +0200
>> > Geert Uytterhoeven <geert@linux-m68k.org> wrote:
>> >> On Tue, May 1, 2018 at 10:00 PM, Sasha Levin
>> >> <Alexander.Levin@microsoft.com> wrote:
>> >> > On Tue, May 01, 2018 at 03:44:50PM -0400, Theodore Y. Ts'o wrote:
>> >> >>On Tue, May 01, 2018 at 04:38:21PM +0000, Sasha Levin wrote:
>> >> > What's worse is that that commit is tagged for stable, which means
>> >> > that (given Greg's schedule) it may find it's way to -stable users
>> >> > even before some -next users/bots had a chance to test it out.
>> >>
>> >> I just noticed a case where a commit was picked up for stable, while a
>> >> bot had flagged it as a build regression 18 hours earlier (with a CC to
>> >> lkml).
>> >
>> > Also, this patch has been on a tree that I know is tested by Fengguang's
>> > robots for more than a week (and in linux-next for 2 days, which, I
>> > agree, is probably not enough), and still, I only received the bug
>> > report when the patch reached mainline. Are there tests that are only
>> > run on Linus' tree?
>>
>> Have your received a success report from Fengguang's bot, listing all
>> configs tested (the broken one should be included; it is included in the
>> configs tested on my branches)?
>
> Yes I did (see below).
>
> -->8--
> From: kbuild test robot <lkp@intel.com>
> To: Boris Brezillon <boris.brezillon@bootlin.com>
> Subject: [bbrezillon-0day:mtd/fixes] BUILD SUCCESS  fc3a9e15b492eef707afd56b7478001fdecfe53f
> Date: Mon, 07 May 2018 20:05:52 +0800
> User-Agent: Heirloom mailx 12.5 6/20/10
>
> tree/branch: https://github.com/bbrezillon/linux-0day  mtd/fixes
> branch HEAD: fc3a9e15b492eef707afd56b7478001fdecfe53f  mtd: rawnand: Make sure we wait tWB before polling the STATUS reg
>
> elapsed time: 49m
>
> configs tested: 142

But the failed config (m68k/allmodconfig) is not listed?

BTW, my last report had:

    configs tested: 178

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Boris Brezillon]

+Fengguang

On Mon, 14 May 2018 10:40:10 +0200
Geert Uytterhoeven <geert@linux-m68k.org> wrote:

> Hi Boris,
> 
> On Mon, May 14, 2018 at 10:34 AM, Boris Brezillon
> <boris.brezillon@bootlin.com> wrote:
> > On Mon, 14 May 2018 10:29:04 +0200
> > Geert Uytterhoeven <geert@linux-m68k.org> wrote:  
> >> On Mon, May 14, 2018 at 10:12 AM, Boris Brezillon
> >> <boris.brezillon@bootlin.com> wrote:  
> >> > On Mon, 14 May 2018 10:00:30 +0200
> >> > Geert Uytterhoeven <geert@linux-m68k.org> wrote:  
> >> >> On Tue, May 1, 2018 at 10:00 PM, Sasha Levin
> >> >> <Alexander.Levin@microsoft.com> wrote:  
> >> >> > On Tue, May 01, 2018 at 03:44:50PM -0400, Theodore Y. Ts'o wrote:  
> >> >> >>On Tue, May 01, 2018 at 04:38:21PM +0000, Sasha Levin wrote:  
> >> >> > What's worse is that that commit is tagged for stable, which means
> >> >> > that (given Greg's schedule) it may find it's way to -stable users
> >> >> > even before some -next users/bots had a chance to test it out.  
> >> >>
> >> >> I just noticed a case where a commit was picked up for stable, while a
> >> >> bot had flagged it as a build regression 18 hours earlier (with a CC to
> >> >> lkml).  
> >> >
> >> > Also, this patch has been on a tree that I know is tested by Fengguang's
> >> > robots for more than a week (and in linux-next for 2 days, which, I
> >> > agree, is probably not enough), and still, I only received the bug
> >> > report when the patch reached mainline. Are there tests that are only
> >> > run on Linus' tree?  
> >>
> >> Have your received a success report from Fengguang's bot, listing all
> >> configs tested (the broken one should be included; it is included in the
> >> configs tested on my branches)?  
> >
> > Yes I did (see below).
> >  
> > -->8--  
> > From: kbuild test robot <lkp@intel.com>
> > To: Boris Brezillon <boris.brezillon@bootlin.com>
> > Subject: [bbrezillon-0day:mtd/fixes] BUILD SUCCESS  fc3a9e15b492eef707afd56b7478001fdecfe53f
> > Date: Mon, 07 May 2018 20:05:52 +0800
> > User-Agent: Heirloom mailx 12.5 6/20/10
> >
> > tree/branch: https://github.com/bbrezillon/linux-0day  mtd/fixes
> > branch HEAD: fc3a9e15b492eef707afd56b7478001fdecfe53f  mtd: rawnand: Make sure we wait tWB before polling the STATUS reg
> >
> > elapsed time: 49m
> >
> > configs tested: 142  
> 
> But the failed config (m68k/allmodconfig) is not listed?

Yes, that's my point. It seems that some configs are only rarely
(never?) tested on my linux-0day tree (probably because they take longer
to build), and I should only take kbuild robot results as an indication
not a guarantee.
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Fengguang Wu]

On Mon, May 14, 2018 at 10:48:03AM +0200, Boris Brezillon wrote:
>+Fengguang
>
>On Mon, 14 May 2018 10:40:10 +0200
>Geert Uytterhoeven <geert@linux-m68k.org> wrote:
>
>> Hi Boris,
>>
>> On Mon, May 14, 2018 at 10:34 AM, Boris Brezillon
>> <boris.brezillon@bootlin.com> wrote:
>> > On Mon, 14 May 2018 10:29:04 +0200
>> > Geert Uytterhoeven <geert@linux-m68k.org> wrote:
>> >> On Mon, May 14, 2018 at 10:12 AM, Boris Brezillon
>> >> <boris.brezillon@bootlin.com> wrote:
>> >> > On Mon, 14 May 2018 10:00:30 +0200
>> >> > Geert Uytterhoeven <geert@linux-m68k.org> wrote:
>> >> >> On Tue, May 1, 2018 at 10:00 PM, Sasha Levin
>> >> >> <Alexander.Levin@microsoft.com> wrote:
>> >> >> > On Tue, May 01, 2018 at 03:44:50PM -0400, Theodore Y. Ts'o wrote:
>> >> >> >>On Tue, May 01, 2018 at 04:38:21PM +0000, Sasha Levin wrote:
>> >> >> > What's worse is that that commit is tagged for stable, which means
>> >> >> > that (given Greg's schedule) it may find it's way to -stable users
>> >> >> > even before some -next users/bots had a chance to test it out.
>> >> >>
>> >> >> I just noticed a case where a commit was picked up for stable, while a
>> >> >> bot had flagged it as a build regression 18 hours earlier (with a CC to
>> >> >> lkml).
>> >> >
>> >> > Also, this patch has been on a tree that I know is tested by Fengguang's
>> >> > robots for more than a week (and in linux-next for 2 days, which, I
>> >> > agree, is probably not enough), and still, I only received the bug
>> >> > report when the patch reached mainline. Are there tests that are only
>> >> > run on Linus' tree?
>> >>
>> >> Have your received a success report from Fengguang's bot, listing all
>> >> configs tested (the broken one should be included; it is included in the
>> >> configs tested on my branches)?
>> >
>> > Yes I did (see below).
>> >
>> > -->8--
>> > From: kbuild test robot <lkp@intel.com>
>> > To: Boris Brezillon <boris.brezillon@bootlin.com>
>> > Subject: [bbrezillon-0day:mtd/fixes] BUILD SUCCESS  fc3a9e15b492eef707afd56b7478001fdecfe53f
>> > Date: Mon, 07 May 2018 20:05:52 +0800
>> > User-Agent: Heirloom mailx 12.5 6/20/10
>> >
>> > tree/branch: https://github.com/bbrezillon/linux-0day  mtd/fixes
>> > branch HEAD: fc3a9e15b492eef707afd56b7478001fdecfe53f  mtd: rawnand: Make sure we wait tWB before polling the STATUS reg
>> >
>> > elapsed time: 49m
>> >
>> > configs tested: 142
>>
>> But the failed config (m68k/allmodconfig) is not listed?
>
>Yes, that's my point. It seems that some configs are only rarely
>(never?) tested on my linux-0day tree (probably because they take longer
>to build), and I should only take kbuild robot results as an indication
>not a guarantee.

Yeah sorry, there is no 100% guarantee. There are 2 main aspects to
this problem.

- Response time vs coverage. Most build errors can be caught within 1
  day. The build success notification email is typically sent within
  half day (a reasonable feedback time). At this time, it can only be
  a rough indication not a guarantee. After sending the 0day build
  success notification, the build tests will actually continue for
  about 1 week to increase test coverage. 

- Merge-test-bisect based workflow. If one branch is hard to merge
  with others, especially if it's based on old kernel, it'll receive
  much less test coverage. Branches with known build/boot errors will
  be excluded from further merges, too.

Thanks,
Fengguang

Re: [Ksummit-discuss] bug-introducing patches

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1081 bytes --]

On Thu, May 10, 2018 at 06:03:22PM +0200, Jiri Kosina wrote:
> On Wed, 9 May 2018, Daniel Vetter wrote:

> > Since Stephen merges all -fixes branches first, before merging all the
> > -next branches, he already generates that as part of linux-next. All
> > he'd need to do is push that intermediate state out to some
> > linux-fixes branch for consumption by test bots.

> What I do for my trees is that I actually merge the '-fixes' branch (that 
> is scheduled to go to Linus in the 'current' cycle) into my for-next 
> branch as well.

> This has the advantage of (a) getting all the coverage linux-next does (b) 
> seeing any potential merge conflicts early

> Is this not feasible for other trees?

That's obviously best practice which I hope everyone who doesn't have a
separate fix branch in -next is doing but it means that the fixes branch
is not getting tested without the changes in your -next branch, and also
reduces the coverage separate to other people's -next branches.  This
means that there's room for implicit dependencies to slip through.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin]

On Thu, May 03, 2018 at 04:09:05PM -0700, Tony Lindgren wrote:
>* Mark Brown <broonie@kernel.org> [180503 22:44]:
>> On Wed, May 02, 2018 at 08:52:29PM -0700, Guenter Roeck wrote:
>>
>> > As for -next, me and others stopped reporting bugs in it, because when we do
>> > we tend to get flamed for the "noise". Is anyone aware (or cares) that mips
>> > and nds32 images don't build ? Soaking clothes in an empty bathtub won't make
>> > them wet, and bugs in code which no one builds, much less tests or uses, won't
>> > be found.
>>
>> You've been flamed for testing -next?  That's not been my experience and
>> frankly it's pretty horrifying that it's happening.  Testing is pretty
>> much the whole point of -next existing in the first place so you have to
>> wonder why people are putting their trees there if they don't want
>> testing.  I have seen a few issues with people reporting bugs on old
>> versions of -next but otherwise...
>
>Yes I agree testing Linux next is very important. That's the best way for
>maintainers to ensure a usable -rc1 after a merge window. And then for
>the -rc cycle, there not much of need for chasing bugs to get things working.
>
>Bugs reported for Linux next often seem to get fixed or reverted faster
>compared to the -rc cycle too. I think that's because people realize that
>their code will not get merged until it's been fixed.
>
>So some daily testing of Linux next can save a lot scrambling after the
>merge window :)
>
>Users don't usually upgrade kernels until after later -rc releases or only
>after major releases so that probably explains some of the -rc cycle fixes.

Tony, I'm curious, how many users are you aware of who actually run
Linus's tree? All the users I've encountered so far on Azure seem to be
running something based on -stable.

I can't really get any solid statistics about that on my end both
because I don't have visibility inside user VMs (I don't actually have
prod access believe it or not), and even if I had it would probably be 
confidential, so I'm just basing this on reports from user's I've seen 
so far.

I think that a question we should be asking ourselves is whether we
should be basing our decisions here on the assumption that (pretty much)
no one runs Linus's tree anymore?

Re: [Ksummit-discuss] bug-introducing patches

[Theodore Y. Ts'o]

On Tue, May 08, 2018 at 02:34:41AM +0000, Sasha Levin via Ksummit-discuss wrote:
> 
> Tony, I'm curious, how many users are you aware of who actually run
> Linus's tree? All the users I've encountered so far on Azure seem to be
> running something based on -stable.

The people who run Linus's tree and test -rc kernels tend to be kernel
developers and individual users who want to run bleeding edge kernels
and who generally are technically clueful.  If you were talking about
SLR cameras, you'd call them the "prosumers" segment of the market.

It tends to be more on desktops and laptops, so it doesn't surprise me
that you don't often see them in a hosting environment where you have
to pay $$$.  (And where you do see them in a hosting environment, it's
probably for things like gce-xfstests.)

> I think that a question we should be asking ourselves is whether we
> should be basing our decisions here on the assumption that (pretty much)
> no one runs Linus's tree anymore?

These people *do* exist, because as a maintainer, I get bug reports
from them.  (And sometimes as a user, I send bug reports when running
-rc kernels to other maintainers, such as the i915 drivers and the
Intel Wireless driver folks.)

Such reports are incredibly valuable and precious to me, since it
allows me to find problems that weren't picked up in my own testing.
(In the case of Intel Wireless, a while back the IWL team didn't have
Aruba Enterprise Access Points in their test hardware library, so I
found a regression after the merge window because I was running -rcX
on my laptop, and wireless access to googleguest network broke.  If I
hadn't been running -rcX, they probably wouldn't have discovered this
problem until after that particular kernel had been released.)

So keeping those users happy is a good thing; since they tend to be
very technically clueful, they can do bisections for you, and they are
able to give a detailed and useful bug report.  If they report that a
regression that was introduced in -rc2 is fixed by a particular patch,
I want to push it into -rc3 immediately, and not let it stall in
linux-next.  If the reason why is because you don't trust my patch
because it "only" got tested by the technically advanced user
reporting the regression, then don't take patches from -rc3 into your
stable branch right away!  Let it bake in Linus's tree anfor a week or
two, instead of demanding that patches stick around in Linux-next
before flowing into Linus's tree.

Because I will guarantee you this --- there are more real users
running Linus's tree than linux-next.  This is because Linus's tree
tends to be far more stable than linux-next, since after -rc2
linux-next starts getting the first set of experiments for what will
be going into the next merge window.  So while I am willing to run
something based on -rc2 or later on my laptop, there is no way in heck
I would be willing to put linux-next on my laptop.  That's just way
too exciting for me....

Would I pull down linux-next, and fire up a VM running gce-xfstests?
Sure.  But that's not a real-life use case; that's just running canned
test cases.  And more often than not, linux-next will be broken while
Linus's -rcX tree is just fine; which is why I do most of my ext4
testing using patches based on top of -rcX, not based on top of
linux-next.

       		     		 	     - Ted

Re: [Ksummit-discuss] bug-introducing patches

[Tony Lindgren]

* Theodore Y. Ts'o <tytso@mit.edu> [180508 03:50]:
> On Tue, May 08, 2018 at 02:34:41AM +0000, Sasha Levin via Ksummit-discuss wrote:
> > 
> > Tony, I'm curious, how many users are you aware of who actually run
> > Linus's tree? All the users I've encountered so far on Azure seem to be
> > running something based on -stable.
> 
> The people who run Linus's tree and test -rc kernels tend to be kernel
> developers and individual users who want to run bleeding edge kernels
> and who generally are technically clueful.  If you were talking about
> SLR cameras, you'd call them the "prosumers" segment of the market.

Yup that's the category. People tinkering with their devices and
using bleeding edge kernels because of some new device driver only
being in thr -rc series for example.

> > I think that a question we should be asking ourselves is whether we
> > should be basing our decisions here on the assumption that (pretty much)
> > no one runs Linus's tree anymore?
> 
> These people *do* exist, because as a maintainer, I get bug reports
> from them.  (And sometimes as a user, I send bug reports when running
> -rc kernels to other maintainers, such as the i915 drivers and the
> Intel Wireless driver folks.)

Yes.

> Such reports are incredibly valuable and precious to me, since it
> allows me to find problems that weren't picked up in my own testing.
> (In the case of Intel Wireless, a while back the IWL team didn't have
> Aruba Enterprise Access Points in their test hardware library, so I
> found a regression after the merge window because I was running -rcX
> on my laptop, and wireless access to googleguest network broke.  If I
> hadn't been running -rcX, they probably wouldn't have discovered this
> problem until after that particular kernel had been released.)

Yes. So as maintainers, we should all test Linux next on frequent
basis to aim for usable -rc1 with no regressions based on that
testing. Then the rest of the -rc cycle should be easy with more
testing and reports from the "prosumer" market :)

> So keeping those users happy is a good thing; since they tend to be
> very technically clueful, they can do bisections for you, and they are
> able to give a detailed and useful bug report.  If they report that a
> regression that was introduced in -rc2 is fixed by a particular patch,
> I want to push it into -rc3 immediately, and not let it stall in
> linux-next.  If the reason why is because you don't trust my patch
> because it "only" got tested by the technically advanced user
> reporting the regression, then don't take patches from -rc3 into your
> stable branch right away!  Let it bake in Linus's tree anfor a week or
> two, instead of demanding that patches stick around in Linux-next
> before flowing into Linus's tree.
> 
> Because I will guarantee you this --- there are more real users
> running Linus's tree than linux-next.  This is because Linus's tree
> tends to be far more stable than linux-next, since after -rc2
> linux-next starts getting the first set of experiments for what will
> be going into the next merge window.  So while I am willing to run
> something based on -rc2 or later on my laptop, there is no way in heck
> I would be willing to put linux-next on my laptop.  That's just way
> too exciting for me....

I follow Linux next on few test systems. Then when I see no regressions,
I might dare try it on my laptop. Something that's usable one week in
next may not be so any longer the next week. So testing minimum few
times a week and carrying occasional reverts are often needed to be
able to test Linux next on daily basis.

> Would I pull down linux-next, and fire up a VM running gce-xfstests?
> Sure.  But that's not a real-life use case; that's just running canned
> test cases.  And more often than not, linux-next will be broken while
> Linus's -rcX tree is just fine; which is why I do most of my ext4
> testing using patches based on top of -rcX, not based on top of
> linux-next.

Ideally we would somehow always end up with an -rc1 that people dare
to use though for the "prosumer" testing :)

Regards,

Tony

Re: [Ksummit-discuss] bug-introducing patches

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 794 bytes --]

On Tue, May 08, 2018 at 07:49:59AM -0700, Tony Lindgren wrote:
> * Theodore Y. Ts'o <tytso@mit.edu> [180508 03:50]:

> > The people who run Linus's tree and test -rc kernels tend to be kernel
> > developers and individual users who want to run bleeding edge kernels
> > and who generally are technically clueful.  If you were talking about
> > SLR cameras, you'd call them the "prosumers" segment of the market.

> Yup that's the category. People tinkering with their devices and
> using bleeding edge kernels because of some new device driver only
> being in thr -rc series for example.

You also get some people who are intending to ship on stable kernels but
are tracking upstream during product development so that they can be on
the most current stable release when they go to production.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Ksummit-discuss] bug-introducing patches

[Tony Lindgren]

* Tony Lindgren <tony@atomide.com> [180508 14:52]:
> * Theodore Y. Ts'o <tytso@mit.edu> [180508 03:50]:
> > Would I pull down linux-next, and fire up a VM running gce-xfstests?
> > Sure.  But that's not a real-life use case; that's just running canned
> > test cases.  And more often than not, linux-next will be broken while
> > Linus's -rcX tree is just fine; which is why I do most of my ext4
> > testing using patches based on top of -rcX, not based on top of
> > linux-next.
> 
> Ideally we would somehow always end up with an -rc1 that people dare
> to use though for the "prosumer" testing :)

BTW, the reason why I think we all should test Linux next on regular
basis is that it's often "some other people's branches(tm)" that cause
the regressions :) Maybe because their own test cases did not show
any regressions, or because they were unable to test the patches.
Or it's because of some "clean-up" work that's completely untested
on some systems.

And that's how we end up with regressions getting merged into -rc1.

Regards,

Tony
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin via Ksummit-discuss]

On Mon, May 07, 2018 at 11:48:20PM -0400, Theodore Y. Ts'o wrote:
>On Tue, May 08, 2018 at 02:34:41AM +0000, Sasha Levin via Ksummit-discuss wrote:
>>
>> Tony, I'm curious, how many users are you aware of who actually run
>> Linus's tree? All the users I've encountered so far on Azure seem to be
>> running something based on -stable.
>
>The people who run Linus's tree and test -rc kernels tend to be kernel
>developers and individual users who want to run bleeding edge kernels
>and who generally are technically clueful.  If you were talking about
>SLR cameras, you'd call them the "prosumers" segment of the market.
>
>It tends to be more on desktops and laptops, so it doesn't surprise me
>that you don't often see them in a hosting environment where you have
>to pay $$$.  (And where you do see them in a hosting environment, it's
>probably for things like gce-xfstests.)
>
>> I think that a question we should be asking ourselves is whether we
>> should be basing our decisions here on the assumption that (pretty much)
>> no one runs Linus's tree anymore?
>
>These people *do* exist, because as a maintainer, I get bug reports
>from them.  (And sometimes as a user, I send bug reports when running
>-rc kernels to other maintainers, such as the i915 drivers and the
>Intel Wireless driver folks.)
>
>Such reports are incredibly valuable and precious to me, since it
>allows me to find problems that weren't picked up in my own testing.
>(In the case of Intel Wireless, a while back the IWL team didn't have
>Aruba Enterprise Access Points in their test hardware library, so I
>found a regression after the merge window because I was running -rcX
>on my laptop, and wireless access to googleguest network broke.  If I
>hadn't been running -rcX, they probably wouldn't have discovered this
>problem until after that particular kernel had been released.)
>
>So keeping those users happy is a good thing; since they tend to be
>very technically clueful, they can do bisections for you, and they are
>able to give a detailed and useful bug report.  If they report that a
>regression that was introduced in -rc2 is fixed by a particular patch,
>I want to push it into -rc3 immediately, and not let it stall in
>linux-next.  If the reason why is because you don't trust my patch
>because it "only" got tested by the technically advanced user
>reporting the regression, then don't take patches from -rc3 into your
>stable branch right away!  Let it bake in Linus's tree anfor a week or
>two, instead of demanding that patches stick around in Linux-next
>before flowing into Linus's tree.
>
>Because I will guarantee you this --- there are more real users
>running Linus's tree than linux-next.  This is because Linus's tree
>tends to be far more stable than linux-next, since after -rc2
>linux-next starts getting the first set of experiments for what will
>be going into the next merge window.  So while I am willing to run
>something based on -rc2 or later on my laptop, there is no way in heck
>I would be willing to put linux-next on my laptop.  That's just way
>too exciting for me....
>
>Would I pull down linux-next, and fire up a VM running gce-xfstests?
>Sure.  But that's not a real-life use case; that's just running canned
>test cases.  And more often than not, linux-next will be broken while
>Linus's -rcX tree is just fine; which is why I do most of my ext4
>testing using patches based on top of -rcX, not based on top of
>linux-next.

This is interesting. We have a group of power users who are testing out
-rc releases, who are usually happy to test out a fast moving target and
provide helpful reports back. We also have a group who run a -stable
kernel (-stable build/distro/android/etc) who want to avoid having to
report bugs to us.

What we don't have is a group of people who use Linus's actual releases
(not the -rc stuff, but the actual point releases). Power users will
move on to the next kernel, and -stable folks won't touch that release
until there's a corresponding -stable.

Even rawhide, like Josh mentioned, will just fill back with the merge
window commits after the release of an older kernel.

So the problem I'm seeing is that since a merge window is open only once
every 2-3 months people will sometimes try to push poorly tested code
just to make that merge window. Additionally, as later -rc releases
start showing up people will again merge poorly tested fixes just to
make it in time for that release.

For both cases, people will push poorly tested code in the kernel just
because they want to make it in time for a kernel release that no one
will actually use.

What if, instead, Linus doesn't actually ever release a point release?
We can make the merge window open more often, and since there's no
actual release, people won't rush to push fixes in later -rc cycles.

We take away the incentive to push poorly tested code. Maintainers still
free to commit anything they'd like, but there's no reason to commit
code they're not confident of just to make it to a random release no one
will use.

Merge window will happen more often, so there's no real reason to rush
things in a particular window, and since -stable releases every week
there's no rush to push a fix in since the next release is just a week
away.
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Matthew Wilcox]

[-- Attachment #1.1: Type: text/plain, Size: 6100 bytes --]

I think your sample size omits some people. I run Debian Testing on my
laptop. That gets something akin to a Linus release pretty soon after he
releases it, and while it gets some amount of -stable patches, it
progresses to the next release fairly rapidly.

Added Ben to the cc for more updates.

I think Fedora does something similar.

On Tue, May 8, 2018, 16:29 Sasha Levin via Ksummit-discuss, <
ksummit-discuss@lists.linuxfoundation.org> wrote:

> On Mon, May 07, 2018 at 11:48:20PM -0400, Theodore Y. Ts'o wrote:
> >On Tue, May 08, 2018 at 02:34:41AM +0000, Sasha Levin via Ksummit-discuss
> wrote:
> >>
> >> Tony, I'm curious, how many users are you aware of who actually run
> >> Linus's tree? All the users I've encountered so far on Azure seem to be
> >> running something based on -stable.
> >
> >The people who run Linus's tree and test -rc kernels tend to be kernel
> >developers and individual users who want to run bleeding edge kernels
> >and who generally are technically clueful.  If you were talking about
> >SLR cameras, you'd call them the "prosumers" segment of the market.
> >
> >It tends to be more on desktops and laptops, so it doesn't surprise me
> >that you don't often see them in a hosting environment where you have
> >to pay $$$.  (And where you do see them in a hosting environment, it's
> >probably for things like gce-xfstests.)
> >
> >> I think that a question we should be asking ourselves is whether we
> >> should be basing our decisions here on the assumption that (pretty much)
> >> no one runs Linus's tree anymore?
> >
> >These people *do* exist, because as a maintainer, I get bug reports
> >from them.  (And sometimes as a user, I send bug reports when running
> >-rc kernels to other maintainers, such as the i915 drivers and the
> >Intel Wireless driver folks.)
> >
> >Such reports are incredibly valuable and precious to me, since it
> >allows me to find problems that weren't picked up in my own testing.
> >(In the case of Intel Wireless, a while back the IWL team didn't have
> >Aruba Enterprise Access Points in their test hardware library, so I
> >found a regression after the merge window because I was running -rcX
> >on my laptop, and wireless access to googleguest network broke.  If I
> >hadn't been running -rcX, they probably wouldn't have discovered this
> >problem until after that particular kernel had been released.)
> >
> >So keeping those users happy is a good thing; since they tend to be
> >very technically clueful, they can do bisections for you, and they are
> >able to give a detailed and useful bug report.  If they report that a
> >regression that was introduced in -rc2 is fixed by a particular patch,
> >I want to push it into -rc3 immediately, and not let it stall in
> >linux-next.  If the reason why is because you don't trust my patch
> >because it "only" got tested by the technically advanced user
> >reporting the regression, then don't take patches from -rc3 into your
> >stable branch right away!  Let it bake in Linus's tree anfor a week or
> >two, instead of demanding that patches stick around in Linux-next
> >before flowing into Linus's tree.
> >
> >Because I will guarantee you this --- there are more real users
> >running Linus's tree than linux-next.  This is because Linus's tree
> >tends to be far more stable than linux-next, since after -rc2
> >linux-next starts getting the first set of experiments for what will
> >be going into the next merge window.  So while I am willing to run
> >something based on -rc2 or later on my laptop, there is no way in heck
> >I would be willing to put linux-next on my laptop.  That's just way
> >too exciting for me....
> >
> >Would I pull down linux-next, and fire up a VM running gce-xfstests?
> >Sure.  But that's not a real-life use case; that's just running canned
> >test cases.  And more often than not, linux-next will be broken while
> >Linus's -rcX tree is just fine; which is why I do most of my ext4
> >testing using patches based on top of -rcX, not based on top of
> >linux-next.
>
> This is interesting. We have a group of power users who are testing out
> -rc releases, who are usually happy to test out a fast moving target and
> provide helpful reports back. We also have a group who run a -stable
> kernel (-stable build/distro/android/etc) who want to avoid having to
> report bugs to us.
>
> What we don't have is a group of people who use Linus's actual releases
> (not the -rc stuff, but the actual point releases). Power users will
> move on to the next kernel, and -stable folks won't touch that release
> until there's a corresponding -stable.
>
> Even rawhide, like Josh mentioned, will just fill back with the merge
> window commits after the release of an older kernel.
>
> So the problem I'm seeing is that since a merge window is open only once
> every 2-3 months people will sometimes try to push poorly tested code
> just to make that merge window. Additionally, as later -rc releases
> start showing up people will again merge poorly tested fixes just to
> make it in time for that release.
>
> For both cases, people will push poorly tested code in the kernel just
> because they want to make it in time for a kernel release that no one
> will actually use.
>
> What if, instead, Linus doesn't actually ever release a point release?
> We can make the merge window open more often, and since there's no
> actual release, people won't rush to push fixes in later -rc cycles.
>
> We take away the incentive to push poorly tested code. Maintainers still
> free to commit anything they'd like, but there's no reason to commit
> code they're not confident of just to make it to a random release no one
> will use.
>
> Merge window will happen more often, so there's no real reason to rush
> things in a particular window, and since -stable releases every week
> there's no rush to push a fix in since the next release is just a week
> away.
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
>

[-- Attachment #1.2: Type: text/html, Size: 7376 bytes --]

[-- Attachment #2: Type: text/plain, Size: 186 bytes --]

_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin]

On Tue, May 08, 2018 at 08:40:02PM +0000, Matthew Wilcox wrote:
>I think your sample size omits some people. I run Debian Testing on my
>laptop. That gets something akin to a Linus release pretty soon after he
>releases it, and while it gets some amount of -stable patches, it
>progresses to the next release fairly rapidly.

Debian testing is pretty much a -stable tree, see the git log history:

https://salsa.debian.org/kernel-team/linux/commits/sid

It follows a current stable tree, and moves on to the next one once it's
available (about a week after Linus releases a new kernel).

>Added Ben to the cc for more updates.
>
>I think Fedora does something similar.

Fedora's rawhide is just (daily?) builds of Linus's tree, they don't
care what stage the tree is in at any point.


My point is that no one picks a release and sticks with it more than a
week. If someone plans to use a release for longer term they use a
-stable tree, and if they are interested in testing, they move on to the
next release once it's available.

There's no one, for example, who picked up vanilla v4.16 and plans to
keep using it for a year.

This leads to my point about rushing fixes: -stable releases for v4.16 are
done weekly, there's no need to rush them in during v4.16-rc8 just to
make some imaginary release no one will pick up.

Re: [Ksummit-discuss] bug-introducing patches

[David Lang]

On Tue, 8 May 2018, Sasha Levin wrote:

> There's no one, for example, who picked up vanilla v4.16 and plans to
> keep using it for a year.

Actually, at a prior job I would do almost exactly that.

I never intended to go a year without updating, but it would happen if nothing 
came up that was related to the hardware/features I was running.

so 'no one uses the Linus kernel is false.

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin via Ksummit-discuss]

On Tue, May 08, 2018 at 01:59:18PM -0700, David Lang wrote:
>On Tue, 8 May 2018, Sasha Levin wrote:
>
>>There's no one, for example, who picked up vanilla v4.16 and plans to
>>keep using it for a year.
>
>Actually, at a prior job I would do almost exactly that.
>
>I never intended to go a year without updating, but it would happen if 
>nothing came up that was related to the hardware/features I was 
>running.
>
>so 'no one uses the Linus kernel is false.

My point is not that "no one ever uses Linus kernel" but that no one
takes one of those kernels and plans to stick with it for 3 months until
the next one comes up, even if there are updates relevant to that user.

Yes, some users will use a .0 release until either Greg releases a
-stable, or until the next -rc is out.

What I'm trying to say is that there is that the .0 release makes some
people rush poorly tested commits in it even though the .0 release is
not significant in any way.
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Dan Williams]

On Tue, May 8, 2018 at 2:43 PM, Sasha Levin via Ksummit-discuss
<ksummit-discuss@lists.linuxfoundation.org> wrote:
> On Tue, May 08, 2018 at 01:59:18PM -0700, David Lang wrote:
>>On Tue, 8 May 2018, Sasha Levin wrote:
>>
>>>There's no one, for example, who picked up vanilla v4.16 and plans to
>>>keep using it for a year.
>>
>>Actually, at a prior job I would do almost exactly that.
>>
>>I never intended to go a year without updating, but it would happen if
>>nothing came up that was related to the hardware/features I was
>>running.
>>
>>so 'no one uses the Linus kernel is false.
>
> My point is not that "no one ever uses Linus kernel" but that no one
> takes one of those kernels and plans to stick with it for 3 months until
> the next one comes up, even if there are updates relevant to that user.
>
> Yes, some users will use a .0 release until either Greg releases a
> -stable, or until the next -rc is out.
>
> What I'm trying to say is that there is that the .0 release makes some
> people rush poorly tested commits in it even though the .0 release is
> not significant in any way.

I think we should take pride in our releases, so I disagree that it is
insignificant. If a maintainer is rushing things into late rc's and
breaking things then they need that feedback, not de-emphasize the
importance of ".0" releases. Could the bar be raised higher on late
fixes, perhaps. I otherwise think the message is already clear
"changes at -rc6,7,8 had better be worthy of and coming in late and be
accompanied with good explanation".
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[James Bottomley]

On Tue, 2018-05-08 at 21:43 +0000, Sasha Levin via Ksummit-discuss
wrote:
> On Tue, May 08, 2018 at 01:59:18PM -0700, David Lang wrote:
> > On Tue, 8 May 2018, Sasha Levin wrote:
> > 
> > > There's no one, for example, who picked up vanilla v4.16 and
> > > plans to keep using it for a year.
> > 
> > Actually, at a prior job I would do almost exactly that.
> > 
> > I never intended to go a year without updating, but it would happen
> > if  nothing came up that was related to the hardware/features I
> > was running.
> > 
> > so 'no one uses the Linus kernel is false.
> 
> My point is not that "no one ever uses Linus kernel" but that no one
> takes one of those kernels and plans to stick with it for 3 months
> until the next one comes up, even if there are updates relevant to
> that user..

Actually, I have sometimes done that.  My current laptop is running the
v4.16 tag now, not because I intended to run it for this long but
because I've run into a Round Tuit shortage as far as the -rc
candidates go.

> Yes, some users will use a .0 release until either Greg releases a
> -stable, or until the next -rc is out.
> 
> What I'm trying to say is that there is that the .0 release makes
> some people rush poorly tested commits in it even though the .0
> release is not significant in any way.

As a milestone, it's extremely significant because it's the cadence
from which everything else flows.  If we as developers stop taking the
-rc cycle seriously, you'll find immediate negative consequences for
your stable kernels.  And I mean way worse consequences than the odd
bad judgment call about a patch that ought not to have gone in right
before a Linus release.

James

_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Justin Forbes]

On Tue, May 8, 2018 at 3:55 PM, Sasha Levin
<Alexander.Levin@microsoft.com> wrote:
> On Tue, May 08, 2018 at 08:40:02PM +0000, Matthew Wilcox wrote:
>>I think your sample size omits some people. I run Debian Testing on my
>>laptop. That gets something akin to a Linus release pretty soon after he
>>releases it, and while it gets some amount of -stable patches, it
>>progresses to the next release fairly rapidly.
>
> Debian testing is pretty much a -stable tree, see the git log history:
>
> https://salsa.debian.org/kernel-team/linux/commits/sid
>
> It follows a current stable tree, and moves on to the next one once it's
> available (about a week after Linus releases a new kernel).
>
>>Added Ben to the cc for more updates.
>>
>>I think Fedora does something similar.
>
> Fedora's rawhide is just (daily?) builds of Linus's tree, they don't
> care what stage the tree is in at any point.

It is, but there is a branch point when Linus releases. If we are
working on a new Fedora release, such as F28, all testing stayed on
4.16.0 until stable updates were released. If there is no release
deadline nearing, we have a "stabilization" repository where people
are using and testing the .0 release until stable updates happen. In
either case, the Linus release is really only tested until the stable
.1 happens, but there are users and testers of .0.

>
> My point is that no one picks a release and sticks with it more than a
> week. If someone plans to use a release for longer term they use a
> -stable tree, and if they are interested in testing, they move on to the
> next release once it's available.
>
> There's no one, for example, who picked up vanilla v4.16 and plans to
> keep using it for a year.
>
> This leads to my point about rushing fixes: -stable releases for v4.16 are
> done weekly, there's no need to rush them in during v4.16-rc8 just to
> make some imaginary release no one will pick up.

Re: [Ksummit-discuss] bug-introducing patches

[Ken Moffat]

On 8 May 2018 at 21:29, Sasha Levin <Alexander.Levin@microsoft.com> wrote:

>
> This is interesting. We have a group of power users who are testing out
> -rc releases, who are usually happy to test out a fast moving target and
> provide helpful reports back. We also have a group who run a -stable
> kernel (-stable build/distro/android/etc) who want to avoid having to
> report bugs to us.
>
> What we don't have is a group of people who use Linus's actual releases
> (not the -rc stuff, but the actual point releases). Power users will
> move on to the next kernel, and -stable folks won't touch that release
> until there's a corresponding -stable.

I resent that assumption :)

As a 'prosumer' in this context, I try to test an early -rc (usually
not until -rc2, sometimes not until later, depending on what I see on
this list), and then intermittently I spread the testing to more of my
desktop machines using later -rc versions. Once linus releases .0 I
hope to move my current systems to that in the next few days. But as
always, other things (sometimes real life, sometimes just new changes
in userspace) intervene.

After that, I will pick up Greg's latest if I build a new system
before the next kernel release, or if I become aware of something
critical (for my usage) in it.  And then probably 4 or 5 weeks after
linus's release I will start the next cycle of testing -rc verisons.

So no, I rarely test Greg's current stable version, but there _is_ a
period of some weeks where I run .0 kernels.

ĸen


>
> Even rawhide, like Josh mentioned, will just fill back with the merge
> window commits after the release of an older kernel.
>
> So the problem I'm seeing is that since a merge window is open only once
> every 2-3 months people will sometimes try to push poorly tested code
> just to make that merge window. Additionally, as later -rc releases
> start showing up people will again merge poorly tested fixes just to
> make it in time for that release.
>
> For both cases, people will push poorly tested code in the kernel just
> because they want to make it in time for a kernel release that no one
> will actually use.
>
> What if, instead, Linus doesn't actually ever release a point release?
> We can make the merge window open more often, and since there's no
> actual release, people won't rush to push fixes in later -rc cycles.
>
> We take away the incentive to push poorly tested code. Maintainers still
> free to commit anything they'd like, but there's no reason to commit
> code they're not confident of just to make it to a random release no one
> will use.
>
> Merge window will happen more often, so there's no real reason to rush
> things in a particular window, and since -stable releases every week
> there's no rush to push a fix in since the next release is just a week
> away.

Re: [Ksummit-discuss] bug-introducing patches

[Theodore Y. Ts'o]

On Tue, May 08, 2018 at 08:29:14PM +0000, Sasha Levin wrote:
> 
> This is interesting. We have a group of power users who are testing out
> -rc releases, who are usually happy to test out a fast moving target and
> provide helpful reports back. We also have a group who run a -stable
> kernel (-stable build/distro/android/etc) who want to avoid having to
> report bugs to us.
> 
> What we don't have is a group of people who use Linus's actual releases
> (not the -rc stuff, but the actual point releases). Power users will
> move on to the next kernel, and -stable folks won't touch that release
> until there's a corresponding -stable.

Linus doesn't release the point releases.  Those are done by the Greg
K-H and use the same process as does the stable kernels.  The only
difference is that the life point release doesn't last very long; just
until the next kernel release from Linus.

There are probably fewer people who use the point releases compared to
the stable kernels.  But I'd hesitate to call it zero.  We once
assumed that companies were all using Distro kernels, and very few
people used the stable kernels except for distribution channels
(enterprise kernels, BSP kernels, etc.).  Then we discovered that
there are people who use the stable kernel and don't go through the
enterprise distro vendors at all.  It wouldn't surprise me if there
are also a silent (and perhaps large) set of users who take Linus's
releases, and then follow along on the dot releases until the next
release from Linus.

> What if, instead, Linus doesn't actually ever release a point
> release?  We can make the merge window open more often, and since
> there's no actual release, people won't rush to push fixes in later
> -rc cycles.

I dont' understand your proposal.  Linus doesn't actually release
point releases.  Those happen during the -rcX cycle for those people
who are too chicken to follow Linus's tree, and just want the bug
fixes.

Getting rid of the point releases isn't going to change how frequently
the merge window opens.  What would do that is being much more strict
about when we only allow regression fixes only into the tree, so
hopefully the tree stablizes itself by -rc5 or -rc6.  

> Merge window will happen more often, so there's no real reason to
> rush things in a particular window, and since -stable releases every
> week there's no rush to push a fix in since the next release is just
> a week away.

Huh?

I can see shortening the release cycle to a six weeks, instead of our
current 8-9 week cycle.  But after the each cycle where we introduce
new features, during the merge window / integration phase, we do need
to have a time when are fixing regression bugs.

						- Ted

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin]

On Tue, May 08, 2018 at 06:15:34PM -0400, Theodore Y. Ts'o wrote:
>On Tue, May 08, 2018 at 08:29:14PM +0000, Sasha Levin wrote:
>>
>> This is interesting. We have a group of power users who are testing out
>> -rc releases, who are usually happy to test out a fast moving target and
>> provide helpful reports back. We also have a group who run a -stable
>> kernel (-stable build/distro/android/etc) who want to avoid having to
>> report bugs to us.
>>
>> What we don't have is a group of people who use Linus's actual releases
>> (not the -rc stuff, but the actual point releases). Power users will
>> move on to the next kernel, and -stable folks won't touch that release
>> until there's a corresponding -stable.
>
>Linus doesn't release the point releases.  Those are done by the Greg
>K-H and use the same process as does the stable kernels.  The only
>difference is that the life point release doesn't last very long; just
>until the next kernel release from Linus.
>
>There are probably fewer people who use the point releases compared to
>the stable kernels.  But I'd hesitate to call it zero.  We once
>assumed that companies were all using Distro kernels, and very few
>people used the stable kernels except for distribution channels
>(enterprise kernels, BSP kernels, etc.).  Then we discovered that
>there are people who use the stable kernel and don't go through the
>enterprise distro vendors at all.  It wouldn't surprise me if there
>are also a silent (and perhaps large) set of users who take Linus's
>releases, and then follow along on the dot releases until the next
>release from Linus.

I was referring to Linus's non-rc releases (4.15, 4.16, etc). While many
users start with, for example, 4.16, most users will either switch to
Greg's releases which start about a week after Linus's release, or
they'll move on to test the 4.17 -rc releases.

There are pretty much no users who pick 4.16, stay with it for 3 months,
switch to 4.17, stay with that for 3 months, and so on.

>> What if, instead, Linus doesn't actually ever release a point
>> release?  We can make the merge window open more often, and since
>> there's no actual release, people won't rush to push fixes in later
>> -rc cycles.
>
>I dont' understand your proposal.  Linus doesn't actually release
>point releases.  Those happen during the -rcX cycle for those people
>who are too chicken to follow Linus's tree, and just want the bug
>fixes.

What I'm suggesting is that "4.16" never gets released. When 4.16-rc8 is
closing and Linus would have tagged 4.16, he'd just open the merge
window and start merging in new features.

At this point Greg could either release 4.16.0 or wait a week and do
4.16.1. This effectively puts the kernel on a weekly release schedule.

>Getting rid of the point releases isn't going to change how frequently
>the merge window opens.  What would do that is being much more strict
>about when we only allow regression fixes only into the tree, so
>hopefully the tree stablizes itself by -rc5 or -rc6.
>
>> Merge window will happen more often, so there's no real reason to
>> rush things in a particular window, and since -stable releases every
>> week there's no rush to push a fix in since the next release is just
>> a week away.
>
>Huh?
>
>I can see shortening the release cycle to a six weeks, instead of our
>current 8-9 week cycle.  But after the each cycle where we introduce
>new features, during the merge window / integration phase, we do need
>to have a time when are fixing regression bugs.

What I'm suggesting is that most of the commits in -rc6/7/8 actually fix
bugs introduced in older kernels rather than the current merge window.
Thus, they don't have much value in "stabilizing" the release.

On the other hand, for some odd reason, folks will try squeezing poorly
tested commits into these late -rc cycles because "Linus is about to
release and we must make it in time for the release", even though in
practice there's no big rush to make it to a particular release since
most folks will just keep updating via -stable or distro kernels.

So since there's not much value in -rc6/7/8, just cancel them.

Re: [Ksummit-discuss] bug-introducing patches

[Willy Tarreau]

On Tue, May 08, 2018 at 08:29:14PM +0000, Sasha Levin wrote:
> What if, instead, Linus doesn't actually ever release a point release?
> We can make the merge window open more often, and since there's no
> actual release, people won't rush to push fixes in later -rc cycles.

And then what's the purpose of these later -rc cycles if you remove one
release ? You're just removing one step and shifting everything down by
one -rc but the issues are the same.

> We take away the incentive to push poorly tested code. Maintainers still
> free to commit anything they'd like, but there's no reason to commit
> code they're not confident of just to make it to a random release no one
> will use.
> 
> Merge window will happen more often, so there's no real reason to rush
> things in a particular window, and since -stable releases every week
> there's no rush to push a fix in since the next release is just a week
> away.

I'm not sure what model you're having in mind but the description above
reminds me of 2.5 which was constantly had something broken and which
used to be unusable for many developers. Many of us even bought some
SCSI cards and disks by then because for a long time IDE was broken.

The primary purpose of Linus' releases and -rc is to synchronise everyone
on the same goal at the same time. The merge window is "send me your crap,
it must be OK but we know problems happen and you'll be allowed to fix it
later". The -rc ones are there so that everyone fixes their crap in
parallel so that we converge towards something acceptable for everyone.

Your argument that the .0 release is useless is wrong in my opinion. It
is as wrong as saying "statistics show that less people use .3 than .7".
And comparing "stable kernels" to ".0" is wrong because there are roughly
10 times more stable kernels than releases so statistically you'll find
10 times more of them in field. The reality is that deploying .0 always
takes a bit more time for end users so statistically it should be a bit
less common in field :
  - you're never certain when the new version is going to be released
    (will rc8/rc9 exist?)
  - when it's released, you have to update your config and it takes
    some time.
  - by the time you find a quiet moment to do all this, it's not
    unlikely that the end of the week is reached with .1 appearing.

And so what ? The .0 release is a stable release like any other one.
It doesn't deserve to be deployed more than any other specific stable
release. It serves as a reference. Before .0 the code experiences some
possibly breaking changes, even some reverts. After .0 it experiences
only small fixes according to the stable rules.

Overall I think the current model is not that bad, and that what is the
most needed is some education regarding how -stable works to encourage
developers to rush their fixes less (after more tests), and to ensure
that those who generally push good quality fixes can submit them at any
moment in the cycle so that we get them as fast as possible in -stable.

Willy

Re: [Ksummit-discuss] bug-introducing patches

[Justin Forbes]

On Mon, May 7, 2018 at 9:34 PM, Sasha Levin
<Alexander.Levin@microsoft.com> wrote:
> On Thu, May 03, 2018 at 04:09:05PM -0700, Tony Lindgren wrote:
>>* Mark Brown <broonie@kernel.org> [180503 22:44]:
>>> On Wed, May 02, 2018 at 08:52:29PM -0700, Guenter Roeck wrote:
>>>
>>> > As for -next, me and others stopped reporting bugs in it, because when we do
>>> > we tend to get flamed for the "noise". Is anyone aware (or cares) that mips
>>> > and nds32 images don't build ? Soaking clothes in an empty bathtub won't make
>>> > them wet, and bugs in code which no one builds, much less tests or uses, won't
>>> > be found.
>>>
>>> You've been flamed for testing -next?  That's not been my experience and
>>> frankly it's pretty horrifying that it's happening.  Testing is pretty
>>> much the whole point of -next existing in the first place so you have to
>>> wonder why people are putting their trees there if they don't want
>>> testing.  I have seen a few issues with people reporting bugs on old
>>> versions of -next but otherwise...
>>
>>Yes I agree testing Linux next is very important. That's the best way for
>>maintainers to ensure a usable -rc1 after a merge window. And then for
>>the -rc cycle, there not much of need for chasing bugs to get things working.
>>
>>Bugs reported for Linux next often seem to get fixed or reverted faster
>>compared to the -rc cycle too. I think that's because people realize that
>>their code will not get merged until it's been fixed.
>>
>>So some daily testing of Linux next can save a lot scrambling after the
>>merge window :)
>>
>>Users don't usually upgrade kernels until after later -rc releases or only
>>after major releases so that probably explains some of the -rc cycle fixes.
>
> Tony, I'm curious, how many users are you aware of who actually run
> Linus's tree? All the users I've encountered so far on Azure seem to be
> running something based on -stable.

I couldn't tell you the number of users we have running rawhide
kernels (daily builds of Linus's tree), but it is a positive integer.
We do get bug reports on things, sometimes a day after Linus commits
them.

>
> I can't really get any solid statistics about that on my end both
> because I don't have visibility inside user VMs (I don't actually have
> prod access believe it or not), and even if I had it would probably be
> confidential, so I'm just basing this on reports from user's I've seen
> so far.
>
> I think that a question we should be asking ourselves is whether we
> should be basing our decisions here on the assumption that (pretty much)
> no one runs Linus's tree anymore?

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin via Ksummit-discuss]

On Fri, May 04, 2018 at 07:42:17AM +0900, Mark Brown wrote:
>On Wed, May 02, 2018 at 08:52:29PM -0700, Guenter Roeck wrote:
>
>> As for -next, me and others stopped reporting bugs in it, because when we do
>> we tend to get flamed for the "noise". Is anyone aware (or cares) that mips
>> and nds32 images don't build ? Soaking clothes in an empty bathtub won't make
>> them wet, and bugs in code which no one builds, much less tests or uses, won't
>> be found.
>
>You've been flamed for testing -next?  That's not been my experience and
>frankly it's pretty horrifying that it's happening.  Testing is pretty
>much the whole point of -next existing in the first place so you have to
>wonder why people are putting their trees there if they don't want
>testing.  I have seen a few issues with people reporting bugs on old
>versions of -next but otherwise...

This is just wrong, what else is -next for?

FWIW, our (MSFT) testing folks should now be reporting issues they see
on our -next testing pipeline directly to LKML. There's not much volume
there given that the 0-day bot catches most of the issues anyways, but
we sometimes see odd regressions given that no one else seems to test
Linux on Hyper-V but us :)
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: bug-introducing patches

[Sasha Levin]

On Tue, May 01, 2018 at 04:54:48PM -0400, Theodore Y. Ts'o wrote:
>On Tue, May 01, 2018 at 08:00:21PM +0000, Sasha Levin wrote:
>>
>> Yes, linux-next users want it fixed *now* and I completely agree it
>> should be done that way, but the fix should not be immediately pushed to
>> Linus as well.
>
>I should have linux-head/linux-rc said testers, sorry.  The fact that
>we have very few live users testing linux-next is a separate problem,
>which I accidentally conflated.  But if a user who is testing -rc2
>finds a problem, it is highly desirable to send a fix for -rc3,
>instead of making that user wait to -rc4 or -rc5.  And *that* is more
>important than AUTOSEL.
>
>> I've just finished reading an interesting article on LWN about the
>> PostgreSQL fsync issues (https://lwn.net/Articles/752952/). If you
>> look at Willy's commit, he wrote the final version of it about 5 days
>> ago, Jeff merged it in 3 days ago, and Linus merged it in the tree
>> today. Did it spend any time getting -next testing? nope.
>
>I agree that having the errseq patch go straight into Linus's tree is
>certainly unfortunate.  The justification was this was a regression
>fix, which I don't think it qualifies, since errseq_t went in some 9+
>months ago.
>
>It might be a good thing to quantify whether the patches you are
>talking about are new features, bug fixes, or fixing a bug that was
>introduced during the merge window or subsequently (e.g., a
>regression).

I see. So something like the following?

 - New feature: 2+ weeks of -next without any code changes/fixes
 - Merge window regression fix: immediate if < -rc3, 2+ weeks of next if
   < -rc6, otherwise consider reverting new feature.
 - bug fix in earlier release: 2+ weeks of -next

>> What's worse is that that commit is tagged for stable, which means
>> that (given Greg's schedule) it may find it's way to -stable users
>> even before some -next users/bots had a chance to test it out.
>
>Well, it used to be that things tagged for stable most-merge window
>are *supposed* to marinate for at least a week or before they would
>get pulled into a stable release.  Part of the whole problem is that
>people are wanting to be a lot more aggressive (both in time and
>volume) in shovelling things into stable.
>
>> This is less about AUTOSEL, and more about asking maintainers to
>> improve the testing commits get before they are sent to Linus.
>> Linus would rant about commits during merge window that didn't go
>> through -next, but for -rc commits this rule is somehow forgiven,
>> which is what I'm trying to change.
>
>I do think it's about AUTOSEL, because when I'm dealing with a
>regression, I want to get it fixed fast.  Because the alternative is
>the merge-window commit getting reverted.  AUTOSEL seems wants perfect
>patches that it can cherry pick once, as opposed to a case where if the
>user confirms that it fixes the regression, I want to send it to Linus
>quickly.  I do *not* want it to marinate in linux-next for 1-2 weeks.
>I would much rather that *stable* hold off on picking up the patch for
>1-2 weeks, but get it fixed in Linux HEAD sooner.  If that means that
>the regression fix needs a further clean up, so be it.

For AUTOSEL, most of the commits that went in so far were from the
v4.9..v4.14 range. Only last week I've sent greg commits picked from
v4.15..v4.16. AUTOSEL is at least a month behind -stable (on average,
9.7 months).

>Post -rc3 or -rc4, in my opinion bug fixes should wait until the next
>merge window before they get merged at all.  (And certainly features
>bugs should be Right Out.)  And sure, bug fixes should certainly get
>more testing.  So I guess my main objection is your making a blanket
>statement about all fixes, instead of breaking out regression fixes
>versus bug fixes.  Since in my opinion they are very different animals...

I understant your point, you want to make fixes available to testers as
soon as possible. This might make sense, as you've mentioned, in < -rc3.

So yes, maybe the solution isn't to force -next, but rather add more
"quiet time" at the end of the cycle? Make special rules for -rc7/8? Or
even add a "test"/"beta" release at the end of the cycle?

>From what I see, the same number of bugs-per-line-of-code applies for
commits accross all -rc releases, so while it makes sense to get a fix
in quickly at -rc1 to allow testing to continue, the same must not
happen during -rc8, but unfourtenately it does now.

Re: bug-introducing patches

[Willy Tarreau]

On Tue, May 01, 2018 at 10:02:30PM +0000, Sasha Levin wrote:
> On Tue, May 01, 2018 at 04:54:48PM -0400, Theodore Y. Ts'o wrote:
> >Post -rc3 or -rc4, in my opinion bug fixes should wait until the next
> >merge window before they get merged at all.  (And certainly features
> >bugs should be Right Out.)  And sure, bug fixes should certainly get
> >more testing.  So I guess my main objection is your making a blanket
> >statement about all fixes, instead of breaking out regression fixes
> >versus bug fixes.  Since in my opinion they are very different animals...
> 
> I understant your point, you want to make fixes available to testers as
> soon as possible. This might make sense, as you've mentioned, in < -rc3.
> 
> So yes, maybe the solution isn't to force -next, but rather add more
> "quiet time" at the end of the cycle? Make special rules for -rc7/8? Or
> even add a "test"/"beta" release at the end of the cycle?

I disagree with the proposals above, and for multiple reasons :
  - leaving a known bug on purpose automatically degrades the quality of
    each release. Given that less than 100% of the fixes introduce
    regressions, by not merging any of these fixes, we'll end up with
    more bugs. That's a very bad idea.

  - this will give a worse image of dot-0 releases, and users will be
    even less interested in testing them, prefering to wait for the
    first stable version. In this case what's the point of dot-0 if it
    is known broken and nobody uses it ?

  - letting fixes rot longer on the developer side will send a very bad
    signal to developers : "we don't care about your bugs". Companies
    relying on contractors will have a harder time including fixes in
    the contract, as it will only cover what's needed to get the feature
    merged. Again this will put the focus on extremely fast and dirty
    development, given that fixes will not be considered during the same
    window.

I'd rather do the exact opposite except for those who introduce too many
regressions : set up a delay penalty to developers who create too many
regressions and make this penalty easy to check. I think it will generally
not affect subsystem maintainers, unless they pull and push lots of crap
without checking, of course. But it could prove very useful for those
developing under contract, because companies employing them will want to
ensure that their work will not be delayed due to a penalty. Thus is will
be important for these developers to be more careful.

After all, the person proposing a fix always knows better than anyone
else if this fix was done seriously or not. Developers who do lots of
testing before sending should not be penalized, and should get their
fix merged immediately. Those who just send untested patches should be
trusted much less.

> From what I see, the same number of bugs-per-line-of-code applies for
> commits accross all -rc releases, so while it makes sense to get a fix
> in quickly at -rc1 to allow testing to continue, the same must not
> happen during -rc8, but unfourtenately it does now.

That's where I strongly disagree, since it would mean releasing with even
more bugs than today.

Willy

Re: bug-introducing patches

[Sasha Levin]

On Wed, May 02, 2018 at 06:30:17AM +0200, Willy Tarreau wrote:
>On Tue, May 01, 2018 at 10:02:30PM +0000, Sasha Levin wrote:
>> On Tue, May 01, 2018 at 04:54:48PM -0400, Theodore Y. Ts'o wrote:
>> >Post -rc3 or -rc4, in my opinion bug fixes should wait until the next
>> >merge window before they get merged at all.  (And certainly features
>> >bugs should be Right Out.)  And sure, bug fixes should certainly get
>> >more testing.  So I guess my main objection is your making a blanket
>> >statement about all fixes, instead of breaking out regression fixes
>> >versus bug fixes.  Since in my opinion they are very different animals...
>>
>> I understant your point, you want to make fixes available to testers as
>> soon as possible. This might make sense, as you've mentioned, in < -rc3.
>>
>> So yes, maybe the solution isn't to force -next, but rather add more
>> "quiet time" at the end of the cycle? Make special rules for -rc7/8? Or
>> even add a "test"/"beta" release at the end of the cycle?
>
>I disagree with the proposals above, and for multiple reasons :
>  - leaving a known bug on purpose automatically degrades the quality of
>    each release. Given that less than 100% of the fixes introduce
>    regressions, by not merging any of these fixes, we'll end up with
>    more bugs. That's a very bad idea.
>
>  - this will give a worse image of dot-0 releases, and users will be
>    even less interested in testing them, prefering to wait for the
>    first stable version. In this case what's the point of dot-0 if it
>    is known broken and nobody uses it ?
>
>  - letting fixes rot longer on the developer side will send a very bad
>    signal to developers : "we don't care about your bugs". Companies
>    relying on contractors will have a harder time including fixes in
>    the contract, as it will only cover what's needed to get the feature
>    merged. Again this will put the focus on extremely fast and dirty
>    development, given that fixes will not be considered during the same
>    window.

I'm not advocating to keep bugs in. If there is a fix, but the developer
can't indicate that proper testing was done on the fix we should revert
the new feature rather than merge the untested fix in.

The way I see it, if a commit can get one or two tested-by, it's a good
alternative to a week in -next.

>I'd rather do the exact opposite except for those who introduce too many
>regressions : set up a delay penalty to developers who create too many
>regressions and make this penalty easy to check. I think it will generally
>not affect subsystem maintainers, unless they pull and push lots of crap
>without checking, of course. But it could prove very useful for those
>developing under contract, because companies employing them will want to
>ensure that their work will not be delayed due to a penalty. Thus is will
>be important for these developers to be more careful.
>
>After all, the person proposing a fix always knows better than anyone
>else if this fix was done seriously or not. Developers who do lots of
>testing before sending should not be penalized, and should get their
>fix merged immediately. Those who just send untested patches should be
>trusted much less.

I'm a bit worried about (social) side effects of a scheme like this.

>> From what I see, the same number of bugs-per-line-of-code applies for
>> commits accross all -rc releases, so while it makes sense to get a fix
>> in quickly at -rc1 to allow testing to continue, the same must not
>> happen during -rc8, but unfourtenately it does now.
>
>That's where I strongly disagree, since it would mean releasing with even
>more bugs than today.

Just don't release it. If we don't have a tested fix for a reported
regression either extend the release cycle (-rc10+) or just revert the
new feature and get it in the next merge window.

Re: bug-introducing patches

[Willy Tarreau]

On Wed, May 02, 2018 at 07:42:33PM +0000, Sasha Levin wrote:
> I'm not advocating to keep bugs in. If there is a fix, but the developer
> can't indicate that proper testing was done on the fix we should revert
> the new feature rather than merge the untested fix in.

If you're exclusively talking about newly merged features, I agree. But
I think that the initial point was not just about newly merged features.
Sometimes it will not work because other changes rely on this new feature
but the way I see it is that this kind of back-pressure should work well
enough to encourage developers to show they have valid reasons to trust
their fix.

> The way I see it, if a commit can get one or two tested-by, it's a good
> alternative to a week in -next.

Agreed. Even their own actually. And I'm not kidding. Those who run large
amounts of tests on certain patches could really mention is in tested-by,
as opposed to the most common cases where the code was just regularly
tested.

> >After all, the person proposing a fix always knows better than anyone
> >else if this fix was done seriously or not. Developers who do lots of
> >testing before sending should not be penalized, and should get their
> >fix merged immediately. Those who just send untested patches should be
> >trusted much less.
> 
> I'm a bit worried about (social) side effects of a scheme like this.

Me as well, because it's still a bit early for this, people might not
be prepared to this yet. But if it were at least discussed and presented
as one of the possibilities for the long term, newcomers would arrive
here with this possibility in mind and would possibly join in better
conditions and possibly that ultimately this solution would only exist
as a threat against bad players but would never be used. Also there are
more and more places where people find it normal to be noted by others,
maybe it will really end up like this over the long term, who knows. At
the very least a first note for a contractor is "I contributed X commits
last year, my work never got reverted for bad quality".

> >> From what I see, the same number of bugs-per-line-of-code applies for
> >> commits accross all -rc releases, so while it makes sense to get a fix
> >> in quickly at -rc1 to allow testing to continue, the same must not
> >> happen during -rc8, but unfourtenately it does now.
> >
> >That's where I strongly disagree, since it would mean releasing with even
> >more bugs than today.
> 
> Just don't release it. If we don't have a tested fix for a reported
> regression either extend the release cycle (-rc10+) or just revert the
> new feature and get it in the next merge window.

I agree in general, but the reality will often be different (think about
contractors for a limited time as I suggested). It should be considered
as a penalty against improper testing so that we don't even have to reach
this situation.

Willy

Re: bug-introducing patches

[Pavel Machek]

Hi!

> > The way I see it, if a commit can get one or two tested-by, it's a good
> > alternative to a week in -next.
> 
> Agreed. Even their own actually. And I'm not kidding. Those who run large
> amounts of tests on certain patches could really mention is in tested-by,
> as opposed to the most common cases where the code was just regularly
> tested.

Actually, it would be cool to get "Tested: no" and "Tested: compile"
tags in the commit mesages. Sometimes it is clear from the context
that patch was not tested (treewide update of time to 64bit), but
sometime it is not.

This is especially problem for -stable, as it seems that lately
patches are backported from new version without any testing.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [Ksummit-discuss] bug-introducing patches

[Guenter Roeck]

On 07/14/2018 10:38 AM, Pavel Machek wrote:
> Hi!
> 
>>> The way I see it, if a commit can get one or two tested-by, it's a good
>>> alternative to a week in -next.
>>
>> Agreed. Even their own actually. And I'm not kidding. Those who run large
>> amounts of tests on certain patches could really mention is in tested-by,
>> as opposed to the most common cases where the code was just regularly
>> tested.
> 
> Actually, it would be cool to get "Tested: no" and "Tested: compile"
> tags in the commit mesages. Sometimes it is clear from the context
> that patch was not tested (treewide update of time to 64bit), but
> sometime it is not.
> 
> This is especially problem for -stable, as it seems that lately
> patches are backported from new version without any testing.


When I started my own testing some five years ago, most architectures
did not even build in stable releases. At that time, the only tests being
done on stable release candidates were a number of build tests, and most
of the results were ignored.

Today, we have 0day, kernelci, kerneltests, Linaro's LKFT, and more, plus
several merge and boot tests done by individuals. Stable release candidates
are build tested on all supported architectures, with hundreds of configurations.
Each stable release candidate is boot tested on qemu with more than 150
configurations on each architecture supported by qemu. A substantial amount of
boot tests run on real hardware. On key architectures, more sophisticated tests
such as kerneltests and LTP ensure that no new regressions are introduced.

What is new is that many more patches are being applied and backported
to stable releases, at least to degree due to Sasha's scripts, but also due
to tools like syzbot running on older kernels and finding problems which
have been fixed upstream, but the fix has not been backported.

At the same time, stable release test coverage has been improved substantially
over the last few years. I am _much_ more confident with stable releases
than I used to be a few of years ago.

Sure, there are regressions. However, the regression rate is very low (last
time I checked it was around 0.1% to 0.3% per stable release for us). Sure,
I would like to further reduce regression rate, to improve stability but also
because each and every regression is used by someone to argue that stable releases
would be unreliable. However, this is more a matter of perception than reality.
Reality is that more than 90% of all CVEs are fixed in stable releases by the time
they are published as affecting a stable release. Reality is that a substantial
percentage of problems reported by syzbot _are_ being fixed in stable releases.
Reality is that, by the time bugs are reported from the field, it is more and
more likely that we find out that the bug has already been fixed in a later
release due to a stable release merge.

Given all that, I think it is quite misleading to claim that the number of
patches applied to stable releases would create additional problems, or that
patches would be applied "without any testing". On the contrary, I would argue
that the additional testing now being performed _enabled_ us to apply more
patches (bug fixes) to stable releases.

Sure, testing is still far from perfect and needs to be improved. However,
requiring that every patch applied to stable releases be tested individually
(where ? on all affected architectures ?) would be the wrong direction.
What we need to do is to further improve test coverage. We should have no
regressions, but we need to get there by improving test coverage, not by
demanding explicit per-patch and per-release testing (which would be all
but impossible anyway - no one has the infrastructure necessary to test
a patch on all affected architectures).

I would encourage everyone interested in kernel testing to attend the
kernel test sessions at Linux Plumbers and ELC later this year to discuss
concerns and possible solutions.

Thanks,
Guenter

Re: [Ksummit-discuss] bug-introducing patches

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 2161 bytes --]

Hi!

> >>>The way I see it, if a commit can get one or two tested-by, it's a good
> >>>alternative to a week in -next.
> >>
> >>Agreed. Even their own actually. And I'm not kidding. Those who run large
> >>amounts of tests on certain patches could really mention is in tested-by,
> >>as opposed to the most common cases where the code was just regularly
> >>tested.
> >
> >Actually, it would be cool to get "Tested: no" and "Tested: compile"
> >tags in the commit mesages. Sometimes it is clear from the context
> >that patch was not tested (treewide update of time to 64bit), but
> >sometime it is not.
> >
> >This is especially problem for -stable, as it seems that lately
> >patches are backported from new version without any testing.
> 
> 
> When I started my own testing some five years ago, most architectures
> did not even build in stable releases. At that time, the only tests being
> done on stable release candidates were a number of build tests, and most
> of the results were ignored.
> 
> Today, we have 0day, kernelci, kerneltests, Linaro's LKFT, and more, plus
> several merge and boot tests done by individuals. Stable release candidates
> are build tested on all supported architectures, with hundreds of
...
> Sure, testing is still far from perfect and needs to be improved. However,
> requiring that every patch applied to stable releases be tested individually
> (where ? on all affected architectures ?) would be the wrong
>direction.

Well, 0day, kernelci etc... is nice... until the change is in the
driver. Most of the kernel are drivers, remember?

I don't know. I'd say that if patch is important enough for -stable,
there should be someone testing it. For core kernel changes, that can
be 0day bot, but for drivers...

And problem exists on mainline, too.

Hmm. Patch for obscure driver. Wow, nice, is WellKnownName actually
using that driver? Aha, no, he is not; he is doing global
search&replace, and did not test the patch...

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [Ksummit-discuss] bug-introducing patches

[Guenter Roeck]

On 07/14/2018 12:47 PM, Pavel Machek wrote:
> Hi!
> 
>>>>> The way I see it, if a commit can get one or two tested-by, it's a good
>>>>> alternative to a week in -next.
>>>>
>>>> Agreed. Even their own actually. And I'm not kidding. Those who run large
>>>> amounts of tests on certain patches could really mention is in tested-by,
>>>> as opposed to the most common cases where the code was just regularly
>>>> tested.
>>>
>>> Actually, it would be cool to get "Tested: no" and "Tested: compile"
>>> tags in the commit mesages. Sometimes it is clear from the context
>>> that patch was not tested (treewide update of time to 64bit), but
>>> sometime it is not.
>>>
>>> This is especially problem for -stable, as it seems that lately
>>> patches are backported from new version without any testing.
>>
>>
>> When I started my own testing some five years ago, most architectures
>> did not even build in stable releases. At that time, the only tests being
>> done on stable release candidates were a number of build tests, and most
>> of the results were ignored.
>>
>> Today, we have 0day, kernelci, kerneltests, Linaro's LKFT, and more, plus
>> several merge and boot tests done by individuals. Stable release candidates
>> are build tested on all supported architectures, with hundreds of
> ...
>> Sure, testing is still far from perfect and needs to be improved. However,
>> requiring that every patch applied to stable releases be tested individually
>> (where ? on all affected architectures ?) would be the wrong
>> direction.
> 
> Well, 0day, kernelci etc... is nice... until the change is in the
> driver. Most of the kernel are drivers, remember?
> 
> I don't know. I'd say that if patch is important enough for -stable,
> there should be someone testing it. For core kernel changes, that can
> be 0day bot, but for drivers...
> 

For my part I am just glad that we were able to pick up a fix in xhci code
just last week, tested or not, from -stable, instead of having to track it
down ourselves. Similar for many other driver patches which _do_ affect us
(like the flurry of ext4 patches this week). Granted, there are lots of
patches which we don't use/need, but even there it is surprising how many
problems are found with existing testing.

For anyone interested in making sure that obscure (whatever that means)
drivers are tested for stable releases, but does not want to spend time on it,
all I can recommend is to implement qemu support for it and let me know,
and I'll be happy to add a respective test to my test farm.

However, ultimately, stable release candidates are public. Everyone is
invited to test them. Anyone interested in a specific release and driver
is invited take stable release candidates and do the necessary testing,
just like I and several others do.

> And problem exists on mainline, too.
> 
> Hmm. Patch for obscure driver. Wow, nice, is WellKnownName actually
> using that driver? Aha, no, he is not; he is doing global
> search&replace, and did not test the patch...
> 

Ah, like me with the strncpy(x, y, strlen(y)) -> memcpy() replacements
I did a week or so ago ? You are right, I only compile tested those and
otherwise trusted my ability to understand C code. If that caused any
problems, please let me know, and hopefully I'll be able to learn something
from it.

Really, there are infrastructure changes all the time. Sometimes I am asked
to run a complete test sequence with those, but most of the time they are
applied to -next and people wait for the fallout. That may not be perfect but,
seriously, the only alternative would be to declare that in-kernel APIs
shall not be changed anymore. I don't think that would be feasible.

Thanks,
Guenter

Re: [Ksummit-discuss] bug-introducing patches

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 2876 bytes --]

Hi!

> >Well, 0day, kernelci etc... is nice... until the change is in the
> >driver. Most of the kernel are drivers, remember?
> >
> >I don't know. I'd say that if patch is important enough for -stable,
> >there should be someone testing it. For core kernel changes, that can
> >be 0day bot, but for drivers...
> >
> 
> For my part I am just glad that we were able to pick up a fix in xhci code
> just last week, tested or not, from -stable, instead of having to track it
> down ourselves. Similar for many other driver patches which _do_ affect us
> (like the flurry of ext4 patches this week). Granted, there are lots of
> patches which we don't use/need, but even there it is surprising how many
> problems are found with existing testing.
> 
> For anyone interested in making sure that obscure (whatever that means)
> drivers are tested for stable releases, but does not want to spend time on it,
> all I can recommend is to implement qemu support for it and let me know,
> and I'll be happy to add a respective test to my test farm.

Umm. Yes, qemu support for every driver would be nice, but will not happen.

> However, ultimately, stable release candidates are public. Everyone is
> invited to test them. Anyone interested in a specific release and
> driver

Yes, they are public. SubmittingPatches says every patch should be
tested, and that's clearly not happening for -stable. And I'd like
those patch marked such.

> >And problem exists on mainline, too.
> >
> >Hmm. Patch for obscure driver. Wow, nice, is WellKnownName actually
> >using that driver? Aha, no, he is not; he is doing global
> >search&replace, and did not test the patch...
> >
> 
> Ah, like me with the strncpy(x, y, strlen(y)) -> memcpy() replacements
> I did a week or so ago ? You are right, I only compile tested those and
> otherwise trusted my ability to understand C code. If that caused any
> problems, please let me know, and hopefully I'll be able to learn something
> from it.

Yes, such stuff. No, I was not talking about you. I did not want to
give concrete example, but...

# > get_monotonic_boottime() is deprecated, so let's convert this to
# > the simpler ktime_get_boot_ns().
# >
# > Signed-off-by: <redacted>
# 
# Have you tested it?
#
...
#   > -	   curr_boot = timespec_to_ns(&boot_time) * cpus;
# 
# Original code is pretty weird (notice the * cpus), so I'm
# double-checking.

Yes, often you can guess that patch was probably not tested, but it
would be nice to have

Tested: compile

annotation to take away the guesswork. It took me a while an some head
scratching in this concrete example, and it is not first time this
happened.

Best regards,
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [Ksummit-discuss] bug-introducing patches

[Willy Tarreau]

On Sat, Jul 14, 2018 at 11:09:13PM +0200, Pavel Machek wrote:
> > For anyone interested in making sure that obscure (whatever that means)
> > drivers are tested for stable releases, but does not want to spend time on it,
> > all I can recommend is to implement qemu support for it and let me know,
> > and I'll be happy to add a respective test to my test farm.
> 
> Umm. Yes, qemu support for every driver would be nice, but will not happen.

Well, I would argue that driver code changes much less than core code
between kernel versions, and that most of the changes in drivers are
mostly infrastructure changes. Drivers don't evolve much in general,
they are written, tested, merged, they receive fixes and then they
only receive infrastructure changes that touch all drivers in the same
category.

When you backport fixes to drivers, it is very common that the code
looks almost the same between even a very old kernel and mainline, and
when not, the adaptations generally look quite straightforward, and if
not it means the driver changed significantly and in this case we don't
backport the fix as we don't even know if it is relevant.

I've always had much more difficulties backporting fixes under the arch/
subdir where stuff changes all the time. Sometimes a patch applies but
doesn't even compile. I learned not to play black magic in this area
because some patches are subtle and if the code changed you often need
the author and/or maintainers to double-check. Some subsystems like KVM
improve a lot over time and are difficult to backport to as well, and
even if you manage to properly backport a fix you're uncertain how to
verify you backported it well. Similarly you don't want to improvise
yourself the backporter of the year in this area.

Drivers are often OK and are the ones harder to test, so in the end
you don't miss much by your limited ability to test a backport there.

What I can certainly say as a stable kernel user is that the regression
rate is so low compared to the fix rate that I never have any problem
upgrading to a more recent version in the same branch, because the
number of problems that will be fixed is much higher than the risk of
a single regression.

As Guenter says, we can always improve, but the most important is to
deliver fixes in a timely manner. When you see that any LTS branch
accumulates around 5000 fixes over time, you understand that any
single new kernel being released contains around 5000 bugs left to be
found. Fixing them quickly is much more important to me (as a user)
than ensuring that I will not reach 5001 by inheriting from a poorly
tested backport.

My hope is that thanks to all the automated testing in place we can
further accelerate the backport rate so that a stable kernel reaches
in 2 months the level of quality that we previously used to reach only
after one year. And I think we're already about there, as both 4.4.x
and 4.9.x in their early versions (x < 10) were already very good for
various use cases. 4.17.5 I'm using on this PC looks pretty slick as
well. Overall it means that we can provide a clean upgrade path for
users so that they don't stick to bogus or insecure kernels by fear
of upgrading. We can always argue that a bug may appear once in a
while but for me while technically this is true, stastistically this
is just FUD and is not relevant to end users' real usage.

Willy

Re: bug-introducing patches

[Greg KH]

On Sat, Jul 14, 2018 at 07:38:12PM +0200, Pavel Machek wrote:
> Hi!
> 
> > > The way I see it, if a commit can get one or two tested-by, it's a good
> > > alternative to a week in -next.
> > 

Pavel, I "love" how you fail to point out that you are responding to a 2
month old thread :(

And that thread was beaten to death, and still you want to revise it,
which is odd to me, perhaps you just don't like stable releases?  Given
that you never mark any of the patches for your subsystem for stable
releases, why do you care about how they are maintained?

> > Agreed. Even their own actually. And I'm not kidding. Those who run large
> > amounts of tests on certain patches could really mention is in tested-by,
> > as opposed to the most common cases where the code was just regularly
> > tested.
> 
> Actually, it would be cool to get "Tested: no" and "Tested: compile"
> tags in the commit mesages. Sometimes it is clear from the context
> that patch was not tested (treewide update of time to 64bit), but
> sometime it is not.
> 
> This is especially problem for -stable, as it seems that lately
> patches are backported from new version without any testing.

As everyone has pointed out numerous times in this thread, there are
more testing of stable patches and releases than _EVER_ before in the
history of stable kernels.  And if you feel there are ways to do more
testing that we somehow are missing, wonderful, please provide
constructive criticism.  If not, and you just want to complain, well, my
killfile can always use a new member...

And as always, you have a choice:
	- if you don't like stable kernels, don't run them.

greg k-h

Re: bug-introducing patches

[Theodore Y. Ts'o]

On Sun, Jul 15, 2018 at 10:54:03AM +0200, Greg KH wrote:
> Pavel, I "love" how you fail to point out that you are responding to a 2
> month old thread :(

And apologies for releasing some ancient messages that were caught in
the ksummit-discuss's moderation queue.  I hadn't been paying
attention to the fact that there had been some messages caught there,
and I figured it might be some people on the ksummit-discuss list who
might have missed the context of that old thread.

That being said, it *was* beaten to death two months ago, so people
who are replying might want to keep that in mind.  :-)

> And as always, you have a choice:
> 	- if you don't like stable kernels, don't run them.

And if you don't like stable kernels, you can pay $$$ to an enterprise
distro kernel.  Although you might find they aren't that much better
at the whole fixing bugs versus introducing regressions tradeoff.
Indeed, because there are crazy people who insist on using 3.18
kernels *and* getting support for the latest hardware, you may find
that it's somewhat worse; not to mention not necessarily being able to
get all of the fixes for the cache-related security problems getting
found...

						- Ted

Re: bug-introducing patches

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 1344 bytes --]

On Sun 2018-07-15 10:54:03, Greg KH wrote:
> On Sat, Jul 14, 2018 at 07:38:12PM +0200, Pavel Machek wrote:
> > Hi!
> > 
> > > > The way I see it, if a commit can get one or two tested-by, it's a good
> > > > alternative to a week in -next.
> > > 
> 
> Pavel, I "love" how you fail to point out that you are responding to a 2
> month old thread :(
> 
> And that thread was beaten to death, and still you want to revise it,
> which is odd to me, perhaps you just don't like stable releases?  Given
> that you never mark any of the patches for your subsystem for stable
> releases, why do you care about how they are maintained?

I do mark patches -- acording to stable kernel rules. But you are
apparently using different rules, not written anywhere, and I get
complains when I don't mark patches according to _those_.

But this was supposed to be about testing.

And I'd like to see Tested: no/compile headers, instead.

[And yes, motivation for this was that broken LED patches were merged
to stable without any testing, and when that was questioned, I was
told that testing was not performed because it would require unusual
hardware called "USB keyboard".]

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [Ksummit-discuss] bug-introducing patches

[Jani Nikula]

On Tue, 01 May 2018, "Theodore Y. Ts'o" <tytso@mit.edu> wrote:
> Post -rc3 or -rc4, in my opinion bug fixes should wait until the next
> merge window before they get merged at all.

What are -rc5 and later for then if not bug fixes? Baffled.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center

Re: [Ksummit-discuss] bug-introducing patches

[James Bottomley]

On Thu, 2018-05-03 at 14:08 +0300, Jani Nikula wrote:
> On Tue, 01 May 2018, "Theodore Y. Ts'o" <tytso@mit.edu> wrote:
> > Post -rc3 or -rc4, in my opinion bug fixes should wait until the
> > next
> > merge window before they get merged at all.
> 
> What are -rc5 and later for then if not bug fixes? Baffled.

They're definitely for bug fixes, but there's a spectrum: obvious bug
fixes with no side effects are easy to justify.  More complex bug fixes
run the risk of having side effects which introduce other bugs, so
could potentially destabilize the -rc process.  In SCSI we tend to look
at what the user visible effects of the bug are in the post -rc5 region
and if they're slight or wouldn't be visible to most users, we'll hold
them over.  If the fix looks complex and we're not sure we caught the
ramifications, we often add it to the merge window tree with a cc to
stable and a note saying to wait X weeks before actually adding to the
stable tree just to make sure no side effects show up with wider
testing.  So, as with most things, it's a judgment call for the
maintainer.

James

_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Willy Tarreau]

On Thu, May 03, 2018 at 07:33:04AM -0700, James Bottomley wrote:
> They're definitely for bug fixes, but there's a spectrum: obvious bug
> fixes with no side effects are easy to justify.  More complex bug fixes
> run the risk of having side effects which introduce other bugs, so
> could potentially destabilize the -rc process.  In SCSI we tend to look
> at what the user visible effects of the bug are in the post -rc5 region
> and if they're slight or wouldn't be visible to most users, we'll hold
> them over.  If the fix looks complex and we're not sure we caught the
> ramifications, we often add it to the merge window tree with a cc to
> stable and a note saying to wait X weeks before actually adding to the
> stable tree just to make sure no side effects show up with wider
> testing.  So, as with most things, it's a judgment call for the
> maintainer.

For me this is the right, and responsible way to deal with bug fixes.
Self-control is much more efficient than random rejection and favors
a good analysis.

Willy

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin via Ksummit-discuss]

On Thu, May 03, 2018 at 04:48:50PM +0200, Willy Tarreau wrote:
>On Thu, May 03, 2018 at 07:33:04AM -0700, James Bottomley wrote:
>> They're definitely for bug fixes, but there's a spectrum: obvious bug
>> fixes with no side effects are easy to justify.  More complex bug fixes
>> run the risk of having side effects which introduce other bugs, so
>> could potentially destabilize the -rc process.  In SCSI we tend to look
>> at what the user visible effects of the bug are in the post -rc5 region
>> and if they're slight or wouldn't be visible to most users, we'll hold
>> them over.  If the fix looks complex and we're not sure we caught the
>> ramifications, we often add it to the merge window tree with a cc to
>> stable and a note saying to wait X weeks before actually adding to the
>> stable tree just to make sure no side effects show up with wider
>> testing.  So, as with most things, it's a judgment call for the
>> maintainer.
>
>For me this is the right, and responsible way to deal with bug fixes.
>Self-control is much more efficient than random rejection and favors
>a good analysis.

I think that the ideal outcome of this discussion, at least for me, is a
tool to go under scripts/ that would allow maintainers to get some sort
of (quantifiable) data that will indicate how well the patch was tested via
the regular channels.

At which point it's the maintainer's judgement call on whether he wants
to grab the patch or wait for more tests or reviews.

This is very similar to what James has described, it just needs to leave
his brain and turn into code :)
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[James Bottomley]

On Thu, 2018-05-03 at 15:06 +0000, Sasha Levin via Ksummit-discuss
wrote:
> On Thu, May 03, 2018 at 04:48:50PM +0200, Willy Tarreau wrote:
> > On Thu, May 03, 2018 at 07:33:04AM -0700, James Bottomley wrote:
> > > They're definitely for bug fixes, but there's a spectrum: obvious
> > > bug fixes with no side effects are easy to justify.  More complex
> > > bug fixes run the risk of having side effects which introduce
> > > other bugs, so could potentially destabilize the -rc process.  In
> > > SCSI we tend to look at what the user visible effects of the bug
> > > are in the post -rc5 region and if they're slight or wouldn't be
> > > visible to most users, we'll hold them over.  If the fix looks
> > > complex and we're not sure we caught the ramifications, we often
> > > add it to the merge window tree with a cc to stable and a note
> > > saying to wait X weeks before actually adding to the
> > > stable tree just to make sure no side effects show up with wider
> > > testing.  So, as with most things, it's a judgment call for the
> > > maintainer.
> > 
> > For me this is the right, and responsible way to deal with bug
> > fixes. Self-control is much more efficient than random rejection
> > and favors a good analysis.
> 
> I think that the ideal outcome of this discussion, at least for me,
> is a tool to go under scripts/ that would allow maintainers to get
> some sort of (quantifiable) data that will indicate how well the
> patch was tested via the regular channels.
> 
> At which point it's the maintainer's judgement call on whether he
> wants to grab the patch or wait for more tests or reviews.
> 
> This is very similar to what James has described, it just needs to
> leave his brain and turn into code :)

I appreciate the sentiment, but if we could script taste, we'd have
replaced Linus with something far less cantankerous a long time ago ...

It's also a sad fact that a lot of things which look like obvious fixes
actually turn out not to be so with later testing.  This is why the
user visibility test is paramount.  If a bug fix has no real user
visible effects, it's often better to defer it no matter how obvious it
looks, which is why the static code checkers often get short shrift
before a merge window.

A script measuring user visibility would be nice, but looks a bit
complex ...

James

_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin via Ksummit-discuss]

On Thu, May 03, 2018 at 08:27:48AM -0700, James Bottomley wrote:
>On Thu, 2018-05-03 at 15:06 +0000, Sasha Levin via Ksummit-discuss
>wrote:
>> On Thu, May 03, 2018 at 04:48:50PM +0200, Willy Tarreau wrote:
>> > On Thu, May 03, 2018 at 07:33:04AM -0700, James Bottomley wrote:
>> > > They're definitely for bug fixes, but there's a spectrum: obvious
>> > > bug fixes with no side effects are easy to justify.  More complex
>> > > bug fixes run the risk of having side effects which introduce
>> > > other bugs, so could potentially destabilize the -rc process.  In
>> > > SCSI we tend to look at what the user visible effects of the bug
>> > > are in the post -rc5 region and if they're slight or wouldn't be
>> > > visible to most users, we'll hold them over.  If the fix looks
>> > > complex and we're not sure we caught the ramifications, we often
>> > > add it to the merge window tree with a cc to stable and a note
>> > > saying to wait X weeks before actually adding to the
>> > > stable tree just to make sure no side effects show up with wider
>> > > testing.  So, as with most things, it's a judgment call for the
>> > > maintainer.
>> >
>> > For me this is the right, and responsible way to deal with bug
>> > fixes. Self-control is much more efficient than random rejection
>> > and favors a good analysis.
>>
>> I think that the ideal outcome of this discussion, at least for me,
>> is a tool to go under scripts/ that would allow maintainers to get
>> some sort of (quantifiable) data that will indicate how well the
>> patch was tested via the regular channels.
>>
>> At which point it's the maintainer's judgement call on whether he
>> wants to grab the patch or wait for more tests or reviews.
>>
>> This is very similar to what James has described, it just needs to
>> leave his brain and turn into code :)
>
>I appreciate the sentiment, but if we could script taste, we'd have
>replaced Linus with something far less cantankerous a long time ago ...

Linus, IMO, is getting replaced. Look at how many functions he used to
do 10 years ago he's no longer responsible for.

One of the most obvious examples is -next, where most integration issues
are resolved before they even reach to Linus.

This is good for the community, as it allows us make the process better
and scale out. It is also good for Linus, as I'm not sure how long he'd
last if he still had to edit patches by hand too often. Instead, he gets
to play with things that interest him more where his is irreplaceable.

>It's also a sad fact that a lot of things which look like obvious fixes
>actually turn out not to be so with later testing.  This is why the
>user visibility test is paramount.  If a bug fix has no real user
>visible effects, it's often better to defer it no matter how obvious it
>looks, which is why the static code checkers often get short shrift
>before a merge window.
>
>A script measuring user visibility would be nice, but looks a bit
>complex ...

It is, but I think it's worthwhile. Would something that'll show you
things like:

 - How long a patch has been in -next?
 - How many replies/reviews/comments it got on a mailing list?
 - Did the 0day bot test it?
 - Did syzbot fuzz it? for how long?
 - If it references a bugzilla of some sort, how many
   comments/reviews/etc it got there?
 - Is it -stable material, or does it fix a regression in the current
   merge window?
 - If subsystem has custom testing rig, results from those tests

be a step in the right way? is it something you'd use to make decisions
on whether you'd take a patch in?
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

[Ksummit-discuss] bug-introducing patches

[Randy Dunlap]

On 05/03/2018 08:43 AM, Sasha Levin wrote:
> On Thu, May 03, 2018 at 08:27:48AM -0700, James Bottomley wrote:
>> On Thu, 2018-05-03 at 15:06 +0000, Sasha Levin via Ksummit-discuss
>> wrote:
>>> On Thu, May 03, 2018 at 04:48:50PM +0200, Willy Tarreau wrote:
>>>> On Thu, May 03, 2018 at 07:33:04AM -0700, James Bottomley wrote:
>>>>> They're definitely for bug fixes, but there's a spectrum: obvious
>>>>> bug fixes with no side effects are easy to justify.  More complex
>>>>> bug fixes run the risk of having side effects which introduce
>>>>> other bugs, so could potentially destabilize the -rc process.  In
>>>>> SCSI we tend to look at what the user visible effects of the bug
>>>>> are in the post -rc5 region and if they're slight or wouldn't be
>>>>> visible to most users, we'll hold them over.  If the fix looks
>>>>> complex and we're not sure we caught the ramifications, we often
>>>>> add it to the merge window tree with a cc to stable and a note
>>>>> saying to wait X weeks before actually adding to the
>>>>> stable tree just to make sure no side effects show up with wider
>>>>> testing.  So, as with most things, it's a judgment call for the
>>>>> maintainer.
>>>>
>>>> For me this is the right, and responsible way to deal with bug
>>>> fixes. Self-control is much more efficient than random rejection
>>>> and favors a good analysis.
>>>
>>> I think that the ideal outcome of this discussion, at least for me,
>>> is a tool to go under scripts/ that would allow maintainers to get
>>> some sort of (quantifiable) data that will indicate how well the
>>> patch was tested via the regular channels.
>>>
>>> At which point it's the maintainer's judgement call on whether he
>>> wants to grab the patch or wait for more tests or reviews.
>>>
>>> This is very similar to what James has described, it just needs to
>>> leave his brain and turn into code :)
>>
>> I appreciate the sentiment, but if we could script taste, we'd have
>> replaced Linus with something far less cantankerous a long time ago ...
> 
> Linus, IMO, is getting replaced. Look at how many functions he used to
> do 10 years ago he's no longer responsible for.

Agree.

> One of the most obvious examples is -next, where most integration issues
> are resolved before they even reach to Linus.
> 
> This is good for the community, as it allows us make the process better
> and scale out. It is also good for Linus, as I'm not sure how long he'd
> last if he still had to edit patches by hand too often. Instead, he gets
> to play with things that interest him more where his is irreplaceable.
> 
>> It's also a sad fact that a lot of things which look like obvious fixes
>> actually turn out not to be so with later testing.  This is why the
>> user visibility test is paramount.  If a bug fix has no real user
>> visible effects, it's often better to defer it no matter how obvious it
>> looks, which is why the static code checkers often get short shrift
>> before a merge window.
>>
>> A script measuring user visibility would be nice, but looks a bit
>> complex ...
> 
> It is, but I think it's worthwhile. Would something that'll show you
> things like:
> 
>  - How long a patch has been in -next?
>  - How many replies/reviews/comments it got on a mailing list?
>  - Did the 0day bot test it?
>  - Did syzbot fuzz it? for how long?
>  - If it references a bugzilla of some sort, how many
>    comments/reviews/etc it got there?
>  - Is it -stable material, or does it fix a regression in the current
>    merge window?
>  - If subsystem has custom testing rig, results from those tests
> 
> be a step in the right way? is it something you'd use to make decisions
> on whether you'd take a patch in?
> 

Reminds me (too much) of checkpatch.  Sure checkpatch has its uses,
as long as its not seen as the only true voice. (some beginners don't
know about that yet)

So with this new script, human evaluation would still be needed.
It's just a tool.  I could be used or misused or abused.
$maintainer still has a job to do, but having a tool could help.

But be careful what you wish for.  Having such a tool could help get
patches merged even quicker.

-- 
~Randy
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin via Ksummit-discuss]

On Thu, May 03, 2018 at 10:17:57AM -0700, Randy Dunlap wrote:
>On 05/03/2018 08:43 AM, Sasha Levin wrote:
>> On Thu, May 03, 2018 at 08:27:48AM -0700, James Bottomley wrote:
>>> On Thu, 2018-05-03 at 15:06 +0000, Sasha Levin via Ksummit-discuss
>>> wrote:
>>>> On Thu, May 03, 2018 at 04:48:50PM +0200, Willy Tarreau wrote:
>>>>> On Thu, May 03, 2018 at 07:33:04AM -0700, James Bottomley wrote:
>>>>>> They're definitely for bug fixes, but there's a spectrum: obvious
>>>>>> bug fixes with no side effects are easy to justify.  More complex
>>>>>> bug fixes run the risk of having side effects which introduce
>>>>>> other bugs, so could potentially destabilize the -rc process.  In
>>>>>> SCSI we tend to look at what the user visible effects of the bug
>>>>>> are in the post -rc5 region and if they're slight or wouldn't be
>>>>>> visible to most users, we'll hold them over.  If the fix looks
>>>>>> complex and we're not sure we caught the ramifications, we often
>>>>>> add it to the merge window tree with a cc to stable and a note
>>>>>> saying to wait X weeks before actually adding to the
>>>>>> stable tree just to make sure no side effects show up with wider
>>>>>> testing.  So, as with most things, it's a judgment call for the
>>>>>> maintainer.
>>>>>
>>>>> For me this is the right, and responsible way to deal with bug
>>>>> fixes. Self-control is much more efficient than random rejection
>>>>> and favors a good analysis.
>>>>
>>>> I think that the ideal outcome of this discussion, at least for me,
>>>> is a tool to go under scripts/ that would allow maintainers to get
>>>> some sort of (quantifiable) data that will indicate how well the
>>>> patch was tested via the regular channels.
>>>>
>>>> At which point it's the maintainer's judgement call on whether he
>>>> wants to grab the patch or wait for more tests or reviews.
>>>>
>>>> This is very similar to what James has described, it just needs to
>>>> leave his brain and turn into code :)
>>>
>>> I appreciate the sentiment, but if we could script taste, we'd have
>>> replaced Linus with something far less cantankerous a long time ago ...
>>
>> Linus, IMO, is getting replaced. Look at how many functions he used to
>> do 10 years ago he's no longer responsible for.
>
>Agree.
>
>> One of the most obvious examples is -next, where most integration issues
>> are resolved before they even reach to Linus.
>>
>> This is good for the community, as it allows us make the process better
>> and scale out. It is also good for Linus, as I'm not sure how long he'd
>> last if he still had to edit patches by hand too often. Instead, he gets
>> to play with things that interest him more where his is irreplaceable.
>>
>>> It's also a sad fact that a lot of things which look like obvious fixes
>>> actually turn out not to be so with later testing.  This is why the
>>> user visibility test is paramount.  If a bug fix has no real user
>>> visible effects, it's often better to defer it no matter how obvious it
>>> looks, which is why the static code checkers often get short shrift
>>> before a merge window.
>>>
>>> A script measuring user visibility would be nice, but looks a bit
>>> complex ...
>>
>> It is, but I think it's worthwhile. Would something that'll show you
>> things like:
>>
>>  - How long a patch has been in -next?
>>  - How many replies/reviews/comments it got on a mailing list?
>>  - Did the 0day bot test it?
>>  - Did syzbot fuzz it? for how long?
>>  - If it references a bugzilla of some sort, how many
>>    comments/reviews/etc it got there?
>>  - Is it -stable material, or does it fix a regression in the current
>>    merge window?
>>  - If subsystem has custom testing rig, results from those tests
>>
>> be a step in the right way? is it something you'd use to make decisions
>> on whether you'd take a patch in?
>>
>
>Reminds me (too much) of checkpatch.  Sure checkpatch has its uses,
>as long as its not seen as the only true voice. (some beginners don't
>know about that yet)
>
>So with this new script, human evaluation would still be needed.
>It's just a tool.  I could be used or misused or abused.
>$maintainer still has a job to do, but having a tool could help.
>
>But be careful what you wish for.  Having such a tool could help get
>patches merged even quicker.

While checkpatch is a tool for both authors and maintainers, I'm hoping
that this tool will only be useful for maintainers, who are less likely
to abuse it. I hope.

Maintainers are still needed. I started this discussion because right
now maintainers don't scale enough, and that in turn causes both delays
and mistakes in the process. We have a bunch of tools to help patch
authors, but not as many for maintainers.

To some extent, I do wish that this will help patches get merged
earlier. If a maintainer sees that the patch spent a while in -next,
passed all his subsystem's internal testing, got a few reviews, he could
just go ahead and merge it in faster without starting to dig through his
mail client and git tree.
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[James Bottomley]

On Thu, 2018-05-03 at 15:43 +0000, Sasha Levin via Ksummit-discuss
wrote:
> On Thu, May 03, 2018 at 08:27:48AM -0700, James Bottomley wrote:
[...]
> > It's also a sad fact that a lot of things which look like obvious
> > fixes actually turn out not to be so with later testing.  This is
> > why the user visibility test is paramount.  If a bug fix has no
> > real user visible effects, it's often better to defer it no matter
> > how obvious it looks, which is why the static code checkers often
> > get short shrift before a merge window.
> > 
> > A script measuring user visibility would be nice, but looks a bit
> > complex ...
> 
> It is, but I think it's worthwhile. Would something that'll show you
> things like:
> 
>  - How long a patch has been in -next?
>  - How many replies/reviews/comments it got on a mailing list?
>  - Did the 0day bot test it?
>  - Did syzbot fuzz it? for how long?
>  - If it references a bugzilla of some sort, how many
>    comments/reviews/etc it got there?
>  - Is it -stable material, or does it fix a regression in the current
>    merge window?
>  - If subsystem has custom testing rig, results from those tests
> 
> be a step in the right way? is it something you'd use to make
> decisions on whether you'd take a patch in?

Actually, no, these are all not what I'm talking about:  They're all
measures of whether the commit triggers another bug.  Which, I agree,
is the fear, so it would be good to have them of course, but they all
take time the maintainer doesn't have when making a quick decision
about a late -rc bug fix.

At late -rc the decision is the current user visible problem set
against the risk of -rc destabilization.  You're measuring the latter
in the above, but in the rule of thumb decision making we just assume
that's constant.  What we're looking to measure is the user visible
effect of not fixing the problem.

So, for instance, a boot failure on a widely used SCSI board is a no
brainer for fix now and tackle consequences later.  An obvious fix to
an error leg of a little used board is the other way: no-one is really
affected, so we don't take the risk.  The judgment call is the spectrum
in between these two extremes.

James

_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Willy Tarreau]

On Thu, May 03, 2018 at 08:27:48AM -0700, James Bottomley wrote:
> It's also a sad fact that a lot of things which look like obvious fixes
> actually turn out not to be so with later testing.  This is why the
> user visibility test is paramount.  If a bug fix has no real user
> visible effects, it's often better to defer it no matter how obvious it
> looks, which is why the static code checkers often get short shrift
> before a merge window.
> 
> A script measuring user visibility would be nice, but looks a bit
> complex ...

I totally agree with this and it matches my experience in haproxy. We
have had series of fixes that broke something else in very subtle ways
that made us want to improve non-reg, but many of the times we noted
that reg testing would hardly spot them given that the failures require
so many conditions to happen only once every million that it's hopeless.
It's just that some users are (un)lucky enough to meet all the conditions
at once very often and to be very sensitive to one error per million.

User exposure is needed. Having multiple stable release ensures everyone
gets their expected level of trust. Those on -rc want to see bugs before
they hit their users. Regressions are bad and require self-moderation and
self-estimation of the amount of trust in one's code, but they're better
in -rc than in -stable. I do happen to write some fixes I'm not totally
sure about and prefer not to backport them immediately. Users value
transparency because that helps them take safe decisions. If I say "this
is my fix, but I'd love more testing as I'm not yet sold on it", I'll get
some testers, but not the ones complaining that I broke their setup. Only
later it makes sense to progressively backport.

I have broken stable releases many times with failed backports. Almost
every time it was my fault due to incomplete testing. I could argue that
once you've built one hundred times in a week-end you're probably a bit
more lenient about next builds, or whatever. But in the end I was the
one breaking a working version. Seeing my branches picked up by Guenter
was a huge relief and it started to spot many build issues that I could
not figure myself. It doesn't make remaining bugs less important but at
least they are easier to swallow, to spot and to address.

What's not acceptable is rushed fixes that have obvious side effects that
could have been caught by closer analysis or better testing. It always
happens but it must not happen too often for the same person/subsystem.
This I think is where the line must be drawn. When Linus shouts once in
a while, it's a reminder for all others. Tune the potentiometer of his
detection threshold a bit lower and we'll get less regressions because
it's never pleasant to be called stupid in public like he does.

Willy

Re: [Ksummit-discuss] bug-introducing patches

[Theodore Y. Ts'o]

On Thu, May 03, 2018 at 02:08:51PM +0300, Jani Nikula wrote:
> On Tue, 01 May 2018, "Theodore Y. Ts'o" <tytso@mit.edu> wrote:
> > Post -rc3 or -rc4, in my opinion bug fixes should wait until the next
> > merge window before they get merged at all.
> 
> What are -rc5 and later for then if not bug fixes? Baffled.

Regression fixes?

Note that if people stopped introducing regressions to the kernel, we
might actually be able to release the final version after -rc6 or even
earlier.

There's nothing which says that we MUST have -rc7, -rc8, -rc9
releases.  If we were actually disciplined in our testing and in what
we push into Linus's tree, we might actually be able to go to a
two-month release cycle, or perhaps even slightly shorter.

But if people insist on trying to fix bugs, and those bugs fixes
introduce regressions, then we end up with a longer release cycle,
which causes people to want to stuck more bug fixes, or worse, even
some feature commits late into the development cycle, and it becomes a
vicious cycle with releases taking longer and longer.

	  	  	    	    	 - Ted
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Geert Uytterhoeven]

Hi Sasha,

On Tue, May 1, 2018 at 6:38 PM, Sasha Levin
<Alexander.Levin@microsoft.com> wrote:
> Working on AUTOSEL, it became even more obvious to me how difficult it is for a
> patch to get a proper review. Maintainers found it difficult to keep up with
> the upstream work for their subsystem, and reviewing additional -stable patches
> put even more load on them which some suggested would be more than what they
> can handle.

Thanks for your work!

>  - For some reason, the odds of a -rc commit to be targetted for -stable is
>    over 20%, while for merge window commits it's about 3%. I can't quite
> explain why that happens, but this would suggest that -rc commits end up
> hurting -stable pretty badly.

Aren't more -rc commits targeted for -stable because they are bugfixes?
Ideally, new features are supposed to be merged during the merge window,
while -rc commits fix bugs.

So they can be categorized like:
  1. Plain -rc commits,
  2. -rc commits fixing a bug:
       a. in the same release cycle,
       b. in a previous release.

2a assumes the bug was backported to -stable, too, doesn't it?

Do you have statistics for which categories are most buggy?

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin via Ksummit-discuss]

On Wed, May 02, 2018 at 05:32:37PM +0200, Geert Uytterhoeven wrote:
>Hi Sasha,
>
>On Tue, May 1, 2018 at 6:38 PM, Sasha Levin
><Alexander.Levin@microsoft.com> wrote:
>> Working on AUTOSEL, it became even more obvious to me how difficult it is for a
>> patch to get a proper review. Maintainers found it difficult to keep up with
>> the upstream work for their subsystem, and reviewing additional -stable patches
>> put even more load on them which some suggested would be more than what they
>> can handle.
>
>Thanks for your work!
>
>>  - For some reason, the odds of a -rc commit to be targetted for -stable is
>>    over 20%, while for merge window commits it's about 3%. I can't quite
>> explain why that happens, but this would suggest that -rc commits end up
>> hurting -stable pretty badly.
>
>Aren't more -rc commits targeted for -stable because they are bugfixes?
>Ideally, new features are supposed to be merged during the merge window,
>while -rc commits fix bugs.

new features can only be merged during a merge window, bug fixes can 
be merged at any point.

>So they can be categorized like:
>  1. Plain -rc commits,

What's this exactly? -rc commits are only supposed to fix bugs.

>  2. -rc commits fixing a bug:
>       a. in the same release cycle,
>       b. in a previous release.
>
>2a assumes the bug was backported to -stable, too, doesn't it?

Bug fixes for features introduced in that release cycle won't be
backported to stable.

>Do you have statistics for which categories are most buggy?

I haven't broken it down to subsystems for a few reasons:

 - My dataset is based on the Fixes: tag, some subsystems use it less
   than others.
 - Maintainers change, so even if one subsystem is being awesome about
   it today, it might not be the case in a year.
 - I don't really want to point fingers at a particular subsystem, I
   think that this is an issue at the kernel level.
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: bug-introducing patches

[Geert Uytterhoeven]

Hi Sasha,

On Wed, May 2, 2018 at 9:51 PM, Sasha Levin
<Alexander.Levin@microsoft.com> wrote:
> On Wed, May 02, 2018 at 05:32:37PM +0200, Geert Uytterhoeven wrote:
>>On Tue, May 1, 2018 at 6:38 PM, Sasha Levin
>><Alexander.Levin@microsoft.com> wrote:
>>> Working on AUTOSEL, it became even more obvious to me how difficult it is for a
>>> patch to get a proper review. Maintainers found it difficult to keep up with
>>> the upstream work for their subsystem, and reviewing additional -stable patches
>>> put even more load on them which some suggested would be more than what they
>>> can handle.
>>
>>Thanks for your work!
>>
>>>  - For some reason, the odds of a -rc commit to be targetted for -stable is
>>>    over 20%, while for merge window commits it's about 3%. I can't quite
>>> explain why that happens, but this would suggest that -rc commits end up
>>> hurting -stable pretty badly.
>>
>>Aren't more -rc commits targeted for -stable because they are bugfixes?
>>Ideally, new features are supposed to be merged during the merge window,
>>while -rc commits fix bugs.
>
> new features can only be merged during a merge window, bug fixes can
> be merged at any point.

I wrote "ideally". There's a big difference between theory and practice...

>>So they can be categorized like:
>>  1. Plain -rc commits,
>
> What's this exactly? -rc commits are only supposed to fix bugs.

... hence not all of them are fixes.

Sometimes fast-tracking a new feature or API reduces dependencies for the
next merge window. This is just one example of IMHO valid non-bugfix
-rc commits.

Between v4.17-rc1 and v4.17-rc3, there are 660 non-merge commits, of which
  - 245 carry a Fixes tag,
  - 196 carry a CC stable,
  - 395 contain the string "fix".
(non-mutually exclusive)

That leaves us with 200 commits not falling in the bugfix category.

>>  2. -rc commits fixing a bug:
>>       a. in the same release cycle,
>>       b. in a previous release.
>>
>>2a assumes the bug was backported to -stable, too, doesn't it?
>
> Bug fixes for features introduced in that release cycle won't be
> backported to stable.

They do, if the original commit was introduced during the same cycle and
backported to stable.

>>Do you have statistics for which categories are most buggy?
>
> I haven't broken it down to subsystems for a few reasons:

I didn't mean break down by subsystem, but by category from the list above
(1, 2a, 2b).

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [Ksummit-discuss] bug-introducing patches

[Theodore Y. Ts'o]

On Wed, May 02, 2018 at 10:41:56PM +0200, Geert Uytterhoeven wrote:
> 
> Between v4.17-rc1 and v4.17-rc3, there are 660 non-merge commits, of which
>   - 245 carry a Fixes tag,
>   - 196 carry a CC stable,
>   - 395 contain the string "fix".
> (non-mutually exclusive)
> 
> That leaves us with 200 commits not falling in the bugfix category.

Some non-bug fixes are allowed in -rc2.  So perhaps what might be
interesting is to look at v4.16 (which is completed), and look at the
distribution of commits:

	* regressions fixes (for bugs introduced during the current
		release cycle)
	* "normal" bug fixes
	* commits which don't touch code (e.g., spelling or
		documentation-only fixes)
	* other commits (features or cleanup fixes)

at each rcX level.  The historic "standard" has been feature commits
in -rc1 and -rc2 (tolerated, but ideally should before the merge
window), bug fixes / regressions in -rc3 and -rc4, and after -rc4,
regression fixes only.  It would be interesting to see how well we
have been holding to the historical ideal.

It would then be intersting to use Sasha's analysis to see whether
there are more bug fixes caused by regression fixes versus normal bug
fixes, and whether or not they are common when fixes come "out of
cycle" --- for example, a non-regression bug fix in -rc5 or -rc6.

Because if that last is the case, then the prescription is very simple
and not controversial --- bug fixes found post -rc4 should be held to
the next merge window.

If the concern is regression fixes which require one or two tries
before they are fixed before 4.16-FINAL is released, then that's a
"life is hard for AUTOSEL" issue, and I suspect Sasha will find that
there is rather less sympathy for holding regression fixes for an
extra week or two.
 
If the concern is bug fixes that show up in -rc3 and -rc4, but which
aren't hitting linux-next first, then holding bug fixes in linux-next
for a week makes sense, and if that means that a bug fix found post
-rc3 needs to marinate in linux-next for a week, and then it then
misses the -rc4 "bug fix" deadline, we can have a discussion about
whether bug fixes should be allowed in -rc5 after a week's marination.

My personal opinion is "to hell with it, just wait until the next
merge window" --- but this can cause more work for the stable
maintainers since a lot of bug fixes would then land in -rc1.

Cheers,

					- Ted
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Guenter Roeck]

On 05/02/2018 05:06 PM, Theodore Y. Ts'o wrote:
> On Wed, May 02, 2018 at 10:41:56PM +0200, Geert Uytterhoeven wrote:
>>
>> Between v4.17-rc1 and v4.17-rc3, there are 660 non-merge commits, of which
>>    - 245 carry a Fixes tag,
>>    - 196 carry a CC stable,
>>    - 395 contain the string "fix".
>> (non-mutually exclusive)
>>
>> That leaves us with 200 commits not falling in the bugfix category.
> 
> Some non-bug fixes are allowed in -rc2.  So perhaps what might be
> interesting is to look at v4.16 (which is completed), and look at the
> distribution of commits:
> 
> 	* regressions fixes (for bugs introduced during the current
> 		release cycle)
> 	* "normal" bug fixes
> 	* commits which don't touch code (e.g., spelling or
> 		documentation-only fixes)
> 	* other commits (features or cleanup fixes)
> 
> at each rcX level.  The historic "standard" has been feature commits
> in -rc1 and -rc2 (tolerated, but ideally should before the merge
> window), bug fixes / regressions in -rc3 and -rc4, and after -rc4,
> regression fixes only.  It would be interesting to see how well we
> have been holding to the historical ideal.
> 
> It would then be intersting to use Sasha's analysis to see whether
> there are more bug fixes caused by regression fixes versus normal bug
> fixes, and whether or not they are common when fixes come "out of
> cycle" --- for example, a non-regression bug fix in -rc5 or -rc6.
> 
> Because if that last is the case, then the prescription is very simple
> and not controversial --- bug fixes found post -rc4 should be held to
> the next merge window.
> 

Holding up even fixes for severe bugs for 4-6 weeks ? Seriously, that is
unrealistic. Holding up the fix for the next SpeckHammer because it was not
ready before -rc4 ? I don't think so.

Even when not counting severe problems, you are adding lots of additional work
for those who do and want to rely on stable releases to merge in bug fixes.
Sure, I am at times annoyed having to deal with a regression in a stable
release, but it very much beats digging through various mailing lists for
pending patches to fix CVEs, or for crashes seen in the field, just because
they are held hostage by some restrictive process. Even worse, I'd end up
picking the regressions anyway because I can _not_ wait those 4-6 weeks
plus the time it takes for the fixes to show up in a stable release.

Really, that just makes the situation worse for everyone. We would be much
better off by further improving test coverage.

Guenter
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Willy Tarreau]

On Wed, May 02, 2018 at 05:38:32PM -0700, Guenter Roeck wrote:
> > Because if that last is the case, then the prescription is very simple
> > and not controversial --- bug fixes found post -rc4 should be held to
> > the next merge window.
> > 
> 
> Holding up even fixes for severe bugs for 4-6 weeks ? Seriously, that is
> unrealistic. Holding up the fix for the next SpeckHammer because it was not
> ready before -rc4 ? I don't think so.

That's exactly what I explained earlier in this thread, it will actually
make the resulting kernels even worse as soon as there is less than 100%
regression (which is the case, since some fixes are valid). Postponing
valid fixes because some of them might be wrong is a bad idea. We need
to trust the developer regarding the test coverage and the developer has
to become trusted by openly indicating the type of testing run on the
patch. From there it will become easier to decide whether to revert a
whole patch set after a few failed fixes, or to take a few more fixes
in hopes that ultimately everything will be good.

Willy

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin]

On Wed, May 02, 2018 at 05:38:32PM -0700, Guenter Roeck wrote:
>On 05/02/2018 05:06 PM, Theodore Y. Ts'o wrote:
>>On Wed, May 02, 2018 at 10:41:56PM +0200, Geert Uytterhoeven wrote:
>>>
>>>Between v4.17-rc1 and v4.17-rc3, there are 660 non-merge commits, of which
>>>   - 245 carry a Fixes tag,
>>>   - 196 carry a CC stable,
>>>   - 395 contain the string "fix".
>>>(non-mutually exclusive)
>>>
>>>That leaves us with 200 commits not falling in the bugfix category.
>>
>>Some non-bug fixes are allowed in -rc2.  So perhaps what might be
>>interesting is to look at v4.16 (which is completed), and look at the
>>distribution of commits:
>>
>>	* regressions fixes (for bugs introduced during the current
>>		release cycle)
>>	* "normal" bug fixes
>>	* commits which don't touch code (e.g., spelling or
>>		documentation-only fixes)
>>	* other commits (features or cleanup fixes)
>>
>>at each rcX level.  The historic "standard" has been feature commits
>>in -rc1 and -rc2 (tolerated, but ideally should before the merge
>>window), bug fixes / regressions in -rc3 and -rc4, and after -rc4,
>>regression fixes only.  It would be interesting to see how well we
>>have been holding to the historical ideal.
>>
>>It would then be intersting to use Sasha's analysis to see whether
>>there are more bug fixes caused by regression fixes versus normal bug
>>fixes, and whether or not they are common when fixes come "out of
>>cycle" --- for example, a non-regression bug fix in -rc5 or -rc6.
>>
>>Because if that last is the case, then the prescription is very simple
>>and not controversial --- bug fixes found post -rc4 should be held to
>>the next merge window.
>>
>
>Holding up even fixes for severe bugs for 4-6 weeks ? Seriously, that is
>unrealistic. Holding up the fix for the next SpeckHammer because it was not
>ready before -rc4 ? I don't think so.

For severe problems, the patch usually gets more than enough reviews and
testing, so I don't see a need to soak it in -next more than some
minimal amount of time to get bot coverage.

However, these things show up only a few times per year. Most of the
fixes even in late -rc cycles are for older bugs that aren't too
critical. We can't base our decision on severe bugs that get exceptional
treatment anyways (see PTI getting pushed in -stable).

>Even when not counting severe problems, you are adding lots of additional work
>for those who do and want to rely on stable releases to merge in bug fixes.
>Sure, I am at times annoyed having to deal with a regression in a stable
>release, but it very much beats digging through various mailing lists for
>pending patches to fix CVEs, or for crashes seen in the field, just because
>they are held hostage by some restrictive process. Even worse, I'd end up
>picking the regressions anyway because I can _not_ wait those 4-6 weeks
>plus the time it takes for the fixes to show up in a stable release.

I think that for -stable we don't have a good idea how soon we want to
merge patches in. On one hand enterprise distro folks complain we're
jumping the gun, and on the other hand folks like yourself claim we're
too slow :)

>Really, that just makes the situation worse for everyone. We would be much
>better off by further improving test coverage.

I'm definitely not saying "no" to more test coverage, but these are work
streams that can happen in parallel.

Re: [Ksummit-discuss] bug-introducing patches

[Guenter Roeck]

On Thu, May 03, 2018 at 02:55:36PM +0000, Sasha Levin wrote:
> On Wed, May 02, 2018 at 05:38:32PM -0700, Guenter Roeck wrote:
> >On 05/02/2018 05:06 PM, Theodore Y. Ts'o wrote:
> >>On Wed, May 02, 2018 at 10:41:56PM +0200, Geert Uytterhoeven wrote:
> >>>
> >>>Between v4.17-rc1 and v4.17-rc3, there are 660 non-merge commits, of which
> >>>   - 245 carry a Fixes tag,
> >>>   - 196 carry a CC stable,
> >>>   - 395 contain the string "fix".
> >>>(non-mutually exclusive)
> >>>
> >>>That leaves us with 200 commits not falling in the bugfix category.
> >>
> >>Some non-bug fixes are allowed in -rc2.  So perhaps what might be
> >>interesting is to look at v4.16 (which is completed), and look at the
> >>distribution of commits:
> >>
> >>	* regressions fixes (for bugs introduced during the current
> >>		release cycle)
> >>	* "normal" bug fixes
> >>	* commits which don't touch code (e.g., spelling or
> >>		documentation-only fixes)
> >>	* other commits (features or cleanup fixes)
> >>
> >>at each rcX level.  The historic "standard" has been feature commits
> >>in -rc1 and -rc2 (tolerated, but ideally should before the merge
> >>window), bug fixes / regressions in -rc3 and -rc4, and after -rc4,
> >>regression fixes only.  It would be interesting to see how well we
> >>have been holding to the historical ideal.
> >>
> >>It would then be intersting to use Sasha's analysis to see whether
> >>there are more bug fixes caused by regression fixes versus normal bug
> >>fixes, and whether or not they are common when fixes come "out of
> >>cycle" --- for example, a non-regression bug fix in -rc5 or -rc6.
> >>
> >>Because if that last is the case, then the prescription is very simple
> >>and not controversial --- bug fixes found post -rc4 should be held to
> >>the next merge window.
> >>
> >
> >Holding up even fixes for severe bugs for 4-6 weeks ? Seriously, that is
> >unrealistic. Holding up the fix for the next SpeckHammer because it was not
> >ready before -rc4 ? I don't think so.
> 
> For severe problems, the patch usually gets more than enough reviews and
> testing, so I don't see a need to soak it in -next more than some
> minimal amount of time to get bot coverage.
> 
> However, these things show up only a few times per year. Most of the
> fixes even in late -rc cycles are for older bugs that aren't too
> critical. We can't base our decision on severe bugs that get exceptional
> treatment anyways (see PTI getting pushed in -stable).
> 
> >Even when not counting severe problems, you are adding lots of additional work
> >for those who do and want to rely on stable releases to merge in bug fixes.
> >Sure, I am at times annoyed having to deal with a regression in a stable
> >release, but it very much beats digging through various mailing lists for
> >pending patches to fix CVEs, or for crashes seen in the field, just because
> >they are held hostage by some restrictive process. Even worse, I'd end up
> >picking the regressions anyway because I can _not_ wait those 4-6 weeks
> >plus the time it takes for the fixes to show up in a stable release.
> 
> I think that for -stable we don't have a good idea how soon we want to
> merge patches in. On one hand enterprise distro folks complain we're
> jumping the gun, and on the other hand folks like yourself claim we're
> too slow :)
> 

You are misquoting me. I am saying that it would be a bad idea to hold up
bug fixes after -rc4, which is quite different to saying that patches
don't make it into stable releases fast enough. I am perfectly happy to
wait a week or so for a patch to soak in _mainline_ before being applied
to stable.

I am absolutely _not_ happy with the number of patches making it into
-stable releases recently. I am especially very concerned that the current
flurry of patches queued for -stable will destabilize pretty much all
stable releases, and pretty badly, for that matter. I am seriously
contemplating not to integrate the next few stable releases into ChromeOS
for that very reason. That would be a different discussion, though.

Guenter

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin via Ksummit-discuss]

On Thu, May 03, 2018 at 08:49:11AM -0700, Guenter Roeck wrote:
>On Thu, May 03, 2018 at 02:55:36PM +0000, Sasha Levin wrote:
>> On Wed, May 02, 2018 at 05:38:32PM -0700, Guenter Roeck wrote:
>> >On 05/02/2018 05:06 PM, Theodore Y. Ts'o wrote:
>> >>On Wed, May 02, 2018 at 10:41:56PM +0200, Geert Uytterhoeven wrote:
>> >>>
>> >>>Between v4.17-rc1 and v4.17-rc3, there are 660 non-merge commits, of which
>> >>>   - 245 carry a Fixes tag,
>> >>>   - 196 carry a CC stable,
>> >>>   - 395 contain the string "fix".
>> >>>(non-mutually exclusive)
>> >>>
>> >>>That leaves us with 200 commits not falling in the bugfix category.
>> >>
>> >>Some non-bug fixes are allowed in -rc2.  So perhaps what might be
>> >>interesting is to look at v4.16 (which is completed), and look at the
>> >>distribution of commits:
>> >>
>> >>	* regressions fixes (for bugs introduced during the current
>> >>		release cycle)
>> >>	* "normal" bug fixes
>> >>	* commits which don't touch code (e.g., spelling or
>> >>		documentation-only fixes)
>> >>	* other commits (features or cleanup fixes)
>> >>
>> >>at each rcX level.  The historic "standard" has been feature commits
>> >>in -rc1 and -rc2 (tolerated, but ideally should before the merge
>> >>window), bug fixes / regressions in -rc3 and -rc4, and after -rc4,
>> >>regression fixes only.  It would be interesting to see how well we
>> >>have been holding to the historical ideal.
>> >>
>> >>It would then be intersting to use Sasha's analysis to see whether
>> >>there are more bug fixes caused by regression fixes versus normal bug
>> >>fixes, and whether or not they are common when fixes come "out of
>> >>cycle" --- for example, a non-regression bug fix in -rc5 or -rc6.
>> >>
>> >>Because if that last is the case, then the prescription is very simple
>> >>and not controversial --- bug fixes found post -rc4 should be held to
>> >>the next merge window.
>> >>
>> >
>> >Holding up even fixes for severe bugs for 4-6 weeks ? Seriously, that is
>> >unrealistic. Holding up the fix for the next SpeckHammer because it was not
>> >ready before -rc4 ? I don't think so.
>>
>> For severe problems, the patch usually gets more than enough reviews and
>> testing, so I don't see a need to soak it in -next more than some
>> minimal amount of time to get bot coverage.
>>
>> However, these things show up only a few times per year. Most of the
>> fixes even in late -rc cycles are for older bugs that aren't too
>> critical. We can't base our decision on severe bugs that get exceptional
>> treatment anyways (see PTI getting pushed in -stable).
>>
>> >Even when not counting severe problems, you are adding lots of additional work
>> >for those who do and want to rely on stable releases to merge in bug fixes.
>> >Sure, I am at times annoyed having to deal with a regression in a stable
>> >release, but it very much beats digging through various mailing lists for
>> >pending patches to fix CVEs, or for crashes seen in the field, just because
>> >they are held hostage by some restrictive process. Even worse, I'd end up
>> >picking the regressions anyway because I can _not_ wait those 4-6 weeks
>> >plus the time it takes for the fixes to show up in a stable release.
>>
>> I think that for -stable we don't have a good idea how soon we want to
>> merge patches in. On one hand enterprise distro folks complain we're
>> jumping the gun, and on the other hand folks like yourself claim we're
>> too slow :)
>>
>
>You are misquoting me. I am saying that it would be a bad idea to hold up
>bug fixes after -rc4, which is quite different to saying that patches
>don't make it into stable releases fast enough. I am perfectly happy to
>wait a week or so for a patch to soak in _mainline_ before being applied
>to stable.

Most bug fixes that go in at that point are fixes for previous released
kernels, what's the harm in keeping them around for longer?

I'm not saying that it should be some arbitrary rule for everyone, but
just suggesting that maintainers should exercise more caution merging
untested commits that don't even fix a current regression.

w.r.t stable, as you just said, you're fine with a week or two, the
enterprise folks (as well as Ted, to some extend, in this thread)
suggest that this should be a month+

>I am absolutely _not_ happy with the number of patches making it into
>-stable releases recently. I am especially very concerned that the current
>flurry of patches queued for -stable will destabilize pretty much all
>stable releases, and pretty badly, for that matter. I am seriously
>contemplating not to integrate the next few stable releases into ChromeOS
>for that very reason. That would be a different discussion, though.

For AUTOSEL, I'd be happy to learn of issues you encounter and address
them in my process.

I've been submitting automatically selected patches for over a year now
and the track record for regressions is on par with patches that are
tagged for stable.
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Justin Forbes]

On Thu, May 3, 2018 at 11:02 AM, Sasha Levin
<Alexander.Levin@microsoft.com> wrote:
> On Thu, May 03, 2018 at 08:49:11AM -0700, Guenter Roeck wrote:
>>On Thu, May 03, 2018 at 02:55:36PM +0000, Sasha Levin wrote:
>>> On Wed, May 02, 2018 at 05:38:32PM -0700, Guenter Roeck wrote:
>>> >On 05/02/2018 05:06 PM, Theodore Y. Ts'o wrote:
>>> >>On Wed, May 02, 2018 at 10:41:56PM +0200, Geert Uytterhoeven wrote:
>>> >>>
>>> >>>Between v4.17-rc1 and v4.17-rc3, there are 660 non-merge commits, of which
>>> >>>   - 245 carry a Fixes tag,
>>> >>>   - 196 carry a CC stable,
>>> >>>   - 395 contain the string "fix".
>>> >>>(non-mutually exclusive)
>>> >>>
>>> >>>That leaves us with 200 commits not falling in the bugfix category.
>>> >>
>>> >>Some non-bug fixes are allowed in -rc2.  So perhaps what might be
>>> >>interesting is to look at v4.16 (which is completed), and look at the
>>> >>distribution of commits:
>>> >>
>>> >>   * regressions fixes (for bugs introduced during the current
>>> >>           release cycle)
>>> >>   * "normal" bug fixes
>>> >>   * commits which don't touch code (e.g., spelling or
>>> >>           documentation-only fixes)
>>> >>   * other commits (features or cleanup fixes)
>>> >>
>>> >>at each rcX level.  The historic "standard" has been feature commits
>>> >>in -rc1 and -rc2 (tolerated, but ideally should before the merge
>>> >>window), bug fixes / regressions in -rc3 and -rc4, and after -rc4,
>>> >>regression fixes only.  It would be interesting to see how well we
>>> >>have been holding to the historical ideal.
>>> >>
>>> >>It would then be intersting to use Sasha's analysis to see whether
>>> >>there are more bug fixes caused by regression fixes versus normal bug
>>> >>fixes, and whether or not they are common when fixes come "out of
>>> >>cycle" --- for example, a non-regression bug fix in -rc5 or -rc6.
>>> >>
>>> >>Because if that last is the case, then the prescription is very simple
>>> >>and not controversial --- bug fixes found post -rc4 should be held to
>>> >>the next merge window.
>>> >>
>>> >
>>> >Holding up even fixes for severe bugs for 4-6 weeks ? Seriously, that is
>>> >unrealistic. Holding up the fix for the next SpeckHammer because it was not
>>> >ready before -rc4 ? I don't think so.
>>>
>>> For severe problems, the patch usually gets more than enough reviews and
>>> testing, so I don't see a need to soak it in -next more than some
>>> minimal amount of time to get bot coverage.
>>>
>>> However, these things show up only a few times per year. Most of the
>>> fixes even in late -rc cycles are for older bugs that aren't too
>>> critical. We can't base our decision on severe bugs that get exceptional
>>> treatment anyways (see PTI getting pushed in -stable).
>>>
>>> >Even when not counting severe problems, you are adding lots of additional work
>>> >for those who do and want to rely on stable releases to merge in bug fixes.
>>> >Sure, I am at times annoyed having to deal with a regression in a stable
>>> >release, but it very much beats digging through various mailing lists for
>>> >pending patches to fix CVEs, or for crashes seen in the field, just because
>>> >they are held hostage by some restrictive process. Even worse, I'd end up
>>> >picking the regressions anyway because I can _not_ wait those 4-6 weeks
>>> >plus the time it takes for the fixes to show up in a stable release.
>>>
>>> I think that for -stable we don't have a good idea how soon we want to
>>> merge patches in. On one hand enterprise distro folks complain we're
>>> jumping the gun, and on the other hand folks like yourself claim we're
>>> too slow :)
>>>
>>
>>You are misquoting me. I am saying that it would be a bad idea to hold up
>>bug fixes after -rc4, which is quite different to saying that patches
>>don't make it into stable releases fast enough. I am perfectly happy to
>>wait a week or so for a patch to soak in _mainline_ before being applied
>>to stable.
>
> Most bug fixes that go in at that point are fixes for previous released
> kernels, what's the harm in keeping them around for longer?
>
> I'm not saying that it should be some arbitrary rule for everyone, but
> just suggesting that maintainers should exercise more caution merging
> untested commits that don't even fix a current regression.
>
There is a balance here. In the past, one of the biggest complaints we
had as distro maintainers was that known regressions get reported, and
fixed, and then the maintainer would sit on the fix until the next
merge window. This happened even for trivial fixes. And not being in
tree does keep it out of stable.  This has improved greatly recently.
Perhaps things have over compensated, but I don' t think that putting
a blanket rule out there is the answer. Just perhaps some best
practices for test coverage.

> w.r.t stable, as you just said, you're fine with a week or two, the
> enterprise folks (as well as Ted, to some extend, in this thread)
> suggest that this should be a month+

I don' t have an issue with some things percolating in mainline for a
bit before being pulled into stable, it might have saved us a lot of
grief with the random patches last week.  But again there isn't a set
rule that seems logical here. Adequate test coverage is the concern,
not some set time, especially for obvious fixes.  I know for Fedora,
we do have (some) people testing rawhide daily, so things in Linus'
tree start getting end user testing usually within 24 hours.

I am not saying that things are great now, or cannot be improved. I am
just concerned that we come up with some "rule" that takes us back to
keeping legitimate fixes out of tree for much longer than necessary.

>>I am absolutely _not_ happy with the number of patches making it into
>>-stable releases recently. I am especially very concerned that the current
>>flurry of patches queued for -stable will destabilize pretty much all
>>stable releases, and pretty badly, for that matter. I am seriously
>>contemplating not to integrate the next few stable releases into ChromeOS
>>for that very reason. That would be a different discussion, though.

There is certainly concern here. If end users stop trusting stable
kernel updates, the next time there is a big security issue, they may
just ignore the fix until there is consensus that it is safe to update
.
>
> For AUTOSEL, I'd be happy to learn of issues you encounter and address
> them in my process.
>
> I've been submitting automatically selected patches for over a year now
> and the track record for regressions is on par with patches that are
> tagged for stable.

Re: [Ksummit-discuss] bug-introducing patches

[Guenter Roeck]

On Thu, May 03, 2018 at 04:02:12PM +0000, Sasha Levin wrote:
> >You are misquoting me. I am saying that it would be a bad idea to hold up
> >bug fixes after -rc4, which is quite different to saying that patches
> >don't make it into stable releases fast enough. I am perfectly happy to
> >wait a week or so for a patch to soak in _mainline_ before being applied
> >to stable.
> 
> Most bug fixes that go in at that point are fixes for previous released
> kernels, what's the harm in keeping them around for longer?
> 

The ones I am mostly concerned about are fixes for CVEs, crashes, file
system corruptions, and similar. Maybe the enterprise folks don't mind
keeping those around for a month or more even though a fix is available.
I do.

> For AUTOSEL, I'd be happy to learn of issues you encounter and address
> them in my process.
> 
> I've been submitting automatically selected patches for over a year now
> and the track record for regressions is on par with patches that are
> tagged for stable.

So far it hasn't been an issue. Or, rather, not much; with more patches
applied, the percentage of regressions may be the same, but the number
of regressions is higher. My "customers" care about the number, not about
the percentage.

However, the set of test results attached below (from last night)
_is_ a problem. I don't know what changed, but something clearly did,
to the point that I am _very_ concerned about the next set of stable
releases.

Guenter

---
For v4.14.39-580-gc8cd674:

Build results:
	total: 146 pass: 98 fail: 48
Qemu test results:
	total: 100 pass: 21 fail: 79

For v4.4.131-268-ga33ce4a:

Build results:
	total: 146 pass: 92 fail: 54
Qemu test results:
	total: 127 pass: 91 fail: 36

Re: [Ksummit-discuss] bug-introducing patches

[Al Viro]

On Wed, May 02, 2018 at 08:06:20PM -0400, Theodore Y. Ts'o wrote:
 
> Because if that last is the case, then the prescription is very simple
> and not controversial --- bug fixes found post -rc4 should be held to
> the next merge window.

Provided it's not a known-to-be-exploited roothole, at least...

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin via Ksummit-discuss]

On Wed, May 02, 2018 at 08:06:20PM -0400, Theodore Y. Ts'o wrote:
>On Wed, May 02, 2018 at 10:41:56PM +0200, Geert Uytterhoeven wrote:
>>
>> Between v4.17-rc1 and v4.17-rc3, there are 660 non-merge commits, of which
>>   - 245 carry a Fixes tag,
>>   - 196 carry a CC stable,
>>   - 395 contain the string "fix".
>> (non-mutually exclusive)
>>
>> That leaves us with 200 commits not falling in the bugfix category.
>
>Some non-bug fixes are allowed in -rc2.  So perhaps what might be
>interesting is to look at v4.16 (which is completed), and look at the
>distribution of commits:
>
>	* regressions fixes (for bugs introduced during the current
>		release cycle)
>	* "normal" bug fixes
>	* commits which don't touch code (e.g., spelling or
>		documentation-only fixes)
>	* other commits (features or cleanup fixes)
>
>at each rcX level.  The historic "standard" has been feature commits
>in -rc1 and -rc2 (tolerated, but ideally should before the merge
>window), bug fixes / regressions in -rc3 and -rc4, and after -rc4,
>regression fixes only.  It would be interesting to see how well we
>have been holding to the historical ideal.
>
>It would then be intersting to use Sasha's analysis to see whether
>there are more bug fixes caused by regression fixes versus normal bug
>fixes, and whether or not they are common when fixes come "out of
>cycle" --- for example, a non-regression bug fix in -rc5 or -rc6.
>
>Because if that last is the case, then the prescription is very simple
>and not controversial --- bug fixes found post -rc4 should be held to
>the next merge window.
>
>If the concern is regression fixes which require one or two tries
>before they are fixed before 4.16-FINAL is released, then that's a
>"life is hard for AUTOSEL" issue, and I suspect Sasha will find that
>there is rather less sympathy for holding regression fixes for an
>extra week or two.
>
>If the concern is bug fixes that show up in -rc3 and -rc4, but which
>aren't hitting linux-next first, then holding bug fixes in linux-next
>for a week makes sense, and if that means that a bug fix found post
>-rc3 needs to marinate in linux-next for a week, and then it then
>misses the -rc4 "bug fix" deadline, we can have a discussion about
>whether bug fixes should be allowed in -rc5 after a week's marination.
>
>My personal opinion is "to hell with it, just wait until the next
>merge window" --- but this can cause more work for the stable
>maintainers since a lot of bug fixes would then land in -rc1.

I'll work on breaking up the 4.16 commits into categories, but one
interesting statistic I've noticed while starting the work is:

Fixes in -rc cycles:
rc1 68
rc2 147
rc3 88
rc4 121
rc5 40
rc6 193
rc7 98

Average days in -next, for a fix, per -rc cycle:
rc1 27.25
rc2 21.4286
rc3 22.5114
rc4 18.281
rc5 14.65
rc6 12.6166
rc7 8.70408

Fixes for bugs not introduced in current merge window:
rc1 40
rc2 113
rc3 61
rc4 79
rc5 25
rc6 139
rc7 72

So for some reason, there is a rush to push fixes for older bugs (that
were not introduced in the current merge window) to the point that rc7
commits that only existed for a few days are merged in to address older
bugs.
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Willy Tarreau]

On Thu, May 03, 2018 at 02:46:14PM +0000, Sasha Levin wrote:
> I'll work on breaking up the 4.16 commits into categories, but one
> interesting statistic I've noticed while starting the work is:
> 
> Fixes in -rc cycles:
> rc1 68
> rc2 147
> rc3 88
> rc4 121
> rc5 40
> rc6 193
> rc7 98
> 
> Average days in -next, for a fix, per -rc cycle:
> rc1 27.25
> rc2 21.4286
> rc3 22.5114
> rc4 18.281
> rc5 14.65
> rc6 12.6166
> rc7 8.70408
> 
> Fixes for bugs not introduced in current merge window:
> rc1 40
> rc2 113
> rc3 61
> rc4 79
> rc5 25
> rc6 139
> rc7 72
> 
> So for some reason, there is a rush to push fixes for older bugs (that
> were not introduced in the current merge window) to the point that rc7
> commits that only existed for a few days are merged in to address older
> bugs.

IMHO it's because it's the time it takes for users to start to trust the
3rd or 4th stable release of the previous version, to switch to it, to
face a bug, to report it, and for the maintainer to write a fix.

I wouldn't be much surprised if you'd find that among those not introduced
in the current merge window, many were introduced in the previous release.

Willy

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin via Ksummit-discuss]

On Thu, May 03, 2018 at 04:52:05PM +0200, Willy Tarreau wrote:
>On Thu, May 03, 2018 at 02:46:14PM +0000, Sasha Levin wrote:
>> I'll work on breaking up the 4.16 commits into categories, but one
>> interesting statistic I've noticed while starting the work is:
>>
>> Fixes in -rc cycles:
>> rc1 68
>> rc2 147
>> rc3 88
>> rc4 121
>> rc5 40
>> rc6 193
>> rc7 98
>>
>> Average days in -next, for a fix, per -rc cycle:
>> rc1 27.25
>> rc2 21.4286
>> rc3 22.5114
>> rc4 18.281
>> rc5 14.65
>> rc6 12.6166
>> rc7 8.70408
>>
>> Fixes for bugs not introduced in current merge window:
>> rc1 40
>> rc2 113
>> rc3 61
>> rc4 79
>> rc5 25
>> rc6 139
>> rc7 72
>>
>> So for some reason, there is a rush to push fixes for older bugs (that
>> were not introduced in the current merge window) to the point that rc7
>> commits that only existed for a few days are merged in to address older
>> bugs.
>
>IMHO it's because it's the time it takes for users to start to trust the
>3rd or 4th stable release of the previous version, to switch to it, to
>face a bug, to report it, and for the maintainer to write a fix.
>
>I wouldn't be much surprised if you'd find that among those not introduced
>in the current merge window, many were introduced in the previous release.

Interesting. Here it is for v4.16-rcX fixes that fix something
introduced before v4.14:

rc1 30
rc2 87
rc3 51
rc4 68
rc5 23
rc6 113
rc7 61

So I'm not sure if what you described is really the case.
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Willy Tarreau]

On Thu, May 03, 2018 at 03:01:08PM +0000, Sasha Levin wrote:
> On Thu, May 03, 2018 at 04:52:05PM +0200, Willy Tarreau wrote:
> >I wouldn't be much surprised if you'd find that among those not introduced
> >in the current merge window, many were introduced in the previous release.
> 
> Interesting. Here it is for v4.16-rcX fixes that fix something
> introduced before v4.14:
> 
> rc1 30
> rc2 87
> rc3 51
> rc4 68
> rc5 23
> rc6 113
> rc7 61
> 
> So I'm not sure if what you described is really the case.

This is rather interesting and probably deserves some analysis or
explanation. I agree that probably a number of the 61 fixes in rc7
could have cooked a little bit more if they fixed 5 months-old bugs.

Willy

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin]

[-- Attachment #1: Type: text/plain, Size: 1253 bytes --]

On Thu, May 03, 2018 at 06:00:46PM +0200, Willy Tarreau wrote:
>On Thu, May 03, 2018 at 03:01:08PM +0000, Sasha Levin wrote:
>> On Thu, May 03, 2018 at 04:52:05PM +0200, Willy Tarreau wrote:
>> >I wouldn't be much surprised if you'd find that among those not introduced
>> >in the current merge window, many were introduced in the previous release.
>>
>> Interesting. Here it is for v4.16-rcX fixes that fix something
>> introduced before v4.14:
>>
>> rc1 30
>> rc2 87
>> rc3 51
>> rc4 68
>> rc5 23
>> rc6 113
>> rc7 61
>>
>> So I'm not sure if what you described is really the case.
>
>This is rather interesting and probably deserves some analysis or
>explanation. I agree that probably a number of the 61 fixes in rc7
>could have cooked a little bit more if they fixed 5 months-old bugs.

I tried looking at a few commits that came in on -rc7, and I see quite a
few cases where a commit was merged to Linus' tree in about 24 hours
after it was authored. Or maintainers who just wrote it, pushed it in,
and shipped in to Linus.

I've attached the data I used. The columns are as follows:

1. Commit ID
2. When was it merged
3. How many days it spent in -next
4. What commit did it fix
5. When was that commit merged

[-- Attachment #2: 416_fixes.txt --]
[-- Type: text/plain, Size: 71347 bytes --]

b6cdbc85234b v4.16-rc7 5 ca254490c8df v4.3
82dd0d2a9a76 v4.16-rc7 5 8f58336d3f78 v4.2
5807b22c9164 v4.16-rc7 5 6c8702c60b88 v4.9
f97c3dc3c0e8 v4.16-rc7 5 4c4dbb4a7363 v4.15
c769accdf3d8 v4.16-rc7 5 cbe7128c4b92 v4.16-rc6
ae4745730cf8 v4.16-rc7 5 4bbb3e0e8239 v4.16-rc6
da5dadb4f116 v4.16-rc7 5 519049afead4 v4.16-rc4
e457edf0b21c v4.16-rc7 5 e8f74a0f0011 v4.16-rc5
d65026c6c62e v4.16-rc7 5 6b1e6cc7855b v4.7
87a73eb5b56f v4.16-rc7 5 5c9c11e1c47c pre-v3.0
2b1b1b4ac716 v4.16-rc7 5 f88fc122cc34 v4.11
f03dbb06dc38 v4.16-rc7 5 009f766ca238 v4.16-rc4
2d2d99ec13f6 v4.16-rc7 5 55d7de9de6c3 v4.2
28913ee8191a v4.16-rc7 5 cc2d58634e0f v4.15
3d942ee079b9 v4.16-rc7 10 31383c6865a5 v4.15-rc1
880cd276dff1 v4.16-rc7 10 510ded33e075 v4.10
bd47a85acd72 v4.16-rc7 5 673aa20c55a1 v4.15
17a1079d9c63 v4.16-rc7 6 4f2f61fc5071 v4.16-rc3
151ad17fbe5e v4.16-rc7 5 dc20b2d52653 v4.13
38759d6175d3 v4.16-rc7 6 a5073d6054f7 v4.15
c8d3bcbfc5ea v4.16-rc7 6 75216638572f pre-v3.0
4b658d1bbc16 v4.16-rc7 6 75216638572f pre-v3.0
9137108cc3d6 v4.16-rc7 6 5fff41e1f89d v4.13-rc4
dc6455a71c7f v4.16-rc7 7 2b8b328b61c7 v3.8-rc6
461d5f1b5949 v4.16-rc7 7 78efed275117 v4.4
6e8814ceb7e8 v4.16-rc7 7 af7d51852631 v4.7
ab6f6dd18ab8 v4.16-rc7 7 d63d271ce2b5 v4.15
cd00edc17986 v4.16-rc7 7 43a0c6751a32 v4.8
1dfe82ebd7d8 v4.16-rc7 7 19acc327258a v3.9
5607dddbfca7 v4.16-rc7 7 02a5d6925cd3 v4.16
a117f73dc243 v4.16-rc7 7 610e89e05c3f v4.10-rc2
423c9db29943 v4.16-rc7 7 232c001398ae v4.11
af1607c37d9d v4.16-rc7 7 232c001398ae v4.11
4246f698dd58 v4.16-rc7 7 cb67b832921c v4.7
b392a2078b5e v4.16-rc7 7 f62b8bb8f2d3 v4.1
f125376b06bc v4.16-rc7 7 e80541ecabd5 v4.13
5ecadff0b6cf v4.16-rc7 7 cb67b832921c v4.7
84c9c8f2ac3c v4.16-rc7 7 20a1ea674783 v4.9
aaa3149bbee9 v4.16-rc7 8 030881372460 v4.5
982cf3b3999d v4.16-rc7 8 cb41c997d444 v4.6-rc1
9f5a90c10774 v4.16-rc7 8 7e1a1ac1fbaa pre-v3.0
ae42cc62a9f0 v4.16-rc7 8 1f718f0f4f97 v3.12
5c78f6bfae2b v4.16-rc7 8 1ff412ad7714 v3.11
eb82a9944792 v4.16-rc7 8 c5ad119fb6c0 v4.15
9066ae7ff5d8 v4.16-rc7 8 657b1989dacf pre-v3.0
10b8a3de603d v4.16-rc7 8 1da177e4c3f4 pre-v3.0
7880287981b6 v4.16-rc7 8 1da177e4c3f4 pre-v3.0
e69647a19c87 v4.16-rc7 8 55d7de9de6c3 v4.2
bc58a1baf2a9 v4.16-rc7 8 60e2a7780793 v4.14
52396500f97c v4.16-rc7 9 f0f558b131db v4.8
cfb2f6f6e0ba v4.16-rc6 7 36735a6a2b5e v4.15
32c1733f0dd4 v4.16-rc7 9 a583636a83ea v4.5
e8a4796ee295 v4.16-rc7 9 9314c442d7dd v4.14
4b6eca9d6817 v4.16-rc7 9 831a0aafe5c3 v4.14
ad43fc9aa24e v4.16-rc7 7 a8e53c151fe7 v4.15
1328f02005bb v4.16-rc7 7 e5b61bafe704 v4.7
771b7bf05339 v4.16-rc7 9 463a9215f3ca v4.14-rc3
a752c0a45248 v4.16-rc7 7 be7af5cf9cae v3.2
f8fb3419ead4 v4.16-rc7 7 1d8ab8d3c176 v3.14
06ace26f4e6f v4.16-rc6 7 d9e9a6418065 v4.15-rc5
1bfa26ff8c4b v4.16-rc7 9 c757faa8bfa2 v4.14
9d1887ef3252 v4.16-rc7 9 8665c3f97320 v4.15
68c3cd925597 v4.16-rc7 9 0063587d3587 v4.14
22b99058dd79 v4.16-rc7 9 0063587d3587 v4.14
f6cc9c054e77 v4.16-rc7 9 1da177e4c3f4 pre-v3.0
a5d4b5891c2f v4.16-rc7 9 1a472c9dba6b v4.6
9a3fb9fb84cc v4.16-rc7 5 acf568ee859f v4.15-rc5
80a4ae202f2d v4.16-rc7 9 03b8abedf4f4 v4.14
cda4a1473313 v4.16-rc7 9 6964e6a4e489 v4.15
9d3c3354bb85 v4.16-rc6 6 2516035499b9 v4.7
1c610d5f93c7 v4.16-rc6 6 bbef938429f5 v4.10
f59f1caf72ba v4.16-rc6 6 b92df1de5d28 v4.10
b3cd54b257ad v4.16-rc6 6 779750d20b93 v4.7
fa41b900c30b v4.16-rc6 6 9a982250f773 v4.4
fece2029a9e6 v4.16-rc6 6 b1caa957ae6d v4.4
28ee90fe6048 v4.16-rc6 12 e61ce6ade404 v4.0
b6bdb7517c3d v4.16-rc6 12 e61ce6ade404 v4.0
1705f7c53416 v4.16-rc6 13 101110f6271c v4.16-rc2
63489f8e8211 v4.16-rc6 13 045c7a3f53d9 v4.11-rc6
2e517d681632 v4.16-rc6 16 d92a8cfcb37e v4.13
8970a63e965b v4.16-rc6 14 fc36b8d3d819 pre-v3.0
ff6781fd1bb4 v4.16-rc7 9 1d607bb3bd60 v4.7
5dcd8400884c v4.16-rc6 6 0759e552bce7 v4.7
40013ff20b1b v4.16-rc6 6 98cd1552ea27 v4.11
7b2ee50c0cd5 v4.16-rc6 6 8195b1396ec8 v4.13
0ef58b0a05c1 v4.16-rc6 6 0cf737808ae7 v4.14
02400fcee254 v4.16-rc6 6 0cf737808ae7 v4.14
8348e0460ab1 v4.16-rc6 6 76bb5db5c749 v4.11
68e2ffdeb5db v4.16-rc6 6 fc1e64e1092f v4.15
6d066734e9f0 v4.16-rc6 6 55454a565836 v4.8
8936ef7604c1 v4.16-rc6 6 6c8702c60b88 v4.9
191f86ca8ef2 v4.16-rc6 6 6c8702c60b88 v4.9
0231b1a074c6 v4.16-rc6 6 b6b5e8a69118 v4.15-rc2
5df7af85ecd8 v4.16-rc6 6 b6b5e8a69118 v4.15-rc2
9a02d3af9ce9 v4.16-rc7 5 511c7023cf23 v4.15
aebfa52a925d v4.16-rc7 10 7b4fdf77a450 v4.11-rc3
88d42b2b45d7 v4.16-rc6 8 4a219ef8f370 v4.12
834814e80268 v4.16-rc6 7 2170dd04316e v4.15
a63d706ea719 v4.16-rc6 8 3af5a67c86a3 v4.5-rc3
891731f6a5db v4.16-rc6 8 c06e836ada59 v3.8
f29cdfbe33d6 v4.16-rc6 6 65a206c01e8e v4.13
d7f200157360 v4.16-rc6 6 4c5b9d9642c8 v4.14
1e46ef1762bb v4.16-rc6 6 65a206c01e8e v4.13
94fa3f929ec0 v4.16-rc6 6 65a206c01e8e v4.13
fe9a552e715d v4.16-rc6 7 96c3a239054a v4.15
5bf7f8185f7c v4.16-rc6 6 65a206c01e8e v4.13
60e10b3adc3b v4.16-rc6 6 65a206c01e8e v4.13
bbc09e7842a5 v4.16-rc6 6 65a206c01e8e v4.13
a821328c2f30 v4.16-rc6 8 2b6639d4c794 v4.13
3223a5a7d3a6 v4.16-rc6 8 dea54fbad332 v4.13
896196dc4e41 v4.16-rc6 7 96c3a239054a v4.15
caf61b1b8b88 v4.16-rc7 10 cecbcddf6461 v4.8
b15606f47b89 v4.16-rc7 10 ec72fce401c6 v4.8
c3594f22302c v4.16-rc7 10 cecbcddf6461 v4.8
5f3e3b85cc0a v4.16-rc7 10 6a21dfc0d0db v4.16-rc4
32927e281c00 v4.16-rc7 10 42cea83f9524 v4.16-rc6
3a088dd1b72d v4.16-rc6 16 14b730fcb8d9 v4.6
60b01bcce971 v4.16-rc6 7 7b6ddeaf27ec v4.15-rc1
d58ac803cfbb v4.16-rc6 7 1b7ba57ecc86 v4.15
b24791fe00f8 v4.16-rc6 7 308e5bcbdb10 v3.2
a8d7bde23e71 v4.16-rc6 9 e79b0006c45c v4.12-rc6
6b00c35138b4 v4.16-rc6 7 656441478ed5 v4.10
0f0e8de334c5 v4.16-rc7 12 cfe17c9bbe6a v4.15-rc7
843c3a59997f v4.16-rc6 7 bccb06c353af v4.15
fa8e6d58c5bc v4.16-rc6 7 82771882d960 v3.3
f005afede992 v4.16-rc6 7 4bebdc7a85aa v4.14
52fda36d63bf v4.16-rc6 7 09584b406742 v4.15
e8980d67d601 v4.16-rc6 7 75216638572f pre-v3.0
5f2fb802eee1 v4.16-rc6 8 2f987a76a977 v4.16-rc6
00777fac28ba v4.16-rc6 8 6eacf31139bf v3.17
467697d289e7 v4.16-rc7 12 8aeff920dcc9 v4.9
7fe4d6dcbcb4 v4.16-rc6 8 1555d204e743 v4.11
ae6153b50f9b v4.16-rc7 12 f92b40a8b264 v4.15
4f2921ca21b7 v4.16-rc7 12 6c03ae210ce3 v4.12
31ad7f8e7dc9 v4.16-rc6 8 49275fef986a v4.15-rc4
1bc659eb2369 v4.16-rc6 8 939d749ad664 v4.14
8250e6cadced v4.16-rc6 8 4b1c924b1fc1 v4.14
9b9322db5c5a v4.16-rc7 12 657308f73e67 v4.16-rc2
320b0651f32b v4.16-rc6 8 cd34cd97b7b4 v4.8
578ae447e7e5 v4.16-rc6 8 dc1dd184c2f0 v4.16-rc3
e5ea9b54a055 v4.16-rc6 8 294fe0f52a44 v4.0
2c2a9bbe7fec v4.16-rc6 26 b8241d20699e pre-v3.0
2399ac42e762 v4.16-rc6 8 a3995460491d v4.12
b1e314462bba v4.16-rc6 17 5ea2355a100a v4.14
b1abf6fc4982 v4.16-rc6 8 058dfc767008 v4.8
68ef3bc31664 v4.16-rc6 10 76d348fadff5 v4.8
80cf79ae4f68 v4.16-rc6 8 02d8883f520e v4.15
ed65a4dc2208 v4.16-rc6 8 75216638572f pre-v3.0
7997f3b2df75 v4.16-rc6 9 41691b8862e2 v4.3
a6618f4aedb2 v4.16-rc6 11 23caaf19b11e pre-v3.0
d418ff56b8f2 v4.16-rc6 9 9c7be59fc519 v4.16-rc6
3bf7b5d6d017 v4.16-rc6 9 b17e5729a630 v4.16-rc6
6de564939e14 v4.16-rc6 11 c2b78452a9db v4.6
4a6d2e525b43 v4.16-rc7 12 cf961e16620f v4.6
f8a554b4aa96 v4.16-rc7 13 b96f9afee4eb v4.9
7a67e69a339a v4.16-rc7 13 ed1efb2aefbb v3.12
c6741fbed6dc v4.16-rc7 13 53c81e95df17 v4.15
03080e5ec727 v4.16-rc7 13 1181412c1a67 v3.5
24fc79798b8d v4.16-rc7 13 c54419321455 v3.9
dd1df2473772 v4.16-rc7 13 b9959fd3b0fa v3.11
fc04fdb2c8a8 v4.16-rc6 6 a73105b8d4c7 v3.0
48881ed56b39 v4.16-rc6 6 a9a08845e9ac v4.15
034f40579389 v4.16-rc6 10 45dac1d6ea04 v4.1
2d433610176d v4.16-rc6 10 86da71b57383 v4.8
1f110e7cae09 v4.16-rc6 10 cadb9c9fdbc6 v4.10
abdadd3cfd3e v4.16-rc6 10 d0f6dd8a914f v4.8
aab378a7eda2 v4.16-rc6 10 9c5f69bbd75a v4.15
1edf8abe0409 v4.16-rc6 10 4c5b9d9642c8 v4.14
bcdd5de80a22 v4.16-rc6 10 117b0dad2d54 v4.9
cce6294cc2ea v4.16-rc6 10 c5ad119fb6c0 v4.15
6f27d2c2a8c2 v4.16-rc6 6 be1db4f6615b v3.12
bc44b78157f6 v4.16-rc6 6 7cdcf6dddc42 v3.7
7b7ef57c0dd2 v4.16-rc6 10 a3c53be55c95 v4.10
6e5d58fdc9be v4.16-rc6 10 1da177e4c3f4 pre-v3.0
02a2385f37a7 v4.16-rc6 10 cb9f7a9a5c96 v4.15
16da09047d3f v4.16-rc6 10 b5c29ca7dab7 v4.12
933e8c91b9f5 v4.16-rc6 10 c7d1d8399994 v4.14
484d802d0f2f v4.16-rc6 10 80105befdb4b v3.15
2cc683e88c0c v4.16-rc6 10 ab7ac4eb9832 v4.5
4bbb3e0e8239 v4.16-rc6 10 a6e18ff11170 v4.4-rc1
093e037ca88e v4.16-rc5 9 3c181c12c431 v4.16-rc3
86a2b2043af7 v4.16-rc7 14 c62446d2b028 v4.8
e829b17caf96 v4.16-rc7 14 85aeb58cec1a v4.12
46c0ef6e1eb9 v4.16-rc7 13 af5d27c4e12b v4.10
dc9e0a9347e9 v4.16-rc6 9 99759869faf1 v4.1
2975d5de6428 v4.16-rc6 10 75216638572f pre-v3.0
9e75dc61eaa9 v4.16-rc5 9 3c66c87dc96b v4.12
76f2e2bc627f v4.16-rc5 9 b53ac1ee12a3 v4.9
95dd77580ccd v4.16-rc5 9 397d425dc26d v4.2
cfb61b5e3e09 v4.16-rc5 9 a8e654f01cb7 v4.15
51d4740f88af v4.16-rc6 11 d0f6dd8a914f v4.8
8a53fc511c5e v4.16-rc6 10 15ed8ce5f84e v4.15
d90c76bb6112 v4.16-rc6 10 6671507f0fbd v4.15
48519232bea9 v4.16-rc6 5 ebae8d07435a v4.15
3d502067599f v4.16-rc6 11 127f49705823 v4.15
8a927d648c2e v4.16-rc6 5 ebae8d07435a v4.15
c658dc58c7ea v4.16-rc6 11 d2f82254e4e8 v4.13
ef75c685869e v4.16-rc5 9 0a53bc07f044 v4.14
b5b6e8c8d3b4 v4.16-rc7 6 84676c1f21e8 v4.15
adbe552349f2 v4.16-rc7 6 84676c1f21e8 v4.15
8b834bff1b73 v4.16-rc7 6 84676c1f21e8 v4.15
3e04040df6d4 v4.16-rc5 10 864b75f9d6b0 v4.16-rc4
bd8602ca42f6 v4.16-rc6 37 872f3578241d v4.15
5388a508479d v4.16-rc6 37 37cb11acf1f7 v4.15
a7e6c7015bf3 v4.16-rc6 12 24b6d4164348 v4.15
18bf591ba975 v4.16-rc5 14 3ec4d3238ab1 v4.13
047fdea63419 v4.16-rc5 14 74d46992e0d9 v4.13
42cea83f9524 v4.16-rc6 11 16c1975f1032 v4.15
b5e324131697 v4.16-rc5 18 4ec6ecf48c64 v4.15
0c81ffc60d52 v4.16-rc6 11 5bc2b7b397b0 v3.10
7688f2c3bbf5 v4.16-rc6 11 c8f6a362bf3e pre-v3.0
9dea9a2ff61c v4.16-rc6 11 19b752a19dce v4.10
8c5c147339d2 v4.16-rc5 11 e2460f2a4bc7 v4.11
f3f134f5260a v4.16-rc6 11 e126ba97dba9 v3.10
18a955219bf7 v4.16-rc5 11 f4eafd8bcd52 v4.5-rc4
d0264c01e758 v4.16-rc5 10 db446a08c23d v3.11
a6d7cff472ee v4.16-rc5 10 e34ecee2ae79 v3.12
27e91ad1e746 v4.16-rc5 10 f68d2b1b73cc v4.4
16ca6a607d84 v4.16-rc5 10 0919e84c0fc1 v4.6
413aa807ae39 v4.16-rc5 10 4c60e360d6df v4.15
e21a4f3a930c v4.16-rc5 25 9b062471e52a v4.15
62b06f8f429c v4.16-rc5 10 96b298000db4 v4.6
ea91df6d8a55 v4.16-rc6 12 506b0a395f26 v4.16-rc2
d52e5a7e7ca4 v4.16-rc6 12 2c8cec5c10bc pre-v3.0
4dcb31d4649d v4.16-rc6 12 ca6fb0651883 v4.3
af1d830bf32b v4.16-rc5 11 dc1dd184c2f0 v4.16-rc3
9bdc00a5c16f v4.16-rc7 7 73102d6fdc32 v4.15
b50697824534 v4.16-rc5 11 5ed92a8ab71f v4.2
c2b37f76485f v4.16-rc6 12 e126ba97dba9 v3.10
2c292dbb398e v4.16-rc6 12 0fb2ed66a14c v4.4
c37366742baa v4.16-rc5 13 8d47e65948dd v4.16-rc4
f89782c2d131 v4.16-rc6 13 9de506a547c0 v4.16-rc6
5a9f8df68ee6 v4.16-rc6 13 4a5b85ffe2a0 v4.14
55c19eee3b47 v4.16-rc6 13 81ac38847a1d v4.15
9903e41ae1f5 v4.16-rc6 13 4f16f7ff3bc0 v4.15
04bf9ab3359f v4.16-rc6 14 0f6cc2b8e94d v4.15
b954f94023dc v4.16-rc6 14 3557baabf280 pre-v3.0
f1430f145eef v4.16-rc5 22 29ecd78d3b79 v4.8
82813ba9b4b3 v4.16-rc5 22 e54ca9774777 v4.11
99652a469df1 v4.16-rc6 14 f8f8f1d04494 v4.15
b6b76dd62c56 v4.16-rc6 14 4b1a29a7f542 v4.15
3c4fe80b32c6 v4.16-rc6 14 c0c050c58d84 v4.3
6ae777eab2f5 v4.16-rc6 14 db1d36a27324 v4.13
832aed16ce7a v4.16-rc6 14 674f50a5b026 v4.15
b9ecc3400bc4 v4.16-rc6 14 d7bc73053024 v4.13
6fc2ffdf1001 v4.16-rc6 14 8f23d638b36b v4.15
40088dc4e1ea v4.16-rc5 13 1ba8f9d30817 v4.16-rc3
9ca8614980d4 v4.16-rc6 14 070badfab767 v4.14
ddc502dfed60 v4.16-rc6 14 96fbc13d7e77 v4.14
13fbcc8dc573 v4.16-rc6 14 d77e38e612a0 v4.11
7f95beea3608 v4.16-rc6 14 9e4d04adeb1a v4.15
932909d9b28d v4.16-rc6 14 b71812168571 v4.16-rc4
3c3e4b3a708a v4.16-rc6 11 3adbf3427330 v4.10
ac68b1b3b9c7 v4.16-rc4 15 d9c6a72d6fa2 v4.12
864b75f9d6b0 v4.16-rc4 15 b92df1de5d28 v4.10
96312e61282a v4.16-rc4 15 ce53053ce378 v4.15
1b4cfe3c0a30 v4.16-rc4 15 b8347c219649 v4.14-rc8
0862ca422b79 v4.16-rc4 15 ad67b74d2469 v4.15-rc1
4704dea36dd9 v4.16-rc4 25 0c397daea1d4 v4.15
28e9091e3119 v4.16-rc4 25 bde51583f49b v3.13
7832f6d12fa2 v4.16-rc5 13 cf140a356971 v4.15
49bae2f3093b v4.16-rc6 14 b2925957ec1a v4.14
df3334c223a0 v4.16-rc5 13 ea6873a45a22 v4.6
e21da1c99200 v4.16-rc4 12 b092201e0020 v4.15
d56e57ca030c v4.16-rc5 13 835e4241e714 v4.11
3a4030761ea8 v4.16-rc6 14 fc72d1d54dd9 v4.15
303fd71b37fb v4.16-rc6 14 c67df11f6e48 v4.12
ab7e34b3431c v4.16-rc6 14 c67df11f6e48 v4.12
ca0edb131bdf v4.16-rc6 14 2c88b5283f60 v4.3
6b0ef92fee2a v4.16-rc4 12 02a7c234e540 v4.14
c4dc56be7e26 v4.16-rc7 18 c69f43fb4f26 v4.9
3f553b308bb0 v4.16-rc6 15 516fb7f2e73d v4.14
c4f24df942a1 v4.16-rc5 22 dc4fd9ab01ab v4.15-rc3
b0dee7910317 v4.16-rc6 15 bee9d41b37ea v4.16-rc4
35a57b7fef13 v4.16-rc6 15 bee9d41b37ea v4.16-rc4
de3d50aadd40 v4.16-rc6 15 009f766ca238 v4.16-rc4
a098dc8b03d1 v4.16-rc5 14 dfba2174dc42 v4.15
97ef0faf575e v4.16-rc5 14 19a7d0d65c4a v4.11
cbcc607e1842 v4.16-rc6 15 9b00cf2d1024 v3.5
55fe6da9efba v4.16-rc4 13 695835511f96 v4.8
61fc470814d8 v4.16-rc4 13 192efb7a1f9b v4.14
93b0beae721b v4.16-rc6 21 30574f0db1b1 v3.6
625504aeff11 v4.16-rc6 21 82d2de5a4f64 v4.15
7bd800915677 v4.16-rc4 13 d15d662e89fc v4.16-rc1
970656b3246d v4.16-rc7 18 b877039aa1fe v4.12
d85739367c6d v4.16-rc4 13 d15d662e89fc v4.16-rc1
a5880b844303 v4.16-rc4 13 75216638572f pre-v3.0
6a21dfc0d0db v4.16-rc4 13 7ce86409adcd pre-v3.0
bb7f8f199c35 v4.16-rc4 13 200298326b27 v4.4
e05836ac07c7 v4.16-rc6 16 f214f915e7db v4.13
17cfe79a65f9 v4.16-rc6 16 fd558d186df2 pre-v3.0
e9fa1495d738 v4.16-rc6 16 fb56be83e43d v4.9-rc5
9de506a547c0 v4.16-rc6 16 dbb799c39717 v4.7
fc110ebdd014 v4.16-rc4 14 a20c7f36bd3d v4.14
2695578b896a v4.16-rc6 16 c8b5d129ee29 v4.11
35d889d10b64 v4.16-rc6 16 6071bd1aa13e v4.6-rc6
d3dcf8eb6155 v4.16-rc6 16 ca26893f05e8 v4.8
dbe8a9dfd7ea v4.16-rc7 20 2b05f6ae1ee5 v4.16-rc2
25b5cdfcce1b v4.16-rc6 16 785ec87483d1 v4.14
933897342d07 v4.16-rc6 16 b69c1df47281 v4.15
87cdf3148b11 v4.16-rc6 16 5efec5c655dd v4.15-rc9
63474dc4ac7e v4.16-rc4 12 b5bc2231b8ad v4.16-rc3
d3b9e8ad425c v4.16-rc4 14 246d8b184c10 v4.15
5d414b178e95 v4.16-rc4 14 427c1e7bcd7e v4.4
210b1f78076f v4.16-rc4 14 32f69e4be269 v4.15
a18177925c25 v4.16-rc4 14 b699a859d17b v4.4
0077416a3d52 v4.16-rc4 14 7e57b85c444c v4.4
6a2cf8d3663e v4.16-rc5 12 d64d6c5671db v4.15
8d47e65948dd v4.16-rc4 16 848b8aefd44d v4.15
99243b922c9d v4.16-rc4 16 22c11858e800 v4.15
da1e148803e0 v4.16-rc4 20 7c29744eccec v4.15
590347e40003 v4.16-rc4 23 41acec624087 v4.15
167f5594b5ef v4.16-rc6 12 e8d513483300 v4.15
655296c8bbef v4.16-rc6 19 03bad714a161 v4.13
6f566af34628 v4.16-rc5 16 57e6f0d7b804 v4.15
cb88a0588717 v4.16-rc5 16 de3af5bf259d v4.13
9a513c905bb9 v4.16-rc5 16 cbeef22fd611 v4.15
8a17eefa235f v4.16-rc6 17 820ee17b8d3b v4.9
55f53567afe5 v4.16-rc6 17 2b2427d06426 v4.4
98409b2bbca3 v4.16-rc6 17 d2fa47d9dd5c v4.4
cd526676de1e v4.16-rc6 17 967dd82ffc52 v4.7
de19e5c3c51f v4.16-rc4 12 3dcc4436fa6f v4.6
40c21898ba53 v4.16-rc4 12 92a61f6412d3 v4.5
f07afa0462b7 v4.16-rc4 15 a6940674c384 v4.8
b0c41b8b6e43 v4.16-rc4 15 02ef6dd8109b v4.15
0d3601d2994a v4.16-rc6 17 6c03ae210ce3 v4.12
d3f468963cd6 v4.16-rc4 14 7041d28115e9 v4.15
859d880cf544 v4.16-rc4 14 b68a68d3dcc1 v4.15
88d3dfb6a690 v4.16-rc4 35 af7a8ffad9c5 v4.14
f616f2830c1e v4.16-rc4 36 701f8231a2fe v4.13
a6f1086e29e9 v4.16-rc4 15 4670d610d592 v4.15
06a3f0c9f272 v4.16-rc4 15 a3e6c1eff548 v3.19-rc7
bd5edbe67794 v4.16-rc4 15 94a47083522e pre-v3.0
4e7dc08e57c9 v4.16-rc6 16 19110cfbb34d v4.14
361a954e6a72 v4.16-rc6 16 4aea7a5c5e94 v4.14
745d0bd3af99 v4.16-rc6 16 4aea7a5c5e94 v4.14
626af862da9c v4.16-rc4 30 ebebbf082357 v4.16-rc1
ee2515d95f9a v4.16-rc4 18 7a407aa5e0d3 v4.15
6cfc70c4321b v4.16-rc4 17 a211a0820d3c v4.15
c7151602255a v4.16-rc6 18 9d9491a7da2a v4.14-rc7
79f3a8e662c1 v4.16-rc4 16 cf797ac49b94 v4.14
0a8a1bf17e3a v4.16-rc4 16 56ade8fe3fe1 v4.3
74c12c630fe3 v4.16-rc6 6 687937ab3489 v4.7
317660940fd9 v4.16-rc4 17 cd34cd97b7b4 v4.8
d02f51cbcf12 v4.16-rc6 19 6578171a7ff0 v4.7
61bd0f66ff92 v4.16-rc4 15 8b24e69fc47e v4.12
949b93250a56 v4.16-rc3 15 71389703839e v4.11
5fdf8e5ba566 v4.16-rc3 15 06e8ccdab15f v4.15
94db151dc892 v4.16-rc3 20 d475c6346a38 v3.19
ba004a2955f7 v4.16-rc4 17 16c513b13477 v4.16-rc3
84eef2b2187e v4.16-rc4 17 0933a578cd55 v4.12
61e18270f604 v4.16-rc4 14 b4e98d9ac775 v4.14
d5078193e56b v4.16-rc4 17 3f2f7c553d07 v4.16-rc1
e4f13ad07823 v4.16-rc6 17 85aeb58cec1a v4.12
63dd5d022f47 v4.16-rc6 17 26d6c16bed53 v4.11
40d53f4a60c9 v4.16-rc6 17 f5d8f50f271d v4.13
a6d50512b4d8 v4.16-rc4 17 1a5f3da20bd9 v4.13
2b5b96473efc v4.16-rc4 16 a4239945b8ad v4.15
e39a97353e53 v4.16-rc4 16 2a842acab109 v4.12
3be8828fc507 v4.16-rc4 16 3bd6f43f5cb3 v4.15
fa83e65885b9 v4.16-rc4 16 3dbec59bdf63 v4.15
07ea4b6026ee v4.16-rc4 16 040036bb0bc1 v4.15
1c6cacf4ea6c v4.16-rc4 16 d8630bb95f46 v4.15
b7e31be38558 v4.16-rc3 15 0b2e9904c159 v4.16-rc3
f0e8c61110c2 v4.16-rc6 16 1fdb92697469 v4.16-rc4
773daa3caf5d v4.16-rc4 18 c7272c2f1229 v4.16-rc4
e82df6702351 v4.16-rc4 18 780bc7903a32 v4.5
158e61865a31 v4.16-rc3 15 74d46992e0d9 v4.13
9c0fb1e313aa v4.16-rc3 15 74d46992e0d9 v4.13
7c5a0dcf557c v4.16-rc3 15 74d46992e0d9 v4.13
f3bc78d2d4b4 v4.16-rc3 15 5700f69178e9 v4.15
3c181c12c431 v4.16-rc3 24 df93589a1737 v3.15
92e222df7b8f v4.16-rc3 24 73c5de005153 pre-v3.0
09a0fb67536a v4.16-rc4 16 d8346b7d9bab v3.8
38c08aa54b50 v4.16-rc3 44 ef884112e55c v4.12
b758dbd57650 v4.16-rc3 44 91f9e850d465 v4.12
945fd17ab6ba v4.16-rc3 15 92a0f81d8957 v4.15-rc4
b8b549eec818 v4.16-rc6 19 d7dbefc45cf5 v4.11
c52232a49e20 v4.16-rc3 15 a683f390b93f v4.7
c2d2e6738a20 v4.16-rc3 17 5b5971df3bc2 v4.15-rc1
910f8befdf5b v4.16-rc3 17 4892c9b4ada9 v3.14
fa89782b4f9c v4.16-rc3 46 fe49789fab97 v4.11
e659d14ed480 v4.16-rc3 49 c7c6e46f913b v4.13
1b0008450f23 v4.16-rc3 50 04416108ccea v4.12
4cd482c12be4 v4.16-rc4 18 200298326b27 v4.4
72a6d72c2cd0 v4.16-rc3 51 9965db26ac05 v4.15
2fb4f4eadd18 v4.16-rc4 18 4be3a4fa51f4 v4.11
65389322b28f v4.16-rc4 18 62ede7779904 v4.13
e7b169f34403 v4.16-rc4 18 09a7d9eca1a6 v4.8
da343b6d90e1 v4.16-rc4 18 b005d3164713 v4.5
b1b13780ab06 v4.16-rc3 52 fec0445caa27 v4.11
8a949fff0302 v4.16-rc4 19 be7be6e161a2 v4.11
b3529af6bb0d v4.16-rc4 19 c57529e1d5d8 v4.12
9d45deb04c59 v4.16-rc4 19 9df552ef3e21 v4.13
77d270967c5f v4.16-rc4 19 ef3116e5403e v4.15
2ddc94c76cc4 v4.16-rc4 19 5f57e0909136 v4.13
a5dcb73b96a9 v4.16-rc4 19 cd6851f30386 v4.10
4e09ff536284 v4.16-rc4 19 4941d472bf95 v4.13
028091f82eef v4.16-rc3 15 bda7b072de99 v4.8
a27fd7a8ed38 v4.16-rc4 19 f214f915e7db v4.13
fc68e171d376 v4.16-rc4 19 89fe18e44f7e v4.10
d4131f09770d v4.16-rc4 19 cc663f4d4c97 v4.11-rc6
c5c48c58b259 v4.16-rc4 19 5f78e29ceebf v4.7
4964c66fd49b v4.16-rc4 19 cb816192d986 v4.13
14d066c3531a v4.16-rc4 19 5f78e29ceebf v4.7
12472af89632 v4.16-rc4 19 2863c61334aa v4.7
b08e5fd90bfc v4.16-rc4 18 6de3f79112cc v4.16-rc2
5753405e27f8 v4.16-rc3 15 e07127a077c7 v4.13
5d7f77ec72d1 v4.16-rc5 20 b8f3bff057b0 v4.15
1f66dd36bb18 v4.16-rc5 49 088da2a17619 v4.5
e7f3e99cb1a6 v4.16-rc5 20 7d8905d06405 v4.13
084a804e0120 v4.16-rc5 17 41ce1456e1db v4.11
c77a6edb6d4d v4.16-rc3 17 b4eb0d522fcb v4.12
350144069abf v4.16-rc3 17 b4eb0d522fcb v4.12
5a23699a39ab v4.16-rc3 17 becf9e5d553c v4.13
0a5aff64f20d v4.16-rc3 16 c53beb47f621 v4.8
0979962f5490 v4.16-rc3 16 e46c7287b1c2 v4.11
c113187d38ff v4.16-rc4 20 3c4d7559159b v4.12
ffc2b6ee4174 v4.16-rc4 20 c54419321455 v3.9
9c2c2e62df3f v4.16-rc4 20 f5e64032a799 v4.15-rc3
1b22bcad7e39 v4.16-rc4 20 60c253069632 v4.15
0373ca74831b v4.16-rc3 15 a307a1e6bc0d v3.12
3d18e4f19f37 v4.16-rc4 20 d9f9b9a4d05f v4.15
b9d17175aeb9 v4.16-rc4 20 d9f9b9a4d05f v4.15
3a574919f0cc v4.16-rc3 18 81196976ed94 v4.15
0d84b9e5631d v4.16-rc3 18 800d78bfccb3 v3.6
a4faa4929ed3 v4.16-rc3 18 800d78bfccb3 v3.6
325501d9360e v4.16-rc3 18 361c7fe9b02e v4.13
c14376de3a1b v4.16-rc3 18 dbdda842fe96 v4.15
6662ae6af82d v4.16-rc3 20 c85823390215 v4.15
d269176e766c v4.16-rc4 20 ce0761419fae v4.8
5c8b2623f6b4 v4.16-rc3 15 343a8d17fa8d v4.15
9326fdf3fbdf v4.16-rc3 15 343a8d17fa8d v4.15
c37406e05d1e v4.16-rc3 16 8bb705e3e79d v4.14
29d1d52b06fb v4.16-rc3 16 dd5c0561db75 v4.15
133390fe497b v4.16-rc3 16 a60d64b15c20 v4.15
230f5a8969d8 v4.16-rc3 20 2bb6d2837083 v4.15-rc1
0e5a82efda87 v4.16-rc4 21 251277598596 v4.12
1bb4f2e868a2 v4.16-rc4 21 761876c857cb v4.13
23e43f07f896 v4.16-rc4 21 761876c857cb v4.13
f249be4d2c27 v4.16-rc4 21 762c330d670e v4.15
0c5661ecc5dd v4.16-rc4 21 42073d91a214 v3.6
13a55372b64e v4.16-rc4 21 4904dbda41c8 v4.3
28f5bfb81919 v4.16-rc4 21 f8ccac0e4493 v3.8
d02ba2a6110c v4.16-rc4 21 ee40fb2e1eb5 v4.14
d00fa9adc528 v4.16-rc4 21 80d84ef3ff1d v3.8-rc6
225eb26489d0 v4.16-rc4 21 fd558d186df2 pre-v3.0
76a6abdb2513 v4.16-rc4 21 309795f4bec2 pre-v3.0
d52987b524cc v4.16-rc3 16 8ddcd653257c v4.14
68d2059be660 v4.16-rc3 19 5db4d286a8ef v4.13
d9c10e5b8863 v4.16-rc3 16 9830f4be159b v4.12
d1fe96c0e4de v4.16-rc4 21 438c84c2f0c7 v4.15-rc3
71db96ddfa72 v4.16-rc3 19 61fcf8ece9b6 v4.16-rc1
753e8abc36b2 v4.16-rc4 18 4e6020565596 v4.15
f287eb9013cc v4.16-rc3 15 2529c3a33079 v3.15
b5095f24e791 v4.16-rc4 21 4b91c30a5a19 v4.15
5a3386790a17 v4.16-rc5 22 7d2993811a1e v4.13
192b2e742c06 v4.16-rc4 16 a08082f8e4e1 v4.15
ab2f336cb7e6 v4.16-rc4 20 3e81a4ca51a1 v4.15
7d98386d55a5 v4.16-rc4 21 ca6fb0651883 v4.3
f22e08932c29 v4.16-rc4 17 c6c8fea29769 pre-v3.0
fce672db548f v4.16-rc4 17 ea4152e11716 v4.8
b0264ecdfeab v4.16-rc4 17 04f3f5bf1883 v4.8
011c935fceae v4.16-rc4 17 b71bb6f924fe v4.8
10d570284258 v4.16-rc4 17 efb766af06e3 v4.8
b078556aecd7 v4.16-rc4 21 58a317f1061c v3.6
3bf2a09da956 v4.16-rc4 17 610bfc6bc99b v3.12
abd6360591d3 v4.16-rc4 17 c6c8fea29769 pre-v3.0
de9647efeaa9 v4.16-rc3 48 30323fb6d552 v4.15
105976f51779 v4.16-rc3 16 bd166ef183c2 v4.10
6de2aeb5746e v4.16-rc6 11 af11143757b7 v4.15
179858efd94f v4.16-rc6 11 bed73904e76f v4.15
c278609bdde9 v4.16-rc6 11 e2e6771c6462 v4.15
4e4f9fbc5620 v4.16-rc6 11 bed73904e76f v4.15
9c5e0afaf157 v4.16-rc3 20 9f5b5b950aa9 v4.15
3e233385ef4a v4.16-rc3 20 0d0736f76347 v4.15
45d0be876308 v4.16-rc3 44 1d57b17c60ff v4.15
fe2a3027e74e v4.16-rc3 20 52a5c155cf79 v4.12
103c763c72dd v4.16-rc3 20 f0d648bdf0a5 v4.3-rc5
99158246208b v4.16-rc3 20 0367f205a3b7 v4.15
0b2e9904c159 v4.16-rc3 20 4191db26b714 v4.14-rc7
ca36960211eb v4.16-rc4 22 17a5267067f3 v3.17
9f2c4d95e088 v4.16-rc3 16 2f330caff577 v4.14
69d7d95452b8 v4.16-rc3 16 2f330caff577 v4.14
a5f7add332b4 v4.16-rc2 17 1c0d32fde5bd v4.9
4e14bf423649 v4.16-rc2 17 d02fd6e7d293 v4.15-rc7
bf8486709ac7 v4.16-rc3 20 3286d3eb906c v4.12
0066c764e7cc v4.16-rc3 20 03fbdb2fc2b8 v4.15
902f4d067a50 v4.16-rc4 22 64b139f97c01 v3.19
1b22b4b28fd5 v4.16-rc4 22 a7473717483e v3.18
ecb586bd29c9 v4.16-rc3 20 d28b387fb74d v4.15
e84cf6aa501c v4.16-rc2 18 69cde0004a4b v4.14
eb0a2d2620ae v4.16-rc2 18 6e032b350cd1 v4.15-rc7
582605a429e2 v4.16-rc2 18 8989d56878a7 v4.15-rc7
2f7d03e05119 v4.16-rc2 18 2b31e3aec1db v4.15
0f9da844d877 v4.16-rc2 16 28128c61e08e v4.16-rc2
16338a9b3ac3 v4.16-rc2 17 ddb55992b04d v4.7
af1da6868437 v4.16-rc3 20 3aaabbf1c39e v4.14
651ca2c00405 v4.16-rc2 18 2f75d9e1c905 v4.14
93c62c45ed5f v4.16-rc2 18 5a924b8951f8 v4.8
370c10522e96 v4.16-rc2 18 23ee07ad3c2f v4.15
6c5f61023c5b v4.16-rc2 18 9a3efb6b661f v4.16-rc2
32fff239de37 v4.16-rc2 18 a10423b87a7e v4.5
657308f73e67 v4.16-rc2 18 73d54c9e74c4 pre-v3.0
a2c0f039bbd0 v4.16-rc2 18 34f0f4e3f488 v4.16-rc2
1fe4b1184c2a v4.16-rc2 18 bf4e0a3db97e v4.11
350c9f484bde v4.16-rc2 18 0f8782ea1497 v4.8
b87b6194be63 v4.16-rc2 18 41c87425a1ac v4.14-rc4
28128c61e08e v4.16-rc2 17 3859a271a003 v4.13-rc1
ad86f605c595 v4.16-rc2 19 52442f9b11b7 v4.8-rc4
d9f4bb1a0f4d v4.16-rc2 17 13100a72f40f v4.6
4b34968e77ad v4.16-rc2 17 a511e1af8b12 v4.6
437499eea429 v4.16-rc2 17 6c2dc5ae4ab7 v4.6
6459ae386699 v4.16-rc2 17 757932e6da6d v3.17
29f4a67c17e1 v4.16-rc2 17 03bb79315ddc v4.11
971b42c038dc v4.16-rc2 17 6c2dc5ae4ab7 v4.6
15122ee2c515 v4.16-rc2 17 324420bf91f6 v4.5
d1fa74520dcd v4.16-rc2 17 ca382f5b38f3 v4.15
7229b12f5da3 v4.16-rc3 21 8a2d6ae1f737 v4.12
dc24b7b49a53 v4.16-rc4 23 f19fbd5ed642 v4.15
5a1e59533380 v4.16-rc3 16 0de5cd367c6a v4.15
083b20907185 v4.16-rc2 19 eb84bab0fb38 v4.0
c7a3275e0f9e v4.16-rc2 19 02ef6dd8109b v4.15
5539d31a04b3 v4.16-rc2 19 3f38000eda48 v4.15
a275b315334d v4.16-rc6 25 59dc3d8c8673 v4.15
7801c545e706 v4.16-rc3 21 721cabf6c660 v4.11
a7dcdf6ea1b2 v4.16-rc2 18 b13d88072172 v4.15
895f7b8e9020 v4.16-rc2 18 f7f99100d8d9 v4.14
908009e832b4 v4.16-rc2 18 d3deafaa8b5c v4.15
7ba716698cc5 v4.16-rc2 18 bd4c82c22c36 v4.13
88913bd8ea2a v4.16-rc2 29 f6302f1bcd75 v3.3-rc3
c3cc39118c36 v4.16-rc2 29 a983b5ebee57 v4.15
d34bc48f8275 v4.16-rc2 29 d70f2a14b72a v4.15
11b7d897ccc1 v4.16-rc2 19 186b3c998c50 v4.14
95dbe9e7b372 v4.16-rc2 19 bb91accf2733 v4.9
7324f5399b06 v4.16-rc2 19 186b3c998c50 v4.14
f45765872e7a v4.16-rc2 18 2f08ee363fe0 v4.16-rc2
f4bc0c813e03 v4.16-rc3 22 1532d9e87e8b v4.15
521ca5a9859a v4.16-rc2 20 856e1eb9bdd4 v4.15
9238e380e823 v4.16-rc2 19 bd71b08ec2ee v4.14
26a0f6e82997 v4.16-rc2 19 b8a0dbe3a90b v4.15
96de67a77293 v4.16-rc2 19 0d235c3fabb7 v4.13
c67f100edae0 v4.16-rc2 19 f32f5bd2eb7e v4.10
2f0db8790169 v4.16-rc2 19 556dd1b9c313 v4.5
001a2fc0c8cc v4.16-rc2 19 8b32580df1cb v4.8
f600c6088018 v4.16-rc2 19 6aace17e64f4 v4.13-rc7
ef7a3518f7dd v4.16-rc2 19 0952da791c97 v4.9
8babd44d2079 v4.16-rc2 19 e586b3b0baee v4.1
0e7def5fb0dc v4.16-rc3 23 8fa1696ea781 v4.13
1575767ef3cf v4.16-rc3 23 8fa1696ea781 v4.13
d16b52cb9cdb v4.16-rc3 23 8fa1696ea781 v4.13
5fe01793dd95 v4.16-rc3 23 8fa1696ea781 v4.13
abe27a885d9e v4.16-rc2 20 d0869c0071e4 v4.16-rc2
e88230a3744a v4.16-rc2 20 bbbe775ec5b5 v4.9
b8ff18028159 v4.16-rc2 20 3fa489dabea9 v4.10
894266466aa7 v4.16-rc2 21 93286261de1b v4.15
013cb81e89f8 v4.16-rc6 27 0f6c480f23f4 v4.15
01a69cab01c1 v4.16-rc3 25 9ad1aefc8ae8 v3.2
39772f0a7be3 v4.16-rc3 25 8430e7e0af9a v4.7
cfc2c7405333 v4.16-rc2 21 0902b469bd25 pre-v3.0
a17f4f032b61 v4.16-rc3 24 7cd9d5fff792 v4.15
4cd6764495f2 v4.16-rc3 24 18cffd68e0c4 v4.15
7e2e5158e700 v4.16-rc3 24 7af4ac8772a8 v4.15
20e6bb17facd v4.16-rc3 25 c3bb33345721 v4.15
d1c95af36696 v4.16-rc2 20 3984d1a89fe7 v4.13
f57bbaae7271 v4.16-rc2 20 3352e6c45760 v4.14
4dba8bbce945 v4.16-rc2 20 192c4b5d48f2 v4.14
b37f78f234bf v4.16-rc2 20 ceed73a2cf4a v4.13
06998a756a38 v4.16-rc2 21 196f954e2509 v4.8-rc1
3addaba8141b v4.16-rc2 21 ea9c260514c1 v4.11
c795f3052b60 v4.16-rc2 21 d2a34232580a v4.11
95f3ce6a7789 v4.16-rc2 21 c9c5962b56c1 v4.6
d78d9ee9d40a v4.16-rc2 21 c9c5962b56c1 v4.6
bee92d06157f v4.16-rc2 21 73da7d5bab79 v3.11
5682e268350f v4.16-rc6 28 c6e6c96d8fa6 v4.8
143a4454daaf v4.16-rc6 27 f5e2bb4f5b22 v4.10
2cb370d615e9 v4.16-rc4 26 f19fbd5ed642 v4.15
4b6c1060eaa6 v4.16-rc3 24 cc27b0c78c79 v4.12
4b242e97d741 v4.16-rc3 24 8db87912c9a8 v4.14
f027e0b3a774 v4.16-rc2 19 ccd2b52f4ac6 v3.7
63e708f826bb v4.16-rc1 22 b4c0a7326f5d v4.14
29fee6eed281 v4.16-rc1 22 fd8aa9095a95 v4.10
aa0aad57909e v4.16-rc2 22 d38ceaf99ed0 v4.1
15734feff2bd v4.16-rc2 22 10ebc0bc0934 v3.12
9ab2323ca184 v4.16-rc2 21 fb23403536ea v4.16-rc2
d61a5c106351 v4.16-rc2 22 5addcf0a5f0f v3.11
43a08e0f58b3 v4.16-rc2 21 90e33d459407 v4.14
15f35d49c93f v4.16-rc2 21 ed70fcfcee95 v3.15
dfec091439bb v4.16-rc2 21 3f34cfae1238 v4.15
7dcf688d4c78 v4.16-rc2 21 67e658794ca1 v4.6-rc4
e6f02a4d57cc v4.16-rc2 21 27887bc7cb7f v4.14
c4e43e14cd46 v4.16-rc2 21 baf5086840ab v4.15
fe9c842695e2 v4.16-rc2 21 d9b8d8e19b07 v3.9
2f08ee363fe0 v4.16-rc2 21 02d8883f520e v4.15
6d79bd5bb6c7 v4.16-rc2 21 204f672255c2 v4.11
4b7f7ee2a5f5 v4.16-rc6 24 93b167c13a3a v4.15
7168179fcf25 v4.16-rc4 27 061701540349 v4.15
de526f401284 v4.16-rc2 22 bea74641e378 v4.13
285cb4f62319 v4.16-rc1 19 7778c4b27cbe v4.13
fc07bd8ce19b v4.16-rc6 24 ee48b72211f8 v4.12-rc4
5ab2ba931255 v4.16-rc6 24 f1ae02b186d9 v4.8
f88982679f54 v4.16-rc2 21 457b9a6f09f0 pre-v3.0
e78c637127ee v4.16-rc3 25 187d7967a5ee v4.4
d207af2eab3f v4.16-rc1 19 c743f0a5c50f v4.12
ca9eee95a2de v4.16-rc3 25 d717f7352ec6 v4.12
510c321b5571 v4.16-rc6 29 52df157f17e5 v4.12
9c2d63b843a5 v4.16-rc2 22 6c9059817432 v4.5
92256269893e v4.16-rc1 18 b138eca661cc v4.15
1f5a6c47aabc v4.16-rc2 21 8b10ba783c9d v4.14
5d4c05c3ee36 v4.16-rc2 21 44c58487d51a v4.11
1ff5325c3ca1 v4.16-rc2 21 fd3c7904db6e v4.11
5c2e1c4f9268 v4.16-rc2 21 fd3c7904db6e v4.11
0cba0efcc723 v4.16-rc2 21 08f294a1524b v4.15
3f802b162dbf v4.16-rc2 21 2dbd5186a39c v4.5
ec6f8401c48a v4.16-rc2 21 4da70da23e9b v4.13
104f268d439b v4.16-rc2 21 3832125624b7 v4.11
6623e3e3cd78 v4.16-rc2 21 3832125624b7 v4.11
d9dc7a3500a5 v4.16-rc2 21 3832125624b7 v4.11
3d89459e2ef9 v4.16-rc2 21 118620d3686b v4.13
415bb699d793 v4.16-rc2 21 02d8883f520e v4.15
14fa91e0fef8 v4.16-rc2 21 771a52584096 v4.11
ec897569ad7d v4.16-rc1 19 2e6522c56552 v4.15
67870eb12042 v4.16-rc3 20 33280b4cd1dc v4.5
d60d8b64280c v4.16-rc3 20 d9e139778376 v4.11
46408ea558df v4.16-rc2 22 977dcfdc6031 v3.16
02a10f061a3f v4.16-rc2 22 a8c06e407ef9 v4.11
11cd764dc9a0 v4.16-rc2 22 02b6fdc2a153 v4.14
8c5a93ebf7ac v4.16-rc2 22 052f71e25a7e v4.15-rc5
d91676717261 v4.16-rc2 22 02b6fdc2a153 v4.14
fa2dfd0ec22e v4.16-rc2 22 dde634057da7 v4.15-rc5
d39b6ea4f8c9 v4.16-rc3 25 566a9b05e1fa v4.15
98112041bcca v4.16-rc2 21 541768b08a40 v4.13-rc1
8874ae5f15f3 v4.16-rc2 21 efed421a94e6 v3.18
dc6c35c679e9 v4.16-rc1 22 7fb4b407a124 v4.15
eca39e7f0cdb v4.16-rc1 19 9862b43624a5 v4.15
6e1d8ea90932 v4.16-rc1 19 44b6b7661132 v4.14
5ce0bad4ccd0 v4.16-rc3 27 903d31e34628 v4.11
52c84d36b7e2 v4.16-rc2 23 004b45c0e51a v4.14
b23af6ad8d2f v4.16-rc7 34 7daa21370075 v4.11
9de29eac8d21 v4.16-rc1 19 9005c6834c0f v4.15
961888b1d76d v4.16-rc1 19 e754aedc26ef v4.7
ea00f301285e v4.16-rc1 20 20ffa1caecca v4.15
c1e150ceb61e v4.16-rc1 18 1d9a090783be v4.16-rc1
8e036c8d30a2 v4.16-rc1 18 eac1e731b59e v4.13
10414014bc08 v4.16-rc2 22 268cb38e1802 pre-v3.0
db93a3632b0f v4.16-rc2 22 d73f33b16883 pre-v3.0
257082e6ae23 v4.16-rc2 23 3c4d7559159b v4.12
a1dfa6812b68 v4.16-rc2 23 3c4d7559159b v4.12
bc3c2431d417 v4.16-rc2 23 f7a6272bf3cb v4.6
57ebd808a97d v4.16-rc2 22 7814b6ec6d0d v4.2
e6dbe9397ea7 v4.16-rc2 23 aa136d0c82fc v4.15
fae8b6f4a6be v4.16-rc2 23 a83863174a61 v4.10
ac5b70198adc v4.16-rc2 23 e6484930d7c7 pre-v3.0
9c481b908b01 v4.16-rc2 24 f371b304f12e v4.15
84c7efd607e7 v4.16-rc3 27 ab8dd3aed011 v4.5
74402055a2d3 v4.16-rc3 27 687c27676151 v4.1
fe27f16794f3 v4.16-rc3 27 f4b9f40ae95b v4.7
7fc17e909edf v4.16-rc2 24 0bf7800f1799 v4.15
118032be3890 v4.16-rc1 19 660fc733bd74 v4.11
fe0e58048f00 v4.16-rc1 19 0a44697627d1 v4.14-rc3
952fad8e3239 v4.16-rc2 24 1aa12bdf1bfb v4.15
9a3efb6b661f v4.16-rc2 24 b95a5c4db09b v4.10
4b8b41d15d9d v4.16-rc1 51 1fe699e30113 v4.16
4c83f0a788cc v4.16-rc1 51 6060b6aec03c v4.15
d3f84c8b0970 v4.16-rc1 51 99e48bf98dd0 v4.16
edb76b01ac16 v4.16-rc1 51 4900727d35bb v4.16
117172c8f9d4 v4.16-rc1 51 d6a2289d9d6b v4.11
4244140d7b8f v4.16-rc1 18 ad70062cdb40 v4.15
d4014d8cc6df v4.16-rc2 24 ebeeb1ad9b8a v4.15
339c21d7c459 v4.16-rc2 24 7fa9d974f3c2 v4.14
bb047ddd1458 v4.16-rc2 24 4861738775d7 v4.15
0f2d2b2736b0 v4.16-rc2 24 76610ebbde18 v4.11
8aa36a8dcde3 v4.16-rc3 28 fa0ce4035d48 v3.2-rc3
fd0e786d9d09 v4.16-rc1 21 ce0fa3e56ad2 v4.13
8f8ca51dbb4d v4.16-rc6 30 da77d76b95a0 v4.16-rc6
01684e72f167 v4.16-rc1 21 540adea3809f v4.15
67a3ba25aa95 v4.16-rc1 21 73fbc1eba7ff v4.10
295cc7eb314e v4.16-rc1 21 30bb9811856f v4.14
910961754572 v4.16-rc1 21 4145f358644b v4.15
82343484a2d4 v4.16-rc1 21 e67e02a544e9 v4.15
ff31e105464d v4.16-rc1 21 bf9a95f9a648 v4.15
4a7aa4fecbbf v4.16-rc1 21 bf9a95f9a648 v4.15
fae2211697c9 v4.16-rc1 21 bf9a95f9a648 v4.15
62e984ddfd6b v4.16-rc1 21 2ee13be34b13 v4.9
b00b62898631 v4.16-rc1 21 6c8e6bb2a52d v4.14
d97ca5d714a5 v4.16-rc6 31 ecd791874523 v3.6-rc6
2471c9816549 v4.16-rc6 31 3ca28286ea80 v4.13
7f95122067ab v4.16-rc1 22 235266b8e11c v4.15
b3ccefaed922 v4.16-rc1 20 f09d160992d1 pre-v3.0
7292b9e65865 v4.16-rc1 52 a091d4ee931b v4.12
33afe065b66f v4.16-rc1 52 ac14fbd460d0 v4.14
73b0fcd24ef1 v4.16-rc1 104 95690a02fb5d v4.15
75b0e73023ef v4.16-rc1 104 4407eaa9b0cc v4.15
6df3877fc962 v4.16-rc4 15 09f984961c13 v4.15
c662f77331c9 v4.16-rc4 15 accb757d798c v4.15
aef17ca12719 v4.16-rc1 22 1b50b776355f v4.14
07a2e1cf3981 v4.16-rc2 25 8c56df372bc1 v4.15
c53593e5cb69 v4.16-rc6 33 0d5936344f30 v4.14
2c1ec6fda2d0 v4.16-rc6 31 ee7fb331c3ac v4.3
9173e5e80729 v4.16-rc6 31 f92a26365a72 pre-v3.0
058f58e235cb v4.16-rc6 31 607126c2a21c pre-v3.0
027d351c5417 v4.16-rc2 25 e69b6c02b4c3 v4.14
8e021a14d908 v4.16-rc2 25 e69b6c02b4c3 v4.14
18a5b052bb1a v4.16-rc2 25 fea23fb591cc v4.15
808cf9e38cd7 v4.16-rc2 25 c134ecb87817 v4.6
fb23403536ea v4.16-rc2 25 94014e8d871a v4.15
27af86bb038d v4.16-rc2 25 cd2b70875058 v4.10
1b12580af1d0 v4.16-rc2 25 9cf637473c85 pre-v3.0
9d37c094dacd v4.16-rc1 18 4fac8076df85 v4.15
16e574d762ac v4.16-rc1 18 ec82b567a74f v4.15
d7212cfb05ba v4.16-rc1 21 f85942207516 v4.15
433986c2c265 v4.16-rc1 21 baa8809f6097 v4.9
69728051f5bf v4.16-rc1 21 c84345597558 v4.10
f23def803861 v4.16-rc1 22 02f26ecf8c77 v4.15
ea56fb282368 v4.16-rc1 22 6a623e076944 v4.11
49527bc0e8d2 v4.16-rc3 29 83c566806a68 v4.15
3cd091a77393 v4.16-rc1 21 662591461c4b v4.12
b16ea8b9492e v4.16-rc2 22 cf6d867d3b57 v4.6
675272d092e4 v4.16-rc2 22 2bfa0719ac2a v4.11-rc1
6cf439e0d374 v4.16-rc2 22 f0175ab51993 v3.16
499350865387 v4.16-rc2 22 e93650994a95 v4.14
c4a5153e87fd v4.16-rc2 22 689bf72c6e0d v4.14
f035d139ffec v4.16-rc2 22 c2da2ff00606 v3.2
6180026341e8 v4.16-rc2 22 72246da40f37 v3.1
1f8ade92a836 v4.16-rc1 22 74426fbff66e v4.14
54e02162d445 v4.16-rc2 25 6e6e41c31122 v4.15
f8d0cbf28df1 v4.15 56 917538e212a2 v4.15
c80c5ec1b2fa v4.16-rc1 23 7b6061627eb8 v4.15
c591c2e36ccc v4.15 4 59f47eff03a0 v4.15
1c5b2216fbb9 v4.15 4 5b54e16f1a54 pre-v3.0
89271c65edd5 v4.15 4 2863c61334aa v4.7
0bf7800f1799 v4.15 4 2e0ab8ca83c1 v4.7
6e6e41c31122 v4.15 4 2e0ab8ca83c1 v4.7
b9058afcd6c7 v4.15 4 23c35f48f5fb v4.15
6016f7d54bc4 v4.15 4 6613d18e9038 v4.9
b81e830c9ad0 v4.15 4 c6acad68eb2d v4.15
8c88181ed452 v4.15 4 cb5f7334d479 v4.15
08f513851218 v4.15 5 84a527a41f38 v4.6-rc7
eb53f7af6f15 v4.15 5 24d3dc6d27ea v4.10
79a8a642bf05 v4.15 39 6d07d1cd300f v4.15
762c330d670e v4.15 5 761876c857cb v4.13
cb9f7a9a5c96 v4.15 5 134e63756d5f pre-v3.0
175e03101d36 v4.15 5 193bcb7b3719 v4.13
2285ae760de2 v4.15 15 e40d99e6183e pre-v3.0
f63248fac563 v4.16-rc4 33 0cdbf481e927 v4.13
07234021410b v4.15 4 60f1d5e3bac4 v4.9
7b6586562708 v4.15 4 3ba009297149 v4.3
d692403e5cf8 v4.15 5 8a2768732a4d v4.12
4dd5f8a99e79 v4.16-rc1 23 4b5d62ca17a1 v4.10
eeb715c3e995 v4.16-rc1 23 371b80447ff3 v4.15-rc2
dedab7f0d313 v4.16-rc1 23 5ef3166e8a32 v4.15
6cc3f91bf69f v4.16-rc1 23 f442d004806e v4.15
5c11d1e52d99 v4.16-rc1 23 f14e953b191f v4.15
b3e456fce9f5 v4.16-rc2 24 6c5d5cfbe3c5 v4.10-rc3
2e7d1d61ea6c v4.16-rc1 24 b8ae5c7387ad v4.13
dd78c832ffaf v4.16-rc1 24 b8ae5c7387ad v4.13
225ece3e7dad v4.16-rc1 24 1005bccd7a4a v3.13-rc1
ba916b6a0339 v4.16-rc1 24 83dee2ce1ae7 v4.15
d9afaaa4ff7a v4.16-rc1 24 7375ae3a0b79 v4.13
eff84b379089 v4.16-rc1 24 45691e2d9b18 v4.7
a827a164b98b v4.15 5 5ccda64d38cc v4.15
55f3538c4923 v4.15 5 49a086c201a9 v4.15
2148481d5d31 v4.15 5 ff69c21a85a4 v4.14
92426820f761 v4.15 5 ff69c21a85a4 v4.14
b7d99235473a v4.15 5 ce4ebfd859c3 v4.15
6e0377212c4e v4.15 5 0fbc0b67a89d v4.15
781198f1f373 v4.15 5 82616f9599a7 v4.13
0725390da9fb v4.15 6 343a8d17fa8d v4.15
57ea5f161a7d v4.15 4 76d837a4c0f9 v4.12-rc1
5c487bb9addd v4.15 5 c24b14c46bb8 v4.14
44750f8483a1 v4.15 5 fc1e64e1092f v4.15
58e354c01b19 v4.15 5 fc1e64e1092f v4.15
17e9e23b130e v4.15 5 3136ef49a14c v4.15-rc1
e729452ec333 v4.15 5 baf5086840ab v4.15
ade7db991b47 v4.15 9 b2a3ad9ca502 v3.4-rc1
b9607de6cf22 v4.15 5 8da58553cc63 v4.15
0a7fe718239e v4.15 5 703f261dd77f v4.15
c713fb071edc v4.15 6 b8b8b16352cd v4.14-rc5
3a0a7b261009 v4.15 12 d349caeb0510 v4.6
a3381e3a65cb v4.15 6 b539cc82d493 v4.11
eb85c50c24cf v4.15 6 343a8d17fa8d v4.15
40e2c4e5a7ef v4.16-rc1 40 75ee94b20b46 v4.14-rc8
9903a91c763e v4.15 20 b0b91d18e2e9 v4.8
85c2dd5473b2 v4.15 20 759c01142a5d v4.4
a77660d231f8 v4.15 15 5c9a8750a640 v4.5
a1be1f3931bf v4.15 11 ba62bafe942b v4.7-rc2
63300eddb1e6 v4.15 7 f9886bc50a8e v4.15
4f7e988e63e3 v4.15 47 92266d6ef60c v3.9
7dc68e98757a v4.15 6 5859034d7eb8 pre-v3.0
50dbd09c56db v4.16-rc2 28 82abdcaf3ede v4.15
f5115e8869e1 v4.15 6 acfb3b883f6d v4.15-rc8
b3ad99ed4591 v4.15 36 351e3db2b363 v3.14
b18224e95cb1 v4.15 36 76ff480ec963 v4.8
d480b28a41a6 v4.15 5 f090a00df9ec v4.14
f85942207516 v4.15 6 1b39e3f813b4 v4.13
d7cdee5ea8d2 v4.15 7 c0d378ef1266 v4.14-rc6
a56c69803f5a v4.15 7 bafbdd527d56 v4.15
7a407aa5e0d3 v4.15 7 f2d0b0d5c171 v4.15
8a0f5b6f3327 v4.15 7 c6acad68eb2d v4.15
45df8462730d v4.15 5 d2ddc776a458 v4.14
8305e579c653 v4.15 5 d2ddc776a458 v4.14
fe4d774c8473 v4.15 5 d2ddc776a458 v4.14
e44150157f42 v4.15 5 d2ddc776a458 v4.14
7b4eb53d95c3 v4.15 7 996139e801fd v4.13
ca66e797120f v4.15 18 e0d02285f16e v4.14
364f56653708 v4.15 7 4bdced5c9a29 v4.14
ad0f1d9d6593 v4.15 7 4bdced5c9a29 v4.14
21a2545bbea0 v4.15 7 a107311d7fdf v4.15
4a31a6b19f9d v4.15 7 410f03831c07 v4.2
957d761cf91c v4.15 7 dbc2b5e9a09e v4.12-rc1
4f1764172a0a v4.15 16 659aefb68eca v4.15-rc1
567af7fc9d87 v4.15 8 23c35f48f5fb v4.15
3a9e9a47092e v4.15 7 bd3dda9ab0fb v4.15
fd649f10c3d2 v4.16-rc1 23 4fde46f0cc71 v4.1
2572cf57d75a v4.15 7 05e0cc84e00c v4.15-rc8
a61a86f8db92 v4.15 8 bd68a2a854ad v4.9
91d9f8fcf7cc v4.15 8 f875a832d202 v4.13
0cde5b44a30f v4.15 8 b27311e1cace pre-v3.0
8dbc1864b74f v4.15 8 17278a91e04f v4.15-rc7
0ef1559a20b9 v4.15 8 f2d0b0d5c171 v4.15
e17ae5cdd10a v4.15 8 f2d0b0d5c171 v4.15
9a9ab3078e27 v4.15 8 eed0eabd12ef v4.8
9b6faee07470 v4.15 7 8ed5eec9d6c4 v4.15
2aed489d163a v4.15 7 0aceb53e73be v4.15
09584b406742 v4.15 8 290af86629b2 v4.15-rc7
511051d509ec v4.16-rc2 19 a83195937151 v4.13
a7770ae19456 v4.15 88 79da4721117f pre-v3.0
8cf4e6a04f73 v4.15 95 d7b1956fed33 pre-v3.0
0911287ce32b v4.15 8 468e2f64d220 v4.14
4db428a7c9ab v4.15 7 ef456144da8e v4.4
058a6c033488 v4.15 7 a1b7c5fd7fe9 v4.5
1d8ef0c07664 v4.15 7 73a0329b057e v4.15
498e7e7ed1fd v4.16-rc4 37 07b8481d4aff v3.5
ebfc15019cfa v4.15 7 a9cdbe72c4e8 v4.15-rc6
af189c95a371 v4.15 6 117cc7a908c8 v4.15-rc7
4bf5d56d429c v4.15 6 fec9434a12f3 v4.15
23fbd7c70aec v4.15 9 ba9c8dd3c222 v4.8
e89e8d8fcdc6 v4.15 4 bebd031866ca v4.11
ba7cd5d95f25 v4.15 8 c38c4597e4bf v4.4
627e08738e43 v4.16-rc1 38 38b5f68e9811 v4.15
c8195a7b1ad5 v4.16-rc1 23 8da6d5815c59 v3.2
900c99816800 v4.16-rc1 24 f32e48e92596 v4.4
1a932ef4e479 v4.16-rc1 23 703c88e03524 v3.12
1846430c24d6 v4.16-rc1 23 681ae50917df v3.12
ea23d5e3bf34 v4.15 8 83f1999caeb1 v4.15
2ff124d597c2 v4.15 9 9ee79fce3642 v4.13
87daac68f77a v4.15 9 44c58487d51a v4.11
b5de809ef6f6 v4.15 9 1b311f8931cf v4.14
78d3633ba9c2 v4.15 9 5b6cabb0db77 v4.13
2b1e7fe16124 v4.15 9 e11ffbd57520 v4.6
f34727a13513 v4.15 9 bf3c5a93c523 v4.15
e7aadb27a541 v4.15 8 a46182b00290 v4.15-rc3
0b1dfa4cc6c6 v4.15 8 76e81d6d5048 v4.15
65073a67331d v4.15 9 02dd3291b2f0 v4.15
751b01cb07eb v4.15 39 e2b763caa6eb v4.11
b26a32a82a90 v4.15 39 4680816be336 v4.9
b5a756a72228 v4.15 40 5b54eddd3920 v4.14
124804c4c4b2 v4.15 40 66df1014efba v4.14
c5bd1fc9a6c8 v4.15 45 4e773c3a8a69 v4.14
d13a8479f358 v4.15 45 bc87229f323e v4.4
cc4f8fc72eb9 v4.15 49 30e17b7847f5 v4.15
25da77f83068 v4.15 50 b03ec3d67ab8 v4.15
c950af50e598 v4.15 50 3176ff49bc3e v4.15
6389902072db v4.15 51 d8fc70b7367b v4.11
2afba81c7909 v4.15 54 90f192c8241e v4.12
a83165f00f16 v4.15 8 936bd486564a v4.8
367dc6586d2d v4.15 8 eb7935830d00 v4.15
ac964d7a5942 v4.15 12 7235acdb1144 v4.7
9862b43624a5 v4.15 9 549b4930f057 v4.14
62a06994ced1 v4.15 9 6882804c916b v4.11
e029f541039f v4.15 9 395cacb5f1a0 v4.15
54f809cfbd6b v4.16-rc2 93 24835e442f28 v4.10
1beaeacdc88b v4.15 8 c942cee46bba v4.14
cbbde7e8d985 v4.15 10 aee5b8cf2477 v4.15
fb208dc73ff1 v4.15 10 aee5b8cf2477 v4.15
17cf68b702a6 v4.15 10 aee5b8cf2477 v4.15
d71ef28636e4 v4.15 10 aee5b8cf2477 v4.15
07ae5389e98c v4.15 4 3d3319b45eea v3.13-rc4
36ee41d161c6 v4.15 4 579006944e0d v4.12
e02a9f048ef7 v4.15 8 67fd707f4681 v4.14
9bb5a391f9a5 v4.15 8 f7f99100d8d9 v4.14
da391d640c52 v4.15 8 aecd6f44266c v3.16
69d763fc6d3a v4.15 23 c82449352854 v3.2
d984187e3a1a v4.15 12 acf8fdbe6afb v4.2
ff26cc10aec1 v4.15 23 1cce4df04f37 v4.4
ee190ca6516b v4.15 12 f729c8c9b24f v4.10-rc3
8de9399a24f9 v4.15 10 37cb11acf1f7 v4.15
580d834fe166 v4.15 10 6613d18e9038 v4.9
d391f1207067 v4.15 10 68c3b4d1676d v3.15
c9cc8d01fb04 v4.15 8 311fc65c9fb9 v4.13-rc6
3f34cfae1238 v4.15 8 22265a5c3c10 pre-v3.0
1a38956cce5e v4.15 8 136e92bbec0a pre-v3.0
1e98ffea5a89 v4.15 8 ec2318904965 v4.10
c028c6309a9f v4.16-rc2 34 1bd773c077de v4.14
56174d9a2149 v4.15 10 e578afab6e5f v4.15
bd9b902798ab v4.15 10 de3ee99b097d v4.14-rc3
5fa4ec9cb2e6 v4.15 10 0092e4346f49 v4.15
761e05a702f5 v4.15 7 2ed077e467ee v4.14
273caa260035 v4.15 11 b8c601e8af2d v4.15
5bf30316991d v4.15 8 77ef56e4f0fb v4.13
4a8842de8db4 v4.16-rc2 28 1d55abc0e98a v4.10
7fcae1118f5f v4.15 7 2961298efe1e v4.15
9471eee9186a v4.15 7 da285121560e v4.15-rc7
a1dfb4c48cc1 v4.15 12 6b5a9492ca0c pre-v3.0
015eb1b89e95 v4.15 9 d4748276ae14 v4.15
66b3bd2356e0 v4.15 9 6f6a23a213be v4.15-rc3
95244db2d3f7 v4.15 5 09ac4fcb3f25 v4.13
30e948a37839 v4.15 9 03aef17bb79b v4.15
40ca54e3a686 v4.15 9 1c0d32fde5bd v4.9
2d85ec8acb01 v4.15 10 dd0273284c74 v4.15
468330e886ee v4.15 9 1ae1df3a1193 v4.15
e64e469b9a2c v4.15 9 f3d9832e56c4 v4.14
31afeb425f7f v4.15 9 1859bac04fb6 v4.14
0bc0b97fca73 v4.15 9 6a8788f25625 v4.15
c76fe2d98c72 v4.15 9 4a6e3c5def13 v4.11
202a0a70e445 v4.15 9 6c389fc931bc v4.8-rc4
eaa160ededfa v4.15 17 22c11858e800 v4.15
4cd879515d68 v4.15 9 3a4d5c94e959 pre-v3.0
88fae87327a2 v4.15 9 362899b8725b v4.7
406de7555424 v4.15 9 c5ad119fb6c0 v4.15
7ece54a60ee2 v4.15 9 e32ea7e74727 v4.4
acd1d71598f7 v4.15 10 66f364649d87 v4.14
9893b905e743 v4.15 12 4d37ece757a8 v3.1
81742be14b6a v4.15 12 0f91c9d6bab9 v4.1
3b28c987fb95 v4.15 13 b83003b3fdc1 pre-v3.0
2e2bc5a9a300 v4.15 13 d637628ce00c pre-v3.0
1536dc063e82 v4.15 13 b83003b3fdc1 pre-v3.0
03aadf9499ba v4.15 14 a9c59ef77458 v4.7
c6fad21a8d03 v4.15 14 1b249475275d pre-v3.0
cbd27448faff v4.15 14 e26a5843f7f5 v4.1
e231c6879cfd v4.15 9 a5864c999de6 v4.7
e7345ba352d1 v4.15 9 2935e3c38228 v4.15
925f7ea7a6ba v4.15 10 16c72e402867 v4.15
b081808a6634 v4.15 10 e126ba97dba9 v3.10
f23a5350e43c v4.15 10 44c58487d51a v4.11
751c45bd828f v4.15 10 00fc0c51e35b v4.12
7d2b8e6aaf9e v4.16-rc2 19 2d6ca60f3284 v4.4
a3b5655ebdb5 v4.16-rc2 19 95e339b6e85d v4.12
e383095c7fe8 v4.15 10 caf7501a1b4e v4.15
5b8b58063029 v4.15 12 16b1d26e77b1 v3.18
384dfd627f1e v4.15 12 ec233ede4c86 v4.14
6dd1ec6c7a2c v4.15 10 b471f2f1de8b v4.15
accf475a5ece v4.15 19 a2e230c7b2ea pre-v3.0
63a83b861c47 v4.15 12 9d49640a21bf v4.10
771fc607e6b9 v4.15 10 d6d4f60c3a09 v4.15
40339af33c70 v4.15 9 36777d9fa24c v4.11
fd492228d403 v4.15 10 73834aec7199 v4.15
ec89ab50a03a v4.15 10 6b88a32c7af6 v4.15
290df4d3ab19 v4.15 14 9e7e3738ab0e v4.14
5b09179e7fa2 v4.15 10 1280c0f8aafc v4.15
a9e6d44ddecc v4.15 14 58eae1416b80 v4.15-rc8
9b046013e583 v4.15 11 4d99b2581eff v4.5
479c03a7173d v4.15 11 92ae112e477a v4.15
2585cd62f098 v4.15 10 40304b2a1567 v4.12
5bdd0c6f89fb v4.15 11 5451f79f5f81 pre-v3.0
8146acff3b80 v4.15 14 3bb0f7c31b1a v4.15
6e49412016f5 v4.15 11 df351ef73789 v4.15
f97f43c9ed1d v4.15 11 795bc112cd5a v4.15
7f55c733b660 v4.15 12 5620a0d1aacd v4.13
003948c54e5b v4.15 12 5620a0d1aacd v4.13
5be2a5011c03 v4.15 12 33b054b867b8 pre-v3.0
f4c6cd1a7f22 v4.15 11 1fef62c1423b v3.18
4c7e95b1b328 v4.15 11 25f815f66a14 v4.15
e36c048a9bd8 v4.15 17 ac69f66e54ca pre-v3.0
9084432c316b v4.15 23 07495ff5d9bc pre-v3.0
4b3e910d7f43 v4.15 11 fd335ab04b3f v3.14
888f22931478 v4.15 11 9474933caf21 v4.10
325694e6c348 v4.15 11 ad75b7d32f25 v4.14
1cbbf69cc83e v4.15 11 7bc04215a66b v4.15
310eb252a783 v4.15 12 f9bb304ce855 v4.15
cc4804448c32 v4.15 10 d0f0f63ac137 v3.19
31e95b61e172 v4.15 11 2b36047e7889 v4.15-rc7
972d0093c2f7 v4.15 14 02bcd1577400 v4.12
1c130ae00b76 v4.15 15 e7942d0633c4 v4.13
0e4d819d0893 v4.15 8 ad67b74d2469 v4.15-rc1
493fb50e958c v4.15 15 cc27b735ad3a v4.14
62314e405fa1 v4.15 11 147b27e4bd08 v4.15
8e6875250a11 v4.15 11 37f1ba0909df v4.14
35136920e100 v4.15 11 111e6b45315c v4.15
2310035fa03f v4.15 11 b471f2f1de8b v4.15
f53d77e19b65 v4.15 11 97a6ec4ac021 v4.15
20d59023c5ec v4.15 12 fbbaf700e7b1 v4.11
4bb7df7d6939 v4.15 11 c19bb00070dd v4.15
9e55e5d30f6c v4.15 11 b2d3bcfa26a7 v4.15
f03e4f8aadc8 v4.15 11 23ee07ad3c2f v4.15
58d6b15e9da5 v4.15 11 67f691976662 v4.6
e58decc9c51e v4.15 11 0dc786219186 v4.12
e71494ae68d2 v4.15 11 da762863edd9 v4.15
7ad81482cad6 v4.15 12 0e70f97f257e pre-v3.0
aae2349c4919 v4.15 11 9d6b04097882 v4.7
791ae273731f v4.15 12 f9036bd43602 v4.8
4852fdca8818 v4.15 15 39c5c4471b8d v4.15
401f69308fe3 v4.15 15 c7cbef1fdb54 v4.15
5eb8c768f493 v4.15 15 2086dd35705f v4.15
631e8f0a9748 v4.15 10 93d10af26bb7 v4.15
04d0e8de8a6a v4.15 15 a60d64b15c20 v4.15
8d25e15d94a2 v4.15 15 f81a8158d4fb v4.15
7d70718de014 v4.15 12 6106c0f82481 v4.15
e2105ca8beb0 v4.15 33 3aa0582fdb82 v4.15
87b970f453d0 v4.15 12 8bc1e4ec79c0 v4.14
a2390348c19d v4.15 12 3515832cc614 v4.13
45596c7889a4 v4.15 12 b416c099472a v4.15
96d5eaa9bb74 v4.15 14 de25deb18016 pre-v3.0
6abe534f0776 v4.15 11 c5e8848fc98e v4.9
c13da21cdb80 v4.15 11 8565d26bcb2f pre-v3.0
03aaa9e2677e v4.15 11 85b352693264 pre-v3.0
e8660ded7f5a v4.15 11 ccfdec908922 v4.15
5e89cfac2e1e v4.15 11 86897c960b49 v4.6
00db63c128dd v4.15 11 abae1b71dd37 v4.3
3267c081e037 v4.15 13 70a65240efb1 v4.15
a5e192335650 v4.15 12 c92316bf8e94 v4.13
51c266561e5a v4.15 12 b0a9c37b0178 v4.15
0198e5b707cf v4.15 13 6592e58c6b68 v4.12
6f794e3c5c8f v4.15 16 319e4d0661e5 v4.4
9f13ce743b1b v4.15 13 ba6d5887946f pre-v3.0
7cfad65297bf v4.15 57 005887f2e3e0 pre-v3.0
9deae9689231 v4.15 63 addc3fa74e5b v3.17
49686cbbb3eb v4.15 11 57e62324e469 v3.3
b75703de1630 v4.15 11 d9f9b9a4d05f v4.15
7f7aeea7cf30 v4.15 12 fa590c222fba v3.13
e749d328b0b4 v4.15 13 dce143c3381c v4.1
c14b65feac9e v4.15 12 de9e33bdfa22 v4.14
0ddcff49b672 v4.15 11 ff4dd73dd2b4 v4.10
35b84bf0614a v4.15 15 7fb59e940f62 v4.15
8c6b8eda7294 v4.15 11 8bfa7e1e03ac v4.15
134aecbc25fd v4.15 12 d7e09d0397e8 v3.10
4a09d0b86bad v4.15 12 b21fc294387e v4.15
35adacd6fc48 v4.15 12 ab9dbf771ff9 v4.15-rc2
8d81296cfcce v4.15 13 0428491cba92 v4.12
ec80eb467171 v4.15 8 cf1b09908a23 v4.14
a5281feafd7b v4.15 8 55de88778f4b v4.15
636e645f1f2a v4.15 8 5c9d9a1b3a54 v4.14
62d1c95d577c v4.15 8 ad475117d201 v4.10
c7780ab56c09 v4.15 8 e377cd8221eb v4.10
4ec591e51a4b v4.15 13 1e0fc9d1eb2b v4.12
c1a72e2dbb4a v4.15 12 8878b126df76 v4.15
c495a9275eec v4.15 12 02f26ecf8c77 v4.15
cf5eebae2cd2 v4.15 14 e522751d605d v4.9
6e36719fbe90 v4.15 12 148b974deea9 v4.15
77dd66a3c67c v4.15 15 9476df7d80df v4.4
10a0cd6e4932 v4.15 15 5f29a77cd957 v4.5-rc7
bcc93a23ca59 v4.15 11 2406e7e546b2 v4.15
10bea9c87304 v4.15 11 d69a24e03659 v4.15
5165674ff502 v4.15 11 c3164d2fc48f v4.12-rc5
8c7a8d1c4b9c v4.15 12 e80a0af4759a v4.15
ba87977a4991 v4.15 14 b7ce40cff0b9 v3.14
a5a927a7c82e v4.15 14 4edb83bb1041 v4.13
e8bfa0422469 v4.15 13 17470b7da11c v4.8
95140ed16db5 v4.15 13 ea8b1c4a6019 v4.9
c3388a075c8a v4.15 14 178f3f75bb4e v4.15
1c9609e3a8cf v4.15 12 4a219ef8f370 v4.12
e9eaa8052fe7 v4.15 11 f77d281713d4 v4.15
7a074fc08389 v4.15 13 d4748276ae14 v4.15
58359193193c v4.15 11 cdf8259d6573 v4.15
ef58ca38dbda v4.15 12 7fdb61b44c0c v4.15
2e6522c56552 v4.15 10 eed0eabd12ef v4.8
5f2483eb2423 v4.15 10 ec3352925b74 pre-v3.0
36447456e1cc v4.15 13 787d8c530af7 v4.12
e7b2823a582a v4.15 12 02dd3291b2f0 v4.15
cbebc6ef4fc8 v4.15 12 57e62324e469 v3.3
06e1902456f3 v4.15 12 8224b2734ab1 v4.13
7233982ade15 v4.15 12 f78e5623f45b v4.15
d680b3524cd2 v4.15 12 7960d1daf278 v4.15
7b7c1763725b v4.15 13 3e507769d15e v4.15
a8bc71d49603 v4.15 13 6cd225cc5d8a v4.15
166a5a33d5a7 v4.15 13 d84b4c7c706f pre-v3.0
e29a22a86a20 v4.15 13 c6059879be29 v4.15
b930151e5b55 v4.15 10 9e80f9064e73 v4.9
0657cb50b5a7 v4.15 10 9e80f9064e73 v4.9
47712a921bb7 v4.15 14 2104180a5369 v4.12
23d4ee19e789 v4.15 13 396eaf21ee17 v4.15
e2e3224122e6 v4.15 12 36e04a2d78d9 v4.15
5ef7e0ba1039 v4.15 12 5f35227ea34b v3.19-rc2
f78e5623f45b v4.15 14 dbb7d2a88d2a v3.6
01353bbea503 v4.15 14 3e507769d15e v4.15
fb7d38a70e1d v4.15 12 566e82516253 v4.8
433c6cab9d29 v4.15 12 566e82516253 v4.8
4f6a71b84e1a v4.15 12 566e82516253 v4.8
37512b42f082 v4.15 12 566e82516253 v4.8
c877154d307f v4.15 51 781f675e2d7e v4.12
de99a346884f v4.15 14 4246a0b63bd8 v4.2
7bed45954b95 v4.15 14 20e4d8139319 v4.15
2d83619de8a6 v4.15 12 c02b3741eb99 pre-v3.0
98051872fd25 v4.15 13 7bc04215a66b v4.15
99ac5327e902 v4.15 13 7bc04215a66b v4.15
4552d128c26e v4.15 14 4388c9b3a6ee v4.13
dc98b8480d8a v4.15 17 928055a01b3f v4.13
71e7673dadfd v4.15 15 428dbf156cc5 v4.14-rc2
1131b0a4af91 v4.15 13 17218e0092f8 v4.15
341a0ffceaa4 v4.15 12 404d1a3edc38 v4.15
605f2d34ea52 v4.15 15 6a537d48461d v4.15
3efc31f76dd7 v4.15 14 82de802ad46e v4.12
ca8dc694045e v4.15 14 daf0cd445a21 v4.11
f20d429511af v4.15 14 d37498417947 v4.5
72b30e986d25 v4.15 14 86dbcd0f12e9 v4.5
7110d80d53f4 v4.15 13 eb54e522a000 v4.13
63c859101ec3 v4.15 13 1b76c13e4b36 v4.2
d77be6895547 v4.15 13 583c90097f72 v4.15
2bc4298f59d2 v4.15 13 c378b5119eb0 v4.4
e7e70fa6784b v4.15 13 6a8788f25625 v4.15
6b88a32c7af6 v4.15 14 27a921e75711 v4.15
14f1889fd4d7 v4.15 16 fbcab13d2e25 v4.14-rc2
fd5f7cde1b85 v4.15 16 6b97a20d3a79 v4.5
42b9f5c58aa8 v4.15 11 0ba6f37012db v4.14
d698c4a02ee0 v4.15 11 05c974669ece v4.8-rc6
a64e7a79dd60 v4.15 14 b014e96d1abb v4.12
ddc212313f16 v4.15 15 e6269c445467 v4.2
f53bad0881a3 v4.15 15 71ae40e4cf33 v4.11
1ff2775a32ef v4.15 15 607ca46e97a1 v3.6
6b6977117f50 v4.15 20 705699a13994 v3.19
851c1a18c541 v4.15 20 963fee165660 v3.16
5c7d4f9ad39d v4.15 20 664f8e26b00c v4.13
4d6bde512a86 v4.15 15 9c656b07997f v4.11
241e3ec0faf5 v4.15 31 4e0b1ab72b8a v4.11
f96adf1ea08a v4.15 13 aaa150089465 v4.15
e3675875c0a5 v4.15 15 6106c0f82481 v4.15
2a7157b137f6 v4.15 12 cee8113a295a v4.15
94d3084a0f8c v4.15 15 dd5ac03e0955 v4.9
29d9bef13085 v4.15 14 084bd29810a5 v3.10
dae06208d310 v4.15 15 a2ecc4df9f84 v4.15
b90c6d105033 v4.15 15 f0ede2c6282b v4.15
b926c1daf3bb v4.15 15 f0ede2c6282b v4.15
82cea533aed4 v4.15 15 a758f50f10cf v4.15
48d0c9becc7f v4.15 22 597d0275736d pre-v3.0
41e4b391157f v4.15 14 902d6a4c2a4f v4.15
b069b37adb14 v4.15 14 902d6a4c2a4f v4.15
e3eeacbac4ad v4.15 14 03d13b6868a2 v4.15
9be9d04b28e7 v4.15 14 a3c90f7a2323 v4.15
a6532e713966 v4.15 14 44c58487d51a v4.11
cd2a6e7d384b v4.15 14 7db20ecd1d97 v4.13
852f6927594d v4.15 14 4196670be786 v3.14-rc2
b2bedfb39541 v4.15 14 a512c2fbef9c v4.13-rc1
4f5e838605c2 v4.15 26 ad67b74d2469 v4.15-rc1
6802f3adcb3f v4.15 14 4c981e28d373 v4.15
c469652bb5e8 v4.15 16 ffcd28d88e4f v3.19
123af9043e93 v4.15 15 b3c70c9ea62a v3.1
39610a68d9a9 v4.15 15 787fd1d019b2 v4.15
98732d1b189b v4.15 15 fa2a8445b1d3 v4.15
6a205420758a v4.15 15 fa2a8445b1d3 v4.15
2f7aacf795e0 v4.15 15 677a60bd2003 v4.15
a22fde8e9794 v4.15 15 7520fa99246d v4.15
9f239fe6a6b9 v4.15 14 fea23fb591cc v4.15
abde587b61a3 v4.15 15 a0c01e4bb92d v4.15
667257e8b298 v4.15 17 e9a823fb34a8 v4.13
bc8d062c36e3 v4.15 17 8ddcd653257c v4.14
c5b40c315a86 v4.15 17 f780429adfbc v4.15
a78182980a2a v4.15 17 f780429adfbc v4.15
be6d447e4f9c v4.15 15 11c8dc419bbc v4.15
a0d605372a27 v4.15 14 a1c55bccf600 v4.15
86a3ae587980 v4.15 14 a0c1214e4764 v4.15
ba4a76f703ab v4.15 15 1ca018d28d96 v4.1
8634ef5e0531 v4.15 15 5b5faaf6df73 v4.12
17218e0092f8 v4.15 13 122a22377a3d v4.7
bb48711800e6 v4.15 16 38fd94b0275c v4.10
bf9ae8c5325c v4.15 17 7c3fb70f0341 v4.15
4adf7bf7bbae v4.15 239 357f4aae859b v4.9
d6088e9a89f7 v4.15 17 e7d214642a19 v4.15
deb869993236 v4.15 15 72f2ff0deb87 v4.10
daaf24c634ab v4.15 15 02dd3291b2f0 v4.15
ce512d79d046 v4.15 19 69110e3cedbb v4.5
a8789a5ae28f v4.15 13 ac503a4cc9e8 v4.15
90bc9fb15942 v4.15 28 019132ff3daf v4.5
6ac1dc736b32 v4.15 16 b920de1b77b7 pre-v3.0
500d58300571 v4.15 16 769a8a96229e v3.0
fa1195ccc0af v4.15 13 9c9f5a2f1944 v4.1
20e4d8139319 v4.15 17 4b855ad37194 v4.12
84676c1f21e8 v4.15 17 4b855ad37194 v4.12
b0076406495c v4.15 17 ef9e1cdf419a v4.15
181d454b7220 v4.15 15 052ece6dc19c v4.15
11af96a47bed v4.15 15 76ad4f0ee747 v4.13
c6ba4f3e68e9 v4.15 17 427988d981c4 v4.15
3b65ca50d24c v4.15 17 dd9d4e7498de v4.9
87e89ce8d0d1 v4.15 17 e47f68587b82 v3.6
6b46d444146e v4.15 19 ca7f85be8d6c v4.9
cdd104099141 v4.15 18 6e8887f60f60 v3.9
6ffb5b4f248f v4.15 18 8113ba917dfa v3.7
23a0d8479929 v4.15 18 f780429adfbc v4.15
db00d38e7b59 v4.15 18 54c47014b474 pre-v3.0
ecb10288a370 v4.15 16 533dd11a12f6 v4.15
8869016d3a58 v4.15 18 c99a2be790b0 v4.15
33f782c49a95 v4.15 18 93b570464cce v4.15
2178790baa86 v4.15 13 d0565132605f v4.15
2a3e83c6f96c v4.15 17 707d4eefbdb3 v3.14-rc7
d5ff18bcd4b5 v4.15 18 e2e6771c6462 v4.15
25140717414c v4.15 36 bed73904e76f v4.15
2353758bc2d4 v4.15 36 bed73904e76f v4.15
6dee6722c606 v4.15 18 55da094824c4 v4.15
5ed001baeeff v4.15 17 2a95183a5e03 v4.15
9cee7972bd0c v4.15 18 55da094824c4 v4.15
76e28f5ffed8 v4.15 17 5a2a30024d8c v4.13
f5843492ecef v4.15 10 b3060044e495 v4.12
89838118a515 v4.15 18 57520751445b v4.11
8d20a1f0ecd5 v4.15 18 3c86aa70bf67 pre-v3.0
cfc350ab0efb v4.15 18 a1367e4adee2 v4.15
bbd16d96d1ec v4.15 18 4b0002275355 v4.15
96cf727fe8f1 v4.15 20 479ad8e9d48c v4.12
a935424bb658 v4.15 18 10da65423fdb v4.12
e213f5b6dd5d v4.15 17 0ddf543226ac v4.15
1125b0087115 v4.15 17 fc72d1d54dd9 v4.15
fd3ba21478d0 v4.15 17 e817f85652c1 v4.15
a0a97f2a1a23 v4.15 17 7c23b629a808 v4.15
72d146578950 v4.15 19 9c40546c012c v4.15
9ae148f80a96 v4.15 19 e2e6771c6462 v4.15
912ec316686d v4.15 20 a33b2d0359a0 v4.14-rc1
03a0120f75df v4.15 18 3b49e2e94e6e v4.15
20651cefd25f v4.15 18 03d13b6868a2 v4.15
a9c324be72b2 v4.15 16 7f1c684a8966 v4.15
82aaff2f6344 v4.15 16 e817f85652c1 v4.15
141b52a98ab4 v4.15 16 e817f85652c1 v4.15
203f8c250e21 v4.15 18 3a0a529971ec v4.14
8abef10b3de1 v4.15 20 0d52af590552 v4.15
16ae30ea17cd v4.15 20 8d52af6795c0 v4.15
08b5a6e2a769 v4.15 20 04ced159cec8 v4.15
c22c0ae5232e v4.15 20 bdc0f9452a39 pre-v3.0
b9820a31691b v4.15 20 e2860e1f62f2 v4.12
38b1f0fb42f7 v4.15 20 bc85734b126f v4.2
7defa77d2bac v4.15 20 39be40ce066d v4.13
2490cdf6435b v4.15 18 033ddf12bcf5 v4.15
f953f0f89663 v4.15 19 27c5b17cd1b1 v4.1
4c8cf8df2f42 v4.15 16 2f3bef4b247e v4.15
918147446463 v4.15 20 f4e8708d3104 v4.15
2b23d9509fd7 v4.15 20 2908d778ab3e pre-v3.0
4a491b1ab11c v4.15 20 2908d778ab3e pre-v3.0
72dd831e24cc v4.15 18 375ef2b1f0d0 v4.14
fb8baefc7b2d v4.15 18 30f1c0421ec5 v4.15
4471879acf92 v4.15 21 f69316d62c70 v4.13
61f5acea8737 v4.15 19 7d06d5895c15 v4.15
b71b25fed1c3 v4.15 23 0e58d0cdb3eb v4.15
a72d3b5dc68d v4.15 41 7df4f9a9f066 v4.14
f102852f980e v4.15 19 fea23fb591cc v4.15
18838d0cc026 v4.15 19 9dc2145d910e v4.14
f9fd82a9f1e4 v4.15 19 46a3df9f9718 v4.13
5bad95a1e55f v4.15 19 a8e8b7ff3517 v4.13
cf72fa63162c v4.15 19 46a3df9f9718 v4.13
57ffee737b36 v4.15 19 496d03e960ae v4.13
94bfaafac9d2 v4.15 19 496d03e960ae v4.13
d2a5dca84048 v4.15 19 76ad4f0ee747 v4.13
7ea5cbdc6609 v4.15 19 46a3df9f9718 v4.13
200a88c69d26 v4.15 19 46a3df9f9718 v4.13
734dc065fc41 v4.15 21 a6f7d2aff623 v4.12
ac1181c60822 v4.15 19 204cc5c44fb6 v3.13
c3929a980b16 v4.15 19 7bc04215a66b v4.15
364bea50dbea v4.15 19 7bc04215a66b v4.15
5896351ea936 v4.15 18 f4d7e40a5b71 v4.15
80cd8f0175d0 v4.15 20 b0a9c37b0178 v4.15
7094dc2b537b v4.15 20 71bb8a1b059e v4.15
0132af05fc8b v4.15 20 d52d7a1be02c v4.15
c6059879be29 v4.15 21 f6a118a800e3 v4.14
c66234cfedfc v4.15 21 f0447f6cbb20 v4.8
eaadb1caa966 v4.15 21 3d9ff34622ba v3.18
b837b28394fb v4.15 21 ab9e00cc72fa v4.14
845ab4009260 v4.15 21 7feb2f786a46 v4.14
0f70db70339d v4.15 20 abefd6741d54 v4.11
313ccb961594 v4.15 17 4af57ef28c2c v4.0
c7fc46fd1410 v4.15 20 9bdd203b4dc8 pre-v3.0
04e491ca9df6 v4.15 20 2763ea0585c9 v4.10
335eaedce461 v4.15 20 290a6ce11d93 v4.10
7b9ebe428266 v4.15 20 290a6ce11d93 v4.10
b9a3589332c2 v4.15 20 bc2b7dab629a v4.7
5c256045b87b v4.15 21 54746dabf770 v4.12
3e8052d90d24 v4.15 21 dfa3cbb83e09 v4.15
a5a86a7f87d7 v4.15 21 a818aa5f967b v4.15
d593574aff0a v4.15 21 9e556dcc5577 v3.12
68021bf4734d v4.15 25 6d0866cbc2d3 pre-v3.0
6c9f0ce0dffe v4.15 25 5e572cab92f0 v4.12
b3e958ce4c58 v4.15 36 5e572cab92f0 v4.12
0b66f2a05a80 v4.15 28 125a22105410 v4.14
2023b0524a63 v4.15 21 a52db659c79c v4.6
5628a8ca1414 v4.15 23 e1e146b1b062 v4.7
0e88bb002a9b v4.15 16 0983b31849bb pre-v3.0
9996b049f641 v4.15 21 9be6a5d788b0 v4.14
883dd4c1b8d9 v4.15 25 c2a6a07afe4a v4.15
2086dd35705f v4.15 25 c2a6a07afe4a v4.15
9b25bd0368d5 v4.15 22 a33801e8b473 v4.14
33c30a8b68cf v4.15 19 8d1a77f974ca v4.15
74bd5d56bfcb v4.15 19 5ed99fb421d4 v4.15
c957f8b3e2e5 v4.15 22 1b44c5a60c13 v4.12
db29bf7b1cb1 v4.15 23 22f37a242925 v4.15
55448d85ae2f v4.15 23 90120d15f4c3 v4.15-rc5
3522f867c13b v4.15 21 1195a098168f pre-v3.0
b4fac96d9a67 v4.15 20 5ccda64d38cc v4.15
404e6bea1066 v4.15 17 9b54470afd83 v4.14
c5313ae8e4e0 v4.15 23 a2d0321dd532 v4.10
f4e8708d3104 v4.15 23 3bc8070fb75b pre-v3.0
5c665aeb65aa v4.15 23 3a8cefbfc5ee pre-v3.0
02ee9da34787 v4.15 23 99db9494035f v4.10
809cb6955650 v4.15 23 8966e28d2e40 v4.14
ee7e16b66a76 v4.15 21 bafbdd527d56 v4.15
9dcfbc182f1a v4.15 30 eec874ce5ff1 v4.14
7dbc7f5f4904 v4.15 25 05359be1176b v4.12
21020812c9e1 v4.15 24 bb06664a534b v4.14
2ab56a59ca88 v4.15 24 35782b233f37 v4.9
21602e1a5578 v4.15 21 2a93c1a3651f v4.15
571e3f3a8e65 v4.15 39 86a10c802362 pre-v3.0
e48e5e198fb6 v4.15 24 1a1c116f3dcf v4.13
9540d977618c v4.15 22 a53851e2c321 v4.15
d0ecbbbe518e v4.15 28 a160a4652973 v4.14
44edf2f89791 v4.15 22 7a4fa29106d9 v4.15-rc3
8234af2db361 v4.15 22 428a68af3a7c v4.15-rc3
bcecb4bbf88a v4.15 22 c5ad119fb6c0 v4.15
912ed8afd643 v4.15 23 8d52af6795c0 v4.15
c127f98ba9ab v4.15 23 db809859c8ce v4.15
30322bcf82d7 v4.15 25 02e479808b5d v4.14
63f1e05f7fe9 v4.15 23 bcf23c79c4e4 v4.10
d1bf2d30728f v4.15 23 8cd84092d35e v3.15
85afe608f5f3 v4.15 26 2defb2729217 v4.15
98fb3a34736d v4.15 27 9d404411091c pre-v3.0
566bd8902e7f v4.15 28 1fb47e0a9ba4 v4.8
b0ab681285aa v4.15 28 305410ffd1b2 v4.8
9cb18db0701f v4.15 29 b985172b328a v3.12
cf4881c12935 v4.15 27 763c5bd045b1 v4.12
3525c7c3bd2b v4.15 27 ed74f8a8a679 v4.12
0aa0c12262fd v4.15 26 49859e55e364 v4.14
cd6f769fdea7 v4.15 26 49859e55e364 v4.14
279a0cd0e02a v4.15 26 49859e55e364 v4.14
5e498ff117c1 v4.15 26 49859e55e364 v4.14
12a26c298d2a v4.15 26 afe76c8fd030 v4.2
9b93ab981e3b v4.15 23 efdc810ba39d v4.6
4484e2994887 v4.15 23 fba53f7b5719 v4.7
18a89ab76681 v4.15 23 1bd27b11c1df v4.9
c94f31b526fe v4.15 18 033ddf12bcf5 v4.15
aeebc6ba88ba v4.15 29 621174a92851 pre-v3.0
bc137dfdbec2 v4.15 54 3c10bbde10fe v4.7
9f3e13c74e1b v4.15 54 b7f0b8ad25f3 v4.13
201b56737f4e v4.15 29 15 03fbdb2fc2b8 v4.15
0136f852be24 v4.15 26 b26ff127c52c v4.15
4f8f7809e47b v4.15 26 78b4af312f91 v4.15
4e2b53a5cb5a v4.15 25 309fa3470fca v4.14
a42b63c1ac19 v4.15 23 564c274c3df0 v3.4
fd4a3e2828b4 v4.15 23 8ad11fb6b073 pre-v3.0
8ec69574031b v4.15 23 09215598119e v4.15
e210a0a9489d v4.15 22 cad4e209c102 v4.13
926aae273019 v4.15 25 8b10ba783c9d v4.14
efac5ac052d9 v4.15 25 233c1955835b v4.12-rc3
836df24a7062 v4.15 23 482d2e9c1cc7 v4.15
3c1490913f3b v4.15 23 8d1a77f974ca v4.15
aada9ce644e5 v4.15 23 f4d7e40a5b71 v4.15
70a87ffea8ac v4.15 23 f4d7e40a5b71 v4.15
820da5357572 v4.15 23 309795f4bec2 pre-v3.0
71b83869a5e3 v4.15 23 cc9bb43ab394 v4.14
f34ffffdcf06 v4.15 23 496d03e960ae v4.13
203dab8395d9 v4.15 16 94162666cd51 v4.15
214bb1c78a4d v4.15 23 afb4c97d90e6 v4.15
2a1c61b38078 v4.15 27 14c416336820 v4.9
86aacdca6677 v4.15 25 007e6e5c5f01 v4.11
fd05e57bb35a v4.15 23 cc2b14d51053 v4.15
fefa06811cc7 v4.15 25 f27b4746f378 v4.5
ce9ce74145aa v4.15 25 f27b4746f378 v4.5
df8b13a1b233 v4.15 25 786c6adb3a94 v4.5
3020f252c3aa v4.15 25 e5e74b61b165 v4.9
fe99afd1febd v4.15 25 86dbcd0f12e9 v4.5
af808ece5ce9 v4.15 25 ee1c60b1bff8 v4.11
757fffcfdffb v4.15 35 94f050246b42 v4.7
0e4b52942b1c v4.15 26 142a27f0a731 v4.14
fc8517bf627c v4.15 26 cce2ea8d90fe v4.12
106840c41096 v4.15 26 cce2ea8d90fe v4.12
d94c3d65df24 v4.15 26 92932d03c2b3 v4.2
1f83f4d15d28 v4.15 26 66008d4230f6 v4.2
6d6e71feb183 v4.15 26 a21eb94fc4d3 v4.13
4e889d62b89d v4.15 34 7e6059020894 v4.15-rc1
f8f8f1d04494 v4.15 26 fc8726a2c021 v4.7
f681e08f671a v4.15 27 3bb48ba5cd60 v4.15
0b598e4f877b v4.15 26 0ba002bc4393 v4.15
ebae8d07435a v4.15 25 7772fdaef939 v4.15
39f55c61da49 v4.15 27 b95800eeef09 v4.15
013e2b722aa9 v4.15 29 c4755fb9064f v4.15
1696784eb7b5 v4.15 26 97f69747d8b1 v4.7
4f74d80971bc v4.15 24 1c2a088a6626 v4.15
8804517e9fc1 v4.15 28 c97c4090ff72 v4.4
afb4c97d90e6 v4.15 25 ef7baf5e083c v4.15
50670b6ee9bc v4.15 25 1a66a836da63 v4.13
a7343211f007 v4.15 25 5a963eb61b7c v4.15
dd8d5b8c5b22 v4.15 25 84e54fe0a5ea v4.13
293a1991cf0d v4.15 25 5a963eb61b7c v4.15
83476bfaf6ac v4.15 29 7f4c9176f760 v4.15
a65eb01ffcc5 v4.15 25 29ad20b22c8f v4.15
14c6547d6df6 v4.15 37 77affa31722b v4.10
a049815229b6 v4.15 28 e1190083b89b v4.15
11cc84e708db v4.15 28 3f268f5d6669 v4.15
981ed1bfbc6c v4.15 28 6e5e959dde0d v3.3
57ad33a307bf v4.15 24 048765ad5af7 v4.7-rc3
6562fbcf3ad5 v4.15 29 1c8e600440c7 v3.19
81ee6f1ef9b1 v4.15 28 06e733e41f87 v4.14
16c8619a7c53 v4.15 26 bccd3b831185 v4.15
1df94c3c5dad v4.15 26 c5ad119fb6c0 v4.15
dc5591dc9c03 v4.15 29 b7e607bf33a2 v4.13
a1c484b3a09b v4.15 32 1502efd2d596 v4.13
4f6c11044f51 v4.15 32 68d9c47b1679 v4.13
b93a25e120d5 v4.15 32 efb9ab67255f v4.9
83c75ddd816e v4.15 29 5d2917d469fa v4.4
94b9d290b753 v4.15 29 9af275be15f7 v4.12
eac56aa3bc8a v4.15 29 0c4ffcfe1fbc v3.17
35ad61921f49 v4.15 30 d74679911610 v4.11
0c47cd7a9b6c v4.15 29 d74679911610 v4.11
ad4a5becc689 v4.15 30 f8aed6ec624f v4.11
793ae04c46cf v4.15 40 8caef1fa9176 v4.14
80d7d7a904fa v4.15 41 f1f0366dd6be v4.10
5ee7f784cd32 v4.15 26 db496944fdaa v4.15
8edcda1266f9 v4.15 40 555e38d27317 v4.12
edf1a84fe37c v4.15 27 c5d76f130b28 v3.18
7baaa49af371 v4.15 27 ce117ffac2e9 v3.10
3fb04cb0bea5 v4.15 27 f6ba181ada55 v4.15
d91e8db5b629 v4.15 26 94d7d8f29287 v4.15
ae3e13373b87 v4.15 26 94d7d8f29287 v4.15
97913bcbe6da v4.15 33 9a761e436843 v3.9
8b874514c11d v4.15 26 9525ae83959b v4.13
2794ffc441dd v4.15 26 73970055450e v4.13
20b56ed9f8ad v4.15 26 73970055450e v4.13
22ec1a2aea73 v4.15 29 f5509cc18daa v4.8-rc1
4e8581eefe72 v4.15 30 bc15ed663e7e v4.10
a31d73c3a721 v4.15 27 7b6da818d86f v4.15
1dfc65cef481 v4.15 31 57f9156bc620 v4.5
e4fdf59bcce3 v4.15 31 cac68a56e34b v4.5
fcab5651fae4 v4.15 31 a8b805f3606f v4.5
19ac50f65537 v4.15 30 f6731485a519 v4.15
d48b8c58c57f v4.15 85 2c7589af3c4d v4.1
5f454bdf6353 v4.15 34 12278dc7c572 v4.12
4a29c0db69e5 v4.15 27 92f0292b35a0 v4.15
24f8d233074b v4.15 54 da2a68b3eb47 v4.10
914d65f3f013 v4.15 54 2501b015313f v4.12
4bcd615fad6a v4.15 54 ee142889e32f v4.5
50a0d71a5d20 v4.15 34 a841178445bb v4.1
e3b03bf29d6b v4.15 31 1c199f2878f6 v4.3
33aa69ed8aac v4.15 28 b056f8f3d6b9 v4.12
6a28fd2bbc94 v4.15 30 3840ed9548f7 v4.13
8af016aa5a27 v4.15 30 23f5b3fdd04e v4.0
024778095aa3 v4.15 27 8fa7b9b6af25 v4.15
28b2d7d04b00 v4.15 27 0e6c4955e149 v4.15
da762863edd9 v4.15 27 e1e798e3fd93 v4.15
c0d451c86ca2 v4.15 27 b93dd49c1a35 v4.7
99cd05c43baa v4.15 41 eef57324d926 v4.10
7cdeb37c66e9 v4.15 28 9271c0ca573e v4.15-rc1
413fd2f5c023 v4.15 29 fb9987d0f748 pre-v3.0
1cb26030f349 v4.15 39 329d88da4df9 v4.5-rc2
45392ff6881d v4.15 35 77e0be12100a pre-v3.0
7de241f3b705 v4.15 29 37bf241b8e7b v4.15
e4779162f737 v4.15 29 2635664e6e4a v4.12
b96a696fb2a8 v4.15 28 0eed8d097612 v4.13
c7b8f77872c7 v4.15 32 1da177e4c3f4 pre-v3.0
f4e2298e63d2 v4.15 28 f371b304f12e v4.15
41f61db2cd24 v4.15 29 5023a6db7319 v4.13
11b2b6531025 v4.15 29 5023a6db7319 v4.13
0cb65d421a2a v4.15 32 1c8f1da5d851 v4.14
302d6424e4a2 v4.15 32 30dc5e63d6a5 v3.15
ac6dbf7fa470 v4.15 32 e4b2d06892c7 v4.14
11e472320668 v4.15 32 d39f0010e409 v4.15
1b0a83ac04e3 v4.15 29 e8fcfcd5684a v3.15
2d2bccef7192 v4.15 53 18bb8a6c899d v4.14
e6756d7c1a7e v4.15 59 2165e2b9cba2 v4.14
b1728ff617f8 v4.15 36 b1cb7372fa82 v4.14
baed3c4bc4c1 v4.15 36 c5f5d0f99794 v4.2
7d622351c941 v4.15 46 81c0ed21aa91 v4.14
1d0d064307cb v4.15 31 e948bc8fbee0 v4.13
0f27cff8597d v4.15 31 9c4aa1eecb48 v4.10-rc2
d72ebbcc4460 v4.15 28 83c566806a68 v4.15
180153efd3cf v4.15 43 95e300c052fd v4.14
330dada5957e v4.15 37 a9cb97c3e628 v4.15
33369d5a2054 v4.15 33 dfba2174dc42 v4.15
1b19b95169cd v4.15 32 14ab8896f5d9 v4.9
87b3524cb505 v4.15 32 261a4351844b v4.8
b7e27bc1d42e v4.15 30 1ac202e978e1 v4.11
a9cb97c3e628 v4.15 38 d73dcf0cdb95 v4.14
81394e3f6df8 v4.15 53 2193e1604eac v3.14
3a0b3bbbff0f v4.15 53 c87b9c601ac8 v3.13
776bc27fd8ab v4.15 48 d6a2289d9d6b v4.11
a0b586fa75a6 v4.15 31 46e6b992c250 v4.15
a0e94598e6b6 v4.15 31 d597580d3737 v4.11
2361bfb055f9 v4.15 35 41e3efd07d5a v4.15
0562315b8637 v4.15 35 a4080225f51d v4.15
23a185254ace v4.15 39 9d4579a85c84 v4.11
d2383318c5a6 v4.15 39 cb39f61e9b1e v4.14
65a12b3aafed v4.15 35 1082e2703a2d pre-v3.0
cd430a244cd5 v4.15 35 fa8cc38165c2 v4.15
df807a19957c v4.15 35 6dad4e8ab3ec v4.15
9db09e3bad65 v4.15 35 eb9ba37dc15a v4.13
461bd6227ede v4.15 29 9f31d1063b43 v4.15
acb1feab320e v4.15 35 7c0482e3d055 v3.4-rc6
5e54b3c12027 v4.15 32 88ca59d1aaf2 v4.14
936e5d8bdfa7 v4.15 32 5342b77c4123 pre-v3.0
5a6a0445d1ed v4.15 32 c22a3f48ef99 v4.11
df45bf84e4f5 v4.15 32 efbf78973978 v4.15
3106507e1004 v4.15 37 b94d22d94ad2 v4.14
0ff6c131c842 v4.15 39 a7e4cfb0a7ca v4.14
5a18c2434f8b v4.15 39 4a5f8ae50b66 v4.12
d5ac225c7d64 v4.15 39 ab33d5071de7 pre-v3.0
124da8f6118b v4.15 31 96f84061620c v4.15
54ce89177988 v4.15 36 4b2d9fe87950 v4.14
d259f91c0efb v4.15 39 d3202d1981dc v4.13
c4b54e138a26 v4.15 34 8a39f597bcfd v4.15
59365b9efd48 v4.15 33 9f66a397c877 v4.15
64082568dd1e v4.15 26 8d46e28fb508 v4.14
3e72be177cf1 v4.15 49 edde361711ef v4.5
9428088c90b6 v4.15 33 1277eed5fecb v4.11
19759033e0d0 v4.15 35 726b85487067 v4.10
bf07ef86e882 v4.15 35 6eb54715b54b v4.4
5d3300a9b8b1 v4.15 35 726b85487067 v4.10
ba743f9148e9 v4.15 35 726b85487067 v4.10
76f9a2dd4c60 v4.15 35 726b85487067 v4.10
b0dcce746b32 v4.15 35 6eb54715b54b v4.4
23c645595dab v4.15 35 15f30a575228 v4.11-rc2
4005a995668b v4.15 35 726b85487067 v4.10
5ef696aa9f3c v4.15 35 726b85487067 v4.10
3be63b1e1838 v4.15 35 bdbe24de281e v4.13
414d9ff3f803 v4.15 35 726b85487067 v4.10
2d73ac6102d9 v4.15 35 726b85487067 v4.10
25ad76b703d9 v4.15 35 726b85487067 v4.10
a084fd68e1d2 v4.15 35 726b85487067 v4.10
d68b850e1bfb v4.15 35 09620eeb62c4 v4.12
a01c77d2cbc4 v4.15 35 726b85487067 v4.10
22e786ea47f8 v4.15 35 726b85487067 v4.10
2e01d0ba868e v4.15 35 726b85487067 v4.10
283c35062f77 v4.15 37 b6b247cd5e37 v4.15
2d9bbd02c540 v4.15 37 3558fe900e8a v3.14
010e3e68cd9c v4.15 34 5888fc9eac3c v4.15
4c3c9a9d0d97 v4.15 35 ad36a27959ca v4.15
33b6cb58cbb6 v4.15 35 a60eac3239f0 v4.15
bc29489f7120 v4.15 34 b862a648de3b v4.15
89f3f35620c7 v4.15 35 c777990fb45b v4.15
92425c40676d v4.15 33 8c95f773b4a3 v4.14
84a1695a6f92 v4.15 39 8428e5ad750d v4.15
46df8824982e v4.15 38 737aed947f9b v4.13
5888fc9eac3c v4.15 52 02bef8f98d26 v4.8
81da3bf6e3f8 v4.15 33 ae8223de3df5 v4.15
b0e9fe1ba7f8 v4.15 33 addf9b90de22 v4.15
35a8a148d8c1 v4.15 37 6de783b6f50f v3.4
66bda1a68ba1 v4.15 36 9d75b8c0b999 v4.12
a2407f4bd1f3 v4.15 36 9d75b8c0b999 v4.12
8472b529e113 v4.15 41 ef1f09eca74a v4.7
8e60b7f195d2 v4.15 29 a2ae95af9646 v4.15
f745e9cc7e40 v4.15 29 e011c6ce2b4f v4.15
53ff2641a817 v4.15 36 dab91783338b v4.15
9753c21f55d4 v4.15 33 80274abafc60 v4.15
e2f930aaa351 v4.15 36 4a97a3da420b v4.13
51bd7125f740 v4.15 37 5da4e04ae480 v4.15
006c23327f8d v4.15 48 e8668bbcb0f9 v4.14
da5f99eaccc1 v4.15 51 d6c0511300dc v4.14
ea26c96d59b2 v4.15 35 b2d6ef70614e v4.15
f5a26acf0162 v4.15 42 7981c0015af2 v4.0
681648e67d43 v4.15 34 8edc3affc077 v4.11-rc2
23a7068ec581 v4.15 52 555e38d27317 v4.12
09edcb647542 v4.15 41 db43e6c473b5 v3.9
bf59fddde1c3 v4.15 41 84edbeeab67c v3.3
7d54d0d38ec4 v4.15 41 dbc4deda03fe v4.14
8fedd64dabc8 v4.15 52 b7792d8b54cc v4.5
5816d9cbc0a0 v4.15 46 51f5a0963984 v4.15
12a3cc8424fe v4.15 34 638f5b90d460 v4.14
b5e0a9418e09 v4.15 36 4e90a6e22272 v4.15
8f52df50d936 v4.15 41 7f866986e705 v4.8
0ba23a211360 v4.15 37 d7dbefc45cf5 v4.11
116ceac9741e v4.15 40 93faccbbfa95 v4.10
bbc3e4710114 v4.15 40 93faccbbfa95 v4.10
51de082570e5 v4.15 35 39323e788cb6 v4.14
7be4b5dc7ffa v4.15 36 14e3e295b2b9 v3.12
e153db03c6b7 v4.15 36 4341881d0562 v3.16
13f9d59cef91 v4.15 42 dbe776c2ca54 v4.14
ecf73eb2d27d v4.15 52 5ab57c702069 v4.8
f250b09c7795 v4.15 37 9aefcab0de47 pre-v3.0
f2eef045de9d v4.15 42 c4f07ecee22e v4.12
bcbec31ce500 v4.15 38 67680d3c0464 v4.14
65c792305768 v4.15 42 0a8adf584759 v3.16
561a10b6a15b v4.15 42 fe304143b0c3 v3.10
59b6d859ff0e v4.15 42 07646d9c0938 v3.6
9987da4b5dcf v4.15 52 b901bb89324a v4.14
87aae50af730 v4.15 43 c694b233295b v4.10
7814f552ff82 v4.15 43 2f47d5804311 v4.14
0f5eb1545907 v4.15 44 0fa20cdfcc1f v4.9
03310a15484a v4.15 42 4f04c210d031 v4.12
3eb23426e174 v4.15 43 7e5b796cde7e v4.13
4d67c2e7f60d v4.15 45 a34a0b6da225 v4.15
10c9ef045a7e v4.15 45 3991c5c80bea v4.15
eb14ed1ad7b6 v4.15 45 291e0c4994d0 v4.12
1b377924841d v4.15 44 420ae8451a22 v3.14
2de2d0b063b0 v4.15 53 1ebaa0b9c2d4 v4.7
75eccf5ed832 v4.15 29 738f66d3211d v4.7
544e92581a2a v4.15 45 1bc021e81565 v3.14
d72f4f03854d v4.15 46 02a5ab18d366 v4.13
ad88d7fc6c03 v4.15 53 366e39b4d2c5 v4.4
aa34a6fc2722 v4.15 36 c41937fd994a v4.15
bb9e0d4bca50 v4.15 53 56f9eac05489 v3.4
c4deb62d7821 v4.15 53 56f9eac05489 v3.4
141a0895d5ad v4.15 36 b46a33e271ed v4.15
b4e3c935b231 v4.15 36 6060b6aec03c v4.15
675f7ff35bd2 v4.15 57 6ac43272768c v4.14
0cfecb7c4b9b v4.15 54 8f067837c4b7 v4.14
41729bf2248b v4.15 87 21cc6431e0c2 v4.14
742015ff12ae v4.15 61 e422267322cd v4.11
2f0af8600e82 v4.15 61 8e99b6d4533c v4.13
19993b82a571 v4.15 61 33974a414ce2 v4.14-rc8
fa48c892645d v4.15 61 a14390fde64e v4.14
82806c3aae74 v4.15 61 e669e833da8d v4.14
c534612e780c v4.15 59 9eb143bbec7d v4.11
f2880e04f3a5 v4.15 55 89ea20b930cb v4.13
24f8a29af4af v4.15 59 1e3197d6ad73 v4.14
8cf80a2e4b31 v4.15 38 0a53bc07f044 v4.14
7469c62cb654 v4.15 38 37cd33006d02 v4.15
34991bd48c92 v4.15 38 0a60797a0efb v4.15
f577a03ba920 v4.15 38 dab91783338b v4.15
0c51ef8684e3 v4.15 38 45d58b40292b v4.15
d3dbf43c56f9 v4.15 61 6619a53ef757 v3.13
556fe36d09da v4.15 38 b6b178a77210 v4.15
94dec87159af v4.15 48 547da76b5777 v4.15
f72b84c677d6 v4.15 48 b7048ea12fbb v4.11-rc3
1ab22356b37a v4.15 48 d07f0e59b2c7 v4.9
2f6a3783833d v4.15 48 f2123818ffad v4.14
5684514ba9dc v4.15 46 894ccebee2b0 v4.11
f991c492aa55 v4.15 48 87701b4b5593 v4.14
ebcaa1ff8b59 v4.15 47 cf6e7bac6357 v4.13
50f365cde4ff v4.15 52 b9f19259b84d v4.14
e5330ac1f50b v4.15 41 bcbd5c33a342 v4.15
1d033beb20d6 v4.15 50 2889caa92321 v4.12
85b4587f8e94 v4.15 39 2f733d6194bd v4.14
4ab09d0ec3c8 v4.15 41 39bf4de89ff7 v4.15
547da76b5777 v4.15 50 d1b48c1e7184 v4.13
bea6e987c1ff v4.15 50 96d776345277 v4.9
886c6b8692ba v4.15 50 9301397a63b3 v4.12
22a8a4fc93b1 v4.15 51 14826673247e v4.14
191f896085cf v4.15 48 eec688e1420d v4.10
64b80085dd36 v4.15 51 1f181225f8ec v4.14
4a118ecbe99c v4.15 51 beecec901790 v4.14
102712a32ff5 v4.15 39 9e32e16e9e98 v4.6

Re: [Ksummit-discuss] bug-introducing patches

[Willy Tarreau]

On Thu, May 03, 2018 at 04:14:57PM +0000, Sasha Levin wrote:
> I tried looking at a few commits that came in on -rc7, and I see quite a
> few cases where a commit was merged to Linus' tree in about 24 hours
> after it was authored. Or maintainers who just wrote it, pushed it in,
> and shipped in to Linus.
> 
> I've attached the data I used. The columns are as follows:
> 
> 1. Commit ID
> 2. When was it merged
> 3. How many days it spent in -next
> 4. What commit did it fix
> 5. When was that commit merged

> b6cdbc85234b v4.16-rc7 5 ca254490c8df v4.3
> 82dd0d2a9a76 v4.16-rc7 5 8f58336d3f78 v4.2
> 5807b22c9164 v4.16-rc7 5 6c8702c60b88 v4.9
> f97c3dc3c0e8 v4.16-rc7 5 4c4dbb4a7363 v4.15
(...)

I like this (not what was done but the analysis).

I'd argue that a small part of them there are very likely valid reasons
(really obvious fix, security issue etc) but it seems there are quite a
large number of them here.

Now I understand what makes me uneasy with what I'm seeing here. As I
mentioned, -rc is for people who want to see bugs before their users.
-rc7 will ensure almost everyone discovers the fix at the same time,
because the next version will be 4.16, the first of a stable release,
the one that users are expected to trust.

So probably that we have to educate/encourage developers *not* to submit
fixes for old bugs that late in the cycle and to rather wait for the next
version so that it cooks in -rc for a while before hitting users, knowing
that these fixes will be backported to stable anyway once considered valid.

Just like Greg has its "WTF" script to remind some developers that their
patch is not suited to -stable, I think you could, based on your work,
try to spot regressions introduced by late patches that fall in the
category you've filtered and emit such WTF messages to the original
patch's authors/committers.

It's important to do it only when these patches cause breakage though,
because we don't want to needlessly delay fixes when they're considered
certain or well tested. Only when they cause trouble.

For me the rule seems simple to understand, every submitter should
think like this late in the cycle :

   "you're sending a patch that is going to be part of a stable kernel
    in no more than 2 weeks, possibly affecting all users upgrading to
    that kernel if you did something wrong. Are you really certain you
    want this patch merged now, that it got sufficient testing and that
    it cannot wait for next -rc1 to get broader exposure first ?"

I'm pretty sure that most of the time it will be "sure I want it now"
and there will be no problem, which is fine as it automatically reduces
the number of bugs in releases. Some may reconsider their submission.
Some may get caught by your automated script if a later commit fixes
an issue introduced by their patch. And there public shaming is the
only option (or maybe only the second time if you really want to be
nice).

Willy

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin via Ksummit-discuss]

On Thu, May 03, 2018 at 06:35:16PM +0200, Willy Tarreau wrote:
>On Thu, May 03, 2018 at 04:14:57PM +0000, Sasha Levin wrote:
>> I tried looking at a few commits that came in on -rc7, and I see quite a
>> few cases where a commit was merged to Linus' tree in about 24 hours
>> after it was authored. Or maintainers who just wrote it, pushed it in,
>> and shipped in to Linus.
>>
>> I've attached the data I used. The columns are as follows:
>>
>> 1. Commit ID
>> 2. When was it merged
>> 3. How many days it spent in -next
>> 4. What commit did it fix
>> 5. When was that commit merged
>
>> b6cdbc85234b v4.16-rc7 5 ca254490c8df v4.3
>> 82dd0d2a9a76 v4.16-rc7 5 8f58336d3f78 v4.2
>> 5807b22c9164 v4.16-rc7 5 6c8702c60b88 v4.9
>> f97c3dc3c0e8 v4.16-rc7 5 4c4dbb4a7363 v4.15
>(...)
>
>I like this (not what was done but the analysis).
>
>I'd argue that a small part of them there are very likely valid reasons
>(really obvious fix, security issue etc) but it seems there are quite a
>large number of them here.
>
>Now I understand what makes me uneasy with what I'm seeing here. As I
>mentioned, -rc is for people who want to see bugs before their users.
>-rc7 will ensure almost everyone discovers the fix at the same time,
>because the next version will be 4.16, the first of a stable release,
>the one that users are expected to trust.
>
>So probably that we have to educate/encourage developers *not* to submit
>fixes for old bugs that late in the cycle and to rather wait for the next
>version so that it cooks in -rc for a while before hitting users, knowing
>that these fixes will be backported to stable anyway once considered valid.
>
>Just like Greg has its "WTF" script to remind some developers that their
>patch is not suited to -stable, I think you could, based on your work,
>try to spot regressions introduced by late patches that fall in the
>category you've filtered and emit such WTF messages to the original
>patch's authors/committers.
>
>It's important to do it only when these patches cause breakage though,
>because we don't want to needlessly delay fixes when they're considered
>certain or well tested. Only when they cause trouble.

I tried pulling all the fixes that went in 4.17 (so far) for bugs that
were introduced as fixes in the v4.16 cycle, I got this list:

d65026c6c62e v4.16-rc7 5 6b1e6cc7855b v4.7   d14d2b78090c
63489f8e8211 v4.16-rc6 13 045c7a3f53d9 v4.11-rc6   5df63c2a149a
5dcd8400884c v4.16-rc6 6 0759e552bce7 v4.7   bd28899dd34f
0ef58b0a05c1 v4.16-rc6 6 0cf737808ae7 v4.14   a56d99d71466 7992894c305e 2afc5d61a719
8936ef7604c1 v4.16-rc6 6 6c8702c60b88 v4.9   a957fa190aa9
bbc09e7842a5 v4.16-rc6 6 65a206c01e8e v4.13   3239534a79ee
6a2cf8d3663e v4.16-rc5 12 d64d6c5671db v4.15   6d6340672ba3
859d880cf544 v4.16-rc4 14 b68a68d3dcc1 v4.15   8420f71943ae
e39a97353e53 v4.16-rc4 16 2a842acab109 v4.12   cbe095e2b584
a27fd7a8ed38 v4.16-rc4 19 f214f915e7db v4.13   bffd168c3fc5
0f9da844d877 v4.16-rc2 16 28128c61e08e v4.16-rc2   a95b37e20db9
7324f5399b06 v4.16-rc2 19 186b3c998c50 v4.14   51568d69407d
e78c637127ee v4.16-rc3 25 187d7967a5ee v4.4   e988867fd774
ca9eee95a2de v4.16-rc3 25 d717f7352ec6 v4.12   e988867fd774

So out of 755 commits, 14 have been fixed, that's about 2% and we're not
even done with 4.17.

>For me the rule seems simple to understand, every submitter should
>think like this late in the cycle :
>
>   "you're sending a patch that is going to be part of a stable kernel
>    in no more than 2 weeks, possibly affecting all users upgrading to
>    that kernel if you did something wrong. Are you really certain you
>    want this patch merged now, that it got sufficient testing and that
>    it cannot wait for next -rc1 to get broader exposure first ?"
>
>I'm pretty sure that most of the time it will be "sure I want it now"
>and there will be no problem, which is fine as it automatically reduces
>the number of bugs in releases. Some may reconsider their submission.
>Some may get caught by your automated script if a later commit fixes
>an issue introduced by their patch. And there public shaming is the
>only option (or maybe only the second time if you really want to be
>nice).

I'd much prefer to blame this on maintainers. Authors should be able to
submit a patch whenever they feel like it, maintainers should only merge
a patch in when it's right.
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Willy Tarreau]

On Thu, May 03, 2018 at 05:29:29PM +0000, Sasha Levin wrote:
> I tried pulling all the fixes that went in 4.17 (so far) for bugs that
> were introduced as fixes in the v4.16 cycle, I got this list:
> 
> d65026c6c62e v4.16-rc7 5 6b1e6cc7855b v4.7   d14d2b78090c
> 63489f8e8211 v4.16-rc6 13 045c7a3f53d9 v4.11-rc6   5df63c2a149a
> 5dcd8400884c v4.16-rc6 6 0759e552bce7 v4.7   bd28899dd34f
> 0ef58b0a05c1 v4.16-rc6 6 0cf737808ae7 v4.14   a56d99d71466 7992894c305e 2afc5d61a719
> 8936ef7604c1 v4.16-rc6 6 6c8702c60b88 v4.9   a957fa190aa9
> bbc09e7842a5 v4.16-rc6 6 65a206c01e8e v4.13   3239534a79ee
> 6a2cf8d3663e v4.16-rc5 12 d64d6c5671db v4.15   6d6340672ba3
> 859d880cf544 v4.16-rc4 14 b68a68d3dcc1 v4.15   8420f71943ae
> e39a97353e53 v4.16-rc4 16 2a842acab109 v4.12   cbe095e2b584
> a27fd7a8ed38 v4.16-rc4 19 f214f915e7db v4.13   bffd168c3fc5
> 0f9da844d877 v4.16-rc2 16 28128c61e08e v4.16-rc2   a95b37e20db9
> 7324f5399b06 v4.16-rc2 19 186b3c998c50 v4.14   51568d69407d
> e78c637127ee v4.16-rc3 25 187d7967a5ee v4.4   e988867fd774
> ca9eee95a2de v4.16-rc3 25 d717f7352ec6 v4.12   e988867fd774
> 
> So out of 755 commits, 14 have been fixed, that's about 2% and we're not
> even done with 4.17.

OK but this is low. Quite frankly, at 2% regressions, even if this is
never fun, it means 98% of the fixes were right. Now just delay them
longer and you'll have 500 commits instead of 755, thus 255 more bugs
unfixed in the release just to try to save 14 wrong ones. *this* is
the problem I'm concerned about by enforcing extra delays on everyone.
This is the reason why in my opinion the most important is to raise
awareness about this so people are more careful or more verbose (and
more detailed commit messages don't hurt, I think all stable maintainers
have many times thought "WTF is this supposed to fix?"), and then remind
everyone that when some get caught abusing, they'll get a public blame.

> >Some may get caught by your automated script if a later commit fixes
> >an issue introduced by their patch. And there public shaming is the
> >only option (or maybe only the second time if you really want to be
> >nice).
> 
> I'd much prefer to blame this on maintainers. Authors should be able to
> submit a patch whenever they feel like it, maintainers should only merge
> a patch in when it's right.

Sorry, wrong word on my side, I also meant maintainers (I very much favor
pushing back to ensure everyone in the chain is responsible for what is
done).

Willy

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin]

On Thu, May 03, 2018 at 07:57:23PM +0200, Willy Tarreau wrote:
>On Thu, May 03, 2018 at 05:29:29PM +0000, Sasha Levin wrote:
>> I tried pulling all the fixes that went in 4.17 (so far) for bugs that
>> were introduced as fixes in the v4.16 cycle, I got this list:
>>
>> d65026c6c62e v4.16-rc7 5 6b1e6cc7855b v4.7   d14d2b78090c
>> 63489f8e8211 v4.16-rc6 13 045c7a3f53d9 v4.11-rc6   5df63c2a149a
>> 5dcd8400884c v4.16-rc6 6 0759e552bce7 v4.7   bd28899dd34f
>> 0ef58b0a05c1 v4.16-rc6 6 0cf737808ae7 v4.14   a56d99d71466 7992894c305e 2afc5d61a719
>> 8936ef7604c1 v4.16-rc6 6 6c8702c60b88 v4.9   a957fa190aa9
>> bbc09e7842a5 v4.16-rc6 6 65a206c01e8e v4.13   3239534a79ee
>> 6a2cf8d3663e v4.16-rc5 12 d64d6c5671db v4.15   6d6340672ba3
>> 859d880cf544 v4.16-rc4 14 b68a68d3dcc1 v4.15   8420f71943ae
>> e39a97353e53 v4.16-rc4 16 2a842acab109 v4.12   cbe095e2b584
>> a27fd7a8ed38 v4.16-rc4 19 f214f915e7db v4.13   bffd168c3fc5
>> 0f9da844d877 v4.16-rc2 16 28128c61e08e v4.16-rc2   a95b37e20db9
>> 7324f5399b06 v4.16-rc2 19 186b3c998c50 v4.14   51568d69407d
>> e78c637127ee v4.16-rc3 25 187d7967a5ee v4.4   e988867fd774
>> ca9eee95a2de v4.16-rc3 25 d717f7352ec6 v4.12   e988867fd774
>>
>> So out of 755 commits, 14 have been fixed, that's about 2% and we're not
>> even done with 4.17.
>
>OK but this is low. Quite frankly, at 2% regressions, even if this is
>never fun, it means 98% of the fixes were right. Now just delay them
>longer and you'll have 500 commits instead of 755, thus 255 more bugs
>unfixed in the release just to try to save 14 wrong ones. *this* is
>the problem I'm concerned about by enforcing extra delays on everyone.
>This is the reason why in my opinion the most important is to raise
>awareness about this so people are more careful or more verbose (and
>more detailed commit messages don't hurt, I think all stable maintainers
>have many times thought "WTF is this supposed to fix?"), and then remind
>everyone that when some get caught abusing, they'll get a public blame.

This is low because we're only about a month in 4.17. Historical 
figures are around 7% for these kinds of commits.

I'm also not trying to argue whether 7% is high or low, only that it's 3
times as many bugs per line of code than what we get from the merge
window.

Isn't the merge window supposed to be the "risky" part?

I understand your concern about delays, I'm not arguing for or against
it, I'm just trying to discuss option.

For example, how about extending the release cycle until the amount of
fixes for regressions introduced in the current merge window drops under
a certain thershold? (so go to -rc20 if we need to).

It has the advantage of less bugs when the kernel is released, it
doesn't stop bug fixes from coming in and it prevents the urge some
folks have to push things in last minute. OTOH, it makes the release
cycle unpredictable time-wise.

Re: [Ksummit-discuss] bug-introducing patches

[Guenter Roeck]

On Thu, May 03, 2018 at 06:12:36PM +0000, Sasha Levin via Ksummit-discuss wrote:
> 
> For example, how about extending the release cycle until the amount of
> fixes for regressions introduced in the current merge window drops under
> a certain thershold? (so go to -rc20 if we need to).
> 
Reminds me of what happened at a previous employer. We had a hard rule that
a product shall not be released unless it has no more than 2 severe or
critical bugs. Solution: Stop testing 2-3 weeks ahead of the scheduled
release day.

Here: We don't want to see -rc20. Solution: Stop applying bug fixes
at -rcX.

Guenter

Re: [Ksummit-discuss] bug-introducing patches

[Willy Tarreau]

On Thu, May 03, 2018 at 06:12:36PM +0000, Sasha Levin wrote:
> I'm also not trying to argue whether 7% is high or low, only that it's 3
> times as many bugs per line of code than what we get from the merge
> window.

Yes but seen differently that's 14 times less bugs than the ones properly
fixed by applying the process which produces these 7%. We can discuss
about ways to improve it, but please consider that it must not reduce
the number of correct fixes represented by the 93% remaining ones.

> Isn't the merge window supposed to be the "risky" part?

"risky" might not be the correct term. Each single line of code comes
with a risk. I'd say the "most risky" part. As I said, what I agree with
the fact that early fixes just before a release have more chances of
affecting users, which in my opinion is the real problem. Education can
help here.

> For example, how about extending the release cycle until the amount of
> fixes for regressions introduced in the current merge window drops under
> a certain thershold? (so go to -rc20 if we need to).

Never works. And Linus already explained it : you cannot stop the development
process. While you're waiting, development continues, and the next merge
window gets twice the number of commits, which causes more than twice the
amount of problems. I've also experienced it in haproxy many years ago. I
made the mistake of saying "I'm finishing this, only 6 months, and I release
1.5". Result: bugs coming in parallel to development stalling progress
forever and it took 4.5 years to release it, or 9 times the expected amount
of time. Now we release approximately on time, missing features go in the
next release, easily testable fixes are merged, complex ones are postponed
for the stable releases with a note in the announce saying "don't play with
this yet, it's broken". We do ship with bugs, we're open about it and we
address them later. Overall this transparency is much appreciated. And we
also do regressions by the way.

Maybe in the end the only thing we're missing is a "known bugs" section in
release announcements, so that those with pending fixes are encouraged to
send a line or two to Linus for inclusion there, having more time to work
on their fixes.

Willy

Re: [Ksummit-discuss] bug-introducing patches

[Al Viro]

On Thu, May 03, 2018 at 02:46:14PM +0000, Sasha Levin via Ksummit-discuss wrote:

> Fixes in -rc cycles:
> rc1 68
> rc2 147
> rc3 88
> rc4 121
> rc5 40
> rc6 193
> rc7 98
> 
> Average days in -next, for a fix, per -rc cycle:
> rc1 27.25
> rc2 21.4286
> rc3 22.5114
> rc4 18.281
> rc5 14.65
> rc6 12.6166
> rc7 8.70408
> 
> Fixes for bugs not introduced in current merge window:
> rc1 40
> rc2 113
> rc3 61
> rc4 79
> rc5 25
> rc6 139
> rc7 72
> 
> So for some reason, there is a rush to push fixes for older bugs (that
> were not introduced in the current merge window) to the point that rc7
> commits that only existed for a few days are merged in to address older
> bugs.

I really wonder how accurate your interpretation of Fixes: is.
Consider e.g. the situation when an old bug is found and partial fixes
applied.  Then we find that those fixes did not cover everything and,
come next cycle, add more on top of those.  Where should Fixes: on
the incrementals point to?  Original commit?  But they won't apply
without the first batch.  The last in the original pile?  But it
would imply (by your metrics) that original fixes had *INTRODUCED*
bugs.

Moreover, what the hell do you suggest in situation when
	* foofs_barf() is b0rken in quite a few ways.  There's an
easily triggerable memory corruptor that can be fixed locally as well
as something else that needs a change of e.g. ->mkdir() calling
conventions to take care of.  The change is mechanical and fairly
simple, but it's already -rc4.

Even though the whole thing is well-understood at that point,
we *can't* apply all 3 - it's too late in the cycle for the
calling conventions' change in the middle of the series.

Should we apply the first fix that cycle, or should it slide to the
next one?  We can't apply #1 + #3 --- #3 won't even compile without
#2.  We can't apply #3 without #1 --- they can be transposed, but
it's nowhere near a mechanical transformation.  So WTF should #3
have in "Fixes:"?

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin via Ksummit-discuss]

On Thu, May 03, 2018 at 05:54:46PM +0100, Al Viro wrote:
>On Thu, May 03, 2018 at 02:46:14PM +0000, Sasha Levin via Ksummit-discuss wrote:
>
>> Fixes in -rc cycles:
>> rc1 68
>> rc2 147
>> rc3 88
>> rc4 121
>> rc5 40
>> rc6 193
>> rc7 98
>>
>> Average days in -next, for a fix, per -rc cycle:
>> rc1 27.25
>> rc2 21.4286
>> rc3 22.5114
>> rc4 18.281
>> rc5 14.65
>> rc6 12.6166
>> rc7 8.70408
>>
>> Fixes for bugs not introduced in current merge window:
>> rc1 40
>> rc2 113
>> rc3 61
>> rc4 79
>> rc5 25
>> rc6 139
>> rc7 72
>>
>> So for some reason, there is a rush to push fixes for older bugs (that
>> were not introduced in the current merge window) to the point that rc7
>> commits that only existed for a few days are merged in to address older
>> bugs.
>
>I really wonder how accurate your interpretation of Fixes: is.
>Consider e.g. the situation when an old bug is found and partial fixes
>applied.  Then we find that those fixes did not cover everything and,
>come next cycle, add more on top of those.  Where should Fixes: on
>the incrementals point to?  Original commit?  But they won't apply
>without the first batch.  The last in the original pile?  But it
>would imply (by your metrics) that original fixes had *INTRODUCED*
>bugs.

It's vaguely close. Beyond the things you mentioned, some commits don't
have a fixes tag, some mention what commit they fixed in the body rather
than a tag, and so on.

This is just an approximation.

>Moreover, what the hell do you suggest in situation when
>	* foofs_barf() is b0rken in quite a few ways.  There's an
>easily triggerable memory corruptor that can be fixed locally as well
>as something else that needs a change of e.g. ->mkdir() calling
>conventions to take care of.  The change is mechanical and fairly
>simple, but it's already -rc4.

I'm not advocating to forcefully block people from submitting patches
after -rc4 (that was Ted's suggesting).

I'm just saying that as a maintainer, you should use your brain and
figure out how critical the bug is, how good is the fix and how well was
it tested, and decide if you want to merge it in or not.

If it fixed the bug and didn't introduce a regression, great! If it
messed something else, you'd have some input on how to address it better
in the future.

I'm trying to come up with a tool/system to help maintainers with
this task because right now it's not working too well. I'm not trying to
introduce arbitrary rules to make your life miserable.

>Even though the whole thing is well-understood at that point,
>we *can't* apply all 3 - it's too late in the cycle for the
>calling conventions' change in the middle of the series.
>
>Should we apply the first fix that cycle, or should it slide to the
>next one?  We can't apply #1 + #3 --- #3 won't even compile without
>#2.  We can't apply #3 without #1 --- they can be transposed, but
>it's nowhere near a mechanical transformation.  So WTF should #3
>have in "Fixes:"?
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Al Viro]

On Thu, May 03, 2018 at 05:34:24PM +0000, Sasha Levin wrote:

> >Moreover, what the hell do you suggest in situation when
> >	* foofs_barf() is b0rken in quite a few ways.  There's an
> >easily triggerable memory corruptor that can be fixed locally as well
> >as something else that needs a change of e.g. ->mkdir() calling
> >conventions to take care of.  The change is mechanical and fairly
> >simple, but it's already -rc4.
> 
> I'm not advocating to forcefully block people from submitting patches
> after -rc4 (that was Ted's suggesting).

I am, though - change of a method signature when we have several dozens
of instances does *not* belong in -rc5; if nothing else, it guarantees
a nightmare pile of conflicts with individual filesystem trees.

> I'm just saying that as a maintainer, you should use your brain and
> figure out how critical the bug is, how good is the fix and how well was
> it tested, and decide if you want to merge it in or not.
> 
> If it fixed the bug and didn't introduce a regression, great! If it
> messed something else, you'd have some input on how to address it better
> in the future.
> 
> I'm trying to come up with a tool/system to help maintainers with
> this task because right now it's not working too well. I'm not trying to
> introduce arbitrary rules to make your life miserable.

And I am asking you what kind of rules do you want/expect/would prefer
for Fixes: pseudo-header.  *I* do not give a flying fuck for its
contents; I can put it in, if there is a good reason, though.  And
the obvious consumers of that thing are -stable maintainers.  Including
yourself.  Which is why I am asking you what should go in there in
situation described above.  And no, that's not a rhetorical question;
I really want to know.

Let me describe it again:
	* a bunch of holes is found in a function; all of them go back
several years
	* a clean fix for the whole pile is a composition of
1) local fix of trivially triggered memory corruptor
2) tree-wide mechanical change of method signature + matching modifications
of callers of that method (say, all five of them).
3) further changes in the function in question and its caller (which happens
to be an instance of the method modified by (2).
	* dependencies between parts: (1) is standalone, (3) has a hard
dependency on (2), (1) can be reordered past (2)+(modified 3), but modifications
needed in (1) and (3) are not trivial.
	* the crap fixed by (1) is much more severe than that fixed by (3)
(and (2) is an equivalent transformation which does not affect behaviour of
anything).
	* too late in the cycle for tree-wide patches like (2).

As far as I'm concerned (and if it makes -stable folks' lives unpleasant,
too fucking bad) the merge order is: (1) as soon as it's sufficiently
reviewed and tested, (2) and (3) - next merge window.  The only question
is what kind of metadata should go into commit messages to minimize the
PITA for -stable folks, *given* *that* *merge* *timing*.
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Greg KH]

On Thu, May 03, 2018 at 07:20:39PM +0100, Al Viro wrote:
> On Thu, May 03, 2018 at 05:34:24PM +0000, Sasha Levin wrote:
> 
> > >Moreover, what the hell do you suggest in situation when
> > >	* foofs_barf() is b0rken in quite a few ways.  There's an
> > >easily triggerable memory corruptor that can be fixed locally as well
> > >as something else that needs a change of e.g. ->mkdir() calling
> > >conventions to take care of.  The change is mechanical and fairly
> > >simple, but it's already -rc4.
> > 
> > I'm not advocating to forcefully block people from submitting patches
> > after -rc4 (that was Ted's suggesting).
> 
> I am, though - change of a method signature when we have several dozens
> of instances does *not* belong in -rc5; if nothing else, it guarantees
> a nightmare pile of conflicts with individual filesystem trees.
> 
> > I'm just saying that as a maintainer, you should use your brain and
> > figure out how critical the bug is, how good is the fix and how well was
> > it tested, and decide if you want to merge it in or not.
> > 
> > If it fixed the bug and didn't introduce a regression, great! If it
> > messed something else, you'd have some input on how to address it better
> > in the future.
> > 
> > I'm trying to come up with a tool/system to help maintainers with
> > this task because right now it's not working too well. I'm not trying to
> > introduce arbitrary rules to make your life miserable.
> 
> And I am asking you what kind of rules do you want/expect/would prefer
> for Fixes: pseudo-header.  *I* do not give a flying fuck for its
> contents; I can put it in, if there is a good reason, though.  And
> the obvious consumers of that thing are -stable maintainers.  Including
> yourself.  Which is why I am asking you what should go in there in
> situation described above.  And no, that's not a rhetorical question;
> I really want to know.
> 
> Let me describe it again:
> 	* a bunch of holes is found in a function; all of them go back
> several years
> 	* a clean fix for the whole pile is a composition of
> 1) local fix of trivially triggered memory corruptor
> 2) tree-wide mechanical change of method signature + matching modifications
> of callers of that method (say, all five of them).
> 3) further changes in the function in question and its caller (which happens
> to be an instance of the method modified by (2).
> 	* dependencies between parts: (1) is standalone, (3) has a hard
> dependency on (2), (1) can be reordered past (2)+(modified 3), but modifications
> needed in (1) and (3) are not trivial.
> 	* the crap fixed by (1) is much more severe than that fixed by (3)
> (and (2) is an equivalent transformation which does not affect behaviour of
> anything).
> 	* too late in the cycle for tree-wide patches like (2).
> 
> As far as I'm concerned (and if it makes -stable folks' lives unpleasant,
> too fucking bad)

Don't care about me for stuff like this.  Fix it correctly and I'll
worry about any dependancy issues later :)

greg k-h
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Willy Tarreau]

On Thu, May 03, 2018 at 11:55:29AM -0700, Greg KH wrote:
> Don't care about me for stuff like this.  Fix it correctly and I'll
> worry about any dependancy issues later :)

For me the real value of the Fixes header is to let the person doing
the backport know if they must search when the patch looks irrelevant
at first glance. On old kernels more than half of the patches don't
apply and sometimes you really do not know if the code moved somewhere
else or if it was not there. Fixes gives some clues what to look for
and sometimes about an exact commit to get more details. I've been
quite happy with those mentioning only "3.2+" in it just like those
copy-pasting the commit ID is pretty fine as well. The former making it
easier to skip a useless patch, the latter providing more information.
But in my opinion these must not add burden on the committer. Even some
vague information like "4.4, maybe 4.2" or "oldest one having subsystem
foo" is extremely helpful there.

The dependency chain however matters less because once you start fighting
with a small patch set for 1 hour you can spend an extra minute testing
several combinations or figuring the dependencies in mainline.

Willy

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin via Ksummit-discuss]

On Thu, May 03, 2018 at 09:14:05PM +0200, Willy Tarreau wrote:
>The dependency chain however matters less because once you start fighting
>with a small patch set for 1 hour you can spend an extra minute testing
>several combinations or figuring the dependencies in mainline.

https://git.kernel.org/pub/scm/linux/kernel/git/sashal/stable-tools.git/tree/stable-deps

:)
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin]

On Thu, May 03, 2018 at 07:20:39PM +0100, Al Viro wrote:
>On Thu, May 03, 2018 at 05:34:24PM +0000, Sasha Levin wrote:
>
>> >Moreover, what the hell do you suggest in situation when
>> >	* foofs_barf() is b0rken in quite a few ways.  There's an
>> >easily triggerable memory corruptor that can be fixed locally as well
>> >as something else that needs a change of e.g. ->mkdir() calling
>> >conventions to take care of.  The change is mechanical and fairly
>> >simple, but it's already -rc4.
>>
>> I'm not advocating to forcefully block people from submitting patches
>> after -rc4 (that was Ted's suggesting).
>
>I am, though - change of a method signature when we have several dozens
>of instances does *not* belong in -rc5; if nothing else, it guarantees
>a nightmare pile of conflicts with individual filesystem trees.
>
>> I'm just saying that as a maintainer, you should use your brain and
>> figure out how critical the bug is, how good is the fix and how well was
>> it tested, and decide if you want to merge it in or not.
>>
>> If it fixed the bug and didn't introduce a regression, great! If it
>> messed something else, you'd have some input on how to address it better
>> in the future.
>>
>> I'm trying to come up with a tool/system to help maintainers with
>> this task because right now it's not working too well. I'm not trying to
>> introduce arbitrary rules to make your life miserable.
>
>And I am asking you what kind of rules do you want/expect/would prefer
>for Fixes: pseudo-header.  *I* do not give a flying fuck for its
>contents; I can put it in, if there is a good reason, though.  And
>the obvious consumers of that thing are -stable maintainers.  Including
>yourself.  Which is why I am asking you what should go in there in
>situation described above.  And no, that's not a rhetorical question;
>I really want to know.
>
>Let me describe it again:
>	* a bunch of holes is found in a function; all of them go back
>several years
>	* a clean fix for the whole pile is a composition of
>1) local fix of trivially triggered memory corruptor
>2) tree-wide mechanical change of method signature + matching modifications
>of callers of that method (say, all five of them).
>3) further changes in the function in question and its caller (which happens
>to be an instance of the method modified by (2).
>	* dependencies between parts: (1) is standalone, (3) has a hard
>dependency on (2), (1) can be reordered past (2)+(modified 3), but modifications
>needed in (1) and (3) are not trivial.
>	* the crap fixed by (1) is much more severe than that fixed by (3)
>(and (2) is an equivalent transformation which does not affect behaviour of
>anything).
>	* too late in the cycle for tree-wide patches like (2).
>
>As far as I'm concerned (and if it makes -stable folks' lives unpleasant,
>too fucking bad) the merge order is: (1) as soon as it's sufficiently
>reviewed and tested, (2) and (3) - next merge window.  The only question
>is what kind of metadata should go into commit messages to minimize the
>PITA for -stable folks, *given* *that* *merge* *timing*.

Right, -stable shouldn't affect how/when you merge your patches in.

Also, in scenarios such as this we have enough tools in place to figure
out the dependency chain, and we could address it when backporting to
stable.

I don't want to create additional work that you wouldn't do anyways.
Keep in mind that the Fixes: tag will also be useful for yourself later
on if you have to go back to this patch a few years later.

As to the question you actually asked, assuming patch (3) is important 
for stable as well (you didn't state it explicitly), the correct 
tagging would be:

For patch (1):
	Fixes: AAA ("Commit from several years ago that introduced the hole")
	Cc: stable@vger.kernel.org

For patch (2):
	No tags

For patch (3):
	Fixes: AAA ("Commit from several years ago that introduced the hole")
	Cc: stable@vger.kernel.org # commit-id-of-(2)

Re: [Ksummit-discuss] bug-introducing patches

[David Howells]

Sasha Levin via Ksummit-discuss wrote:

> 	Cc: stable@vger.kernel.org # commit-id-of-(2)

Can you please not do this?  This screws up email address parsing in some
tools.

David
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Jani Nikula]

On Fri, 04 May 2018, David Howells <dhowells@redhat.com> wrote:
> Sasha Levin via Ksummit-discuss wrote:
>
>> 	Cc: stable@vger.kernel.org # commit-id-of-(2)
>
> Can you please not do this?  This screws up email address parsing in some
> tools.

This has been documented since

commit 8e9b9362266dd16255473c080d846b13e27247bf
Author: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Date:   Sun Dec 6 12:24:31 2009 +0100

    Doc/stable rules: add new cherry-pick logic

in v2.6.33 so seems like there should have been enough time to fix the
tools.


BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Theodore Y. Ts'o]

On Fri, May 04, 2018 at 03:31:17PM +0300, Jani Nikula wrote:
> On Fri, 04 May 2018, David Howells <dhowells@redhat.com> wrote:
> > Sasha Levin via Ksummit-discuss wrote:
> >
> >> 	Cc: stable@vger.kernel.org # commit-id-of-(2)
>
> This has been documented since
> 
> commit 8e9b9362266dd16255473c080d846b13e27247bf
> Author: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
> Date:   Sun Dec 6 12:24:31 2009 +0100
> 
>     Doc/stable rules: add new cherry-pick logic
> 
> in v2.6.33 so seems like there should have been enough time to fix the
> tools.

The problem is that it's not being *used* that way.  In fact, that
documentation is arguably out of date.  When it does get used, it's
used to indicate which kernels the stable patch applies.  You have to
go pretty far back before you find that suggested usage.  Run:

git log --grep stable@kernel.org | grep -i cc: | grep stable | grep \#

and see for yourself.  The first couple of hits:

    Cc: stable@kernel.org # 3.11
    Cc: stable@kernel.org # 4.8+
    Cc: stable@kernel.org # 4.8+
    Cc: stable@kernel.org # 4.13+
    Cc: stable@kernel.org # 4.8+
    Cc: stable@kernel.org   # 4.13 - together with 890da9cf0983
    Cc: stable@kernel.org # 4.13
    Cc: stable@kernel.org # 4.13
    Cc: stable@kernel.org   # v4.8+
    Cc: stable@kernel.org # v4.10+
    Cc: stable@kernel.org # v4.10+
    Cc: stable@kernel.org # v4.10+
    Cc: stable@kernel.org  # reverted commits were marked for stable
    Cc: stable@kernel.org   # for the backport of the original commit
    Cc: stable@kernel.org # v4.8+

At this point, my suggestion would be to delete the text added by the
above-mentioned commit, and add a new syntax.  We're much more willing
to support multiple headers, so something like this:

Stable-prereq: DEADBEEF1234: subsystem: bork bork bork....

With multiple Stable-preeq: lines allowed, where the order is
significant, might be one way to do things.

	     	      	      	    - Ted

Re: [Ksummit-discuss] bug-introducing patches

[Greg KH]

On Fri, May 04, 2018 at 09:09:32AM -0400, Theodore Y. Ts'o wrote:
> On Fri, May 04, 2018 at 03:31:17PM +0300, Jani Nikula wrote:
> > On Fri, 04 May 2018, David Howells <dhowells@redhat.com> wrote:
> > > Sasha Levin via Ksummit-discuss wrote:
> > >
> > >> 	Cc: stable@vger.kernel.org # commit-id-of-(2)
> >
> > This has been documented since
> > 
> > commit 8e9b9362266dd16255473c080d846b13e27247bf
> > Author: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
> > Date:   Sun Dec 6 12:24:31 2009 +0100
> > 
> >     Doc/stable rules: add new cherry-pick logic
> > 
> > in v2.6.33 so seems like there should have been enough time to fix the
> > tools.
> 
> The problem is that it's not being *used* that way.  In fact, that
> documentation is arguably out of date.  When it does get used, it's
> used to indicate which kernels the stable patch applies.  You have to
> go pretty far back before you find that suggested usage.  Run:
> 
> git log --grep stable@kernel.org | grep -i cc: | grep stable | grep \#
> 
> and see for yourself.  The first couple of hits:
> 
>     Cc: stable@kernel.org # 3.11
>     Cc: stable@kernel.org # 4.8+
>     Cc: stable@kernel.org # 4.8+
>     Cc: stable@kernel.org # 4.13+
>     Cc: stable@kernel.org # 4.8+
>     Cc: stable@kernel.org   # 4.13 - together with 890da9cf0983
>     Cc: stable@kernel.org # 4.13
>     Cc: stable@kernel.org # 4.13
>     Cc: stable@kernel.org   # v4.8+
>     Cc: stable@kernel.org # v4.10+
>     Cc: stable@kernel.org # v4.10+
>     Cc: stable@kernel.org # v4.10+
>     Cc: stable@kernel.org  # reverted commits were marked for stable
>     Cc: stable@kernel.org   # for the backport of the original commit
>     Cc: stable@kernel.org # v4.8+
> 
> At this point, my suggestion would be to delete the text added by the
> above-mentioned commit, and add a new syntax.  We're much more willing
> to support multiple headers, so something like this:
> 
> Stable-prereq: DEADBEEF1234: subsystem: bork bork bork....
> 
> With multiple Stable-preeq: lines allowed, where the order is
> significant, might be one way to do things.

Ugh, what?  I don't understand what you are proposing here, what we have
today is just fine, what is broken with it?

thanks,

greg k-h
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Theodore Y. Ts'o]

On Fri, May 04, 2018 at 10:40:55AM -0700, Greg KH wrote:
> Ugh, what?  I don't understand what you are proposing here, what we have
> today is just fine, what is broken with it?

What we have today is this:

     Cc: stable@kernel.org # 3.11
     Cc: stable@kernel.org # 4.8+
     Cc: stable@kernel.org # 4.8+

Jani was suggesting something documented which doesn't match current
practice.  See commit 8e9b9362266d, which describes something like this:

     Cc: <stable@kernel.org> # .32.x: a1f84a3: sched: Check for idle
     Cc: <stable@kernel.org> # .32.x: 1b9508f: sched: Rate-limit newidle
     Cc: <stable@kernel.org> # .32.x: fd21073: sched: Fix affinity logic

... to specify prereqisite commits needed to backport the commit in
question.  I am proposing that we delete what is in
stable_kernel_rules.rst, because it doesn't match with current
practice.

If it is necessary to explicitly specify prerequisites (as opposed to
having scripts or stable maintainers guess or figure things out
manually), then something like this might be better:

Stable-prereq: DEADBEEF1234: subsystem: bork bork bork....

If it's not necessary, fine.  But we should still delete what is
currently documented in stable_kernel_rules and was introduced in
8e9b9362266d, because it doesn't describe current practice.

						- Ted

Re: [Ksummit-discuss] bug-introducing patches

[James Bottomley]

On Fri, 2018-05-04 at 17:13 -0400, Theodore Y. Ts'o wrote:
> If it's not necessary, fine.  But we should still delete what is
> currently documented in stable_kernel_rules and was introduced in
> 8e9b9362266d, because it doesn't describe current practice.

It definitely doesn't seem to describe current practice.  It looks like
it got applied because the commit description bears a somewhat strange
relation to the actual text that was added:  The commit talks about the
 original script that used to forward to stable (although it got me and
hpa confused) which seems to refer to a tiny deletion and the rest is
adding an Ingo one off proposal for dependencies.

For the record: Greg runs his own script now and I'm not involved.

Current process (at least from the SCSI centric view) is that if we
screw up and forward a commit with missing dependencies to stable via a
cc: tag, it won't apply and Greg tells us to fix it, which we do.  That
seems to be an adequately functional process for the odd times we run
into this.

James

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin]

On Fri, May 04, 2018 at 02:38:01PM -0700, James Bottomley wrote:
>On Fri, 2018-05-04 at 17:13 -0400, Theodore Y. Ts'o wrote:
>> If it's not necessary, fine.  But we should still delete what is
>> currently documented in stable_kernel_rules and was introduced in
>> 8e9b9362266d, because it doesn't describe current practice.
>
>It definitely doesn't seem to describe current practice.  It looks like
>it got applied because the commit description bears a somewhat strange
>relation to the actual text that was added:  The commit talks about the
> original script that used to forward to stable (although it got me and
>hpa confused) which seems to refer to a tiny deletion and the rest is
>adding an Ingo one off proposal for dependencies.

The usage for something like this is only if a commit that we didn't
previously think would go to stable now has to because it's a dependency
of a new -stable commit, so the expected usage will be pretty small
anyways.

I don't have an objection to moving this to it's own tag. It will make
my scripts somewhat simpler for sure.

>For the record: Greg runs his own script now and I'm not involved.
>
>Current process (at least from the SCSI centric view) is that if we
>screw up and forward a commit with missing dependencies to stable via a
>cc: tag, it won't apply and Greg tells us to fix it, which we do.  That
>seems to be an adequately functional process for the odd times we run
>into this.

Assuming a commit won't apply/fail to build because of dependencies is
really not a safe approach, which I keep getting reminded of quite
often.

See for example this patch:

	https://patchwork.kernel.org/patch/10243707/

It will apply and build, but will fail to boot on a particular flavor 
of ARMv7, and this is just the obvious failure modes of approaches like
these.

So again, I don't have a an objection to changing the docs or the way
it's being done now, but the new way should make it very easy for folks
to list dependency chains if they want to.

Re: [Ksummit-discuss] bug-introducing patches

[Theodore Y. Ts'o]

On Fri, May 04, 2018 at 09:51:14PM +0000, Sasha Levin wrote:
> I don't have an objection to moving this to it's own tag. It will make
> my scripts somewhat simpler for sure.

It's not a matter "moving this it's own tag", but creating a new tag
--- because what is in the docs is a lie.  It does not describe what
we do today.  And current practice is the reality, not what is in the
docs.

As to whether we should create a new tag to support explicit
dependencies, I'll leave that between you and Greg K-H and the rest of
the stable maintainers.  :-)

						- Ted
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Willy Tarreau]

On Fri, May 04, 2018 at 07:35:42PM -0400, Theodore Y. Ts'o wrote:
> On Fri, May 04, 2018 at 09:51:14PM +0000, Sasha Levin wrote:
> > I don't have an objection to moving this to it's own tag. It will make
> > my scripts somewhat simpler for sure.
> 
> It's not a matter "moving this it's own tag", but creating a new tag
> --- because what is in the docs is a lie.  It does not describe what
> we do today.  And current practice is the reality, not what is in the
> docs.
> 
> As to whether we should create a new tag to support explicit
> dependencies, I'll leave that between you and Greg K-H and the rest of
> the stable maintainers.  :-)

Guys, *personally*, I've sometimes been a bit annoyed by the huge amount
of irregular extra headers trying to compensate for horribly vague commit
messages, and I'm pretty sure it pisses off patch authors who don't know
anymore what to put in their description. We need to keep in mind that
authors are humans and not machines, and that natural language remains
the best to explain complex dependencies. I'd prefer to see :

    This patch needs to be backported to all stable branches that contain
    717d3133 and 207f5b3c (that's 3.10+) or their respective backports but
    must be adapted (contact me) if only a backport of 717d3133 is present.

    Cc: stable # v3.10+

Rather than horrible stuff like this :

    Cc: stable # v3.10+ (717d3133 && 207f5b3c) || WARN_ON(back(717d3133))

Of course it's a bit made up, but not too far from what is being discussed
here, probably only the next step. People will often get complex rules
wrong, both on the producer and on the consumer side. The day we need a
compiler to emit commit messages, we'll have to wonder if we didn't go
too far.

Also I've found the Fixes header pretty useful. It allows patch authors
to mention what is being fixed without necessarily copying stable,
because sometimes you'd rather not see your patch immediately backported
or you think the risks are higher than the bug. And here as well, it's
only suited for simple situations with a single commit ID, complex
desriptions have to be part of the commit message body.

I think that what we have now works pretty well but that some descriptions
lack a bit of detail, especially on the impact of the bug which would help
decide to backport or drop. This is understandable because often the person
fixing a bug documents it for people knowing the same subsystem well. But
when you backport fixes into other kernel versions, you don't know well
how each subsystem works, and guessing the impact of a bug is not always
obvious. Most of the time, authors who add Fixes: and/or Cc: stable take
care of providing enough information, though I'd suspect that sometimes
they're making efforts trying to figure how to place the information
there and possibly try to avoid redundancy by writing a shorter body.

At this point, I'm really not seeing what we're trying to improve or
optimize, and to be honest this discussion worries me a bit, by just
thinking that it could result in annoying changes...

Willy

Re: [Ksummit-discuss] bug-introducing patches

[Eric W. Biederman]

Willy Tarreau <w@1wt.eu> writes:

> On Fri, May 04, 2018 at 07:35:42PM -0400, Theodore Y. Ts'o wrote:
>> On Fri, May 04, 2018 at 09:51:14PM +0000, Sasha Levin wrote:
>> > I don't have an objection to moving this to it's own tag. It will make
>> > my scripts somewhat simpler for sure.
>> 
>> It's not a matter "moving this it's own tag", but creating a new tag
>> --- because what is in the docs is a lie.  It does not describe what
>> we do today.  And current practice is the reality, not what is in the
>> docs.
>> 
>> As to whether we should create a new tag to support explicit
>> dependencies, I'll leave that between you and Greg K-H and the rest of
>> the stable maintainers.  :-)
>
> Guys, *personally*, I've sometimes been a bit annoyed by the huge amount
> of irregular extra headers trying to compensate for horribly vague commit
> messages, and I'm pretty sure it pisses off patch authors who don't know
> anymore what to put in their description. We need to keep in mind that
> authors are humans and not machines, and that natural language remains
> the best to explain complex dependencies. I'd prefer to see :
>
>     This patch needs to be backported to all stable branches that contain
>     717d3133 and 207f5b3c (that's 3.10+) or their respective backports but
>     must be adapted (contact me) if only a backport of 717d3133 is present.
>
>     Cc: stable # v3.10+
>
> Rather than horrible stuff like this :
>
>     Cc: stable # v3.10+ (717d3133 && 207f5b3c) || WARN_ON(back(717d3133))
>
> Of course it's a bit made up, but not too far from what is being discussed
> here, probably only the next step. People will often get complex rules
> wrong, both on the producer and on the consumer side. The day we need a
> compiler to emit commit messages, we'll have to wonder if we didn't go
> too far.
>
> Also I've found the Fixes header pretty useful. It allows patch authors
> to mention what is being fixed without necessarily copying stable,
> because sometimes you'd rather not see your patch immediately backported
> or you think the risks are higher than the bug. And here as well, it's
> only suited for simple situations with a single commit ID, complex
> desriptions have to be part of the commit message body.
>
> I think that what we have now works pretty well but that some descriptions
> lack a bit of detail, especially on the impact of the bug which would help
> decide to backport or drop. This is understandable because often the person
> fixing a bug documents it for people knowing the same subsystem well. But
> when you backport fixes into other kernel versions, you don't know well
> how each subsystem works, and guessing the impact of a bug is not always
> obvious. Most of the time, authors who add Fixes: and/or Cc: stable take
> care of providing enough information, though I'd suspect that sometimes
> they're making efforts trying to figure how to place the information
> there and possibly try to avoid redundancy by writing a shorter body.
>
> At this point, I'm really not seeing what we're trying to improve or
> optimize, and to be honest this discussion worries me a bit, by just
> thinking that it could result in annoying changes...

So the way I use headers today is:
Cc: stable@vger.kernel.org
Fixes: sha1hash "commit subject"

I will use "Fixes: v2.0.1" if something is so old that it isn't in git.
If it was in bitkeeper and now in tglx's tree I will use:
History Tree: https://git.kernel.org/pub/scm/linux/kernel/git/tglx/history.git

Just because you won't find the commit in Linus's git tree.

I tend not to find particularly serious bugs, just ancient ones so I
generally figure if it doesn't backport easily it probably is not a
candidate for stable.  The bug has existed for ages without anyone
really carring anyway.

Eric
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Greg KH]

On Sat, May 05, 2018 at 12:02:47AM -0500, Eric W. Biederman wrote:
> So the way I use headers today is:
> Cc: stable@vger.kernel.org
> Fixes: sha1hash "commit subject"

And that makes my life _so_ much easier.  The Fixes: tag is great
(thanks James!), I have scripts that I use to track if a fix was applied
to a stable tree to know if it needs to go into that branch as well.
Without that, the "# 4.9" marking just doesn't work, as it doesn't tell
me if that commit got backported to 4.4.y as well.  I used to do that
type of detection by hand, but automating it is so much better and I
miss less patches that way.

Anyway, just my two cents, let's try to keep this simple for both
maintainers, and stable developers, if at all possible.  I think what we
have now works well, but if people think the documentation should be
cleaned up, great, send patches :)

thanks,

greg k-h
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Sasha Levin via Ksummit-discuss]

On Fri, May 04, 2018 at 07:35:42PM -0400, Theodore Y. Ts'o wrote:
>On Fri, May 04, 2018 at 09:51:14PM +0000, Sasha Levin wrote:
>> I don't have an objection to moving this to it's own tag. It will make
>> my scripts somewhat simpler for sure.
>
>It's not a matter "moving this it's own tag", but creating a new tag
>--- because what is in the docs is a lie.  It does not describe what
>we do today.  And current practice is the reality, not what is in the
>docs.

I'm really confused here. What do you mean with "not describe what we do
today"?

The doc allows for three ways to tag a patch:

1. Empty tag: "Cc: stable@vger.kernel.org"
2. With a version, quoting from the doc:

	Also, some patches may have kernel version prerequisites.  This can be
	specified in the following format in the sign-off area:

	     Cc: <stable@vger.kernel.org> # 3.3.x

	The tag has the meaning of:

	     git cherry-pick <this commit>

	For each "-stable" tree starting with the specified version.

3. With a prereq commit, which is in the form of:

	Cc: <stable@vger.kernel.org> # 3.3.x: a1f84a3: sched: Check for idle

We expect this to be used rarely used, and indeed it's not used as much.

>As to whether we should create a new tag to support explicit
>dependencies, I'll leave that between you and Greg K-H and the rest of
>the stable maintainers.  :-)
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Al Viro]

On Wed, May 02, 2018 at 10:41:56PM +0200, Geert Uytterhoeven wrote:

> Between v4.17-rc1 and v4.17-rc3, there are 660 non-merge commits, of which
>   - 245 carry a Fixes tag,
>   - 196 carry a CC stable,
>   - 395 contain the string "fix".
> (non-mutually exclusive)

BTW, what about situations when we have a fix naturally split into
a series of 2-3 massaging equivalent transformations + one-liner fix
in the resulting tree?
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] bug-introducing patches

[Geert Uytterhoeven]

Hi Sasha,

On Tue, May 1, 2018 at 6:38 PM, Sasha Levin
<Alexander.Levin@microsoft.com> wrote:
> Working on AUTOSEL, it became even more obvious to me how difficult it is for a
> patch to get a proper review. Maintainers found it difficult to keep up with
> the upstream work for their subsystem, and reviewing additional -stable patches
> put even more load on them which some suggested would be more than what they
> can handle.

Thanks for your work!

>  - For some reason, the odds of a -rc commit to be targetted for -stable is
>    over 20%, while for merge window commits it's about 3%. I can't quite
> explain why that happens, but this would suggest that -rc commits end up
> hurting -stable pretty badly.

Aren't more -rc commits targeted for -stable because they are bugfixes?
Ideally, new features are supposed to be merged during the merge window,
while -rc commits fix bugs.

So they can be categorized like:
  1. Plain -rc commits,
  2. -rc commits fixing a bug:
       a. in the same release cycle,
       b. in a previous release.

2a assumes the bug was backported to -stable, too, doesn't it?

Do you have statistics for which categories are most buggy?

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss