[Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Sasha Levin]

Hi folks,

I'd like to propose a topic discussing issues that are caused as a result of the
way development happens upstream, and the way we integrate with distros that affect
the quality of stable trees:

 1. During the RC cycles bug fixes tend to get sent to Linus without going through
linux-next. This is very risky, but it seems to work(?). The problem is that Linus
doesn't restrict those fixes to bugs that were introduced in the current merge window
but takes anything that is labelled as a "fix".

The result is that there is a significant amount of mostly untested RC patches
trickling down into stable trees, causing breakage for folks who assume that they
are running a tested kernel but end up with commits that haven't even been in
linux-next for more than a few days.

Since for RC kernel it's expected to see issues, and it's easy to correct, this is
less than a problem, but consider this flow for stable:

 * 4.0: bug "A" introduced.
 * 4.2-rc1: bug "A" fixed, but fix unknowingly introduced bug "B".
 * 4.1.1: ships with fix for "A", and new bug "B".
 * Stable user machines suffer from breakage.
 * 4.2-rc7: bug "B" fixed.
 * Stable users still suffer until the next kernel release.

So while it was quickly fixed for RC, this seriously affects stable.

I actually just had this happen with "nfs: take extra reference to fl->fl_file when
running a LOCKU operation" on the day I was writing this mail.

 2. The review cycle: I've *never* ended up receiving comments during review cycle
of a stable release. I've received comments either when I've sent my "added to the..."
mails when I've added a patch in, which usually came from the authors of the patch
or the maintainers of the subsystem, and I've received comments after the tree has
shipped - when it actually broke something.

We need to explore ways to integrate the review process better with the end users,
possibly by extending it to allow distributions to ship "proposed" review kernels
rather than waiting for us to finalize a stable kernel before they start working
on shipping it.

 3. Cross tree verification and auditing: There seems to be a fair amount of LTS
kernels that are maintained openly on the stable@ ML, and even a bigger amount
if the Canonical folks decide to play ball at any stage. While ideally each tree
should contain (if required, correctly backported) patches that are relevant only
to that given tree, we have no standard way to verify that.

We need a mechanism that would let us audit the existence (and non-existence) of
patches in an easy way, and to compare backports between stable trees to help verify
their correctness.

 4. Upstream monitoring: I've suggested to Greg that we have a bot looking at
commits going upstream, and for every commit marked as for stable it would attempt
to apply it to all relevant stable trees and build them, and on failure would
notify the author.

Greg objected for two main reasons: the first is that we should put more effort
into trying to fix any possible issues which arise from failure to build and
backport before we send mails out, which I accepted. The other issue was that he
doesn't want to generate too much noise, and if the patch doesn't look important
enough and only applies to the latest stable it's enough, and no need to bother
people to backport it.

So this is mostly an open discussion: what do people expect to do (if anything)
as a result of marking a patch for stable? What do people think about the increased
noise? Is there a better way to do it rather than by mails?


Thanks,
Sasha

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Geert Uytterhoeven]

On Sat, Jul 11, 2015 at 6:12 PM, Sasha Levin <sasha.levin@oracle.com> wrote:
>  1. During the RC cycles bug fixes tend to get sent to Linus without going through
> linux-next. This is very risky, but it seems to work(?). The problem is that Linus
> doesn't restrict those fixes to bugs that were introduced in the current merge window
> but takes anything that is labelled as a "fix".
>
> The result is that there is a significant amount of mostly untested RC patches
> trickling down into stable trees, causing breakage for folks who assume that they
> are running a tested kernel but end up with commits that haven't even been in
> linux-next for more than a few days.
>
> Since for RC kernel it's expected to see issues, and it's easy to correct, this is
> less than a problem, but consider this flow for stable:
>
>  * 4.0: bug "A" introduced.
>  * 4.2-rc1: bug "A" fixed, but fix unknowingly introduced bug "B".

And the fix for "A" was not in -next?
It will be the next day, though.

>  * 4.1.1: ships with fix for "A", and new bug "B".

For serious bugs, the fix may indeed be applied Really Quick.

>  * Stable user machines suffer from breakage.
>  * 4.2-rc7: bug "B" fixed.
>  * Stable users still suffer until the next kernel release.
>
> So while it was quickly fixed for RC, this seriously affects stable.

I wouldn't say "quickly": there's quite some time period (6 weeks) between
-rc1 and -rc7.

If the fix for "B" has a proper "Fixes:" tag, I guess it will be applied to
the next -stable soon. Hence make sure to always provide a "Fixes:" tag when
fixing bugs.

So it boils down to: "How soon to apply fixes to -stable?", and the trade-off
between applying fixes early, but risking to break something unknown and new,
vs. applying fixes late (after more validation), causing more breakage from a
known issue.

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Sasha Levin]

On 07/12/2015 06:02 AM, Geert Uytterhoeven wrote:
> On Sat, Jul 11, 2015 at 6:12 PM, Sasha Levin <sasha.levin@oracle.com> wrote:
>>  1. During the RC cycles bug fixes tend to get sent to Linus without going through
>> linux-next. This is very risky, but it seems to work(?). The problem is that Linus
>> doesn't restrict those fixes to bugs that were introduced in the current merge window
>> but takes anything that is labelled as a "fix".
>>
>> The result is that there is a significant amount of mostly untested RC patches
>> trickling down into stable trees, causing breakage for folks who assume that they
>> are running a tested kernel but end up with commits that haven't even been in
>> linux-next for more than a few days.
>>
>> Since for RC kernel it's expected to see issues, and it's easy to correct, this is
>> less than a problem, but consider this flow for stable:
>>
>>  * 4.0: bug "A" introduced.
>>  * 4.2-rc1: bug "A" fixed, but fix unknowingly introduced bug "B".
> 
> And the fix for "A" was not in -next?
> It will be the next day, though.

It will, but there's a huge difference between being in -next for a day or a month before
getting merged in.

Usually the amount of time it spends in -next equals to the amount of time it takes to
trickle from the author through the maintainer chain all the way to Linus.

>>  * 4.1.1: ships with fix for "A", and new bug "B".
> 
> For serious bugs, the fix may indeed be applied Really Quick.

Right.

>>  * Stable user machines suffer from breakage.
>>  * 4.2-rc7: bug "B" fixed.
>>  * Stable users still suffer until the next kernel release.
>>
>> So while it was quickly fixed for RC, this seriously affects stable.
> 
> I wouldn't say "quickly": there's quite some time period (6 weeks) between
> -rc1 and -rc7.

So it would be the same scenario if I wrote there "-rc1..-rc2". LTS kernel users
tend to upgrade their kernel less often than those 6 weeks anyhow.

> If the fix for "B" has a proper "Fixes:" tag, I guess it will be applied to
> the next -stable soon. Hence make sure to always provide a "Fixes:" tag when
> fixing bugs.

True. I'm not saying it wouldn't be fixed in the next stable release, I'm saying
that we're sending untested patches all the way to users who expect the most stable
kernel.

> So it boils down to: "How soon to apply fixes to -stable?", and the trade-off
> between applying fixes early, but risking to break something unknown and new,
> vs. applying fixes late (after more validation), causing more breakage from a
> known issue.

That's just one solution, but there are a few more (which is why it's worth discussing
it :) ).

Consider also:

 - Aligning the stable release process with the kernel where we'd do a few release
candidates for the stable kernel before releasing it.

 - Tightening what is allowed to go in as -rc patches, requiring some time in -next
before it even gets into Linus's hands. Even for "serious" things (does it matter if
a fix for a privesc gets in -rc2 or -rc6, beyond that it would be pulled to stable
earlier?)

 - Differentiate the type of patches going into "regular" -stable, and LTS?


Thanks,
Sasha

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[NeilBrown]

I've been bitten by this a couple of times too.  At least two fairly
serious md bugs *never* got into a release from Linus, but did get into
-stable and at least one into a vendor kernel.


On Sun, 12 Jul 2015 09:32:11 -0400 Sasha Levin <sasha.levin@oracle.com>
wrote:


> > So it boils down to: "How soon to apply fixes to -stable?", and the trade-off
> > between applying fixes early, but risking to break something unknown and new,
> > vs. applying fixes late (after more validation), causing more breakage from a
> > known issue.
> 
> That's just one solution, but there are a few more (which is why it's worth discussing
> it :) ).
> 
> Consider also:
> 
>  - Aligning the stable release process with the kernel where we'd do a few release
> candidates for the stable kernel before releasing it.
> 
>  - Tightening what is allowed to go in as -rc patches, requiring some time in -next
> before it even gets into Linus's hands. Even for "serious" things (does it matter if
> a fix for a privesc gets in -rc2 or -rc6, beyond that it would be pulled to stable
> earlier?)
> 
>  - Differentiate the type of patches going into "regular" -stable, and LTS?
> 

My proposal would be to change the default timing.
Currently patches tagged for 'stable' go into the next -stable release
after they get into Linus's tree.  You can ask for an exception
(sooner, later, different patch) and Greg (or any other stable
maintainer) tries to be accommodating.  But you have to remember to ask.

I would rather that the default was that patches don't go into -stable
until they have
  - been in a full release from Linus and
  - been in a Linus's tree for at least 2 weeks.
    (or 1 week times the age of the target in releases.
     So a fix in 4.4 get to 4.3-stable after a week, 4.2-stable
     after 2 weeks etc .... maybe I'm going over-board here).

Many fixes are important but simply aren't that urgent so the two or
more weeks is no great cost.

If a developer/maintainer thinks a fix is urgent, then they need to
explicitly ask for a quick release, and that could be as easy as saying:

  Cc: stable@vger.kernel.org (URGENT v3.0 and later)

An 'URGENT' fix *should* come with an independent
"Reviewed-by:"  (well ... everything should of course, but if URGENT
stable patches with no Reviewed-by got some push-back, I think that
would be a "very good thing" all around).

I don't think that tightening the criteria for going into any
particular tree will really help.  I'm not sure there is even real
agreement on what is or is not allowed in -stable (we have clearly
written rules, but the practice is really whatever a maintainer
chooses).
-rc prereleases for -stable would only help if people tested them.
Given that the same bugs are in -linus, the testing done there should
be sufficient providing it is given enough time.

Thanks,
NeilBrown

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Andy Lutomirski]

On Sun, Jul 12, 2015 at 5:52 PM, NeilBrown <neilb@suse.com> wrote:
>
> I've been bitten by this a couple of times too.  At least two fairly
> serious md bugs *never* got into a release from Linus, but did get into
> -stable and at least one into a vendor kernel.
>

I've had -stable regression issues, too (both regressions I caused and
regressions I've been bitten by).

>
> I would rather that the default was that patches don't go into -stable
> until they have
>   - been in a full release from Linus and
>   - been in a Linus's tree for at least 2 weeks.
>     (or 1 week times the age of the target in releases.
>      So a fix in 4.4 get to 4.3-stable after a week, 4.2-stable
>      after 2 weeks etc .... maybe I'm going over-board here).
>
> Many fixes are important but simply aren't that urgent so the two or
> more weeks is no great cost.

I tend to agree.  It makes me quite nervous when brand-new
non-critical fixes get backported before they've had a nice soak in
some -rc releases.

--Andy

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Sasha Levin]

On 07/12/2015 08:52 PM, NeilBrown wrote:
> My proposal would be to change the default timing.
> Currently patches tagged for 'stable' go into the next -stable release
> after they get into Linus's tree.  You can ask for an exception
> (sooner, later, different patch) and Greg (or any other stable
> maintainer) tries to be accommodating.  But you have to remember to ask.
> 
> I would rather that the default was that patches don't go into -stable
> until they have
>   - been in a full release from Linus and
>   - been in a Linus's tree for at least 2 weeks.
>     (or 1 week times the age of the target in releases.
>      So a fix in 4.4 get to 4.3-stable after a week, 4.2-stable
>      after 2 weeks etc .... maybe I'm going over-board here).
> 
> Many fixes are important but simply aren't that urgent so the two or
> more weeks is no great cost.

I'd actually argue that Linus shouldn't be pulling *anything* that wasn't in
-next for 2+ weeks. There's no good excuse to want something pulled immediately
as the only benefit Linus's tree provides in that aspect is the wider testing
it receives, so it would make sense to weed out more bugs in -next before they
get to Linus.

I think that this is a small mind-shift from thinking about Linus's tree as
an integration tree to considering it as mostly bug-free code, and stop
merging in risky patches. We already have -next for that.

> If a developer/maintainer thinks a fix is urgent, then they need to
> explicitly ask for a quick release, and that could be as easy as saying:
> 
>   Cc: stable@vger.kernel.org (URGENT v3.0 and later)
> 
> An 'URGENT' fix *should* come with an independent
> "Reviewed-by:"  (well ... everything should of course, but if URGENT
> stable patches with no Reviewed-by got some push-back, I think that
> would be a "very good thing" all around).

I suppose that this is something Linus/maintainers would need to enforce? No
patch unless it lived in -next unless it was acked by the maintainer of that
subsystem?

> I don't think that tightening the criteria for going into any
> particular tree will really help.  I'm not sure there is even real
> agreement on what is or is not allowed in -stable (we have clearly
> written rules, but the practice is really whatever a maintainer
> chooses).
> -rc prereleases for -stable would only help if people tested them.
> Given that the same bugs are in -linus, the testing done there should
> be sufficient providing it is given enough time.

My reasoning for -rc prereleases for stable was to test out the backports that
go into stable kernels: they don't see the light of day until they're shipped
out to folks who want *stable* kernels, but end up being the first testers of
a backport.

I don't want to suggest that we do a few of those per stable kernel, but even
one -rc release that would end up in distros (marked as "proposed/devel") and
would let users test that would be a great step forward.

The reason I've suggested it for Ksummit rather than hashing it out on stable@
is that I believe that the solution is more complex and would need more discussion
than just slapping a "cooldown period" on patches. We need to make sure less
bugs/untested code ends up in Linus's tree to begin with, and we need a way to
validate proposed stable releases before releasing them, since -stable users
aren't interested in being testers.


Thanks,
Sasha

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[NeilBrown]

On Mon, 13 Jul 2015 00:27:52 -0400 Sasha Levin <sasha.levin@oracle.com>
wrote:

> On 07/12/2015 08:52 PM, NeilBrown wrote:
> > My proposal would be to change the default timing.
> > Currently patches tagged for 'stable' go into the next -stable release
> > after they get into Linus's tree.  You can ask for an exception
> > (sooner, later, different patch) and Greg (or any other stable
> > maintainer) tries to be accommodating.  But you have to remember to ask.
> > 
> > I would rather that the default was that patches don't go into -stable
> > until they have
> >   - been in a full release from Linus and
> >   - been in a Linus's tree for at least 2 weeks.
> >     (or 1 week times the age of the target in releases.
> >      So a fix in 4.4 get to 4.3-stable after a week, 4.2-stable
> >      after 2 weeks etc .... maybe I'm going over-board here).
> > 
> > Many fixes are important but simply aren't that urgent so the two or
> > more weeks is no great cost.
> 
> I'd actually argue that Linus shouldn't be pulling *anything* that wasn't in
> -next for 2+ weeks. There's no good excuse to want something pulled immediately
> as the only benefit Linus's tree provides in that aspect is the wider testing
> it receives, so it would make sense to weed out more bugs in -next before they
> get to Linus.

As long as there is a clear 2 week window between a patch being "out
there for wide testing" and being auto-pulled into a -stable, I would
see an improvement.

However I'm not at all convinced that being in -next is really "out
there for wide testing".  Certainly some testing happens in -next, but
my understand is that it is mainly about integration testing, not
burn-in testing.

Isn't the point of the 2-week merge window is that we all stop writing
more bugs and instead start testing to find each others bugs?  You seem
to want to make the previous two weeks fill that role.  I don't think
that would work.



> 
> I think that this is a small mind-shift from thinking about Linus's tree as
> an integration tree to considering it as mostly bug-free code, and stop
> merging in risky patches. We already have -next for that.

I certainly do see Linus's tree as mostly bug-free code.  Certainly I
don't submit something until I'm reasonably confident.  Unfortunately I
am sometimes wrong.  Usually by rc8 it is a lot closer to bug-free.
That steady improvement is the whole point.  So going from rc8 to
stable makes lots more sense than going from rc1 to stable.


> 
> > If a developer/maintainer thinks a fix is urgent, then they need to
> > explicitly ask for a quick release, and that could be as easy as saying:
> > 
> >   Cc: stable@vger.kernel.org (URGENT v3.0 and later)
> > 
> > An 'URGENT' fix *should* come with an independent
> > "Reviewed-by:"  (well ... everything should of course, but if URGENT
> > stable patches with no Reviewed-by got some push-back, I think that
> > would be a "very good thing" all around).
> 
> I suppose that this is something Linus/maintainers would need to enforce? No
> patch unless it lived in -next unless it was acked by the maintainer of that
> subsystem?
> 
> > I don't think that tightening the criteria for going into any
> > particular tree will really help.  I'm not sure there is even real
> > agreement on what is or is not allowed in -stable (we have clearly
> > written rules, but the practice is really whatever a maintainer
> > chooses).
> > -rc prereleases for -stable would only help if people tested them.
> > Given that the same bugs are in -linus, the testing done there should
> > be sufficient providing it is given enough time.
> 
> My reasoning for -rc prereleases for stable was to test out the backports that
> go into stable kernels: they don't see the light of day until they're shipped
> out to folks who want *stable* kernels, but end up being the first testers of
> a backport.

That is certainly a worthy goal - if the testing actually eventuates.

Thanks,
NeilBrown


> 
> I don't want to suggest that we do a few of those per stable kernel, but even
> one -rc release that would end up in distros (marked as "proposed/devel") and
> would let users test that would be a great step forward.
> 
> The reason I've suggested it for Ksummit rather than hashing it out on stable@
> is that I believe that the solution is more complex and would need more discussion
> than just slapping a "cooldown period" on patches. We need to make sure less
> bugs/untested code ends up in Linus's tree to begin with, and we need a way to
> validate proposed stable releases before releasing them, since -stable users
> aren't interested in being testers.
> 
> 
> Thanks,
> Sasha

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Rafael J. Wysocki]

On Monday, July 13, 2015 03:10:12 PM NeilBrown wrote:
> On Mon, 13 Jul 2015 00:27:52 -0400 Sasha Levin <sasha.levin@oracle.com>
> wrote:
> 
> > On 07/12/2015 08:52 PM, NeilBrown wrote:
> > > My proposal would be to change the default timing.
> > > Currently patches tagged for 'stable' go into the next -stable release
> > > after they get into Linus's tree.  You can ask for an exception
> > > (sooner, later, different patch) and Greg (or any other stable
> > > maintainer) tries to be accommodating.  But you have to remember to ask.
> > > 
> > > I would rather that the default was that patches don't go into -stable
> > > until they have
> > >   - been in a full release from Linus and
> > >   - been in a Linus's tree for at least 2 weeks.
> > >     (or 1 week times the age of the target in releases.
> > >      So a fix in 4.4 get to 4.3-stable after a week, 4.2-stable
> > >      after 2 weeks etc .... maybe I'm going over-board here).
> > > 
> > > Many fixes are important but simply aren't that urgent so the two or
> > > more weeks is no great cost.
> > 
> > I'd actually argue that Linus shouldn't be pulling *anything* that wasn't in
> > -next for 2+ weeks. There's no good excuse to want something pulled immediately
> > as the only benefit Linus's tree provides in that aspect is the wider testing
> > it receives, so it would make sense to weed out more bugs in -next before they
> > get to Linus.
> 
> As long as there is a clear 2 week window between a patch being "out
> there for wide testing" and being auto-pulled into a -stable, I would
> see an improvement.
> 
> However I'm not at all convinced that being in -next is really "out
> there for wide testing".

Agreed.  I'm not even sure about -rcs from Linus for that matter.

Unfortunately, -stable often get's wide testing before anything else.

> Certainly some testing happens in -next, but my understand is that it is
> mainly about integration testing, not burn-in testing.
> 
> Isn't the point of the 2-week merge window is that we all stop writing
> more bugs and instead start testing to find each others bugs?  You seem
> to want to make the previous two weeks fill that role.  I don't think
> that would work.

I guess the idea is to avoid exposing -stable users to commits that
mainline users have not been really exposed to yet.

It really is a matter of what is marked for -stable IMO.  Some of those things
should not really carry a "Cc: stable" tag until they've received some wider
testing in the mainline.


> > I think that this is a small mind-shift from thinking about Linus's tree as
> > an integration tree to considering it as mostly bug-free code, and stop
> > merging in risky patches. We already have -next for that.
> 
> I certainly do see Linus's tree as mostly bug-free code.  Certainly I
> don't submit something until I'm reasonably confident.  Unfortunately I
> am sometimes wrong.  Usually by rc8 it is a lot closer to bug-free.
> That steady improvement is the whole point.  So going from rc8 to
> stable makes lots more sense than going from rc1 to stable.

Agreed.

Thanks,
Rafael

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Steven Rostedt]

On Mon, 13 Jul 2015 00:27:52 -0400
Sasha Levin <sasha.levin@oracle.com> wrote:


> > Many fixes are important but simply aren't that urgent so the two or
> > more weeks is no great cost.
> 
> I'd actually argue that Linus shouldn't be pulling *anything* that wasn't in
> -next for 2+ weeks. There's no good excuse to want something pulled immediately
> as the only benefit Linus's tree provides in that aspect is the wider testing
> it receives, so it would make sense to weed out more bugs in -next before they
> get to Linus.

I disagree. I thought next was a place to have integration of new
development, and not just a place to test. Really, how many people test
next compared to Linus's tree? I trip over bugs all the times in
Linus's tree that's been in -next for almost a whole release cycle.

The only bugs that I find that come from -next is integration issues,
where an interface changes and another subsystem stumbles over it.
That's exactly what it was for and what it's good at.

I like the idea that patches marked for stable should sit in Linus's
tree for at least 2 weeks before being pulled into stable. Linus's tree
gets the most testing than any other tree and that's where new fixes
should go.


> 
> I think that this is a small mind-shift from thinking about Linus's tree as
> an integration tree to considering it as mostly bug-free code, and stop
> merging in risky patches. We already have -next for that.

No, we have -next as a way incorporate new code and see how things
interact with other subsystems.

> 
> > If a developer/maintainer thinks a fix is urgent, then they need to
> > explicitly ask for a quick release, and that could be as easy as saying:
> > 
> >   Cc: stable@vger.kernel.org (URGENT v3.0 and later)
> > 
> > An 'URGENT' fix *should* come with an independent
> > "Reviewed-by:"  (well ... everything should of course, but if URGENT
> > stable patches with no Reviewed-by got some push-back, I think that
> > would be a "very good thing" all around).
> 
> I suppose that this is something Linus/maintainers would need to enforce? No
> patch unless it lived in -next unless it was acked by the maintainer of that
> subsystem?
> 
> > I don't think that tightening the criteria for going into any
> > particular tree will really help.  I'm not sure there is even real
> > agreement on what is or is not allowed in -stable (we have clearly
> > written rules, but the practice is really whatever a maintainer
> > chooses).
> > -rc prereleases for -stable would only help if people tested them.
> > Given that the same bugs are in -linus, the testing done there should
> > be sufficient providing it is given enough time.
> 
> My reasoning for -rc prereleases for stable was to test out the backports that
> go into stable kernels: they don't see the light of day until they're shipped
> out to folks who want *stable* kernels, but end up being the first testers of
> a backport.

Which to me looks like rational to let it sit in Linus's tree for a bit.

> 
> I don't want to suggest that we do a few of those per stable kernel, but even
> one -rc release that would end up in distros (marked as "proposed/devel") and
> would let users test that would be a great step forward.
> 
> The reason I've suggested it for Ksummit rather than hashing it out on stable@
> is that I believe that the solution is more complex and would need more discussion
> than just slapping a "cooldown period" on patches. We need to make sure less
> bugs/untested code ends up in Linus's tree to begin with, and we need a way to
> validate proposed stable releases before releasing them, since -stable users
> aren't interested in being testers.

I'm all for discussing this in person.

-- Steve

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 2006 bytes --]

On Mon, Jul 13, 2015 at 02:21:32PM -0400, Steven Rostedt wrote:

> I disagree. I thought next was a place to have integration of new
> development, and not just a place to test. Really, how many people test
> next compared to Linus's tree? I trip over bugs all the times in
> Linus's tree that's been in -next for almost a whole release cycle.

> The only bugs that I find that come from -next is integration issues,
> where an interface changes and another subsystem stumbles over it.
> That's exactly what it was for and what it's good at.

In the embedded space it's much more common to track -next as people are
often working with multiple subsystems so the integration is important.
Most of my code is developed against -next then moved to topic branches
for submission.

We also catch quite a lot of issues in -next as a result of the work on
boot testing that kernelci.org and Olof's bots are doing, hopefully
that'll start to build out to include test suites like kselftest (I know
there's work in progress there but no ETA as of yet).  Things get
exposed to a lot more systems and configurations than individual
maintainers have access to which can shake out issues in code that deals
with hardware.

> > I think that this is a small mind-shift from thinking about Linus's tree as
> > an integration tree to considering it as mostly bug-free code, and stop
> > merging in risky patches. We already have -next for that.

> No, we have -next as a way incorporate new code and see how things
> interact with other subsystems.

I think you're mostly agreeing with each other here (or at least talking
about the definition of "mostly") - it's not an either/or thing.
Exposure in -next does expand the audience and provide a lot of risk
mitigation to what gets sent to Linus' tree as a result of that.  That
catches more than just integration issues between different trees, it
also catches integration issues with a wider range of hardware and
usage.  Not as wide as you see for Linus' tree but wider.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Olof Johansson]

On Mon, Jul 13, 2015 at 2:51 PM, Mark Brown <broonie@kernel.org> wrote:
> On Mon, Jul 13, 2015 at 02:21:32PM -0400, Steven Rostedt wrote:
>
>> I disagree. I thought next was a place to have integration of new
>> development, and not just a place to test. Really, how many people test
>> next compared to Linus's tree? I trip over bugs all the times in
>> Linus's tree that's been in -next for almost a whole release cycle.
>
>> The only bugs that I find that come from -next is integration issues,
>> where an interface changes and another subsystem stumbles over it.
>> That's exactly what it was for and what it's good at.
>
> In the embedded space it's much more common to track -next as people are
> often working with multiple subsystems so the integration is important.
> Most of my code is developed against -next then moved to topic branches
> for submission.
>
> We also catch quite a lot of issues in -next as a result of the work on
> boot testing that kernelci.org and Olof's bots are doing, hopefully
> that'll start to build out to include test suites like kselftest (I know
> there's work in progress there but no ETA as of yet).  Things get
> exposed to a lot more systems and configurations than individual
> maintainers have access to which can shake out issues in code that deals
> with hardware.

I've stopped running -stable releases through the tester. It didn't
fit the way I kept track of what's been built very well and it was
hard to capture a useful state in which to test.

For a while I tried to capture the current-state-of-the-queue from
gregkh's public quilt series ever so often, but it's quite churny.
There's an -rc that's posted for review but not tagged and not
provided as a git branch of applied patches, so it's hard to
automatically test just those.

That would be the ideal setup for me though -- tagged or branched -rc
candidates of stable releases that I'd be happy to put through the
build/boot test at my end.


-Olof

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Guenter Roeck]

On Wed, Jul 15, 2015 at 10:52:13AM -0400, Olof Johansson wrote:
> On Mon, Jul 13, 2015 at 2:51 PM, Mark Brown <broonie@kernel.org> wrote:
> > On Mon, Jul 13, 2015 at 02:21:32PM -0400, Steven Rostedt wrote:
> >
> >> I disagree. I thought next was a place to have integration of new
> >> development, and not just a place to test. Really, how many people test
> >> next compared to Linus's tree? I trip over bugs all the times in
> >> Linus's tree that's been in -next for almost a whole release cycle.
> >
> >> The only bugs that I find that come from -next is integration issues,
> >> where an interface changes and another subsystem stumbles over it.
> >> That's exactly what it was for and what it's good at.
> >
> > In the embedded space it's much more common to track -next as people are
> > often working with multiple subsystems so the integration is important.
> > Most of my code is developed against -next then moved to topic branches
> > for submission.
> >
> > We also catch quite a lot of issues in -next as a result of the work on
> > boot testing that kernelci.org and Olof's bots are doing, hopefully
> > that'll start to build out to include test suites like kselftest (I know
> > there's work in progress there but no ETA as of yet).  Things get
> > exposed to a lot more systems and configurations than individual
> > maintainers have access to which can shake out issues in code that deals
> > with hardware.
> 
> I've stopped running -stable releases through the tester. It didn't
> fit the way I kept track of what's been built very well and it was
> hard to capture a useful state in which to test.
> 
> For a while I tried to capture the current-state-of-the-queue from
> gregkh's public quilt series ever so often, but it's quite churny.
> There's an -rc that's posted for review but not tagged and not
> provided as a git branch of applied patches, so it's hard to
> automatically test just those.
> 
Seems to work quite nicely for me. Sure, there is churn, but it tends
to come in bursts which can be handled easily. The posted -rc matches
the tip of the quilt series, so my auto-builder picks it up automatically.

I use a two-stage approach for my testing: A change in the quilt tree
triggers a script which converts the quilt tree into a git tree. The
changes in the git tree are then picked up by the actual builder(s).
Changes from the quilt tree are only picked up after the quilt tree
has not changed for a couple of hours (to match Greg's workflow).
This takes care of the churn.

Guenter

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Greg KH]

On Wed, Jul 15, 2015 at 10:52:13AM -0400, Olof Johansson wrote:
> On Mon, Jul 13, 2015 at 2:51 PM, Mark Brown <broonie@kernel.org> wrote:
> > On Mon, Jul 13, 2015 at 02:21:32PM -0400, Steven Rostedt wrote:
> >
> >> I disagree. I thought next was a place to have integration of new
> >> development, and not just a place to test. Really, how many people test
> >> next compared to Linus's tree? I trip over bugs all the times in
> >> Linus's tree that's been in -next for almost a whole release cycle.
> >
> >> The only bugs that I find that come from -next is integration issues,
> >> where an interface changes and another subsystem stumbles over it.
> >> That's exactly what it was for and what it's good at.
> >
> > In the embedded space it's much more common to track -next as people are
> > often working with multiple subsystems so the integration is important.
> > Most of my code is developed against -next then moved to topic branches
> > for submission.
> >
> > We also catch quite a lot of issues in -next as a result of the work on
> > boot testing that kernelci.org and Olof's bots are doing, hopefully
> > that'll start to build out to include test suites like kselftest (I know
> > there's work in progress there but no ETA as of yet).  Things get
> > exposed to a lot more systems and configurations than individual
> > maintainers have access to which can shake out issues in code that deals
> > with hardware.
> 
> I've stopped running -stable releases through the tester. It didn't
> fit the way I kept track of what's been built very well and it was
> hard to capture a useful state in which to test.
> 
> For a while I tried to capture the current-state-of-the-queue from
> gregkh's public quilt series ever so often, but it's quite churny.
> There's an -rc that's posted for review but not tagged and not
> provided as a git branch of applied patches, so it's hard to
> automatically test just those.
> 
> That would be the ideal setup for me though -- tagged or branched -rc
> candidates of stable releases that I'd be happy to put through the
> build/boot test at my end.

kernelci now handles -rc stable releases, so would this just be a
duplication of that work?

thanks,

greg k-h

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Sasha Levin]

On 07/15/2015 12:03 PM, Greg KH wrote:
>> > That would be the ideal setup for me though -- tagged or branched -rc
>> > candidates of stable releases that I'd be happy to put through the
>> > build/boot test at my end.
> kernelci now handles -rc stable releases, so would this just be a
> duplication of that work?

I've actually suggested -rc stable cycles for the purpose of getting -rc
stable kernels into users hands by making it possible for distros to
ship these as proposed/test kernels by giving them enough time (right now
the review cycle is only a few days, making it impossible for distros
to ship them).

As Guenter and Greg mentioned, there's already a fair amount of testing
being done on the queue branches, but there's really no testing being
done by end users before they receive a final kernel.


Thanks,
Sasha

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Greg KH]

On Wed, Jul 15, 2015 at 12:15:50PM -0400, Sasha Levin wrote:
> On 07/15/2015 12:03 PM, Greg KH wrote:
> >> > That would be the ideal setup for me though -- tagged or branched -rc
> >> > candidates of stable releases that I'd be happy to put through the
> >> > build/boot test at my end.
> > kernelci now handles -rc stable releases, so would this just be a
> > duplication of that work?
> 
> I've actually suggested -rc stable cycles for the purpose of getting -rc
> stable kernels into users hands by making it possible for distros to
> ship these as proposed/test kernels by giving them enough time (right now
> the review cycle is only a few days, making it impossible for distros
> to ship them).

Given that I do a new -stable release on the average of one a week,
getting these into a distro as a testing kernel and getting feedback is
going to be pretty much impossible.

> As Guenter and Greg mentioned, there's already a fair amount of testing
> being done on the queue branches, but there's really no testing being
> done by end users before they receive a final kernel.

I get feedback from Fedora, they always let me know when something goes
really wrong, which is pretty unusual.  So I don't know if "more
testing" is going to really pay off.

thanks,

greg k-h

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Josh Boyer]

On Wed, Jul 15, 2015 at 12:40 PM, Greg KH <greg@kroah.com> wrote:
> On Wed, Jul 15, 2015 at 12:15:50PM -0400, Sasha Levin wrote:
>> On 07/15/2015 12:03 PM, Greg KH wrote:
>> >> > That would be the ideal setup for me though -- tagged or branched -rc
>> >> > candidates of stable releases that I'd be happy to put through the
>> >> > build/boot test at my end.
>> > kernelci now handles -rc stable releases, so would this just be a
>> > duplication of that work?
>>
>> I've actually suggested -rc stable cycles for the purpose of getting -rc
>> stable kernels into users hands by making it possible for distros to
>> ship these as proposed/test kernels by giving them enough time (right now
>> the review cycle is only a few days, making it impossible for distros
>> to ship them).
>
> Given that I do a new -stable release on the average of one a week,
> getting these into a distro as a testing kernel and getting feedback is
> going to be pretty much impossible.
>
>> As Guenter and Greg mentioned, there's already a fair amount of testing
>> being done on the queue branches, but there's really no testing being
>> done by end users before they receive a final kernel.
>
> I get feedback from Fedora, they always let me know when something goes
> really wrong, which is pretty unusual.  So I don't know if "more
> testing" is going to really pay off.

Well... yes.  We do give feedback, sometimes even other than "oh my
this is broken."  However, our rollout plans tend to skip all the .1
stable releases because we've been bitten a few times in the past with
patches that went into -rc1 that were wrong.  Pretty much the
scenarios being discussed.  So we have held off until .2 comes out to
rebase our releases.

Now, we switched to that a while ago before you started the "must be
in a Linus release" rule, so maybe .1 releases are better now.  We
just don't have data to back it up on the Fedora side though, and I
wanted to be clear on why that is.

Another reason we don't ship .1 releases is because the backlog of
patches is usually huge still, and the prior 4.X-1.y release is
usually still getting updates at this point.  So shipping a .1 rebase
to turn around and ship .2 a week later is a lot of churn for
relatively little gain.  We can get the rebase and the extra patches
in .2 by simply waiting.

josh

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Sasha Levin]

On 07/15/2015 12:40 PM, Greg KH wrote:
>> On 07/15/2015 12:03 PM, Greg KH wrote:
>>>>> > >> > That would be the ideal setup for me though -- tagged or branched -rc
>>>>> > >> > candidates of stable releases that I'd be happy to put through the
>>>>> > >> > build/boot test at my end.
>>> > > kernelci now handles -rc stable releases, so would this just be a
>>> > > duplication of that work?
>> > 
>> > I've actually suggested -rc stable cycles for the purpose of getting -rc
>> > stable kernels into users hands by making it possible for distros to
>> > ship these as proposed/test kernels by giving them enough time (right now
>> > the review cycle is only a few days, making it impossible for distros
>> > to ship them).
> Given that I do a new -stable release on the average of one a week,
> getting these into a distro as a testing kernel and getting feedback is
> going to be pretty much impossible.

Agreed, it's probably better for the LTS kernels where it's better to trade
off release frequency for stability.

What's the reasoning for doing it 1 per week? To get fixes out earlier?


Thanks,
Sasha

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Greg KH]

On Wed, Jul 15, 2015 at 05:21:57PM -0400, Sasha Levin wrote:
> On 07/15/2015 12:40 PM, Greg KH wrote:
> >> On 07/15/2015 12:03 PM, Greg KH wrote:
> >>>>> > >> > That would be the ideal setup for me though -- tagged or branched -rc
> >>>>> > >> > candidates of stable releases that I'd be happy to put through the
> >>>>> > >> > build/boot test at my end.
> >>> > > kernelci now handles -rc stable releases, so would this just be a
> >>> > > duplication of that work?
> >> > 
> >> > I've actually suggested -rc stable cycles for the purpose of getting -rc
> >> > stable kernels into users hands by making it possible for distros to
> >> > ship these as proposed/test kernels by giving them enough time (right now
> >> > the review cycle is only a few days, making it impossible for distros
> >> > to ship them).
> > Given that I do a new -stable release on the average of one a week,
> > getting these into a distro as a testing kernel and getting feedback is
> > going to be pretty much impossible.
> 
> Agreed, it's probably better for the LTS kernels where it's better to trade
> off release frequency for stability.
> 
> What's the reasoning for doing it 1 per week? To get fixes out earlier?

Because if I don't, the number of patches are just too large for one
release.  And also it's good to get fixes out as soon as possible, right
now I'm running a few weeks behind and people are getting annoyed...

thanks,

greg k-h

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Sasha Levin]

On 07/15/2015 06:34 PM, Greg KH wrote:
>> What's the reasoning for doing it 1 per week? To get fixes out earlier?
> Because if I don't, the number of patches are just too large for one
> release.  And also it's good to get fixes out as soon as possible, right
> now I'm running a few weeks behind and people are getting annoyed...

If the issue is just the size of the release then -rc might work here:
rather than release once a week, you can finalize a release a week
after it's been an -rc:

week 1: 4.2.1-rc1
week 2: 4.2.1, 4.2.2-rc1
week 3: 4.2.2, 4.2.3-rc1
[...]

This would benefit both folks who won't their fixes in immediately, because
they can always grab the -rc, and folks who want more stable kernels since
we could only allow fixes to a give -rc release to be added to the tree.

You can make an argument that people can just use the release from one week
earlier, but that reduces from the potential stability of the release since
only fixes to a given tree would go between the -rc and the final release.


Thanks,
Sasha

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Greg KH]

On Wed, Jul 15, 2015 at 06:40:04PM -0400, Sasha Levin wrote:
> On 07/15/2015 06:34 PM, Greg KH wrote:
> >> What's the reasoning for doing it 1 per week? To get fixes out earlier?
> > Because if I don't, the number of patches are just too large for one
> > release.  And also it's good to get fixes out as soon as possible, right
> > now I'm running a few weeks behind and people are getting annoyed...
> 
> If the issue is just the size of the release then -rc might work here:
> rather than release once a week, you can finalize a release a week
> after it's been an -rc:
> 
> week 1: 4.2.1-rc1
> week 2: 4.2.1, 4.2.2-rc1
> week 3: 4.2.2, 4.2.3-rc1
> [...]

Ugh, that's just going to drive everyone crazy and confused.  Not to
mention a lot more work for me, and honestly, I'm pushed to my limit
with the number of stable/LTS kernels I'm handling right now.

And I doubt anyone will actually run the -rc releases, other than the
testers I have today :)

So I'll just leave things as-is, because it has worked pretty well for
the past decade.  There's always going to be bugs and regressions,
that's software.  What matters is how quickly they can be responded to.

thanks,

greg k-h

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Rafael J. Wysocki]

On Wednesday, July 15, 2015 08:36:42 PM Greg KH wrote:
> On Wed, Jul 15, 2015 at 06:40:04PM -0400, Sasha Levin wrote:
> > On 07/15/2015 06:34 PM, Greg KH wrote:
> > >> What's the reasoning for doing it 1 per week? To get fixes out earlier?
> > > Because if I don't, the number of patches are just too large for one
> > > release.  And also it's good to get fixes out as soon as possible, right
> > > now I'm running a few weeks behind and people are getting annoyed...
> > 
> > If the issue is just the size of the release then -rc might work here:
> > rather than release once a week, you can finalize a release a week
> > after it's been an -rc:
> > 
> > week 1: 4.2.1-rc1
> > week 2: 4.2.1, 4.2.2-rc1
> > week 3: 4.2.2, 4.2.3-rc1
> > [...]
> 
> Ugh, that's just going to drive everyone crazy and confused.  Not to
> mention a lot more work for me, and honestly, I'm pushed to my limit
> with the number of stable/LTS kernels I'm handling right now.
> 
> And I doubt anyone will actually run the -rc releases, other than the
> testers I have today :)
> 
> So I'll just leave things as-is, because it has worked pretty well for
> the past decade.  There's always going to be bugs and regressions,
> that's software.  What matters is how quickly they can be responded to.

I tend to agree here.

To me, there is a tradeoff between being fairly aggressive with pushing
fixes into -stable and spending more time on making them receive wider
testing before they go there.  The lack of balance either way will make
someone unhappy.

Thanks,
Rafael

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Zefan Li]

On 2015/7/16 0:15, Sasha Levin wrote:
> On 07/15/2015 12:03 PM, Greg KH wrote:
>>>> That would be the ideal setup for me though -- tagged or branched -rc
>>>> candidates of stable releases that I'd be happy to put through the
>>>> build/boot test at my end.
>> kernelci now handles -rc stable releases, so would this just be a
>> duplication of that work?
> 
> I've actually suggested -rc stable cycles for the purpose of getting -rc
> stable kernels into users hands by making it possible for distros to
> ship these as proposed/test kernels by giving them enough time (right now
> the review cycle is only a few days, making it impossible for distros
> to ship them).
> 
> As Guenter and Greg mentioned, there's already a fair amount of testing
> being done on the queue branches,

Now 0day has been testing stable releases on linux-stable.git, but I don't
think it tests queue branches? I asked Fengguang to run 0day on my 3.4.y
stable queue some time ago.

> but there's really no testing being
> done by end users before they receive a final kernel.
> 

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Olof Johansson]

On Wed, Jul 15, 2015 at 9:03 AM, Greg KH <greg@kroah.com> wrote:
> On Wed, Jul 15, 2015 at 10:52:13AM -0400, Olof Johansson wrote:
>> On Mon, Jul 13, 2015 at 2:51 PM, Mark Brown <broonie@kernel.org> wrote:
>> > On Mon, Jul 13, 2015 at 02:21:32PM -0400, Steven Rostedt wrote:
>> >
>> >> I disagree. I thought next was a place to have integration of new
>> >> development, and not just a place to test. Really, how many people test
>> >> next compared to Linus's tree? I trip over bugs all the times in
>> >> Linus's tree that's been in -next for almost a whole release cycle.
>> >
>> >> The only bugs that I find that come from -next is integration issues,
>> >> where an interface changes and another subsystem stumbles over it.
>> >> That's exactly what it was for and what it's good at.
>> >
>> > In the embedded space it's much more common to track -next as people are
>> > often working with multiple subsystems so the integration is important.
>> > Most of my code is developed against -next then moved to topic branches
>> > for submission.
>> >
>> > We also catch quite a lot of issues in -next as a result of the work on
>> > boot testing that kernelci.org and Olof's bots are doing, hopefully
>> > that'll start to build out to include test suites like kselftest (I know
>> > there's work in progress there but no ETA as of yet).  Things get
>> > exposed to a lot more systems and configurations than individual
>> > maintainers have access to which can shake out issues in code that deals
>> > with hardware.
>>
>> I've stopped running -stable releases through the tester. It didn't
>> fit the way I kept track of what's been built very well and it was
>> hard to capture a useful state in which to test.
>>
>> For a while I tried to capture the current-state-of-the-queue from
>> gregkh's public quilt series ever so often, but it's quite churny.
>> There's an -rc that's posted for review but not tagged and not
>> provided as a git branch of applied patches, so it's hard to
>> automatically test just those.
>>
>> That would be the ideal setup for me though -- tagged or branched -rc
>> candidates of stable releases that I'd be happy to put through the
>> build/boot test at my end.
>
> kernelci now handles -rc stable releases, so would this just be a
> duplication of that work?

Great, then I won't waste my time on it.


-Olof

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Sasha Levin]

On 07/13/2015 02:21 PM, Steven Rostedt wrote:
> On Mon, 13 Jul 2015 00:27:52 -0400
> Sasha Levin <sasha.levin@oracle.com> wrote:
> 
> 
>>> Many fixes are important but simply aren't that urgent so the two or
>>> more weeks is no great cost.
>>
>> I'd actually argue that Linus shouldn't be pulling *anything* that wasn't in
>> -next for 2+ weeks. There's no good excuse to want something pulled immediately
>> as the only benefit Linus's tree provides in that aspect is the wider testing
>> it receives, so it would make sense to weed out more bugs in -next before they
>> get to Linus.
> 
> I disagree. I thought next was a place to have integration of new
> development, and not just a place to test. Really, how many people test
> next compared to Linus's tree? I trip over bugs all the times in
> Linus's tree that's been in -next for almost a whole release cycle.

I didn't say that it's not for integration, I just said that with the
recent increase in testing by various people/organizations the focus
seems to be shifting to catching things in -next before they get into
Linus's tree.

> The only bugs that I find that come from -next is integration issues,
> where an interface changes and another subsystem stumbles over it.
> That's exactly what it was for and what it's good at.
> 
> I like the idea that patches marked for stable should sit in Linus's
> tree for at least 2 weeks before being pulled into stable. Linus's tree
> gets the most testing than any other tree and that's where new fixes
> should go.
> 
> 
>>
>> I think that this is a small mind-shift from thinking about Linus's tree as
>> an integration tree to considering it as mostly bug-free code, and stop
>> merging in risky patches. We already have -next for that.
> 
> No, we have -next as a way incorporate new code and see how things
> interact with other subsystems.

I don't see why we're disagreeing?

>>
>>> If a developer/maintainer thinks a fix is urgent, then they need to
>>> explicitly ask for a quick release, and that could be as easy as saying:
>>>
>>>   Cc: stable@vger.kernel.org (URGENT v3.0 and later)
>>>
>>> An 'URGENT' fix *should* come with an independent
>>> "Reviewed-by:"  (well ... everything should of course, but if URGENT
>>> stable patches with no Reviewed-by got some push-back, I think that
>>> would be a "very good thing" all around).
>>
>> I suppose that this is something Linus/maintainers would need to enforce? No
>> patch unless it lived in -next unless it was acked by the maintainer of that
>> subsystem?
>>
>>> I don't think that tightening the criteria for going into any
>>> particular tree will really help.  I'm not sure there is even real
>>> agreement on what is or is not allowed in -stable (we have clearly
>>> written rules, but the practice is really whatever a maintainer
>>> chooses).
>>> -rc prereleases for -stable would only help if people tested them.
>>> Given that the same bugs are in -linus, the testing done there should
>>> be sufficient providing it is given enough time.
>>
>> My reasoning for -rc prereleases for stable was to test out the backports that
>> go into stable kernels: they don't see the light of day until they're shipped
>> out to folks who want *stable* kernels, but end up being the first testers of
>> a backport.
> 
> Which to me looks like rational to let it sit in Linus's tree for a bit.

Backports don't end up in Linus's tree, they only live in our stable trees and
never see the light of day before we ship that tree out.

Thanks,
Sasha

>>
>> I don't want to suggest that we do a few of those per stable kernel, but even
>> one -rc release that would end up in distros (marked as "proposed/devel") and
>> would let users test that would be a great step forward.
>>
>> The reason I've suggested it for Ksummit rather than hashing it out on stable@
>> is that I believe that the solution is more complex and would need more discussion
>> than just slapping a "cooldown period" on patches. We need to make sure less
>> bugs/untested code ends up in Linus's tree to begin with, and we need a way to
>> validate proposed stable releases before releasing them, since -stable users
>> aren't interested in being testers.
> 
> I'm all for discussing this in person.
> 
> -- Steve
> 

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Steven Rostedt]

On Mon, 13 Jul 2015 20:42:00 -0400
Sasha Levin <sasha.levin@oracle.com> wrote:


> > I disagree. I thought next was a place to have integration of new
> > development, and not just a place to test. Really, how many people test
> > next compared to Linus's tree? I trip over bugs all the times in
> > Linus's tree that's been in -next for almost a whole release cycle.
> 
> I didn't say that it's not for integration, I just said that with the
> recent increase in testing by various people/organizations the focus
> seems to be shifting to catching things in -next before they get into
> Linus's tree.

Yes, it's great if we can catch things in -next. But I don't believe
that patches that fix bugs found in Linus's tree should sit in next
before going into Linus's tree, because those patches are basically
fixing stuff that was already in next and wasn't discovered until it
hit Linus's tree. Which is why I say it's a waste of time to put it in
next before sending straight to Linus.

 
> >>
> >> I think that this is a small mind-shift from thinking about Linus's tree as
> >> an integration tree to considering it as mostly bug-free code, and stop
> >> merging in risky patches. We already have -next for that.
> > 
> > No, we have -next as a way incorporate new code and see how things
> > interact with other subsystems.
> 
> I don't see why we're disagreeing?

I guess it's a point of view we are having. Maybe I'm misunderstanding
you. Are you concerned about fixes that are sent straight to Linus
without ever going into next? I do that quite often, and I too am
guilty of having those fixes cause other bugs that need to be fixed.
But I highly doubt that having them sit in next for two weeks would
change that. Again, the code that the patches fix was already in next,
and the original bug was never found there. What makes you think bugs
in the fix patches will be any different?


> >> My reasoning for -rc prereleases for stable was to test out the backports that
> >> go into stable kernels: they don't see the light of day until they're shipped
> >> out to folks who want *stable* kernels, but end up being the first testers of
> >> a backport.
> > 
> > Which to me looks like rational to let it sit in Linus's tree for a bit.
> 
> Backports don't end up in Linus's tree, they only live in our stable trees and
> never see the light of day before we ship that tree out.
> 

Oops, sorry. I misread what you wrote. I was thinking of it being in
next and missed the -rc prerelease comment about stable.

-- Steve

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Sasha Levin]

On 07/13/2015 09:02 PM, Steven Rostedt wrote:
> On Mon, 13 Jul 2015 20:42:00 -0400
> Sasha Levin <sasha.levin@oracle.com> wrote:
> 
> 
>>> I disagree. I thought next was a place to have integration of new
>>> development, and not just a place to test. Really, how many people test
>>> next compared to Linus's tree? I trip over bugs all the times in
>>> Linus's tree that's been in -next for almost a whole release cycle.
>>
>> I didn't say that it's not for integration, I just said that with the
>> recent increase in testing by various people/organizations the focus
>> seems to be shifting to catching things in -next before they get into
>> Linus's tree.
> 
> Yes, it's great if we can catch things in -next. But I don't believe
> that patches that fix bugs found in Linus's tree should sit in next
> before going into Linus's tree, because those patches are basically
> fixing stuff that was already in next and wasn't discovered until it
> hit Linus's tree. Which is why I say it's a waste of time to put it in
> next before sending straight to Linus.

I disagree: quite a few of the patches I send out fix issues in -next rather
than Linus's tree. What makes you say that "those patches are basically
fixing stuff that was already in next and wasn't discovered until it git
Linus's tree"?

For example, looking at the workflow of mm/, I'm pretty sure that Andrew
could attest to the amount of fixes that go into mmotm for patches that
are still in mmotm.

>>>> I think that this is a small mind-shift from thinking about Linus's tree as
>>>> an integration tree to considering it as mostly bug-free code, and stop
>>>> merging in risky patches. We already have -next for that.
>>>
>>> No, we have -next as a way incorporate new code and see how things
>>> interact with other subsystems.
>>
>> I don't see why we're disagreeing?
> 
> I guess it's a point of view we are having. Maybe I'm misunderstanding
> you. Are you concerned about fixes that are sent straight to Linus
> without ever going into next? I do that quite often, and I too am
> guilty of having those fixes cause other bugs that need to be fixed.
> But I highly doubt that having them sit in next for two weeks would
> change that. Again, the code that the patches fix was already in next,
> and the original bug was never found there. What makes you think bugs
> in the fix patches will be any different?

Well, look at the last issue I've reported and you fixed:

 - "tracing: Have filter check for balanced ops" was merged in 4.1-rc8 as a fix, as
far as I can tell it never went through -next.

 - 4.1 was released shortly after, and that fix was backported into stable kernels
(3.14, for example, has that patch).

 - I've reported an issue with the patch, which was fixed in "tracing/filter: Do not
WARN on operand count going below zero", and got into 4.2.

 - The second fix, however, didn't make it into all stable kernels, which still have
only the original buggy fix.

This is exactly the scenario I'm saying we should prevent. I would even claim that
if the original fix would go through -next I would have spotted the issue before
it got into Linus's hands/stable and all this could be avoided.



Thanks,
Sasha

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Jonathan Corbet]

On Mon, 13 Jul 2015 21:02:26 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> Yes, it's great if we can catch things in -next. But I don't believe
> that patches that fix bugs found in Linus's tree should sit in next
> before going into Linus's tree, because those patches are basically
> fixing stuff that was already in next and wasn't discovered until it
> hit Linus's tree. Which is why I say it's a waste of time to put it in
> next before sending straight to Linus.

That, of course, assumes that these fixes don't introduce *other* bugs
that might just be caught in -next...

In general, though, I think a lot of people see -next as -rc1 without the
quality control; it's volatile and scary.  So it's not surprising that it
doesn't get a lot of real-world testing.  And, as long as that's the case,
there's going to be a lot of bugs that are never caught in -next.

jon

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Stephen Rothwell]

Hi Jon,

On Mon, 13 Jul 2015 20:28:18 -0600 Jonathan Corbet <corbet@lwn.net> wrote:
>
> In general, though, I think a lot of people see -next as -rc1 without the
> quality control; it's volatile and scary.  So it's not surprising that it
> doesn't get a lot of real-world testing.  And, as long as that's the case,
> there's going to be a lot of bugs that are never caught in -next.

The rule for next is "the code is as ready as can be to be merged by
Linus (or other upstream maintainer)", so, in a sense, it should no be
so scary.  However in actuality, that rule is not followed as well as I
would like (as evidenced by the amount of rebasing that still goes
on).  :-(

Sure we say "release early, release often", but *not* too early,
please.  The cycle should be something like "merge, test, (repeat),
release" - not "merge, release, test (repeat)" as it seems sometimes.

-- 
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Geert Uytterhoeven]

On Tue, Jul 14, 2015 at 4:28 AM, Jonathan Corbet <corbet@lwn.net> wrote:
> On Mon, 13 Jul 2015 21:02:26 -0400
> Steven Rostedt <rostedt@goodmis.org> wrote:
>
>> Yes, it's great if we can catch things in -next. But I don't believe
>> that patches that fix bugs found in Linus's tree should sit in next
>> before going into Linus's tree, because those patches are basically
>> fixing stuff that was already in next and wasn't discovered until it
>> hit Linus's tree. Which is why I say it's a waste of time to put it in
>> next before sending straight to Linus.

It's about not introducing regressions to fix other regressions (and not
getting flamed by Linus)...

> That, of course, assumes that these fixes don't introduce *other* bugs
> that might just be caught in -next...
>
> In general, though, I think a lot of people see -next as -rc1 without the
> quality control; it's volatile and scary.  So it's not surprising that it
> doesn't get a lot of real-world testing.  And, as long as that's the case,
> there's going to be a lot of bugs that are never caught in -next.

In the old (pre-git/bk?) days, -rc1 was volatile and scary, and "release
candidate" was really a misnomer. You didn't dare to trust any system
you cared a bit about with an -rc before -rc3 or -rc4.

These days that role is played by -next, and -rc1 is usually fairly usable.
But it seems we can't get rid of the "early -rc is to be avoided" stamp.

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[James Bottomley]

On Mon, 2015-07-13 at 20:28 -0600, Jonathan Corbet wrote:
> On Mon, 13 Jul 2015 21:02:26 -0400
> Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> > Yes, it's great if we can catch things in -next. But I don't believe
> > that patches that fix bugs found in Linus's tree should sit in next
> > before going into Linus's tree, because those patches are basically
> > fixing stuff that was already in next and wasn't discovered until it
> > hit Linus's tree. Which is why I say it's a waste of time to put it in
> > next before sending straight to Linus.
> 
> That, of course, assumes that these fixes don't introduce *other* bugs
> that might just be caught in -next...
> 
> In general, though, I think a lot of people see -next as -rc1 without the
> quality control; it's volatile and scary.  So it's not surprising that it
> doesn't get a lot of real-world testing.  And, as long as that's the case,
> there's going to be a lot of bugs that are never caught in -next.

Yes, I'm with this.  Instantly into Linus' tree means we get a lot of
bug introducing fixes which we then have to sort out.  One of the
complaints the stable tree maintainers and the distros are making is
that it's hard to track the set of patches required for a fix that was
first done wrongly.

No harm comes to us from running regression fixes into -next and thus in
the 0day tests because they eventually get into the correct kernel and
the benefit is that bogus fixes may be picked up by the tests.  Why
would we not incubate for a while in -next when there's no down side and
plenty of upside?

The idea that fixes have to go ASAP without our standard review (and
-next and 0day are now part of our review) processes is completely wrong
in my opinion.

I think we can argue about the time length (or just leave it up to the
maintainer) but saying we should bypass the standard process is wrong.

James

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Steven Rostedt]

On Tue, 14 Jul 2015 12:03:15 +0100
James Bottomley <James.Bottomley@HansenPartnership.com> wrote:

> No harm comes to us from running regression fixes into -next and thus in
> the 0day tests because they eventually get into the correct kernel and
> the benefit is that bogus fixes may be picked up by the tests.  Why
> would we not incubate for a while in -next when there's no down side and
> plenty of upside?

Mark Brown mentioned the down side. Depending on what the bug is,
especially if it breaks Linus's build, or causes some other major
breakage, to wait in next means that Linus's tree (that everyone is
based on) will be broken for that long too. Which could stop other
types of testing of Linus's tree.

In fact, I have a series of tests I run before pushing any patch to
Linus. If my tests break because of other breakage in Linus's tree, I
have to stop what I'm doing and find those bugs before I can proceed.
With v4.2-rc1, I found 3 bugs that I had to fix or find workarounds
before I could push my fixes to Linus.

I've been trying to get time to test against -next before the merge
window opens, because my tests usually discover these there. But I
don't always have time to do so.

-- Steve

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[James Bottomley]

On Tue, 2015-07-14 at 09:29 -0400, Steven Rostedt wrote:
> On Tue, 14 Jul 2015 12:03:15 +0100
> James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> 
> > No harm comes to us from running regression fixes into -next and thus in
> > the 0day tests because they eventually get into the correct kernel and
> > the benefit is that bogus fixes may be picked up by the tests.  Why
> > would we not incubate for a while in -next when there's no down side and
> > plenty of upside?
> 
> Mark Brown mentioned the down side. Depending on what the bug is,
> especially if it breaks Linus's build, or causes some other major
> breakage, to wait in next means that Linus's tree (that everyone is
> based on) will be broken for that long too. Which could stop other
> types of testing of Linus's tree.

If you followed process in the first place, how could you possibly break
the build except for some corner case configuration which can wait for
the fix?  linux-next and 0day pick up this kind of breakage fairly
instantly.  If you wait before sending a pull, you'll see the reports in
time to correct.

> In fact, I have a series of tests I run before pushing any patch to
> Linus. If my tests break because of other breakage in Linus's tree, I
> have to stop what I'm doing and find those bugs before I can proceed.
> With v4.2-rc1, I found 3 bugs that I had to fix or find workarounds
> before I could push my fixes to Linus.

I'd encourage maintainers to run tests (I do).  But I wouldn't encourage
them to think that their tests are the pinnacle of testing ... hence the
process should be everything through our standard test suite by
publishing a for-next branch before sending a pull.

> I've been trying to get time to test against -next before the merge
> window opens, because my tests usually discover these there. But I
> don't always have time to do so.

Well, the more, the merrier.  If you have a suite of tests, just package
it up and send it off to Fengguang if you don't have time to run it.

James


> -- Steve
> 
> 
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
> 

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1708 bytes --]

On Tue, Jul 14, 2015 at 11:17:26PM +0300, James Bottomley wrote:
> On Tue, 2015-07-14 at 09:29 -0400, Steven Rostedt wrote:

> > Mark Brown mentioned the down side. Depending on what the bug is,
> > especially if it breaks Linus's build, or causes some other major
> > breakage, to wait in next means that Linus's tree (that everyone is
> > based on) will be broken for that long too. Which could stop other
> > types of testing of Linus's tree.

> If you followed process in the first place, how could you possibly break
> the build except for some corner case configuration which can wait for
> the fix?  linux-next and 0day pick up this kind of breakage fairly
> instantly.  If you wait before sending a pull, you'll see the reports in
> time to correct.

The specific case I'm thinking of had a dependency on some other changes
in -next that weren't targeted for Linus' tree and which triggered only
non-x86 architectures (but fairly obviously on some of them IIRC,
probably something like all*config).  With the other changes in -next
everything was fine, and with x86 everything was fine.  Most of the
testing on -next is on the integrated tree, though 0day probably should
have triggered.  

That interaction with other bits of -next definitely feels like a weak
spot for me when I'm sending changes to Linus.

> > I've been trying to get time to test against -next before the merge
> > window opens, because my tests usually discover these there. But I
> > don't always have time to do so.

> Well, the more, the merrier.  If you have a suite of tests, just package
> it up and send it off to Fengguang if you don't have time to run it.

Or get them into kselftest!

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Steven Rostedt]

On Tue, 14 Jul 2015 23:17:26 +0300
James Bottomley <James.Bottomley@HansenPartnership.com> wrote:


> > I've been trying to get time to test against -next before the merge
> > window opens, because my tests usually discover these there. But I
> > don't always have time to do so.
> 
> Well, the more, the merrier.  If you have a suite of tests, just package
> it up and send it off to Fengguang if you don't have time to run it.

A lot of my tests are big time hacks. When I get time, I start to
convert them over to the ftrace selftests. Some have already been
converted. Some of those tests wont go because they take up too much
time, and the selftests are suppose to be rather quick.

Perhaps we should talk at KS about what to do with tests that are not
suitable for selftests but would be nice for others to run. Should we
have a selftests section for "this may take a while to finish"?

-- Steve

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Andy Lutomirski]

On Tue, Jul 14, 2015 at 3:12 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
> On Tue, 14 Jul 2015 23:17:26 +0300
> James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
>
>
>> > I've been trying to get time to test against -next before the merge
>> > window opens, because my tests usually discover these there. But I
>> > don't always have time to do so.
>>
>> Well, the more, the merrier.  If you have a suite of tests, just package
>> it up and send it off to Fengguang if you don't have time to run it.
>
> A lot of my tests are big time hacks. When I get time, I start to
> convert them over to the ftrace selftests. Some have already been
> converted. Some of those tests wont go because they take up too much
> time, and the selftests are suppose to be rather quick.
>
> Perhaps we should talk at KS about what to do with tests that are not
> suitable for selftests but would be nice for others to run. Should we
> have a selftests section for "this may take a while to finish"?

I think we should, and I think we should move a bunch of existing
tests there.  I want to run quick smoke tests when I'm programming,
and kselftests is slow right now.

--Andy

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Jani Nikula]

On Tue, 14 Jul 2015, James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> On Mon, 2015-07-13 at 20:28 -0600, Jonathan Corbet wrote:
>> On Mon, 13 Jul 2015 21:02:26 -0400
>> Steven Rostedt <rostedt@goodmis.org> wrote:
>> 
>> > Yes, it's great if we can catch things in -next. But I don't believe
>> > that patches that fix bugs found in Linus's tree should sit in next
>> > before going into Linus's tree, because those patches are basically
>> > fixing stuff that was already in next and wasn't discovered until it
>> > hit Linus's tree. Which is why I say it's a waste of time to put it in
>> > next before sending straight to Linus.
>> 
>> That, of course, assumes that these fixes don't introduce *other* bugs
>> that might just be caught in -next...
>> 
>> In general, though, I think a lot of people see -next as -rc1 without the
>> quality control; it's volatile and scary.  So it's not surprising that it
>> doesn't get a lot of real-world testing.  And, as long as that's the case,
>> there's going to be a lot of bugs that are never caught in -next.
>
> Yes, I'm with this.  Instantly into Linus' tree means we get a lot of
> bug introducing fixes which we then have to sort out.  One of the
> complaints the stable tree maintainers and the distros are making is
> that it's hard to track the set of patches required for a fix that was
> first done wrongly.

Digging up an old thread, sorry...

I think one of the issues with the stable process is that when we add
stable tags (or Fixes: references) they are cast in stone in the commit
history. We can fix the code and everyone sees the current version, but
when you look at a commit intended for stable, it's not always as
trivial to figure out whether that was a good idea in hindsight.

When I sort out the drm/i915 fixes for current development kernels (or
-next) I often spend quite a bit of time doing git log/blame archeology
figuring out if it's a regression fix and what the regressing commit
was, etc. I've thought about scripting the git history to add git notes
to the commits that are referred to by later commits, as this would be
helpful in figuring out if there are *other* commits fixing issues in
the same regressing one. Alas I've never found the time.

I'm wondering if people would think it worthwhile to have a
collaborative effort of annotating commits using git notes, both
automatically and manually. The manual annotations might be things like,
"Cc: stable", "whoops, this was a bad idea for stable", or "this also
needs commit foo in stable". Of course, with more structure, but you get
the idea.

BR,
Jani.


>
> No harm comes to us from running regression fixes into -next and thus in
> the 0day tests because they eventually get into the correct kernel and
> the benefit is that bogus fixes may be picked up by the tests.  Why
> would we not incubate for a while in -next when there's no down side and
> plenty of upside?
>
> The idea that fixes have to go ASAP without our standard review (and
> -next and 0day are now part of our review) processes is completely wrong
> in my opinion.
>
> I think we can argue about the time length (or just leave it up to the
> maintainer) but saying we should bypass the standard process is wrong.
>
> James
>
>
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

-- 
Jani Nikula, Intel Open Source Technology Center

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Greg KH]

On Tue, Sep 01, 2015 at 11:44:40AM +0300, Jani Nikula wrote:
> On Tue, 14 Jul 2015, James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> > On Mon, 2015-07-13 at 20:28 -0600, Jonathan Corbet wrote:
> >> On Mon, 13 Jul 2015 21:02:26 -0400
> >> Steven Rostedt <rostedt@goodmis.org> wrote:
> >> 
> >> > Yes, it's great if we can catch things in -next. But I don't believe
> >> > that patches that fix bugs found in Linus's tree should sit in next
> >> > before going into Linus's tree, because those patches are basically
> >> > fixing stuff that was already in next and wasn't discovered until it
> >> > hit Linus's tree. Which is why I say it's a waste of time to put it in
> >> > next before sending straight to Linus.
> >> 
> >> That, of course, assumes that these fixes don't introduce *other* bugs
> >> that might just be caught in -next...
> >> 
> >> In general, though, I think a lot of people see -next as -rc1 without the
> >> quality control; it's volatile and scary.  So it's not surprising that it
> >> doesn't get a lot of real-world testing.  And, as long as that's the case,
> >> there's going to be a lot of bugs that are never caught in -next.
> >
> > Yes, I'm with this.  Instantly into Linus' tree means we get a lot of
> > bug introducing fixes which we then have to sort out.  One of the
> > complaints the stable tree maintainers and the distros are making is
> > that it's hard to track the set of patches required for a fix that was
> > first done wrongly.
> 
> Digging up an old thread, sorry...
> 
> I think one of the issues with the stable process is that when we add
> stable tags (or Fixes: references) they are cast in stone in the commit
> history. We can fix the code and everyone sees the current version, but
> when you look at a commit intended for stable, it's not always as
> trivial to figure out whether that was a good idea in hindsight.
> 
> When I sort out the drm/i915 fixes for current development kernels (or
> -next) I often spend quite a bit of time doing git log/blame archeology
> figuring out if it's a regression fix and what the regressing commit
> was, etc. I've thought about scripting the git history to add git notes
> to the commits that are referred to by later commits, as this would be
> helpful in figuring out if there are *other* commits fixing issues in
> the same regressing one. Alas I've never found the time.
> 
> I'm wondering if people would think it worthwhile to have a
> collaborative effort of annotating commits using git notes, both
> automatically and manually. The manual annotations might be things like,
> "Cc: stable", "whoops, this was a bad idea for stable", or "this also
> needs commit foo in stable". Of course, with more structure, but you get
> the idea.

git notes are a pain to propagate around, so while I like the idea of
this, I don't know how well the proposed solution would work.

thanks,

greg k-h

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Sasha Levin]

On 09/01/2015 04:52 PM, Greg KH wrote:
>> I'm wondering if people would think it worthwhile to have a
>> > collaborative effort of annotating commits using git notes, both
>> > automatically and manually. The manual annotations might be things like,
>> > "Cc: stable", "whoops, this was a bad idea for stable", or "this also
>> > needs commit foo in stable". Of course, with more structure, but you get
>> > the idea.
> git notes are a pain to propagate around, so while I like the idea of
> this, I don't know how well the proposed solution would work.

I agree, the way we work right now doesn't really help with collaboration between
us when working on multiple stable trees.

I do have a plan in mind to create a "standard" set of scripts (rather than each
using his own), where sharing of this metadata could be scripted in the background
and used transparently without having to deal with notes/note conflicts/etc.

However, we all do things a bit differently, so that's one of the reasons I think
the -stable workshow will be useful: we could figure out a common base and start
working from there.

In the future, it would be interesting to get distro kernel maintainers into
this mix as well...


Thanks,
Sasha

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Jiri Kosina]

On Tue, 1 Sep 2015, Sasha Levin wrote:

> In the future, it would be interesting to get distro kernel maintainers 
> into this mix as well...

You mean to the mix of people maintaining stable branches? There are 
already distro folks doing that (at least for 3.12 that's definitely the 
case).

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Sasha Levin]

On 09/01/2015 05:08 PM, Jiri Kosina wrote:
> On Tue, 1 Sep 2015, Sasha Levin wrote:
> 
>> In the future, it would be interesting to get distro kernel maintainers 
>> into this mix as well...
> 
> You mean to the mix of people maintaining stable branches? There are 
> already distro folks doing that (at least for 3.12 that's definitely the 
> case).
> 

Right, I guess what I was meaning to say was "all of the other distro folks".

Hopefully having a common base will make this integration easier.


Thanks,
Sasha

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Luis Henriques]

[ I'm also adding Kamal to the CC list as I'm not sure he's subscribed
  to this ML.  I don't know about the other maintainers listed in [1]. ]

On Tue, Sep 01, 2015 at 05:00:48PM -0400, Sasha Levin wrote:
> On 09/01/2015 04:52 PM, Greg KH wrote:
> >> I'm wondering if people would think it worthwhile to have a
> >> > collaborative effort of annotating commits using git notes, both
> >> > automatically and manually. The manual annotations might be things like,
> >> > "Cc: stable", "whoops, this was a bad idea for stable", or "this also
> >> > needs commit foo in stable". Of course, with more structure, but you get
> >> > the idea.
> > git notes are a pain to propagate around, so while I like the idea of
> > this, I don't know how well the proposed solution would work.
> 
> I agree, the way we work right now doesn't really help with collaboration between
> us when working on multiple stable trees.
> 
> I do have a plan in mind to create a "standard" set of scripts (rather than each
> using his own), where sharing of this metadata could be scripted in the background
> and used transparently without having to deal with notes/note conflicts/etc.
> 
> However, we all do things a bit differently, so that's one of the reasons I think
> the -stable workshow will be useful: we could figure out a common base and start
> working from there.
> 
> In the future, it would be interesting to get distro kernel maintainers into
> this mix as well...
> 

I would definitely be interested in being involved in this discussion.  We
currently maintain our set of scripts (as everyone else), but we still
follow the same stable process to maintain our kernels.

BTW, Kamal was already trying to incorporate git notes into our current
workflow in a similar way as Jani Nikula was suggesting before (and just
for reference, our workflow is documented here[2]).  The problem for
merging the notes is obviously the difficult bit ('git notes maintainers' :-)

[1] https://www.kernel.org/category/releases.html
[2] https://wiki.ubuntu.com/Kernel/Dev/ExtendedStable/Maintenance

Cheers,
--
Luís

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Rafael J. Wysocki]

On Monday, July 13, 2015 08:28:18 PM Jonathan Corbet wrote:
> On Mon, 13 Jul 2015 21:02:26 -0400
> Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> > Yes, it's great if we can catch things in -next. But I don't believe
> > that patches that fix bugs found in Linus's tree should sit in next
> > before going into Linus's tree, because those patches are basically
> > fixing stuff that was already in next and wasn't discovered until it
> > hit Linus's tree. Which is why I say it's a waste of time to put it in
> > next before sending straight to Linus.
> 
> That, of course, assumes that these fixes don't introduce *other* bugs
> that might just be caught in -next...
> 
> In general, though, I think a lot of people see -next as -rc1 without the
> quality control; it's volatile and scary.  So it's not surprising that it
> doesn't get a lot of real-world testing.  And, as long as that's the case,
> there's going to be a lot of bugs that are never caught in -next.

Well stated.

And in addition to that, -next is simply evolving too fast to get any deeper
testing at any point.

Thanks,
Rafael

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 847 bytes --]

On Thu, Jul 16, 2015 at 02:53:55AM +0200, Rafael J. Wysocki wrote:
> On Monday, July 13, 2015 08:28:18 PM Jonathan Corbet wrote:

> > In general, though, I think a lot of people see -next as -rc1 without the
> > quality control; it's volatile and scary.  So it's not surprising that it
> > doesn't get a lot of real-world testing.  And, as long as that's the case,
> > there's going to be a lot of bugs that are never caught in -next.

> Well stated.

> And in addition to that, -next is simply evolving too fast to get any deeper
> testing at any point.

Right.  The one advantage -next does have for testing over individual
development trees is that it's a single point to pick up the majority of
code targetted for -rc1 so where people are doing explicit testing it's
going to be one of the first trees they start looking at after Linus'
tree.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Stephen Rothwell]

Hi Steven,

On Mon, 13 Jul 2015 21:02:26 -0400 Steven Rostedt <rostedt@goodmis.org> wrote:
>
> Yes, it's great if we can catch things in -next. But I don't believe
> that patches that fix bugs found in Linus's tree should sit in next
> before going into Linus's tree, because those patches are basically
> fixing stuff that was already in next and wasn't discovered until it
> hit Linus's tree. Which is why I say it's a waste of time to put it in
> next before sending straight to Linus.

Those bugs are not always for stuff that was in -next - there is a
significant amount of code that still does not see -next until the
merge window opens (if ever) ... and sometimes the fixes do get to sit
in -next (in my fixes tree) because they do not get forwarded to Linus
quickly (or Linus is asleep :-)) or I find them and need them so that I
can get on with further merges.

-- 
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Geert Uytterhoeven]

On Tue, Jul 14, 2015 at 3:02 AM, Steven Rostedt <rostedt@goodmis.org> wrote:
> On Mon, 13 Jul 2015 20:42:00 -0400
> Sasha Levin <sasha.levin@oracle.com> wrote:
>> > I disagree. I thought next was a place to have integration of new
>> > development, and not just a place to test. Really, how many people test
>> > next compared to Linus's tree? I trip over bugs all the times in
>> > Linus's tree that's been in -next for almost a whole release cycle.
>>
>> I didn't say that it's not for integration, I just said that with the
>> recent increase in testing by various people/organizations the focus
>> seems to be shifting to catching things in -next before they get into
>> Linus's tree.
>
> Yes, it's great if we can catch things in -next. But I don't believe
> that patches that fix bugs found in Linus's tree should sit in next
> before going into Linus's tree, because those patches are basically
> fixing stuff that was already in next and wasn't discovered until it
> hit Linus's tree. Which is why I say it's a waste of time to put it in
> next before sending straight to Linus.

Worse, sometimes issues were detected in -next, but the problematic
commits still migrated to Linus' tree, despite the detection (and sometimes
even despite the availability of a fix for several weeks).

I'm not aware of any of these having migrated to -stable too, but I guess
it might happen.

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 918 bytes --]

On Mon, Jul 13, 2015 at 09:02:26PM -0400, Steven Rostedt wrote:

> But I highly doubt that having them sit in next for two weeks would
> change that. Again, the code that the patches fix was already in next,
> and the original bug was never found there. What makes you think bugs
> in the fix patches will be any different?

Two weeks seems excessive but having them there for a -next build means
the automated stuff has a chance to kick in which is 90% of what -next
gives us in terms of quality.  The benefit does depend on how likely the
change is to be affected by system/platform/config specific issues.

Indeed depending on the bug having things forced to sit in -next for too
long can be a real problem - I've had people refusing to fix build
errors in Linus' tree for a week as they wanted things to cook in -next
which took out automated testing of Linus' tree (and anything that
merged it) for the duration.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Sasha Levin]

On 07/14/2015 06:46 AM, Mark Brown wrote:
> On Mon, Jul 13, 2015 at 09:02:26PM -0400, Steven Rostedt wrote:
> 
>>> But I highly doubt that having them sit in next for two weeks would change that. Again, the code that the patches fix was already in next, and the original bug was never found there. What makes you think bugs in the fix patches will be any different?
> Two weeks seems excessive but having them there for a -next build means the automated stuff has a chance to kick in which is 90% of what -next gives us in terms of quality.  The benefit does depend on how likely the change is to be affected by system/platform/config specific issues.
> 
> Indeed depending on the bug having things forced to sit in -next for too long can be a real problem - I've had people refusing to fix build errors in Linus' tree for a week as they wanted things to cook in -next which took out automated testing of Linus' tree (and anything that merged it) for the duration.

I don't agree that letting things cook in -next is hurting anyone: if it's
a critical bug then Linus should probably delay releasing a final before
it's been resolved, and if it's not then there's no harm for letting it
sit outside for a bit.

It seems that the current process is to just release at rc7/8 and deal with
breakage during the next window, consider that 4.1-rc6..4.1-rc7 had 128
non-merge commits, and 4.1-rc7..4.1-rc8 had 98. While it's a decrease, it's
not a significant one, and I don't think that anyone would claim that if
there would have been an -rc9 it would have an insignificant amount of commits.

In summary: if there's a critical bug, hold on releasing a final version If not,
leave it alone to cook in -next.


Thanks,
Sasha

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1080 bytes --]

On Tue, Jul 14, 2015 at 09:57:28AM -0400, Sasha Levin wrote:

Re-wrapped the text, it looks like something joined all the paragraphs
up into single lines.

> On 07/14/2015 06:46 AM, Mark Brown wrote:

> > Indeed depending on the bug having things forced to sit in -next for
> > too long can be a real problem - I've had people refusing to fix
> > build errors in Linus' tree for a week as they wanted things to cook
> > in -next which took out automated testing of Linus' tree (and
> > anything that merged it) for the duration.

> I don't agree that letting things cook in -next is hurting anyone: if it's
> a critical bug then Linus should probably delay releasing a final before
> it's been resolved, and if it's not then there's no harm for letting it
> sit outside for a bit.

If the issue being fixed is serious enough to take out substantial
portion of our test coverage or affect a lot of other development usage
then that's really disruptive to other work, it impacts things like
bisection for example.  A strong rule does nobody any good, it's
overkill for the problem.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Sasha Levin]

On 07/14/2015 11:25 AM, Mark Brown wrote:
> On Tue, Jul 14, 2015 at 09:57:28AM -0400, Sasha Levin wrote:
> 
> Re-wrapped the text, it looks like something joined all the paragraphs up into single lines.
> 
>> On 07/14/2015 06:46 AM, Mark Brown wrote:
> 
>>> Indeed depending on the bug having things forced to sit in -next for too long can be a real problem - I've had people refusing to fix build errors in Linus' tree for a week as they wanted things to cook in -next which took out automated testing of Linus' tree (and anything that merged it) for the duration.
> 
>> I don't agree that letting things cook in -next is hurting anyone: if it's a critical bug then Linus should probably delay releasing a final before it's been resolved, and if it's not then there's no harm for letting it sit outside for a bit.
> 
> If the issue being fixed is serious enough to take out substantial portion of our test coverage or affect a lot of other development usage then that's really disruptive to other work, it impacts things like bisection for example.  A strong rule does nobody any good, it's overkill for the problem.

If there's an issue that causes that effect then the original commit that
caused it should be reverted rather than introducing an untested fix right
away.

Obviously not a hard rule, but it should be the case in general.


Thanks,
Sasha

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Steven Rostedt]

On Tue, 14 Jul 2015 11:32:27 -0400
Sasha Levin <sasha.levin@oracle.com> wrote:

> If there's an issue that causes that effect then the original commit that
> caused it should be reverted rather than introducing an untested fix right
> away.
> 

You'd think. But there's been times that I sent a revert patch where
Linus himself would just make the fix instead. So Linus is just as
guilty as the rest of us for adding in patches that were not tested in
-next.

-- Steve

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Sasha Levin]

On 07/14/2015 11:38 AM, Steven Rostedt wrote:
> On Tue, 14 Jul 2015 11:32:27 -0400
> Sasha Levin <sasha.levin@oracle.com> wrote:
> 
>> > If there's an issue that causes that effect then the original commit that
>> > caused it should be reverted rather than introducing an untested fix right
>> > away.
>> > 
> You'd think. But there's been times that I sent a revert patch where
> Linus himself would just make the fix instead. So Linus is just as
> guilty as the rest of us for adding in patches that were not tested in
> -next.

I suppose it depends on how trivial the patch in: I'm not saying not to do
it for simple things (spelling errors, trivial compile failures and such), but
yes - if Linus prefers to pull a more than trivial untested change to fix a
bug that makes him as guilty as the rest of us.

The point I'm trying to make is that a bad patch in Linus' tree has a wider
ripple effect than what it were in the past, while Linus might consider a bad
patch in one of the -rc releases something minor since "no one should be using
it for production" it really isn't the case any more, those patches can and
will end up with the folks who don't want to have them.


Thanks,
Sasha

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Steven Rostedt]

On Tue, 14 Jul 2015 11:53:24 -0400
Sasha Levin <sasha.levin@oracle.com> wrote:

 
> The point I'm trying to make is that a bad patch in Linus' tree has a wider
> ripple effect than what it were in the past, while Linus might consider a bad
> patch in one of the -rc releases something minor since "no one should be using
> it for production" it really isn't the case any more, those patches can and
> will end up with the folks who don't want to have them.

I have to ask, why?

Just because a stable tag is on a patch it automatically gets pulled
into stable? What about waiting a while before pulling in those
patches? Wait till -rc2 is out before pulling in any patches marked for
stable in -rc1. Then wait for -rc3 to pull in the patches that were
added in -rc2. But don't pull in any patches that has a "Fixes" to it
in the next -rc release.

That is, when -rc2 is released, only pull in the patches marked for
stable in -rc1 if there were no Fixes tags for them in -rc2. And so on.

Again, just placing stuff in -next isn't going to solve this. It may
help, but you will still have fixes that breaks things when they get
into Linus's tree no matter how long they were in -next. This is simply
because Linus's tree has a wider audience. But hopefully, the next
release candidate will have the fixes for anything that breaks in the
previous release candidate.

-- Steve

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Sasha Levin]

On 07/14/2015 12:02 PM, Steven Rostedt wrote:
> On Tue, 14 Jul 2015 11:53:24 -0400
> Sasha Levin <sasha.levin@oracle.com> wrote:
> 
>  
>> > The point I'm trying to make is that a bad patch in Linus' tree has a wider
>> > ripple effect than what it were in the past, while Linus might consider a bad
>> > patch in one of the -rc releases something minor since "no one should be using
>> > it for production" it really isn't the case any more, those patches can and
>> > will end up with the folks who don't want to have them.
> I have to ask, why?
> 
> Just because a stable tag is on a patch it automatically gets pulled
> into stable? What about waiting a while before pulling in those
> patches? Wait till -rc2 is out before pulling in any patches marked for
> stable in -rc1. Then wait for -rc3 to pull in the patches that were
> added in -rc2. But don't pull in any patches that has a "Fixes" to it
> in the next -rc release.
> 
> That is, when -rc2 is released, only pull in the patches marked for
> stable in -rc1 if there were no Fixes tags for them in -rc2. And so on.
> 
> Again, just placing stuff in -next isn't going to solve this. It may
> help, but you will still have fixes that breaks things when they get
> into Linus's tree no matter how long they were in -next. This is simply
> because Linus's tree has a wider audience. But hopefully, the next
> release candidate will have the fixes for anything that breaks in the
> previous release candidate.

I agree that this would be enough for -stable.

But wouldn't you agree that the policy of not passing patches in -rc cycles
through -next at all is incorrect?

I'm fine with not having a minimal time it must live in -next, but I really
think that it should be in -next at some point.



Thanks,
Sasha

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Steven Rostedt]

On Tue, 14 Jul 2015 15:30:37 -0400
Sasha Levin <sasha.levin@oracle.com> wrote:


> I agree that this would be enough for -stable.
> 
> But wouldn't you agree that the policy of not passing patches in -rc cycles
> through -next at all is incorrect?
> 
> I'm fine with not having a minimal time it must live in -next, but I really
> think that it should be in -next at some point.

I'm not sure I see this as totally necessary. But perhaps this should
be a KS topic, as I think this discussion is better resolved as a
discussion than over email.

-- Steve

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[NeilBrown]

On Tue, 14 Jul 2015 15:30:37 -0400 Sasha Levin <sasha.levin@oracle.com>
wrote:

> On 07/14/2015 12:02 PM, Steven Rostedt wrote:
> > On Tue, 14 Jul 2015 11:53:24 -0400
> > Sasha Levin <sasha.levin@oracle.com> wrote:
> > 
> >  
> >> > The point I'm trying to make is that a bad patch in Linus' tree has a wider
> >> > ripple effect than what it were in the past, while Linus might consider a bad
> >> > patch in one of the -rc releases something minor since "no one should be using
> >> > it for production" it really isn't the case any more, those patches can and
> >> > will end up with the folks who don't want to have them.
> > I have to ask, why?
> > 
> > Just because a stable tag is on a patch it automatically gets pulled
> > into stable? What about waiting a while before pulling in those
> > patches? Wait till -rc2 is out before pulling in any patches marked for
> > stable in -rc1. Then wait for -rc3 to pull in the patches that were
> > added in -rc2. But don't pull in any patches that has a "Fixes" to it
> > in the next -rc release.
> > 
> > That is, when -rc2 is released, only pull in the patches marked for
> > stable in -rc1 if there were no Fixes tags for them in -rc2. And so on.
> > 
> > Again, just placing stuff in -next isn't going to solve this. It may
> > help, but you will still have fixes that breaks things when they get
> > into Linus's tree no matter how long they were in -next. This is simply
> > because Linus's tree has a wider audience. But hopefully, the next
> > release candidate will have the fixes for anything that breaks in the
> > previous release candidate.
> 
> I agree that this would be enough for -stable.
> 
> But wouldn't you agree that the policy of not passing patches in -rc cycles
> through -next at all is incorrect?
> 
> I'm fine with not having a minimal time it must live in -next, but I really
> think that it should be in -next at some point.

What exactly is the value of sitting in -next for a while.
-next was originally to catch integration issues, and a "simple" bug
fix shouldn't have those.

0-day runs of -next, but then it runs on lots of other trees too.  So
if you want 0-day coverage (which I do), then the rule doesn't have to
"in -next" but only "in a 0-day tree".

So what, specifically, is the value that a bug-fix patch gets from
-next that it cannot get elsewhere?

Thanks,
NeilBrown


> 
> 
> 
> Thanks,
> Sasha
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Sasha Levin]

On 07/14/2015 09:49 PM, NeilBrown wrote:
> On Tue, 14 Jul 2015 15:30:37 -0400 Sasha Levin <sasha.levin@oracle.com>
> wrote:
> 
>> On 07/14/2015 12:02 PM, Steven Rostedt wrote:
>>> On Tue, 14 Jul 2015 11:53:24 -0400
>>> Sasha Levin <sasha.levin@oracle.com> wrote:
>>>
>>>  
>>>>> The point I'm trying to make is that a bad patch in Linus' tree has a wider
>>>>> ripple effect than what it were in the past, while Linus might consider a bad
>>>>> patch in one of the -rc releases something minor since "no one should be using
>>>>> it for production" it really isn't the case any more, those patches can and
>>>>> will end up with the folks who don't want to have them.
>>> I have to ask, why?
>>>
>>> Just because a stable tag is on a patch it automatically gets pulled
>>> into stable? What about waiting a while before pulling in those
>>> patches? Wait till -rc2 is out before pulling in any patches marked for
>>> stable in -rc1. Then wait for -rc3 to pull in the patches that were
>>> added in -rc2. But don't pull in any patches that has a "Fixes" to it
>>> in the next -rc release.
>>>
>>> That is, when -rc2 is released, only pull in the patches marked for
>>> stable in -rc1 if there were no Fixes tags for them in -rc2. And so on.
>>>
>>> Again, just placing stuff in -next isn't going to solve this. It may
>>> help, but you will still have fixes that breaks things when they get
>>> into Linus's tree no matter how long they were in -next. This is simply
>>> because Linus's tree has a wider audience. But hopefully, the next
>>> release candidate will have the fixes for anything that breaks in the
>>> previous release candidate.
>>
>> I agree that this would be enough for -stable.
>>
>> But wouldn't you agree that the policy of not passing patches in -rc cycles
>> through -next at all is incorrect?
>>
>> I'm fine with not having a minimal time it must live in -next, but I really
>> think that it should be in -next at some point.
> 
> What exactly is the value of sitting in -next for a while.
> -next was originally to catch integration issues, and a "simple" bug
> fix shouldn't have those.
> 
> 0-day runs of -next, but then it runs on lots of other trees too.  So
> if you want 0-day coverage (which I do), then the rule doesn't have to
> "in -next" but only "in a 0-day tree".
> 
> So what, specifically, is the value that a bug-fix patch gets from
> -next that it cannot get elsewhere?

Right, -next was originally there to catch integration but this is no longer
the case: between Fengguang's tests which go beyond just building, kernelci.org
which also does boot tests on multiple platforms, and various people (myself
included) who do various testing on -next, a good chunk of non-integration
bugs is getting caught in -next before it even reaches Linus.

We could keep closing our eyes and claiming that -next is there only to deal
with the likes of merge conflicts, but reality is different and it's actually
better than what you're suggesting the intention of -next is.


Thanks,
Sasha

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[NeilBrown]

On Tue, 14 Jul 2015 22:09:50 -0400 Sasha Levin <sasha.levin@oracle.com>
wrote:

> On 07/14/2015 09:49 PM, NeilBrown wrote:
> > On Tue, 14 Jul 2015 15:30:37 -0400 Sasha Levin <sasha.levin@oracle.com>
> > wrote:
> > 
> >> On 07/14/2015 12:02 PM, Steven Rostedt wrote:
> >>> On Tue, 14 Jul 2015 11:53:24 -0400
> >>> Sasha Levin <sasha.levin@oracle.com> wrote:
> >>>
> >>>  
> >>>>> The point I'm trying to make is that a bad patch in Linus' tree has a wider
> >>>>> ripple effect than what it were in the past, while Linus might consider a bad
> >>>>> patch in one of the -rc releases something minor since "no one should be using
> >>>>> it for production" it really isn't the case any more, those patches can and
> >>>>> will end up with the folks who don't want to have them.
> >>> I have to ask, why?
> >>>
> >>> Just because a stable tag is on a patch it automatically gets pulled
> >>> into stable? What about waiting a while before pulling in those
> >>> patches? Wait till -rc2 is out before pulling in any patches marked for
> >>> stable in -rc1. Then wait for -rc3 to pull in the patches that were
> >>> added in -rc2. But don't pull in any patches that has a "Fixes" to it
> >>> in the next -rc release.
> >>>
> >>> That is, when -rc2 is released, only pull in the patches marked for
> >>> stable in -rc1 if there were no Fixes tags for them in -rc2. And so on.
> >>>
> >>> Again, just placing stuff in -next isn't going to solve this. It may
> >>> help, but you will still have fixes that breaks things when they get
> >>> into Linus's tree no matter how long they were in -next. This is simply
> >>> because Linus's tree has a wider audience. But hopefully, the next
> >>> release candidate will have the fixes for anything that breaks in the
> >>> previous release candidate.
> >>
> >> I agree that this would be enough for -stable.
> >>
> >> But wouldn't you agree that the policy of not passing patches in -rc cycles
> >> through -next at all is incorrect?
> >>
> >> I'm fine with not having a minimal time it must live in -next, but I really
> >> think that it should be in -next at some point.
> > 
> > What exactly is the value of sitting in -next for a while.
> > -next was originally to catch integration issues, and a "simple" bug
> > fix shouldn't have those.
> > 
> > 0-day runs of -next, but then it runs on lots of other trees too.  So
> > if you want 0-day coverage (which I do), then the rule doesn't have to
> > "in -next" but only "in a 0-day tree".
> > 
> > So what, specifically, is the value that a bug-fix patch gets from
> > -next that it cannot get elsewhere?
> 
> Right, -next was originally there to catch integration but this is no longer
> the case: between Fengguang's tests which go beyond just building, kernelci.org
> which also does boot tests on multiple platforms, and various people (myself
> included) who do various testing on -next, a good chunk of non-integration
> bugs is getting caught in -next before it even reaches Linus.
> 
> We could keep closing our eyes and claiming that -next is there only to deal
> with the likes of merge conflicts, but reality is different and it's actually
> better than what you're suggesting the intention of -next is.

But 0-day doesn't *only* pull from -next.  Neither (as far as I can
tell) does kernelci.org.

Maybe I'm suggesting that it is wrong to add all this extra meaning to
linux-next.
It is definitely good to have this extra testing before code gets to
Linus.  It is definitely good to encourage people to make use of it.
I'm not sure that directing everything through -next (which has a delay
because it is curated by a human) is the right approach.

I want bugfixes to go quickly though automated tests (and 0-day is
delightfully quick - no need to wait for morning in .au for -next), and
then to Linus to sit for an extended time to be tested by the army of
beta-testers (who try things that no automated test would ever think
of).

Thanks,
NeilBrown

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[James Bottomley]

On Wed, 2015-07-15 at 12:28 +1000, NeilBrown wrote:
> On Tue, 14 Jul 2015 22:09:50 -0400 Sasha Levin <sasha.levin@oracle.com>
> wrote:
> 
> > On 07/14/2015 09:49 PM, NeilBrown wrote:
> > > On Tue, 14 Jul 2015 15:30:37 -0400 Sasha Levin <sasha.levin@oracle.com>
> > > wrote:
> > > 
> > >> On 07/14/2015 12:02 PM, Steven Rostedt wrote:
> > >>> On Tue, 14 Jul 2015 11:53:24 -0400
> > >>> Sasha Levin <sasha.levin@oracle.com> wrote:
> > >>>
> > >>>  
> > >>>>> The point I'm trying to make is that a bad patch in Linus' tree has a wider
> > >>>>> ripple effect than what it were in the past, while Linus might consider a bad
> > >>>>> patch in one of the -rc releases something minor since "no one should be using
> > >>>>> it for production" it really isn't the case any more, those patches can and
> > >>>>> will end up with the folks who don't want to have them.
> > >>> I have to ask, why?
> > >>>
> > >>> Just because a stable tag is on a patch it automatically gets pulled
> > >>> into stable? What about waiting a while before pulling in those
> > >>> patches? Wait till -rc2 is out before pulling in any patches marked for
> > >>> stable in -rc1. Then wait for -rc3 to pull in the patches that were
> > >>> added in -rc2. But don't pull in any patches that has a "Fixes" to it
> > >>> in the next -rc release.
> > >>>
> > >>> That is, when -rc2 is released, only pull in the patches marked for
> > >>> stable in -rc1 if there were no Fixes tags for them in -rc2. And so on.
> > >>>
> > >>> Again, just placing stuff in -next isn't going to solve this. It may
> > >>> help, but you will still have fixes that breaks things when they get
> > >>> into Linus's tree no matter how long they were in -next. This is simply
> > >>> because Linus's tree has a wider audience. But hopefully, the next
> > >>> release candidate will have the fixes for anything that breaks in the
> > >>> previous release candidate.
> > >>
> > >> I agree that this would be enough for -stable.
> > >>
> > >> But wouldn't you agree that the policy of not passing patches in -rc cycles
> > >> through -next at all is incorrect?
> > >>
> > >> I'm fine with not having a minimal time it must live in -next, but I really
> > >> think that it should be in -next at some point.
> > > 
> > > What exactly is the value of sitting in -next for a while.
> > > -next was originally to catch integration issues, and a "simple" bug
> > > fix shouldn't have those.
> > > 
> > > 0-day runs of -next, but then it runs on lots of other trees too.  So
> > > if you want 0-day coverage (which I do), then the rule doesn't have to
> > > "in -next" but only "in a 0-day tree".
> > > 
> > > So what, specifically, is the value that a bug-fix patch gets from
> > > -next that it cannot get elsewhere?
> > 
> > Right, -next was originally there to catch integration but this is no longer
> > the case: between Fengguang's tests which go beyond just building, kernelci.org
> > which also does boot tests on multiple platforms, and various people (myself
> > included) who do various testing on -next, a good chunk of non-integration
> > bugs is getting caught in -next before it even reaches Linus.
> > 
> > We could keep closing our eyes and claiming that -next is there only to deal
> > with the likes of merge conflicts, but reality is different and it's actually
> > better than what you're suggesting the intention of -next is.
> 
> But 0-day doesn't *only* pull from -next.  Neither (as far as I can
> tell) does kernelci.org.
> 
> Maybe I'm suggesting that it is wrong to add all this extra meaning to
> linux-next.
> It is definitely good to have this extra testing before code gets to
> Linus.  It is definitely good to encourage people to make use of it.
> I'm not sure that directing everything through -next (which has a delay
> because it is curated by a human) is the right approach.
> 
> I want bugfixes to go quickly though automated tests (and 0-day is
> delightfully quick - no need to wait for morning in .au for -next), and
> then to Linus to sit for an extended time to be tested by the army of
> beta-testers (who try things that no automated test would ever think
> of).

OK, granted 0day runs on all branches, not just for-next; but how do you
know it's run?  If all tests passed, you don't get a success report.
I'd be happy skipping linux-next for 0day if I knew it had run.

I just use the rule of thumb: must be in -next for two days because then
I know 0day (and the other checkers) have also run.  Effectively this
means that all -rc fixes have to be in by Thursday for the Sunday -rc
release.  Anything after Thursday goes in the next -rc.  Even if I'm
waiting only for 0day, the rule means I know it's run.

I don't consider this to be an onerous delay and it has, on occasion,
caught bogus fixes.

James

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[NeilBrown]

On Wed, 15 Jul 2015 13:13:54 +0300 James Bottomley
<James.Bottomley@HansenPartnership.com> wrote:

> On Wed, 2015-07-15 at 12:28 +1000, NeilBrown wrote:

> > I want bugfixes to go quickly though automated tests (and 0-day is
> > delightfully quick - no need to wait for morning in .au for -next), and
> > then to Linus to sit for an extended time to be tested by the army of
> > beta-testers (who try things that no automated test would ever think
> > of).
> 
> OK, granted 0day runs on all branches, not just for-next; but how do you
> know it's run?  If all tests passed, you don't get a success report.
> I'd be happy skipping linux-next for 0day if I knew it had run.
> 
> I just use the rule of thumb: must be in -next for two days because then
> I know 0day (and the other checkers) have also run.

Do you *really* know that?
Given that there seems to be value in knowing "0day has run
successfully", should we ask for an interface to do exactly that?

Hey .. we could even have a bot which watches lkml for pull requests,
grabs the git hash, and checks that 0day has run on it.  If not -
automatic public shaming ensues :-)


>  Effectively this
> means that all -rc fixes have to be in by Thursday for the Sunday -rc
> release.  Anything after Thursday goes in the next -rc.  Even if I'm
> waiting only for 0day, the rule means I know it's run.
> 
> I don't consider this to be an onerous delay and it has, on occasion,
> caught bogus fixes.

I agree - not particularly onerous (though I'd really like the timing
to work so that pull requests can be sent on Friday and get into the
next rc - but that is just me being fussy).
My point was really about using 'the right tool for the job', and I
don't think -next is the 'right tool for the job' for validating bug
fixes.

Thanks,
NeilBrown

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Andy Lutomirski]

On Wed, Jul 15, 2015 at 4:24 PM, NeilBrown <neilb@suse.com> wrote:
> On Wed, 15 Jul 2015 13:13:54 +0300 James Bottomley
>
>>
>> I just use the rule of thumb: must be in -next for two days because then
>> I know 0day (and the other checkers) have also run.
>
> Do you *really* know that?
> Given that there seems to be value in knowing "0day has run
> successfully", should we ask for an interface to do exactly that?
>
> Hey .. we could even have a bot which watches lkml for pull requests,
> grabs the git hash, and checks that 0day has run on it.  If not -
> automatic public shaming ensues :-)
>

As of today, I'm much less convinced that -next is suitable for this
purpose.  I have a really dumb, blatant, obvious regression sitting in
-next right now, and it's been there for a couple of weeks with no one
spotting it.  The 0-day bot didn't spot it either (and yes, I emailed
Fengguang suggesting another test for the bot).

I think it's hard to find any real substitute for a soak in -rc or a
real release.

--Andy

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Linus Torvalds]

On Wed, Jul 15, 2015 at 6:05 PM, Andy Lutomirski <luto@amacapital.net> wrote:
>
> As of today, I'm much less convinced that -next is suitable for this
> purpose.  I have a really dumb, blatant, obvious regression sitting in
> -next right now, and it's been there for a couple of weeks with no one
> spotting it.  The 0-day bot didn't spot it either (and yes, I emailed
> Fengguang suggesting another test for the bot).

I think -next is supremely useful, but let's not kid ourselves: it's
useful because it catches compile problems (particularly on odd
architectures) and integration issues.

Very few people actually *run* linux-next. At best, it gets some
boot-testing on a farm or two. Maybe one or two brave souls actually
run it on their own machines (I think Andrew may, for example). But
that's it.

I suspect that (possibly with the exception of the merge window)
keeping bugfixes in -next for a few days is more likely to *delay*
testing than it is to make it better. I suspect a lot more people run
my -git trees than run -next.

So I don't think -next is a panacea for testing. Not at all. It is
very good for what it does, but I don't think people should believe
that "testing" is what it does.

                  Linus

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Steven Rostedt]

On Thu, 16 Jul 2015 09:24:10 +1000
NeilBrown <neilb@suse.com> wrote:

> > I just use the rule of thumb: must be in -next for two days because then
> > I know 0day (and the other checkers) have also run.
> 
> Do you *really* know that?
> Given that there seems to be value in knowing "0day has run
> successfully", should we ask for an interface to do exactly that?
> 

There's an opt-in with Fengguang Wu's 0day tests to have it email you
when it finishes with "SUCCESS". I have that, as I don't do pushes for
any patches until I see that.

-- Steve

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[James Bottomley]

On Wed, 2015-07-15 at 21:25 -0400, Steven Rostedt wrote:
> On Thu, 16 Jul 2015 09:24:10 +1000
> NeilBrown <neilb@suse.com> wrote:
> 
> > > I just use the rule of thumb: must be in -next for two days because then
> > > I know 0day (and the other checkers) have also run.
> > 
> > Do you *really* know that?
> > Given that there seems to be value in knowing "0day has run
> > successfully", should we ask for an interface to do exactly that?
> > 
> 
> There's an opt-in with Fengguang Wu's 0day tests to have it email you
> when it finishes with "SUCCESS". I have that, as I don't do pushes for
> any patches until I see that.

How?  Is there actually some sort of interface to 0day? (/me has been
rather lazy and simply emails requests to Fengguang).  Detail would be
great to share.

James

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Jonathan Cameron]

On 16 July 2015 10:19:04 BST, James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
>On Wed, 2015-07-15 at 21:25 -0400, Steven Rostedt wrote:
>> On Thu, 16 Jul 2015 09:24:10 +1000
>> NeilBrown <neilb@suse.com> wrote:
>> 
>> > > I just use the rule of thumb: must be in -next for two days
>because then
>> > > I know 0day (and the other checkers) have also run.
>> > 
>> > Do you *really* know that?
>> > Given that there seems to be value in knowing "0day has run
>> > successfully", should we ask for an interface to do exactly that?
>> > 
>> 
>> There's an opt-in with Fengguang Wu's 0day tests to have it email you
>> when it finishes with "SUCCESS". I have that, as I don't do pushes
>for
>> any patches until I see that.
>
>How?  Is there actually some sort of interface to 0day? (/me has been
>rather lazy and simply emails requests to Fengguang).  Detail would be
>great to share.

Fengguang said back in Edinburgh to email him if you want success messages.

Don't think that has changed. I emailed across the room and been getting them ever since :) 
As with others I hold pull requests until they come in.

Jonathan
>
>James
>
>
>_______________________________________________
>Ksummit-discuss mailing list
>Ksummit-discuss@lists.linuxfoundation.org
>https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Fengguang Wu]

On Thu, Jul 16, 2015 at 12:19:04PM +0300, James Bottomley wrote:
> On Wed, 2015-07-15 at 21:25 -0400, Steven Rostedt wrote:
> > On Thu, 16 Jul 2015 09:24:10 +1000
> > NeilBrown <neilb@suse.com> wrote:
> > 
> > > > I just use the rule of thumb: must be in -next for two days because then
> > > > I know 0day (and the other checkers) have also run.
> > > 
> > > Do you *really* know that?
> > > Given that there seems to be value in knowing "0day has run
> > > successfully", should we ask for an interface to do exactly that?
> > > 
> > 
> > There's an opt-in with Fengguang Wu's 0day tests to have it email you
> > when it finishes with "SUCCESS". I have that, as I don't do pushes for
> > any patches until I see that.
> 
> How?  Is there actually some sort of interface to 0day? (/me has been
> rather lazy and simply emails requests to Fengguang).  Detail would be
> great to share.

Just send me an email and say something like

        I'd like to get build completion notification for git URLs

            git://neil.brown.name/md
            git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi.git

Then I'll add them to the notification list.

The build completion notification will be private emails to the
committer of each branch's HEAD commit. It will typically be sent
out 1-2 hours after your git push.

But note the email only means the "quick" set of about 100 kconfigs
are finished and the build farm is in healthy state.

To improve coverage, we added 2 features in the past year to let
0day system run kind of endless more tests in machine idle time:

- hundreds more randconfig + all defconfigs in arch/*/configs/**
- commit by commit testing
 
So if you want to be reasonably safe, consider waiting for 1 day
before sending out your patch.

Thanks,
Fengguang

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 783 bytes --]

On Tue, Jul 14, 2015 at 11:32:27AM -0400, Sasha Levin wrote:

Re-wrapped the text, it looks like something joined all the paragraphs
up into single lines.

> > If the issue being fixed is serious enough to take out substantial
> > portion of our test coverage or affect a lot of other development
> > usage then that's really disruptive to other work, it impacts things
> > like bisection for example.  A strong rule does nobody any good,
> > it's overkill for the problem.

> If there's an issue that causes that effect then the original commit that
> caused it should be reverted rather than introducing an untested fix right
> away.

> Obviously not a hard rule, but it should be the case in general.

That's still sending a change of course which was what was being
objected to.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Sasha Levin]

On 07/14/2015 11:56 AM, Mark Brown wrote:
>>> If the issue being fixed is serious enough to take out substantial
>>> > > portion of our test coverage or affect a lot of other development
>>> > > usage then that's really disruptive to other work, it impacts things
>>> > > like bisection for example.  A strong rule does nobody any good,
>>> > > it's overkill for the problem.
>> > If there's an issue that causes that effect then the original commit that
>> > caused it should be reverted rather than introducing an untested fix right
>> > away.
>> > Obviously not a hard rule, but it should be the case in general.
> That's still sending a change of course which was what was being
> objected to.

Lesser of two evils? I can't really come up with a safer solution rather than
reverting it (not blindly, the revert will need an ack too, and that's assuming
that the fix is not trivial).

Thanks,
Sasha

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Geert Uytterhoeven]

On Tue, Jul 14, 2015 at 9:01 PM, Sasha Levin <sasha.levin@oracle.com> wrote:
> On 07/14/2015 11:56 AM, Mark Brown wrote:
>>>> If the issue being fixed is serious enough to take out substantial
>>>> > > portion of our test coverage or affect a lot of other development
>>>> > > usage then that's really disruptive to other work, it impacts things
>>>> > > like bisection for example.  A strong rule does nobody any good,
>>>> > > it's overkill for the problem.
>>> > If there's an issue that causes that effect then the original commit that
>>> > caused it should be reverted rather than introducing an untested fix right
>>> > away.
>>> > Obviously not a hard rule, but it should be the case in general.
>> That's still sending a change of course which was what was being
>> objected to.
>
> Lesser of two evils? I can't really come up with a safer solution rather than
> reverting it (not blindly, the revert will need an ack too, and that's assuming
> that the fix is not trivial).

And that it can be reverted, i.e. there are no later commits that depend on it.

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Sasha Levin]

On 07/14/2015 03:18 PM, Geert Uytterhoeven wrote:
> On Tue, Jul 14, 2015 at 9:01 PM, Sasha Levin <sasha.levin@oracle.com> wrote:
>> > On 07/14/2015 11:56 AM, Mark Brown wrote:
>>>>> >>>> If the issue being fixed is serious enough to take out substantial
>>>>>>> >>>> > > portion of our test coverage or affect a lot of other development
>>>>>>> >>>> > > usage then that's really disruptive to other work, it impacts things
>>>>>>> >>>> > > like bisection for example.  A strong rule does nobody any good,
>>>>>>> >>>> > > it's overkill for the problem.
>>>>> >>> > If there's an issue that causes that effect then the original commit that
>>>>> >>> > caused it should be reverted rather than introducing an untested fix right
>>>>> >>> > away.
>>>>> >>> > Obviously not a hard rule, but it should be the case in general.
>>> >> That's still sending a change of course which was what was being
>>> >> objected to.
>> >
>> > Lesser of two evils? I can't really come up with a safer solution rather than
>> > reverting it (not blindly, the revert will need an ack too, and that's assuming
>> > that the fix is not trivial).
> And that it can be reverted, i.e. there are no later commits that depend on it.

Right, this it tricky - one would need to audit the commit the depend on it for
correctness knowing that a dependency has a bug.

The premise here is that if we're not dealing with trivial breakage, reverting is
generally safer than fixing, but for some reason we rarely take that path.


Thanks,
Sasha

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Jan Kara]

On Tue 14-07-15 15:31:46, Sasha Levin wrote:
> On 07/14/2015 03:18 PM, Geert Uytterhoeven wrote:
> > On Tue, Jul 14, 2015 at 9:01 PM, Sasha Levin <sasha.levin@oracle.com> wrote:
> >> > On 07/14/2015 11:56 AM, Mark Brown wrote:
> >>>>> >>>> If the issue being fixed is serious enough to take out substantial
> >>>>>>> >>>> > > portion of our test coverage or affect a lot of other development
> >>>>>>> >>>> > > usage then that's really disruptive to other work, it impacts things
> >>>>>>> >>>> > > like bisection for example.  A strong rule does nobody any good,
> >>>>>>> >>>> > > it's overkill for the problem.
> >>>>> >>> > If there's an issue that causes that effect then the original commit that
> >>>>> >>> > caused it should be reverted rather than introducing an untested fix right
> >>>>> >>> > away.
> >>>>> >>> > Obviously not a hard rule, but it should be the case in general.
> >>> >> That's still sending a change of course which was what was being
> >>> >> objected to.
> >> >
> >> > Lesser of two evils? I can't really come up with a safer solution rather than
> >> > reverting it (not blindly, the revert will need an ack too, and that's assuming
> >> > that the fix is not trivial).
> > And that it can be reverted, i.e. there are no later commits that depend on it.
> 
> Right, this it tricky - one would need to audit the commit the depend on it for
> correctness knowing that a dependency has a bug.
> 
> The premise here is that if we're not dealing with trivial breakage, reverting is
> generally safer than fixing, but for some reason we rarely take that path.

Frankly at least in my experience reverting usually happens if:

1) no other commits depend on the buggy one
2) the state after the commit is actually worse than before it (sometimes
you fix 95% of the problem and forget to fix some corner case - then
reverting would be counterproductive).

So at least in areas where I usually work things work fine in this regard.

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 452 bytes --]

On Tue, Jul 14, 2015 at 03:01:57PM -0400, Sasha Levin wrote:
> On 07/14/2015 11:56 AM, Mark Brown wrote:

> > That's still sending a change of course which was what was being
> > objected to.

> Lesser of two evils? I can't really come up with a safer solution rather than
> reverting it (not blindly, the revert will need an ack too, and that's assuming
> that the fix is not trivial).

AFAICT any change of any kind would have got the same response.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Jan Kara]

  Hello,

On Mon 13-07-15 10:52:10, NeilBrown wrote:
> 
> I've been bitten by this a couple of times too.  At least two fairly
> serious md bugs *never* got into a release from Linus, but did get into
> -stable and at least one into a vendor kernel.
>
> On Sun, 12 Jul 2015 09:32:11 -0400 Sasha Levin <sasha.levin@oracle.com>
> wrote:
> 
> 
> > > So it boils down to: "How soon to apply fixes to -stable?", and the trade-off
> > > between applying fixes early, but risking to break something unknown and new,
> > > vs. applying fixes late (after more validation), causing more breakage from a
> > > known issue.
> > 
> > That's just one solution, but there are a few more (which is why it's worth discussing
> > it :) ).
> > 
> > Consider also:
> > 
> >  - Aligning the stable release process with the kernel where we'd do a few release
> > candidates for the stable kernel before releasing it.
> > 
> >  - Tightening what is allowed to go in as -rc patches, requiring some time in -next
> > before it even gets into Linus's hands. Even for "serious" things (does it matter if
> > a fix for a privesc gets in -rc2 or -rc6, beyond that it would be pulled to stable
> > earlier?)
> > 
> >  - Differentiate the type of patches going into "regular" -stable, and LTS?
> > 
> 
> My proposal would be to change the default timing.
> Currently patches tagged for 'stable' go into the next -stable release
> after they get into Linus's tree.  You can ask for an exception
> (sooner, later, different patch) and Greg (or any other stable
> maintainer) tries to be accommodating.  But you have to remember to ask.
> 
> I would rather that the default was that patches don't go into -stable
> until they have
>   - been in a full release from Linus and
>   - been in a Linus's tree for at least 2 weeks.
>     (or 1 week times the age of the target in releases.
>      So a fix in 4.4 get to 4.3-stable after a week, 4.2-stable
>      after 2 weeks etc .... maybe I'm going over-board here).
> 
> Many fixes are important but simply aren't that urgent so the two or
> more weeks is no great cost.

I agree with this and it seems as a sensible thing. In the last merge window
I had introduced two regression (:-|) - one in audit and one in ext4.
Neither of these two patches was marked for stable but that doesn't really
make a difference. Now both patches passed a review, testing in maintainer
tree, testing in linux-next for quite a while and only once they went into
Linus' tree people found the regression relatively quickly (couple of days,
definitely less than two weeks). And it's not like maintainers were
mindlessly applying patches or not testing their tree, just they happened
to not hit the bugs. It's the breadth of testing Linus' tree gets which
helps to shake out more subtle bugs. So from that point of view leting the
patch live for some time in Linus' tree before merging it into stable tree
makes sense to me.

One could argue that we should be doing more testing of linux-next but
people tend to develop against Linus' tree rather than linux-next which
also makes Linus' tree natural testing target... So I think that would be
a difficult fight for doubtful gain.

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Greg KH]

On Mon, Jul 13, 2015 at 10:52:10AM +1000, NeilBrown wrote:
> 
> I've been bitten by this a couple of times too.  At least two fairly
> serious md bugs *never* got into a release from Linus, but did get into
> -stable and at least one into a vendor kernel.

How did that happen?  Did I not wait for a -rc release?  Did I miss
something else?  What broke down that caused this?

> On Sun, 12 Jul 2015 09:32:11 -0400 Sasha Levin <sasha.levin@oracle.com>
> wrote:
> 
> 
> > > So it boils down to: "How soon to apply fixes to -stable?", and the trade-off
> > > between applying fixes early, but risking to break something unknown and new,
> > > vs. applying fixes late (after more validation), causing more breakage from a
> > > known issue.
> > 
> > That's just one solution, but there are a few more (which is why it's worth discussing
> > it :) ).
> > 
> > Consider also:
> > 
> >  - Aligning the stable release process with the kernel where we'd do a few release
> > candidates for the stable kernel before releasing it.
> > 
> >  - Tightening what is allowed to go in as -rc patches, requiring some time in -next
> > before it even gets into Linus's hands. Even for "serious" things (does it matter if
> > a fix for a privesc gets in -rc2 or -rc6, beyond that it would be pulled to stable
> > earlier?)
> > 
> >  - Differentiate the type of patches going into "regular" -stable, and LTS?
> > 
> 
> My proposal would be to change the default timing.
> Currently patches tagged for 'stable' go into the next -stable release
> after they get into Linus's tree.  You can ask for an exception
> (sooner, later, different patch) and Greg (or any other stable
> maintainer) tries to be accommodating.  But you have to remember to ask.
> 
> I would rather that the default was that patches don't go into -stable
> until they have
>   - been in a full release from Linus and
>   - been in a Linus's tree for at least 2 weeks.
>     (or 1 week times the age of the target in releases.
>      So a fix in 4.4 get to 4.3-stable after a week, 4.2-stable
>      after 2 weeks etc .... maybe I'm going over-board here).
> 
> Many fixes are important but simply aren't that urgent so the two or
> more weeks is no great cost.

Really?  Based on the traffic I get, I have people asking me why a patch
is in Linus's tree is not yet in a -stable release about once a week or
so.

A year or so ago we made the decision to wait for a patch to show up in
a -rc release before adding it to -stable because people felt I was
being "too fast".  So we did that, and now people want to wait even
longer?  I don't buy it, and feel that will only delay the problem
another week.

If you look closely, you will note that for the past 3 months or so,
I've already been waiting an extra -rc release as it is, just due to
spare time issues on my side (travel, outreachy review, etc.).  And even
with that delay (which people do keep bugging me about, see the ALSA
email this weekend for an example of such a thing), we get bugs
introduced into -stable releases.

So I don't think that a manditory 2 week waiting period is going to help
out much here, sorry.

If it gets into Linus's tree you had better think it is correct,
especially if you tag it for stable.  Yes, bugs happen, that's part of
life, but let's not slow down everyone just because we get 1-2 bugs
introduced into -stable every 6 months or so.

> If a developer/maintainer thinks a fix is urgent, then they need to
> explicitly ask for a quick release, and that could be as easy as saying:
> 
>   Cc: stable@vger.kernel.org (URGENT v3.0 and later)

No, do the opposite, which I have seen and follow:
    Cc: stable@v.k.o # wait for -rc5 to be out

or even:
    Cc: stable@v.k.o # wait for -final to be out for 2 weeks

Mark the things that you think should be delayed, not the ones that you
think should be urgent.

That being said, if I have missed a patch that you did mark for stable
and want to see it go in faster (not waiting for a -rc or for my queue
to drain down to your patch), just drop an email to stable@ and let me
know, and I will make it happen.

> I don't think that tightening the criteria for going into any
> particular tree will really help.  I'm not sure there is even real
> agreement on what is or is not allowed in -stable (we have clearly
> written rules, but the practice is really whatever a maintainer
> chooses).

If I am not following the rules that are documented, please let me know,
as I really try hard to do so.  But it's a tough balancing act here,
some patches don't really fall into the defined rules, yet fix issues
that people hit, or fix performance issues, or other things.  I take
those on a case-by-case basis, as rules don't work for everything.

thanks,

greg k-h

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Sasha Levin]

On 07/13/2015 04:51 PM, Greg KH wrote:
> On Mon, Jul 13, 2015 at 10:52:10AM +1000, NeilBrown wrote:
>>
>> I've been bitten by this a couple of times too.  At least two fairly
>> serious md bugs *never* got into a release from Linus, but did get into
>> -stable and at least one into a vendor kernel.
> 
> How did that happen?  Did I not wait for a -rc release?  Did I miss
> something else?  What broke down that caused this?

This is a good point in general: it seems that a few folks have reported issues
where stable bit them in the arse, and if it happens to kernel folks it happens
to regular users too.

It'll be interesting to hear those incidents and figure out what went wrong if
you end up getting bitten by one, the stable folks do really want to know when
things don't work out right.

>> On Sun, 12 Jul 2015 09:32:11 -0400 Sasha Levin <sasha.levin@oracle.com>
>> wrote:
>>
>>
>>>> So it boils down to: "How soon to apply fixes to -stable?", and the trade-off
>>>> between applying fixes early, but risking to break something unknown and new,
>>>> vs. applying fixes late (after more validation), causing more breakage from a
>>>> known issue.
>>>
>>> That's just one solution, but there are a few more (which is why it's worth discussing
>>> it :) ).
>>>
>>> Consider also:
>>>
>>>  - Aligning the stable release process with the kernel where we'd do a few release
>>> candidates for the stable kernel before releasing it.
>>>
>>>  - Tightening what is allowed to go in as -rc patches, requiring some time in -next
>>> before it even gets into Linus's hands. Even for "serious" things (does it matter if
>>> a fix for a privesc gets in -rc2 or -rc6, beyond that it would be pulled to stable
>>> earlier?)
>>>
>>>  - Differentiate the type of patches going into "regular" -stable, and LTS?
>>>
>>
>> My proposal would be to change the default timing.
>> Currently patches tagged for 'stable' go into the next -stable release
>> after they get into Linus's tree.  You can ask for an exception
>> (sooner, later, different patch) and Greg (or any other stable
>> maintainer) tries to be accommodating.  But you have to remember to ask.
>>
>> I would rather that the default was that patches don't go into -stable
>> until they have
>>   - been in a full release from Linus and
>>   - been in a Linus's tree for at least 2 weeks.
>>     (or 1 week times the age of the target in releases.
>>      So a fix in 4.4 get to 4.3-stable after a week, 4.2-stable
>>      after 2 weeks etc .... maybe I'm going over-board here).
>>
>> Many fixes are important but simply aren't that urgent so the two or
>> more weeks is no great cost.
> 
> Really?  Based on the traffic I get, I have people asking me why a patch
> is in Linus's tree is not yet in a -stable release about once a week or
> so.
> 
> A year or so ago we made the decision to wait for a patch to show up in
> a -rc release before adding it to -stable because people felt I was
> being "too fast".  So we did that, and now people want to wait even
> longer?  I don't buy it, and feel that will only delay the problem
> another week.
> 
> If you look closely, you will note that for the past 3 months or so,
> I've already been waiting an extra -rc release as it is, just due to
> spare time issues on my side (travel, outreachy review, etc.).  And even
> with that delay (which people do keep bugging me about, see the ALSA
> email this weekend for an example of such a thing), we get bugs
> introduced into -stable releases.
> 
> So I don't think that a manditory 2 week waiting period is going to help
> out much here, sorry.
> 
> If it gets into Linus's tree you had better think it is correct,
> especially if you tag it for stable.  Yes, bugs happen, that's part of
> life, but let's not slow down everyone just because we get 1-2 bugs
> introduced into -stable every 6 months or so.

Looking back at the git log and my notes, it would have prevented 6 bugs
that I had to revert/apply a different fix to. I guess it depends if
there's a magic number here that will make everyone change their mind.

What's your opinion about introducing RC pre-releases for stable kernel?
if someone cares about getting a particular fix he can use the RC, if
he cares more about having a tested kernel he'll wait another week or
two for the release.


Thanks,
Sasha

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[NeilBrown]

On Mon, 13 Jul 2015 13:51:25 -0700 Greg KH <greg@kroah.com> wrote:

> On Mon, Jul 13, 2015 at 10:52:10AM +1000, NeilBrown wrote:
> > 
> > I've been bitten by this a couple of times too.  At least two fairly
> > serious md bugs *never* got into a release from Linus, but did get into
> > -stable and at least one into a vendor kernel.
> 
> How did that happen?  Did I not wait for a -rc release?  Did I miss
> something else?  What broke down that caused this?

"rc" is "release candidate", so not a release - at least not the way I
was using the word.
I meant that the bugs never appeared in a x.y, or 2.6.y kernel from
Linus.

> 
> > On Sun, 12 Jul 2015 09:32:11 -0400 Sasha Levin <sasha.levin@oracle.com>
> > wrote:
> > 
> > 
> > > > So it boils down to: "How soon to apply fixes to -stable?", and the trade-off
> > > > between applying fixes early, but risking to break something unknown and new,
> > > > vs. applying fixes late (after more validation), causing more breakage from a
> > > > known issue.
> > > 
> > > That's just one solution, but there are a few more (which is why it's worth discussing
> > > it :) ).
> > > 
> > > Consider also:
> > > 
> > >  - Aligning the stable release process with the kernel where we'd do a few release
> > > candidates for the stable kernel before releasing it.
> > > 
> > >  - Tightening what is allowed to go in as -rc patches, requiring some time in -next
> > > before it even gets into Linus's hands. Even for "serious" things (does it matter if
> > > a fix for a privesc gets in -rc2 or -rc6, beyond that it would be pulled to stable
> > > earlier?)
> > > 
> > >  - Differentiate the type of patches going into "regular" -stable, and LTS?
> > > 
> > 
> > My proposal would be to change the default timing.
> > Currently patches tagged for 'stable' go into the next -stable release
> > after they get into Linus's tree.  You can ask for an exception
> > (sooner, later, different patch) and Greg (or any other stable
> > maintainer) tries to be accommodating.  But you have to remember to ask.
> > 
> > I would rather that the default was that patches don't go into -stable
> > until they have
> >   - been in a full release from Linus and
> >   - been in a Linus's tree for at least 2 weeks.
> >     (or 1 week times the age of the target in releases.
> >      So a fix in 4.4 get to 4.3-stable after a week, 4.2-stable
> >      after 2 weeks etc .... maybe I'm going over-board here).
> > 
> > Many fixes are important but simply aren't that urgent so the two or
> > more weeks is no great cost.
> 
> Really?  Based on the traffic I get, I have people asking me why a patch
> is in Linus's tree is not yet in a -stable release about once a week or
> so.

Certainly some patches are urgent.  And some people are impatient.
Creating a general expectation that "it takes at least 2 weeks unless
the maintainer explicit asks otherwise" would at least give you an easy
concrete answer.



> 
> A year or so ago we made the decision to wait for a patch to show up in
> a -rc release before adding it to -stable because people felt I was
> being "too fast".  So we did that, and now people want to wait even
> longer?  I don't buy it, and feel that will only delay the problem
> another week.
> 
> If you look closely, you will note that for the past 3 months or so,
> I've already been waiting an extra -rc release as it is, just due to
> spare time issues on my side (travel, outreachy review, etc.).  And even
> with that delay (which people do keep bugging me about, see the ALSA
> email this weekend for an example of such a thing), we get bugs
> introduced into -stable releases.
> 
> So I don't think that a manditory 2 week waiting period is going to help
> out much here, sorry.

My focus was more "wait for it to get into a -final release from Linus"
- the "two weeks" was only for fixes that got into -final late.

When i have minor bug fixes I usually keep them for the merge window
but also mark them for 'stable'.
What should I add to request that they don't migrate to -stable until
they are in a -final release from Linus?  I'll make that tagging my
default, and only over-ride if I have a strong reason (and a
reviewed-by)

And I never suggested a "mandatory 2 week" I suggested a "default 2
week".  Very different.


> 
> If it gets into Linus's tree you had better think it is correct,
> especially if you tag it for stable.  Yes, bugs happen, that's part of
> life, but let's not slow down everyone just because we get 1-2 bugs
> introduced into -stable every 6 months or so.
> 
> > If a developer/maintainer thinks a fix is urgent, then they need to
> > explicitly ask for a quick release, and that could be as easy as saying:
> > 
> >   Cc: stable@vger.kernel.org (URGENT v3.0 and later)
> 
> No, do the opposite, which I have seen and follow:
>     Cc: stable@v.k.o # wait for -rc5 to be out
> 
> or even:
>     Cc: stable@v.k.o # wait for -final to be out for 2 weeks
> 
> Mark the things that you think should be delayed, not the ones that you
> think should be urgent.

But most should be delayed and few are really urgent....

I'll try to make the later of your examples my default.

> 
> That being said, if I have missed a patch that you did mark for stable
> and want to see it go in faster (not waiting for a -rc or for my queue
> to drain down to your patch), just drop an email to stable@ and let me
> know, and I will make it happen.
> 
> > I don't think that tightening the criteria for going into any
> > particular tree will really help.  I'm not sure there is even real
> > agreement on what is or is not allowed in -stable (we have clearly
> > written rules, but the practice is really whatever a maintainer
> > chooses).
> 
> If I am not following the rules that are documented, please let me know,
> as I really try hard to do so.  But it's a tough balancing act here,
> some patches don't really fall into the defined rules, yet fix issues
> that people hit, or fix performance issues, or other things.  I take
> those on a case-by-case basis, as rules don't work for everything.

I have no reason to think you aren't following the rules as documented
and have always found you suitably responsive and helpful.

I'm just suggesting that maybe the current rules aren't the best
possible, and am looking to see how other people respond to the
suggestion.  I think I've decided how I'll respond to the suggestion
for now.

Thanks,
NeilBrown

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Steven Rostedt]

On Mon, 13 Jul 2015 13:51:25 -0700
Greg KH <greg@kroah.com> wrote:


> So I don't think that a manditory 2 week waiting period is going to help
> out much here, sorry.

The exact same thing can be said about having patches brew in -next for
2 weeks (or whatever) before going to Linus. If issues are not caught
in Linus's tree after two weeks, I highly doubt two weeks in -next will
be any better.

-- Steve

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[James Bottomley]

On July 15, 2015 10:41:51 PM GMT+03:00, Steven Rostedt <rostedt@goodmis.org> wrote:
>On Mon, 13 Jul 2015 13:51:25 -0700
>Greg KH <greg@kroah.com> wrote:
>
>
>> So I don't think that a manditory 2 week waiting period is going to
>help
>> out much here, sorry.
>
>The exact same thing can be said about having patches brew in -next for
>2 weeks (or whatever) before going to Linus. If issues are not caught
>in Linus's tree after two weeks, I highly doubt two weeks in -next will
>be any better.

Can we get off the two week delay hobby horse?  The contention is that we need some incubation for 0day and -next to run.  I think a couple of days is sufficient but we can debate that.  The thing I think is irresponsible is immediate onward transmission of alleged fixes without exposing them to our test and integration infrastructure because that induces unnecessary regressions into the -rc cycle.

James

>-- Steve
>_______________________________________________
>Ksummit-discuss mailing list
>Ksummit-discuss@lists.linuxfoundation.org
>https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Masami Hiramatsu]

On 2015/07/12 1:12, Sasha Levin wrote:
> Hi folks,
> 
> I'd like to propose a topic discussing issues that are caused as a result of the
> way development happens upstream, and the way we integrate with distros that affect
> the quality of stable trees:
> 
[...]
>  2. The review cycle: I've *never* ended up receiving comments during review cycle
> of a stable release. I've received comments either when I've sent my "added to the..."
> mails when I've added a patch in, which usually came from the authors of the patch
> or the maintainers of the subsystem, and I've received comments after the tree has
> shipped - when it actually broke something.
> 
> We need to explore ways to integrate the review process better with the end users,
> possibly by extending it to allow distributions to ship "proposed" review kernels
> rather than waiting for us to finalize a stable kernel before they start working
> on shipping it.

Hmm, it sounds like distro's business...

> 
>  3. Cross tree verification and auditing: There seems to be a fair amount of LTS
> kernels that are maintained openly on the stable@ ML, and even a bigger amount
> if the Canonical folks decide to play ball at any stage. While ideally each tree
> should contain (if required, correctly backported) patches that are relevant only
> to that given tree, we have no standard way to verify that.
> 
> We need a mechanism that would let us audit the existence (and non-existence) of
> patches in an easy way, and to compare backports between stable trees to help verify
> their correctness.

Agreed. Can't we use bugzilla.kernel.org for this purpose? Of course, more automated-way
is better :)

>  4. Upstream monitoring: I've suggested to Greg that we have a bot looking at
> commits going upstream, and for every commit marked as for stable it would attempt
> to apply it to all relevant stable trees and build them, and on failure would
> notify the author.
> 
> Greg objected for two main reasons: the first is that we should put more effort
> into trying to fix any possible issues which arise from failure to build and
> backport before we send mails out, which I accepted. The other issue was that he
> doesn't want to generate too much noise, and if the patch doesn't look important
> enough and only applies to the latest stable it's enough, and no need to bother
> people to backport it.

Hmm, I think it depends on how many stable trees to be maintained officially.
We can see 2 stables and 7 longterms on www.kernel.org, porting each bugfix to
all of them is hard...

Thank you,

> 
> So this is mostly an open discussion: what do people expect to do (if anything)
> as a result of marking a patch for stable? What do people think about the increased
> noise? Is there a better way to do it rather than by mails?
> 
> 
> Thanks,
> Sasha
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
> 


-- 
Masami HIRAMATSU
Linux Technology Research Center, System Productivity Research Dept.
Center for Technology Innovation - Systems Engineering
Hitachi, Ltd., Research & Development Group
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Zefan Li]

On 2015/7/12 0:12, Sasha Levin wrote:
> Hi folks,
> 
> I'd like to propose a topic discussing issues that are caused as a result of the
> way development happens upstream, and the way we integrate with distros that affect
> the quality of stable trees:
> 
>  1. During the RC cycles bug fixes tend to get sent to Linus without going through
> linux-next. This is very risky, but it seems to work(?). The problem is that Linus
> doesn't restrict those fixes to bugs that were introduced in the current merge window
> but takes anything that is labelled as a "fix".
> 
> The result is that there is a significant amount of mostly untested RC patches
> trickling down into stable trees, causing breakage for folks who assume that they
> are running a tested kernel but end up with commits that haven't even been in
> linux-next for more than a few days.
> 
> Since for RC kernel it's expected to see issues, and it's easy to correct, this is
> less than a problem, but consider this flow for stable:
> 
>  * 4.0: bug "A" introduced.
>  * 4.2-rc1: bug "A" fixed, but fix unknowingly introduced bug "B".
>  * 4.1.1: ships with fix for "A", and new bug "B".
>  * Stable user machines suffer from breakage.
>  * 4.2-rc7: bug "B" fixed.
>  * Stable users still suffer until the next kernel release.
> 
> So while it was quickly fixed for RC, this seriously affects stable.
> 

For 3.4.y, when I finish backporting patches for 4.x and about to release new 3.4.y,
4.(x+1) has released. My scripts will go through all the commits between 4.x and 
4.(x+1) to find out fixes for regressions introduced by my backports. Besides I'll
pick up some other important fixes.

One reason is I don't have enough time to catch up with upstream. Another reason is
to avoid the issue you described here.

> I actually just had this happen with "nfs: take extra reference to fl->fl_file when
> running a LOCKU operation" on the day I was writing this mail.
> 
>  2. The review cycle: I've *never* ended up receiving comments during review cycle
> of a stable release. I've received comments either when I've sent my "added to the..."
> mails when I've added a patch in, which usually came from the authors of the patch
> or the maintainers of the subsystem, and I've received comments after the tree has
> shipped - when it actually broke something.
> 

I do get some replies during review cycle, and so does Ben. We don't send out mails to
remind people when we add a patch to the stable queue.

> We need to explore ways to integrate the review process better with the end users,
> possibly by extending it to allow distributions to ship "proposed" review kernels
> rather than waiting for us to finalize a stable kernel before they start working
> on shipping it.
> 
>  3. Cross tree verification and auditing: There seems to be a fair amount of LTS
> kernels that are maintained openly on the stable@ ML, and even a bigger amount
> if the Canonical folks decide to play ball at any stage. While ideally each tree
> should contain (if required, correctly backported) patches that are relevant only
> to that given tree, we have no standard way to verify that.
> 
> We need a mechanism that would let us audit the existence (and non-existence) of
> patches in an easy way, and to compare backports between stable trees to help verify
> their correctness.
> 

I've once compared 3.2.y and 3.4.y when I haven't taken over 3.4 from Greg, and I
found there were hundreds of fixes in 3.2.y that are applicable to 3.4.y, and it's
mainly because Ben has been manually analyzing commits which have stable tags but
can't be backported automatically while Greg doesn't work like this because of
time budget.

The basic principle is if a fix has been backported to an older stable tree, newer
stable trees probably also need it.

>  4. Upstream monitoring: I've suggested to Greg that we have a bot looking at
> commits going upstream, and for every commit marked as for stable it would attempt
> to apply it to all relevant stable trees and build them, and on failure would
> notify the author.
> 

I would never do this for 3.4.y, because for a kernel as old as 3.4 such failure
is very common.

> Greg objected for two main reasons: the first is that we should put more effort
> into trying to fix any possible issues which arise from failure to build and
> backport before we send mails out, which I accepted. The other issue was that he
> doesn't want to generate too much noise, and if the patch doesn't look important
> enough and only applies to the latest stable it's enough, and no need to bother
> people to backport it.
> 
> So this is mostly an open discussion: what do people expect to do (if anything)
> as a result of marking a patch for stable? What do people think about the increased
> noise? Is there a better way to do it rather than by mails?
> 

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Sasha Levin]

On 07/13/2015 06:15 AM, Zefan Li wrote:
> On 2015/7/12 0:12, Sasha Levin wrote:
>> Hi folks,
>>
>> I'd like to propose a topic discussing issues that are caused as a result of the
>> way development happens upstream, and the way we integrate with distros that affect
>> the quality of stable trees:
>>
>>  1. During the RC cycles bug fixes tend to get sent to Linus without going through
>> linux-next. This is very risky, but it seems to work(?). The problem is that Linus
>> doesn't restrict those fixes to bugs that were introduced in the current merge window
>> but takes anything that is labelled as a "fix".
>>
>> The result is that there is a significant amount of mostly untested RC patches
>> trickling down into stable trees, causing breakage for folks who assume that they
>> are running a tested kernel but end up with commits that haven't even been in
>> linux-next for more than a few days.
>>
>> Since for RC kernel it's expected to see issues, and it's easy to correct, this is
>> less than a problem, but consider this flow for stable:
>>
>>  * 4.0: bug "A" introduced.
>>  * 4.2-rc1: bug "A" fixed, but fix unknowingly introduced bug "B".
>>  * 4.1.1: ships with fix for "A", and new bug "B".
>>  * Stable user machines suffer from breakage.
>>  * 4.2-rc7: bug "B" fixed.
>>  * Stable users still suffer until the next kernel release.
>>
>> So while it was quickly fixed for RC, this seriously affects stable.
>>
> 
> For 3.4.y, when I finish backporting patches for 4.x and about to release new 3.4.y,
> 4.(x+1) has released. My scripts will go through all the commits between 4.x and 
> 4.(x+1) to find out fixes for regressions introduced by my backports. Besides I'll
> pick up some other important fixes.
> 
> One reason is I don't have enough time to catch up with upstream. Another reason is
> to avoid the issue you described here.

That sounds like the sort of delay we should be doing, but you could still get bitten
by pulling a bad patch from the last RC of the 4.(x) release, no?

>> I actually just had this happen with "nfs: take extra reference to fl->fl_file when
>> running a LOCKU operation" on the day I was writing this mail.
>>
>>  2. The review cycle: I've *never* ended up receiving comments during review cycle
>> of a stable release. I've received comments either when I've sent my "added to the..."
>> mails when I've added a patch in, which usually came from the authors of the patch
>> or the maintainers of the subsystem, and I've received comments after the tree has
>> shipped - when it actually broke something.
>>
> 
> I do get some replies during review cycle, and so does Ben. We don't send out mails to
> remind people when we add a patch to the stable queue.

I guess it would make sense to either send emails when it was added or for the review
cycle, but not for both?

>> We need to explore ways to integrate the review process better with the end users,
>> possibly by extending it to allow distributions to ship "proposed" review kernels
>> rather than waiting for us to finalize a stable kernel before they start working
>> on shipping it.
>>
>>  3. Cross tree verification and auditing: There seems to be a fair amount of LTS
>> kernels that are maintained openly on the stable@ ML, and even a bigger amount
>> if the Canonical folks decide to play ball at any stage. While ideally each tree
>> should contain (if required, correctly backported) patches that are relevant only
>> to that given tree, we have no standard way to verify that.
>>
>> We need a mechanism that would let us audit the existence (and non-existence) of
>> patches in an easy way, and to compare backports between stable trees to help verify
>> their correctness.
>>
> 
> I've once compared 3.2.y and 3.4.y when I haven't taken over 3.4 from Greg, and I
> found there were hundreds of fixes in 3.2.y that are applicable to 3.4.y, and it's
> mainly because Ben has been manually analyzing commits which have stable tags but
> can't be backported automatically while Greg doesn't work like this because of
> time budget.
> 
> The basic principle is if a fix has been backported to an older stable tree, newer
> stable trees probably also need it.
> 
>>  4. Upstream monitoring: I've suggested to Greg that we have a bot looking at
>> commits going upstream, and for every commit marked as for stable it would attempt
>> to apply it to all relevant stable trees and build them, and on failure would
>> notify the author.
>>
> 
> I would never do this for 3.4.y, because for a kernel as old as 3.4 such failure
> is very common.

So this is an interesting subject: right now we have a few stable/LTS trees and each
of us does his own thing (while following a common pattern and discussing things),
including picking patches on his own, sending mails for added/review/released and
dealing with testing/backports (we may share testing resources and look for backports
in other trees, but it's up to each of us individually to get it right).

What if instead of keeping things separated, we try combining our efforts and work
on a single tree? It would mean that people would only get one mail when their patch
was added, reviewed and released, making it easier to follow discussions and manage
patches.

You don't send out mails when things failed to apply on 3.4 because it's common, and
we don't want to flood everyone with this, but what if authors would only get one mail
stating which versions the patch applied successfully to and which it didn't, making the
review process easier?

It would also mean that if I, for example, look at a patch and mark is as not-relevant
for 3.18, you wouldn't be looking at it again to decide if it applies to 3.4, saving
everyone lots of work.


Thanks,
Sasha

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Zefan Li]

> So this is an interesting subject: right now we have a few stable/LTS trees and each
> of us does his own thing (while following a common pattern and discussing things),
> including picking patches on his own, sending mails for added/review/released and
> dealing with testing/backports (we may share testing resources and look for backports
> in other trees, but it's up to each of us individually to get it right).
> 
> What if instead of keeping things separated, we try combining our efforts and work
> on a single tree? It would mean that people would only get one mail when their patch
> was added, reviewed and released, making it easier to follow discussions and manage
> patches.
> 
> You don't send out mails when things failed to apply on 3.4 because it's common, and
> we don't want to flood everyone with this, but what if authors would only get one mail
> stating which versions the patch applied successfully to and which it didn't, making the
> review process easier?
> 
> It would also mean that if I, for example, look at a patch and mark is as not-relevant
> for 3.18, you wouldn't be looking at it again to decide if it applies to 3.4, saving
> everyone lots of work.
> 

I agree it will save us a lot of work if we work together, and that will better ensure
a fix won't be missing from some stable trees.

It helps more for maintainers who work on stable tress that are very close in version.
Take Ben and I as an example, if I already manually backport a commit to 3.4.y, Ben will
get notified. Now Ben can just pick it for 3.2.y, and it probably can be applied cleanly,
thus save his time.

Maybe there should be a workshop during KS for stable tree maintainers to get together
and discuss how to work on how to share each other's work.

Another related issue is we all have our own scripts. If we share a set of common scripts,
then we can benefit from each other's improvements on those scripts. Ben once brought this
up in stable mailing list:

http://www.spinics.net/lists/stable/msg89430.html

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Sasha Levin]

On 07/14/2015 06:08 AM, Zefan Li wrote:
> I agree it will save us a lot of work if we work together, and that will better ensure
> a fix won't be missing from some stable trees.
> 
> It helps more for maintainers who work on stable tress that are very close in version.
> Take Ben and I as an example, if I already manually backport a commit to 3.4.y, Ben will
> get notified. Now Ben can just pick it for 3.2.y, and it probably can be applied cleanly,
> thus save his time.
> 
> Maybe there should be a workshop during KS for stable tree maintainers to get together
> and discuss how to work on how to share each other's work.
> 
> Another related issue is we all have our own scripts. If we share a set of common scripts,
> then we can benefit from each other's improvements on those scripts. Ben once brought this
> up in stable mailing list:
> 
> http://www.spinics.net/lists/stable/msg89430.html

I agree, a workshop could be beneficial in improving the coordination between stable tree
maintainers, and hopefully reducing duplicate work in the future.

Greg, do you have any objections to pursuing this idea?


Thanks,
Sasha

Re: [Ksummit-discuss] [CORE TOPIC] Issues with stable process

[Greg Kroah-Hartman]

On Tue, Jul 14, 2015 at 10:00:18AM -0400, Sasha Levin wrote:
> On 07/14/2015 06:08 AM, Zefan Li wrote:
> > I agree it will save us a lot of work if we work together, and that will better ensure
> > a fix won't be missing from some stable trees.
> > 
> > It helps more for maintainers who work on stable tress that are very close in version.
> > Take Ben and I as an example, if I already manually backport a commit to 3.4.y, Ben will
> > get notified. Now Ben can just pick it for 3.2.y, and it probably can be applied cleanly,
> > thus save his time.
> > 
> > Maybe there should be a workshop during KS for stable tree maintainers to get together
> > and discuss how to work on how to share each other's work.
> > 
> > Another related issue is we all have our own scripts. If we share a set of common scripts,
> > then we can benefit from each other's improvements on those scripts. Ben once brought this
> > up in stable mailing list:
> > 
> > http://www.spinics.net/lists/stable/msg89430.html
> 
> I agree, a workshop could be beneficial in improving the coordination between stable tree
> maintainers, and hopefully reducing duplicate work in the future.
> 
> Greg, do you have any objections to pursuing this idea?

No objection, but I really don't want to have to rewrite my scripts
again :)

As for "duplicate work", I don't see a ton of it happening in that all
of us publish our patches and trees and it's easy to pick from them if
you want to.

But we can talk about it in Seoul, in person would make it easier.

thanks,

greg k-h