KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[James Bottomley]

Before the "3.10.1-stable review" thread degenerated into a disagreement
about habits of politeness, there were some solid points being made
which, I think, bear consideration and which may now be lost.

The problem, as Jiří Kosina put is succinctly is that the distributions
are finding stable less useful because it contains to much stuff they'd
classify as not stable material.

The question that arises from this is who is stable aiming at ...
because if it's the distributions (and that's what people seem to be
using it for) then we need to take this feedback seriously.

The next question is how should we, the maintainers, be policing commits
to stable.  As I think has been demonstrated in the discussion the
"stable rules" are more sort of guidelines (apologies for the pirates
reference).  In many ways, this is as it should be, because people
should have enough taste to know what constitutes a stable fix.  The
real root cause of the problem is that the cc: stable tag can't be
stripped once it's in the tree, so maintainers only get to police things
they put in the tree.  Stuff they pull from others is already tagged and
that tag can't be changed.  This effectively pushes the problem out to
the lowest (and possibly more inexperienced) leaves of the Maintainer
tree.  In theory we have a review stage for stable, but the review
patches don't automatically get routed to the right mailing list and the
first round usually comes out in the merge window when Maintainers'
attention is elsewhere.

The solution, to me, looks simple:  Let's co-opt a process we already
know how to do: mailing list review and tree handling.  So the proposal
is simple:

     1. Drop the cc: stable@ tag: it makes it way too easy to add an ill
        reviewed patch to stable
     2. All patches to stable should follow current review rules: They
        should go to the mailing list the original patch was sent to
        once the original is upstream as a request for stable.
     3. Following debate on the list, the original maintainer would be
        responsible for collecting the patches (including the upstream
        commit) adjudicating on them and passing them on to stable after
        list review (either by git tree pull or email to stable@).

I contend this raises the bar for adding patches to stable much higher,
which seems to be needed, and adds a review stage which involves all the
original reviewers.

Oh, and did someone mention plum brandy ...?

James

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Steven Rostedt]

On Mon, 2013-07-15 at 23:27 +0400, James Bottomley wrote:
> Before the "3.10.1-stable review" thread degenerated into a disagreement
> about habits of politeness, there were some solid points being made
> which, I think, bear consideration and which may now be lost.

Party pooper ;-)

> 
> The problem, as Jiří Kosina put is succinctly is that the distributions
> are finding stable less useful because it contains to much stuff they'd
> classify as not stable material.
> 
> The question that arises from this is who is stable aiming at ...
> because if it's the distributions (and that's what people seem to be
> using it for) then we need to take this feedback seriously.
> 
> The next question is how should we, the maintainers, be policing commits
> to stable.  As I think has been demonstrated in the discussion the
> "stable rules" are more sort of guidelines (apologies for the pirates
> reference).  In many ways, this is as it should be, because people
> should have enough taste to know what constitutes a stable fix.  The
> real root cause of the problem is that the cc: stable tag can't be
> stripped once it's in the tree, so maintainers only get to police things
> they put in the tree.  Stuff they pull from others is already tagged and
> that tag can't be changed.  This effectively pushes the problem out to
> the lowest (and possibly more inexperienced) leaves of the Maintainer
> tree.  In theory we have a review stage for stable, but the review
> patches don't automatically get routed to the right mailing list and the
> first round usually comes out in the merge window when Maintainers'
> attention is elsewhere.
> 
> The solution, to me, looks simple:  Let's co-opt a process we already
> know how to do: mailing list review and tree handling.  So the proposal
> is simple:
> 
>      1. Drop the cc: stable@ tag: it makes it way too easy to add an ill
>         reviewed patch to stable
>      2. All patches to stable should follow current review rules: They
>         should go to the mailing list the original patch was sent to
>         once the original is upstream as a request for stable.
>      3. Following debate on the list, the original maintainer would be
>         responsible for collecting the patches (including the upstream
>         commit) adjudicating on them and passing them on to stable after
>         list review (either by git tree pull or email to stable@).
> 
> I contend this raises the bar for adding patches to stable much higher,
> which seems to be needed, and adds a review stage which involves all the
> original reviewers.

How about this as a proposal.

Keep the Cc: stable@ tag as it is today.

Have Greg, or whoever, change his script to not take commits marked for
stable, but instead, forward the commit to the maintainer. Or as it
already does today, to everyone on the Cc, and -by: tags. Change the
script from being "stable review" to say "Look to see what you want, and
resend what is needed for stable".

Then the maintainer needs to look at all the patches that were marked
for stable, and resend them to Greg saying which ones should go into
stable.

This has a few advantages.

1) lets anyone still mark their patch as a stable fix, even if at the
end isn't.

2) Some commits can be marked for stable, for those that want to read
git logs, but not send to stable (the Fixes: tag as described in another
thread).

2) Makes the stable decision go where it belongs. To the maintainer of
that system. It lets the maintainer think twice about what goes to
stable or not.

3) gives a little more time if needed, to let stable commits stir in
mainline. If its a security fix, the maintainer should be on the ball to
get in to stable, and not just let Greg worry about it.

4) Probably the most important. It should unload most of the work off of
Greg, and push it more on the maintainers. Greg would just need to make
sure the patch goes to the right person (probably not an issue, as it is
most likely on the Signed off by line). Then the maintainers could
review the commits that should go to stable, and if needed, not send
them. They could also send commits that were not marked for stable, as
long as they are already in mainline.

Also, we could mandate that the maintainers do the backports too.

> 
> Oh, and did someone mention plum brandy ...?

No, but I was looking forward to cookies.

-- Steve

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Willy Tarreau]

Hi Steven,

On Mon, Jul 15, 2013 at 03:45:17PM -0400, Steven Rostedt wrote:
> How about this as a proposal.
> 
> Keep the Cc: stable@ tag as it is today.
> 
> Have Greg, or whoever, change his script to not take commits marked for
> stable, but instead, forward the commit to the maintainer. Or as it
> already does today, to everyone on the Cc, and -by: tags. Change the
> script from being "stable review" to say "Look to see what you want, and
> resend what is needed for stable".
> 
> Then the maintainer needs to look at all the patches that were marked
> for stable, and resend them to Greg saying which ones should go into
> stable.
> 
> This has a few advantages.
> 
> 1) lets anyone still mark their patch as a stable fix, even if at the
> end isn't.
> 
> 2) Some commits can be marked for stable, for those that want to read
> git logs, but not send to stable (the Fixes: tag as described in another
> thread).
> 
> 2) Makes the stable decision go where it belongs. To the maintainer of
> that system. It lets the maintainer think twice about what goes to
> stable or not.
> 
> 3) gives a little more time if needed, to let stable commits stir in
> mainline. If its a security fix, the maintainer should be on the ball to
> get in to stable, and not just let Greg worry about it.
> 
> 4) Probably the most important. It should unload most of the work off of
> Greg, and push it more on the maintainers. Greg would just need to make
> sure the patch goes to the right person (probably not an issue, as it is
> most likely on the Signed off by line). Then the maintainers could
> review the commits that should go to stable, and if needed, not send
> them. They could also send commits that were not marked for stable, as
> long as they are already in mainline.
> 
> Also, we could mandate that the maintainers do the backports too.

I disagree with your proposal. All these points are already covered by
the stable review and the early notification that the greg-bot does when
the patch is included in the queue. If submitters/maintainers do not read
these e-mails sent to them about changes going to happen in the subsystems
they're responsible for, having them resend the patches will only punish
the honnest ones. The other ones will simply reply without doing anything
else. In the end, we get more work to get fixes merged so less efficiency.

I tend to think the merge should be slightly less automatic or at least
add some delay (post release for late submissions), but what I'm worried
about is that adding exceptions to the workflow will mean more work for
Greg, so we should be careful about this, as we have a single one and we
don't want to burn him out.

Regards,
Willy

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Steven Rostedt]

On Mon, 2013-07-15 at 21:55 +0200, Willy Tarreau wrote:

> I disagree with your proposal. All these points are already covered by
> the stable review and the early notification that the greg-bot does when
> the patch is included in the queue. If submitters/maintainers do not read
> these e-mails sent to them about changes going to happen in the subsystems
> they're responsible for, having them resend the patches will only punish
> the honnest ones. The other ones will simply reply without doing anything
> else. In the end, we get more work to get fixes merged so less efficiency.
> 
> I tend to think the merge should be slightly less automatic or at least
> add some delay (post release for late submissions), but what I'm worried
> about is that adding exceptions to the workflow will mean more work for
> Greg, so we should be careful about this, as we have a single one and we
> don't want to burn him out.
> 

I'm temporarily maintaining a 3.6 stable release (can't wait till I
don't have to do that anymore). And I cheat. I use the trees that Greg
uses, and I still spend days getting it ready.

I think the current method does not scale. It's only been doing so well
only because Greg has been putting a lot of time and effort into it. But
I still think the process is broken.

Do I think this will add more work to the maintainer? Yes, definitely!
But we have hundreds of maintainers, and only one Greg. Where do you
think we should be adding the work too?

In another KS topic, we talked about backup maintainers. This could be
the job of #2.

Yes, there's already a automatic response, but who really looks a those.
I know I'm guilty of seeing that and saying to myself "oh good, Greg
added that patch" and not actually review it. This process may force me
to look at it better.

It may not be efficient for maintainers, but as maintainers we should
spend a bit more time on stable releases. If you do that up front before
marking commits with the stable tag, then just setup a mail filter that
simply forwards the email to the second address that Greg will take. If
you abuse that, then Greg can get nasty with you ;-)

-- Steve

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Joe Perches]

On Mon, 2013-07-15 at 16:56 -0400, Steven Rostedt wrote:
> It may not be efficient for maintainers, but as maintainers we should
> spend a bit more time on stable releases.

The MAINTAINERS file specifies a difference between a
section that's Maintained vs Supported.

Do please remember there's a difference between a maintainer
and someone that's paid to support a particular thing.

My feeling is if you're a maintainer you can just say
"use the current version" and not spend even a moments
effort or thought on backports.

When you're getting paid however, different issues...

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Steven Rostedt]

On Mon, 2013-07-15 at 14:09 -0700, Joe Perches wrote:
> On Mon, 2013-07-15 at 16:56 -0400, Steven Rostedt wrote:
> > It may not be efficient for maintainers, but as maintainers we should
> > spend a bit more time on stable releases.
> 
> The MAINTAINERS file specifies a difference between a
> section that's Maintained vs Supported.

I never noticed this before. Hmm, Red Hat pays me to work on real-time,
but lets me also work on ftrace. I wounder if I should switch that to
"Supported", although I push tracing support behind having to do
real-time, as that's really my job.

> 
> Do please remember there's a difference between a maintainer
> and someone that's paid to support a particular thing.
> 
> My feeling is if you're a maintainer you can just say
> "use the current version" and not spend even a moments
> effort or thought on backports.

If you're doing this as a volunteer, then you have every right to say
such a thing. But I have to ask. How many maintainers are really just
volunteers? And of those, how many have critical parts of the kernel
that distros rely on. If they do, then one would think that a distro
would hire them.

> 
> When you're getting paid however, different issues...
> 

Yep.

-- Steve

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Joe Perches]

On Mon, 2013-07-15 at 17:21 -0400, Steven Rostedt wrote:
> How many maintainers are really just volunteers?

No idea.  Here's a data point.

$ git grep "^S:" MAINTAINERS|sed -r 's/\s+/ /g'|sort|uniq -c|sort -rn
    818 MAINTAINERS:S: Maintained
    248 MAINTAINERS:S: Supported
     49 MAINTAINERS:S: Odd Fixes
     32 MAINTAINERS:S: Orphan
     10 MAINTAINERS:S: Odd fixes
      2 MAINTAINERS:S: Unmaintained
      1 MAINTAINERS:S: Orphan / Obsolete
      1 MAINTAINERS:S: Odd Fixes (e.g., new signatures)
      1 MAINTAINERS:S: Obsolete
      1 MAINTAINERS:S: Maintained for 2.6.
      1 MAINTAINERS:S: Maintained:
      1 MAINTAINERS:S: Buried alive in reporters

I will note that most MAINTAINER section entries
don't actually have people actively shepherding
patches.  There's no aging out of inactive entries.

> And of those, how many have critical parts of the kernel
> that distros rely on.

No idea.  I'd guess most distributions don't go
out of their way to hire folk when "good enough"
seems to be working.

> If they do, then one would think that a distro
> would hire them.

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Rob Landley]

On 07/15/2013 04:09:53 PM, Joe Perches wrote:
> On Mon, 2013-07-15 at 16:56 -0400, Steven Rostedt wrote:
> > It may not be efficient for maintainers, but as maintainers we  
> should
> > spend a bit more time on stable releases.
> 
> The MAINTAINERS file specifies a difference between a
> section that's Maintained vs Supported.
> 
> Do please remember there's a difference between a maintainer
> and someone that's paid to support a particular thing.
> 
> My feeling is if you're a maintainer you can just say
> "use the current version" and not spend even a moments
> effort or thought on backports.
> 
> When you're getting paid however, different issues...

Then you get to say "you're not a customer of my company, I don't have  
to listen to you at all". (See Renesas with sh4 circa 2010.)

Rob

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Willy Tarreau]

On Mon, Jul 15, 2013 at 04:56:19PM -0400, Steven Rostedt wrote:
> I'm temporarily maintaining a 3.6 stable release (can't wait till I
> don't have to do that anymore). And I cheat. I use the trees that Greg
> uses, and I still spend days getting it ready.

I've been doing the same for a long time (still doing it for 2.6.32).
I couldn't do it without Greg's selection of patches and help, quite
frankly. I still don't know how he manages to support that many trees
in parallel with that regular frequency. Maybe he's more resistant to
effort than us.

> I think the current method does not scale. It's only been doing so well
> only because Greg has been putting a lot of time and effort into it. But
> I still think the process is broken.

In my opinion, it's not broken, but it's fragile.

> Do I think this will add more work to the maintainer? Yes, definitely!
> But we have hundreds of maintainers, and only one Greg. Where do you
> think we should be adding the work too?

I agree on this point, except that maintainers exchange with Greg, and
more exchanges inevitably means more work for Greg.

> In another KS topic, we talked about backup maintainers. This could be
> the job of #2.

Backup is for an active-backup setup. You're suggesting an active-active
one, which can be quite different, even maybe in terms of funding.

> Yes, there's already a automatic response, but who really looks a those.

I think more than you imagine on average. And anyway, when some patches
break and require a fix that was known at the time of the review, we
should shout (sorry Sarah) on the authors for not mentionning it during
the review. A review is a responsible process. You're giving your consent
for some of your work being merged in kernels that will be deployed in
zillions of devices. It's not just a formality.

> I know I'm guilty of seeing that and saying to myself "oh good, Greg
> added that patch" and not actually review it. This process may force me
> to look at it better.

Then better save a mail exchange and start reviewing them with more
attention. Maybe Greg should let more time for reviews. For example I
have learned that old kernels are of less importance to maintainers
(which is expected) and if some want to perform a deep review, they
need some time to set up an environment (we all have dirty things in
our trees and don't want to checkout -f to an old code just for a
review, do we?). So I increased from 3 days to one week lately and
it increased the feedback. I don't know if current maintainers already
get bored by reviews, but I think this is the point which needs to be
reinforced if they do.

> It may not be efficient for maintainers, but as maintainers we should
> spend a bit more time on stable releases. If you do that up front before
> marking commits with the stable tag, then just setup a mail filter that
> simply forwards the email to the second address that Greg will take. If
> you abuse that, then Greg can get nasty with you ;-)

Then let's save one step and remind people that by putting a Cc: stable tag,
you agree to take care of reviewing what you have sent and to notify about
known missing fixes or any pending issues which require the patch to be
postponed.

That's probably where the Cc:stable unveils some laziness we discussed
about with Greg. It's too easy to add "Cc:stable" and consider that it's
someone else's job as of now to backport and check.

If you're not willing to review fixes sent by you, please do not Cc
stable in your patches, that's really better for the process. (I'm not
saying that to you specifically, but as a rule to put in the doc).

Best regards,
Willy

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1146 bytes --]

On Mon, Jul 15, 2013 at 03:45:17PM -0400, Steven Rostedt wrote:

> Have Greg, or whoever, change his script to not take commits marked for
> stable, but instead, forward the commit to the maintainer. Or as it
> already does today, to everyone on the Cc, and -by: tags. Change the
> script from being "stable review" to say "Look to see what you want, and
> resend what is needed for stable".

This sounds fairly close what James was suggesting, though with central
automation of the mail it out stage.

> Then the maintainer needs to look at all the patches that were marked
> for stable, and resend them to Greg saying which ones should go into
> stable.

One thing I don't particularly like about this is having to resend the
patches in response to mail; it seems cumbersome to do that rather than
reply to mail or something.  Requiring a positive acknowledgement or
action seems useful but the particular one seems a bit annoying; I'd
rather either just send the patch as part of the mark for stable
workflow or ack something in mail.

> Also, we could mandate that the maintainers do the backports too.

That's what happens already isn't it?

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Steven Rostedt]

On Mon, 2013-07-15 at 21:15 +0100, Mark Brown wrote:

> One thing I don't particularly like about this is having to resend the
> patches in response to mail; it seems cumbersome to do that rather than
> reply to mail or something.  Requiring a positive acknowledgement or
> action seems useful but the particular one seems a bit annoying; I'd
> rather either just send the patch as part of the mark for stable
> workflow or ack something in mail.

A reply to the email before it goes in would work as well. I'm just
saying that the stable tag alone should not be the criteria of what goes
into stable.

> 
> > Also, we could mandate that the maintainers do the backports too.
> 
> That's what happens already isn't it?

Only if it breaks. But I've been quite impressed at a lot of the patches
that Greg seems to get working himself. He gets things backported that
don't look to be automated. I guess if it's trivial enough he does the
work himself. But I've had a few commits where I thought for sure I
would get a slew of "FAILED to apply" stable messages that ended up
going in without my help.

-- Steve

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Guenter Roeck]

On Mon, Jul 15, 2013 at 11:27:56PM +0400, James Bottomley wrote:
[ ... ]
> 
> The solution, to me, looks simple:  Let's co-opt a process we already
> know how to do: mailing list review and tree handling.  So the proposal
> is simple:
> 
>      1. Drop the cc: stable@ tag: it makes it way too easy to add an ill
>         reviewed patch to stable

That seems to be a bit drastic. It is quite useful to have the tag,
but maybe it should only be added by the maintainer and not in the initial
patch submission. This would ensure that the maintainer(s) made the decision.
If the original patch submitter thinks that the patch is stable material,
that information could be added in the comments section.

This would also ensure that patches are not accidentially sent to the stable
mailing list prior to being accepted into the upstream kernel, and if necessary
triggers a discussion on the subject mailing list if the patch is stable
material or not - which addresses points 2 and 3 below.

Guenter

>      2. All patches to stable should follow current review rules: They
>         should go to the mailing list the original patch was sent to
>         once the original is upstream as a request for stable.
>      3. Following debate on the list, the original maintainer would be
>         responsible for collecting the patches (including the upstream
>         commit) adjudicating on them and passing them on to stable after
>         list review (either by git tree pull or email to stable@).
> 
> I contend this raises the bar for adding patches to stable much higher,
> which seems to be needed, and adds a review stage which involves all the
> original reviewers.
> 
> Oh, and did someone mention plum brandy ...?
> 
> James
> 
> 
> _______________________________________________
> Ksummit-2013-discuss mailing list
> Ksummit-2013-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-2013-discuss

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 549 bytes --]

On Mon, 2013-07-15 at 13:19 -0700, Guenter Roeck wrote:
> That seems to be a bit drastic. It is quite useful to have the tag,
> but maybe it should only be added by the maintainer and not in the initial
> patch submission. This would ensure that the maintainer(s) made the decision.
> If the original patch submitter thinks that the patch is stable material,
> that information could be added in the comments section.

In the case where a maintainer applies a patch with 'git am', surely
they can *see* that it's cc:stable?

-- 
dwmw2


[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5745 bytes --]

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Guenter Roeck]

On Mon, Jul 15, 2013 at 11:04:28PM +0100, David Woodhouse wrote:
> On Mon, 2013-07-15 at 13:19 -0700, Guenter Roeck wrote:
> > That seems to be a bit drastic. It is quite useful to have the tag,
> > but maybe it should only be added by the maintainer and not in the initial
> > patch submission. This would ensure that the maintainer(s) made the decision.
> > If the original patch submitter thinks that the patch is stable material,
> > that information could be added in the comments section.
> 
> In the case where a maintainer applies a patch with 'git am', surely
> they can *see* that it's cc:stable?
> 
If that maintainer is careful, yes. But that isn't the point or idea. The
difference is that the maintainer would have to make an active decision
to add the cc:stable tag vs. just going along with it.

Guenter

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[H. Peter Anvin]

On 07/15/2013 03:07 PM, Guenter Roeck wrote:
> On Mon, Jul 15, 2013 at 11:04:28PM +0100, David Woodhouse wrote:
>> On Mon, 2013-07-15 at 13:19 -0700, Guenter Roeck wrote:
>>> That seems to be a bit drastic. It is quite useful to have the tag,
>>> but maybe it should only be added by the maintainer and not in the initial
>>> patch submission. This would ensure that the maintainer(s) made the decision.
>>> If the original patch submitter thinks that the patch is stable material,
>>> that information could be added in the comments section.
>>
>> In the case where a maintainer applies a patch with 'git am', surely
>> they can *see* that it's cc:stable?
>>
> If that maintainer is careful, yes. But that isn't the point or idea. The
> difference is that the maintainer would have to make an active decision
> to add the cc:stable tag vs. just going along with it.
> 

WTF?  If a maintainer applies a patch and misses that the thing had a
Cc: <stable> tag, that maintainer should never have applied the patch in
the first place.

It gets stickier in the case of submaintainers where there are git pulls
involved.

	-hpa

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Guenter Roeck]

On Mon, Jul 15, 2013 at 03:38:08PM -0700, H. Peter Anvin wrote:
> On 07/15/2013 03:07 PM, Guenter Roeck wrote:
> > On Mon, Jul 15, 2013 at 11:04:28PM +0100, David Woodhouse wrote:
> >> On Mon, 2013-07-15 at 13:19 -0700, Guenter Roeck wrote:
> >>> That seems to be a bit drastic. It is quite useful to have the tag,
> >>> but maybe it should only be added by the maintainer and not in the initial
> >>> patch submission. This would ensure that the maintainer(s) made the decision.
> >>> If the original patch submitter thinks that the patch is stable material,
> >>> that information could be added in the comments section.
> >>
> >> In the case where a maintainer applies a patch with 'git am', surely
> >> they can *see* that it's cc:stable?
> >>
> > If that maintainer is careful, yes. But that isn't the point or idea. The
> > difference is that the maintainer would have to make an active decision
> > to add the cc:stable tag vs. just going along with it.
> > 
> 
> WTF?  If a maintainer applies a patch and misses that the thing had a
> Cc: <stable> tag, that maintainer should never have applied the patch in
> the first place.
> 

I agree, _should_. But again, that is not the point I was trying to make.
The keyword is _active_ decision vs. passive acceptance of a stable tag.

If the stable tag is not added by the maintainer, it can always be added to
the stable queue after the code was pushed upstream. Nothing lost but a bit
of convenience.

Guenter

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[H. Peter Anvin]

On 07/15/2013 04:22 PM, Guenter Roeck wrote:
> 
> I agree, _should_. But again, that is not the point I was trying to make.
> The keyword is _active_ decision vs. passive acceptance of a stable tag.
> 
> If the stable tag is not added by the maintainer, it can always be added to
> the stable queue after the code was pushed upstream. Nothing lost but a bit
> of convenience.
> 

... and yet another opportunity for things to fall between the cracks,
which is in my opinion MUCH more likely than something inappropriate
being tagged Cc: stable.

However, it doesn't seem to happen too often, but it does underscore the
need for a maintainer to be able to *retroactively* NAK a patch for
stable, if it is uncovered that it isn't appropriate after all.

	-hpa

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Greg KH]

On Mon, Jul 15, 2013 at 05:13:42PM -0700, H. Peter Anvin wrote:
> On 07/15/2013 04:22 PM, Guenter Roeck wrote:
> > 
> > I agree, _should_. But again, that is not the point I was trying to make.
> > The keyword is _active_ decision vs. passive acceptance of a stable tag.
> > 
> > If the stable tag is not added by the maintainer, it can always be added to
> > the stable queue after the code was pushed upstream. Nothing lost but a bit
> > of convenience.
> > 
> 
> ... and yet another opportunity for things to fall between the cracks,
> which is in my opinion MUCH more likely than something inappropriate
> being tagged Cc: stable.
> 
> However, it doesn't seem to happen too often, but it does underscore the
> need for a maintainer to be able to *retroactively* NAK a patch for
> stable, if it is uncovered that it isn't appropriate after all.

I give maintainers 2 different chances to NAK a patch, and if they miss
those, I can also easily revert a patch that got applied and do a new
release, which I have done in the past.

greg k-h

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[H. Peter Anvin]

On 07/15/2013 05:21 PM, Greg KH wrote:
>>
>> However, it doesn't seem to happen too often, but it does underscore the
>> need for a maintainer to be able to *retroactively* NAK a patch for
>> stable, if it is uncovered that it isn't appropriate after all.
> 
> I give maintainers 2 different chances to NAK a patch, and if they miss
> those, I can also easily revert a patch that got applied and do a new
> release, which I have done in the past.
> 

Yes, it doesn't actually seem to be a problem in practice.

In other words, the current system seems to work well, and unless
someone wants to show cases where it doesn't work I don't see a reason
to switch it...

	-hpa

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Paul Gortmaker]

On 13-07-15 08:25 PM, H. Peter Anvin wrote:
> On 07/15/2013 05:21 PM, Greg KH wrote:
>>>
>>> However, it doesn't seem to happen too often, but it does underscore the
>>> need for a maintainer to be able to *retroactively* NAK a patch for
>>> stable, if it is uncovered that it isn't appropriate after all.
>>
>> I give maintainers 2 different chances to NAK a patch, and if they miss
>> those, I can also easily revert a patch that got applied and do a new
>> release, which I have done in the past.
>>
> 
> Yes, it doesn't actually seem to be a problem in practice.
> 
> In other words, the current system seems to work well, and unless
> someone wants to show cases where it doesn't work I don't see a reason
> to switch it...

I'd agree with that; I recently found something in 3.4.x that wasn't
quite right and Greg reverted it without hesitation.  I think that
instance could have been observed earlier in a semi-automated fashion
by adding a check to ensure the patched function(s) in the cherry
pick match the patched function in the original.  I had a hacked up
script to do this for my 2.6.34.x longterm commits:

http://git.kernel.org/cgit/linux/kernel/git/paulg/longterm-queue-2.6.34.git/tree/scripts/reviewbot

It might be worthwhile having some kind of similar thing looking
at the stable-queue.git content as it flows in?

Some additional possible points for discussion:

The risk of an incorrectly applied (or simply inappropriate use
of) a patch goes up exponentially with the size of the gap
between mainline where it appeared and the baseline of the
older target stable tree.

Meaning that commit in 3.10 will most likely seamlessly apply
to 3.9, but perhaps not so easy to 3.4, and it may not even
be appropriate for 3.0 at all.  Those are the worst ones to
detect; the commits which will happily "git am" but simply are
not appropriate because the baseline never had the bug/issue.

The patch creator and/or the maintainer are most likely the
best people who can comment on what versions a commit is
applicable to -- we have that now as a pseudo comment on
the Cc: stable line -- encouraging more active use of it
might be worthwhile?

Anyway, the effort for maintaining the older/longterm trees is
generally going to be more than for the "current minus one"
tree.  And if an increase in the strictness of what goes into
stable is made, it would have the most effect in terms of
making a stable maintainer's life easier there.

I guess what I'm driving at here, is that there is the option
to be more strict with the older releases vs. the "current-1"
release, and I wouldn't be surprised if the commit counts on
the various active stable branches reflect that Greg is already
implicitly doing this to some degree.

Maybe that opens discussion on longterm releases, such as how
many there should be, what they should contain, how long they
should live and so on?  To some degree, the answers to those
questions are largely at the mercy of the effort the maintainer
is willing to put in, but at the same time, having a certain
level of consistency and a uniform message to present to the
world about the different releases is good as well.

Paul.
--

> 
> 	-hpa
> 
> 
> _______________________________________________
> Ksummit-2013-discuss mailing list
> Ksummit-2013-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-2013-discuss
> 
> 

Re: KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Jason Cooper]

On Mon, Jul 15, 2013 at 11:27:56PM +0400, James Bottomley wrote:
...
> The solution, to me, looks simple:  Let's co-opt a process we already
> know how to do: mailing list review and tree handling.  So the proposal
> is simple:
> 
>      1. Drop the cc: stable@ tag: it makes it way too easy to add an ill
>         reviewed patch to stable
>      2. All patches to stable should follow current review rules: They
>         should go to the mailing list the original patch was sent to
>         once the original is upstream as a request for stable.
>      3. Following debate on the list, the original maintainer would be
>         responsible for collecting the patches (including the upstream
>         commit) adjudicating on them and passing them on to stable after
>         list review (either by git tree pull or email to stable@).
> 
> I contend this raises the bar for adding patches to stable much higher,
> which seems to be needed, and adds a review stage which involves all the
> original reviewers.

If I may expand on the 'tree-handling' with an idea:

When I'm collecting up patches for mvebu, the first thing I do is
classify the patch as a fix for the current -rc cycle, or for next.  It
would be very easy to maintain a stable branch "ahead of" fixes.  More
accurately, one for each stable version.  eg:

stable-3.2
stable-3.4
stable-3.9
stable-3.10
fixes
cleanup (or next)
...

Each branch would need to be based on a commit in the mainline tree, eg
stable-3.2 would be based on v3.2, not v3.2.48.

Anything I suspect is stable material goes in the earliest relevant
stable branch, and I send an email series to the linux-stable list.
Discussion ensues, maybe I remove one or two patches from the stable
branch and only put them in fixes (so, for -rcX), and send the PR.  The
stable team can then merge the PR, once they see the commit in mainline.

The big advantage of this proposal is that the patch would have the same
commit id in both trees.  'git branch -a --contains <commit-ish>' would
work.

The main problem with this is that maintainers would need to do the
legwork themselves to figure out how far back a patch is relevant for.
Maybe this would be a feature? ;-)

Unfortunately, since most of the time there's just a single patch, I'm
not sure the cost of the merge commits is worth the benefit of having
the same commit id in stable and mainline.

thx,

Jason.

Re: KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Greg KH]

On Mon, Jul 15, 2013 at 11:27:56PM +0400, James Bottomley wrote:
> Before the "3.10.1-stable review" thread degenerated into a disagreement
> about habits of politeness, there were some solid points being made
> which, I think, bear consideration and which may now be lost.
> 
> The problem, as Ji???? Kosina put is succinctly is that the distributions
> are finding stable less useful because it contains to much stuff they'd
> classify as not stable material.
> 
> The question that arises from this is who is stable aiming at ...
> because if it's the distributions (and that's what people seem to be
> using it for) then we need to take this feedback seriously.
> 
> The next question is how should we, the maintainers, be policing commits
> to stable.  As I think has been demonstrated in the discussion the
> "stable rules" are more sort of guidelines (apologies for the pirates
> reference).  In many ways, this is as it should be, because people
> should have enough taste to know what constitutes a stable fix.  The
> real root cause of the problem is that the cc: stable tag can't be
> stripped once it's in the tree, so maintainers only get to police things
> they put in the tree.  Stuff they pull from others is already tagged and
> that tag can't be changed.  This effectively pushes the problem out to
> the lowest (and possibly more inexperienced) leaves of the Maintainer
> tree.  In theory we have a review stage for stable, but the review
> patches don't automatically get routed to the right mailing list and the
> first round usually comes out in the merge window when Maintainers'
> attention is elsewhere.
> 
> The solution, to me, looks simple:  Let's co-opt a process we already
> know how to do: mailing list review and tree handling.  So the proposal
> is simple:
> 
>      1. Drop the cc: stable@ tag: it makes it way too easy to add an ill
>         reviewed patch to stable
>      2. All patches to stable should follow current review rules: They
>         should go to the mailing list the original patch was sent to
>         once the original is upstream as a request for stable.
>      3. Following debate on the list, the original maintainer would be
>         responsible for collecting the patches (including the upstream
>         commit) adjudicating on them and passing them on to stable after
>         list review (either by git tree pull or email to stable@).
> 
> I contend this raises the bar for adding patches to stable much higher,
> which seems to be needed, and adds a review stage which involves all the
> original reviewers.

I don't like this at all, just for the simple reason that it will push
the majority of the work of stable kernel development on to the
subsystem maintainers, who have enough work to do as it is.

Stable tree stuff should cause almost _no_ extra burden on the kernel
developers, because it is something that I, and a few other people, have
agreed to do with our time.  It has taken me 8 _years_ to finally get
maintainers to agree to mark stuff for the stable tree, and fine-tune a
development process that makes it easy for us to do this backport work.

I _want_ the exact same commit that is in Linus's tree for the backport
because if I have to rely on a maintainer to do the backport and resend
it, I _know_ it will usually be a changed patch, and the git commit id
will be lost.

I know this because it happens today.  I get patches from maintainers
that are radically changed from what is in Linus's tree without any
justification for why that is.  That's not ok.

Let me work with the distros on the issues they have raised.  So far I
have the following issues they have complained about:
  - patches that shouldn't be there because they don't really do
    anything.
  - patches that aren't obvious why they are there.

The first one I am going to be pushing back on, and have already, much
to the dismay of some subsystem maintainers.

The second one is almost always due to security issues that were unknown
to the distro.  The announcement of security problems to the distros has
now been addressed, and since that has changed, I haven't heard any
problems about this.

Have I missed anything else that the distros are objecting to?  The
"smaller" distros (i.e. ones without lots of kernel developers) have
been giving me nothing but _thanks_ and appreciation with the way that
I've been sucking in all of the different fixes.  Do we want to mess
with a process that is really working out well for them, and only causes
annoyance at times by the larger ones?

thanks,

greg k-h

Re: KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Greg KH]

On Mon, Jul 15, 2013 at 02:44:22PM -0700, Greg KH wrote:
> On Mon, Jul 15, 2013 at 11:27:56PM +0400, James Bottomley wrote:
> > Before the "3.10.1-stable review" thread degenerated into a disagreement
> > about habits of politeness, there were some solid points being made
> > which, I think, bear consideration and which may now be lost.
> > 
> > The problem, as Ji???? Kosina put is succinctly is that the distributions
> > are finding stable less useful because it contains to much stuff they'd
> > classify as not stable material.
> > 
> > The question that arises from this is who is stable aiming at ...
> > because if it's the distributions (and that's what people seem to be
> > using it for) then we need to take this feedback seriously.
> > 
> > The next question is how should we, the maintainers, be policing commits
> > to stable.  As I think has been demonstrated in the discussion the
> > "stable rules" are more sort of guidelines (apologies for the pirates
> > reference).  In many ways, this is as it should be, because people
> > should have enough taste to know what constitutes a stable fix.  The
> > real root cause of the problem is that the cc: stable tag can't be
> > stripped once it's in the tree, so maintainers only get to police things
> > they put in the tree.  Stuff they pull from others is already tagged and
> > that tag can't be changed.  This effectively pushes the problem out to
> > the lowest (and possibly more inexperienced) leaves of the Maintainer
> > tree.  In theory we have a review stage for stable, but the review
> > patches don't automatically get routed to the right mailing list and the
> > first round usually comes out in the merge window when Maintainers'
> > attention is elsewhere.
> > 
> > The solution, to me, looks simple:  Let's co-opt a process we already
> > know how to do: mailing list review and tree handling.  So the proposal
> > is simple:
> > 
> >      1. Drop the cc: stable@ tag: it makes it way too easy to add an ill
> >         reviewed patch to stable
> >      2. All patches to stable should follow current review rules: They
> >         should go to the mailing list the original patch was sent to
> >         once the original is upstream as a request for stable.
> >      3. Following debate on the list, the original maintainer would be
> >         responsible for collecting the patches (including the upstream
> >         commit) adjudicating on them and passing them on to stable after
> >         list review (either by git tree pull or email to stable@).
> > 
> > I contend this raises the bar for adding patches to stable much higher,
> > which seems to be needed, and adds a review stage which involves all the
> > original reviewers.
> 
> I don't like this at all, just for the simple reason that it will push
> the majority of the work of stable kernel development on to the
> subsystem maintainers, who have enough work to do as it is.

Oh, and one other thing, some subsystem maintainers do do exactly this,
and I don't object to it at all.

Specifically David Miller batches up stable kernel patches, tests them,
and then resends them to us for application.  He doesn't mark anything
with Cc: stable, so I don't automatically pull them in, and it's been
working really well for many years now.

There are a few other "smaller" subsystems also doing this, one even
going so far as to give me git trees to pull from, as it works out
better for their workflow, and that's fine.

One major subsystem told me to flat-out-refuse any stable patches that
get marked for it, as they wanted to do the work to verify them all
beforehand and resend them to me.  In the end, the work got to be too
much, and they gave up, but they sure tried :)

So if you want to change the way you, as a subsystem maintainer have the
end-say as to how I will apply stable patches for your area of the
kernel, please do so.  The Cc: stable stuff evolved over time to make it
_easy_ for maintainers to get patches to me with the least amount of
work.  But if you want to do more work on your end, sure, I'll be glad
to let you do that...

thanks,

greg k-h

Re: KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[H. Peter Anvin]

Perhaps the KS topic should be about different stable workflows and what
the maintainers' options are, rather than about a specific proposal.
This seems like a good discussion topic.

	-hpa

Re: KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Greg KH]

On Mon, Jul 15, 2013 at 03:01:18PM -0700, H. Peter Anvin wrote:
> Perhaps the KS topic should be about different stable workflows and what
> the maintainers' options are, rather than about a specific proposal.
> This seems like a good discussion topic.

I agree, that sounds good to me.

greg k-h

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Rafael J. Wysocki]

On Monday, July 15, 2013 04:08:25 PM Greg KH wrote:
> On Mon, Jul 15, 2013 at 03:01:18PM -0700, H. Peter Anvin wrote:
> > Perhaps the KS topic should be about different stable workflows and what
> > the maintainers' options are, rather than about a specific proposal.
> > This seems like a good discussion topic.
> 
> I agree, that sounds good to me.

+1

Rafael

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Jiri Kosina]

On Mon, 15 Jul 2013, H. Peter Anvin wrote:

> Perhaps the KS topic should be about different stable workflows and what
> the maintainers' options are, rather than about a specific proposal.
> This seems like a good discussion topic.

I agree as well; I believe all the proposals related to -stable can be 
folded into (this) one.

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Steven Rostedt]

On Mon, 2013-07-15 at 14:44 -0700, Greg KH wrote:

> I don't like this at all, just for the simple reason that it will push
> the majority of the work of stable kernel development on to the
> subsystem maintainers, who have enough work to do as it is.
> 
> Stable tree stuff should cause almost _no_ extra burden on the kernel
> developers, because it is something that I, and a few other people, have
> agreed to do with our time.  It has taken me 8 _years_ to finally get
> maintainers to agree to mark stuff for the stable tree, and fine-tune a
> development process that makes it easy for us to do this backport work.

Although, since those 8 years, the stable tree has proven its
importance.

Is a extra "ack" also too much to ask?

> 
> I _want_ the exact same commit that is in Linus's tree for the backport
> because if I have to rely on a maintainer to do the backport and resend
> it, I _know_ it will usually be a changed patch, and the git commit id
> will be lost.

This is a good point and a valid concern.


> 
> I know this because it happens today.  I get patches from maintainers
> that are radically changed from what is in Linus's tree without any
> justification for why that is.  That's not ok.
> 
> Let me work with the distros on the issues they have raised.  So far I
> have the following issues they have complained about:
>   - patches that shouldn't be there because they don't really do
>     anything.
>   - patches that aren't obvious why they are there.
> 
> The first one I am going to be pushing back on, and have already, much
> to the dismay of some subsystem maintainers.
> 
> The second one is almost always due to security issues that were unknown
> to the distro.  The announcement of security problems to the distros has
> now been addressed, and since that has changed, I haven't heard any
> problems about this.
> 
> Have I missed anything else that the distros are objecting to?  The
> "smaller" distros (i.e. ones without lots of kernel developers) have
> been giving me nothing but _thanks_ and appreciation with the way that
> I've been sucking in all of the different fixes.  Do we want to mess

This is proof of how important stable has been.

> with a process that is really working out well for them, and only causes
> annoyance at times by the larger ones?


The big problem with the above is that the process depends highly on
this guy named "Greg".

If "Greg" gets tired of doing this, or gets sick, or "see other KS topic
about mortality of maintainers", then the entire process fails.

OK, you are not the only one that does the stable release. There's Ben
and Luis doing it as well, and various others. But there seems to be a
bit of an issue here. How much should go into stable? Who really
decides?

How important is the stable releases? Are maintainers willing to do a
little more work now to make sure their subsystems work fine in older
kernels? This isn't the same stable as it was 8 years ago.

I guess this is a great topic for KS, as it seems to be going all over
the place via email, and would probably be much better sorted out face
to face.

-- Steve

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Greg KH]

On Mon, Jul 15, 2013 at 06:01:39PM -0400, Steven Rostedt wrote:
> On Mon, 2013-07-15 at 14:44 -0700, Greg KH wrote:
> 
> > I don't like this at all, just for the simple reason that it will push
> > the majority of the work of stable kernel development on to the
> > subsystem maintainers, who have enough work to do as it is.
> > 
> > Stable tree stuff should cause almost _no_ extra burden on the kernel
> > developers, because it is something that I, and a few other people, have
> > agreed to do with our time.  It has taken me 8 _years_ to finally get
> > maintainers to agree to mark stuff for the stable tree, and fine-tune a
> > development process that makes it easy for us to do this backport work.
> 
> Although, since those 8 years, the stable tree has proven its
> importance.
> 
> Is a extra "ack" also too much to ask?

Maintainers are our most limited resource, I'm getting their "ack" when
they themselves tag the patch to be backported with the Cc: line.

I then cc: them when the patch goes into the patch queue.

I then cc: them again when the patch is in the -rc1 phase.

How many times do I need to do this to give people a chance to say
"nak"?

> The big problem with the above is that the process depends highly on
> this guy named "Greg".
> 
> If "Greg" gets tired of doing this, or gets sick, or "see other KS topic
> about mortality of maintainers", then the entire process fails.

"Greg" has documented how he does all of this work, and the scripts I
use are all published as well.

"Greg" has also asked for help with all of this, a number of times, with
a specific list of things that he needs help with.  And almost no one
has ever offered to help.  That's fine, I can live with that, we all
have jobs to do in other areas, and the LF now lets me do this as part
of my job, so I can focus on it.

> OK, you are not the only one that does the stable release. There's Ben
> and Luis doing it as well, and various others. But there seems to be a
> bit of an issue here. How much should go into stable? Who really
> decides?

The subsystem maintainers do by tagging the patches.  Every once in a
while I dig stuff up on my own, but again, I give the subsystem
maintainer at least two separate chances to NAK the patch, so it's not
like I'm doing anything in private here.

> How important is the stable releases? Are maintainers willing to do a
> little more work now to make sure their subsystems work fine in older
> kernels? This isn't the same stable as it was 8 years ago.

And that annoys the hell out of some Linux companies who feel that the
stable kernels compete with them.  So people working for those companies
might not get as much help with doing any additional work for stable
kernel releases (this is not just idle gossip, I've heard it directly
from management's mouths.)

So I need to, for both a resource management issue, and a business
issue, make this as painless for maintainers as possible, which I have
tried to do.

And given the uptick in the past few years of maintainers tagging
patches, I think it's worked out so well that people are now doing it
too much, which gets back to my original complaints last week that I am
now working to address in a different manner.

thanks,

greg k-h

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Steven Rostedt]

On Mon, 2013-07-15 at 17:06 -0700, Greg KH wrote:

> Maintainers are our most limited resource, I'm getting their "ack" when
> they themselves tag the patch to be backported with the Cc: line.

I find stable maintainers even more limited.

I'm not sure our maintainers are the most limited resource, we just have
the few that have proved themselves and it's hard to find people we
trust to expand. There's plenty of good people out there to help, but
the issue is more trust than anything else.

> 
> I then cc: them when the patch goes into the patch queue.
> 
> I then cc: them again when the patch is in the -rc1 phase.
> 
> How many times do I need to do this to give people a chance to say
> "nak"?

It can be missed by a big INBOX. Or put off till a later time that never
comes. The issue is that the default is to go in unless there is a
complaint, but if the maintainer needs to acknowledge before it goes
further, then it goes in when the maintainer is ready, not a short time
after it gets pulled into Linus's tree.

If the maintainer wants some patches to spend a bit of time in Linus's
tree before going to stable, then they need to not put the stable tag on
them, and manually do the work. Which may end up never going to stable
in the first place.

> 
> > The big problem with the above is that the process depends highly on
> > this guy named "Greg".
> > 
> > If "Greg" gets tired of doing this, or gets sick, or "see other KS topic
> > about mortality of maintainers", then the entire process fails.
> 
> "Greg" has documented how he does all of this work, and the scripts I
> use are all published as well.
> 
> "Greg" has also asked for help with all of this, a number of times, with
> a specific list of things that he needs help with.  And almost no one
> has ever offered to help.  That's fine, I can live with that, we all
> have jobs to do in other areas, and the LF now lets me do this as part
> of my job, so I can focus on it.

This is exactly my point. "Greg"'s problem is that he does too good of a
job at this. People look at what "Greg" does and say "he must be a God"
and start worshiping Him. I'm sure there are distros of people that have
sacrificed the sacred gecko in a burning altar chanting the subjects of
the stable patch bombs "Greg" does every week, praying that next week
there will come another holy trinity of the three stable releases "Greg"
maintains.

> 
> > OK, you are not the only one that does the stable release. There's Ben
> > and Luis doing it as well, and various others. But there seems to be a
> > bit of an issue here. How much should go into stable? Who really
> > decides?
> 
> The subsystem maintainers do by tagging the patches.  Every once in a
> while I dig stuff up on my own, but again, I give the subsystem
> maintainer at least two separate chances to NAK the patch, so it's not
> like I'm doing anything in private here.

It's not about doing it in private. But there seems to be some questions
about what should and should not be tagged stable. And the worse thing
that can happen is that a stable patch causes a regression in the stable
tree.

> 
> > How important is the stable releases? Are maintainers willing to do a
> > little more work now to make sure their subsystems work fine in older
> > kernels? This isn't the same stable as it was 8 years ago.
> 
> And that annoys the hell out of some Linux companies who feel that the
> stable kernels compete with them.  So people working for those companies
> might not get as much help with doing any additional work for stable
> kernel releases (this is not just idle gossip, I've heard it directly
> from management's mouths.)

Hmm, this is new to me. Really, I thought the whole point of the stable
releases was to help Linux companies.


> 
> So I need to, for both a resource management issue, and a business
> issue, make this as painless for maintainers as possible, which I have
> tried to do.
> 
> And given the uptick in the past few years of maintainers tagging
> patches, I think it's worked out so well that people are now doing it
> too much, which gets back to my original complaints last week that I am
> now working to address in a different manner.

Well, the final decision hangs with you and Linus. But would be good to
talk about this at Kernel Summit anyway. From other comments in the
thread, it looks like this will definitely be a topic there.

-- Steve

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 1039 bytes --]

On Mon, 2013-07-15 at 22:09 -0400, Steven Rostedt wrote:
[...] 
> > > How important is the stable releases? Are maintainers willing to do a
> > > little more work now to make sure their subsystems work fine in older
> > > kernels? This isn't the same stable as it was 8 years ago.
> > 
> > And that annoys the hell out of some Linux companies who feel that the
> > stable kernels compete with them.  So people working for those companies
> > might not get as much help with doing any additional work for stable
> > kernel releases (this is not just idle gossip, I've heard it directly
> > from management's mouths.)
> 
> Hmm, this is new to me. Really, I thought the whole point of the stable
> releases was to help Linux companies.
[...]

I also heard some managers decided their kernel source packages should
have all the patches squashed together to make them harder to cherry-
pick... could it have been the same company?

Ben.

-- 
Ben Hutchings
Humans are not rational beings; they are rationalising beings.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Dave Airlie]

On Tue, Jul 16, 2013 at 12:41 PM, Ben Hutchings <ben@decadent.org.uk> wrote:
> On Mon, 2013-07-15 at 22:09 -0400, Steven Rostedt wrote:
> [...]
>> > > How important is the stable releases? Are maintainers willing to do a
>> > > little more work now to make sure their subsystems work fine in older
>> > > kernels? This isn't the same stable as it was 8 years ago.
>> >
>> > And that annoys the hell out of some Linux companies who feel that the
>> > stable kernels compete with them.  So people working for those companies
>> > might not get as much help with doing any additional work for stable
>> > kernel releases (this is not just idle gossip, I've heard it directly
>> > from management's mouths.)
>>
>> Hmm, this is new to me. Really, I thought the whole point of the stable
>> releases was to help Linux companies.
> [...]
>
> I also heard some managers decided their kernel source packages should
> have all the patches squashed together to make them harder to cherry-
> pick... could it have been the same company?

Greg loves to tell stories about RH management, but really if he can
find any engineer who works for RH that says he can't work on stable
due to being told by management, I'd be surprised. Maybe when stable
first surfaced there was a hope of it being close to RHEL, but at this
point stable has little to no usefulness from a RHEL point of view,
and since nearly all the RH employed maintainers all do stable work, I
can't see why Greg would think this matters.

In fact Greg how much of stable queue does come from Red Hatters?

Dave.

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Steven Rostedt]

On Tue, 2013-07-16 at 13:27 +1000, Dave Airlie wrote:

> > I also heard some managers decided their kernel source packages should
> > have all the patches squashed together to make them harder to cherry-
> > pick... could it have been the same company?
> 
> Greg loves to tell stories about RH management, but really if he can
> find any engineer who works for RH that says he can't work on stable
> due to being told by management, I'd be surprised. Maybe when stable
> first surfaced there was a hope of it being close to RHEL, but at this
> point stable has little to no usefulness from a RHEL point of view,
> and since nearly all the RH employed maintainers all do stable work, I
> can't see why Greg would think this matters.

And to stress that point. I myself work for Red Hat, and I'm very big
into the stable tree. Heck, I maintain a 3.6 stable (not as well as
"Greg" does, but I'm just a mere mortal).

I don't know what managers Greg talked to, but I've been encouraged by
management to mark bug fixes with the stable tag.

-- Steve

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 2165 bytes --]

On Tue, 2013-07-16 at 13:27 +1000, Dave Airlie wrote:
> On Tue, Jul 16, 2013 at 12:41 PM, Ben Hutchings <ben@decadent.org.uk> wrote:
> > On Mon, 2013-07-15 at 22:09 -0400, Steven Rostedt wrote:
> > [...]
> >> > > How important is the stable releases? Are maintainers willing to do a
> >> > > little more work now to make sure their subsystems work fine in older
> >> > > kernels? This isn't the same stable as it was 8 years ago.
> >> >
> >> > And that annoys the hell out of some Linux companies who feel that the
> >> > stable kernels compete with them.  So people working for those companies
> >> > might not get as much help with doing any additional work for stable
> >> > kernel releases (this is not just idle gossip, I've heard it directly
> >> > from management's mouths.)
> >>
> >> Hmm, this is new to me. Really, I thought the whole point of the stable
> >> releases was to help Linux companies.
> > [...]
> >
> > I also heard some managers decided their kernel source packages should
> > have all the patches squashed together to make them harder to cherry-
> > pick... could it have been the same company?
> 
> Greg loves to tell stories about RH management, but really if he can
> find any engineer who works for RH that says he can't work on stable
> due to being told by management, I'd be surprised.

To be clear, I very much appreciate the level of contribution to stable
from RH developers.  And it's a little too high to believe these are
just a few rogues flouting policy. :-)

> Maybe when stable
> first surfaced there was a hope of it being close to RHEL,

The lack of non-trivial driver backports means it's not really
competitive with any commercial distribution.  (But it looks a little
better if you add in compat-drivers.)

> but at this
> point stable has little to no usefulness from a RHEL point of view,
> and since nearly all the RH employed maintainers all do stable work, I
> can't see why Greg would think this matters.
> 
> In fact Greg how much of stable queue does come from Red Hatters?

Ben.

-- 
Ben Hutchings
Humans are not rational beings; they are rationalising beings.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Greg KH]

On Tue, Jul 16, 2013 at 01:27:42PM +1000, Dave Airlie wrote:
> On Tue, Jul 16, 2013 at 12:41 PM, Ben Hutchings <ben@decadent.org.uk> wrote:
> > On Mon, 2013-07-15 at 22:09 -0400, Steven Rostedt wrote:
> > [...]
> >> > > How important is the stable releases? Are maintainers willing to do a
> >> > > little more work now to make sure their subsystems work fine in older
> >> > > kernels? This isn't the same stable as it was 8 years ago.
> >> >
> >> > And that annoys the hell out of some Linux companies who feel that the
> >> > stable kernels compete with them.  So people working for those companies
> >> > might not get as much help with doing any additional work for stable
> >> > kernel releases (this is not just idle gossip, I've heard it directly
> >> > from management's mouths.)
> >>
> >> Hmm, this is new to me. Really, I thought the whole point of the stable
> >> releases was to help Linux companies.
> > [...]
> >
> > I also heard some managers decided their kernel source packages should
> > have all the patches squashed together to make them harder to cherry-
> > pick... could it have been the same company?
> 
> Greg loves to tell stories about RH management, but really if he can
> find any engineer who works for RH that says he can't work on stable
> due to being told by management, I'd be surprised. Maybe when stable
> first surfaced there was a hope of it being close to RHEL, but at this
> point stable has little to no usefulness from a RHEL point of view,
> and since nearly all the RH employed maintainers all do stable work, I
> can't see why Greg would think this matters.
> 
> In fact Greg how much of stable queue does come from Red Hatters?

I always separate RH managers from engineers in their doings because I
know that one does not feel the same as the other.

My point is, if I start asking developers to do more work for the stable
trees, that has the potential to make people not like the stable trees
more for resource issues, and I don't want to do that at all.

greg k-h

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[James Bottomley]

On Mon, 2013-07-15 at 17:06 -0700, Greg KH wrote:
> On Mon, Jul 15, 2013 at 06:01:39PM -0400, Steven Rostedt wrote:
> > On Mon, 2013-07-15 at 14:44 -0700, Greg KH wrote:
> > 
> > > I don't like this at all, just for the simple reason that it will push
> > > the majority of the work of stable kernel development on to the
> > > subsystem maintainers, who have enough work to do as it is.
> > > 
> > > Stable tree stuff should cause almost _no_ extra burden on the kernel
> > > developers, because it is something that I, and a few other people, have
> > > agreed to do with our time.  It has taken me 8 _years_ to finally get
> > > maintainers to agree to mark stuff for the stable tree, and fine-tune a
> > > development process that makes it easy for us to do this backport work.
> > 
> > Although, since those 8 years, the stable tree has proven its
> > importance.
> > 
> > Is a extra "ack" also too much to ask?
> 
> Maintainers are our most limited resource, I'm getting their "ack" when
> they themselves tag the patch to be backported with the Cc: line.
> 
> I then cc: them when the patch goes into the patch queue.
> 
> I then cc: them again when the patch is in the -rc1 phase.
> 
> How many times do I need to do this to give people a chance to say
> "nak"?

Just to pick up on this, the problem from my perspective is that this
cc: goes into my personal inbox.  From a list perspective this just
doesn't work.  The entirety of my workflow is set up to operate from the
mailing lists.  My inbox is for my day job.  It gets about 100 emails or
more a day and anything that goes in there and doesn't get looked at for
a day gets lost.  I sometimes feel guilty about seeing stable reviews
whiz by, but not necessarily guilty enough to go back and try to find
them.  I have thought of using filtering to manually place these into a
deferred mailbox for later use.  However, the problem is that my work
inbox is exchange, and the only tags I could filter on seem to be in the
body (exchange does body filtering about as elegantly as a penguin
flies).

That's where the suggestion to drop cc: stable@ came from.  I realise
the workflow just isn't working for me.

I say we have the discussion at KS then I'll investigate a different
workflow for SCSI.

James

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Greg KH]

On Tue, Jul 16, 2013 at 10:10:31AM +0400, James Bottomley wrote:
> On Mon, 2013-07-15 at 17:06 -0700, Greg KH wrote:
> > On Mon, Jul 15, 2013 at 06:01:39PM -0400, Steven Rostedt wrote:
> > > On Mon, 2013-07-15 at 14:44 -0700, Greg KH wrote:
> > > 
> > > > I don't like this at all, just for the simple reason that it will push
> > > > the majority of the work of stable kernel development on to the
> > > > subsystem maintainers, who have enough work to do as it is.
> > > > 
> > > > Stable tree stuff should cause almost _no_ extra burden on the kernel
> > > > developers, because it is something that I, and a few other people, have
> > > > agreed to do with our time.  It has taken me 8 _years_ to finally get
> > > > maintainers to agree to mark stuff for the stable tree, and fine-tune a
> > > > development process that makes it easy for us to do this backport work.
> > > 
> > > Although, since those 8 years, the stable tree has proven its
> > > importance.
> > > 
> > > Is a extra "ack" also too much to ask?
> > 
> > Maintainers are our most limited resource, I'm getting their "ack" when
> > they themselves tag the patch to be backported with the Cc: line.
> > 
> > I then cc: them when the patch goes into the patch queue.
> > 
> > I then cc: them again when the patch is in the -rc1 phase.
> > 
> > How many times do I need to do this to give people a chance to say
> > "nak"?
> 
> Just to pick up on this, the problem from my perspective is that this
> cc: goes into my personal inbox.  From a list perspective this just
> doesn't work.  The entirety of my workflow is set up to operate from the
> mailing lists.  My inbox is for my day job.  It gets about 100 emails or
> more a day and anything that goes in there and doesn't get looked at for
> a day gets lost.  I sometimes feel guilty about seeing stable reviews
> whiz by, but not necessarily guilty enough to go back and try to find
> them.  I have thought of using filtering to manually place these into a
> deferred mailbox for later use.  However, the problem is that my work
> inbox is exchange, and the only tags I could filter on seem to be in the
> body (exchange does body filtering about as elegantly as a penguin
> flies).
> 
> That's where the suggestion to drop cc: stable@ came from.  I realise
> the workflow just isn't working for me.
> 
> I say we have the discussion at KS then I'll investigate a different
> workflow for SCSI.

KS is in November, feel free to not tag patches Cc: stable for scsi if
you don't want to use the existing workflow, and try something else now.

All I need, at the least, is a list of git ids to apply to the stable
tree(s), send them to stable@vger.kernel.org and I can take it from
there with my tools.

If you want to test things separately, send me a tree to pull, or a mbox
to apply, that works as well.  You can do all of these without having to
wait until November.

thanks,

greg k-h

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Jiri Kosina]

On Mon, 15 Jul 2013, Greg KH wrote:

> > The solution, to me, looks simple:  Let's co-opt a process we already
> > know how to do: mailing list review and tree handling.  So the proposal
> > is simple:
> > 
> >      1. Drop the cc: stable@ tag: it makes it way too easy to add an ill
> >         reviewed patch to stable
> >      2. All patches to stable should follow current review rules: They
> >         should go to the mailing list the original patch was sent to
> >         once the original is upstream as a request for stable.
> >      3. Following debate on the list, the original maintainer would be
> >         responsible for collecting the patches (including the upstream
> >         commit) adjudicating on them and passing them on to stable after
> >         list review (either by git tree pull or email to stable@).
> > 
> > I contend this raises the bar for adding patches to stable much higher,
> > which seems to be needed, and adds a review stage which involves all the
> > original reviewers.
> 
> I don't like this at all, just for the simple reason that it will push
> the majority of the work of stable kernel development on to the
> subsystem maintainers, who have enough work to do as it is.

Sorry Greg, but I disagree.

If the point of the stable tree really is about rock-solid stability, the 
"it applies without fuzz and there was no explicit NACK" just isn't 
enough. Someone who actually understands the code (maintainer) should 
absolutely give his Acked-for-stable-by: (*), otherwise the result is much 
less trustworthy.

I think 991f76f83 might serve as a good example. It has been marked "Cc: 
stable", it applied without cleanly, so it has been applied to all the 
existing stable trees, including 3.0.

The problem is that one has to actually perform a review of the patch with 
respect to 3.0.x codebase to notice that the pre-requisity for this patch 
(ef3d0fd27e) is only present in 3.2 and later, and hasn't been marked for 
stable (which is correct, it has no business there).

(*) For me personally, the best mode of operation would actually be to 
    have for-stable/3.x branches in my git tree, cherry-pick from other 
    topic branches once the patches are in Linus' tree, and send you pull 
    request for stable regularly (for each stable branch separately of 
    course)
   
    This model would make maintainers clearly responsible for the contents 
    of stable tree, wouldn't cause any extra work for you (quite the 
    contrary, I'd say), and it'd follow the development model we have for
    Linus' tree.

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Jiri Kosina]

On Tue, 16 Jul 2013, Jiri Kosina wrote:

> (*) For me personally, the best mode of operation would actually be to 
>     have for-stable/3.x branches in my git tree, cherry-pick from other 
>     topic branches once the patches are in Linus' tree, and send you pull 
>     request for stable regularly (for each stable branch separately of 
>     course)
>    
>     This model would make maintainers clearly responsible for the contents 
>     of stable tree, wouldn't cause any extra work for you (quite the 
>     contrary, I'd say), and it'd follow the development model we have for
>     Linus' tree.

... and it will actually have a nice self-regulatory feature (in case 
maintainers who are pushing too much stuff your way are part of the 
problem).

Preparing a proper pull request requires much more care, "stopping and 
thinking for a while", formulating the pull request, than just blindly 
tossing "Cc: stable" everywhere in a headless-chicken mode.

To sum it up:

- if the maintainers really do care about their patches being in the 
  -stable tree, this wouldn't cost them too much extra time

- if they are just randomly pushing everything to -stable because "hey, 
  why not", this will be an extra hurdle for them to overcome, and they 
  will be revealed easily by poor pull request justification

- if the maintainers are lazy and don't care about preparing stable 
  branches, it's their responsibility (and their shame) that -stable will 
  be missing the code they are responsible for

- it offloads the work from single point of failure (you) to the 
  maintainers, which makes a lot of sense to me

- it aligns the workflow with the workflow that's in place for Linus' 
  already (and not only there) to be more or less a proper git workflow

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Greg KH]

On Tue, Jul 16, 2013 at 12:22:16AM +0200, Jiri Kosina wrote:
> On Mon, 15 Jul 2013, Greg KH wrote:
> 
> > > The solution, to me, looks simple:  Let's co-opt a process we already
> > > know how to do: mailing list review and tree handling.  So the proposal
> > > is simple:
> > > 
> > >      1. Drop the cc: stable@ tag: it makes it way too easy to add an ill
> > >         reviewed patch to stable
> > >      2. All patches to stable should follow current review rules: They
> > >         should go to the mailing list the original patch was sent to
> > >         once the original is upstream as a request for stable.
> > >      3. Following debate on the list, the original maintainer would be
> > >         responsible for collecting the patches (including the upstream
> > >         commit) adjudicating on them and passing them on to stable after
> > >         list review (either by git tree pull or email to stable@).
> > > 
> > > I contend this raises the bar for adding patches to stable much higher,
> > > which seems to be needed, and adds a review stage which involves all the
> > > original reviewers.
> > 
> > I don't like this at all, just for the simple reason that it will push
> > the majority of the work of stable kernel development on to the
> > subsystem maintainers, who have enough work to do as it is.
> 
> Sorry Greg, but I disagree.
> 
> If the point of the stable tree really is about rock-solid stability, the 
> "it applies without fuzz and there was no explicit NACK" just isn't 
> enough. Someone who actually understands the code (maintainer) should 
> absolutely give his Acked-for-stable-by: (*), otherwise the result is much 
> less trustworthy.
> 
> I think 991f76f83 might serve as a good example. It has been marked "Cc: 
> stable", it applied without cleanly, so it has been applied to all the 
> existing stable trees, including 3.0.
> 
> The problem is that one has to actually perform a review of the patch with 
> respect to 3.0.x codebase to notice that the pre-requisity for this patch 
> (ef3d0fd27e) is only present in 3.2 and later, and hasn't been marked for 
> stable (which is correct, it has no business there).

Ok, that's a bug / fault that was my fault.  It got caught (right?),
that will always happen at times, all we can do is recover and move on.

And you really did want this patch in the stable kernel at the time, as
it could crash the kernel, so no matter what the procedure in place
was, I would have applied it.  Heck, I am the maintainer there, so I
messed up, how could I have prevented myself from applying the patch?  :)

> (*) For me personally, the best mode of operation would actually be to 
>     have for-stable/3.x branches in my git tree, cherry-pick from other 
>     topic branches once the patches are in Linus' tree, and send you pull 
>     request for stable regularly (for each stable branch separately of 
>     course)
>    
>     This model would make maintainers clearly responsible for the contents 
>     of stable tree, wouldn't cause any extra work for you (quite the 
>     contrary, I'd say), and it'd follow the development model we have for
>     Linus' tree.

I don't object to that, and again, some maintainers do this.  If you
want to do this for your trees, fine with me.  But for others, it might
not be the best workflow.

It also increases the workload on maintainers to support stable
releases, which is what I do _not_ want to do at all.  Maintainers are
the most limited of resources that we possibly have at the moment, I
will almost never ask them to do extra work that is not needed.

thanks,

greg k-h

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 910 bytes --]

On Mon, 2013-07-15 at 14:44 -0700, Greg KH wrote:
[...]
> The second one is almost always due to security issues that were unknown
> to the distro.  The announcement of security problems to the distros has
> now been addressed, and since that has changed, I haven't heard any
> problems about this.
[...]

I'm not elite enough to be on these security lists, though sometimes I
hear about embargoed issues through the Debian security team.

Anything that's being reviewed on the stable list is public.  I know
this is an old argument, but if you point out a fix you *know* has a
security impact then you'll help general distribution maintainers and
users a lot more than you help the black-hats who are quite capable of
recognising such a fix (if they haven't already spotted and exploited
the bug).

Ben.

-- 
Ben Hutchings
Humans are not rational beings; they are rationalising beings.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Greg KH]

On Tue, Jul 16, 2013 at 03:30:01AM +0100, Ben Hutchings wrote:
> Anything that's being reviewed on the stable list is public.  I know
> this is an old argument, but if you point out a fix you *know* has a
> security impact then you'll help general distribution maintainers and
> users a lot more than you help the black-hats who are quite capable of
> recognising such a fix (if they haven't already spotted and exploited
> the bug).

I'm sorry, but you know I will not do that, so asking about it isn't
going to change this behavior.

greg k-h

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Jiri Kosina]

On Mon, 15 Jul 2013, Greg KH wrote:

> > Anything that's being reviewed on the stable list is public.  I know
> > this is an old argument, but if you point out a fix you *know* has a
> > security impact then you'll help general distribution maintainers and
> > users a lot more than you help the black-hats who are quite capable of
> > recognising such a fix (if they haven't already spotted and exploited
> > the bug).
> 
> I'm sorry, but you know I will not do that, so asking about it isn't
> going to change this behavior.

I just followed up in the other thread, where Ted was explaining why the 
huge /dev/random rework was a -stable material.

Why specifically would it be wrong to be open about this being security 
related, and providing the necessary data (i.e. at least reference to 
http://factorable.net/) publically?

I fail to see what the point behind hiding this would be.

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Greg KH]

On Tue, Jul 16, 2013 at 11:11:24AM +0200, Jiri Kosina wrote:
> On Mon, 15 Jul 2013, Greg KH wrote:
> 
> > > Anything that's being reviewed on the stable list is public.  I know
> > > this is an old argument, but if you point out a fix you *know* has a
> > > security impact then you'll help general distribution maintainers and
> > > users a lot more than you help the black-hats who are quite capable of
> > > recognising such a fix (if they haven't already spotted and exploited
> > > the bug).
> > 
> > I'm sorry, but you know I will not do that, so asking about it isn't
> > going to change this behavior.
> 
> I just followed up in the other thread, where Ted was explaining why the 
> huge /dev/random rework was a -stable material.
> 
> Why specifically would it be wrong to be open about this being security 
> related, and providing the necessary data (i.e. at least reference to 
> http://factorable.net/) publically?
> 
> I fail to see what the point behind hiding this would be.

I'm not "hiding" anything, all I'm doing is using the exact same
changelog comments that are in Linus's tree, and nothing else.

greg k-h

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 1590 bytes --]

On Tue, 2013-07-16 at 09:36 -0700, Greg KH wrote:
> On Tue, Jul 16, 2013 at 11:11:24AM +0200, Jiri Kosina wrote:
> > On Mon, 15 Jul 2013, Greg KH wrote:
> > 
> > > > Anything that's being reviewed on the stable list is public.  I know
> > > > this is an old argument, but if you point out a fix you *know* has a
> > > > security impact then you'll help general distribution maintainers and
> > > > users a lot more than you help the black-hats who are quite capable of
> > > > recognising such a fix (if they haven't already spotted and exploited
> > > > the bug).
> > > 
> > > I'm sorry, but you know I will not do that, so asking about it isn't
> > > going to change this behavior.
> > 
> > I just followed up in the other thread, where Ted was explaining why the 
> > huge /dev/random rework was a -stable material.
> > 
> > Why specifically would it be wrong to be open about this being security 
> > related, and providing the necessary data (i.e. at least reference to 
> > http://factorable.net/) publically?
> > 
> > I fail to see what the point behind hiding this would be.
> 
> I'm not "hiding" anything, all I'm doing is using the exact same
> changelog comments that are in Linus's tree, and nothing else.

Right, and I wouldn't expect you to edit commit messages.  But if a fix
was privately proposed to you for stable on the grounds that the bug is
found to be exploitable, maybe you could include that information in the
cover message for the review.

Ben.

-- 
Ben Hutchings
Humans are not rational beings; they are rationalising beings.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Greg KH]

On Wed, Jul 17, 2013 at 04:53:58AM +0100, Ben Hutchings wrote:
> On Tue, 2013-07-16 at 09:36 -0700, Greg KH wrote:
> > On Tue, Jul 16, 2013 at 11:11:24AM +0200, Jiri Kosina wrote:
> > > On Mon, 15 Jul 2013, Greg KH wrote:
> > > 
> > > > > Anything that's being reviewed on the stable list is public.  I know
> > > > > this is an old argument, but if you point out a fix you *know* has a
> > > > > security impact then you'll help general distribution maintainers and
> > > > > users a lot more than you help the black-hats who are quite capable of
> > > > > recognising such a fix (if they haven't already spotted and exploited
> > > > > the bug).
> > > > 
> > > > I'm sorry, but you know I will not do that, so asking about it isn't
> > > > going to change this behavior.
> > > 
> > > I just followed up in the other thread, where Ted was explaining why the 
> > > huge /dev/random rework was a -stable material.
> > > 
> > > Why specifically would it be wrong to be open about this being security 
> > > related, and providing the necessary data (i.e. at least reference to 
> > > http://factorable.net/) publically?
> > > 
> > > I fail to see what the point behind hiding this would be.
> > 
> > I'm not "hiding" anything, all I'm doing is using the exact same
> > changelog comments that are in Linus's tree, and nothing else.
> 
> Right, and I wouldn't expect you to edit commit messages.  But if a fix
> was privately proposed to you for stable on the grounds that the bug is
> found to be exploitable, maybe you could include that information in the
> cover message for the review.

Sorry, my position on that has not changed, I will not do that for the
obvious reasons discussed many times in the past.

greg k-h

Re: KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[James Bottomley]

On Mon, 2013-07-15 at 14:44 -0700, Greg KH wrote:
> On Mon, Jul 15, 2013 at 11:27:56PM +0400, James Bottomley wrote:
> > Before the "3.10.1-stable review" thread degenerated into a disagreement
> > about habits of politeness, there were some solid points being made
> > which, I think, bear consideration and which may now be lost.
> > 
> > The problem, as Ji???? Kosina put is succinctly is that the distributions
> > are finding stable less useful because it contains to much stuff they'd
> > classify as not stable material.
> > 
> > The question that arises from this is who is stable aiming at ...
> > because if it's the distributions (and that's what people seem to be
> > using it for) then we need to take this feedback seriously.
> > 
> > The next question is how should we, the maintainers, be policing commits
> > to stable.  As I think has been demonstrated in the discussion the
> > "stable rules" are more sort of guidelines (apologies for the pirates
> > reference).  In many ways, this is as it should be, because people
> > should have enough taste to know what constitutes a stable fix.  The
> > real root cause of the problem is that the cc: stable tag can't be
> > stripped once it's in the tree, so maintainers only get to police things
> > they put in the tree.  Stuff they pull from others is already tagged and
> > that tag can't be changed.  This effectively pushes the problem out to
> > the lowest (and possibly more inexperienced) leaves of the Maintainer
> > tree.  In theory we have a review stage for stable, but the review
> > patches don't automatically get routed to the right mailing list and the
> > first round usually comes out in the merge window when Maintainers'
> > attention is elsewhere.
> > 
> > The solution, to me, looks simple:  Let's co-opt a process we already
> > know how to do: mailing list review and tree handling.  So the proposal
> > is simple:
> > 
> >      1. Drop the cc: stable@ tag: it makes it way too easy to add an ill
> >         reviewed patch to stable
> >      2. All patches to stable should follow current review rules: They
> >         should go to the mailing list the original patch was sent to
> >         once the original is upstream as a request for stable.
> >      3. Following debate on the list, the original maintainer would be
> >         responsible for collecting the patches (including the upstream
> >         commit) adjudicating on them and passing them on to stable after
> >         list review (either by git tree pull or email to stable@).
> > 
> > I contend this raises the bar for adding patches to stable much higher,
> > which seems to be needed, and adds a review stage which involves all the
> > original reviewers.
> 
> I don't like this at all, just for the simple reason that it will push
> the majority of the work of stable kernel development on to the
> subsystem maintainers, who have enough work to do as it is.

Well, I think of this as a feature not a bug because it enables scaling,
but we can certainly debate the topic; it's probably one of the more
important things that should be discussed.

> Stable tree stuff should cause almost _no_ extra burden on the kernel
> developers, because it is something that I, and a few other people, have
> agreed to do with our time.  It has taken me 8 _years_ to finally get
> maintainers to agree to mark stuff for the stable tree, and fine-tune a
> development process that makes it easy for us to do this backport work.
> 
> I _want_ the exact same commit that is in Linus's tree for the backport
> because if I have to rely on a maintainer to do the backport and resend
> it, I _know_ it will usually be a changed patch, and the git commit id
> will be lost.

This is fixable with process and tools, surely.

It's always surprised me that the number one stable_kernel_rules.txt
rule wasn't you have tobe upstream and label in the the first line with
Upstream-Commit: commitid.  (I know it's in the procedure, but that's
not rules).  But if it were, a script can reject with a response all
non-conforming emails.  Now you have the commitid, interdiff can tell
you if it's the same patch as went into the kernel, so you can put this
into the automatic apply queue.  Everything else goes into a different
queue.

> I know this because it happens today.  I get patches from maintainers
> that are radically changed from what is in Linus's tree without any
> justification for why that is.  That's not ok.

So you could even script this.  An automatic request for justification
of patches that interdiff flags as being different (or a request to
resubmit with the identical diff so they pass the interdiff test).

> Let me work with the distros on the issues they have raised.  So far I
> have the following issues they have complained about:
>   - patches that shouldn't be there because they don't really do
>     anything.
>   - patches that aren't obvious why they are there.
> 
> The first one I am going to be pushing back on, and have already, much
> to the dismay of some subsystem maintainers.
> 
> The second one is almost always due to security issues that were unknown
> to the distro.  The announcement of security problems to the distros has
> now been addressed, and since that has changed, I haven't heard any
> problems about this.
> 
> Have I missed anything else that the distros are objecting to?  The
> "smaller" distros (i.e. ones without lots of kernel developers) have
> been giving me nothing but _thanks_ and appreciation with the way that
> I've been sucking in all of the different fixes.  Do we want to mess
> with a process that is really working out well for them, and only causes
> annoyance at times by the larger ones?

I still think the scaling problem remains plus, if you push back more,
it's going to increase friction: you aren't necessarily in the best
position to know what should be a stable fix for a given subsystem, and
if you push back incorrectly it's going to annoy the maintainer.

James

Re: KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Greg KH]

On Tue, Jul 16, 2013 at 09:17:32AM +0400, James Bottomley wrote:
> On Mon, 2013-07-15 at 14:44 -0700, Greg KH wrote:
> > On Mon, Jul 15, 2013 at 11:27:56PM +0400, James Bottomley wrote:
> > > Before the "3.10.1-stable review" thread degenerated into a disagreement
> > > about habits of politeness, there were some solid points being made
> > > which, I think, bear consideration and which may now be lost.
> > > 
> > > The problem, as Ji???? Kosina put is succinctly is that the distributions
> > > are finding stable less useful because it contains to much stuff they'd
> > > classify as not stable material.

And I'm going to be working on that.

But I need, from the distros, specific examples of what they object to.
So far all I've gotten is one security patch (that was needed), and one
patch for sysfs that I backported too far in the version numbers (my
fault.)

Given the huge number of stable patches over the past few years, only
having 2 patches be an issue sounds like things are working well for
people here.

If I don't get pushback, with specifics, from the distros, I will not
know what to change, so if people can provide me that, it will help out
a lot.

> > I don't like this at all, just for the simple reason that it will push
> > the majority of the work of stable kernel development on to the
> > subsystem maintainers, who have enough work to do as it is.
> 
> Well, I think of this as a feature not a bug because it enables scaling,
> but we can certainly debate the topic; it's probably one of the more
> important things that should be discussed.

How does this scale?  It makes the maintainers do more work, which does
not scale at all.

> > Stable tree stuff should cause almost _no_ extra burden on the kernel
> > developers, because it is something that I, and a few other people, have
> > agreed to do with our time.  It has taken me 8 _years_ to finally get
> > maintainers to agree to mark stuff for the stable tree, and fine-tune a
> > development process that makes it easy for us to do this backport work.
> > 
> > I _want_ the exact same commit that is in Linus's tree for the backport
> > because if I have to rely on a maintainer to do the backport and resend
> > it, I _know_ it will usually be a changed patch, and the git commit id
> > will be lost.
> 
> This is fixable with process and tools, surely.

That's crazy, and it implies that there is a problem today with the Cc:
tag.

Is there?  What is the problem that you are seeing that you wish to help
resolve?

I don't think it's the same problem that I am seeing, as adding more
tools and processes sure isn't going to fix that.

> > Have I missed anything else that the distros are objecting to?  The
> > "smaller" distros (i.e. ones without lots of kernel developers) have
> > been giving me nothing but _thanks_ and appreciation with the way that
> > I've been sucking in all of the different fixes.  Do we want to mess
> > with a process that is really working out well for them, and only causes
> > annoyance at times by the larger ones?
> 
> I still think the scaling problem remains plus, if you push back more,
> it's going to increase friction: you aren't necessarily in the best
> position to know what should be a stable fix for a given subsystem, and
> if you push back incorrectly it's going to annoy the maintainer.

Making a subsystem maintainer answer "why is this needed" is really
going to annoy people?  Seriously?

Let's try it out, and see what happens.

I think the real stable issue that _everyone_ keeps ignoring, is my
original complaint, in that people are using the -rc1 merge window to
get fixes in they should have sent to Linus for .0.

I don't see anything you have written so far that will help resolve that
issue, which is not good, as that needs to be taken care of.

greg k-h

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[James Bottomley]

On Mon, 2013-07-15 at 23:20 -0700, Greg KH wrote:
> On Tue, Jul 16, 2013 at 09:17:32AM +0400, James Bottomley wrote:
> > On Mon, 2013-07-15 at 14:44 -0700, Greg KH wrote:
> > > On Mon, Jul 15, 2013 at 11:27:56PM +0400, James Bottomley wrote:
> > > > Before the "3.10.1-stable review" thread degenerated into a disagreement
> > > > about habits of politeness, there were some solid points being made
> > > > which, I think, bear consideration and which may now be lost.
> > > > 
> > > > The problem, as Ji???? Kosina put is succinctly is that the distributions
> > > > are finding stable less useful because it contains to much stuff they'd
> > > > classify as not stable material.
> 
> And I'm going to be working on that.

OK, I accept that.

> But I need, from the distros, specific examples of what they object to.
> So far all I've gotten is one security patch (that was needed), and one
> patch for sysfs that I backported too far in the version numbers (my
> fault.)
> 
> Given the huge number of stable patches over the past few years, only
> having 2 patches be an issue sounds like things are working well for
> people here.
> 
> If I don't get pushback, with specifics, from the distros, I will not
> know what to change, so if people can provide me that, it will help out
> a lot.

I agree ... I think Jiří and his Red Hat equivalent need to pipe up and
give us more examples of the problems they've been having.

> > > I don't like this at all, just for the simple reason that it will push
> > > the majority of the work of stable kernel development on to the
> > > subsystem maintainers, who have enough work to do as it is.
> > 
> > Well, I think of this as a feature not a bug because it enables scaling,
> > but we can certainly debate the topic; it's probably one of the more
> > important things that should be discussed.
> 
> How does this scale?  It makes the maintainers do more work, which does
> not scale at all.

We obviously have different ideas of what "scaling" means.  Being a
mathematician I think of a process were you review everything as o(1)
and where the maintainers review everything as o(n).  Since the number
of patches (np) tends to grow as the number of maintainers, np/o(n) is
an approximate constant meaning the amount of work per maintainer
scales.  np/o(1) grows.

> > > Stable tree stuff should cause almost _no_ extra burden on the kernel
> > > developers, because it is something that I, and a few other people, have
> > > agreed to do with our time.  It has taken me 8 _years_ to finally get
> > > maintainers to agree to mark stuff for the stable tree, and fine-tune a
> > > development process that makes it easy for us to do this backport work.
> > > 
> > > I _want_ the exact same commit that is in Linus's tree for the backport
> > > because if I have to rely on a maintainer to do the backport and resend
> > > it, I _know_ it will usually be a changed patch, and the git commit id
> > > will be lost.
> > 
> > This is fixable with process and tools, surely.
> 
> That's crazy,

Fixing a problem you just complained about with a suggestion involving
process and tools is crazy? I think I want some of what you're on.

>  and it implies that there is a problem today with the Cc:
> tag.
> 
> Is there?  What is the problem that you are seeing that you wish to help
> resolve?

To repeat: the problem, as I see it, is that the tag gets applied at the
leaves of the tree.  It can't be stripped in a normal git workflow, so
the process for adding it doesn't get enough review to ensure our stable
tree contains the correct patches.

> I don't think it's the same problem that I am seeing, as adding more
> tools and processes sure isn't going to fix that.
> 
> > > Have I missed anything else that the distros are objecting to?  The
> > > "smaller" distros (i.e. ones without lots of kernel developers) have
> > > been giving me nothing but _thanks_ and appreciation with the way that
> > > I've been sucking in all of the different fixes.  Do we want to mess
> > > with a process that is really working out well for them, and only causes
> > > annoyance at times by the larger ones?
> > 
> > I still think the scaling problem remains plus, if you push back more,
> > it's going to increase friction: you aren't necessarily in the best
> > position to know what should be a stable fix for a given subsystem, and
> > if you push back incorrectly it's going to annoy the maintainer.
> 
> Making a subsystem maintainer answer "why is this needed" is really
> going to annoy people?  Seriously?

Pushing back wrongly is, yes, it's just human nature.

>From a human nature perspective, the friction to inclusion rises
globally as the current kernel goes through the -rc cycle.  There's no
such natural cadence for stable.  It's the general level of friction
that needs to rise for stable and I'm not sure that individual pushback
does that. 

> Let's try it out, and see what happens.

Well, experiment is the best teacher, yes, let's see how it works out.

> I think the real stable issue that _everyone_ keeps ignoring, is my
> original complaint, in that people are using the -rc1 merge window to
> get fixes in they should have sent to Linus for .0.

You mean we delay fixes to the merge window (tagged for stable) because
we can't get them into Linus' tree at -rc5 on?  Guilty ... that's
because the friction for getting stuff in rises.  It's a big fight to
get something marginal in after -rc5 ... it's easy to silently tag it
for stable (did I mention that I think the tag part enables this
behaviour?).

> I don't see anything you have written so far that will help resolve that
> issue, which is not good, as that needs to be taken care of.

Removing the tag removes the behaviour ... it looks like a fairly simple
(albeit naïve) solution to me.

James

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Jiri Kosina]

On Tue, 16 Jul 2013, James Bottomley wrote:

> > But I need, from the distros, specific examples of what they object to.
> > So far all I've gotten is one security patch (that was needed), and one
> > patch for sysfs that I backported too far in the version numbers (my
> > fault.)
> > 
> > Given the huge number of stable patches over the past few years, only
> > having 2 patches be an issue sounds like things are working well for
> > people here.
> > 
> > If I don't get pushback, with specifics, from the distros, I will not
> > know what to change, so if people can provide me that, it will help out
> > a lot.
> 
> I agree ... I think Jiří and his Red Hat equivalent need to pipe up and
> give us more examples of the problems they've been having.

I am still continuing with my pushback against the /dev/random revamp that 
happened in -stable; at least in the form it happened. I still strongly 
believe it's something that's not a stable material. But that's happening 
in parallel in a different thread already.

Okay, if you want another example:

	commit a6aa749906b92eaec6ca0469f90f35de26044d90
	Author: Zhenzhong Duan <zhenzhong.duan@oracle.com>
	Date:   Thu Dec 20 15:05:14 2012 -0800

	    drivers/firmware/dmi_scan.c: fetch dmi version from SMBIOS if it exists

While this is a correct fix for major kernel release, as it achieves 
correctness by checking SMBIOS version properly and behaving according to 
the spec, it actually causes an userspace ABI regression in some sense, as 
it just changes byte order of /sys/class/dmi/id/product_uuid on certain 
systems.

Which is something I absolutely *do not* expect from a minor release of 
branch which is called "stable".

As has been pointed out in other thread already, this all might be a 
matter of taste; I am just trying to point out where the stable process is 
failing for us.

> > > > I don't like this at all, just for the simple reason that it will push
> > > > the majority of the work of stable kernel development on to the
> > > > subsystem maintainers, who have enough work to do as it is.
> > > 
> > > Well, I think of this as a feature not a bug because it enables scaling,
> > > but we can certainly debate the topic; it's probably one of the more
> > > important things that should be discussed.
> > 
> > How does this scale?  It makes the maintainers do more work, which does
> > not scale at all.
> 
> We obviously have different ideas of what "scaling" means.  Being a
> mathematician I think of a process were you review everything as o(1)
> and where the maintainers review everything as o(n).  Since the number
> of patches (np) tends to grow as the number of maintainers, np/o(n) is
> an approximate constant meaning the amount of work per maintainer
> scales.  np/o(1) grows.

Having a degree from mathematical insitute as well, I agree with James :)


On Mon, 15 Jul 2013, Greg KH wrote:

> I give maintainers 2 different chances to NAK a patch, and if they miss
> those, I can also easily revert a patch that got applied and do a new
> release, which I have done in the past.

This is exactly what I wanted to change with my proposal yesterday -- 
asking for NAKing of something that is already upstream is just "boring" 
and going to be ignored by many people. That's reality, unfortunately. 
It's basically a passive approval scenario.

If you completely push the burden out to maintainers, to have them prepare 
the branch and send it to you for pulling, that requires an *active* 
action from them, which means much more thinking about what code is going 
into the branch.

It could mean that less code will be going to stable, yes. But it will be 
documented *why* it's going there (in the pull request), there will be 
clearly someone responsible for it (the person who conducted the pull 
request) and the changes will have much higher chances to be reviewed in 
terms of applicability to -stable.

Oh, and I need to add: I am not pushing against you personally or the 
stable branch in general, at all. We are heavy users of -stable releases 
of course, and it saves us a lot of work. I am just trying to figure out 
ways how to avoid another lot of extra work on our side when change that 
shouldn't be pulled into stable lands in our tree through it. Once that 
happens, it causes us quite some headache, as we are implicitly relying on 
stable to actually be "stable", in our understanting of the word :)

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Jiri Kosina]

On Tue, 16 Jul 2013, James Bottomley wrote:

> > But I need, from the distros, specific examples of what they object to.
> > So far all I've gotten is one security patch (that was needed), and one
> > patch for sysfs that I backported too far in the version numbers (my
> > fault.)
> > 
> > Given the huge number of stable patches over the past few years, only
> > having 2 patches be an issue sounds like things are working well for
> > people here.
> > 
> > If I don't get pushback, with specifics, from the distros, I will not
> > know what to change, so if people can provide me that, it will help out
> > a lot.
> 
> I agree ... I think Ji�� and his Red Hat equivalent need to pipe up and
> give us more examples of the problems they've been having.

I am still continuing with my pushback against the /dev/random revamp that 
happened in -stable; at least in the form it happened. I still strongly 
believe it's something that's not a stable material. But that's happening 
in parallel in a different thread already.

Okay, if you want another example:

	commit a6aa749906b92eaec6ca0469f90f35de26044d90
	Author: Zhenzhong Duan <zhenzhong.duan@oracle.com>
	Date:   Thu Dec 20 15:05:14 2012 -0800

	    drivers/firmware/dmi_scan.c: fetch dmi version from SMBIOS if it exists

While this is a correct fix for major kernel release, as it achieves 
correctness by checking SMBIOS version properly and behaving according to 
the spec, it actually causes an userspace ABI regression in some sense, as 
it just changes byte order of /sys/class/dmi/id/product_uuid on certain 
systems.

Which is something I absolutely *do not* expect from a minor release of 
branch which is called "stable".

As has been pointed out in other thread already, this all might be a 
matter of taste; I am just trying to point out where the stable process is 
failing for us.

> > > > I don't like this at all, just for the simple reason that it will push
> > > > the majority of the work of stable kernel development on to the
> > > > subsystem maintainers, who have enough work to do as it is.
> > > 
> > > Well, I think of this as a feature not a bug because it enables scaling,
> > > but we can certainly debate the topic; it's probably one of the more
> > > important things that should be discussed.
> > 
> > How does this scale?  It makes the maintainers do more work, which does
> > not scale at all.
> 
> We obviously have different ideas of what "scaling" means.  Being a
> mathematician I think of a process were you review everything as o(1)
> and where the maintainers review everything as o(n).  Since the number
> of patches (np) tends to grow as the number of maintainers, np/o(n) is
> an approximate constant meaning the amount of work per maintainer
> scales.  np/o(1) grows.

Having a degree from mathematical insitute as well, I agree with James :)


On Mon, 15 Jul 2013, Greg KH wrote:

> I give maintainers 2 different chances to NAK a patch, and if they miss
> those, I can also easily revert a patch that got applied and do a new
> release, which I have done in the past.

This is exactly what I wanted to change with my proposal yesterday -- 
asking for NAKing of something that is already upstream is just "boring" 
and going to be ignored by many people. That's reality, unfortunately. 
It's basically a passive approval scenario.

If you completely push the burden out to maintainers, to have them prepare 
the branch and send it to you for pulling, that requires an *active* 
action from them, which means much more thinking about what code is going 
into the branch.

It could mean that less code will be going to stable, yes. But it will be 
documented *why* it's going there (in the pull request), there will be 
clearly someone responsible for it (the person who conducted the pull 
request) and the changes will have much higher chances to be reviewed in 
terms of applicability to -stable.

Oh, and I need to add: I am not pushing against you personally or the 
stable branch in general, at all. We are heavy users of -stable releases 
of course, and it saves us a lot of work. I am just trying to figure out 
ways how to avoid another lot of extra work on our side when change that 
shouldn't be pulled into stable lands in our tree through it. Once that 
happens, it causes us quite some headache, as we are implicitly relying on 
stable to actually be "stable", in our understanting of the word :)

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 1963 bytes --]

On Tue, 2013-07-16 at 11:46 +0200, Jiri Kosina wrote:
> On Tue, 16 Jul 2013, James Bottomley wrote:
> 
> > > But I need, from the distros, specific examples of what they object to.
> > > So far all I've gotten is one security patch (that was needed), and one
> > > patch for sysfs that I backported too far in the version numbers (my
> > > fault.)
> > > 
> > > Given the huge number of stable patches over the past few years, only
> > > having 2 patches be an issue sounds like things are working well for
> > > people here.
> > > 
> > > If I don't get pushback, with specifics, from the distros, I will not
> > > know what to change, so if people can provide me that, it will help out
> > > a lot.
> > 
> > I agree ... I think Jiří and his Red Hat equivalent need to pipe up and
> > give us more examples of the problems they've been having.
> 
> I am still continuing with my pushback against the /dev/random revamp that 
> happened in -stable; at least in the form it happened. I still strongly 
> believe it's something that's not a stable material. But that's happening 
> in parallel in a different thread already.
> 
> Okay, if you want another example:
> 
> 	commit a6aa749906b92eaec6ca0469f90f35de26044d90
> 	Author: Zhenzhong Duan <zhenzhong.duan@oracle.com>
> 	Date:   Thu Dec 20 15:05:14 2012 -0800
> 
> 	    drivers/firmware/dmi_scan.c: fetch dmi version from SMBIOS if it exists
> 
> While this is a correct fix for major kernel release, as it achieves 
> correctness by checking SMBIOS version properly and behaving according to 
> the spec, it actually causes an userspace ABI regression in some sense, as 
> it just changes byte order of /sys/class/dmi/id/product_uuid on certain 
> systems.
[...]

It wasn't even a correct fix, as it caused the DMI header to be detected
in the wrong place on some machines.

Ben.

-- 
Ben Hutchings
Humans are not rational beings; they are rationalising beings.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Greg KH]

On Tue, Jul 16, 2013 at 11:46:05AM +0200, Jiri Kosina wrote:
> On Tue, 16 Jul 2013, James Bottomley wrote:
> 
> > > But I need, from the distros, specific examples of what they object to.
> > > So far all I've gotten is one security patch (that was needed), and one
> > > patch for sysfs that I backported too far in the version numbers (my
> > > fault.)
> > > 
> > > Given the huge number of stable patches over the past few years, only
> > > having 2 patches be an issue sounds like things are working well for
> > > people here.
> > > 
> > > If I don't get pushback, with specifics, from the distros, I will not
> > > know what to change, so if people can provide me that, it will help out
> > > a lot.
> > 
> > I agree ... I think Ji?? and his Red Hat equivalent need to pipe up and
> > give us more examples of the problems they've been having.
> 
> I am still continuing with my pushback against the /dev/random revamp that 
> happened in -stable; at least in the form it happened. I still strongly 
> believe it's something that's not a stable material. But that's happening 
> in parallel in a different thread already.
> 
> Okay, if you want another example:
> 
> 	commit a6aa749906b92eaec6ca0469f90f35de26044d90
> 	Author: Zhenzhong Duan <zhenzhong.duan@oracle.com>
> 	Date:   Thu Dec 20 15:05:14 2012 -0800
> 
> 	    drivers/firmware/dmi_scan.c: fetch dmi version from SMBIOS if it exists
> 
> While this is a correct fix for major kernel release, as it achieves 
> correctness by checking SMBIOS version properly and behaving according to 
> the spec, it actually causes an userspace ABI regression in some sense, as 
> it just changes byte order of /sys/class/dmi/id/product_uuid on certain 
> systems.
> 
> Which is something I absolutely *do not* expect from a minor release of 
> branch which is called "stable".

As you pointed out, your definition of "stable" is one that the
enterprise distros have created, and might be totally different from my
view of what "stable" is :)

And also, as Ben pointed out, this was probably wrong, and someone
should have told me about this, so I could have reverted it.  Please do
so the next time.

thanks,

greg k-h

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Jiri Kosina]

On Tue, 16 Jul 2013, Greg KH wrote:

> > > > But I need, from the distros, specific examples of what they object to.
> > > > So far all I've gotten is one security patch (that was needed), and one
> > > > patch for sysfs that I backported too far in the version numbers (my
> > > > fault.)
> > > > 
> > > > Given the huge number of stable patches over the past few years, only
> > > > having 2 patches be an issue sounds like things are working well for
> > > > people here.
> > > > 
> > > > If I don't get pushback, with specifics, from the distros, I will not
> > > > know what to change, so if people can provide me that, it will help out
> > > > a lot.
> > > 
> > > I agree ... I think Ji?? and his Red Hat equivalent need to pipe up and
> > > give us more examples of the problems they've been having.
> > 
> > I am still continuing with my pushback against the /dev/random revamp that 
> > happened in -stable; at least in the form it happened. I still strongly 
> > believe it's something that's not a stable material. But that's happening 
> > in parallel in a different thread already.
> > 
> > Okay, if you want another example:
> > 
> > 	commit a6aa749906b92eaec6ca0469f90f35de26044d90
> > 	Author: Zhenzhong Duan <zhenzhong.duan@oracle.com>
> > 	Date:   Thu Dec 20 15:05:14 2012 -0800
> > 
> > 	    drivers/firmware/dmi_scan.c: fetch dmi version from SMBIOS if it exists
> > 
> > While this is a correct fix for major kernel release, as it achieves 
> > correctness by checking SMBIOS version properly and behaving according to 
> > the spec, it actually causes an userspace ABI regression in some sense, as 
> > it just changes byte order of /sys/class/dmi/id/product_uuid on certain 
> > systems.
> > 
> > Which is something I absolutely *do not* expect from a minor release of 
> > branch which is called "stable".
> 
> As you pointed out, your definition of "stable" is one that the
> enterprise distros have created, and might be totally different from my
> view of what "stable" is :)

Absolutely, this is why I think discussing this would be very healthy.

- first to clarify what the consumers of your branch are expecting it to 
  be
- then making clear what your understanding of "stable" is
- then clarifying whether the stable branch is there for you or for its 
  consumers :)
- finally finding some intersection (some time in the future :) ) that'll 
  suit all parties

> And also, as Ben pointed out, this was probably wrong, and someone
> should have told me about this, so I could have reverted it.  Please do
> so the next time.

I thought that one of your golas was to improve scalability.

"Everyone is reviewing everything" is not my understanding of scalability; 
hence my proposals.

Thanks, Greg.

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Josh Boyer]

On Tue, Jul 16, 2013 at 3:43 AM, James Bottomley
<James.Bottomley@hansenpartnership.com> wrote:
> On Mon, 2013-07-15 at 23:20 -0700, Greg KH wrote:
>> On Tue, Jul 16, 2013 at 09:17:32AM +0400, James Bottomley wrote:
>> > On Mon, 2013-07-15 at 14:44 -0700, Greg KH wrote:
>> > > On Mon, Jul 15, 2013 at 11:27:56PM +0400, James Bottomley wrote:
>> > > > Before the "3.10.1-stable review" thread degenerated into a disagreement
>> > > > about habits of politeness, there were some solid points being made
>> > > > which, I think, bear consideration and which may now be lost.
>> > > >
>> > > > The problem, as Ji???? Kosina put is succinctly is that the distributions
>> > > > are finding stable less useful because it contains to much stuff they'd
>> > > > classify as not stable material.
>>
>> And I'm going to be working on that.
>
> OK, I accept that.
>
>> But I need, from the distros, specific examples of what they object to.
>> So far all I've gotten is one security patch (that was needed), and one
>> patch for sysfs that I backported too far in the version numbers (my
>> fault.)
>>
>> Given the huge number of stable patches over the past few years, only
>> having 2 patches be an issue sounds like things are working well for
>> people here.
>>
>> If I don't get pushback, with specifics, from the distros, I will not
>> know what to change, so if people can provide me that, it will help out
>> a lot.
>
> I agree ... I think Jiří and his Red Hat equivalent need to pipe up and
> give us more examples of the problems they've been having.

I'm actually more concerned about the glut of patches showing up in
the .1 releases because people didn't bother to send fixes to Linus
for the .0 release.  I think it is a clear sign that the stable tree
is working fairly well when we don't bother to push out a rebase to a
new release until the .1 or .2 stable tree is out there.  It's a
warning sign for the actual .0 release though.

Now, that being said, there are cases where .1 can be broken too.
There have been a couple of changes that showed up during a merge
window that were sucked into -stable through the CC tag, and then
later found to be broken.  Those resulted in reverts or significant
fixes that then took a while to get reverted in -stable or required
different temporary fixes because the upstream fix was too invasive.
That's mainly a problem of them not getting enough testing before
hitting Linus' tree I suppose, but the CC: stable tag being on them
doesn't help.

Forgive me for not having immediate concrete examples here.  I'm
sitting in the middle of nowhere on a lake using tethering for
internet and hiding from my wife so she doesn't catch me doing "work".
 Crap, I think she's coming...

josh

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[John W. Linville]

On Tue, Jul 16, 2013 at 11:43:36AM +0400, James Bottomley wrote:
> On Mon, 2013-07-15 at 23:20 -0700, Greg KH wrote:
> 
> > I think the real stable issue that _everyone_ keeps ignoring, is my
> > original complaint, in that people are using the -rc1 merge window to
> > get fixes in they should have sent to Linus for .0.
> 
> You mean we delay fixes to the merge window (tagged for stable) because
> we can't get them into Linus' tree at -rc5 on?  Guilty ... that's
> because the friction for getting stuff in rises.  It's a big fight to
> get something marginal in after -rc5 ... it's easy to silently tag it
> for stable (did I mention that I think the tag part enables this
> behaviour?).

I'll just chime-in here and agree that I have delayed minor fixes,
preventing them from being merged during an -rc6 or -rc7 but tagging
them for -stable.  It has long been 'tradition' (at least in the
networking space) that fixes so late in the cycle should tend to be
really small (i.e. "one-liners") and/or really, really, important.
In other words, they need to avoid potentially delaying a release
unless they are absolutely necessary.

Fixes merged early in the next release cycle at least have a fighting
chance of getting some testing before getting into the hands of the
unwashed masses.  If they have problems in 3.<previous>.1 then they
can still be reverted in -stable, but they can never be removed from
the .0 release -- does this matter?  I'm not sure.  Is having a flood
of fixes in x.y.1 any worse than having to got to an -rc8 or an -rc9?

John
-- 
John W. Linville		Someday the world will need a hero, and you
linville@tuxdriver.com			might be all we have.  Be ready.

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Kalle Valo]

"John W. Linville" <linville@tuxdriver.com> writes:

> Is having a flood of fixes in x.y.1 any worse than having to got to an
> -rc8 or an -rc9?

I think it's better to send less fixes to -rc8 or -rc9 and focus more on
testing. That way there should be less regressions in later stages of
-rc releases and especially in the final x.y.0 release.

Most of the time I don't even consider updating to the stable releases,
I just wait for the next .0 release from Linus. For this kind of
workflow there is a clear advantage of slowing down patch flow before
the release, better chances of not needing to update the kernel that
often.

-- 
Kalle Valo

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Jan Kara]

On Mon 15-07-13 23:20:58, Greg KH wrote:
> I think the real stable issue that _everyone_ keeps ignoring, is my
> original complaint, in that people are using the -rc1 merge window to
> get fixes in they should have sent to Linus for .0.
> 
> I don't see anything you have written so far that will help resolve that
> issue, which is not good, as that needs to be taken care of.
  Well, I remember Linus has said several times in case of more risky fixes
that he'd rather see them in -rc1 with -stable tag than merging them (very)
late in the -rc series. And frankly if the patch gets in in the merge
window it doesn't get too much real testing before it is released in stable
so how to treat such patches is a question. So I think we need to find a
balance between what you and Linus ask for.

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[David Lang]

On Mon, 15 Jul 2013, Greg KH wrote:

> I don't like this at all, just for the simple reason that it will push
> the majority of the work of stable kernel development on to the
> subsystem maintainers, who have enough work to do as it is.
>
> Stable tree stuff should cause almost _no_ extra burden on the kernel
> developers, because it is something that I, and a few other people, have
> agreed to do with our time.  It has taken me 8 _years_ to finally get
> maintainers to agree to mark stuff for the stable tree, and fine-tune a
> development process that makes it easy for us to do this backport work.

It's important to remember that the subsystem maintainers have essentially no 
input on how many stable trees there are or how long they are maintained.

That is entirely up to the people maintaining the stable trees.

Just because some crazy person ;-) decides to maintain 2.4 for many years 
doesn't mean that every subsystem maintainer needs to worry about backporting 
patches from 3.11 all the way back to 2.4. The fact that they are as willing as 
they are to review the results of the backports for problems in amazing.

David Lang

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Steven Rostedt]

On Mon, 2013-07-15 at 23:24 -0700, David Lang wrote:

> Just because some crazy person ;-) decides to maintain 2.4 for many years 
> doesn't mean that every subsystem maintainer needs to worry about backporting 
> patches from 3.11 all the way back to 2.4. The fact that they are as willing as 
> they are to review the results of the backports for problems in amazing.

Any "process" that we come up with for maintainers with respect to the
stable tree, should only matter to the official ones "last release" and
"last release-1". All others are the responsibility of those maintaining
it. Long term stable trees shouldn't burden the maintainer unless they
want to help.

I don't expect much help from maintainers for the 3.6 stable I maintain.
Any help I do get I'm greatly appreciated of. I also love it when
maintainers add which versions of the kernel a bug fix goes back too.
That is, I love seeing

 Cc: stable@vger.kernel.org # 3.2+

instead of just:

 Cc: stable@vger.kernel.org


But even that, I'm glad to see and don't expect to. The maintainers
don't need that burden as well. I'll read the change log and try to
figure out if it's required or not. And if it applies, I add it,
otherwise, I skip it.

-- Steve

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 3957 bytes --]

On Mon, 2013-07-15 at 23:27 +0400, James Bottomley wrote:
> Before the "3.10.1-stable review" thread degenerated into a disagreement
> about habits of politeness, there were some solid points being made
> which, I think, bear consideration and which may now be lost.
> 
> The problem, as Jiří Kosina put is succinctly is that the distributions
> are finding stable less useful because it contains to much stuff they'd
> classify as not stable material.

Well, some distributions may find that.  Some distribution maintainers
have taken on stable branches, officially or otherwise - Steven Rostedt
(MRG), several Ubuntu developers, and myself (Debian), and of course
Greg was working for SUSE not so long ago.

Myself, I suspect I've been too eager to backport fixes to 3.2 and that
this has led to some avoidable regressions.  I hope I'm striking
approximately the right balance now.

> The question that arises from this is who is stable aiming at ...
> because if it's the distributions (and that's what people seem to be
> using it for) then we need to take this feedback seriously.
> 
> The next question is how should we, the maintainers, be policing commits
> to stable.  As I think has been demonstrated in the discussion the
> "stable rules" are more sort of guidelines (apologies for the pirates
> reference).  In many ways, this is as it should be, because people
> should have enough taste to know what constitutes a stable fix.  The
> real root cause of the problem is that the cc: stable tag can't be
> stripped once it's in the tree, so maintainers only get to police things
> they put in the tree.  Stuff they pull from others is already tagged and
> that tag can't be changed.

Sure it can.  Subsystem maintainers can reject a pull request and demand
changes that include rebasing.  (But only reasonably for leaf
repositories.)

> This effectively pushes the problem out to
> the lowest (and possibly more inexperienced) leaves of the Maintainer
> tree.  In theory we have a review stage for stable, but the review
> patches don't automatically get routed to the right mailing list

This is a simple matter of programming.

> and the
> first round usually comes out in the merge window when Maintainers'
> attention is elsewhere.

That is a real problem.  It's also a time when those changes have not
had much testing together.  (Yes there is linux-next, but I seriously
doubt it gets as much run-time testing as mainline.)

> The solution, to me, looks simple:  Let's co-opt a process we already
> know how to do: mailing list review and tree handling.  So the proposal
> is simple:
> 
>      1. Drop the cc: stable@ tag: it makes it way too easy to add an ill
>         reviewed patch to stable
>      2. All patches to stable should follow current review rules: They
>         should go to the mailing list the original patch was sent to
>         once the original is upstream as a request for stable.
>      3. Following debate on the list, the original maintainer would be
>         responsible for collecting the patches (including the upstream
>         commit) adjudicating on them and passing them on to stable after
>         list review (either by git tree pull or email to stable@).
>
> I contend this raises the bar for adding patches to stable much higher,
> which seems to be needed,

I think it would be an over-correction...

> and adds a review stage which involves all the original reviewers.

...but this is important.  Also, I'm not convinced that the current
standard review period (48 hours starting whenever) is really long
enough.  Especially when there can be 4 large series at once (for 3.0,
3.4, 3.latest-1 and 3.latest) or even more if other stable maintainers
start overlapping reviews.

Ben.

> Oh, and did someone mention plum brandy ...?

-- 
Ben Hutchings
Humans are not rational beings; they are rationalising beings.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1177 bytes --]

On Tue, Jul 16, 2013 at 03:00:29AM +0100, Ben Hutchings wrote:
> On Mon, 2013-07-15 at 23:27 +0400, James Bottomley wrote:

> > The problem, as Jiří Kosina put is succinctly is that the distributions
> > are finding stable less useful because it contains to much stuff they'd
> > classify as not stable material.

> Well, some distributions may find that.  Some distribution maintainers
> have taken on stable branches, officially or otherwise - Steven Rostedt
> (MRG), several Ubuntu developers, and myself (Debian), and of course
> Greg was working for SUSE not so long ago.

There's going to be people using it via the stable kernels from Linaro
too soon (mostly underlying distros provided by silicon vendors for
embedded systems in the first instance).

> ...but this is important.  Also, I'm not convinced that the current
> standard review period (48 hours starting whenever) is really long
> enough.  Especially when there can be 4 large series at once (for 3.0,
> 3.4, 3.latest-1 and 3.latest) or even more if other stable maintainers
> start overlapping reviews.

I definitely agree with this, it can be a bit of a DoS when things are
busy.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Rob Landley]

On 07/15/2013 02:27:56 PM, James Bottomley wrote:
> Before the "3.10.1-stable review" thread degenerated into a  
> disagreement
> about habits of politeness, there were some solid points being made
> which, I think, bear consideration and which may now be lost.
> 
> The problem, as Jiří Kosina put is succinctly is that the  
> distributions
> are finding stable less useful because it contains to much stuff  
> they'd
> classify as not stable material.

Does anybody actually review hundred-patch dumps for stable stuff? I  
don't. Any regression fixes or security fixes are buried in 90% chaff.  
"Could be improved" is why you upgrade to new versions, yet "random  
improvements that we don't think will break anything' is the vast  
majority of stable.

Bar's a bit low for my tastes. Oh well.

Rob

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 934 bytes --]

On Sat, 2013-07-20 at 23:11 -0500, Rob Landley wrote:
> On 07/15/2013 02:27:56 PM, James Bottomley wrote:
> > Before the "3.10.1-stable review" thread degenerated into a  
> > disagreement
> > about habits of politeness, there were some solid points being made
> > which, I think, bear consideration and which may now be lost.
> > 
> > The problem, as Jiří Kosina put is succinctly is that the  
> > distributions
> > are finding stable less useful because it contains to much stuff  
> > they'd
> > classify as not stable material.
> 
> Does anybody actually review hundred-patch dumps for stable stuff?
[...]

I review what I can, but recently have often missed the 2 day review
period.

Review from the authors and maintainers is probably more valuable than
that from generalists on the stable list.

Ben.

-- 
Ben Hutchings
The most exhausting thing in life is being insincere. - Anne Morrow Lindberg

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[KOSAKI Motohiro]

> I review what I can, but recently have often missed the 2 day review
> period.
>
> Review from the authors and maintainers is probably more valuable than
> that from generalists on the stable list.

>From point of subsystem developers view, the problem is there are too many
stable branches. I can't memorize what is different each other.
Currently www.kernel.org
lists two stable and five longterm branches. Who know all of them?
Eventually I only
checked them when I have enough much spare time.

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Li Zefan]

On 2013/7/23 5:24, KOSAKI Motohiro wrote:
>> I review what I can, but recently have often missed the 2 day review
>> period.
>>
>> Review from the authors and maintainers is probably more valuable than
>> that from generalists on the stable list.
> 
>>From point of subsystem developers view, the problem is there are too many
> stable branches. I can't memorize what is different each other.
> Currently www.kernel.org
> lists two stable and five longterm branches. Who know all of them?
> Eventually I only
> checked them when I have enough much spare time.

I only check those maintained by Greg.

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Li Zefan]

On 2013/7/23 5:24, KOSAKI Motohiro wrote:
>> I review what I can, but recently have often missed the 2 day review
>> period.
>>
>> Review from the authors and maintainers is probably more valuable than
>> that from generalists on the stable list.
> 
>>>From point of subsystem developers view, the problem is there are too many
> stable branches. I can't memorize what is different each other.
> Currently www.kernel.org
> lists two stable and five longterm branches. Who know all of them?
> Eventually I only
> checked them when I have enough much spare time.

I only check those maintained by Greg.

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Myklebust, Trond]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 3155 bytes --]

On Mon, 2013-07-15 at 23:27 +0400, James Bottomley wrote:
> Before the "3.10.1-stable review" thread degenerated into a disagreement
> about habits of politeness, there were some solid points being made
> which, I think, bear consideration and which may now be lost.
> 
> The problem, as Jiří Kosina put is succinctly is that the distributions
> are finding stable less useful because it contains to much stuff they'd
> classify as not stable material.
> 
> The question that arises from this is who is stable aiming at ...
> because if it's the distributions (and that's what people seem to be
> using it for) then we need to take this feedback seriously.
> 
> The next question is how should we, the maintainers, be policing commits
> to stable.  As I think has been demonstrated in the discussion the
> "stable rules" are more sort of guidelines (apologies for the pirates
> reference).  In many ways, this is as it should be, because people
> should have enough taste to know what constitutes a stable fix.  The
> real root cause of the problem is that the cc: stable tag can't be
> stripped once it's in the tree, so maintainers only get to police things
> they put in the tree.  Stuff they pull from others is already tagged and
> that tag can't be changed.  This effectively pushes the problem out to
> the lowest (and possibly more inexperienced) leaves of the Maintainer
> tree.  In theory we have a review stage for stable, but the review
> patches don't automatically get routed to the right mailing list and the
> first round usually comes out in the merge window when Maintainers'
> attention is elsewhere.
> 
> The solution, to me, looks simple:  Let's co-opt a process we already
> know how to do: mailing list review and tree handling.  So the proposal
> is simple:
> 
>      1. Drop the cc: stable@ tag: it makes it way too easy to add an ill
>         reviewed patch to stable
>      2. All patches to stable should follow current review rules: They
>         should go to the mailing list the original patch was sent to
>         once the original is upstream as a request for stable.
>      3. Following debate on the list, the original maintainer would be
>         responsible for collecting the patches (including the upstream
>         commit) adjudicating on them and passing them on to stable after
>         list review (either by git tree pull or email to stable@).
> 
> I contend this raises the bar for adding patches to stable much higher,
> which seems to be needed, and adds a review stage which involves all the
> original reviewers.

Could we keep the Cc: stable tag itself, since the dependency
information ("Cc: <stable@vger.kernel.org> # 3.3.x: a1f84a3: sched:
Check for idle") is actually very useful? If we discard that, then we
really should revise the whole stable system, since it would mean that
we are in effect discarding the 'upstream first' rule.
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Myklebust, Trond]

On Mon, 2013-07-15 at 23:27 +0400, James Bottomley wrote:
> Before the "3.10.1-stable review" thread degenerated into a disagreement
> about habits of politeness, there were some solid points being made
> which, I think, bear consideration and which may now be lost.
> 
> The problem, as Jiří Kosina put is succinctly is that the distributions
> are finding stable less useful because it contains to much stuff they'd
> classify as not stable material.
> 
> The question that arises from this is who is stable aiming at ...
> because if it's the distributions (and that's what people seem to be
> using it for) then we need to take this feedback seriously.
> 
> The next question is how should we, the maintainers, be policing commits
> to stable.  As I think has been demonstrated in the discussion the
> "stable rules" are more sort of guidelines (apologies for the pirates
> reference).  In many ways, this is as it should be, because people
> should have enough taste to know what constitutes a stable fix.  The
> real root cause of the problem is that the cc: stable tag can't be
> stripped once it's in the tree, so maintainers only get to police things
> they put in the tree.  Stuff they pull from others is already tagged and
> that tag can't be changed.  This effectively pushes the problem out to
> the lowest (and possibly more inexperienced) leaves of the Maintainer
> tree.  In theory we have a review stage for stable, but the review
> patches don't automatically get routed to the right mailing list and the
> first round usually comes out in the merge window when Maintainers'
> attention is elsewhere.
> 
> The solution, to me, looks simple:  Let's co-opt a process we already
> know how to do: mailing list review and tree handling.  So the proposal
> is simple:
> 
>      1. Drop the cc: stable@ tag: it makes it way too easy to add an ill
>         reviewed patch to stable
>      2. All patches to stable should follow current review rules: They
>         should go to the mailing list the original patch was sent to
>         once the original is upstream as a request for stable.
>      3. Following debate on the list, the original maintainer would be
>         responsible for collecting the patches (including the upstream
>         commit) adjudicating on them and passing them on to stable after
>         list review (either by git tree pull or email to stable@).
> 
> I contend this raises the bar for adding patches to stable much higher,
> which seems to be needed, and adds a review stage which involves all the
> original reviewers.

Could we keep the Cc: stable tag itself, since the dependency
information ("Cc: <stable@vger.kernel.org> # 3.3.x: a1f84a3: sched:
Check for idle") is actually very useful? If we discard that, then we
really should revise the whole stable system, since it would mean that
we are in effect discarding the 'upstream first' rule.
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[James Bottomley]

On Tue, 2013-07-23 at 02:40 +0000, Myklebust, Trond wrote:
> On Mon, 2013-07-15 at 23:27 +0400, James Bottomley wrote:
> > The solution, to me, looks simple:  Let's co-opt a process we already
> > know how to do: mailing list review and tree handling.  So the proposal
> > is simple:
> > 
> >      1. Drop the cc: stable@ tag: it makes it way too easy to add an ill
> >         reviewed patch to stable
> >      2. All patches to stable should follow current review rules: They
> >         should go to the mailing list the original patch was sent to
> >         once the original is upstream as a request for stable.
> >      3. Following debate on the list, the original maintainer would be
> >         responsible for collecting the patches (including the upstream
> >         commit) adjudicating on them and passing them on to stable after
> >         list review (either by git tree pull or email to stable@).
> > 
> > I contend this raises the bar for adding patches to stable much higher,
> > which seems to be needed, and adds a review stage which involves all the
> > original reviewers.
> 
> Could we keep the Cc: stable tag itself, since the dependency
> information ("Cc: <stable@vger.kernel.org> # 3.3.x: a1f84a3: sched:
> Check for idle") is actually very useful? If we discard that, then we
> really should revise the whole stable system, since it would mean that
> we are in effect discarding the 'upstream first' rule.

The two don't follow.  No-one's proposing to dump the must be upstream
rule.  The proposal is to modify the automatic behaviour that leads to
over tagging for stable and consequently too many "stable" patches that
aren't really.

James

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[James Bottomley]

On Tue, 2013-07-23 at 02:40 +0000, Myklebust, Trond wrote:
> On Mon, 2013-07-15 at 23:27 +0400, James Bottomley wrote:
> > The solution, to me, looks simple:  Let's co-opt a process we already
> > know how to do: mailing list review and tree handling.  So the proposal
> > is simple:
> > 
> >      1. Drop the cc: stable@ tag: it makes it way too easy to add an ill
> >         reviewed patch to stable
> >      2. All patches to stable should follow current review rules: They
> >         should go to the mailing list the original patch was sent to
> >         once the original is upstream as a request for stable.
> >      3. Following debate on the list, the original maintainer would be
> >         responsible for collecting the patches (including the upstream
> >         commit) adjudicating on them and passing them on to stable after
> >         list review (either by git tree pull or email to stable@).
> > 
> > I contend this raises the bar for adding patches to stable much higher,
> > which seems to be needed, and adds a review stage which involves all the
> > original reviewers.
> 
> Could we keep the Cc: stable tag itself, since the dependency
> information ("Cc: <stable@vger.kernel.org> # 3.3.x: a1f84a3: sched:
> Check for idle") is actually very useful? If we discard that, then we
> really should revise the whole stable system, since it would mean that
> we are in effect discarding the 'upstream first' rule.

The two don't follow.  No-one's proposing to dump the must be upstream
rule.  The proposal is to modify the automatic behaviour that leads to
over tagging for stable and consequently too many "stable" patches that
aren't really.

James

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Myklebust, Trond]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 2472 bytes --]

On Mon, 2013-07-22 at 19:47 -0700, James Bottomley wrote:
> On Tue, 2013-07-23 at 02:40 +0000, Myklebust, Trond wrote:
> > On Mon, 2013-07-15 at 23:27 +0400, James Bottomley wrote:
> > > The solution, to me, looks simple:  Let's co-opt a process we already
> > > know how to do: mailing list review and tree handling.  So the proposal
> > > is simple:
> > > 
> > >      1. Drop the cc: stable@ tag: it makes it way too easy to add an ill
> > >         reviewed patch to stable
> > >      2. All patches to stable should follow current review rules: They
> > >         should go to the mailing list the original patch was sent to
> > >         once the original is upstream as a request for stable.
> > >      3. Following debate on the list, the original maintainer would be
> > >         responsible for collecting the patches (including the upstream
> > >         commit) adjudicating on them and passing them on to stable after
> > >         list review (either by git tree pull or email to stable@).
> > > 
> > > I contend this raises the bar for adding patches to stable much higher,
> > > which seems to be needed, and adds a review stage which involves all the
> > > original reviewers.
> > 
> > Could we keep the Cc: stable tag itself, since the dependency
> > information ("Cc: <stable@vger.kernel.org> # 3.3.x: a1f84a3: sched:
> > Check for idle") is actually very useful? If we discard that, then we
> > really should revise the whole stable system, since it would mean that
> > we are in effect discarding the 'upstream first' rule.
> 
> The two don't follow.  No-one's proposing to dump the must be upstream
> rule.  The proposal is to modify the automatic behaviour that leads to
> over tagging for stable and consequently too many "stable" patches that
> aren't really.

My point was that the _tag_ is useful as a list of dependencies for
something that we thing might need to be backported to older kernels.
I'd like to see us keep that information somehow.

Whether or not we interpret it as being an automatic "for stable"
request is a different matter. I'd be quite happy to see the "propose
for stable" step as reverting to being a manual step that occurs only
after we've upstreamed the fix.

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

Re: [Ksummit-2013-discuss] KS Topic request: Handling the Stable kernel, let's dump the cc: stable tag

[Myklebust, Trond]

On Mon, 2013-07-22 at 19:47 -0700, James Bottomley wrote:
> On Tue, 2013-07-23 at 02:40 +0000, Myklebust, Trond wrote:
> > On Mon, 2013-07-15 at 23:27 +0400, James Bottomley wrote:
> > > The solution, to me, looks simple:  Let's co-opt a process we already
> > > know how to do: mailing list review and tree handling.  So the proposal
> > > is simple:
> > > 
> > >      1. Drop the cc: stable@ tag: it makes it way too easy to add an ill
> > >         reviewed patch to stable
> > >      2. All patches to stable should follow current review rules: They
> > >         should go to the mailing list the original patch was sent to
> > >         once the original is upstream as a request for stable.
> > >      3. Following debate on the list, the original maintainer would be
> > >         responsible for collecting the patches (including the upstream
> > >         commit) adjudicating on them and passing them on to stable after
> > >         list review (either by git tree pull or email to stable@).
> > > 
> > > I contend this raises the bar for adding patches to stable much higher,
> > > which seems to be needed, and adds a review stage which involves all the
> > > original reviewers.
> > 
> > Could we keep the Cc: stable tag itself, since the dependency
> > information ("Cc: <stable@vger.kernel.org> # 3.3.x: a1f84a3: sched:
> > Check for idle") is actually very useful? If we discard that, then we
> > really should revise the whole stable system, since it would mean that
> > we are in effect discarding the 'upstream first' rule.
> 
> The two don't follow.  No-one's proposing to dump the must be upstream
> rule.  The proposal is to modify the automatic behaviour that leads to
> over tagging for stable and consequently too many "stable" patches that
> aren't really.

My point was that the _tag_ is useful as a list of dependencies for
something that we thing might need to be backported to older kernels.
I'd like to see us keep that information somehow.

Whether or not we interpret it as being an automatic "for stable"
request is a different matter. I'd be quite happy to see the "propose
for stable" step as reverting to being a manual step that occurs only
after we've upstreamed the fix.

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com