Future of the -longterm kernel releases (i.e. how we pick them).

Future of the -longterm kernel releases (i.e. how we pick them).

[Greg KH]

As I'm giving a talk about the -stable and -longterm kernels this week
at LinuxCon in Vancouver, and I've been talking with a lot of different
people about the future of the -longterm kernels, here's some thoughts
as to what I'm considering:


tl;dr;
	* -stable kernel releases stay the same
	* this proposal is how we pick the -longterm releases
	* -longterm kernels will be picked every year, and maintained
	  for 2 years before being dropped.
	* the same Documentation/stable_kernel_rules.txt will apply for
	  -longterm kernels, as before.

History:

2.6.16 became a "longterm" kernel because my day job (at SUSE) picked
the 2.6.16 kernel for its "enterprise" release and it made things a lot
easier for me to keep working at applying bugfixes and other stable
patches to it to make my job simpler (applying a known-good bunch of
patches in one stable update was easier than a set of smaller patches
that were only tested by a smaller group of people.)

Seeing that this worked well, a cabal of developers got together at a
few different Linux conferences and determined that based on their
future distro release cycles, we could all aim for standardizing on the
2.6.32 kernel, saving us all time and energy in the long run.  We turned
around and planted the proper seeds within the different organizations
and low-and-behold, project managers figured that this was their idea
and sold it to the rest of the groups and made it happen.  Right now all
of the major "enterprise" and "stable" distro releases are based on the
2.6.32 kernel, making this trial a huge success.

Last year, two different community members (Andi and Paul) asked me
if they could maintain the 2.6.34 and 2.6.35 kernels as -longterm kernel
releases as their companies needed this type of support.  I agreed, and
they have done a great job at this.

Andi reports that the 2.6.35 kernel is being used by a number of
different distros, but they will be phased out as their support lifetime
expires.  There are also a number of embedded users of the kernel as
well as some individual ones.  So that -longterm kernel is having a lot
of benefit for a wide range of users.


Today:

Now that 2.6.32 is over a year and a half, and the enterprise distros
are off doing their thing with their multi-year upgrade cycles, there's
no real need from the distros for a new longterm kernel release.  But it
turns out that the distros are not the only user of the kernel, other
groups and companies have been approaching me over the past year, asking
how they could pick the next longterm kernel, or what the process is in
determining this.

To keep this all out in the open, let's figure out what to do here.
Consumer devices have a 1-2 year lifespan, and want and need the
experience of the kernel community maintaining their "base" kernel for
them.  There is no real "enterprise" embedded distro out there from what
I can see.  montaVista and WindRiver have some offerings in this area, but
they are not that widely used and are usually more "deep embedded".
There's also talk that the CELF group and Linaro are wanting to do
something on a "longterm" basis, and are fishing around for how to
properly handle this with the community to share the workload.  Android
also is another huge player here, upgrading their kernel every major
release, and they could use the support of a longterm kernel as well.

Proposal:

Here's a first cut at a proposal, let me know if you like it, hate it,
would work for you and your company, or not at all:

- a new -longterm kernel is picked every year.
- a -longterm kernel is maintained for 2 years and then dropped.
- -stable kernels keep the same schedule that they have been (dropping
  the last one after a new release happens.)  These releases are best
  for products that require new hardware updates (desktop distros,
  community distros, fast-moving embedded distros (like Yocto)).
- the normal -stable rules apply to these -longterm kernels as described
  in Documentation/stable_kernel_rules.txt

This means that there are 2 -longterm kernels being maintained at the
same time, and one -stable kernel.  I'm volunteering to do this work, as
it's pretty much what I'm doing today anyway, and I have all of the
scripts and workflow down.

Public Notifications:

The current kernel.org site doesn't properly show what is and is not
being maintained as a -stable and -longterm kernel.  I have a proposal
for how to fix this involving 'git notes', I just need to sit down and
do the work with the kernel.org admins to get this running properly.

Thoughts?


greg k-h

Re: Future of the -longterm kernel releases (i.e. how we pick them).

[Jörg-Volker Peetz]

Greg KH wrote, on 08/15/11 06:15:
<snip>
> 
> - a new -longterm kernel is picked every year.
> - a -longterm kernel is maintained for 2 years and then dropped.
<snip>

Just a little nitpick: with this scheme you will accumulate longterm kernels. If
I understand it right, one longterm kernel is added every year.

Re: Future of the -longterm kernel releases (i.e. how we pick them).

[david]

On Mon, 15 Aug 2011, J?rg-Volker Peetz wrote:

> Greg KH wrote, on 08/15/11 06:15:
> <snip>
>>
>> - a new -longterm kernel is picked every year.
>> - a -longterm kernel is maintained for 2 years and then dropped.
> <snip>
>
> Just a little nitpick: with this scheme you will accumulate longterm kernels. If
> I understand it right, one longterm kernel is added every year.

but with one new kernel being added each year, and then being dropped two 
years later, there are only 2 long term kernels at any one time (that Greg 
will be maintaining at least, nothing stops other people from maintaining 
other long term kernels in addition)

David Lang

Re: Future of the -longterm kernel releases (i.e. how we pick them).

[Teck Choon Giam]

2011/8/15  <david@lang.hm>:
> On Mon, 15 Aug 2011, J?rg-Volker Peetz wrote:
>
>> Greg KH wrote, on 08/15/11 06:15:
>> <snip>
>>>
>>> - a new -longterm kernel is picked every year.
>>> - a -longterm kernel is maintained for 2 years and then dropped.
>>
>> <snip>
>>
>> Just a little nitpick: with this scheme you will accumulate longterm
>> kernels. If
>> I understand it right, one longterm kernel is added every year.
>
> but with one new kernel being added each year, and then being dropped two
> years later, there are only 2 long term kernels at any one time (that Greg
> will be maintaining at least, nothing stops other people from maintaining
> other long term kernels in addition)

Err... sorry from my understanding like this year... one new long term
kernel added so it is N+1 then next year is N+1+1 and two years drop
one so it is N+1+1-1... so every year there will be one new long term
added and every two years there will be one overall long term kernel
added to total number of long term kernels... since two are added
within 2 years and one dropped every 2 years.  So there are not only 2
long term kernels at any one time in this case... ... someone correct
me if I am wrong.

Thanks.

Kindest regards,
Giam Teck Choon

Re: Future of the -longterm kernel releases (i.e. how we pick them).

[Jörg-Volker Peetz]

Jörg-Volker Peetz wrote, on 08/15/11 08:48:
> Greg KH wrote, on 08/15/11 06:15:
> <snip>
>>
>> - a new -longterm kernel is picked every year.
>> - a -longterm kernel is maintained for 2 years and then dropped.
> <snip>
> 
> Just a little nitpick: with this scheme you will accumulate longterm kernels. If
> I understand it right, one longterm kernel is added every year.

Correction: this should read "one longterm kernel is added every two years".
-- 
Best regards,
Jörg-Volker.

Re: Future of the -longterm kernel releases (i.e. how we pick them).

[david]

On Sun, 14 Aug 2011, Greg KH wrote:

> Proposal:
>
> Here's a first cut at a proposal, let me know if you like it, hate it,
> would work for you and your company, or not at all:
>
> - a new -longterm kernel is picked every year.
> - a -longterm kernel is maintained for 2 years and then dropped.
> - -stable kernels keep the same schedule that they have been (dropping
>  the last one after a new release happens.)  These releases are best
>  for products that require new hardware updates (desktop distros,
>  community distros, fast-moving embedded distros (like Yocto)).
> - the normal -stable rules apply to these -longterm kernels as described
>  in Documentation/stable_kernel_rules.txt
>
> This means that there are 2 -longterm kernels being maintained at the
> same time, and one -stable kernel.  I'm volunteering to do this work, as
> it's pretty much what I'm doing today anyway, and I have all of the
> scripts and workflow down.

I am one of the people who runs kernel.org kernels on my companies 
systems, and so I think I am well within the target of this proposal.

First off, let me say 'thank you' for your work here, now on to the ugly 
details :-)

There is definantly a need for a kernel that's maintained longer than the 
current -stable series is, there just isn't enough time to get comfortable 
with a new kernel before the -stable stops being maintained. What I end up 
doing is watching the vendor lists for alerts on things that relate to my 
configurations, and when I see them, work to jump to the latest kernel. 
this works, but not especially well.

a more regular -longterm kernel is appealing, but as I have been doing 
this since the kernel 1.3 days, I have seen a lot of kernels that ended up 
with just not being that good in subtle ways, and I'm not sure that just 
trying to backport smallish patches can really solve the problem.

rather than having a hard schedule (the first kernel released after July 1 
each year for example I know this is not the exact proposal), I think that 
it would be better to pick the -longterm kernel a few months after it has 
been released (3.4 is looking very good, the normal minor driver fixes in 
-stable, but no fundamental regressions have been reported, it's the new 
-longerm kernel for 
example)

doing so doesn't give the predictability that some people will want in 
knowing that their September release will always have a fresh new -longerm 
kernel, but I think the result would be better -longterm kernels. However, 
to get the information about how good the kernels are, I think that the 
-stable timeframe would need to be extended to give the kernels time to 
settle and gather reports. I would then suggest scheduling that once a 
year you look at the last couple -stable kernels and pick one of them 
rather than designating the new -longterm kernel ahead of time.

I hope my midnight rambling makes some sort of sense :-)

David Lang

Re: Future of the -longterm kernel releases (i.e. how we pick them).

[david]

On Mon, 15 Aug 2011, Teck Choon Giam wrote:

> 2011/8/15  <david@lang.hm>:
>> On Mon, 15 Aug 2011, J?rg-Volker Peetz wrote:
>>
>>> Greg KH wrote, on 08/15/11 06:15:
>>> <snip>
>>>>
>>>> - a new -longterm kernel is picked every year.
>>>> - a -longterm kernel is maintained for 2 years and then dropped.
>>>
>>> <snip>
>>>
>>> Just a little nitpick: with this scheme you will accumulate longterm
>>> kernels. If
>>> I understand it right, one longterm kernel is added every year.
>>
>> but with one new kernel being added each year, and then being dropped two
>> years later, there are only 2 long term kernels at any one time (that Greg
>> will be maintaining at least, nothing stops other people from maintaining
>> other long term kernels in addition)
>
> Err... sorry from my understanding like this year... one new long term
> kernel added so it is N+1 then next year is N+1+1 and two years drop
> one so it is N+1+1-1... so every year there will be one new long term
> added and every two years there will be one overall long term kernel
> added to total number of long term kernels... since two are added
> within 2 years and one dropped every 2 years.  So there are not only 2
> long term kernels at any one time in this case... ... someone correct
> me if I am wrong.

in 2011 there will be a -longterm kernel picked (1 maintained)

in 2012 there will be a -longterm kernel picked (1+1=2 maintained)

in 2013 there will be a -longterm kernel picked, but the 2011 one will no 
longer be maintained (1+1+1-1=2 maintained)

in 2014 there will be a -longterm kernel picked, but the 2012 one will no 
longer be maintained (1+1+1+1-1-1=2 maintained)

David Lang

Re: [Stable-review] Future of the -longterm kernel releases (i.e. how we pick them).

[Willy Tarreau]

Hi Greg,

On Sun, Aug 14, 2011 at 09:15:24PM -0700, Greg KH wrote:
> As I'm giving a talk about the -stable and -longterm kernels this week
> at LinuxCon in Vancouver, and I've been talking with a lot of different
> people about the future of the -longterm kernels, here's some thoughts
> as to what I'm considering:
(...)
> Thoughts?

I think this is all good, and knowing in advance what next kernel will
be picked will help a lot of users because they'll be able to focus on
a given version.

I just have one comment about the 2-years : new devices generally ship
with something which is expected to be stable, so they won't ship with
the shiny new 2-digit kernel that was just released. This means that a
two-years lifecycle for the longterm kernels will allow less than 2
years of life for the device.

But possibly there are several types of products :
  - the ones where an upgrade may be forced on the user about once a
    year, so that the kernel can be between 1 and 2-years old
  - the ones that have to live longer without upgrades

I'm realizing that the second type is mostly the case for products
that are not easy to validate and are kept as-is without update as
long as they're supported, which is common in the network appliances
world (this is the reason I picked 2.6.27 ;-)).

Also, since your kernels are pretty stable after 2 years, updates are
quite rare and generally of minor importance. This is also the time I
choose to pick them for use in products where planning a reboot takes
weeks and an upgrade takes many months (I even know some people who are
currently ordering hardware to deploy 2.4). So it's likely that in 6
months I'll be interested in an almost frozen 2.6.32, and I will then
propose you to take it over and prolong its life when you drop it.

And in the end, I think this development model makes a lot of sense :
  - developers need a fast moving target
  - desktop users need something up to date with latest drivers and
    don't need the best stability => latest release or -stable are OK
  - most servers are happy with -stable (mine are)
  - distros need to focus on a recent -longterm and will contribute
    to its stability
  - some products need to ship with a very stable -longterm with
    rare updates

You're interested in the 4th category above and I'm in the 5th. So
in the end, I think that your proposal fits your expected target
perfectly well, and that if I want it to fit mine, I'll just have
to extend it myself as we've done till now. So that sounds good !

Thanks,
Willy

Re: Future of the -longterm kernel releases (i.e. how we pick them).

[Teck Choon Giam]

On Mon, Aug 15, 2011 at 3:25 PM,  <david@lang.hm> wrote:
> On Mon, 15 Aug 2011, Teck Choon Giam wrote:
>
>> 2011/8/15  <david@lang.hm>:
>>>
>>> On Mon, 15 Aug 2011, J?rg-Volker Peetz wrote:
>>>
>>>> Greg KH wrote, on 08/15/11 06:15:
>>>> <snip>
>>>>>
>>>>> - a new -longterm kernel is picked every year.
>>>>> - a -longterm kernel is maintained for 2 years and then dropped.
>>>>
>>>> <snip>
>>>>
>>>> Just a little nitpick: with this scheme you will accumulate longterm
>>>> kernels. If
>>>> I understand it right, one longterm kernel is added every year.
>>>
>>> but with one new kernel being added each year, and then being dropped two
>>> years later, there are only 2 long term kernels at any one time (that
>>> Greg
>>> will be maintaining at least, nothing stops other people from maintaining
>>> other long term kernels in addition)
>>
>> Err... sorry from my understanding like this year... one new long term
>> kernel added so it is N+1 then next year is N+1+1 and two years drop
>> one so it is N+1+1-1... so every year there will be one new long term
>> added and every two years there will be one overall long term kernel
>> added to total number of long term kernels... since two are added
>> within 2 years and one dropped every 2 years.  So there are not only 2
>> long term kernels at any one time in this case... ... someone correct
>> me if I am wrong.
>
> in 2011 there will be a -longterm kernel picked (1 maintained)
>
> in 2012 there will be a -longterm kernel picked (1+1=2 maintained)
>
> in 2013 there will be a -longterm kernel picked, but the 2011 one will no
> longer be maintained (1+1+1-1=2 maintained)
>
> in 2014 there will be a -longterm kernel picked, but the 2012 one will no
> longer be maintained (1+1+1+1-1-1=2 maintained)

Thanks for explaining... :)

Kindest regards,
Giam Teck Choon

Re: Future of the -longterm kernel releases (i.e. how we pick them).

[Jörg-Volker Peetz]

david@lang.hm wrote, on 08/15/11 09:25:
> 
> in 2011 there will be a -longterm kernel picked (1 maintained)
> 
> in 2012 there will be a -longterm kernel picked (1+1=2 maintained)
> 
> in 2013 there will be a -longterm kernel picked, but the 2011 one will no longer
> be maintained (1+1+1-1=2 maintained)
> 
> in 2014 there will be a -longterm kernel picked, but the 2012 one will no longer
> be maintained (1+1+1+1-1-1=2 maintained)
> 
> David Lang

Yes, you are right. My error in reasoning. Please excuse the noise.
-- 
Best regards,
Jörg-Volker.

Re: [stable] [Stable-review] Future of the -longterm kernel releases (i.e. how we pick them).

[Greg KH]

On Mon, Aug 15, 2011 at 09:33:09AM +0200, Willy Tarreau wrote:
> You're interested in the 4th category above and I'm in the 5th. So
> in the end, I think that your proposal fits your expected target
> perfectly well, and that if I want it to fit mine, I'll just have
> to extend it myself as we've done till now. So that sounds good !

Wonderful, thanks for confirming that this would work out for you, and
that you will be willing to take on .32 in 6+ months when I am planning
on dropping it.

Keep up the great work,

greg k-h

Re: Future of the -longterm kernel releases (i.e. how we pick them).

[Greg KH]

On Mon, Aug 15, 2011 at 12:21:59AM -0700, david@lang.hm wrote:
> rather than having a hard schedule (the first kernel released after
> July 1 each year for example I know this is not the exact proposal),
> I think that it would be better to pick the -longterm kernel a few
> months after it has been released (3.4 is looking very good, the
> normal minor driver fixes in -stable, but no fundamental regressions
> have been reported, it's the new -longerm kernel for example)
> 
> doing so doesn't give the predictability that some people will want
> in knowing that their September release will always have a fresh new
> -longerm kernel, but I think the result would be better -longterm
> kernels. However, to get the information about how good the kernels
> are, I think that the -stable timeframe would need to be extended to
> give the kernels time to settle and gather reports. I would then
> suggest scheduling that once a year you look at the last couple
> -stable kernels and pick one of them rather than designating the new
> -longterm kernel ahead of time.

Yes, that's a very good idea.  I've seen problems in the past when
distros have made a time-based decision to pick a kernel version and
then the problems that this can cause if it happens that a subsystem
really had issues for that release.

So yes, I'll take a look at the bug reports and how things are working
out to pick the next -longterm.  I'll also take into consideration any
companies/major users that are going to be using that release as well,
so it greatly behooves people to talk to me about their plans (hint,
hint...)

> I hope my midnight rambling makes some sort of sense :-)

It did, thanks for the response.

greg k-h

Re: [stable] Future of the -longterm kernel releases (i.e. how we pick them).

[Tim Gardner]

On 08/14/2011 10:15 PM, Greg KH wrote:
> As I'm giving a talk about the -stable and -longterm kernels this week
> at LinuxCon in Vancouver, and I've been talking with a lot of different
> people about the future of the -longterm kernels, here's some thoughts
> as to what I'm considering:
>
>
> tl;dr;
> 	* -stable kernel releases stay the same
> 	* this proposal is how we pick the -longterm releases
> 	* -longterm kernels will be picked every year, and maintained
> 	  for 2 years before being dropped.
> 	* the same Documentation/stable_kernel_rules.txt will apply for
> 	  -longterm kernels, as before.
>
> History:
>
> 2.6.16 became a "longterm" kernel because my day job (at SUSE) picked
> the 2.6.16 kernel for its "enterprise" release and it made things a lot
> easier for me to keep working at applying bugfixes and other stable
> patches to it to make my job simpler (applying a known-good bunch of
> patches in one stable update was easier than a set of smaller patches
> that were only tested by a smaller group of people.)
>
> Seeing that this worked well, a cabal of developers got together at a
> few different Linux conferences and determined that based on their
> future distro release cycles, we could all aim for standardizing on the
> 2.6.32 kernel, saving us all time and energy in the long run.  We turned
> around and planted the proper seeds within the different organizations
> and low-and-behold, project managers figured that this was their idea
> and sold it to the rest of the groups and made it happen.  Right now all
> of the major "enterprise" and "stable" distro releases are based on the
> 2.6.32 kernel, making this trial a huge success.
>
> Last year, two different community members (Andi and Paul) asked me
> if they could maintain the 2.6.34 and 2.6.35 kernels as -longterm kernel
> releases as their companies needed this type of support.  I agreed, and
> they have done a great job at this.
>
> Andi reports that the 2.6.35 kernel is being used by a number of
> different distros, but they will be phased out as their support lifetime
> expires.  There are also a number of embedded users of the kernel as
> well as some individual ones.  So that -longterm kernel is having a lot
> of benefit for a wide range of users.
>
>
> Today:
>
> Now that 2.6.32 is over a year and a half, and the enterprise distros
> are off doing their thing with their multi-year upgrade cycles, there's
> no real need from the distros for a new longterm kernel release.  But it
> turns out that the distros are not the only user of the kernel, other
> groups and companies have been approaching me over the past year, asking
> how they could pick the next longterm kernel, or what the process is in
> determining this.
>
> To keep this all out in the open, let's figure out what to do here.
> Consumer devices have a 1-2 year lifespan, and want and need the
> experience of the kernel community maintaining their "base" kernel for
> them.  There is no real "enterprise" embedded distro out there from what
> I can see.  montaVista and WindRiver have some offerings in this area, but
> they are not that widely used and are usually more "deep embedded".
> There's also talk that the CELF group and Linaro are wanting to do
> something on a "longterm" basis, and are fishing around for how to
> properly handle this with the community to share the workload.  Android
> also is another huge player here, upgrading their kernel every major
> release, and they could use the support of a longterm kernel as well.
>
> Proposal:
>
> Here's a first cut at a proposal, let me know if you like it, hate it,
> would work for you and your company, or not at all:
>
> - a new -longterm kernel is picked every year.
> - a -longterm kernel is maintained for 2 years and then dropped.
> - -stable kernels keep the same schedule that they have been (dropping
>    the last one after a new release happens.)  These releases are best
>    for products that require new hardware updates (desktop distros,
>    community distros, fast-moving embedded distros (like Yocto)).
> - the normal -stable rules apply to these -longterm kernels as described
>    in Documentation/stable_kernel_rules.txt
>
> This means that there are 2 -longterm kernels being maintained at the
> same time, and one -stable kernel.  I'm volunteering to do this work, as
> it's pretty much what I'm doing today anyway, and I have all of the
> scripts and workflow down.
>
> Public Notifications:
>
> The current kernel.org site doesn't properly show what is and is not
> being maintained as a -stable and -longterm kernel.  I have a proposal
> for how to fix this involving 'git notes', I just need to sit down and
> do the work with the kernel.org admins to get this running properly.
>
> Thoughts?
>
>
> greg k-h
>

Speaking for Ubuntu, your proposal seems reasonable to me. We're going 
to have an LTS release every couple of years with the next one being 
12.04 (April 2012) so it would be good to coordinate on that kernel 
version choice (if possible).

The current non-LTS stable kernel releases have also been working well 
since by the time their natural support cycle elapses the Ubuntu 
community is already on to the next shiny release. Maintenance of those 
non-LTS kernels thereafter consists primarily of CVE backports.

rtg
-- 
Tim Gardner tim.gardner@canonical.com

Re: [Stable-review] Future of the -longterm kernel releases (i.e. how we pick them).

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 2645 bytes --]

On Sun, 2011-08-14 at 21:15 -0700, Greg KH wrote:
[...]
> Today:
> 
> Now that 2.6.32 is over a year and a half, and the enterprise distros
> are off doing their thing with their multi-year upgrade cycles, there's
> no real need from the distros for a new longterm kernel release.  But it
> turns out that the distros are not the only user of the kernel, other
> groups and companies have been approaching me over the past year, asking
> how they could pick the next longterm kernel, or what the process is in
> determining this.
> 
> To keep this all out in the open, let's figure out what to do here.
> Consumer devices have a 1-2 year lifespan, and want and need the
> experience of the kernel community maintaining their "base" kernel for
> them.

This timespan is both somewhat optimistic with respect to current
reality, and also rather depressing in that it sets a very low bar.  I
would hope that we don't collectively treat consumer electronics as
disposable and that responsible vendors would like to provide security
support for a lifetime of 5-10 years.  (But no, I don't think that's
easy.)

[...]
> Proposal:
> 
> Here's a first cut at a proposal, let me know if you like it, hate it,
> would work for you and your company, or not at all:
> 
> - a new -longterm kernel is picked every year.
> - a -longterm kernel is maintained for 2 years and then dropped.

This is nowhere near the maintenance lifetime for any of the vendors
that are following 2.6.32.y now:

Debian: ~3 years (stated as 1 year after next release)
Oracle: ??? (probably as long as anyone pays for it)
SUSE (SLES): 7 years
Ubuntu (LTS): 5 years

(and those distributions were released well after the original 2.6.32
release).  Given that we'll want to carry on sharing the maintenance
burden, it seems silly to set such an early cut-off date.

[...]
> This means that there are 2 -longterm kernels being maintained at the
> same time, and one -stable kernel.  I'm volunteering to do this work, as
> it's pretty much what I'm doing today anyway, and I have all of the
> scripts and workflow down.
[...]

I entirely understand that *you* don't want to be stuck maintaining
longterm releases from 1, 2, 3, ... n years back.  If you could set a
policy of handing off longterm series around the 2 year mark (and
terminating them if there is no volunteer) then that would be more
reasonable.

I see that you've accepted Willy Tarreau's offer to take over 2.6.32.y,
so if you could just formalise that before /. goes wild over the looming
end of all the above distributions, that would be nice. :-)

Ben.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [Stable-review] Future of the -longterm kernel releases (i.e. how we pick them).

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 2603 bytes --]

On Mon, 2011-08-15 at 07:21 -0700, Greg KH wrote:
> On Mon, Aug 15, 2011 at 12:21:59AM -0700, david@lang.hm wrote:
> > rather than having a hard schedule (the first kernel released after
> > July 1 each year for example I know this is not the exact proposal),
> > I think that it would be better to pick the -longterm kernel a few
> > months after it has been released (3.4 is looking very good, the
> > normal minor driver fixes in -stable, but no fundamental regressions
> > have been reported, it's the new -longerm kernel for example)
> > 
> > doing so doesn't give the predictability that some people will want
> > in knowing that their September release will always have a fresh new
> > -longerm kernel, but I think the result would be better -longterm
> > kernels. However, to get the information about how good the kernels
> > are, I think that the -stable timeframe would need to be extended to
> > give the kernels time to settle and gather reports. I would then
> > suggest scheduling that once a year you look at the last couple
> > -stable kernels and pick one of them rather than designating the new
> > -longterm kernel ahead of time.
> 
> Yes, that's a very good idea.  I've seen problems in the past when
> distros have made a time-based decision to pick a kernel version and
> then the problems that this can cause if it happens that a subsystem
> really had issues for that release.
> 
> So yes, I'll take a look at the bug reports and how things are working
> out to pick the next -longterm.  I'll also take into consideration any
> companies/major users that are going to be using that release as well,
> so it greatly behooves people to talk to me about their plans (hint,
> hint...)
[...]

This might be a good discussion to have at Kernel Summit, if enough
distro people are going to be there.

The next stable release of Debian will have a feature-freeze in June
2012, so we need to make a choice some time before then.  However I know
that Ubuntu will need to make a decision rather earlier for the 12.04
LTS release.

Are any other distributions aiming to make long-term-supported releases
in 2012?

Also, in Debian we just added an 'rt' build of the kernel using the
PREEMPT_RT patches.  We don't want to use different upstream versions as
a basis for specific builds, so if we're going to include PREEMPT_RT in
a stable distribution release we would ideally want to see some interest
in maintaining this patchset on top of the longterm series.  (Hopefully
it's going to be significantly smaller by then, of course.)

Ben.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [stable] [Stable-review] Future of the -longterm kernel releases (i.e. how we pick them).

[Greg KH]

On Tue, Aug 16, 2011 at 03:26:56AM +0100, Ben Hutchings wrote:
> On Mon, 2011-08-15 at 07:21 -0700, Greg KH wrote:
> > On Mon, Aug 15, 2011 at 12:21:59AM -0700, david@lang.hm wrote:
> > > rather than having a hard schedule (the first kernel released after
> > > July 1 each year for example I know this is not the exact proposal),
> > > I think that it would be better to pick the -longterm kernel a few
> > > months after it has been released (3.4 is looking very good, the
> > > normal minor driver fixes in -stable, but no fundamental regressions
> > > have been reported, it's the new -longerm kernel for example)
> > > 
> > > doing so doesn't give the predictability that some people will want
> > > in knowing that their September release will always have a fresh new
> > > -longerm kernel, but I think the result would be better -longterm
> > > kernels. However, to get the information about how good the kernels
> > > are, I think that the -stable timeframe would need to be extended to
> > > give the kernels time to settle and gather reports. I would then
> > > suggest scheduling that once a year you look at the last couple
> > > -stable kernels and pick one of them rather than designating the new
> > > -longterm kernel ahead of time.
> > 
> > Yes, that's a very good idea.  I've seen problems in the past when
> > distros have made a time-based decision to pick a kernel version and
> > then the problems that this can cause if it happens that a subsystem
> > really had issues for that release.
> > 
> > So yes, I'll take a look at the bug reports and how things are working
> > out to pick the next -longterm.  I'll also take into consideration any
> > companies/major users that are going to be using that release as well,
> > so it greatly behooves people to talk to me about their plans (hint,
> > hint...)
> [...]
> 
> This might be a good discussion to have at Kernel Summit, if enough
> distro people are going to be there.

I doubt the proper distro people will be at the Kernel Summit, sorry.
Perhaps at Plumbers or at LinuxCon Europe?

greg k-h

Re: [stable] [Stable-review] Future of the -longterm kernel releases (i.e. how we pick them).

[Greg KH]

On Tue, Aug 16, 2011 at 03:09:02AM +0100, Ben Hutchings wrote:
> 
> I see that you've accepted Willy Tarreau's offer to take over 2.6.32.y,
> so if you could just formalise that before /. goes wild over the looming
> end of all the above distributions, that would be nice. :-)

The distros were doing fine before any of this -stable and -longterm
stuff ever happened, so I'm sure that's not going to happen :)

greg k-h

Re: [stable] [Stable-review] Future of the -longterm kernel releases (i.e. how we pick them).

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 824 bytes --]

On Mon, 2011-08-15 at 19:56 -0700, Greg KH wrote:
> On Tue, Aug 16, 2011 at 03:26:56AM +0100, Ben Hutchings wrote:
> > On Mon, 2011-08-15 at 07:21 -0700, Greg KH wrote:
[...]
> > > So yes, I'll take a look at the bug reports and how things are working
> > > out to pick the next -longterm.  I'll also take into consideration any
> > > companies/major users that are going to be using that release as well,
> > > so it greatly behooves people to talk to me about their plans (hint,
> > > hint...)
> > [...]
> > 
> > This might be a good discussion to have at Kernel Summit, if enough
> > distro people are going to be there.
> 
> I doubt the proper distro people will be at the Kernel Summit, sorry.
> Perhaps at Plumbers or at LinuxCon Europe?

KS is the only one of the three that I'm going to.

Ben.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

RE: Future of the -longterm kernel releases (i.e. how we pick them).

[Daniel Taylor]

 

> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org 
> [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Greg KH
> Sent: Monday, August 15, 2011 7:22 AM
> To: david@lang.hm
> Cc: linux-kernel@vger.kernel.org; 
> torvalds@linux-foundation.org; akpm@linux-foundation.org; 
> alan@lxorguk.ukuu.org.uk; stable-review@kernel.org; stable@kernel.org
> Subject: Re: Future of the -longterm kernel releases (i.e. 
> how we pick them).
> 
> On Mon, Aug 15, 2011 at 12:21:59AM -0700, david@lang.hm wrote:
> > rather than having a hard schedule (the first kernel released after
> > July 1 each year for example I know this is not the exact proposal),
> > I think that it would be better to pick the -longterm kernel a few
> > months after it has been released (3.4 is looking very good, the
> > normal minor driver fixes in -stable, but no fundamental regressions
> > have been reported, it's the new -longerm kernel for example)
> > 
> > doing so doesn't give the predictability that some people will want
> > in knowing that their September release will always have a fresh new
> > -longerm kernel, but I think the result would be better -longterm
> > kernels. However, to get the information about how good the kernels
> > are, I think that the -stable timeframe would need to be extended to
> > give the kernels time to settle and gather reports. I would then
> > suggest scheduling that once a year you look at the last couple
> > -stable kernels and pick one of them rather than designating the new
> > -longterm kernel ahead of time.
> 
> Yes, that's a very good idea.  I've seen problems in the past when
> distros have made a time-based decision to pick a kernel version and
> then the problems that this can cause if it happens that a subsystem
> really had issues for that release.
> 
> So yes, I'll take a look at the bug reports and how things are working
> out to pick the next -longterm.  I'll also take into consideration any
> companies/major users that are going to be using that release as well,
> so it greatly behooves people to talk to me about their plans (hint,
> hint...)
> 
> > I hope my midnight rambling makes some sort of sense :-)
> 
> It did, thanks for the response.
> 
> greg k-h
> --
> To unsubscribe from this list: send the line "unsubscribe 
> linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

I also appreciate the amount of work it takes to maintain a kernel; thanks.

For embedded systems, which often have multi- (not just two-) year
warranties, it sounds like we would then be maintaining the -longterm
ourselves.  The "easy" side of this is that embedded systems don't
(usually) have users trying to plug in the latest and greatest widget,
except on external busses, such as USB.  So back-porting drivers is
not generally a big deal.  Security patches, OTOH, are a real concern.
Other than net and drivers/net, which are exposed to non-local attacks,
are there other subsystems we should watch for security fixes?  There
have, for example, been a lot of security upgrades in /sys, but there's
also been a lot of restructing for new interfaces and "arch" models,
which would make back-porting those much more complicated.

Re: [Stable-review] Future of the -longterm kernel releases (i.e. how we pick them).

[Jeremiah C. Foster]

On Aug 16, 2011, at 4:09, Ben Hutchings wrote:

> On Sun, 2011-08-14 at 21:15 -0700, Greg KH wrote:
> [...]
>> Today:
>> 
>> Now that 2.6.32 is over a year and a half, and the enterprise distros
>> are off doing their thing with their multi-year upgrade cycles, there's
>> no real need from the distros for a new longterm kernel release.  But it
>> turns out that the distros are not the only user of the kernel, other
>> groups and companies have been approaching me over the past year, asking
>> how they could pick the next longterm kernel, or what the process is in
>> determining this.
>> 
>> To keep this all out in the open, let's figure out what to do here.
>> Consumer devices have a 1-2 year lifespan, and want and need the
>> experience of the kernel community maintaining their "base" kernel for
>> them.
> 
> This timespan is both somewhat optimistic with respect to current
> reality, and also rather depressing in that it sets a very low bar.  I
> would hope that we don't collectively treat consumer electronics as
> disposable and that responsible vendors would like to provide security
> support for a lifetime of 5-10 years.  (But no, I don't think that's
> easy.)

I'd like to echo Ben's sentiment, particularly in the area of automotive. 
A car has to be supported with parts for at least ten years, often longer, 
and this includes the build system for the infotainment software.
The GENIVI Alliance is now building infotainment systems for their member 
companies (BMW, GM, PSA, Hyundai, etc.) which will have to preserve a 
working kernel for a long time, like lark's tongues in aspic. So there is an 
interest in a "longterm, stable" kernel in the automotive industry. Furthermore,
know-how around choosing a long term kernel relevant to a car is in short 
supply, so there is a lot of reliance on the distros and commercial OSVs in 
this regard.

Regards,

Jeremiah

Re: [stable] [Stable-review] Future of the -longterm kernel releases (i.e. how we pick them).

[Greg KH]

On Tue, Aug 16, 2011 at 09:26:24PM +0200, Jeremiah C. Foster wrote:
> I'd like to echo Ben's sentiment, particularly in the area of automotive. 
> A car has to be supported with parts for at least ten years, often longer, 
> and this includes the build system for the infotainment software.
> The GENIVI Alliance is now building infotainment systems for their member 
> companies (BMW, GM, PSA, Hyundai, etc.) which will have to preserve a 
> working kernel for a long time, like lark's tongues in aspic. So there is an 
> interest in a "longterm, stable" kernel in the automotive industry. Furthermore,
> know-how around choosing a long term kernel relevant to a car is in short 
> supply, so there is a lot of reliance on the distros and commercial OSVs in 
> this regard.

Isn't that the job of the distros and commercial OSVs today?  Are they
somehow not doing this job well?  Do they need help from the community
instead to help define, implement, and maintain this for them?

I'm genuinely curious about this, I haven't heard this directly from
users before, only from companies who are in this line of work, wanting
help in doing this for them, for a variety of odd reasons.

If so, doesn't this imply that maybe those users should be choosing a
different company for this support, or that they have given up on this
and want to work directly with the community instead?  If the latter,
I'd be very happy to work with them, contacts are greatly appreciated.

greg k-h

Re: Future of the -longterm kernel releases (i.e. how we pick them).

[Tim Bird]

>Greg KH wrote:
> To keep this all out in the open, let's figure out what to do here.
> Consumer devices have a 1-2 year lifespan, and want and need the
> experience of the kernel community maintaining their "base" kernel for
> them.  There is no real "enterprise" embedded distro out there from what
> I can see.  montaVista and WindRiver have some offerings in this area, but
> they are not that widely used and are usually more "deep embedded".
> There's also talk that the CELF group and Linaro are wanting to do
> something on a "longterm" basis, and are fishing around for how to
> properly handle this with the community to share the workload.  Android
> also is another huge player here, upgrading their kernel every major
> release, and they could use the support of a longterm kernel as well.

Well I certainly hope that there's more participation on sharing the
workload by embedded companies than there has historically been.
I'm not at LinuxCon this week, but I want to follow up with you
(maybe at KS?) on good ways that CE companies can help with
this effort.

> Proposal:
>
> Here's a first cut at a proposal, let me know if you like it, hate it,
> would work for you and your company, or not at all:
>
> - a new -longterm kernel is picked every year.
> - a -longterm kernel is maintained for 2 years and then dropped.
> - -stable kernels keep the same schedule that they have been (dropping
>  the last one after a new release happens.)  These releases are best
>  for products that require new hardware updates (desktop distros,
>  community distros, fast-moving embedded distros (like Yocto)).
> - the normal -stable rules apply to these -longterm kernels as described
>  in Documentation/stable_kernel_rules.txt
>
> This means that there are 2 -longterm kernels being maintained at the
> same time, and one -stable kernel.  I'm volunteering to do this work, as
> it's pretty much what I'm doing today anyway, and I have all of the
> scripts and workflow down.

This all sounds great to me. As you know CEWG (formerly CELF) is very
interested in supporting LTS work.  What you describe
would be a good base for what we envision.

That said, I echo the concerns about version selection.  It would be
good to have a "settle" period longer than 1 stable release for selecting
a kernel (but then again, we don't want to wait too long to pick each
longterm release).

Rolling, overlapping, longterm versions seems like a nice idea.
I like the idea of looking for a volunteer to continue maintenance after
2 years, and if none is found dropping the release.  That way, someone
can step up if there is continued interest in a particular longterm version.

Also, the more people using a particular LTS kernel, the better. In the past
the embedded guys didn't coordinate well enough with other parties.  If
we could get enterprise, desktop and embedded on a single LT release, that
would be really nice.  I'm not sure how you're keeping track of interested
parties, but if there's a mailing list (besides 'stable') let me know so I can
sign up.

With regard to version selection, I know different companies will have
different versions they'd like to see selected.  Having consensus on
a version is more important than the particular version chosen.

Having said that, here are some (non-BSP) things we look at for selecting
kernel versions at Sony:

One specific issue I have is support for PREEMPT_RT, so that's a big
factor in selecting Sony kernel versions.  Thus, coordinating with the
RT patchset kernel versions is important to me.  Currently
that would mean 3.0 is a good candidate.

The other major out-of-tree patches we usually integrate are
1) Android
2) LTTng, but I've heard that this might soon be buildable as
a loadable module (eliminating the need for any patch integration).

Thanks,
  -- Tim

Re: Future of the -longterm kernel releases (i.e. how we pick them).

[Greg KH]

On Tue, Aug 16, 2011 at 04:01:57PM -0700, Tim Bird wrote:
> >Greg KH wrote:
> > To keep this all out in the open, let's figure out what to do here.
> > Consumer devices have a 1-2 year lifespan, and want and need the
> > experience of the kernel community maintaining their "base" kernel for
> > them.  There is no real "enterprise" embedded distro out there from what
> > I can see.  montaVista and WindRiver have some offerings in this area, but
> > they are not that widely used and are usually more "deep embedded".
> > There's also talk that the CELF group and Linaro are wanting to do
> > something on a "longterm" basis, and are fishing around for how to
> > properly handle this with the community to share the workload.  Android
> > also is another huge player here, upgrading their kernel every major
> > release, and they could use the support of a longterm kernel as well.
> 
> Well I certainly hope that there's more participation on sharing the
> workload by embedded companies than there has historically been.
> I'm not at LinuxCon this week, but I want to follow up with you
> (maybe at KS?) on good ways that CE companies can help with
> this effort.

There is a BOF this week at LinuxCon with the CE companies about this
topic, and I've been talking with them about this as well, so if anyone
is there and interested, we can talk about it then.

And yes, ELC/KS/LinuxCon Europe is also a good place to discuss it as
well.

> That said, I echo the concerns about version selection.  It would be
> good to have a "settle" period longer than 1 stable release for selecting
> a kernel (but then again, we don't want to wait too long to pick each
> longterm release).

I understand, and will be working on this.  I'm trying to talk to
everyone I can about their plans for support and products to try to
determine this, and will see how it goes.

> Also, the more people using a particular LTS kernel, the better. In the past
> the embedded guys didn't coordinate well enough with other parties.  If
> we could get enterprise, desktop and embedded on a single LT release, that
> would be really nice.  I'm not sure how you're keeping track of interested
> parties, but if there's a mailing list (besides 'stable') let me know so I can
> sign up.

stable is the best place for this.  Or emails/phone calls/hallway
discussions with me as well.  Note, I also have the ability to sign NDAs
if needed.

> One specific issue I have is support for PREEMPT_RT, so that's a big
> factor in selecting Sony kernel versions.  Thus, coordinating with the
> RT patchset kernel versions is important to me.  Currently
> that would mean 3.0 is a good candidate.

3.0 is looking like a good candidate, but we haven't seen 3.1 yet :)

> The other major out-of-tree patches we usually integrate are
> 1) Android

I would love to get some feedback/contacts with the Android teams to
talk about this with them.  If anyone knows anyone I should talk to
here, please let me know.

thanks,

greg k-h

Re: [stable] [Stable-review] Future of the -longterm kernel releases (i.e. how we pick them).

[Jeremiah Foster]

On Aug 17, 2011, at 00:33, Greg KH wrote:
> On Tue, Aug 16, 2011 at 09:26:24PM +0200, Jeremiah C. Foster wrote:
>> I'd like to echo Ben's sentiment, particularly in the area of automotive. 
>> A car has to be supported with parts for at least ten years, often longer, 
>> and this includes the build system for the infotainment software.
>> The GENIVI Alliance is now building infotainment systems for their member 
>> companies (BMW, GM, PSA, Hyundai, etc.) which will have to preserve a 
>> working kernel for a long time, like lark's tongues in aspic. So there is an 
>> interest in a "longterm, stable" kernel in the automotive industry. Furthermore,
>> know-how around choosing a long term kernel relevant to a car is in short 
>> supply, so there is a lot of reliance on the distros and commercial OSVs in 
>> this regard.
> 
> Isn't that the job of the distros and commercial OSVs today?  

My understanding is yes. It appears to be a business opportunity for many OSVs and others as well, but the distro's are doing a good job so increasingly commercial companies are turning to distros, at least initially.

> Are they
> somehow not doing this job well?  

I think they are doing the job well which is why there is increasing choice; use a distro or pay for an OSV? Rely on the community or develop in-house competence? These questions are new, at least for the automotive industry, since previously it was all proprietary all the time.

> Do they need help from the community
> instead to help define, implement, and maintain this for them?

I think the answer is yes.

> I'm genuinely curious about this, I haven't heard this directly from
> users before, only from companies who are in this line of work, wanting
> help in doing this for them, for a variety of odd reasons.

If it helps at all, I can bring up this topic inside GENIVI and ask if there are OEMs, Tier 1s and others who would be interested in how to identify a kernel that is appropriate for their long-term needs. I have participated in GENIVI discussions like this previously and there has not necessarily been clarity. Having your perspective and the perspective of others with experience in kernel maintenance would be helpful.

> If so, doesn't this imply that maybe those users should be choosing a
> different company for this support, or that they have given up on this
> and want to work directly with the community instead?  

That is the eternal question. For the auto industry it often boils down to the cost / benefit ratio and the cost sensitivity for production vehicles per unit is a major factor in what they choose. I think if they can find a reasonable long-term kernel they'll help maintain it in conjunction with the community.

> If the latter,
> I'd be very happy to work with them, contacts are greatly appreciated.

Very generous of you. Let me see if I can pull together a list of people where this can be discussed.

@everyone: Should I trim the CC list and limit this to a mailing list? Or would people prefer to remain on CC?

Regards,

Jeremiah

Re: Future of the -longterm kernel releases (i.e. how we pick them).

[James Courtier-Dutton]

On 15 August 2011 08:16, Teck Choon Giam <giamteckchoon@gmail.com> wrote:
> 2011/8/15  <david@lang.hm>:
>> On Mon, 15 Aug 2011, J?rg-Volker Peetz wrote:
>>
>>> Greg KH wrote, on 08/15/11 06:15:
>>> <snip>
>>>>
>>>> - a new -longterm kernel is picked every year.
>>>> - a -longterm kernel is maintained for 2 years and then dropped.
>>>
>>> <snip>
>>>
>>> Just a little nitpick: with this scheme you will accumulate longterm
>>> kernels. If
>>> I understand it right, one longterm kernel is added every year.
>>
>> but with one new kernel being added each year, and then being dropped two
>> years later, there are only 2 long term kernels at any one time (that Greg
>> will be maintaining at least, nothing stops other people from maintaining
>> other long term kernels in addition)
>
> Err... sorry from my understanding like this year... one new long term
> kernel added so it is N+1 then next year is N+1+1 and two years drop
> one so it is N+1+1-1... so every year there will be one new long term
> added and every two years there will be one overall long term kernel
> added to total number of long term kernels... since two are added
> within 2 years and one dropped every 2 years.  So there are not only 2
> long term kernels at any one time in this case... ... someone correct
> me if I am wrong.
>

I think your maths might be wrong.

Year 1, long term kernel 1 starts  (total concurrent long term = 1)
Year 2, long term kernel 2 starts  (total concurrent long term = 2)
Year 3, long term kernel 3 starts, long term kernel 1 stops. (total
concurrent long term = 2)
Year 4, long term kernel 4 starts, long term kernel 2 stops. (total
concurrent long term = 2)
Year 5, long term kernel 5 starts, long term kernel 3 stops. (total
concurrent long term = 2)

Re: Future of the -longterm kernel releases (i.e. how we pick them).

[Mark Brown]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 546 bytes --]

On Tue, Aug 16, 2011 at 09:58:56PM -0700, Greg KH wrote:
> On Tue, Aug 16, 2011 at 04:01:57PM -0700, Tim Bird wrote:

> > The other major out-of-tree patches we usually integrate are
> > 1) Android

> I would love to get some feedback/contacts with the Android teams to
> talk about this with them.  If anyone knows anyone I should talk to
> here, please let me know.

Brian Swetland and Arve Hjønnevåshould be able to point you at the
right people if they aren't themselves the right people.

[Sorry if my software has mangled your name Arve]

Re: Future of the -longterm kernel releases (i.e. how we pick them).

[Brian Swetland]

On Wed, Aug 17, 2011 at 6:21 AM, Mark Brown
<broonie@opensource.wolfsonmicro.com> wrote:
> On Tue, Aug 16, 2011 at 09:58:56PM -0700, Greg KH wrote:
>> On Tue, Aug 16, 2011 at 04:01:57PM -0700, Tim Bird wrote:
>
>> > The other major out-of-tree patches we usually integrate are
>> > 1) Android
>
>> I would love to get some feedback/contacts with the Android teams to
>> talk about this with them.  If anyone knows anyone I should talk to
>> here, please let me know.
>
> Brian Swetland and Arve Hjønnevåg should be able to point you at the
> right people if they aren't themselves the right people.
>
> [Sorry if my software has mangled your name Arve]

(something ate the final 'g' (fixed above) but otherwise his name
seems to have survived)

As far as long-term kernels goes, from the Android perspective we
strongly prefer to snap up to the most recent released kernel on every
platform/device release.  I prefer to be as up to date on bugfixes and
features from mainline as possible and minimize the deltas on our
stack 'o patches as much as possible.

We've been getting more aggressive about merging in the -rc#s and then
rebasing on the final during development (before final stabilization
freeze for a release) in the last year or so, and it seems to work
pretty well.

Not all OEMs or silicon vendors are as comfortable with this approach
and I expect some would welcome a long term tree they can count on to
get critical fixes for, though.  Device release teams (even our own,
for lead devices we work on) are notoriously risk-averse (often for
good reasons), and tend to get really nervous when they see the total
change count or diff stat for pulling up to a new kernel release close
to "final rom" for a product.

Brian

(apologies to anyone who got this twice, gmail snafu with html
formatted email. I should use mutt for lkml...)

Re: [stable] [Stable-review] Future of the -longterm kernel releases (i.e. how we pick them).

[david]

On Wed, 17 Aug 2011, Jeremiah Foster wrote:

>> Do they need help from the community
>> instead to help define, implement, and maintain this for them?
>
> I think the answer is yes.
>

to expand on this a bit.

it's a lot easier to look at changelogs and see if a -stable or -longterm 
update is relavent to your systems than it is to watch the flood of 
merges to the latest kerenl and then figure out how to backport them. BUt 
people who start up just compiling their own kernels frequently become 
testers, if not contributers to the kernel. If everyone only runs the 
distro kernels, then the upstream kernel quality will suffer because 
nobody is testing it.

people running -longterm kernels on their productionsystems will not be 
testing the llatest -rc kernel on those systems, but they are likely to be 
more interested in watching and testng new kernel releases (at least on 
lab machines) than people who just wait for things to be backported to the 
distro kernel.

David Lang

Re: [Stable-review] Future of the -longterm kernel releases (i.e. how we pick them).

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 1198 bytes --]

On Tue, 2011-08-16 at 21:58 -0700, Greg KH wrote:
> On Tue, Aug 16, 2011 at 04:01:57PM -0700, Tim Bird wrote:
[...]
> > One specific issue I have is support for PREEMPT_RT, so that's a big
> > factor in selecting Sony kernel versions.  Thus, coordinating with the
> > RT patchset kernel versions is important to me.  Currently
> > that would mean 3.0 is a good candidate.
> 
> 3.0 is looking like a good candidate, but we haven't seen 3.1 yet :)
[...]

I'm quite happy with 3.0 in terms of stability and features, though it's
early days yet.  It's not so good for Debian in terms of timing.  We're
not going to freeze for another 10 months, then the release will likely
be at least 6 months after that.  The earlier a kernel release we pick
with, the more hardware support needs to be backported later.  So,
extrapolating the likely kernel release dates, I think I would prefer to
use something between 3.2 and 3.4.

As for the RT patchset, I haven't seen any announcement that it will
continue to be developed against 3.0 after the release of 3.1.  Of
course, the fact that there *is* a working RT patchset for 3.0 is an
advancement over recent upstream releases.

Ben.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [Stable-review] Future of the -longterm kernel releases (i.e. how we pick them).

[Pasi Kärkkäinen]

On Thu, Aug 18, 2011 at 01:33:34AM +0100, Ben Hutchings wrote:
> On Tue, 2011-08-16 at 21:58 -0700, Greg KH wrote:
> > On Tue, Aug 16, 2011 at 04:01:57PM -0700, Tim Bird wrote:
> [...]
> > > One specific issue I have is support for PREEMPT_RT, so that's a big
> > > factor in selecting Sony kernel versions.  Thus, coordinating with the
> > > RT patchset kernel versions is important to me.  Currently
> > > that would mean 3.0 is a good candidate.
> > 
> > 3.0 is looking like a good candidate, but we haven't seen 3.1 yet :)
> [...]
> 
> I'm quite happy with 3.0 in terms of stability and features, though it's
> early days yet.  It's not so good for Debian in terms of timing.  We're
> not going to freeze for another 10 months, then the release will likely
> be at least 6 months after that.  The earlier a kernel release we pick
> with, the more hardware support needs to be backported later.  So,
> extrapolating the likely kernel release dates, I think I would prefer to
> use something between 3.2 and 3.4.
> 

I'd *guess* Ubuntu 12.04 LTS goes for Linux 3.2 .. 

-- Pasi


> As for the RT patchset, I haven't seen any announcement that it will
> continue to be developed against 3.0 after the release of 3.1.  Of
> course, the fact that there *is* a working RT patchset for 3.0 is an
> advancement over recent upstream releases.
> 
> Ben.
> 

Re: [stable] [Stable-review] Future of the -longterm kernel releases (i.e. how we pick them).

[Greg KH]

On Wed, Aug 17, 2011 at 12:33:56PM +0200, Jeremiah Foster wrote:
> 
> On Aug 17, 2011, at 00:33, Greg KH wrote:
> > On Tue, Aug 16, 2011 at 09:26:24PM +0200, Jeremiah C. Foster wrote:
> >> I'd like to echo Ben's sentiment, particularly in the area of automotive. 
> >> A car has to be supported with parts for at least ten years, often longer, 
> >> and this includes the build system for the infotainment software.
> >> The GENIVI Alliance is now building infotainment systems for their member 
> >> companies (BMW, GM, PSA, Hyundai, etc.) which will have to preserve a 
> >> working kernel for a long time, like lark's tongues in aspic. So there is an 
> >> interest in a "longterm, stable" kernel in the automotive industry. Furthermore,
> >> know-how around choosing a long term kernel relevant to a car is in short 
> >> supply, so there is a lot of reliance on the distros and commercial OSVs in 
> >> this regard.
> > 
> > Isn't that the job of the distros and commercial OSVs today?  
> 
> My understanding is yes. It appears to be a business opportunity for
> many OSVs and others as well, but the distro's are doing a good job so
> increasingly commercial companies are turning to distros, at least
> initially.
> 
> > Are they
> > somehow not doing this job well?  
> 
> I think they are doing the job well which is why there is increasing
> choice; use a distro or pay for an OSV? Rely on the community or
> develop in-house competence? These questions are new, at least for the
> automotive industry, since previously it was all proprietary all the
> time.

Yes, it's a new model that they need to get used to :)

> > Do they need help from the community
> > instead to help define, implement, and maintain this for them?
> 
> I think the answer is yes.
> 
> > I'm genuinely curious about this, I haven't heard this directly from
> > users before, only from companies who are in this line of work, wanting
> > help in doing this for them, for a variety of odd reasons.
> 
> If it helps at all, I can bring up this topic inside GENIVI and ask if
> there are OEMs, Tier 1s and others who would be interested in how to
> identify a kernel that is appropriate for their long-term needs. I
> have participated in GENIVI discussions like this previously and there
> has not necessarily been clarity. Having your perspective and the
> perspective of others with experience in kernel maintenance would be
> helpful.

Please do.  My view of GENIVI from the outside is that it reminds me a
lot of the problems that plagued the old CGL initiative.  Hopefully that
is incorrect on my part.

If there's anyone, or any group, I should be talking to about this, or
any meeting I could attend to help explain it all better, please let me
know, I am more than willing to do so.

> > If so, doesn't this imply that maybe those users should be choosing a
> > different company for this support, or that they have given up on this
> > and want to work directly with the community instead?  
> 
> That is the eternal question. For the auto industry it often boils
> down to the cost / benefit ratio and the cost sensitivity for
> production vehicles per unit is a major factor in what they choose. I
> think if they can find a reasonable long-term kernel they'll help
> maintain it in conjunction with the community.

That's good to hear, any help is appreciated.

Mostly I want to know what patches should be applied, that fix problems
they have.  That and testing the -rc releases would be wonderful.

> > If the latter,
> > I'd be very happy to work with them, contacts are greatly appreciated.
> 
> Very generous of you. Let me see if I can pull together a list of
> people where this can be discussed.

That would be great, odds are, a new thread can be started, and everyone
on cc: taken off, as I doubt they care about this :)

thanks,

greg k-h

Re: [stable] [Stable-review] Future of the -longterm kernel releases (i.e. how we pick them).

[Greg KH]

On Wed, Aug 17, 2011 at 01:20:37PM -0700, david@lang.hm wrote:
> On Wed, 17 Aug 2011, Jeremiah Foster wrote:
> 
> >>Do they need help from the community
> >>instead to help define, implement, and maintain this for them?
> >
> >I think the answer is yes.
> >
> 
> to expand on this a bit.
> 
> it's a lot easier to look at changelogs and see if a -stable or
> -longterm update is relavent to your systems than it is to watch the
> flood of merges to the latest kerenl and then figure out how to
> backport them. BUt people who start up just compiling their own
> kernels frequently become testers, if not contributers to the
> kernel. If everyone only runs the distro kernels, then the upstream
> kernel quality will suffer because nobody is testing it.

I totally agree.

> people running -longterm kernels on their productionsystems will not
> be testing the llatest -rc kernel on those systems, but they are
> likely to be more interested in watching and testng new kernel
> releases (at least on lab machines) than people who just wait for
> things to be backported to the distro kernel.

That's good to hear, hopefully the -longterm kernels are useful for them
and their usage model.  If not, please let me know.

greg k-h

Re: Future of the -longterm kernel releases (i.e. how we pick them).

[Greg KH]

On Wed, Aug 17, 2011 at 10:33:05AM -0700, Brian Swetland wrote:
> As far as long-term kernels goes, from the Android perspective we
> strongly prefer to snap up to the most recent released kernel on every
> platform/device release.  I prefer to be as up to date on bugfixes and
> features from mainline as possible and minimize the deltas on our
> stack 'o patches as much as possible.

That's good to hear.

> We've been getting more aggressive about merging in the -rc#s and then
> rebasing on the final during development (before final stabilization
> freeze for a release) in the last year or so, and it seems to work
> pretty well.

Is your kernel git tree public during this merge cycle so that others
can track it?  I tried to dig through android.kernel.org but there are a
lot of different kernel trees there :(

> Not all OEMs or silicon vendors are as comfortable with this approach
> and I expect some would welcome a long term tree they can count on to
> get critical fixes for, though.  Device release teams (even our own,
> for lead devices we work on) are notoriously risk-averse (often for
> good reasons), and tend to get really nervous when they see the total
> change count or diff stat for pulling up to a new kernel release close
> to "final rom" for a product.

That's understandable.

So that kind of means they would like such a longterm tree, but they
have the problem that your patches move forward to a newer kernel, but
are not maintained for that longterm kernel.  Are they responsible for
maintaining this themselves then, or do you help with providing your
patchset on top of all of the different kernel versions that the OEMs
and others need?

thanks,

greg k-h

Re: [stable] [Stable-review] Future of the -longterm kernel releases (i.e. how we pick them).

[Jeremiah Foster]

On Aug 24, 2011, at 06:46, Greg KH wrote:

> On Wed, Aug 17, 2011 at 12:33:56PM +0200, Jeremiah Foster wrote:
>> 
>> On Aug 17, 2011, at 00:33, Greg KH wrote:
>>> On Tue, Aug 16, 2011 at 09:26:24PM +0200, Jeremiah C. Foster wrote:

[snip]

>>> Are they
>>> somehow not doing this job well?  
>> 
>> I think they are doing the job well which is why there is increasing
>> choice; use a distro or pay for an OSV? Rely on the community or
>> develop in-house competence? These questions are new, at least for the
>> automotive industry, since previously it was all proprietary all the
>> time.
> 
> Yes, it's a new model that they need to get used to :)

I think this is what they have realized. I think they are actually trying to use that model but that has changed things for them. When I said they maintain build systems and software for ten years I was speaking of the current model. When I went back and spoke to the software developers who work for car makers they said that things are changing. The pace of development in consumer electronics has affected what you have to put in a car. People expect a certain level of software functionality in a high end car so the car makers have had to adapt. 

The ten year model may be coming to an end. Over the air updates, once a year dealer updates, and mileage-based service updates are all now opportunities to ship bug fixes and potentially new features. So it looks like a 5 year model, or even shorter, might be usable.

Currently there are agreements between car makers and their software partners and as these groups both get more familiar with open source hopefully they'll be better prepared to work with the mainline kernel maintainers and to be open about their requirements. Right now unfortunately they are not able to do that.

[snip]

>>> I'm genuinely curious about this, I haven't heard this directly from
>>> users before, only from companies who are in this line of work, wanting
>>> help in doing this for them, for a variety of odd reasons.
>> 
>> If it helps at all, I can bring up this topic inside GENIVI and ask if
>> there are OEMs, Tier 1s and others who would be interested in how to
>> identify a kernel that is appropriate for their long-term needs. I
>> have participated in GENIVI discussions like this previously and there
>> has not necessarily been clarity. Having your perspective and the
>> perspective of others with experience in kernel maintenance would be
>> helpful.
> 
> Please do.  My view of GENIVI from the outside is that it reminds me a
> lot of the problems that plagued the old CGL initiative.  Hopefully that
> is incorrect on my part.

There are certain overlaps between the two projects that's for sure.

> If there's anyone, or any group, I should be talking to about this, or
> any meeting I could attend to help explain it all better, please let me
> know, I am more than willing to do so.

I proposed that the GENIVI system architects meet with you to discuss this, but currently there is no clear picture inside the OEMs about identifying even a particular kernel version let alone how long it needs to be maintained, so it was deemed that a meeting now would not be very productive. I've encouraged individual companies to get in touch with you on one of the kernel lists or drop you an email.

Individual GENIVI members attend the regular Linux conferences and there may be an automotive BoF in Prague if you're there.

>>> If so, doesn't this imply that maybe those users should be choosing a
>>> different company for this support, or that they have given up on this
>>> and want to work directly with the community instead?  
>> 
>> That is the eternal question. For the auto industry it often boils
>> down to the cost / benefit ratio and the cost sensitivity for
>> production vehicles per unit is a major factor in what they choose. I
>> think if they can find a reasonable long-term kernel they'll help
>> maintain it in conjunction with the community.
> 
> That's good to hear, any help is appreciated.
> 
> Mostly I want to know what patches should be applied, that fix problems
> they have.  That and testing the -rc releases would be wonderful.

I'll make sure to communicate this since the GENIVI releases are really more of a developer's baseline than they are polished production images. So it seems like the opportunity to test -rc releases is there.

>>> If the latter,
>>> I'd be very happy to work with them, contacts are greatly appreciated.
>> 
>> Very generous of you. Let me see if I can pull together a list of
>> people where this can be discussed.
> 
> That would be great, odds are, a new thread can be started, and everyone
> on cc: taken off, as I doubt they care about this :)

I'm following up with everyone on cc just so that they can see the outcome of this thread, which sadly I think is anti-climactic. I hope to drag GENIVI members onto the public kernel lists when they have questions.

Regards,

Jeremiah

Re: Future of the -longterm kernel releases (i.e. how we pick them).

[Greg KH]

On Mon, Aug 15, 2011 at 10:26:54PM -0700, Daniel Taylor wrote:
> For embedded systems, which often have multi- (not just two-) year
> warranties, it sounds like we would then be maintaining the -longterm
> ourselves.  The "easy" side of this is that embedded systems don't
> (usually) have users trying to plug in the latest and greatest widget,
> except on external busses, such as USB.  So back-porting drivers is
> not generally a big deal.  Security patches, OTOH, are a real concern.
> Other than net and drivers/net, which are exposed to non-local attacks,
> are there other subsystems we should watch for security fixes?  There
> have, for example, been a lot of security upgrades in /sys, but there's
> also been a lot of restructing for new interfaces and "arch" models,
> which would make back-porting those much more complicated.

You should watch out for all bugfixes, that's the only way to have a
chance at catching all of the issues.

good luck,

greg k-h

Re: Future of the -longterm kernel releases (i.e. how we pick them).

[Brian Swetland]

On Tue, Aug 23, 2011 at 9:57 PM, Greg KH <greg@kroah.com> wrote:
> On Wed, Aug 17, 2011 at 10:33:05AM -0700, Brian Swetland wrote:
>> As far as long-term kernels goes, from the Android perspective we
>> strongly prefer to snap up to the most recent released kernel on every
>> platform/device release.  I prefer to be as up to date on bugfixes and
>> features from mainline as possible and minimize the deltas on our
>> stack 'o patches as much as possible.
>
> That's good to hear.
>
>> We've been getting more aggressive about merging in the -rc#s and then
>> rebasing on the final during development (before final stabilization
>> freeze for a release) in the last year or so, and it seems to work
>> pretty well.
>
> Is your kernel git tree public during this merge cycle so that others
> can track it?  I tried to dig through android.kernel.org but there are a
> lot of different kernel trees there :(

We really need a "which branch is which" quick guide that's easily
findable.  kernel/common is always where our generic patch stack
lives, and it looks like android-3.0 is the most recent (which has
3.0.1 merged in).

http://android.git.kernel.org/?p=kernel/common.git;a=summary

>> Not all OEMs or silicon vendors are as comfortable with this approach
>> and I expect some would welcome a long term tree they can count on to
>> get critical fixes for, though.  Device release teams (even our own,
>> for lead devices we work on) are notoriously risk-averse (often for
>> good reasons), and tend to get really nervous when they see the total
>> change count or diff stat for pulling up to a new kernel release close
>> to "final rom" for a product.
>
> That's understandable.
>
> So that kind of means they would like such a longterm tree, but they
> have the problem that your patches move forward to a newer kernel, but
> are not maintained for that longterm kernel.  Are they responsible for
> maintaining this themselves then, or do you help with providing your
> patchset on top of all of the different kernel versions that the OEMs
> and others need?

So far the silicon vendors have been doing more of the longer term
kernel stuff, as it better matches their model of providing "boxed"
stable BSPs of some sort.  It might make sense for us to ensure
backports of fixes for core patches (our common tree) are available
for the LT kernels, but we are unlikely to have the necessary staff to
maintain LT trees for every SoC we interact with.

Brian

Re: Future of the -longterm kernel releases (i.e. how we pick them).

[Greg KH]

On Wed, Aug 24, 2011 at 09:49:22PM -0700, Brian Swetland wrote:
> On Tue, Aug 23, 2011 at 9:57 PM, Greg KH <greg@kroah.com> wrote:
> > On Wed, Aug 17, 2011 at 10:33:05AM -0700, Brian Swetland wrote:
> >> As far as long-term kernels goes, from the Android perspective we
> >> strongly prefer to snap up to the most recent released kernel on every
> >> platform/device release.  I prefer to be as up to date on bugfixes and
> >> features from mainline as possible and minimize the deltas on our
> >> stack 'o patches as much as possible.
> >
> > That's good to hear.
> >
> >> We've been getting more aggressive about merging in the -rc#s and then
> >> rebasing on the final during development (before final stabilization
> >> freeze for a release) in the last year or so, and it seems to work
> >> pretty well.
> >
> > Is your kernel git tree public during this merge cycle so that others
> > can track it?  I tried to dig through android.kernel.org but there are a
> > lot of different kernel trees there :(
> 
> We really need a "which branch is which" quick guide that's easily
> findable.  kernel/common is always where our generic patch stack
> lives, and it looks like android-3.0 is the most recent (which has
> 3.0.1 merged in).
> 
> http://android.git.kernel.org/?p=kernel/common.git;a=summary

Thanks for the pointer.

If you ever get such a quick guide, I'd appreciate a link to it.

greg k-h