* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-17 19:48 ` Scott Branden
0 siblings, 0 replies; 133+ messages in thread
From: Scott Branden @ 2021-02-17 19:48 UTC (permalink / raw)
To: Greg Kroah-Hartman; +Cc: BCM Kernel Feedback, LKML, Linux ARM
[-- Attachment #1.1: Type: text/plain, Size: 6621 bytes --]
Hi Greg,
On 2021-02-17 1:40 a.m., Greg Kroah-Hartman wrote:
> On Tue, Jan 26, 2021 at 07:51:18PM +0100, Greg Kroah-Hartman wrote:
>> On Tue, Jan 26, 2021 at 10:30:16AM -0800, Scott Branden wrote:
>>> Hi Greg,
>>>
>>>
>>> On 2021-01-25 11:29 p.m., Greg Kroah-Hartman wrote:
>>>> On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote:
>>>>> Hi All,
>>>>>
>>>>> The 5.10 LTS kernel being officially LTS supported for 2 years presents a problem:
>>>>> why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel has a 6 year LTS.
>>>> Because they want to use all of the latest stuff that 5.10 provides
>>>> them. Don't you want faster and more secure kernels for your devices?
>>> Yes, 5.10 is a more secure and less buggy kernel than 5.4.
>>
>> Great, use it, ship it to your customers and we are all happy. What do
>> you need me for any of this? :)
>>
>>>>> And AOSP has already declared the use
>>>>> of 5.10 kernel in their Android S and T releases.
>>>> Publically? Where? And is that really the name of the new Android
>>>> releases, I thought they switched to numbers now (hence the naming of
>>>> the current android-common kernel branches, marketing is fun...)
>>> https://source.android.com/devices/architecture/kernel/android-common
>>> Feature and launch kernels provides kernels supported per version.
>>
>> Oh nice, didn't know that.
>>
>> But note, Android kernels do not reflect the lifespan of LTS kernels.
>> If that were the case, I would still be supporting 3.18 as they are
>> doing that at the moment for their devices and customers, and will be
>> doing so for I think another full year.
>>
>> So while Android is nice to see here, remember that is what Google is
>> promising to support for their users. You can do the same thing for
>> your users, what do you need me here for this? You can do the same
>> thing that Google is doing for 3.18 right now, pick the stable fixes
>> from upstream, backport them, test them, and push them out to their
>> users.
>>
>> While Google is a great help to me in the LTS effort, providing huge
>> amounts of resources to enable my life easier with this (i.e. funding
>> Linaro's testing efforts), their promise to their customers/users does
>> not depend on me keeping LTS kernels alive, if I stopped tomorrow their
>> contracts are still in place and they know how to do this work
>> themselves (as is proof with 3.18).
>>
>> So you can provide the same kind of guarantee to support any kernel
>> version for any amount of time to any customer you like, it shouldn't
>> require me to do that work for you, right?
>>
>>>>> Is there some way we could make the LTS support more clear.
>>>>> A 2 year declaration is not LTS any more.
>>>> Not true at all, a "normal" stable kernel is dropped after the next
>>>> release happens, making their lifespan about 4 months long. 2 years is
>>>> much longer than 4 months, so it still is a "long term supported" kernel
>>>> in contrast, correct?
>>> Perhaps a new name needs to be made for "LTS" for 6 years to distinguish it from 2 years.
>>> The timeframes are very different.
>>
>> At this point in time, anyone wanting a kernel longer than 2 years
>> should know how this all works.
>>
>> If not, please do some basic research, I have written whitepapers on
>> this and given numerous talks. The information is out there...
>>
>>>>> If 5.10 is "actually" going to be supported for 6 years it would be quite valuable to make such a declaration.
>>>>> https://www.kernel.org/category/releases.html
>>>> Why? What would that change?
>>>>
>>>> Ok, seriously, this happens every year, and every year we go through the
>>>> same thing, it's not like this is somehow new, right?
>>> No, but why do we need to keep playing the same game every year now.
>>
>> Because, 5.4 almost did not become "6 years" of support from me. That
>> was because in the beginning, no one said they were going to use it in
>> their devices and offer me help in testing and backporting. Only when I
>> knew for sure that we had people helping this out did I change the date
>> on kernel.org.
>>
>> So far the jury is still out for 5.10, are you willing to help with
>> this? If not, why are you willing to hope that others are going to do
>> your work for you? I am talking to some companies, but am not willing
>> to commit to anything in public just yet, because no one has committed
>> to me yet.
>
> Following up on this as I did not hear back from you. Are you and/or
> your company willing to help out with the testing of 5.10 to ensure that
> it is a LTS kernel? So far I have not had any companies agree to help
> out with this effort, which is sad to see as it seems that companies
> want 6 years of stable kernels, yet do not seem to be able to at the
> least, do a test-build/run of those kernels, which is quite odd...
I personally cannot commit to supporting this kernel for 6 years
(and personally do not want to backport new features to a 6 year old kernel).
And customers are finicky and ask for one thing and then change their mind later.
We'll have to see what decisions are made at a company level for this as there
are added costs to run tests on LTS kernel branches. We already run extensive QA on
whatever active development branches are in use and a subset on the mainline
branch as well. QA resources are finite and committing those for 6 years is
not something that makes sense if customers drop that kernel version.
Testing of the LTS kernel changes really moves out of our hands and into the
customer's testing after our major releases to them.
Other difficulty with the LTS version is the frequency it is updated. We would not
pickup the changes that frequently to test. A quarterly, bi-annually, or when a critical fix
is identified would be when we update and perform any meaningful testing when in maintainence.
>
> If you want to point people at your company this link that explains it
> all in a single location instead of an email thread:
> http://www.kroah.com/log/blog/2021/02/03/helping-out-with-lts-kernel-releases/
> that would be great.
Thanks for the link. Will forward it to see if it helps make any commitment from customers
or company.
The link Pavel sent looks interesting.
"CIP project intends to maintain 5.10 for a long time, so you can expect testing/help from them:
https://www.cip-project.org/blog/2020/12/02/cip-to-embark-on-kernel-5-10-development-for-slts
"
If the CIP project has committed to 10 years you would think they would be in contact
with you to add their support to the 5.10 LTS effort.
>
> thanks,
>
> greg k-h
>
Thanks,
Scott
[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4169 bytes --]
[-- Attachment #2: Type: text/plain, Size: 176 bytes --]
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-17 19:48 ` Scott Branden
@ 2021-02-18 7:43 ` Greg Kroah-Hartman
-1 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-18 7:43 UTC (permalink / raw)
To: Scott Branden; +Cc: Linux ARM, LKML, BCM Kernel Feedback
On Wed, Feb 17, 2021 at 11:48:21AM -0800, Scott Branden wrote:
> Hi Greg,
>
> On 2021-02-17 1:40 a.m., Greg Kroah-Hartman wrote:
> > On Tue, Jan 26, 2021 at 07:51:18PM +0100, Greg Kroah-Hartman wrote:
> >> On Tue, Jan 26, 2021 at 10:30:16AM -0800, Scott Branden wrote:
> >>> Hi Greg,
> >>>
> >>>
> >>> On 2021-01-25 11:29 p.m., Greg Kroah-Hartman wrote:
> >>>> On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote:
> >>>>> Hi All,
> >>>>>
> >>>>> The 5.10 LTS kernel being officially LTS supported for 2 years presents a problem:
> >>>>> why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel has a 6 year LTS.
> >>>> Because they want to use all of the latest stuff that 5.10 provides
> >>>> them. Don't you want faster and more secure kernels for your devices?
> >>> Yes, 5.10 is a more secure and less buggy kernel than 5.4.
> >>
> >> Great, use it, ship it to your customers and we are all happy. What do
> >> you need me for any of this? :)
> >>
> >>>>> And AOSP has already declared the use
> >>>>> of 5.10 kernel in their Android S and T releases.
> >>>> Publically? Where? And is that really the name of the new Android
> >>>> releases, I thought they switched to numbers now (hence the naming of
> >>>> the current android-common kernel branches, marketing is fun...)
> >>> https://source.android.com/devices/architecture/kernel/android-common
> >>> Feature and launch kernels provides kernels supported per version.
> >>
> >> Oh nice, didn't know that.
> >>
> >> But note, Android kernels do not reflect the lifespan of LTS kernels.
> >> If that were the case, I would still be supporting 3.18 as they are
> >> doing that at the moment for their devices and customers, and will be
> >> doing so for I think another full year.
> >>
> >> So while Android is nice to see here, remember that is what Google is
> >> promising to support for their users. You can do the same thing for
> >> your users, what do you need me here for this? You can do the same
> >> thing that Google is doing for 3.18 right now, pick the stable fixes
> >> from upstream, backport them, test them, and push them out to their
> >> users.
> >>
> >> While Google is a great help to me in the LTS effort, providing huge
> >> amounts of resources to enable my life easier with this (i.e. funding
> >> Linaro's testing efforts), their promise to their customers/users does
> >> not depend on me keeping LTS kernels alive, if I stopped tomorrow their
> >> contracts are still in place and they know how to do this work
> >> themselves (as is proof with 3.18).
> >>
> >> So you can provide the same kind of guarantee to support any kernel
> >> version for any amount of time to any customer you like, it shouldn't
> >> require me to do that work for you, right?
> >>
> >>>>> Is there some way we could make the LTS support more clear.
> >>>>> A 2 year declaration is not LTS any more.
> >>>> Not true at all, a "normal" stable kernel is dropped after the next
> >>>> release happens, making their lifespan about 4 months long. 2 years is
> >>>> much longer than 4 months, so it still is a "long term supported" kernel
> >>>> in contrast, correct?
> >>> Perhaps a new name needs to be made for "LTS" for 6 years to distinguish it from 2 years.
> >>> The timeframes are very different.
> >>
> >> At this point in time, anyone wanting a kernel longer than 2 years
> >> should know how this all works.
> >>
> >> If not, please do some basic research, I have written whitepapers on
> >> this and given numerous talks. The information is out there...
> >>
> >>>>> If 5.10 is "actually" going to be supported for 6 years it would be quite valuable to make such a declaration.
> >>>>> https://www.kernel.org/category/releases.html
> >>>> Why? What would that change?
> >>>>
> >>>> Ok, seriously, this happens every year, and every year we go through the
> >>>> same thing, it's not like this is somehow new, right?
> >>> No, but why do we need to keep playing the same game every year now.
> >>
> >> Because, 5.4 almost did not become "6 years" of support from me. That
> >> was because in the beginning, no one said they were going to use it in
> >> their devices and offer me help in testing and backporting. Only when I
> >> knew for sure that we had people helping this out did I change the date
> >> on kernel.org.
> >>
> >> So far the jury is still out for 5.10, are you willing to help with
> >> this? If not, why are you willing to hope that others are going to do
> >> your work for you? I am talking to some companies, but am not willing
> >> to commit to anything in public just yet, because no one has committed
> >> to me yet.
> >
> > Following up on this as I did not hear back from you. Are you and/or
> > your company willing to help out with the testing of 5.10 to ensure that
> > it is a LTS kernel? So far I have not had any companies agree to help
> > out with this effort, which is sad to see as it seems that companies
> > want 6 years of stable kernels, yet do not seem to be able to at the
> > least, do a test-build/run of those kernels, which is quite odd...
> I personally cannot commit to supporting this kernel for 6 years
> (and personally do not want to backport new features to a 6 year old kernel).
Why would you backport new features to an old kernel? That's not what
they are there for.
> And customers are finicky and ask for one thing and then change their mind later.
Sure, but it's up to you to define what you will support, for your
customers, which is what I am asking here.
> We'll have to see what decisions are made at a company level for this as there
> are added costs to run tests on LTS kernel branches. We already run extensive QA on
> whatever active development branches are in use and a subset on the mainline
> branch as well. QA resources are finite and committing those for 6 years is
> not something that makes sense if customers drop that kernel version.
> Testing of the LTS kernel changes really moves out of our hands and into the
> customer's testing after our major releases to them.
That feels wrong, but hey, I'm not your customer. If I was, I would be
demanding all LTS updates for the specific kernel branch you gave me as
I am paying for that support :)
> Other difficulty with the LTS version is the frequency it is updated. We would not
> pickup the changes that frequently to test. A quarterly, bi-annually, or when a critical fix
> is identified would be when we update and perform any meaningful testing when in maintainence.
How are you "identifying" these "critical fixes"? We fix at least one
known security issue a week, and probably multitudes of
unknown-at-this-moment ones. How are you determining when you need to
send a new base kernel update off to your customers? At such long
intervals it feels like anyone using your kernel releases is woefully
insecure.
Although if you do get people to update on a quarterly basis no matter
what, that's better than nothing, I'll take that. What's the odds that
this really happens?
> > If you want to point people at your company this link that explains it
> > all in a single location instead of an email thread:
> > http://www.kroah.com/log/blog/2021/02/03/helping-out-with-lts-kernel-releases/
> > that would be great.
> Thanks for the link. Will forward it to see if it helps make any commitment from customers
> or company.
Thanks. When can I expect to hear something back, even if it is a "no,
we don't care about supporting 5.10"?
> The link Pavel sent looks interesting.
> "CIP project intends to maintain 5.10 for a long time, so you can expect testing/help from them:
> https://www.cip-project.org/blog/2020/12/02/cip-to-embark-on-kernel-5-10-development-for-slts
> "
>
> If the CIP project has committed to 10 years you would think they would be in contact
> with you to add their support to the 5.10 LTS effort.
They are doing testing right now, see the announcements where they test
each stable -rc release. But they have not talked to me about 5.10
yet, Their model is that they will, somehow in a way that is yet to be
determined, take over maintaining these releases _after_ I drop them.
But they are only going to do so for a very specific hardware platform
or two, so anyone using anything other than their specific boards, is
going to be out of luck.
Which makes sense, scoping support like this to those that actually care
about a specific hardware platform is much easier than the work that I
and Sasha do in support it for all platforms. So if you are interested
in 10 years, please work with CIP to get your platform into that list.
thanks,
greg k-h
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 7:43 ` Greg Kroah-Hartman
0 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-18 7:43 UTC (permalink / raw)
To: Scott Branden; +Cc: BCM Kernel Feedback, LKML, Linux ARM
On Wed, Feb 17, 2021 at 11:48:21AM -0800, Scott Branden wrote:
> Hi Greg,
>
> On 2021-02-17 1:40 a.m., Greg Kroah-Hartman wrote:
> > On Tue, Jan 26, 2021 at 07:51:18PM +0100, Greg Kroah-Hartman wrote:
> >> On Tue, Jan 26, 2021 at 10:30:16AM -0800, Scott Branden wrote:
> >>> Hi Greg,
> >>>
> >>>
> >>> On 2021-01-25 11:29 p.m., Greg Kroah-Hartman wrote:
> >>>> On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote:
> >>>>> Hi All,
> >>>>>
> >>>>> The 5.10 LTS kernel being officially LTS supported for 2 years presents a problem:
> >>>>> why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel has a 6 year LTS.
> >>>> Because they want to use all of the latest stuff that 5.10 provides
> >>>> them. Don't you want faster and more secure kernels for your devices?
> >>> Yes, 5.10 is a more secure and less buggy kernel than 5.4.
> >>
> >> Great, use it, ship it to your customers and we are all happy. What do
> >> you need me for any of this? :)
> >>
> >>>>> And AOSP has already declared the use
> >>>>> of 5.10 kernel in their Android S and T releases.
> >>>> Publically? Where? And is that really the name of the new Android
> >>>> releases, I thought they switched to numbers now (hence the naming of
> >>>> the current android-common kernel branches, marketing is fun...)
> >>> https://source.android.com/devices/architecture/kernel/android-common
> >>> Feature and launch kernels provides kernels supported per version.
> >>
> >> Oh nice, didn't know that.
> >>
> >> But note, Android kernels do not reflect the lifespan of LTS kernels.
> >> If that were the case, I would still be supporting 3.18 as they are
> >> doing that at the moment for their devices and customers, and will be
> >> doing so for I think another full year.
> >>
> >> So while Android is nice to see here, remember that is what Google is
> >> promising to support for their users. You can do the same thing for
> >> your users, what do you need me here for this? You can do the same
> >> thing that Google is doing for 3.18 right now, pick the stable fixes
> >> from upstream, backport them, test them, and push them out to their
> >> users.
> >>
> >> While Google is a great help to me in the LTS effort, providing huge
> >> amounts of resources to enable my life easier with this (i.e. funding
> >> Linaro's testing efforts), their promise to their customers/users does
> >> not depend on me keeping LTS kernels alive, if I stopped tomorrow their
> >> contracts are still in place and they know how to do this work
> >> themselves (as is proof with 3.18).
> >>
> >> So you can provide the same kind of guarantee to support any kernel
> >> version for any amount of time to any customer you like, it shouldn't
> >> require me to do that work for you, right?
> >>
> >>>>> Is there some way we could make the LTS support more clear.
> >>>>> A 2 year declaration is not LTS any more.
> >>>> Not true at all, a "normal" stable kernel is dropped after the next
> >>>> release happens, making their lifespan about 4 months long. 2 years is
> >>>> much longer than 4 months, so it still is a "long term supported" kernel
> >>>> in contrast, correct?
> >>> Perhaps a new name needs to be made for "LTS" for 6 years to distinguish it from 2 years.
> >>> The timeframes are very different.
> >>
> >> At this point in time, anyone wanting a kernel longer than 2 years
> >> should know how this all works.
> >>
> >> If not, please do some basic research, I have written whitepapers on
> >> this and given numerous talks. The information is out there...
> >>
> >>>>> If 5.10 is "actually" going to be supported for 6 years it would be quite valuable to make such a declaration.
> >>>>> https://www.kernel.org/category/releases.html
> >>>> Why? What would that change?
> >>>>
> >>>> Ok, seriously, this happens every year, and every year we go through the
> >>>> same thing, it's not like this is somehow new, right?
> >>> No, but why do we need to keep playing the same game every year now.
> >>
> >> Because, 5.4 almost did not become "6 years" of support from me. That
> >> was because in the beginning, no one said they were going to use it in
> >> their devices and offer me help in testing and backporting. Only when I
> >> knew for sure that we had people helping this out did I change the date
> >> on kernel.org.
> >>
> >> So far the jury is still out for 5.10, are you willing to help with
> >> this? If not, why are you willing to hope that others are going to do
> >> your work for you? I am talking to some companies, but am not willing
> >> to commit to anything in public just yet, because no one has committed
> >> to me yet.
> >
> > Following up on this as I did not hear back from you. Are you and/or
> > your company willing to help out with the testing of 5.10 to ensure that
> > it is a LTS kernel? So far I have not had any companies agree to help
> > out with this effort, which is sad to see as it seems that companies
> > want 6 years of stable kernels, yet do not seem to be able to at the
> > least, do a test-build/run of those kernels, which is quite odd...
> I personally cannot commit to supporting this kernel for 6 years
> (and personally do not want to backport new features to a 6 year old kernel).
Why would you backport new features to an old kernel? That's not what
they are there for.
> And customers are finicky and ask for one thing and then change their mind later.
Sure, but it's up to you to define what you will support, for your
customers, which is what I am asking here.
> We'll have to see what decisions are made at a company level for this as there
> are added costs to run tests on LTS kernel branches. We already run extensive QA on
> whatever active development branches are in use and a subset on the mainline
> branch as well. QA resources are finite and committing those for 6 years is
> not something that makes sense if customers drop that kernel version.
> Testing of the LTS kernel changes really moves out of our hands and into the
> customer's testing after our major releases to them.
That feels wrong, but hey, I'm not your customer. If I was, I would be
demanding all LTS updates for the specific kernel branch you gave me as
I am paying for that support :)
> Other difficulty with the LTS version is the frequency it is updated. We would not
> pickup the changes that frequently to test. A quarterly, bi-annually, or when a critical fix
> is identified would be when we update and perform any meaningful testing when in maintainence.
How are you "identifying" these "critical fixes"? We fix at least one
known security issue a week, and probably multitudes of
unknown-at-this-moment ones. How are you determining when you need to
send a new base kernel update off to your customers? At such long
intervals it feels like anyone using your kernel releases is woefully
insecure.
Although if you do get people to update on a quarterly basis no matter
what, that's better than nothing, I'll take that. What's the odds that
this really happens?
> > If you want to point people at your company this link that explains it
> > all in a single location instead of an email thread:
> > http://www.kroah.com/log/blog/2021/02/03/helping-out-with-lts-kernel-releases/
> > that would be great.
> Thanks for the link. Will forward it to see if it helps make any commitment from customers
> or company.
Thanks. When can I expect to hear something back, even if it is a "no,
we don't care about supporting 5.10"?
> The link Pavel sent looks interesting.
> "CIP project intends to maintain 5.10 for a long time, so you can expect testing/help from them:
> https://www.cip-project.org/blog/2020/12/02/cip-to-embark-on-kernel-5-10-development-for-slts
> "
>
> If the CIP project has committed to 10 years you would think they would be in contact
> with you to add their support to the 5.10 LTS effort.
They are doing testing right now, see the announcements where they test
each stable -rc release. But they have not talked to me about 5.10
yet, Their model is that they will, somehow in a way that is yet to be
determined, take over maintaining these releases _after_ I drop them.
But they are only going to do so for a very specific hardware platform
or two, so anyone using anything other than their specific boards, is
going to be out of luck.
Which makes sense, scoping support like this to those that actually care
about a specific hardware platform is much easier than the work that I
and Sasha do in support it for all platforms. So if you are interested
in 10 years, please work with CIP to get your platform into that list.
thanks,
greg k-h
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 7:43 ` Greg Kroah-Hartman
@ 2021-02-18 11:31 ` Willy Tarreau
-1 siblings, 0 replies; 133+ messages in thread
From: Willy Tarreau @ 2021-02-18 11:31 UTC (permalink / raw)
To: Greg Kroah-Hartman, Scott Branden; +Cc: Linux ARM, LKML, BCM Kernel Feedback
On Thu, Feb 18, 2021 at 08:43:48AM +0100, Greg Kroah-Hartman wrote:
> On Wed, Feb 17, 2021 at 11:48:21AM -0800, Scott Branden wrote:
> > Other difficulty with the LTS version is the frequency it is updated.
What a stange statement! So basically if fixes come in quickly so that
customers are not exposed too long to well-known issues, it's a difficulty ?
I guess by now every serious OS vendor provides at least weekly fixes, and
at an era where devices are all interconnected, it's really necessary
(unless of course you don't care about your customer's security).
> > We would not
> > pickup the changes that frequently to test. A quarterly, bi-annually, or when a critical fix
> > is identified would be when we update and perform any meaningful testing when in maintainence.
>
> How are you "identifying" these "critical fixes"? We fix at least one
> known security issue a week, and probably multitudes of
> unknown-at-this-moment ones. How are you determining when you need to
> send a new base kernel update off to your customers? At such long
> intervals it feels like anyone using your kernel releases is woefully
> insecure.
+1! It seems like this dangerous practice will never end :-(
Let me explain a personal experience. When I took over 2.6.32 many years
ago, Greg asked me to adapt to the new maintenance process involving the
patch reviews. At first I feared that it would increase my amount of work.
And it did. But I also discovered how important these reviews were, because
I started to get lots of "don't take this one in this version" and more
importantly "if you merge this you'll need these ones as well". And very
quickly I discovered how bogus the branches I used to maintain before
had been, given the high feedback ratio!
So based on this experience, I can assure anyone doing cherry-picks in
their garage from LTS kernels that they're doing crap and that they must
not distribute these kernels to anyone because THESE KERNELS ARE DANGEROUS.
It's even very easy to introduce vulnerabilities by doing this!
The only set of fixes that can be trusted are the "official" stable
kernels, because they are the only ones that are approved by the patches
authors themselves. Adding more stuff on top of stable kernels is fine
(and done at your own risk), but randomly dropping stuff from stable
kernels just because you don't think you need that is totally non-sense
and must not be done anymore!
Willy
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 11:31 ` Willy Tarreau
0 siblings, 0 replies; 133+ messages in thread
From: Willy Tarreau @ 2021-02-18 11:31 UTC (permalink / raw)
To: Greg Kroah-Hartman, Scott Branden; +Cc: BCM Kernel Feedback, LKML, Linux ARM
On Thu, Feb 18, 2021 at 08:43:48AM +0100, Greg Kroah-Hartman wrote:
> On Wed, Feb 17, 2021 at 11:48:21AM -0800, Scott Branden wrote:
> > Other difficulty with the LTS version is the frequency it is updated.
What a stange statement! So basically if fixes come in quickly so that
customers are not exposed too long to well-known issues, it's a difficulty ?
I guess by now every serious OS vendor provides at least weekly fixes, and
at an era where devices are all interconnected, it's really necessary
(unless of course you don't care about your customer's security).
> > We would not
> > pickup the changes that frequently to test. A quarterly, bi-annually, or when a critical fix
> > is identified would be when we update and perform any meaningful testing when in maintainence.
>
> How are you "identifying" these "critical fixes"? We fix at least one
> known security issue a week, and probably multitudes of
> unknown-at-this-moment ones. How are you determining when you need to
> send a new base kernel update off to your customers? At such long
> intervals it feels like anyone using your kernel releases is woefully
> insecure.
+1! It seems like this dangerous practice will never end :-(
Let me explain a personal experience. When I took over 2.6.32 many years
ago, Greg asked me to adapt to the new maintenance process involving the
patch reviews. At first I feared that it would increase my amount of work.
And it did. But I also discovered how important these reviews were, because
I started to get lots of "don't take this one in this version" and more
importantly "if you merge this you'll need these ones as well". And very
quickly I discovered how bogus the branches I used to maintain before
had been, given the high feedback ratio!
So based on this experience, I can assure anyone doing cherry-picks in
their garage from LTS kernels that they're doing crap and that they must
not distribute these kernels to anyone because THESE KERNELS ARE DANGEROUS.
It's even very easy to introduce vulnerabilities by doing this!
The only set of fixes that can be trusted are the "official" stable
kernels, because they are the only ones that are approved by the patches
authors themselves. Adding more stuff on top of stable kernels is fine
(and done at your own risk), but randomly dropping stuff from stable
kernels just because you don't think you need that is totally non-sense
and must not be done anymore!
Willy
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 11:31 ` Willy Tarreau
@ 2021-02-18 14:15 ` Jari Ruusu
-1 siblings, 0 replies; 133+ messages in thread
From: Jari Ruusu @ 2021-02-18 14:15 UTC (permalink / raw)
To: Willy Tarreau
Cc: Greg Kroah-Hartman, Scott Branden, Linux ARM, LKML, BCM Kernel Feedback
Willy Tarreau wrote:
> The only set of fixes that can be trusted are the "official" stable
> kernels, because they are the only ones that are approved by the patches
> authors themselves. Adding more stuff on top of stable kernels is fine
> (and done at your own risk), but randomly dropping stuff from stable
> kernels just because you don't think you need that is totally non-sense
> and must not be done anymore!
This may be little bit off-topic... but stable kernel.org kernels
can also bit-rot badly because of "selective" backporting... as in
anything that does not apply cleanly gets dropped regardless of
how critical they are.
I will give you one example: Intel WiFi (iwlwifi) on 4.19.y
kernel.org stable kernels is currently missing many critical
locking fixes. As a result, that in-tree iwlwifi driver causes
erratic behavior to random unrelated processes, and has been doing
so for many months now. My not-so-politically correct opinion is
that in-tree iwlwifi is completely FUBAR unless someone steps up
to do professional quality backport of those locking fixes from
upstream out-of-tree Intel version [1] [2] of the driver. For me
only way to get properly working WiFi on my laptop computer is to
compile that Intel out-of-tree version. Sad, but true.
[1] https://wireless.wiki.kernel.org/en/users/drivers/iwlwifi/core_release
[2] https://git.kernel.org/pub/scm/linux/kernel/git/iwlwifi/backport-iwlwifi.git/
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 14:15 ` Jari Ruusu
0 siblings, 0 replies; 133+ messages in thread
From: Jari Ruusu @ 2021-02-18 14:15 UTC (permalink / raw)
To: Willy Tarreau
Cc: Greg Kroah-Hartman, BCM Kernel Feedback, LKML, Linux ARM, Scott Branden
Willy Tarreau wrote:
> The only set of fixes that can be trusted are the "official" stable
> kernels, because they are the only ones that are approved by the patches
> authors themselves. Adding more stuff on top of stable kernels is fine
> (and done at your own risk), but randomly dropping stuff from stable
> kernels just because you don't think you need that is totally non-sense
> and must not be done anymore!
This may be little bit off-topic... but stable kernel.org kernels
can also bit-rot badly because of "selective" backporting... as in
anything that does not apply cleanly gets dropped regardless of
how critical they are.
I will give you one example: Intel WiFi (iwlwifi) on 4.19.y
kernel.org stable kernels is currently missing many critical
locking fixes. As a result, that in-tree iwlwifi driver causes
erratic behavior to random unrelated processes, and has been doing
so for many months now. My not-so-politically correct opinion is
that in-tree iwlwifi is completely FUBAR unless someone steps up
to do professional quality backport of those locking fixes from
upstream out-of-tree Intel version [1] [2] of the driver. For me
only way to get properly working WiFi on my laptop computer is to
compile that Intel out-of-tree version. Sad, but true.
[1] https://wireless.wiki.kernel.org/en/users/drivers/iwlwifi/core_release
[2] https://git.kernel.org/pub/scm/linux/kernel/git/iwlwifi/backport-iwlwifi.git/
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 14:15 ` Jari Ruusu
@ 2021-02-18 14:29 ` Greg Kroah-Hartman
-1 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-18 14:29 UTC (permalink / raw)
To: Jari Ruusu
Cc: Willy Tarreau, Scott Branden, Linux ARM, LKML, BCM Kernel Feedback
On Thu, Feb 18, 2021 at 04:15:11PM +0200, Jari Ruusu wrote:
> Willy Tarreau wrote:
> > The only set of fixes that can be trusted are the "official" stable
> > kernels, because they are the only ones that are approved by the patches
> > authors themselves. Adding more stuff on top of stable kernels is fine
> > (and done at your own risk), but randomly dropping stuff from stable
> > kernels just because you don't think you need that is totally non-sense
> > and must not be done anymore!
>
> This may be little bit off-topic... but stable kernel.org kernels
> can also bit-rot badly because of "selective" backporting... as in
> anything that does not apply cleanly gets dropped regardless of
> how critical they are.
>
> I will give you one example: Intel WiFi (iwlwifi) on 4.19.y
> kernel.org stable kernels is currently missing many critical
> locking fixes.
Why has no one asked for the specific upstream commits to be backported
if this is the case?
> As a result, that in-tree iwlwifi driver causes
> erratic behavior to random unrelated processes, and has been doing
> so for many months now. My not-so-politically correct opinion is
> that in-tree iwlwifi is completely FUBAR unless someone steps up
> to do professional quality backport of those locking fixes from
> upstream out-of-tree Intel version [1] [2] of the driver.
Why does any out-of-tree driver come into play here? What is wrong with
the in-kernel code?
> For me
> only way to get properly working WiFi on my laptop computer is to
> compile that Intel out-of-tree version. Sad, but true.
Why use 4.19.y on a laptop in the firstplace? That feels very wrong and
is not the recommended thing to use the LTS kernels for.
thanks,
greg k-h
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 14:29 ` Greg Kroah-Hartman
0 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-18 14:29 UTC (permalink / raw)
To: Jari Ruusu
Cc: LKML, BCM Kernel Feedback, Willy Tarreau, Linux ARM, Scott Branden
On Thu, Feb 18, 2021 at 04:15:11PM +0200, Jari Ruusu wrote:
> Willy Tarreau wrote:
> > The only set of fixes that can be trusted are the "official" stable
> > kernels, because they are the only ones that are approved by the patches
> > authors themselves. Adding more stuff on top of stable kernels is fine
> > (and done at your own risk), but randomly dropping stuff from stable
> > kernels just because you don't think you need that is totally non-sense
> > and must not be done anymore!
>
> This may be little bit off-topic... but stable kernel.org kernels
> can also bit-rot badly because of "selective" backporting... as in
> anything that does not apply cleanly gets dropped regardless of
> how critical they are.
>
> I will give you one example: Intel WiFi (iwlwifi) on 4.19.y
> kernel.org stable kernels is currently missing many critical
> locking fixes.
Why has no one asked for the specific upstream commits to be backported
if this is the case?
> As a result, that in-tree iwlwifi driver causes
> erratic behavior to random unrelated processes, and has been doing
> so for many months now. My not-so-politically correct opinion is
> that in-tree iwlwifi is completely FUBAR unless someone steps up
> to do professional quality backport of those locking fixes from
> upstream out-of-tree Intel version [1] [2] of the driver.
Why does any out-of-tree driver come into play here? What is wrong with
the in-kernel code?
> For me
> only way to get properly working WiFi on my laptop computer is to
> compile that Intel out-of-tree version. Sad, but true.
Why use 4.19.y on a laptop in the firstplace? That feels very wrong and
is not the recommended thing to use the LTS kernels for.
thanks,
greg k-h
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 14:29 ` Greg Kroah-Hartman
@ 2021-02-18 20:55 ` Pavel Machek
-1 siblings, 0 replies; 133+ messages in thread
From: Pavel Machek @ 2021-02-18 20:55 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Jari Ruusu, Willy Tarreau, Scott Branden, Linux ARM, LKML,
BCM Kernel Feedback
[-- Attachment #1: Type: text/plain, Size: 534 bytes --]
Hi!
> > For me
> > only way to get properly working WiFi on my laptop computer is to
> > compile that Intel out-of-tree version. Sad, but true.
>
> Why use 4.19.y on a laptop in the firstplace? That feels very wrong and
> is not the recommended thing to use the LTS kernels for.
Well, that's actually what distributions are doing, for example Debian
10.8 is on 4.19...
I expect -stable is what most users are running on their notebooks.
Best regards,
Pavel
--
http://www.livejournal.com/~pavelmachek
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 20:55 ` Pavel Machek
0 siblings, 0 replies; 133+ messages in thread
From: Pavel Machek @ 2021-02-18 20:55 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Jari Ruusu, Scott Branden, LKML, BCM Kernel Feedback,
Willy Tarreau, Linux ARM
[-- Attachment #1.1: Type: text/plain, Size: 534 bytes --]
Hi!
> > For me
> > only way to get properly working WiFi on my laptop computer is to
> > compile that Intel out-of-tree version. Sad, but true.
>
> Why use 4.19.y on a laptop in the firstplace? That feels very wrong and
> is not the recommended thing to use the LTS kernels for.
Well, that's actually what distributions are doing, for example Debian
10.8 is on 4.19...
I expect -stable is what most users are running on their notebooks.
Best regards,
Pavel
--
http://www.livejournal.com/~pavelmachek
[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
[-- Attachment #2: Type: text/plain, Size: 176 bytes --]
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 20:55 ` Pavel Machek
@ 2021-02-18 22:43 ` Ondrej Zary
-1 siblings, 0 replies; 133+ messages in thread
From: Ondrej Zary @ 2021-02-18 22:43 UTC (permalink / raw)
To: Pavel Machek
Cc: Greg Kroah-Hartman, Jari Ruusu, Willy Tarreau, Scott Branden,
Linux ARM, LKML, BCM Kernel Feedback
On Thursday 18 February 2021 21:55:34 Pavel Machek wrote:
> Hi!
>
> > > For me
> > > only way to get properly working WiFi on my laptop computer is to
> > > compile that Intel out-of-tree version. Sad, but true.
> >
> > Why use 4.19.y on a laptop in the firstplace? That feels very wrong and
> > is not the recommended thing to use the LTS kernels for.
>
> Well, that's actually what distributions are doing, for example Debian
> 10.8 is on 4.19...
There's 5.10 in buster-backports. That's probably the easiest way to get support for new HW.
> I expect -stable is what most users are running on their notebooks.
>
> Best regards,
> Pavel
--
Ondrej Zary
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 22:43 ` Ondrej Zary
0 siblings, 0 replies; 133+ messages in thread
From: Ondrej Zary @ 2021-02-18 22:43 UTC (permalink / raw)
To: Pavel Machek
Cc: Jari Ruusu, Scott Branden, Greg Kroah-Hartman, LKML,
BCM Kernel Feedback, Willy Tarreau, Linux ARM
On Thursday 18 February 2021 21:55:34 Pavel Machek wrote:
> Hi!
>
> > > For me
> > > only way to get properly working WiFi on my laptop computer is to
> > > compile that Intel out-of-tree version. Sad, but true.
> >
> > Why use 4.19.y on a laptop in the firstplace? That feels very wrong and
> > is not the recommended thing to use the LTS kernels for.
>
> Well, that's actually what distributions are doing, for example Debian
> 10.8 is on 4.19...
There's 5.10 in buster-backports. That's probably the easiest way to get support for new HW.
> I expect -stable is what most users are running on their notebooks.
>
> Best regards,
> Pavel
--
Ondrej Zary
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 22:43 ` Ondrej Zary
@ 2021-02-19 8:00 ` Pavel Machek
-1 siblings, 0 replies; 133+ messages in thread
From: Pavel Machek @ 2021-02-19 8:00 UTC (permalink / raw)
To: Ondrej Zary
Cc: Greg Kroah-Hartman, Jari Ruusu, Willy Tarreau, Scott Branden,
Linux ARM, LKML, BCM Kernel Feedback
[-- Attachment #1: Type: text/plain, Size: 806 bytes --]
Hi!
> > > > For me
> > > > only way to get properly working WiFi on my laptop computer is to
> > > > compile that Intel out-of-tree version. Sad, but true.
> > >
> > > Why use 4.19.y on a laptop in the firstplace? That feels very wrong and
> > > is not the recommended thing to use the LTS kernels for.
> >
> > Well, that's actually what distributions are doing, for example Debian
> > 10.8 is on 4.19...
>
> There's 5.10 in buster-backports. That's probably the easiest way to get support for new HW.
>
I can compile my own kernel, too. But if you go up the thread, it is
about iwlwifi becoming broken in 4.19, and Greg saying it is wrong
to put -stable on laptop. And -stable on laptop is norm, not the
exception.
Pavel
--
http://www.livejournal.com/~pavelmachek
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-19 8:00 ` Pavel Machek
0 siblings, 0 replies; 133+ messages in thread
From: Pavel Machek @ 2021-02-19 8:00 UTC (permalink / raw)
To: Ondrej Zary
Cc: Jari Ruusu, Scott Branden, Greg Kroah-Hartman, LKML,
BCM Kernel Feedback, Willy Tarreau, Linux ARM
[-- Attachment #1.1: Type: text/plain, Size: 806 bytes --]
Hi!
> > > > For me
> > > > only way to get properly working WiFi on my laptop computer is to
> > > > compile that Intel out-of-tree version. Sad, but true.
> > >
> > > Why use 4.19.y on a laptop in the firstplace? That feels very wrong and
> > > is not the recommended thing to use the LTS kernels for.
> >
> > Well, that's actually what distributions are doing, for example Debian
> > 10.8 is on 4.19...
>
> There's 5.10 in buster-backports. That's probably the easiest way to get support for new HW.
>
I can compile my own kernel, too. But if you go up the thread, it is
about iwlwifi becoming broken in 4.19, and Greg saying it is wrong
to put -stable on laptop. And -stable on laptop is norm, not the
exception.
Pavel
--
http://www.livejournal.com/~pavelmachek
[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]
[-- Attachment #2: Type: text/plain, Size: 176 bytes --]
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-19 8:00 ` Pavel Machek
@ 2021-02-19 8:30 ` Greg Kroah-Hartman
-1 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-19 8:30 UTC (permalink / raw)
To: Pavel Machek
Cc: Ondrej Zary, Jari Ruusu, Willy Tarreau, Scott Branden, Linux ARM,
LKML, BCM Kernel Feedback
On Fri, Feb 19, 2021 at 09:00:27AM +0100, Pavel Machek wrote:
> Hi!
>
> > > > > For me
> > > > > only way to get properly working WiFi on my laptop computer is to
> > > > > compile that Intel out-of-tree version. Sad, but true.
> > > >
> > > > Why use 4.19.y on a laptop in the firstplace? That feels very wrong and
> > > > is not the recommended thing to use the LTS kernels for.
> > >
> > > Well, that's actually what distributions are doing, for example Debian
> > > 10.8 is on 4.19...
> >
> > There's 5.10 in buster-backports. That's probably the easiest way to get support for new HW.
> >
>
> I can compile my own kernel, too. But if you go up the thread, it is
> about iwlwifi becoming broken in 4.19, and Greg saying it is wrong
> to put -stable on laptop. And -stable on laptop is norm, not the
> exception.
If distros want to "camp out" on an old kernel version, that's fine, as
I describe in:
http://www.kroah.com/log/blog/2018/08/24/what-stable-kernel-should-i-use/
many years ago.
But for someone using a device where they want to use "new" hardware,
that was made _after_ the kernel version was released, that's just
someone who needs to pick a better distro :)
thanks,
greg k-h
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-19 8:30 ` Greg Kroah-Hartman
0 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-19 8:30 UTC (permalink / raw)
To: Pavel Machek
Cc: Jari Ruusu, Scott Branden, Ondrej Zary, LKML,
BCM Kernel Feedback, Willy Tarreau, Linux ARM
On Fri, Feb 19, 2021 at 09:00:27AM +0100, Pavel Machek wrote:
> Hi!
>
> > > > > For me
> > > > > only way to get properly working WiFi on my laptop computer is to
> > > > > compile that Intel out-of-tree version. Sad, but true.
> > > >
> > > > Why use 4.19.y on a laptop in the firstplace? That feels very wrong and
> > > > is not the recommended thing to use the LTS kernels for.
> > >
> > > Well, that's actually what distributions are doing, for example Debian
> > > 10.8 is on 4.19...
> >
> > There's 5.10 in buster-backports. That's probably the easiest way to get support for new HW.
> >
>
> I can compile my own kernel, too. But if you go up the thread, it is
> about iwlwifi becoming broken in 4.19, and Greg saying it is wrong
> to put -stable on laptop. And -stable on laptop is norm, not the
> exception.
If distros want to "camp out" on an old kernel version, that's fine, as
I describe in:
http://www.kroah.com/log/blog/2018/08/24/what-stable-kernel-should-i-use/
many years ago.
But for someone using a device where they want to use "new" hardware,
that was made _after_ the kernel version was released, that's just
someone who needs to pick a better distro :)
thanks,
greg k-h
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 14:15 ` Jari Ruusu
@ 2021-02-18 14:33 ` Willy Tarreau
-1 siblings, 0 replies; 133+ messages in thread
From: Willy Tarreau @ 2021-02-18 14:33 UTC (permalink / raw)
To: Jari Ruusu
Cc: Greg Kroah-Hartman, Scott Branden, Linux ARM, LKML, BCM Kernel Feedback
On Thu, Feb 18, 2021 at 04:15:11PM +0200, Jari Ruusu wrote:
> Willy Tarreau wrote:
> > The only set of fixes that can be trusted are the "official" stable
> > kernels, because they are the only ones that are approved by the patches
> > authors themselves. Adding more stuff on top of stable kernels is fine
> > (and done at your own risk), but randomly dropping stuff from stable
> > kernels just because you don't think you need that is totally non-sense
> > and must not be done anymore!
>
> This may be little bit off-topic... but stable kernel.org kernels
> can also bit-rot badly because of "selective" backporting... as in
> anything that does not apply cleanly gets dropped regardless of
> how critical they are.
Sure it will. And the huge difference is that it usually gets quickly
spotted and fixed. For sensitive servers I tend to apply the principle
of not necessarily updating to the latest stable kernel but one or two
versions before it which nobody complained about. And I'm pretty fine
with skipping a significant number of updates (we all do that anyway).
> I will give you one example: Intel WiFi (iwlwifi) on 4.19.y
> kernel.org stable kernels is currently missing many critical
> locking fixes. As a result, that in-tree iwlwifi driver causes
> erratic behavior to random unrelated processes, and has been doing
> so for many months now. My not-so-politically correct opinion is
> that in-tree iwlwifi is completely FUBAR unless someone steps up
> to do professional quality backport of those locking fixes from
> upstream out-of-tree Intel version [1] [2] of the driver.
I see, and it happens with plenty of other drivers or subsystems. Is
it in any way the stable branch's or stable maintainer's fault if
someone doesn't correctly do the backporting job on their driver ? No.
Is it expected that a driver works perfectly from its inclusion ? No.
Is it expected that a driver can always be fixed without a significant
rework that risks more breakage than fixes ? No. Some design limitations
or errors can require so many changes that they're unfixable in place.
I even had to *document* security issues in 2.4 because fixing them was
riskier than keeping them. This happens in any piece of software.
It's always been the case that some older kernels work less well than
some newer ones due to limited features, partially wrong drivers etc,
and getting better drivers is a valid reason for upgrading to a more
recent one. However the older driver ought to continue to be maintained
in a working state for those for whom it works fine.
> For me
> only way to get properly working WiFi on my laptop computer is to
> compile that Intel out-of-tree version. Sad, but true.
That's perfectly fine from my point of view. I've been doing the same
for certain driver (e.g. e100 vs eepro100 15 years ago) and have been
pleased to be able to stop using those out-of-tree versions. This is
also in order to make this possible for those who need to do it that
LTS kernels provide a lot of value: such out-of-tree drivers tend to
take some time to resynchronize with latest updates, and once they're
updated, you can use your machine for quite some time with them.
Obviously if somemone is able to figure the required fixes for the locking
bugs you mentioned above and to submit patches for stable branches, I'm
sure Greg will appreciate! But maybe that's not fixable there and you need
to upgrade. Usually you pick an LTS kernel for a specific hardware. If it
works that's great. But you cannot expect hardware to suddenly start to
work in the middle of a stable kernel. Sometimes it happens (PCI IDs) but
that's basically all and that's not their purpose.
Willy
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 14:33 ` Willy Tarreau
0 siblings, 0 replies; 133+ messages in thread
From: Willy Tarreau @ 2021-02-18 14:33 UTC (permalink / raw)
To: Jari Ruusu
Cc: Greg Kroah-Hartman, BCM Kernel Feedback, LKML, Linux ARM, Scott Branden
On Thu, Feb 18, 2021 at 04:15:11PM +0200, Jari Ruusu wrote:
> Willy Tarreau wrote:
> > The only set of fixes that can be trusted are the "official" stable
> > kernels, because they are the only ones that are approved by the patches
> > authors themselves. Adding more stuff on top of stable kernels is fine
> > (and done at your own risk), but randomly dropping stuff from stable
> > kernels just because you don't think you need that is totally non-sense
> > and must not be done anymore!
>
> This may be little bit off-topic... but stable kernel.org kernels
> can also bit-rot badly because of "selective" backporting... as in
> anything that does not apply cleanly gets dropped regardless of
> how critical they are.
Sure it will. And the huge difference is that it usually gets quickly
spotted and fixed. For sensitive servers I tend to apply the principle
of not necessarily updating to the latest stable kernel but one or two
versions before it which nobody complained about. And I'm pretty fine
with skipping a significant number of updates (we all do that anyway).
> I will give you one example: Intel WiFi (iwlwifi) on 4.19.y
> kernel.org stable kernels is currently missing many critical
> locking fixes. As a result, that in-tree iwlwifi driver causes
> erratic behavior to random unrelated processes, and has been doing
> so for many months now. My not-so-politically correct opinion is
> that in-tree iwlwifi is completely FUBAR unless someone steps up
> to do professional quality backport of those locking fixes from
> upstream out-of-tree Intel version [1] [2] of the driver.
I see, and it happens with plenty of other drivers or subsystems. Is
it in any way the stable branch's or stable maintainer's fault if
someone doesn't correctly do the backporting job on their driver ? No.
Is it expected that a driver works perfectly from its inclusion ? No.
Is it expected that a driver can always be fixed without a significant
rework that risks more breakage than fixes ? No. Some design limitations
or errors can require so many changes that they're unfixable in place.
I even had to *document* security issues in 2.4 because fixing them was
riskier than keeping them. This happens in any piece of software.
It's always been the case that some older kernels work less well than
some newer ones due to limited features, partially wrong drivers etc,
and getting better drivers is a valid reason for upgrading to a more
recent one. However the older driver ought to continue to be maintained
in a working state for those for whom it works fine.
> For me
> only way to get properly working WiFi on my laptop computer is to
> compile that Intel out-of-tree version. Sad, but true.
That's perfectly fine from my point of view. I've been doing the same
for certain driver (e.g. e100 vs eepro100 15 years ago) and have been
pleased to be able to stop using those out-of-tree versions. This is
also in order to make this possible for those who need to do it that
LTS kernels provide a lot of value: such out-of-tree drivers tend to
take some time to resynchronize with latest updates, and once they're
updated, you can use your machine for quite some time with them.
Obviously if somemone is able to figure the required fixes for the locking
bugs you mentioned above and to submit patches for stable branches, I'm
sure Greg will appreciate! But maybe that's not fixable there and you need
to upgrade. Usually you pick an LTS kernel for a specific hardware. If it
works that's great. But you cannot expect hardware to suddenly start to
work in the middle of a stable kernel. Sometimes it happens (PCI IDs) but
that's basically all and that's not their purpose.
Willy
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 14:33 ` Willy Tarreau
@ 2021-02-18 17:19 ` Jari Ruusu
-1 siblings, 0 replies; 133+ messages in thread
From: Jari Ruusu @ 2021-02-18 17:19 UTC (permalink / raw)
To: Willy Tarreau
Cc: Jari Ruusu, Greg Kroah-Hartman, Scott Branden, Linux ARM, LKML,
BCM Kernel Feedback
On Thursday, February 18, 2021 4:33 PM, Willy Tarreau <w@1wt.eu> wrote:
> Usually you pick an LTS kernel for a specific hardware. If it
> works that's great. But you cannot expect hardware to suddenly start to
> work in the middle of a stable kernel. Sometimes it happens (PCI IDs) but
> that's basically all and that's not their purpose.
It was the other way around. Fine working in-tree driver got
broken by backported "fixes". I did mention bit-rot.
In-tree iwlwifi worked half-ok on early 4.9.y stable. If
connection somehow de-autheticated (out of radio range or
whatever) it crashed the kernel spectacularly. Eventually that was
fixed and in-tree iwlwifi worked fine on 4.9.y and 4.14.y stable
kernels. On second half of year 2020 (don't remember exactly when)
iwlwifi started causing erratic behavior when some random process
terminated, as if some exit processing left some resources
un-freed or something weird like that. Upgraded to 4.19.y kernels
in hope to fix the issue. Nope, same problems continued there as
well. Replacing in-tree iwlwifi with out-of-tree upstream Intel
version solved the problem for me.
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 17:19 ` Jari Ruusu
0 siblings, 0 replies; 133+ messages in thread
From: Jari Ruusu @ 2021-02-18 17:19 UTC (permalink / raw)
To: Willy Tarreau
Cc: Jari Ruusu, Scott Branden, Greg Kroah-Hartman, LKML,
BCM Kernel Feedback, Linux ARM
On Thursday, February 18, 2021 4:33 PM, Willy Tarreau <w@1wt.eu> wrote:
> Usually you pick an LTS kernel for a specific hardware. If it
> works that's great. But you cannot expect hardware to suddenly start to
> work in the middle of a stable kernel. Sometimes it happens (PCI IDs) but
> that's basically all and that's not their purpose.
It was the other way around. Fine working in-tree driver got
broken by backported "fixes". I did mention bit-rot.
In-tree iwlwifi worked half-ok on early 4.9.y stable. If
connection somehow de-autheticated (out of radio range or
whatever) it crashed the kernel spectacularly. Eventually that was
fixed and in-tree iwlwifi worked fine on 4.9.y and 4.14.y stable
kernels. On second half of year 2020 (don't remember exactly when)
iwlwifi started causing erratic behavior when some random process
terminated, as if some exit processing left some resources
un-freed or something weird like that. Upgraded to 4.19.y kernels
in hope to fix the issue. Nope, same problems continued there as
well. Replacing in-tree iwlwifi with out-of-tree upstream Intel
version solved the problem for me.
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 17:19 ` Jari Ruusu
@ 2021-02-18 17:22 ` Russell King - ARM Linux admin
-1 siblings, 0 replies; 133+ messages in thread
From: Russell King - ARM Linux admin @ 2021-02-18 17:22 UTC (permalink / raw)
To: Jari Ruusu
Cc: Willy Tarreau, Jari Ruusu, Scott Branden, Greg Kroah-Hartman,
LKML, BCM Kernel Feedback, Linux ARM
On Thu, Feb 18, 2021 at 05:19:54PM +0000, Jari Ruusu wrote:
> In-tree iwlwifi worked half-ok on early 4.9.y stable. If
> connection somehow de-autheticated (out of radio range or
> whatever) it crashed the kernel spectacularly. Eventually that was
> fixed and in-tree iwlwifi worked fine on 4.9.y and 4.14.y stable
> kernels. On second half of year 2020 (don't remember exactly when)
> iwlwifi started causing erratic behavior when some random process
> terminated, as if some exit processing left some resources
> un-freed or something weird like that.
So you bisected the stable kernel, and reported the problem so the
problem commit could be dealt with?
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 17:22 ` Russell King - ARM Linux admin
0 siblings, 0 replies; 133+ messages in thread
From: Russell King - ARM Linux admin @ 2021-02-18 17:22 UTC (permalink / raw)
To: Jari Ruusu
Cc: Jari Ruusu, Scott Branden, Greg Kroah-Hartman, LKML,
BCM Kernel Feedback, Willy Tarreau, Linux ARM
On Thu, Feb 18, 2021 at 05:19:54PM +0000, Jari Ruusu wrote:
> In-tree iwlwifi worked half-ok on early 4.9.y stable. If
> connection somehow de-autheticated (out of radio range or
> whatever) it crashed the kernel spectacularly. Eventually that was
> fixed and in-tree iwlwifi worked fine on 4.9.y and 4.14.y stable
> kernels. On second half of year 2020 (don't remember exactly when)
> iwlwifi started causing erratic behavior when some random process
> terminated, as if some exit processing left some resources
> un-freed or something weird like that.
So you bisected the stable kernel, and reported the problem so the
problem commit could be dealt with?
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 17:19 ` Jari Ruusu
@ 2021-02-18 17:44 ` Greg Kroah-Hartman
-1 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-18 17:44 UTC (permalink / raw)
To: Jari Ruusu
Cc: Willy Tarreau, Jari Ruusu, Scott Branden, Linux ARM, LKML,
BCM Kernel Feedback
On Thu, Feb 18, 2021 at 05:19:54PM +0000, Jari Ruusu wrote:
> On Thursday, February 18, 2021 4:33 PM, Willy Tarreau <w@1wt.eu> wrote:
> > Usually you pick an LTS kernel for a specific hardware. If it
> > works that's great. But you cannot expect hardware to suddenly start to
> > work in the middle of a stable kernel. Sometimes it happens (PCI IDs) but
> > that's basically all and that's not their purpose.
>
> It was the other way around. Fine working in-tree driver got
> broken by backported "fixes". I did mention bit-rot.
It did? Please let us stable maintainers know about, we will always
gladly revert problems patches. What commits caused the problem?
> In-tree iwlwifi worked half-ok on early 4.9.y stable. If
> connection somehow de-autheticated (out of radio range or
> whatever) it crashed the kernel spectacularly. Eventually that was
> fixed and in-tree iwlwifi worked fine on 4.9.y and 4.14.y stable
> kernels. On second half of year 2020 (don't remember exactly when)
> iwlwifi started causing erratic behavior when some random process
> terminated, as if some exit processing left some resources
> un-freed or something weird like that. Upgraded to 4.19.y kernels
> in hope to fix the issue. Nope, same problems continued there as
> well. Replacing in-tree iwlwifi with out-of-tree upstream Intel
> version solved the problem for me.
So something in the 4.9.y and 4.14.y stable kernels caused a regression,
can you please do 'git bisect' to let us know what broke?
And if 4.19.0 was always broken, why didn't you report that as well?
How about 5.11, have you tried that? If not, please do so and report it
to the developers, otherwise how can it ever get fixed?
thanks,
greg k-h
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 17:44 ` Greg Kroah-Hartman
0 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-18 17:44 UTC (permalink / raw)
To: Jari Ruusu
Cc: Jari Ruusu, Scott Branden, LKML, BCM Kernel Feedback,
Willy Tarreau, Linux ARM
On Thu, Feb 18, 2021 at 05:19:54PM +0000, Jari Ruusu wrote:
> On Thursday, February 18, 2021 4:33 PM, Willy Tarreau <w@1wt.eu> wrote:
> > Usually you pick an LTS kernel for a specific hardware. If it
> > works that's great. But you cannot expect hardware to suddenly start to
> > work in the middle of a stable kernel. Sometimes it happens (PCI IDs) but
> > that's basically all and that's not their purpose.
>
> It was the other way around. Fine working in-tree driver got
> broken by backported "fixes". I did mention bit-rot.
It did? Please let us stable maintainers know about, we will always
gladly revert problems patches. What commits caused the problem?
> In-tree iwlwifi worked half-ok on early 4.9.y stable. If
> connection somehow de-autheticated (out of radio range or
> whatever) it crashed the kernel spectacularly. Eventually that was
> fixed and in-tree iwlwifi worked fine on 4.9.y and 4.14.y stable
> kernels. On second half of year 2020 (don't remember exactly when)
> iwlwifi started causing erratic behavior when some random process
> terminated, as if some exit processing left some resources
> un-freed or something weird like that. Upgraded to 4.19.y kernels
> in hope to fix the issue. Nope, same problems continued there as
> well. Replacing in-tree iwlwifi with out-of-tree upstream Intel
> version solved the problem for me.
So something in the 4.9.y and 4.14.y stable kernels caused a regression,
can you please do 'git bisect' to let us know what broke?
And if 4.19.0 was always broken, why didn't you report that as well?
How about 5.11, have you tried that? If not, please do so and report it
to the developers, otherwise how can it ever get fixed?
thanks,
greg k-h
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 17:44 ` Greg Kroah-Hartman
@ 2021-02-19 7:10 ` Jari Ruusu
-1 siblings, 0 replies; 133+ messages in thread
From: Jari Ruusu @ 2021-02-19 7:10 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Willy Tarreau, Jari Ruusu, Scott Branden, Linux ARM, LKML,
BCM Kernel Feedback
On Thursday, February 18, 2021 7:44 PM, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> > It was the other way around. Fine working in-tree driver got
> > broken by backported "fixes". I did mention bit-rot.
>
> It did? Please let us stable maintainers know about, we will always
> gladly revert problems patches. What commits caused the problem?
I don't have a list of commits for you. It took me long time to
figure out that it was iwlwifi that was causing those problems.
In-tree iwlwifi on 4.19.y kernels needs professional quality
locking audit and backporting of necessary fixes from upstream
Intel out-of-tree version.
> So something in the 4.9.y and 4.14.y stable kernels caused a regression,
> can you please do 'git bisect' to let us know what broke?
My ability to do WiFi tests on that laptop computer are limited
for now. Earlier that laptop's connectivity to world was through
mobile WiFi router. That mobile WiFi router no longer has a
SIM-card, so no connectivity through that anymore. That laptop's
connectivity to world is now through wired ethernet to fiber.
It was actually this switch to ethernet/fiber that made me realize
the brokenness on in-tree iwlwifi on 4.19.y kernels. When in-tree
WiFi was not used, those problems never triggered. Switched back
to in-tree WiFi, and problems came back. Switched to out-of-tree
Intel version of iwlwifi, problems went away again. Then I looked
at the fixes in out-of-tree Intel version of iwlwifi that were
missing in in-tree version, and I understood why that was so.
Those stability issues on in-tree iwlwifi on 4.19.y kernels are
difficult to trigger. Sometimes it may take days to trigger it
once. Sometimes I was unlucky enough to trigger them more than
once a day. I say that two weeks of operation without issues are
needed to pronounce those issues gone.
Currently, special arrangements are needed for me to test WiFi at
all on that laptop computer, and those arrangements are something
I am not willing to commit for multi-week testing run. Sorry.
> And if 4.19.0 was always broken, why didn't you report that as well?
I didn't test early 4.19.y kernels.
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-19 7:10 ` Jari Ruusu
0 siblings, 0 replies; 133+ messages in thread
From: Jari Ruusu @ 2021-02-19 7:10 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Jari Ruusu, Scott Branden, LKML, BCM Kernel Feedback,
Willy Tarreau, Linux ARM
On Thursday, February 18, 2021 7:44 PM, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> > It was the other way around. Fine working in-tree driver got
> > broken by backported "fixes". I did mention bit-rot.
>
> It did? Please let us stable maintainers know about, we will always
> gladly revert problems patches. What commits caused the problem?
I don't have a list of commits for you. It took me long time to
figure out that it was iwlwifi that was causing those problems.
In-tree iwlwifi on 4.19.y kernels needs professional quality
locking audit and backporting of necessary fixes from upstream
Intel out-of-tree version.
> So something in the 4.9.y and 4.14.y stable kernels caused a regression,
> can you please do 'git bisect' to let us know what broke?
My ability to do WiFi tests on that laptop computer are limited
for now. Earlier that laptop's connectivity to world was through
mobile WiFi router. That mobile WiFi router no longer has a
SIM-card, so no connectivity through that anymore. That laptop's
connectivity to world is now through wired ethernet to fiber.
It was actually this switch to ethernet/fiber that made me realize
the brokenness on in-tree iwlwifi on 4.19.y kernels. When in-tree
WiFi was not used, those problems never triggered. Switched back
to in-tree WiFi, and problems came back. Switched to out-of-tree
Intel version of iwlwifi, problems went away again. Then I looked
at the fixes in out-of-tree Intel version of iwlwifi that were
missing in in-tree version, and I understood why that was so.
Those stability issues on in-tree iwlwifi on 4.19.y kernels are
difficult to trigger. Sometimes it may take days to trigger it
once. Sometimes I was unlucky enough to trigger them more than
once a day. I say that two weeks of operation without issues are
needed to pronounce those issues gone.
Currently, special arrangements are needed for me to test WiFi at
all on that laptop computer, and those arrangements are something
I am not willing to commit for multi-week testing run. Sorry.
> And if 4.19.0 was always broken, why didn't you report that as well?
I didn't test early 4.19.y kernels.
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-19 7:10 ` Jari Ruusu
@ 2021-02-19 8:22 ` Greg Kroah-Hartman
-1 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-19 8:22 UTC (permalink / raw)
To: Jari Ruusu
Cc: Willy Tarreau, Jari Ruusu, Scott Branden, Linux ARM, LKML,
BCM Kernel Feedback
On Fri, Feb 19, 2021 at 07:10:35AM +0000, Jari Ruusu wrote:
> On Thursday, February 18, 2021 7:44 PM, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> > > It was the other way around. Fine working in-tree driver got
> > > broken by backported "fixes". I did mention bit-rot.
> >
> > It did? Please let us stable maintainers know about, we will always
> > gladly revert problems patches. What commits caused the problem?
>
> I don't have a list of commits for you. It took me long time to
> figure out that it was iwlwifi that was causing those problems.
>
> In-tree iwlwifi on 4.19.y kernels needs professional quality
> locking audit and backporting of necessary fixes from upstream
> Intel out-of-tree version.
That's not the goal of stable kernel releases/trees. If the driver
version that is in 4.19.y does not work for you on release 4.19.0, odds
of that "changing" in later stable releases is slim to none.
Especially without any specific bug reports or emails to the developers
to tell them what is broken and not working for you.
If however, you wish to stick with an out-of-tree driver, wonderful,
that's your choice. But don't claim that somehow the stable kernel
process is broken because of that, as it has nothing to do with this
type of thing.
Best of luck!
greg k-h
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-19 8:22 ` Greg Kroah-Hartman
0 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-19 8:22 UTC (permalink / raw)
To: Jari Ruusu
Cc: Jari Ruusu, Scott Branden, LKML, BCM Kernel Feedback,
Willy Tarreau, Linux ARM
On Fri, Feb 19, 2021 at 07:10:35AM +0000, Jari Ruusu wrote:
> On Thursday, February 18, 2021 7:44 PM, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> > > It was the other way around. Fine working in-tree driver got
> > > broken by backported "fixes". I did mention bit-rot.
> >
> > It did? Please let us stable maintainers know about, we will always
> > gladly revert problems patches. What commits caused the problem?
>
> I don't have a list of commits for you. It took me long time to
> figure out that it was iwlwifi that was causing those problems.
>
> In-tree iwlwifi on 4.19.y kernels needs professional quality
> locking audit and backporting of necessary fixes from upstream
> Intel out-of-tree version.
That's not the goal of stable kernel releases/trees. If the driver
version that is in 4.19.y does not work for you on release 4.19.0, odds
of that "changing" in later stable releases is slim to none.
Especially without any specific bug reports or emails to the developers
to tell them what is broken and not working for you.
If however, you wish to stick with an out-of-tree driver, wonderful,
that's your choice. But don't claim that somehow the stable kernel
process is broken because of that, as it has nothing to do with this
type of thing.
Best of luck!
greg k-h
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-19 8:22 ` Greg Kroah-Hartman
@ 2021-02-19 10:31 ` Jari Ruusu
-1 siblings, 0 replies; 133+ messages in thread
From: Jari Ruusu @ 2021-02-19 10:31 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Willy Tarreau, Jari Ruusu, Scott Branden, Linux ARM, LKML,
BCM Kernel Feedback
On Friday, February 19, 2021 10:22 AM, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> That's not the goal of stable kernel releases/trees. If the driver
> version that is in 4.19.y does not work for you on release 4.19.0, odds
> of that "changing" in later stable releases is slim to none.
But in-tree iwlwifi worked fine on 4.9.y and 4.14.y , and then got broken
in later 4.14.y versions. I tried later versions of 4.19.y in an attempt to
fix the breakage. Basically you are are saying that I should have NOT
attempted to upgrage 4.9.y -> 4.14.y -> 4.19.y
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-19 10:31 ` Jari Ruusu
0 siblings, 0 replies; 133+ messages in thread
From: Jari Ruusu @ 2021-02-19 10:31 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Jari Ruusu, Scott Branden, LKML, BCM Kernel Feedback,
Willy Tarreau, Linux ARM
On Friday, February 19, 2021 10:22 AM, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> That's not the goal of stable kernel releases/trees. If the driver
> version that is in 4.19.y does not work for you on release 4.19.0, odds
> of that "changing" in later stable releases is slim to none.
But in-tree iwlwifi worked fine on 4.9.y and 4.14.y , and then got broken
in later 4.14.y versions. I tried later versions of 4.19.y in an attempt to
fix the breakage. Basically you are are saying that I should have NOT
attempted to upgrage 4.9.y -> 4.14.y -> 4.19.y
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-19 10:31 ` Jari Ruusu
@ 2021-02-19 10:37 ` Greg Kroah-Hartman
-1 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-19 10:37 UTC (permalink / raw)
To: Jari Ruusu
Cc: Willy Tarreau, Jari Ruusu, Scott Branden, Linux ARM, LKML,
BCM Kernel Feedback
On Fri, Feb 19, 2021 at 10:31:49AM +0000, Jari Ruusu wrote:
> On Friday, February 19, 2021 10:22 AM, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> > That's not the goal of stable kernel releases/trees. If the driver
> > version that is in 4.19.y does not work for you on release 4.19.0, odds
> > of that "changing" in later stable releases is slim to none.
>
> But in-tree iwlwifi worked fine on 4.9.y and 4.14.y , and then got broken
> in later 4.14.y versions.
Did you report that breakage to us and the developers of the driver?
Sounds like a regression that people would love to hear about and get
fixed.
> I tried later versions of 4.19.y in an attempt to
> fix the breakage. Basically you are are saying that I should have NOT
> attempted to upgrage 4.9.y -> 4.14.y -> 4.19.y
Without anyone reporting a problem, how can anyone know to fix it?
thanks,
greg k-h
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-19 10:37 ` Greg Kroah-Hartman
0 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-19 10:37 UTC (permalink / raw)
To: Jari Ruusu
Cc: Jari Ruusu, Scott Branden, LKML, BCM Kernel Feedback,
Willy Tarreau, Linux ARM
On Fri, Feb 19, 2021 at 10:31:49AM +0000, Jari Ruusu wrote:
> On Friday, February 19, 2021 10:22 AM, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> > That's not the goal of stable kernel releases/trees. If the driver
> > version that is in 4.19.y does not work for you on release 4.19.0, odds
> > of that "changing" in later stable releases is slim to none.
>
> But in-tree iwlwifi worked fine on 4.9.y and 4.14.y , and then got broken
> in later 4.14.y versions.
Did you report that breakage to us and the developers of the driver?
Sounds like a regression that people would love to hear about and get
fixed.
> I tried later versions of 4.19.y in an attempt to
> fix the breakage. Basically you are are saying that I should have NOT
> attempted to upgrage 4.9.y -> 4.14.y -> 4.19.y
Without anyone reporting a problem, how can anyone know to fix it?
thanks,
greg k-h
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-19 10:37 ` Greg Kroah-Hartman
@ 2021-02-19 10:57 ` Jari Ruusu
-1 siblings, 0 replies; 133+ messages in thread
From: Jari Ruusu @ 2021-02-19 10:57 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Willy Tarreau, Jari Ruusu, Scott Branden, Linux ARM, LKML,
BCM Kernel Feedback
On Friday, February 19, 2021 12:37 PM, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> Did you report that breakage to us and the developers of the driver?
> Sounds like a regression that people would love to hear about and get
> fixed.
At that time I didn't know where the problem was, so I did not report it.
I only recently connected-the-dots that problem is in-tree iwlwifi, so I
am reporting it now.
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-19 10:57 ` Jari Ruusu
0 siblings, 0 replies; 133+ messages in thread
From: Jari Ruusu @ 2021-02-19 10:57 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Jari Ruusu, Scott Branden, LKML, BCM Kernel Feedback,
Willy Tarreau, Linux ARM
On Friday, February 19, 2021 12:37 PM, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> Did you report that breakage to us and the developers of the driver?
> Sounds like a regression that people would love to hear about and get
> fixed.
At that time I didn't know where the problem was, so I did not report it.
I only recently connected-the-dots that problem is in-tree iwlwifi, so I
am reporting it now.
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-19 10:57 ` Jari Ruusu
@ 2021-02-19 11:16 ` Greg Kroah-Hartman
-1 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-19 11:16 UTC (permalink / raw)
To: Jari Ruusu
Cc: Willy Tarreau, Jari Ruusu, Scott Branden, Linux ARM, LKML,
BCM Kernel Feedback
On Fri, Feb 19, 2021 at 10:57:30AM +0000, Jari Ruusu wrote:
> On Friday, February 19, 2021 12:37 PM, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> > Did you report that breakage to us and the developers of the driver?
> > Sounds like a regression that people would love to hear about and get
> > fixed.
>
> At that time I didn't know where the problem was, so I did not report it.
> I only recently connected-the-dots that problem is in-tree iwlwifi, so I
> am reporting it now.
Great! Can you run 'git bisect' on the 4.14.y stable tree to find the
offending change?
thanks,
greg k-h
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-19 11:16 ` Greg Kroah-Hartman
0 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-19 11:16 UTC (permalink / raw)
To: Jari Ruusu
Cc: Jari Ruusu, Scott Branden, LKML, BCM Kernel Feedback,
Willy Tarreau, Linux ARM
On Fri, Feb 19, 2021 at 10:57:30AM +0000, Jari Ruusu wrote:
> On Friday, February 19, 2021 12:37 PM, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> > Did you report that breakage to us and the developers of the driver?
> > Sounds like a regression that people would love to hear about and get
> > fixed.
>
> At that time I didn't know where the problem was, so I did not report it.
> I only recently connected-the-dots that problem is in-tree iwlwifi, so I
> am reporting it now.
Great! Can you run 'git bisect' on the 4.14.y stable tree to find the
offending change?
thanks,
greg k-h
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-19 11:16 ` Greg Kroah-Hartman
@ 2021-02-19 15:23 ` Jari Ruusu
-1 siblings, 0 replies; 133+ messages in thread
From: Jari Ruusu @ 2021-02-19 15:23 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Willy Tarreau, Jari Ruusu, Scott Branden, Linux ARM, LKML,
BCM Kernel Feedback
On Friday, February 19, 2021 1:16 PM, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> Great! Can you run 'git bisect' on the 4.14.y stable tree to find the
> offending change?
As I tried to explain earlier, that laptop's connection to world is no
longer iwlwifi based but ethernet connection to fibre. I can't do much
bisecting when iwlwifi is not being used at all. Sorry. My contribution
here is trying to point you guys to right direction.
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-19 15:23 ` Jari Ruusu
0 siblings, 0 replies; 133+ messages in thread
From: Jari Ruusu @ 2021-02-19 15:23 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Jari Ruusu, Scott Branden, LKML, BCM Kernel Feedback,
Willy Tarreau, Linux ARM
On Friday, February 19, 2021 1:16 PM, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> Great! Can you run 'git bisect' on the 4.14.y stable tree to find the
> offending change?
As I tried to explain earlier, that laptop's connection to world is no
longer iwlwifi based but ethernet connection to fibre. I can't do much
bisecting when iwlwifi is not being used at all. Sorry. My contribution
here is trying to point you guys to right direction.
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-19 15:23 ` Jari Ruusu
@ 2021-02-20 13:29 ` Jari Ruusu
-1 siblings, 0 replies; 133+ messages in thread
From: Jari Ruusu @ 2021-02-20 13:29 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Willy Tarreau, Jari Ruusu, Scott Branden, Linux ARM, LKML,
BCM Kernel Feedback
On Friday, February 19, 2021 5:23 PM, Jari Ruusu <jariruusu@protonmail.com> wrote:
> My contribution here is trying to point you guys to right direction.
I have been able to narrow the beginning of the problem to these kernels:
4.14.188 ... 4.14.202
Same "fix" that went info 4.14.y is also bugging 4.19.y kernels.
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-20 13:29 ` Jari Ruusu
0 siblings, 0 replies; 133+ messages in thread
From: Jari Ruusu @ 2021-02-20 13:29 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Jari Ruusu, Scott Branden, LKML, BCM Kernel Feedback,
Willy Tarreau, Linux ARM
On Friday, February 19, 2021 5:23 PM, Jari Ruusu <jariruusu@protonmail.com> wrote:
> My contribution here is trying to point you guys to right direction.
I have been able to narrow the beginning of the problem to these kernels:
4.14.188 ... 4.14.202
Same "fix" that went info 4.14.y is also bugging 4.19.y kernels.
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-20 13:29 ` Jari Ruusu
@ 2021-02-20 16:05 ` Greg Kroah-Hartman
-1 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-20 16:05 UTC (permalink / raw)
To: Jari Ruusu
Cc: Willy Tarreau, Jari Ruusu, Scott Branden, Linux ARM, LKML,
BCM Kernel Feedback
On Sat, Feb 20, 2021 at 01:29:21PM +0000, Jari Ruusu wrote:
> On Friday, February 19, 2021 5:23 PM, Jari Ruusu <jariruusu@protonmail.com> wrote:
> > My contribution here is trying to point you guys to right direction.
>
> I have been able to narrow the beginning of the problem to these kernels:
> 4.14.188 ... 4.14.202
>
> Same "fix" that went info 4.14.y is also bugging 4.19.y kernels.
Great, any chance you can narrow this down to the commit itself?
thanks,
greg k-h
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-20 16:05 ` Greg Kroah-Hartman
0 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-20 16:05 UTC (permalink / raw)
To: Jari Ruusu
Cc: Jari Ruusu, Scott Branden, LKML, BCM Kernel Feedback,
Willy Tarreau, Linux ARM
On Sat, Feb 20, 2021 at 01:29:21PM +0000, Jari Ruusu wrote:
> On Friday, February 19, 2021 5:23 PM, Jari Ruusu <jariruusu@protonmail.com> wrote:
> > My contribution here is trying to point you guys to right direction.
>
> I have been able to narrow the beginning of the problem to these kernels:
> 4.14.188 ... 4.14.202
>
> Same "fix" that went info 4.14.y is also bugging 4.19.y kernels.
Great, any chance you can narrow this down to the commit itself?
thanks,
greg k-h
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-20 16:05 ` Greg Kroah-Hartman
@ 2021-02-20 17:06 ` Willy Tarreau
-1 siblings, 0 replies; 133+ messages in thread
From: Willy Tarreau @ 2021-02-20 17:06 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Jari Ruusu, Jari Ruusu, Scott Branden, Linux ARM, LKML,
BCM Kernel Feedback
On Sat, Feb 20, 2021 at 05:05:14PM +0100, Greg Kroah-Hartman wrote:
> On Sat, Feb 20, 2021 at 01:29:21PM +0000, Jari Ruusu wrote:
> > On Friday, February 19, 2021 5:23 PM, Jari Ruusu <jariruusu@protonmail.com> wrote:
> > > My contribution here is trying to point you guys to right direction.
> >
> > I have been able to narrow the beginning of the problem to these kernels:
> > 4.14.188 ... 4.14.202
> >
> > Same "fix" that went info 4.14.y is also bugging 4.19.y kernels.
>
> Great, any chance you can narrow this down to the commit itself?
What is strnage is that there was no iwlwifi driver change in this
interval. Only iwlegacy (don't know if this one was used instead).
Note, my laptop uses iwlwifi as well. I've met random issues with it
a year ago when I started to use wifi, such as wifi randomly freezing
during audio meetings, and automatically fixing itself after 1 minute.
I also found that a down-up cycle on the interface would fix it. It
happened less than once a day so it was not easy to diagnose, and given
how crappy all wifi chips are, I naturally attributed this to the
hardware. But since I upgraded to 5.4 months ago, I don't remember having
met that issue anymore, so it was likely related to the driver. All I can
say is that 4.19.68 was exhibiting this issue. I can't say for older ones
because I didn't use wifi before.
Just my two cents,
Willy
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-20 17:06 ` Willy Tarreau
0 siblings, 0 replies; 133+ messages in thread
From: Willy Tarreau @ 2021-02-20 17:06 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Jari Ruusu, Scott Branden, Jari Ruusu, LKML, BCM Kernel Feedback,
Linux ARM
On Sat, Feb 20, 2021 at 05:05:14PM +0100, Greg Kroah-Hartman wrote:
> On Sat, Feb 20, 2021 at 01:29:21PM +0000, Jari Ruusu wrote:
> > On Friday, February 19, 2021 5:23 PM, Jari Ruusu <jariruusu@protonmail.com> wrote:
> > > My contribution here is trying to point you guys to right direction.
> >
> > I have been able to narrow the beginning of the problem to these kernels:
> > 4.14.188 ... 4.14.202
> >
> > Same "fix" that went info 4.14.y is also bugging 4.19.y kernels.
>
> Great, any chance you can narrow this down to the commit itself?
What is strnage is that there was no iwlwifi driver change in this
interval. Only iwlegacy (don't know if this one was used instead).
Note, my laptop uses iwlwifi as well. I've met random issues with it
a year ago when I started to use wifi, such as wifi randomly freezing
during audio meetings, and automatically fixing itself after 1 minute.
I also found that a down-up cycle on the interface would fix it. It
happened less than once a day so it was not easy to diagnose, and given
how crappy all wifi chips are, I naturally attributed this to the
hardware. But since I upgraded to 5.4 months ago, I don't remember having
met that issue anymore, so it was likely related to the driver. All I can
say is that 4.19.68 was exhibiting this issue. I can't say for older ones
because I didn't use wifi before.
Just my two cents,
Willy
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-20 16:05 ` Greg Kroah-Hartman
@ 2021-02-21 11:38 ` Jari Ruusu
-1 siblings, 0 replies; 133+ messages in thread
From: Jari Ruusu @ 2021-02-21 11:38 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Willy Tarreau, Jari Ruusu, Scott Branden, Linux ARM, LKML,
BCM Kernel Feedback
On Saturday, February 20, 2021 6:05 PM, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> On Sat, Feb 20, 2021 at 01:29:21PM +0000, Jari Ruusu wrote:
> > I have been able to narrow the beginning of the problem to these kernels:
> > 4.14.188 ... 4.14.202
> > Same "fix" that went info 4.14.y is also bugging 4.19.y kernels.
>
> Great, any chance you can narrow this down to the commit itself?
I am not able to test WiFi on that laptop computer anymore,
because that laptop now connects to world using wired connection.
It was that WiFI->wired connection change that led me to realize
the buggyness of in-tree iwlwifi in 4.19.y kernels.
I did that narrowing to specific kernel versions by digging my
archived backup files and their notes. No tests were run.
Problems started triggering in those kernels that I mentioned in
earlier email. That does not mean that the bugs were not already
there in kernels older that those. Maybe some change just widened
"window of opportunity" enough for me to see the issues.
In-tree iwlwifi in 4.19.y is missing locking fixes. No amount of
smooth-talking is going to change that.
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-21 11:38 ` Jari Ruusu
0 siblings, 0 replies; 133+ messages in thread
From: Jari Ruusu @ 2021-02-21 11:38 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Jari Ruusu, Scott Branden, LKML, BCM Kernel Feedback,
Willy Tarreau, Linux ARM
On Saturday, February 20, 2021 6:05 PM, Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote:
> On Sat, Feb 20, 2021 at 01:29:21PM +0000, Jari Ruusu wrote:
> > I have been able to narrow the beginning of the problem to these kernels:
> > 4.14.188 ... 4.14.202
> > Same "fix" that went info 4.14.y is also bugging 4.19.y kernels.
>
> Great, any chance you can narrow this down to the commit itself?
I am not able to test WiFi on that laptop computer anymore,
because that laptop now connects to world using wired connection.
It was that WiFI->wired connection change that led me to realize
the buggyness of in-tree iwlwifi in 4.19.y kernels.
I did that narrowing to specific kernel versions by digging my
archived backup files and their notes. No tests were run.
Problems started triggering in those kernels that I mentioned in
earlier email. That does not mean that the bugs were not already
there in kernels older that those. Maybe some change just widened
"window of opportunity" enough for me to see the issues.
In-tree iwlwifi in 4.19.y is missing locking fixes. No amount of
smooth-talking is going to change that.
--
Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-19 11:16 ` Greg Kroah-Hartman
@ 2021-02-19 16:50 ` Theodore Ts'o
-1 siblings, 0 replies; 133+ messages in thread
From: Theodore Ts'o @ 2021-02-19 16:50 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Jari Ruusu, Willy Tarreau, Jari Ruusu, Scott Branden, Linux ARM,
LKML, BCM Kernel Feedback
On Fri, Feb 19, 2021 at 12:16:12PM +0100, Greg Kroah-Hartman wrote:
>
> Great! Can you run 'git bisect' on the 4.14.y stable tree to find the
> offending change?
To be fair, especially with WiFi bugs, you may need to run for hours
or days before you are absolutely sure that a particular bisection
point will result in the the locking/kernel crash bug to manifest
itself. Worse, you have to be actively using the Wifi in order to see
the problem, and in some cases, it only triggers when you switch
between AP's, so you have to be actively using it in the work office,
and taking it between conference rooms, only to see your machine crash
taking your unsaved work, email drafts, etc. with it.
That being said, users should at least report the bug. (That's what I
did, when I ran into this a bunch of years ago, with an explanation of
"I'm trying to do a bisect, but it may take a few weeks for me to
figure out what the !@#!? is going on. In my case, I was trying to
use upstream -rcX kernels to dogfood on my work laptop, but the
principle is the same.)
Ultiumately, I solved the problem, by switching laptops to one that
didn't use an NVidia GPU (which sometimes forced me to stay 1-2
upstream versions behind, making life even more difficult when
debugging these issues, until the out-of-tree video driver got updated
to work with newer upstream), and which also had WiFi hardware which
was less subject to these issues.
It's unfortunate, but not all hardware is as well supported on Linux.
And in my case, because $WORK was using Enterprise WiFi systems with
AP's that don't get as much testing by developers, very few other
people could repro the bugs. That's life, and sometimes the only
solution is to switch hardware. And/or you just use a Chromebook in
those sorts of situations, separating your work/enterprise and
upstream development hardware, and be done with it. :-)
Cheers,
- Ted
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-19 16:50 ` Theodore Ts'o
0 siblings, 0 replies; 133+ messages in thread
From: Theodore Ts'o @ 2021-02-19 16:50 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Jari Ruusu, Scott Branden, Jari Ruusu, LKML, BCM Kernel Feedback,
Willy Tarreau, Linux ARM
On Fri, Feb 19, 2021 at 12:16:12PM +0100, Greg Kroah-Hartman wrote:
>
> Great! Can you run 'git bisect' on the 4.14.y stable tree to find the
> offending change?
To be fair, especially with WiFi bugs, you may need to run for hours
or days before you are absolutely sure that a particular bisection
point will result in the the locking/kernel crash bug to manifest
itself. Worse, you have to be actively using the Wifi in order to see
the problem, and in some cases, it only triggers when you switch
between AP's, so you have to be actively using it in the work office,
and taking it between conference rooms, only to see your machine crash
taking your unsaved work, email drafts, etc. with it.
That being said, users should at least report the bug. (That's what I
did, when I ran into this a bunch of years ago, with an explanation of
"I'm trying to do a bisect, but it may take a few weeks for me to
figure out what the !@#!? is going on. In my case, I was trying to
use upstream -rcX kernels to dogfood on my work laptop, but the
principle is the same.)
Ultiumately, I solved the problem, by switching laptops to one that
didn't use an NVidia GPU (which sometimes forced me to stay 1-2
upstream versions behind, making life even more difficult when
debugging these issues, until the out-of-tree video driver got updated
to work with newer upstream), and which also had WiFi hardware which
was less subject to these issues.
It's unfortunate, but not all hardware is as well supported on Linux.
And in my case, because $WORK was using Enterprise WiFi systems with
AP's that don't get as much testing by developers, very few other
people could repro the bugs. That's life, and sometimes the only
solution is to switch hardware. And/or you just use a Chromebook in
those sorts of situations, separating your work/enterprise and
upstream development hardware, and be done with it. :-)
Cheers,
- Ted
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 11:31 ` Willy Tarreau
@ 2021-02-18 17:16 ` Florian Fainelli
-1 siblings, 0 replies; 133+ messages in thread
From: Florian Fainelli @ 2021-02-18 17:16 UTC (permalink / raw)
To: Willy Tarreau, Greg Kroah-Hartman, Scott Branden
Cc: Linux ARM, LKML, BCM Kernel Feedback
On 2/18/2021 3:31 AM, Willy Tarreau wrote:
> On Thu, Feb 18, 2021 at 08:43:48AM +0100, Greg Kroah-Hartman wrote:
>> On Wed, Feb 17, 2021 at 11:48:21AM -0800, Scott Branden wrote:
>>> Other difficulty with the LTS version is the frequency it is updated.
>
> What a stange statement! So basically if fixes come in quickly so that
> customers are not exposed too long to well-known issues, it's a difficulty ?
> I guess by now every serious OS vendor provides at least weekly fixes, and
> at an era where devices are all interconnected, it's really necessary
> (unless of course you don't care about your customer's security).
>
>>> We would not
>>> pickup the changes that frequently to test. A quarterly, bi-annually, or when a critical fix
>>> is identified would be when we update and perform any meaningful testing when in maintainence.
>>
>> How are you "identifying" these "critical fixes"? We fix at least one
>> known security issue a week, and probably multitudes of
>> unknown-at-this-moment ones. How are you determining when you need to
>> send a new base kernel update off to your customers? At such long
>> intervals it feels like anyone using your kernel releases is woefully
>> insecure.
>
> +1! It seems like this dangerous practice will never end :-(
>
> Let me explain a personal experience. When I took over 2.6.32 many years
> ago, Greg asked me to adapt to the new maintenance process involving the
> patch reviews. At first I feared that it would increase my amount of work.
> And it did. But I also discovered how important these reviews were, because
> I started to get lots of "don't take this one in this version" and more
> importantly "if you merge this you'll need these ones as well". And very
> quickly I discovered how bogus the branches I used to maintain before
> had been, given the high feedback ratio!
>
> So based on this experience, I can assure anyone doing cherry-picks in
> their garage from LTS kernels that they're doing crap and that they must
> not distribute these kernels to anyone because THESE KERNELS ARE DANGEROUS.
> It's even very easy to introduce vulnerabilities by doing this!
Yes absolutely.
>
> The only set of fixes that can be trusted are the "official" stable
> kernels, because they are the only ones that are approved by the patches
> authors themselves.
Well, let us say that the authors had a chance to review the backports
being applied but given the volume maybe they did and silence means
agreement, or maybe they did not get a chance to review those changes.
Let us say that the trust level of the offical stable kernels is just
the highest of all kernels that are out there?
> Adding more stuff on top of stable kernels is fine
> (and done at your own risk), but randomly dropping stuff from stable
> kernels just because you don't think you need that is totally non-sense
> and must not be done anymore!
Yes, definitively not setting up for success.
--
Florian
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 17:16 ` Florian Fainelli
0 siblings, 0 replies; 133+ messages in thread
From: Florian Fainelli @ 2021-02-18 17:16 UTC (permalink / raw)
To: Willy Tarreau, Greg Kroah-Hartman, Scott Branden
Cc: BCM Kernel Feedback, LKML, Linux ARM
On 2/18/2021 3:31 AM, Willy Tarreau wrote:
> On Thu, Feb 18, 2021 at 08:43:48AM +0100, Greg Kroah-Hartman wrote:
>> On Wed, Feb 17, 2021 at 11:48:21AM -0800, Scott Branden wrote:
>>> Other difficulty with the LTS version is the frequency it is updated.
>
> What a stange statement! So basically if fixes come in quickly so that
> customers are not exposed too long to well-known issues, it's a difficulty ?
> I guess by now every serious OS vendor provides at least weekly fixes, and
> at an era where devices are all interconnected, it's really necessary
> (unless of course you don't care about your customer's security).
>
>>> We would not
>>> pickup the changes that frequently to test. A quarterly, bi-annually, or when a critical fix
>>> is identified would be when we update and perform any meaningful testing when in maintainence.
>>
>> How are you "identifying" these "critical fixes"? We fix at least one
>> known security issue a week, and probably multitudes of
>> unknown-at-this-moment ones. How are you determining when you need to
>> send a new base kernel update off to your customers? At such long
>> intervals it feels like anyone using your kernel releases is woefully
>> insecure.
>
> +1! It seems like this dangerous practice will never end :-(
>
> Let me explain a personal experience. When I took over 2.6.32 many years
> ago, Greg asked me to adapt to the new maintenance process involving the
> patch reviews. At first I feared that it would increase my amount of work.
> And it did. But I also discovered how important these reviews were, because
> I started to get lots of "don't take this one in this version" and more
> importantly "if you merge this you'll need these ones as well". And very
> quickly I discovered how bogus the branches I used to maintain before
> had been, given the high feedback ratio!
>
> So based on this experience, I can assure anyone doing cherry-picks in
> their garage from LTS kernels that they're doing crap and that they must
> not distribute these kernels to anyone because THESE KERNELS ARE DANGEROUS.
> It's even very easy to introduce vulnerabilities by doing this!
Yes absolutely.
>
> The only set of fixes that can be trusted are the "official" stable
> kernels, because they are the only ones that are approved by the patches
> authors themselves.
Well, let us say that the authors had a chance to review the backports
being applied but given the volume maybe they did and silence means
agreement, or maybe they did not get a chance to review those changes.
Let us say that the trust level of the offical stable kernels is just
the highest of all kernels that are out there?
> Adding more stuff on top of stable kernels is fine
> (and done at your own risk), but randomly dropping stuff from stable
> kernels just because you don't think you need that is totally non-sense
> and must not be done anymore!
Yes, definitively not setting up for success.
--
Florian
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 7:43 ` Greg Kroah-Hartman
@ 2021-02-18 12:51 ` Pavel Machek
-1 siblings, 0 replies; 133+ messages in thread
From: Pavel Machek @ 2021-02-18 12:51 UTC (permalink / raw)
To: Greg Kroah-Hartman; +Cc: Scott Branden, Linux ARM, LKML, BCM Kernel Feedback
[-- Attachment #1: Type: text/plain, Size: 2209 bytes --]
Hi!
> Why would you backport new features to an old kernel? That's not what
> they are there for.
For CIP project, this is one of advantages for "supported" boards, as
we'll backport patches improving their support as long as those
patches are mainline.
> > If the CIP project has committed to 10 years you would think they would be in contact
> > with you to add their support to the 5.10 LTS effort.
>
> They are doing testing right now, see the announcements where they test
> each stable -rc release. But they have not talked to me about 5.10
> yet, Their model is that they will, somehow in a way that is yet to be
> determined, take over maintaining these releases _after_ I drop them.
> But they are only going to do so for a very specific hardware platform
> or two, so anyone using anything other than their specific boards, is
> going to be out of luck.
I'd say this is a bit inaccurate :-). We care about different boards
at armv7, armv8 and x86-64 architectures, which actually means we care
about quite big part of the kernel already. Rough estimate is that 50%
of patches in stable touch our configurations, I could collect better
statistics.
As you said, I'm not sure everything is fully determined at this
point, but... we'll certainly try to support community effort for 4.4,
4.19 and 5.10 maintainance for as long as possible.
It is well possible that we'll continue to maintain even
configurations we can't test (we don't have s390 to test on) on
best-effort basis (to help community and because applying a patch may
be easier than determining if someone in CIP community depends on it).
> Which makes sense, scoping support like this to those that actually care
> about a specific hardware platform is much easier than the work that I
> and Sasha do in support it for all platforms. So if you are interested
> in 10 years, please work with CIP to get your platform into that list.
And yes, this is very good suggestion.
Best regards,
Pavel
PS: Again, I'm speaking for myself.
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 12:51 ` Pavel Machek
0 siblings, 0 replies; 133+ messages in thread
From: Pavel Machek @ 2021-02-18 12:51 UTC (permalink / raw)
To: Greg Kroah-Hartman; +Cc: BCM Kernel Feedback, LKML, Linux ARM, Scott Branden
[-- Attachment #1.1: Type: text/plain, Size: 2209 bytes --]
Hi!
> Why would you backport new features to an old kernel? That's not what
> they are there for.
For CIP project, this is one of advantages for "supported" boards, as
we'll backport patches improving their support as long as those
patches are mainline.
> > If the CIP project has committed to 10 years you would think they would be in contact
> > with you to add their support to the 5.10 LTS effort.
>
> They are doing testing right now, see the announcements where they test
> each stable -rc release. But they have not talked to me about 5.10
> yet, Their model is that they will, somehow in a way that is yet to be
> determined, take over maintaining these releases _after_ I drop them.
> But they are only going to do so for a very specific hardware platform
> or two, so anyone using anything other than their specific boards, is
> going to be out of luck.
I'd say this is a bit inaccurate :-). We care about different boards
at armv7, armv8 and x86-64 architectures, which actually means we care
about quite big part of the kernel already. Rough estimate is that 50%
of patches in stable touch our configurations, I could collect better
statistics.
As you said, I'm not sure everything is fully determined at this
point, but... we'll certainly try to support community effort for 4.4,
4.19 and 5.10 maintainance for as long as possible.
It is well possible that we'll continue to maintain even
configurations we can't test (we don't have s390 to test on) on
best-effort basis (to help community and because applying a patch may
be easier than determining if someone in CIP community depends on it).
> Which makes sense, scoping support like this to those that actually care
> about a specific hardware platform is much easier than the work that I
> and Sasha do in support it for all platforms. So if you are interested
> in 10 years, please work with CIP to get your platform into that list.
And yes, this is very good suggestion.
Best regards,
Pavel
PS: Again, I'm speaking for myself.
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]
[-- Attachment #2: Type: text/plain, Size: 176 bytes --]
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-17 19:48 ` Scott Branden
@ 2021-02-18 16:51 ` Sasha Levin
-1 siblings, 0 replies; 133+ messages in thread
From: Sasha Levin @ 2021-02-18 16:51 UTC (permalink / raw)
To: Scott Branden; +Cc: Greg Kroah-Hartman, BCM Kernel Feedback, LKML, Linux ARM
On Wed, Feb 17, 2021 at 11:48:21AM -0800, Scott Branden wrote:
>On 2021-02-17 1:40 a.m., Greg Kroah-Hartman wrote:
>> Following up on this as I did not hear back from you. Are you and/or
>> your company willing to help out with the testing of 5.10 to ensure that
>> it is a LTS kernel? So far I have not had any companies agree to help
>> out with this effort, which is sad to see as it seems that companies
>> want 6 years of stable kernels, yet do not seem to be able to at the
>> least, do a test-build/run of those kernels, which is quite odd...
>I personally cannot commit to supporting this kernel for 6 years
>(and personally do not want to backport new features to a 6 year old kernel).
>And customers are finicky and ask for one thing and then change their mind later.
Why would we commit to maintining an upstream LTS for 6 years then? If
no one ends up using it (and we don't want anyone using older LTS
kernels) we're still stuck maintaining it.
>We'll have to see what decisions are made at a company level for this as there
>are added costs to run tests on LTS kernel branches. We already run extensive QA on
This sounds very wrong: it's ok to get volunteers to commit to 6 years
while the company that is asking for it won't do the same?
Shouldn't Broadcom commit to the work involved here first?
>whatever active development branches are in use and a subset on the mainline
>branch as well. QA resources are finite and committing those for 6 years is
>not something that makes sense if customers drop that kernel version.
>Testing of the LTS kernel changes really moves out of our hands and into the
>customer's testing after our major releases to them.
Keep in mind that QA resources are generally more abundant than
engineering resources that need to actually backport stuff to old
kernels.
--
Thanks,
Sasha
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 16:51 ` Sasha Levin
0 siblings, 0 replies; 133+ messages in thread
From: Sasha Levin @ 2021-02-18 16:51 UTC (permalink / raw)
To: Scott Branden; +Cc: Greg Kroah-Hartman, BCM Kernel Feedback, LKML, Linux ARM
On Wed, Feb 17, 2021 at 11:48:21AM -0800, Scott Branden wrote:
>On 2021-02-17 1:40 a.m., Greg Kroah-Hartman wrote:
>> Following up on this as I did not hear back from you. Are you and/or
>> your company willing to help out with the testing of 5.10 to ensure that
>> it is a LTS kernel? So far I have not had any companies agree to help
>> out with this effort, which is sad to see as it seems that companies
>> want 6 years of stable kernels, yet do not seem to be able to at the
>> least, do a test-build/run of those kernels, which is quite odd...
>I personally cannot commit to supporting this kernel for 6 years
>(and personally do not want to backport new features to a 6 year old kernel).
>And customers are finicky and ask for one thing and then change their mind later.
Why would we commit to maintining an upstream LTS for 6 years then? If
no one ends up using it (and we don't want anyone using older LTS
kernels) we're still stuck maintaining it.
>We'll have to see what decisions are made at a company level for this as there
>are added costs to run tests on LTS kernel branches. We already run extensive QA on
This sounds very wrong: it's ok to get volunteers to commit to 6 years
while the company that is asking for it won't do the same?
Shouldn't Broadcom commit to the work involved here first?
>whatever active development branches are in use and a subset on the mainline
>branch as well. QA resources are finite and committing those for 6 years is
>not something that makes sense if customers drop that kernel version.
>Testing of the LTS kernel changes really moves out of our hands and into the
>customer's testing after our major releases to them.
Keep in mind that QA resources are generally more abundant than
engineering resources that need to actually backport stuff to old
kernels.
--
Thanks,
Sasha
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 16:51 ` Sasha Levin
@ 2021-02-18 17:21 ` Florian Fainelli
-1 siblings, 0 replies; 133+ messages in thread
From: Florian Fainelli @ 2021-02-18 17:21 UTC (permalink / raw)
To: Sasha Levin, Scott Branden
Cc: Greg Kroah-Hartman, BCM Kernel Feedback, LKML, Linux ARM
On 2/18/2021 8:51 AM, Sasha Levin wrote:
> On Wed, Feb 17, 2021 at 11:48:21AM -0800, Scott Branden wrote:
>> On 2021-02-17 1:40 a.m., Greg Kroah-Hartman wrote:
>>> Following up on this as I did not hear back from you. Are you and/or
>>> your company willing to help out with the testing of 5.10 to ensure that
>>> it is a LTS kernel? So far I have not had any companies agree to help
>>> out with this effort, which is sad to see as it seems that companies
>>> want 6 years of stable kernels, yet do not seem to be able to at the
>>> least, do a test-build/run of those kernels, which is quite odd...
>> I personally cannot commit to supporting this kernel for 6 years
>> (and personally do not want to backport new features to a 6 year old
>> kernel).
>> And customers are finicky and ask for one thing and then change their
>> mind later.
>
> Why would we commit to maintining an upstream LTS for 6 years then? If
> no one ends up using it (and we don't want anyone using older LTS
> kernels) we're still stuck maintaining it.
>
>> We'll have to see what decisions are made at a company level for this
>> as there
>> are added costs to run tests on LTS kernel branches. We already run
>> extensive QA on
>
> This sounds very wrong: it's ok to get volunteers to commit to 6 years
> while the company that is asking for it won't do the same?
>
> Shouldn't Broadcom commit to the work involved here first?
There are different groups within Broadcom, and Scott and I belong to
different groups, so I can only speak for mine. We can talk about why I
do not use the same email domain as Scott if you would like, and no, I
cannot help you fix the Broadcom Wi-Fi drivers :)
My group is committed to using the 5.10 kernel for the next 6 years
because of Android TV and we will do our best to test the 5.10 stable RC
as they come. We are a small team of 7 people in the grand scheme of our
larger business activities, and we get pulled into way too many things,
so we may skip testing a few stable releases once in a while. I do not
know how to make it more official than that.
As a company, we are most likely shooting ourselves in the foot by not
having a point of coordination with the Linux Foundation and key people
like you, Greg and other participants in the stable kernel. The usual
left hand and right hand not having yet discovered each other, hey hi
there left hand! I can see about remedying that at least for the
interest of the group I work in.
--
Florian
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 17:21 ` Florian Fainelli
0 siblings, 0 replies; 133+ messages in thread
From: Florian Fainelli @ 2021-02-18 17:21 UTC (permalink / raw)
To: Sasha Levin, Scott Branden
Cc: Greg Kroah-Hartman, BCM Kernel Feedback, LKML, Linux ARM
On 2/18/2021 8:51 AM, Sasha Levin wrote:
> On Wed, Feb 17, 2021 at 11:48:21AM -0800, Scott Branden wrote:
>> On 2021-02-17 1:40 a.m., Greg Kroah-Hartman wrote:
>>> Following up on this as I did not hear back from you. Are you and/or
>>> your company willing to help out with the testing of 5.10 to ensure that
>>> it is a LTS kernel? So far I have not had any companies agree to help
>>> out with this effort, which is sad to see as it seems that companies
>>> want 6 years of stable kernels, yet do not seem to be able to at the
>>> least, do a test-build/run of those kernels, which is quite odd...
>> I personally cannot commit to supporting this kernel for 6 years
>> (and personally do not want to backport new features to a 6 year old
>> kernel).
>> And customers are finicky and ask for one thing and then change their
>> mind later.
>
> Why would we commit to maintining an upstream LTS for 6 years then? If
> no one ends up using it (and we don't want anyone using older LTS
> kernels) we're still stuck maintaining it.
>
>> We'll have to see what decisions are made at a company level for this
>> as there
>> are added costs to run tests on LTS kernel branches. We already run
>> extensive QA on
>
> This sounds very wrong: it's ok to get volunteers to commit to 6 years
> while the company that is asking for it won't do the same?
>
> Shouldn't Broadcom commit to the work involved here first?
There are different groups within Broadcom, and Scott and I belong to
different groups, so I can only speak for mine. We can talk about why I
do not use the same email domain as Scott if you would like, and no, I
cannot help you fix the Broadcom Wi-Fi drivers :)
My group is committed to using the 5.10 kernel for the next 6 years
because of Android TV and we will do our best to test the 5.10 stable RC
as they come. We are a small team of 7 people in the grand scheme of our
larger business activities, and we get pulled into way too many things,
so we may skip testing a few stable releases once in a while. I do not
know how to make it more official than that.
As a company, we are most likely shooting ourselves in the foot by not
having a point of coordination with the Linux Foundation and key people
like you, Greg and other participants in the stable kernel. The usual
left hand and right hand not having yet discovered each other, hey hi
there left hand! I can see about remedying that at least for the
interest of the group I work in.
--
Florian
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 17:21 ` Florian Fainelli
@ 2021-02-18 17:53 ` Greg Kroah-Hartman
-1 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-18 17:53 UTC (permalink / raw)
To: Florian Fainelli
Cc: Sasha Levin, Scott Branden, BCM Kernel Feedback, LKML, Linux ARM
On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
> As a company, we are most likely shooting ourselves in the foot by not
> having a point of coordination with the Linux Foundation and key people
> like you, Greg and other participants in the stable kernel.
What does the LF have to do with this?
We are here, on the mailing lists, working with everyone. Just test the
-rc releases we make and let us know if they work or not for you, it's
not a lot of "coordination" needed at all.
Otherwise, if no one is saying that they are going to need these for 6
years and are willing to use it in their project (i.e. and test it),
there's no need for us to maintain it for that long, right?
thanks,
greg k-h
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 17:53 ` Greg Kroah-Hartman
0 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-18 17:53 UTC (permalink / raw)
To: Florian Fainelli
Cc: Sasha Levin, BCM Kernel Feedback, LKML, Linux ARM, Scott Branden
On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
> As a company, we are most likely shooting ourselves in the foot by not
> having a point of coordination with the Linux Foundation and key people
> like you, Greg and other participants in the stable kernel.
What does the LF have to do with this?
We are here, on the mailing lists, working with everyone. Just test the
-rc releases we make and let us know if they work or not for you, it's
not a lot of "coordination" needed at all.
Otherwise, if no one is saying that they are going to need these for 6
years and are willing to use it in their project (i.e. and test it),
there's no need for us to maintain it for that long, right?
thanks,
greg k-h
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 17:53 ` Greg Kroah-Hartman
@ 2021-02-18 17:57 ` Florian Fainelli
-1 siblings, 0 replies; 133+ messages in thread
From: Florian Fainelli @ 2021-02-18 17:57 UTC (permalink / raw)
To: Greg Kroah-Hartman, Florian Fainelli
Cc: Sasha Levin, Scott Branden, BCM Kernel Feedback, LKML, Linux ARM
On 2/18/2021 9:53 AM, Greg Kroah-Hartman wrote:
> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>> As a company, we are most likely shooting ourselves in the foot by not
>> having a point of coordination with the Linux Foundation and key people
>> like you, Greg and other participants in the stable kernel.
>
> What does the LF have to do with this?
I did not know whether the commitment/plan to use a given LTS kernel had
to be funneled through the Linux Foundation and then down to you, but I
am happy that this can be done publicly via a mailing list instead.
>
> We are here, on the mailing lists, working with everyone. Just test the
> -rc releases we make and let us know if they work or not for you, it's
> not a lot of "coordination" needed at all.
>
> Otherwise, if no one is saying that they are going to need these for 6
> years and are willing to use it in their project (i.e. and test it),
> there's no need for us to maintain it for that long, right?
Right, that is straight forward, works for me!
--
Florian
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 17:57 ` Florian Fainelli
0 siblings, 0 replies; 133+ messages in thread
From: Florian Fainelli @ 2021-02-18 17:57 UTC (permalink / raw)
To: Greg Kroah-Hartman, Florian Fainelli
Cc: Sasha Levin, BCM Kernel Feedback, LKML, Linux ARM, Scott Branden
On 2/18/2021 9:53 AM, Greg Kroah-Hartman wrote:
> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>> As a company, we are most likely shooting ourselves in the foot by not
>> having a point of coordination with the Linux Foundation and key people
>> like you, Greg and other participants in the stable kernel.
>
> What does the LF have to do with this?
I did not know whether the commitment/plan to use a given LTS kernel had
to be funneled through the Linux Foundation and then down to you, but I
am happy that this can be done publicly via a mailing list instead.
>
> We are here, on the mailing lists, working with everyone. Just test the
> -rc releases we make and let us know if they work or not for you, it's
> not a lot of "coordination" needed at all.
>
> Otherwise, if no one is saying that they are going to need these for 6
> years and are willing to use it in their project (i.e. and test it),
> there's no need for us to maintain it for that long, right?
Right, that is straight forward, works for me!
--
Florian
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 17:53 ` Greg Kroah-Hartman
@ 2021-02-18 18:20 ` Willy Tarreau
-1 siblings, 0 replies; 133+ messages in thread
From: Willy Tarreau @ 2021-02-18 18:20 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Florian Fainelli, Sasha Levin, BCM Kernel Feedback, LKML,
Linux ARM, Scott Branden
On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
> > As a company, we are most likely shooting ourselves in the foot by not
> > having a point of coordination with the Linux Foundation and key people
> > like you, Greg and other participants in the stable kernel.
>
> What does the LF have to do with this?
>
> We are here, on the mailing lists, working with everyone. Just test the
> -rc releases we make and let us know if they work or not for you, it's
> not a lot of "coordination" needed at all.
>
> Otherwise, if no one is saying that they are going to need these for 6
> years and are willing to use it in their project (i.e. and test it),
> there's no need for us to maintain it for that long, right?
Greg, please remember I expressed I really need them for slightly more than
3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
this saves me from having to take over these kernels after you, like in the
past, but I cannot engage on the regularity of my availability.
Overall I think that a lot of people completely underestimate the amount
of work it requires to maintain stable kernels, and how much it could be
distributed. By having a bunch of users participate a little bit more
(e.g. by sometimes backporting the patches that are essential to them,
by testing what's relevant to their use case), it already offloads a lot
of work. I don't think the extra work requires to be much organized if
there are enough participants to share the efforts.
Regards,
Willy
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 18:20 ` Willy Tarreau
0 siblings, 0 replies; 133+ messages in thread
From: Willy Tarreau @ 2021-02-18 18:20 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Sasha Levin, Florian Fainelli, Scott Branden, LKML,
BCM Kernel Feedback, Linux ARM
On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
> > As a company, we are most likely shooting ourselves in the foot by not
> > having a point of coordination with the Linux Foundation and key people
> > like you, Greg and other participants in the stable kernel.
>
> What does the LF have to do with this?
>
> We are here, on the mailing lists, working with everyone. Just test the
> -rc releases we make and let us know if they work or not for you, it's
> not a lot of "coordination" needed at all.
>
> Otherwise, if no one is saying that they are going to need these for 6
> years and are willing to use it in their project (i.e. and test it),
> there's no need for us to maintain it for that long, right?
Greg, please remember I expressed I really need them for slightly more than
3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
this saves me from having to take over these kernels after you, like in the
past, but I cannot engage on the regularity of my availability.
Overall I think that a lot of people completely underestimate the amount
of work it requires to maintain stable kernels, and how much it could be
distributed. By having a bunch of users participate a little bit more
(e.g. by sometimes backporting the patches that are essential to them,
by testing what's relevant to their use case), it already offloads a lot
of work. I don't think the extra work requires to be much organized if
there are enough participants to share the efforts.
Regards,
Willy
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 18:20 ` Willy Tarreau
@ 2021-02-18 18:36 ` Greg Kroah-Hartman
-1 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-18 18:36 UTC (permalink / raw)
To: Willy Tarreau
Cc: Florian Fainelli, Sasha Levin, BCM Kernel Feedback, LKML,
Linux ARM, Scott Branden
On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
> > On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
> > > As a company, we are most likely shooting ourselves in the foot by not
> > > having a point of coordination with the Linux Foundation and key people
> > > like you, Greg and other participants in the stable kernel.
> >
> > What does the LF have to do with this?
> >
> > We are here, on the mailing lists, working with everyone. Just test the
> > -rc releases we make and let us know if they work or not for you, it's
> > not a lot of "coordination" needed at all.
> >
> > Otherwise, if no one is saying that they are going to need these for 6
> > years and are willing to use it in their project (i.e. and test it),
> > there's no need for us to maintain it for that long, right?
>
> Greg, please remember I expressed I really need them for slightly more than
> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
> this saves me from having to take over these kernels after you, like in the
> past, but I cannot engage on the regularity of my availability.
Ok, great!
That's one person/company saying they can help out (along with what CIP
has been stating.)
What about others? Broadcom started this conversation, odd that they
don't seem to want to help out :)
thanks,
greg k-h
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 18:36 ` Greg Kroah-Hartman
0 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-18 18:36 UTC (permalink / raw)
To: Willy Tarreau
Cc: Sasha Levin, Florian Fainelli, Scott Branden, LKML,
BCM Kernel Feedback, Linux ARM
On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
> > On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
> > > As a company, we are most likely shooting ourselves in the foot by not
> > > having a point of coordination with the Linux Foundation and key people
> > > like you, Greg and other participants in the stable kernel.
> >
> > What does the LF have to do with this?
> >
> > We are here, on the mailing lists, working with everyone. Just test the
> > -rc releases we make and let us know if they work or not for you, it's
> > not a lot of "coordination" needed at all.
> >
> > Otherwise, if no one is saying that they are going to need these for 6
> > years and are willing to use it in their project (i.e. and test it),
> > there's no need for us to maintain it for that long, right?
>
> Greg, please remember I expressed I really need them for slightly more than
> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
> this saves me from having to take over these kernels after you, like in the
> past, but I cannot engage on the regularity of my availability.
Ok, great!
That's one person/company saying they can help out (along with what CIP
has been stating.)
What about others? Broadcom started this conversation, odd that they
don't seem to want to help out :)
thanks,
greg k-h
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 18:36 ` Greg Kroah-Hartman
@ 2021-02-18 20:16 ` Scott Branden
-1 siblings, 0 replies; 133+ messages in thread
From: Scott Branden @ 2021-02-18 20:16 UTC (permalink / raw)
To: Greg Kroah-Hartman, Willy Tarreau
Cc: Florian Fainelli, Sasha Levin, BCM Kernel Feedback, LKML, Linux ARM
[-- Attachment #1: Type: text/plain, Size: 2670 bytes --]
On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>>>> As a company, we are most likely shooting ourselves in the foot by not
>>>> having a point of coordination with the Linux Foundation and key people
>>>> like you, Greg and other participants in the stable kernel.
>>>
>>> What does the LF have to do with this?
>>>
>>> We are here, on the mailing lists, working with everyone. Just test the
>>> -rc releases we make and let us know if they work or not for you, it's
>>> not a lot of "coordination" needed at all.
>>>
>>> Otherwise, if no one is saying that they are going to need these for 6
>>> years and are willing to use it in their project (i.e. and test it),
>>> there's no need for us to maintain it for that long, right?
>>
>> Greg, please remember I expressed I really need them for slightly more than
>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
>> this saves me from having to take over these kernels after you, like in the
>> past, but I cannot engage on the regularity of my availability.
>
> Ok, great!
>
> That's one person/company saying they can help out (along with what CIP
> has been stating.)
>
> What about others? Broadcom started this conversation, odd that they
> don't seem to want to help out :)
Greg, I'm sorry but I'm not in a position to provide such a commitment.
My original question arose because the 5.10 kernel is declared as 2 years LTS while older LTS kernels are now 6 years.
One problem this has created is requests to provide silicon support in an older kernel version (for a new project) rather than starting from a newer kernel version that more properly supports the (silicon and non-silicon) features.
If all LTS kernels were declared as 3.5-4 years as Willy commented this would solve a few issues.
6 year LTS kernels would only have a maximum 1 year lifespan over the latest declared LTS kernel.
Also, many products take a year or more to develop, there isn't any life left in an LTS kernel if it is only 2 years.
After 1-3 years of kernel age the relevant parties that want to invest and care about supporting specific kernel versions longer should become apparent and could commit to longer support. Perhaps you move the burden of 6 years LTS elsewhere to longer term projects. But, I'm sure many are happy because you continue doing such a great job in a central location, especially those whose product lifespan is around 6 years.
>
> thanks,
>
> greg k-h
>
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4169 bytes --]
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 20:16 ` Scott Branden
0 siblings, 0 replies; 133+ messages in thread
From: Scott Branden @ 2021-02-18 20:16 UTC (permalink / raw)
To: Greg Kroah-Hartman, Willy Tarreau
Cc: Sasha Levin, Florian Fainelli, BCM Kernel Feedback, LKML, Linux ARM
[-- Attachment #1.1: Type: text/plain, Size: 2670 bytes --]
On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>>>> As a company, we are most likely shooting ourselves in the foot by not
>>>> having a point of coordination with the Linux Foundation and key people
>>>> like you, Greg and other participants in the stable kernel.
>>>
>>> What does the LF have to do with this?
>>>
>>> We are here, on the mailing lists, working with everyone. Just test the
>>> -rc releases we make and let us know if they work or not for you, it's
>>> not a lot of "coordination" needed at all.
>>>
>>> Otherwise, if no one is saying that they are going to need these for 6
>>> years and are willing to use it in their project (i.e. and test it),
>>> there's no need for us to maintain it for that long, right?
>>
>> Greg, please remember I expressed I really need them for slightly more than
>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
>> this saves me from having to take over these kernels after you, like in the
>> past, but I cannot engage on the regularity of my availability.
>
> Ok, great!
>
> That's one person/company saying they can help out (along with what CIP
> has been stating.)
>
> What about others? Broadcom started this conversation, odd that they
> don't seem to want to help out :)
Greg, I'm sorry but I'm not in a position to provide such a commitment.
My original question arose because the 5.10 kernel is declared as 2 years LTS while older LTS kernels are now 6 years.
One problem this has created is requests to provide silicon support in an older kernel version (for a new project) rather than starting from a newer kernel version that more properly supports the (silicon and non-silicon) features.
If all LTS kernels were declared as 3.5-4 years as Willy commented this would solve a few issues.
6 year LTS kernels would only have a maximum 1 year lifespan over the latest declared LTS kernel.
Also, many products take a year or more to develop, there isn't any life left in an LTS kernel if it is only 2 years.
After 1-3 years of kernel age the relevant parties that want to invest and care about supporting specific kernel versions longer should become apparent and could commit to longer support. Perhaps you move the burden of 6 years LTS elsewhere to longer term projects. But, I'm sure many are happy because you continue doing such a great job in a central location, especially those whose product lifespan is around 6 years.
>
> thanks,
>
> greg k-h
>
[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4169 bytes --]
[-- Attachment #2: Type: text/plain, Size: 176 bytes --]
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 20:16 ` Scott Branden
@ 2021-02-18 21:00 ` Willy Tarreau
-1 siblings, 0 replies; 133+ messages in thread
From: Willy Tarreau @ 2021-02-18 21:00 UTC (permalink / raw)
To: Scott Branden
Cc: Greg Kroah-Hartman, Sasha Levin, Florian Fainelli,
BCM Kernel Feedback, LKML, Linux ARM
On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
> > On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
> >> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
> >>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
> >>>> As a company, we are most likely shooting ourselves in the foot by not
> >>>> having a point of coordination with the Linux Foundation and key people
> >>>> like you, Greg and other participants in the stable kernel.
> >>>
> >>> What does the LF have to do with this?
> >>>
> >>> We are here, on the mailing lists, working with everyone. Just test the
> >>> -rc releases we make and let us know if they work or not for you, it's
> >>> not a lot of "coordination" needed at all.
> >>>
> >>> Otherwise, if no one is saying that they are going to need these for 6
> >>> years and are willing to use it in their project (i.e. and test it),
> >>> there's no need for us to maintain it for that long, right?
> >>
> >> Greg, please remember I expressed I really need them for slightly more than
> >> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
> >> this saves me from having to take over these kernels after you, like in the
> >> past, but I cannot engage on the regularity of my availability.
> >
> > Ok, great!
> >
> > That's one person/company saying they can help out (along with what CIP
> > has been stating.)
> >
> > What about others? Broadcom started this conversation, odd that they
> > don't seem to want to help out :)
> Greg, I'm sorry but I'm not in a position to provide such a commitment.
Are you at least in a position to defend that ? There are necessarily
some people in your company who understand the benefits of using open
source provdided for free by others and who understand that devoting
a few people's time to this task is extremely cheap compared to the
amount of work required by having to do it entirely yourself for a
lower quality.
> My original question arose because the 5.10 kernel is declared as 2 years LTS
> while older LTS kernels are now 6 years.
(...)
> If all LTS kernels were declared as 3.5-4 years as Willy commented this would
> solve a few issues. 6 year LTS kernels would only have a maximum 1 year
> lifespan over the latest declared LTS kernel. Also, many products take a year
> or more to develop, there isn't any life left in an LTS kernel if it is only
> 2 years.
We all have the same problem regarding this but how do you want Greg to
engage into such a task by himself if he's not certain he can count on
others to help ? The few of us having worked on extended kernels know
that there's a limit around 2.5 years beyond which backports become much
harder to perform and to test. Doing it every year would result in 6 LTS
kernels to maintain in addition to the last 1-2 stable ones. That becomes
a huge amount of work! I even think that having one LTS kernel every 2
years but maintained one extra year (e.g. 5 vs 4 in my case) would reduce
the effort.
> After 1-3 years of kernel age the relevant parties that want to invest and
> care about supporting specific kernel versions longer should become apparent
> and could commit to longer support.
But that's exactly what's currently being done. Greg initially commits
to 2 years hoping to get some help to pursue this longer, and this causes
trouble to some of us not being certain upfront whether or not we're choosing
the right kernel. So only the solution I'm seeing is for Greg to know
early who jumps in so that those of us without the power or skill to
entirely maintain a kernel by themselves know early which version to
choose. Quite frankly if we ship an LTS kernel in a product, the least
we can do is to give back a little bit to make sure the situation remains
durable.
As such even if you are not in a position to provide such a commitment,
I'd appreciate it if you would bring these arguments to those who are in
such a position, so that I don't end up as one of the too few ones having
to share a significant part of that task to make sure this valuable kernel
continues to exist.
Thanks,
Willy
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 21:00 ` Willy Tarreau
0 siblings, 0 replies; 133+ messages in thread
From: Willy Tarreau @ 2021-02-18 21:00 UTC (permalink / raw)
To: Scott Branden
Cc: Sasha Levin, Florian Fainelli, Greg Kroah-Hartman, LKML,
BCM Kernel Feedback, Linux ARM
On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
> > On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
> >> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
> >>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
> >>>> As a company, we are most likely shooting ourselves in the foot by not
> >>>> having a point of coordination with the Linux Foundation and key people
> >>>> like you, Greg and other participants in the stable kernel.
> >>>
> >>> What does the LF have to do with this?
> >>>
> >>> We are here, on the mailing lists, working with everyone. Just test the
> >>> -rc releases we make and let us know if they work or not for you, it's
> >>> not a lot of "coordination" needed at all.
> >>>
> >>> Otherwise, if no one is saying that they are going to need these for 6
> >>> years and are willing to use it in their project (i.e. and test it),
> >>> there's no need for us to maintain it for that long, right?
> >>
> >> Greg, please remember I expressed I really need them for slightly more than
> >> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
> >> this saves me from having to take over these kernels after you, like in the
> >> past, but I cannot engage on the regularity of my availability.
> >
> > Ok, great!
> >
> > That's one person/company saying they can help out (along with what CIP
> > has been stating.)
> >
> > What about others? Broadcom started this conversation, odd that they
> > don't seem to want to help out :)
> Greg, I'm sorry but I'm not in a position to provide such a commitment.
Are you at least in a position to defend that ? There are necessarily
some people in your company who understand the benefits of using open
source provdided for free by others and who understand that devoting
a few people's time to this task is extremely cheap compared to the
amount of work required by having to do it entirely yourself for a
lower quality.
> My original question arose because the 5.10 kernel is declared as 2 years LTS
> while older LTS kernels are now 6 years.
(...)
> If all LTS kernels were declared as 3.5-4 years as Willy commented this would
> solve a few issues. 6 year LTS kernels would only have a maximum 1 year
> lifespan over the latest declared LTS kernel. Also, many products take a year
> or more to develop, there isn't any life left in an LTS kernel if it is only
> 2 years.
We all have the same problem regarding this but how do you want Greg to
engage into such a task by himself if he's not certain he can count on
others to help ? The few of us having worked on extended kernels know
that there's a limit around 2.5 years beyond which backports become much
harder to perform and to test. Doing it every year would result in 6 LTS
kernels to maintain in addition to the last 1-2 stable ones. That becomes
a huge amount of work! I even think that having one LTS kernel every 2
years but maintained one extra year (e.g. 5 vs 4 in my case) would reduce
the effort.
> After 1-3 years of kernel age the relevant parties that want to invest and
> care about supporting specific kernel versions longer should become apparent
> and could commit to longer support.
But that's exactly what's currently being done. Greg initially commits
to 2 years hoping to get some help to pursue this longer, and this causes
trouble to some of us not being certain upfront whether or not we're choosing
the right kernel. So only the solution I'm seeing is for Greg to know
early who jumps in so that those of us without the power or skill to
entirely maintain a kernel by themselves know early which version to
choose. Quite frankly if we ship an LTS kernel in a product, the least
we can do is to give back a little bit to make sure the situation remains
durable.
As such even if you are not in a position to provide such a commitment,
I'd appreciate it if you would bring these arguments to those who are in
such a position, so that I don't end up as one of the too few ones having
to share a significant part of that task to make sure this valuable kernel
continues to exist.
Thanks,
Willy
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 21:00 ` Willy Tarreau
@ 2021-02-18 22:38 ` Scott Branden
-1 siblings, 0 replies; 133+ messages in thread
From: Scott Branden @ 2021-02-18 22:38 UTC (permalink / raw)
To: Willy Tarreau
Cc: Greg Kroah-Hartman, Sasha Levin, Florian Fainelli,
BCM Kernel Feedback, LKML, Linux ARM
[-- Attachment #1: Type: text/plain, Size: 4383 bytes --]
Hi Willy,
On 2021-02-18 1:00 p.m., Willy Tarreau wrote:
> On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
>> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
>>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
>>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
>>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>>>>>> As a company, we are most likely shooting ourselves in the foot by not
>>>>>> having a point of coordination with the Linux Foundation and key people
>>>>>> like you, Greg and other participants in the stable kernel.
>>>>>
>>>>> What does the LF have to do with this?
>>>>>
>>>>> We are here, on the mailing lists, working with everyone. Just test the
>>>>> -rc releases we make and let us know if they work or not for you, it's
>>>>> not a lot of "coordination" needed at all.
>>>>>
>>>>> Otherwise, if no one is saying that they are going to need these for 6
>>>>> years and are willing to use it in their project (i.e. and test it),
>>>>> there's no need for us to maintain it for that long, right?
>>>>
>>>> Greg, please remember I expressed I really need them for slightly more than
>>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
>>>> this saves me from having to take over these kernels after you, like in the
>>>> past, but I cannot engage on the regularity of my availability.
>>>
>>> Ok, great!
>>>
>>> That's one person/company saying they can help out (along with what CIP
>>> has been stating.)
>>>
>>> What about others? Broadcom started this conversation, odd that they
>>> don't seem to want to help out :)
>> Greg, I'm sorry but I'm not in a position to provide such a commitment.
>
> Are you at least in a position to defend that ? There are necessarily
> some people in your company who understand the benefits of using open
> source provdided for free by others and who understand that devoting
> a few people's time to this task is extremely cheap compared to the
> amount of work required by having to do it entirely yourself for a
> lower quality.
>
>> My original question arose because the 5.10 kernel is declared as 2 years LTS
>> while older LTS kernels are now 6 years.
> (...)
>> If all LTS kernels were declared as 3.5-4 years as Willy commented this would
>> solve a few issues. 6 year LTS kernels would only have a maximum 1 year
>> lifespan over the latest declared LTS kernel. Also, many products take a year
>> or more to develop, there isn't any life left in an LTS kernel if it is only
>> 2 years.
>
> We all have the same problem regarding this but how do you want Greg to
> engage into such a task by himself if he's not certain he can count on
> others to help ? The few of us having worked on extended kernels know
> that there's a limit around 2.5 years beyond which backports become much
> harder to perform and to test. Doing it every year would result in 6 LTS
> kernels to maintain in addition to the last 1-2 stable ones. That becomes
> a huge amount of work! I even think that having one LTS kernel every 2
> years but maintained one extra year (e.g. 5 vs 4 in my case) would reduce
> the effort.
>
>> After 1-3 years of kernel age the relevant parties that want to invest and
>> care about supporting specific kernel versions longer should become apparent
>> and could commit to longer support.
>
> But that's exactly what's currently being done. Greg initially commits
> to 2 years hoping to get some help to pursue this longer, and this causes
> trouble to some of us not being certain upfront whether or not we're choosing
> the right kernel. So only the solution I'm seeing is for Greg to know
> early who jumps in so that those of us without the power or skill to
> entirely maintain a kernel by themselves know early which version to
> choose. Quite frankly if we ship an LTS kernel in a product, the least
> we can do is to give back a little bit to make sure the situation remains
> durable.
>
> As such even if you are not in a position to provide such a commitment,
> I'd appreciate it if you would bring these arguments to those who are in
> such a position, so that I don't end up as one of the too few ones having
> to share a significant part of that task to make sure this valuable kernel
> continues to exist.
Thanks - will forward such info as necessary.
>
> Thanks,
> Willy
>
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4169 bytes --]
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 22:38 ` Scott Branden
0 siblings, 0 replies; 133+ messages in thread
From: Scott Branden @ 2021-02-18 22:38 UTC (permalink / raw)
To: Willy Tarreau
Cc: Sasha Levin, Florian Fainelli, Greg Kroah-Hartman, LKML,
BCM Kernel Feedback, Linux ARM
[-- Attachment #1.1: Type: text/plain, Size: 4383 bytes --]
Hi Willy,
On 2021-02-18 1:00 p.m., Willy Tarreau wrote:
> On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
>> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
>>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
>>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
>>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>>>>>> As a company, we are most likely shooting ourselves in the foot by not
>>>>>> having a point of coordination with the Linux Foundation and key people
>>>>>> like you, Greg and other participants in the stable kernel.
>>>>>
>>>>> What does the LF have to do with this?
>>>>>
>>>>> We are here, on the mailing lists, working with everyone. Just test the
>>>>> -rc releases we make and let us know if they work or not for you, it's
>>>>> not a lot of "coordination" needed at all.
>>>>>
>>>>> Otherwise, if no one is saying that they are going to need these for 6
>>>>> years and are willing to use it in their project (i.e. and test it),
>>>>> there's no need for us to maintain it for that long, right?
>>>>
>>>> Greg, please remember I expressed I really need them for slightly more than
>>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
>>>> this saves me from having to take over these kernels after you, like in the
>>>> past, but I cannot engage on the regularity of my availability.
>>>
>>> Ok, great!
>>>
>>> That's one person/company saying they can help out (along with what CIP
>>> has been stating.)
>>>
>>> What about others? Broadcom started this conversation, odd that they
>>> don't seem to want to help out :)
>> Greg, I'm sorry but I'm not in a position to provide such a commitment.
>
> Are you at least in a position to defend that ? There are necessarily
> some people in your company who understand the benefits of using open
> source provdided for free by others and who understand that devoting
> a few people's time to this task is extremely cheap compared to the
> amount of work required by having to do it entirely yourself for a
> lower quality.
>
>> My original question arose because the 5.10 kernel is declared as 2 years LTS
>> while older LTS kernels are now 6 years.
> (...)
>> If all LTS kernels were declared as 3.5-4 years as Willy commented this would
>> solve a few issues. 6 year LTS kernels would only have a maximum 1 year
>> lifespan over the latest declared LTS kernel. Also, many products take a year
>> or more to develop, there isn't any life left in an LTS kernel if it is only
>> 2 years.
>
> We all have the same problem regarding this but how do you want Greg to
> engage into such a task by himself if he's not certain he can count on
> others to help ? The few of us having worked on extended kernels know
> that there's a limit around 2.5 years beyond which backports become much
> harder to perform and to test. Doing it every year would result in 6 LTS
> kernels to maintain in addition to the last 1-2 stable ones. That becomes
> a huge amount of work! I even think that having one LTS kernel every 2
> years but maintained one extra year (e.g. 5 vs 4 in my case) would reduce
> the effort.
>
>> After 1-3 years of kernel age the relevant parties that want to invest and
>> care about supporting specific kernel versions longer should become apparent
>> and could commit to longer support.
>
> But that's exactly what's currently being done. Greg initially commits
> to 2 years hoping to get some help to pursue this longer, and this causes
> trouble to some of us not being certain upfront whether or not we're choosing
> the right kernel. So only the solution I'm seeing is for Greg to know
> early who jumps in so that those of us without the power or skill to
> entirely maintain a kernel by themselves know early which version to
> choose. Quite frankly if we ship an LTS kernel in a product, the least
> we can do is to give back a little bit to make sure the situation remains
> durable.
>
> As such even if you are not in a position to provide such a commitment,
> I'd appreciate it if you would bring these arguments to those who are in
> such a position, so that I don't end up as one of the too few ones having
> to share a significant part of that task to make sure this valuable kernel
> continues to exist.
Thanks - will forward such info as necessary.
>
> Thanks,
> Willy
>
[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4169 bytes --]
[-- Attachment #2: Type: text/plain, Size: 176 bytes --]
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 20:16 ` Scott Branden
@ 2021-02-18 21:39 ` Sasha Levin
-1 siblings, 0 replies; 133+ messages in thread
From: Sasha Levin @ 2021-02-18 21:39 UTC (permalink / raw)
To: Scott Branden
Cc: Greg Kroah-Hartman, Willy Tarreau, Florian Fainelli,
BCM Kernel Feedback, LKML, Linux ARM
On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
>On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>>>>> As a company, we are most likely shooting ourselves in the foot by not
>>>>> having a point of coordination with the Linux Foundation and key people
>>>>> like you, Greg and other participants in the stable kernel.
>>>>
>>>> What does the LF have to do with this?
>>>>
>>>> We are here, on the mailing lists, working with everyone. Just test the
>>>> -rc releases we make and let us know if they work or not for you, it's
>>>> not a lot of "coordination" needed at all.
>>>>
>>>> Otherwise, if no one is saying that they are going to need these for 6
>>>> years and are willing to use it in their project (i.e. and test it),
>>>> there's no need for us to maintain it for that long, right?
>>>
>>> Greg, please remember I expressed I really need them for slightly more than
>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
>>> this saves me from having to take over these kernels after you, like in the
>>> past, but I cannot engage on the regularity of my availability.
>>
>> Ok, great!
>>
>> That's one person/company saying they can help out (along with what CIP
>> has been stating.)
>>
>> What about others? Broadcom started this conversation, odd that they
>> don't seem to want to help out :)
>Greg, I'm sorry but I'm not in a position to provide such a commitment.
>
>My original question arose because the 5.10 kernel is declared as 2 years LTS while older LTS kernels are now 6 years.
>One problem this has created is requests to provide silicon support in an older kernel version (for a new project) rather than starting from a newer kernel version that more properly supports the (silicon and non-silicon) features.
So this sounds like you have boatloads of out-of-tree code and need a
stable kernel to avoid having to rebase that code. This is not why the
LTS trees are around.
For new projects, the easiest route is to upstream your stuff and ship
the latest kernel.
>If all LTS kernels were declared as 3.5-4 years as Willy commented this would solve a few issues.
>6 year LTS kernels would only have a maximum 1 year lifespan over the latest declared LTS kernel.
>Also, many products take a year or more to develop, there isn't any life left in an LTS kernel if it is only 2 years.
Products are supposed to upgrade their kernel. If you released something
with, for example, a 5.4 kernel, doesn't mean you're forever stuck on a
5.4 kernel for that product.
>After 1-3 years of kernel age the relevant parties that want to invest and care about supporting specific kernel versions longer should become apparent and could commit to longer support. Perhaps you move the burden of 6 years LTS elsewhere to longer term projects. But, I'm sure many are happy because you continue doing such a great job in a central location, especially those whose product lifespan is around 6 years.
But this is exactly what's happening now: we support LTS kernels for two
years, and after that interested parties can figure it out on their own
if it's worth it for them to keep going.
--
Thanks,
Sasha
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 21:39 ` Sasha Levin
0 siblings, 0 replies; 133+ messages in thread
From: Sasha Levin @ 2021-02-18 21:39 UTC (permalink / raw)
To: Scott Branden
Cc: Florian Fainelli, Greg Kroah-Hartman, LKML, BCM Kernel Feedback,
Willy Tarreau, Linux ARM
On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
>On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>>>>> As a company, we are most likely shooting ourselves in the foot by not
>>>>> having a point of coordination with the Linux Foundation and key people
>>>>> like you, Greg and other participants in the stable kernel.
>>>>
>>>> What does the LF have to do with this?
>>>>
>>>> We are here, on the mailing lists, working with everyone. Just test the
>>>> -rc releases we make and let us know if they work or not for you, it's
>>>> not a lot of "coordination" needed at all.
>>>>
>>>> Otherwise, if no one is saying that they are going to need these for 6
>>>> years and are willing to use it in their project (i.e. and test it),
>>>> there's no need for us to maintain it for that long, right?
>>>
>>> Greg, please remember I expressed I really need them for slightly more than
>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
>>> this saves me from having to take over these kernels after you, like in the
>>> past, but I cannot engage on the regularity of my availability.
>>
>> Ok, great!
>>
>> That's one person/company saying they can help out (along with what CIP
>> has been stating.)
>>
>> What about others? Broadcom started this conversation, odd that they
>> don't seem to want to help out :)
>Greg, I'm sorry but I'm not in a position to provide such a commitment.
>
>My original question arose because the 5.10 kernel is declared as 2 years LTS while older LTS kernels are now 6 years.
>One problem this has created is requests to provide silicon support in an older kernel version (for a new project) rather than starting from a newer kernel version that more properly supports the (silicon and non-silicon) features.
So this sounds like you have boatloads of out-of-tree code and need a
stable kernel to avoid having to rebase that code. This is not why the
LTS trees are around.
For new projects, the easiest route is to upstream your stuff and ship
the latest kernel.
>If all LTS kernels were declared as 3.5-4 years as Willy commented this would solve a few issues.
>6 year LTS kernels would only have a maximum 1 year lifespan over the latest declared LTS kernel.
>Also, many products take a year or more to develop, there isn't any life left in an LTS kernel if it is only 2 years.
Products are supposed to upgrade their kernel. If you released something
with, for example, a 5.4 kernel, doesn't mean you're forever stuck on a
5.4 kernel for that product.
>After 1-3 years of kernel age the relevant parties that want to invest and care about supporting specific kernel versions longer should become apparent and could commit to longer support. Perhaps you move the burden of 6 years LTS elsewhere to longer term projects. But, I'm sure many are happy because you continue doing such a great job in a central location, especially those whose product lifespan is around 6 years.
But this is exactly what's happening now: we support LTS kernels for two
years, and after that interested parties can figure it out on their own
if it's worth it for them to keep going.
--
Thanks,
Sasha
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 21:39 ` Sasha Levin
@ 2021-02-18 22:00 ` Florian Fainelli
-1 siblings, 0 replies; 133+ messages in thread
From: Florian Fainelli @ 2021-02-18 22:00 UTC (permalink / raw)
To: Sasha Levin, Scott Branden
Cc: Greg Kroah-Hartman, Willy Tarreau, Florian Fainelli,
BCM Kernel Feedback, LKML, Linux ARM
On 2/18/2021 1:39 PM, Sasha Levin wrote:
> On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
>> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
>>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
>>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
>>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>>>>>> As a company, we are most likely shooting ourselves in the foot by
>>>>>> not
>>>>>> having a point of coordination with the Linux Foundation and key
>>>>>> people
>>>>>> like you, Greg and other participants in the stable kernel.
>>>>>
>>>>> What does the LF have to do with this?
>>>>>
>>>>> We are here, on the mailing lists, working with everyone. Just
>>>>> test the
>>>>> -rc releases we make and let us know if they work or not for you, it's
>>>>> not a lot of "coordination" needed at all.
>>>>>
>>>>> Otherwise, if no one is saying that they are going to need these for 6
>>>>> years and are willing to use it in their project (i.e. and test it),
>>>>> there's no need for us to maintain it for that long, right?
>>>>
>>>> Greg, please remember I expressed I really need them for slightly
>>>> more than
>>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time
>>>> permits if
>>>> this saves me from having to take over these kernels after you, like
>>>> in the
>>>> past, but I cannot engage on the regularity of my availability.
>>>
>>> Ok, great!
>>>
>>> That's one person/company saying they can help out (along with what CIP
>>> has been stating.)
>>>
>>> What about others? Broadcom started this conversation, odd that they
>>> don't seem to want to help out :)
>> Greg, I'm sorry but I'm not in a position to provide such a commitment.
>>
>> My original question arose because the 5.10 kernel is declared as 2
>> years LTS while older LTS kernels are now 6 years.
>> One problem this has created is requests to provide silicon support in
>> an older kernel version (for a new project) rather than starting from
>> a newer kernel version that more properly supports the (silicon and
>> non-silicon) features.
>
> So this sounds like you have boatloads of out-of-tree code and need a
> stable kernel to avoid having to rebase that code. This is not why the
> LTS trees are around.
>
> For new projects, the easiest route is to upstream your stuff and ship
> the latest kernel.
Agreed, however no matter how good you are at upstreaming your changes,
there is always a delta that you keep accumulating, it is just the
reality of things be it because a specific piece of code was just harder
to upstream you missed multiple kernel versions, or it was not suitable,
or it changed, or anything. Not everything goes upstream even if that
gets closer and closer to zero if you do a good job at it.
>
>> If all LTS kernels were declared as 3.5-4 years as Willy commented
>> this would solve a few issues.
>> 6 year LTS kernels would only have a maximum 1 year lifespan over the
>> latest declared LTS kernel.
>> Also, many products take a year or more to develop, there isn't any
>> life left in an LTS kernel if it is only 2 years.
>
> Products are supposed to upgrade their kernel. If you released something
> with, for example, a 5.4 kernel, doesn't mean you're forever stuck on a
> 5.4 kernel for that product.
The are lots of Android devices manufacturers that launch using of the 3
kernel versions supported by the given Android release they targeted and
they do not update the kernel version ever after that.
>
>> After 1-3 years of kernel age the relevant parties that want to invest
>> and care about supporting specific kernel versions longer should
>> become apparent and could commit to longer support. Perhaps you move
>> the burden of 6 years LTS elsewhere to longer term projects. But, I'm
>> sure many are happy because you continue doing such a great job in a
>> central location, especially those whose product lifespan is around 6
>> years.
>
> But this is exactly what's happening now: we support LTS kernels for two
> years, and after that interested parties can figure it out on their own
> if it's worth it for them to keep going.
>
--
Florian
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 22:00 ` Florian Fainelli
0 siblings, 0 replies; 133+ messages in thread
From: Florian Fainelli @ 2021-02-18 22:00 UTC (permalink / raw)
To: Sasha Levin, Scott Branden
Cc: Florian Fainelli, Greg Kroah-Hartman, LKML, BCM Kernel Feedback,
Willy Tarreau, Linux ARM
On 2/18/2021 1:39 PM, Sasha Levin wrote:
> On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
>> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
>>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
>>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
>>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>>>>>> As a company, we are most likely shooting ourselves in the foot by
>>>>>> not
>>>>>> having a point of coordination with the Linux Foundation and key
>>>>>> people
>>>>>> like you, Greg and other participants in the stable kernel.
>>>>>
>>>>> What does the LF have to do with this?
>>>>>
>>>>> We are here, on the mailing lists, working with everyone. Just
>>>>> test the
>>>>> -rc releases we make and let us know if they work or not for you, it's
>>>>> not a lot of "coordination" needed at all.
>>>>>
>>>>> Otherwise, if no one is saying that they are going to need these for 6
>>>>> years and are willing to use it in their project (i.e. and test it),
>>>>> there's no need for us to maintain it for that long, right?
>>>>
>>>> Greg, please remember I expressed I really need them for slightly
>>>> more than
>>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time
>>>> permits if
>>>> this saves me from having to take over these kernels after you, like
>>>> in the
>>>> past, but I cannot engage on the regularity of my availability.
>>>
>>> Ok, great!
>>>
>>> That's one person/company saying they can help out (along with what CIP
>>> has been stating.)
>>>
>>> What about others? Broadcom started this conversation, odd that they
>>> don't seem to want to help out :)
>> Greg, I'm sorry but I'm not in a position to provide such a commitment.
>>
>> My original question arose because the 5.10 kernel is declared as 2
>> years LTS while older LTS kernels are now 6 years.
>> One problem this has created is requests to provide silicon support in
>> an older kernel version (for a new project) rather than starting from
>> a newer kernel version that more properly supports the (silicon and
>> non-silicon) features.
>
> So this sounds like you have boatloads of out-of-tree code and need a
> stable kernel to avoid having to rebase that code. This is not why the
> LTS trees are around.
>
> For new projects, the easiest route is to upstream your stuff and ship
> the latest kernel.
Agreed, however no matter how good you are at upstreaming your changes,
there is always a delta that you keep accumulating, it is just the
reality of things be it because a specific piece of code was just harder
to upstream you missed multiple kernel versions, or it was not suitable,
or it changed, or anything. Not everything goes upstream even if that
gets closer and closer to zero if you do a good job at it.
>
>> If all LTS kernels were declared as 3.5-4 years as Willy commented
>> this would solve a few issues.
>> 6 year LTS kernels would only have a maximum 1 year lifespan over the
>> latest declared LTS kernel.
>> Also, many products take a year or more to develop, there isn't any
>> life left in an LTS kernel if it is only 2 years.
>
> Products are supposed to upgrade their kernel. If you released something
> with, for example, a 5.4 kernel, doesn't mean you're forever stuck on a
> 5.4 kernel for that product.
The are lots of Android devices manufacturers that launch using of the 3
kernel versions supported by the given Android release they targeted and
they do not update the kernel version ever after that.
>
>> After 1-3 years of kernel age the relevant parties that want to invest
>> and care about supporting specific kernel versions longer should
>> become apparent and could commit to longer support. Perhaps you move
>> the burden of 6 years LTS elsewhere to longer term projects. But, I'm
>> sure many are happy because you continue doing such a great job in a
>> central location, especially those whose product lifespan is around 6
>> years.
>
> But this is exactly what's happening now: we support LTS kernels for two
> years, and after that interested parties can figure it out on their own
> if it's worth it for them to keep going.
>
--
Florian
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 21:39 ` Sasha Levin
@ 2021-02-18 22:26 ` Scott Branden
-1 siblings, 0 replies; 133+ messages in thread
From: Scott Branden @ 2021-02-18 22:26 UTC (permalink / raw)
To: Sasha Levin
Cc: Greg Kroah-Hartman, Willy Tarreau, Florian Fainelli,
BCM Kernel Feedback, LKML, Linux ARM
[-- Attachment #1: Type: text/plain, Size: 3963 bytes --]
On 2021-02-18 1:39 p.m., Sasha Levin wrote:
> On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
>> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
>>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
>>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
>>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>>>>>> As a company, we are most likely shooting ourselves in the foot by not
>>>>>> having a point of coordination with the Linux Foundation and key people
>>>>>> like you, Greg and other participants in the stable kernel.
>>>>>
>>>>> What does the LF have to do with this?
>>>>>
>>>>> We are here, on the mailing lists, working with everyone. Just test the
>>>>> -rc releases we make and let us know if they work or not for you, it's
>>>>> not a lot of "coordination" needed at all.
>>>>>
>>>>> Otherwise, if no one is saying that they are going to need these for 6
>>>>> years and are willing to use it in their project (i.e. and test it),
>>>>> there's no need for us to maintain it for that long, right?
>>>>
>>>> Greg, please remember I expressed I really need them for slightly more than
>>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
>>>> this saves me from having to take over these kernels after you, like in the
>>>> past, but I cannot engage on the regularity of my availability.
>>>
>>> Ok, great!
>>>
>>> That's one person/company saying they can help out (along with what CIP
>>> has been stating.)
>>>
>>> What about others? Broadcom started this conversation, odd that they
>>> don't seem to want to help out :)
>> Greg, I'm sorry but I'm not in a position to provide such a commitment.
>>
>> My original question arose because the 5.10 kernel is declared as 2 years LTS while older LTS kernels are now 6 years.
>> One problem this has created is requests to provide silicon support in an older kernel version (for a new project) rather than starting from a newer kernel version that more properly supports the (silicon and non-silicon) features.
>
> So this sounds like you have boatloads of out-of-tree code and need a
> stable kernel to avoid having to rebase that code. This is not why the
> LTS trees are around.
My question centered around the inconsistency of a 2 year LTS kernel and some of the problems
of having older LTS kernels declared as 6 years.
The discussion has gone sideways from there with many other assumptions.
There is not always control over what kernel version a customer uses,
how many hacks they have in it, when they upgrade it, what other vendor changes are in it, etc.
But, will keep this email chain to forward to others. It may help influence some decisions.
>
> For new projects, the easiest route is to upstream your stuff and ship
> the latest kernel.
>
>> If all LTS kernels were declared as 3.5-4 years as Willy commented this would solve a few issues.
>> 6 year LTS kernels would only have a maximum 1 year lifespan over the latest declared LTS kernel.
>> Also, many products take a year or more to develop, there isn't any life left in an LTS kernel if it is only 2 years.
>
> Products are supposed to upgrade their kernel. If you released something
> with, for example, a 5.4 kernel, doesn't mean you're forever stuck on a
> 5.4 kernel for that product.
>
>> After 1-3 years of kernel age the relevant parties that want to invest and care about supporting specific kernel versions longer should become apparent and could commit to longer support. Perhaps you move the burden of 6 years LTS elsewhere to longer term projects. But, I'm sure many are happy because you continue doing such a great job in a central location, especially those whose product lifespan is around 6 years.
>
> But this is exactly what's happening now: we support LTS kernels for two
> years, and after that interested parties can figure it out on their own
> if it's worth it for them to keep going.
>
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4169 bytes --]
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 22:26 ` Scott Branden
0 siblings, 0 replies; 133+ messages in thread
From: Scott Branden @ 2021-02-18 22:26 UTC (permalink / raw)
To: Sasha Levin
Cc: Florian Fainelli, Greg Kroah-Hartman, LKML, BCM Kernel Feedback,
Willy Tarreau, Linux ARM
[-- Attachment #1.1: Type: text/plain, Size: 3963 bytes --]
On 2021-02-18 1:39 p.m., Sasha Levin wrote:
> On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
>> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
>>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
>>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
>>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>>>>>> As a company, we are most likely shooting ourselves in the foot by not
>>>>>> having a point of coordination with the Linux Foundation and key people
>>>>>> like you, Greg and other participants in the stable kernel.
>>>>>
>>>>> What does the LF have to do with this?
>>>>>
>>>>> We are here, on the mailing lists, working with everyone. Just test the
>>>>> -rc releases we make and let us know if they work or not for you, it's
>>>>> not a lot of "coordination" needed at all.
>>>>>
>>>>> Otherwise, if no one is saying that they are going to need these for 6
>>>>> years and are willing to use it in their project (i.e. and test it),
>>>>> there's no need for us to maintain it for that long, right?
>>>>
>>>> Greg, please remember I expressed I really need them for slightly more than
>>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
>>>> this saves me from having to take over these kernels after you, like in the
>>>> past, but I cannot engage on the regularity of my availability.
>>>
>>> Ok, great!
>>>
>>> That's one person/company saying they can help out (along with what CIP
>>> has been stating.)
>>>
>>> What about others? Broadcom started this conversation, odd that they
>>> don't seem to want to help out :)
>> Greg, I'm sorry but I'm not in a position to provide such a commitment.
>>
>> My original question arose because the 5.10 kernel is declared as 2 years LTS while older LTS kernels are now 6 years.
>> One problem this has created is requests to provide silicon support in an older kernel version (for a new project) rather than starting from a newer kernel version that more properly supports the (silicon and non-silicon) features.
>
> So this sounds like you have boatloads of out-of-tree code and need a
> stable kernel to avoid having to rebase that code. This is not why the
> LTS trees are around.
My question centered around the inconsistency of a 2 year LTS kernel and some of the problems
of having older LTS kernels declared as 6 years.
The discussion has gone sideways from there with many other assumptions.
There is not always control over what kernel version a customer uses,
how many hacks they have in it, when they upgrade it, what other vendor changes are in it, etc.
But, will keep this email chain to forward to others. It may help influence some decisions.
>
> For new projects, the easiest route is to upstream your stuff and ship
> the latest kernel.
>
>> If all LTS kernels were declared as 3.5-4 years as Willy commented this would solve a few issues.
>> 6 year LTS kernels would only have a maximum 1 year lifespan over the latest declared LTS kernel.
>> Also, many products take a year or more to develop, there isn't any life left in an LTS kernel if it is only 2 years.
>
> Products are supposed to upgrade their kernel. If you released something
> with, for example, a 5.4 kernel, doesn't mean you're forever stuck on a
> 5.4 kernel for that product.
>
>> After 1-3 years of kernel age the relevant parties that want to invest and care about supporting specific kernel versions longer should become apparent and could commit to longer support. Perhaps you move the burden of 6 years LTS elsewhere to longer term projects. But, I'm sure many are happy because you continue doing such a great job in a central location, especially those whose product lifespan is around 6 years.
>
> But this is exactly what's happening now: we support LTS kernels for two
> years, and after that interested parties can figure it out on their own
> if it's worth it for them to keep going.
>
[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4169 bytes --]
[-- Attachment #2: Type: text/plain, Size: 176 bytes --]
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 20:16 ` Scott Branden
@ 2021-02-19 8:25 ` Greg Kroah-Hartman
-1 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-19 8:25 UTC (permalink / raw)
To: Scott Branden
Cc: Willy Tarreau, Florian Fainelli, Sasha Levin,
BCM Kernel Feedback, LKML, Linux ARM
On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
> > On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
> >> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
> >>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
> >>>> As a company, we are most likely shooting ourselves in the foot by not
> >>>> having a point of coordination with the Linux Foundation and key people
> >>>> like you, Greg and other participants in the stable kernel.
> >>>
> >>> What does the LF have to do with this?
> >>>
> >>> We are here, on the mailing lists, working with everyone. Just test the
> >>> -rc releases we make and let us know if they work or not for you, it's
> >>> not a lot of "coordination" needed at all.
> >>>
> >>> Otherwise, if no one is saying that they are going to need these for 6
> >>> years and are willing to use it in their project (i.e. and test it),
> >>> there's no need for us to maintain it for that long, right?
> >>
> >> Greg, please remember I expressed I really need them for slightly more than
> >> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
> >> this saves me from having to take over these kernels after you, like in the
> >> past, but I cannot engage on the regularity of my availability.
> >
> > Ok, great!
> >
> > That's one person/company saying they can help out (along with what CIP
> > has been stating.)
> >
> > What about others? Broadcom started this conversation, odd that they
> > don't seem to want to help out :)
> Greg, I'm sorry but I'm not in a position to provide such a commitment.
Ok, who at Broadcom do I need to talk to to get that type of commitment?
> My original question arose because the 5.10 kernel is declared as 2 years LTS while older LTS kernels are now 6 years.
> One problem this has created is requests to provide silicon support in an older kernel version (for a new project) rather than starting from a newer kernel version that more properly supports the (silicon and non-silicon) features.
Sounds like your development model is broken, again, who do I need to
talk to in order to help you all fix this?
thanks,
greg k-h
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-19 8:25 ` Greg Kroah-Hartman
0 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-19 8:25 UTC (permalink / raw)
To: Scott Branden
Cc: Sasha Levin, Florian Fainelli, LKML, BCM Kernel Feedback,
Willy Tarreau, Linux ARM
On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
> > On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
> >> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
> >>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
> >>>> As a company, we are most likely shooting ourselves in the foot by not
> >>>> having a point of coordination with the Linux Foundation and key people
> >>>> like you, Greg and other participants in the stable kernel.
> >>>
> >>> What does the LF have to do with this?
> >>>
> >>> We are here, on the mailing lists, working with everyone. Just test the
> >>> -rc releases we make and let us know if they work or not for you, it's
> >>> not a lot of "coordination" needed at all.
> >>>
> >>> Otherwise, if no one is saying that they are going to need these for 6
> >>> years and are willing to use it in their project (i.e. and test it),
> >>> there's no need for us to maintain it for that long, right?
> >>
> >> Greg, please remember I expressed I really need them for slightly more than
> >> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
> >> this saves me from having to take over these kernels after you, like in the
> >> past, but I cannot engage on the regularity of my availability.
> >
> > Ok, great!
> >
> > That's one person/company saying they can help out (along with what CIP
> > has been stating.)
> >
> > What about others? Broadcom started this conversation, odd that they
> > don't seem to want to help out :)
> Greg, I'm sorry but I'm not in a position to provide such a commitment.
Ok, who at Broadcom do I need to talk to to get that type of commitment?
> My original question arose because the 5.10 kernel is declared as 2 years LTS while older LTS kernels are now 6 years.
> One problem this has created is requests to provide silicon support in an older kernel version (for a new project) rather than starting from a newer kernel version that more properly supports the (silicon and non-silicon) features.
Sounds like your development model is broken, again, who do I need to
talk to in order to help you all fix this?
thanks,
greg k-h
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-19 8:25 ` Greg Kroah-Hartman
@ 2021-02-19 15:05 ` Florian Fainelli
-1 siblings, 0 replies; 133+ messages in thread
From: Florian Fainelli @ 2021-02-19 15:05 UTC (permalink / raw)
To: Greg Kroah-Hartman, Scott Branden
Cc: Willy Tarreau, Florian Fainelli, Sasha Levin,
BCM Kernel Feedback, LKML, Linux ARM
On 2/19/2021 12:25 AM, Greg Kroah-Hartman wrote:
> On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
>> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
>>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
>>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
>>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>>>>>> As a company, we are most likely shooting ourselves in the foot by not
>>>>>> having a point of coordination with the Linux Foundation and key people
>>>>>> like you, Greg and other participants in the stable kernel.
>>>>>
>>>>> What does the LF have to do with this?
>>>>>
>>>>> We are here, on the mailing lists, working with everyone. Just test the
>>>>> -rc releases we make and let us know if they work or not for you, it's
>>>>> not a lot of "coordination" needed at all.
>>>>>
>>>>> Otherwise, if no one is saying that they are going to need these for 6
>>>>> years and are willing to use it in their project (i.e. and test it),
>>>>> there's no need for us to maintain it for that long, right?
>>>>
>>>> Greg, please remember I expressed I really need them for slightly more than
>>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
>>>> this saves me from having to take over these kernels after you, like in the
>>>> past, but I cannot engage on the regularity of my availability.
>>>
>>> Ok, great!
>>>
>>> That's one person/company saying they can help out (along with what CIP
>>> has been stating.)
>>>
>>> What about others? Broadcom started this conversation, odd that they
>>> don't seem to want to help out :)
>> Greg, I'm sorry but I'm not in a position to provide such a commitment.
>
> Ok, who at Broadcom do I need to talk to to get that type of commitment?
I am not sure if I was too subtle before, we (Broadcom) cannot give you
an unified voice to speak with because we are divided in silos/business
units that make their independent decisions.
The group I work in (STB/CM, different from Scott's) is committed to
using the 5.10 kernel for 6 years and that is a decision that has been
taken.
I could give you names of other decision makers in other business units
I know who also deliver Linux for their respective business units
however some of them may not make public appearances on mailing lists,
let alone care about upstreaming their changes so I do not know whether
a 6 years 5.10 kernel is even something they remotely entertain.
Hope this helps.
--
Florian
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-19 15:05 ` Florian Fainelli
0 siblings, 0 replies; 133+ messages in thread
From: Florian Fainelli @ 2021-02-19 15:05 UTC (permalink / raw)
To: Greg Kroah-Hartman, Scott Branden
Cc: Sasha Levin, Florian Fainelli, LKML, BCM Kernel Feedback,
Willy Tarreau, Linux ARM
On 2/19/2021 12:25 AM, Greg Kroah-Hartman wrote:
> On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
>> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
>>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
>>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
>>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>>>>>> As a company, we are most likely shooting ourselves in the foot by not
>>>>>> having a point of coordination with the Linux Foundation and key people
>>>>>> like you, Greg and other participants in the stable kernel.
>>>>>
>>>>> What does the LF have to do with this?
>>>>>
>>>>> We are here, on the mailing lists, working with everyone. Just test the
>>>>> -rc releases we make and let us know if they work or not for you, it's
>>>>> not a lot of "coordination" needed at all.
>>>>>
>>>>> Otherwise, if no one is saying that they are going to need these for 6
>>>>> years and are willing to use it in their project (i.e. and test it),
>>>>> there's no need for us to maintain it for that long, right?
>>>>
>>>> Greg, please remember I expressed I really need them for slightly more than
>>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
>>>> this saves me from having to take over these kernels after you, like in the
>>>> past, but I cannot engage on the regularity of my availability.
>>>
>>> Ok, great!
>>>
>>> That's one person/company saying they can help out (along with what CIP
>>> has been stating.)
>>>
>>> What about others? Broadcom started this conversation, odd that they
>>> don't seem to want to help out :)
>> Greg, I'm sorry but I'm not in a position to provide such a commitment.
>
> Ok, who at Broadcom do I need to talk to to get that type of commitment?
I am not sure if I was too subtle before, we (Broadcom) cannot give you
an unified voice to speak with because we are divided in silos/business
units that make their independent decisions.
The group I work in (STB/CM, different from Scott's) is committed to
using the 5.10 kernel for 6 years and that is a decision that has been
taken.
I could give you names of other decision makers in other business units
I know who also deliver Linux for their respective business units
however some of them may not make public appearances on mailing lists,
let alone care about upstreaming their changes so I do not know whether
a 6 years 5.10 kernel is even something they remotely entertain.
Hope this helps.
--
Florian
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-19 15:05 ` Florian Fainelli
@ 2021-02-19 15:53 ` Greg Kroah-Hartman
-1 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-19 15:53 UTC (permalink / raw)
To: Florian Fainelli
Cc: Scott Branden, Willy Tarreau, Sasha Levin, BCM Kernel Feedback,
LKML, Linux ARM
On Fri, Feb 19, 2021 at 07:05:41AM -0800, Florian Fainelli wrote:
>
>
> On 2/19/2021 12:25 AM, Greg Kroah-Hartman wrote:
> > On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
> >> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
> >>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
> >>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
> >>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
> >>>>>> As a company, we are most likely shooting ourselves in the foot by not
> >>>>>> having a point of coordination with the Linux Foundation and key people
> >>>>>> like you, Greg and other participants in the stable kernel.
> >>>>>
> >>>>> What does the LF have to do with this?
> >>>>>
> >>>>> We are here, on the mailing lists, working with everyone. Just test the
> >>>>> -rc releases we make and let us know if they work or not for you, it's
> >>>>> not a lot of "coordination" needed at all.
> >>>>>
> >>>>> Otherwise, if no one is saying that they are going to need these for 6
> >>>>> years and are willing to use it in their project (i.e. and test it),
> >>>>> there's no need for us to maintain it for that long, right?
> >>>>
> >>>> Greg, please remember I expressed I really need them for slightly more than
> >>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
> >>>> this saves me from having to take over these kernels after you, like in the
> >>>> past, but I cannot engage on the regularity of my availability.
> >>>
> >>> Ok, great!
> >>>
> >>> That's one person/company saying they can help out (along with what CIP
> >>> has been stating.)
> >>>
> >>> What about others? Broadcom started this conversation, odd that they
> >>> don't seem to want to help out :)
> >> Greg, I'm sorry but I'm not in a position to provide such a commitment.
> >
> > Ok, who at Broadcom do I need to talk to to get that type of commitment?
>
> I am not sure if I was too subtle before, we (Broadcom) cannot give you
> an unified voice to speak with because we are divided in silos/business
> units that make their independent decisions.
That's fine, I'm totally used to that, large (and even small) companies
always have different groups with different roadmaps and policies.
> The group I work in (STB/CM, different from Scott's) is committed to
> using the 5.10 kernel for 6 years and that is a decision that has been
> taken.
Great! Will you all be testing the -rc releases and letting me know how
they work for your systems?
> I could give you names of other decision makers in other business units
> I know who also deliver Linux for their respective business units
> however some of them may not make public appearances on mailing lists,
> let alone care about upstreaming their changes so I do not know whether
> a 6 years 5.10 kernel is even something they remotely entertain.
That's fine, I'm not expecting emails from the list, we can take this
off-list if you like as it sounds like I need to talk to some different
managers there, right? :)
thanks,
greg k-h
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-19 15:53 ` Greg Kroah-Hartman
0 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-19 15:53 UTC (permalink / raw)
To: Florian Fainelli
Cc: Sasha Levin, Scott Branden, LKML, BCM Kernel Feedback,
Willy Tarreau, Linux ARM
On Fri, Feb 19, 2021 at 07:05:41AM -0800, Florian Fainelli wrote:
>
>
> On 2/19/2021 12:25 AM, Greg Kroah-Hartman wrote:
> > On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
> >> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
> >>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
> >>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
> >>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
> >>>>>> As a company, we are most likely shooting ourselves in the foot by not
> >>>>>> having a point of coordination with the Linux Foundation and key people
> >>>>>> like you, Greg and other participants in the stable kernel.
> >>>>>
> >>>>> What does the LF have to do with this?
> >>>>>
> >>>>> We are here, on the mailing lists, working with everyone. Just test the
> >>>>> -rc releases we make and let us know if they work or not for you, it's
> >>>>> not a lot of "coordination" needed at all.
> >>>>>
> >>>>> Otherwise, if no one is saying that they are going to need these for 6
> >>>>> years and are willing to use it in their project (i.e. and test it),
> >>>>> there's no need for us to maintain it for that long, right?
> >>>>
> >>>> Greg, please remember I expressed I really need them for slightly more than
> >>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
> >>>> this saves me from having to take over these kernels after you, like in the
> >>>> past, but I cannot engage on the regularity of my availability.
> >>>
> >>> Ok, great!
> >>>
> >>> That's one person/company saying they can help out (along with what CIP
> >>> has been stating.)
> >>>
> >>> What about others? Broadcom started this conversation, odd that they
> >>> don't seem to want to help out :)
> >> Greg, I'm sorry but I'm not in a position to provide such a commitment.
> >
> > Ok, who at Broadcom do I need to talk to to get that type of commitment?
>
> I am not sure if I was too subtle before, we (Broadcom) cannot give you
> an unified voice to speak with because we are divided in silos/business
> units that make their independent decisions.
That's fine, I'm totally used to that, large (and even small) companies
always have different groups with different roadmaps and policies.
> The group I work in (STB/CM, different from Scott's) is committed to
> using the 5.10 kernel for 6 years and that is a decision that has been
> taken.
Great! Will you all be testing the -rc releases and letting me know how
they work for your systems?
> I could give you names of other decision makers in other business units
> I know who also deliver Linux for their respective business units
> however some of them may not make public appearances on mailing lists,
> let alone care about upstreaming their changes so I do not know whether
> a 6 years 5.10 kernel is even something they remotely entertain.
That's fine, I'm not expecting emails from the list, we can take this
off-list if you like as it sounds like I need to talk to some different
managers there, right? :)
thanks,
greg k-h
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-19 15:53 ` Greg Kroah-Hartman
@ 2021-02-19 17:44 ` Florian Fainelli
-1 siblings, 0 replies; 133+ messages in thread
From: Florian Fainelli @ 2021-02-19 17:44 UTC (permalink / raw)
To: Greg Kroah-Hartman, Florian Fainelli
Cc: Scott Branden, Willy Tarreau, Sasha Levin, BCM Kernel Feedback,
LKML, Linux ARM
On 2/19/2021 7:53 AM, Greg Kroah-Hartman wrote:
> On Fri, Feb 19, 2021 at 07:05:41AM -0800, Florian Fainelli wrote:
>>
>>
>> On 2/19/2021 12:25 AM, Greg Kroah-Hartman wrote:
>>> On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
>>>> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
>>>>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
>>>>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
>>>>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>>>>>>>> As a company, we are most likely shooting ourselves in the foot by not
>>>>>>>> having a point of coordination with the Linux Foundation and key people
>>>>>>>> like you, Greg and other participants in the stable kernel.
>>>>>>>
>>>>>>> What does the LF have to do with this?
>>>>>>>
>>>>>>> We are here, on the mailing lists, working with everyone. Just test the
>>>>>>> -rc releases we make and let us know if they work or not for you, it's
>>>>>>> not a lot of "coordination" needed at all.
>>>>>>>
>>>>>>> Otherwise, if no one is saying that they are going to need these for 6
>>>>>>> years and are willing to use it in their project (i.e. and test it),
>>>>>>> there's no need for us to maintain it for that long, right?
>>>>>>
>>>>>> Greg, please remember I expressed I really need them for slightly more than
>>>>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
>>>>>> this saves me from having to take over these kernels after you, like in the
>>>>>> past, but I cannot engage on the regularity of my availability.
>>>>>
>>>>> Ok, great!
>>>>>
>>>>> That's one person/company saying they can help out (along with what CIP
>>>>> has been stating.)
>>>>>
>>>>> What about others? Broadcom started this conversation, odd that they
>>>>> don't seem to want to help out :)
>>>> Greg, I'm sorry but I'm not in a position to provide such a commitment.
>>>
>>> Ok, who at Broadcom do I need to talk to to get that type of commitment?
>>
>> I am not sure if I was too subtle before, we (Broadcom) cannot give you
>> an unified voice to speak with because we are divided in silos/business
>> units that make their independent decisions.
>
> That's fine, I'm totally used to that, large (and even small) companies
> always have different groups with different roadmaps and policies.
>
>> The group I work in (STB/CM, different from Scott's) is committed to
>> using the 5.10 kernel for 6 years and that is a decision that has been
>> taken.
>
> Great! Will you all be testing the -rc releases and letting me know how
> they work for your systems?
Yes I will, can you add me to your CC list for the stable candidates?
That helps me not having to dig for those announcements specifically.
>
>> I could give you names of other decision makers in other business units
>> I know who also deliver Linux for their respective business units
>> however some of them may not make public appearances on mailing lists,
>> let alone care about upstreaming their changes so I do not know whether
>> a 6 years 5.10 kernel is even something they remotely entertain.
>
> That's fine, I'm not expecting emails from the list, we can take this
> off-list if you like as it sounds like I need to talk to some different
> managers there, right? :)
I will put together a list and send you an email off list.
--
Florian
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-19 17:44 ` Florian Fainelli
0 siblings, 0 replies; 133+ messages in thread
From: Florian Fainelli @ 2021-02-19 17:44 UTC (permalink / raw)
To: Greg Kroah-Hartman, Florian Fainelli
Cc: Sasha Levin, Scott Branden, LKML, BCM Kernel Feedback,
Willy Tarreau, Linux ARM
On 2/19/2021 7:53 AM, Greg Kroah-Hartman wrote:
> On Fri, Feb 19, 2021 at 07:05:41AM -0800, Florian Fainelli wrote:
>>
>>
>> On 2/19/2021 12:25 AM, Greg Kroah-Hartman wrote:
>>> On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
>>>> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
>>>>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
>>>>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
>>>>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
>>>>>>>> As a company, we are most likely shooting ourselves in the foot by not
>>>>>>>> having a point of coordination with the Linux Foundation and key people
>>>>>>>> like you, Greg and other participants in the stable kernel.
>>>>>>>
>>>>>>> What does the LF have to do with this?
>>>>>>>
>>>>>>> We are here, on the mailing lists, working with everyone. Just test the
>>>>>>> -rc releases we make and let us know if they work or not for you, it's
>>>>>>> not a lot of "coordination" needed at all.
>>>>>>>
>>>>>>> Otherwise, if no one is saying that they are going to need these for 6
>>>>>>> years and are willing to use it in their project (i.e. and test it),
>>>>>>> there's no need for us to maintain it for that long, right?
>>>>>>
>>>>>> Greg, please remember I expressed I really need them for slightly more than
>>>>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
>>>>>> this saves me from having to take over these kernels after you, like in the
>>>>>> past, but I cannot engage on the regularity of my availability.
>>>>>
>>>>> Ok, great!
>>>>>
>>>>> That's one person/company saying they can help out (along with what CIP
>>>>> has been stating.)
>>>>>
>>>>> What about others? Broadcom started this conversation, odd that they
>>>>> don't seem to want to help out :)
>>>> Greg, I'm sorry but I'm not in a position to provide such a commitment.
>>>
>>> Ok, who at Broadcom do I need to talk to to get that type of commitment?
>>
>> I am not sure if I was too subtle before, we (Broadcom) cannot give you
>> an unified voice to speak with because we are divided in silos/business
>> units that make their independent decisions.
>
> That's fine, I'm totally used to that, large (and even small) companies
> always have different groups with different roadmaps and policies.
>
>> The group I work in (STB/CM, different from Scott's) is committed to
>> using the 5.10 kernel for 6 years and that is a decision that has been
>> taken.
>
> Great! Will you all be testing the -rc releases and letting me know how
> they work for your systems?
Yes I will, can you add me to your CC list for the stable candidates?
That helps me not having to dig for those announcements specifically.
>
>> I could give you names of other decision makers in other business units
>> I know who also deliver Linux for their respective business units
>> however some of them may not make public appearances on mailing lists,
>> let alone care about upstreaming their changes so I do not know whether
>> a 6 years 5.10 kernel is even something they remotely entertain.
>
> That's fine, I'm not expecting emails from the list, we can take this
> off-list if you like as it sounds like I need to talk to some different
> managers there, right? :)
I will put together a list and send you an email off list.
--
Florian
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-19 17:44 ` Florian Fainelli
@ 2021-02-22 14:17 ` Greg Kroah-Hartman
-1 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-22 14:17 UTC (permalink / raw)
To: Florian Fainelli
Cc: Scott Branden, Willy Tarreau, Sasha Levin, BCM Kernel Feedback,
LKML, Linux ARM
On Fri, Feb 19, 2021 at 09:44:23AM -0800, Florian Fainelli wrote:
>
>
> On 2/19/2021 7:53 AM, Greg Kroah-Hartman wrote:
> > On Fri, Feb 19, 2021 at 07:05:41AM -0800, Florian Fainelli wrote:
> >>
> >>
> >> On 2/19/2021 12:25 AM, Greg Kroah-Hartman wrote:
> >>> On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
> >>>> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
> >>>>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
> >>>>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
> >>>>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
> >>>>>>>> As a company, we are most likely shooting ourselves in the foot by not
> >>>>>>>> having a point of coordination with the Linux Foundation and key people
> >>>>>>>> like you, Greg and other participants in the stable kernel.
> >>>>>>>
> >>>>>>> What does the LF have to do with this?
> >>>>>>>
> >>>>>>> We are here, on the mailing lists, working with everyone. Just test the
> >>>>>>> -rc releases we make and let us know if they work or not for you, it's
> >>>>>>> not a lot of "coordination" needed at all.
> >>>>>>>
> >>>>>>> Otherwise, if no one is saying that they are going to need these for 6
> >>>>>>> years and are willing to use it in their project (i.e. and test it),
> >>>>>>> there's no need for us to maintain it for that long, right?
> >>>>>>
> >>>>>> Greg, please remember I expressed I really need them for slightly more than
> >>>>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
> >>>>>> this saves me from having to take over these kernels after you, like in the
> >>>>>> past, but I cannot engage on the regularity of my availability.
> >>>>>
> >>>>> Ok, great!
> >>>>>
> >>>>> That's one person/company saying they can help out (along with what CIP
> >>>>> has been stating.)
> >>>>>
> >>>>> What about others? Broadcom started this conversation, odd that they
> >>>>> don't seem to want to help out :)
> >>>> Greg, I'm sorry but I'm not in a position to provide such a commitment.
> >>>
> >>> Ok, who at Broadcom do I need to talk to to get that type of commitment?
> >>
> >> I am not sure if I was too subtle before, we (Broadcom) cannot give you
> >> an unified voice to speak with because we are divided in silos/business
> >> units that make their independent decisions.
> >
> > That's fine, I'm totally used to that, large (and even small) companies
> > always have different groups with different roadmaps and policies.
> >
> >> The group I work in (STB/CM, different from Scott's) is committed to
> >> using the 5.10 kernel for 6 years and that is a decision that has been
> >> taken.
> >
> > Great! Will you all be testing the -rc releases and letting me know how
> > they work for your systems?
>
> Yes I will, can you add me to your CC list for the stable candidates?
> That helps me not having to dig for those announcements specifically.
Now added. Of course _after_ I just sent out a round of stable -rc
releases, sorry about that :(
greg k-h
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-22 14:17 ` Greg Kroah-Hartman
0 siblings, 0 replies; 133+ messages in thread
From: Greg Kroah-Hartman @ 2021-02-22 14:17 UTC (permalink / raw)
To: Florian Fainelli
Cc: Sasha Levin, Scott Branden, LKML, BCM Kernel Feedback,
Willy Tarreau, Linux ARM
On Fri, Feb 19, 2021 at 09:44:23AM -0800, Florian Fainelli wrote:
>
>
> On 2/19/2021 7:53 AM, Greg Kroah-Hartman wrote:
> > On Fri, Feb 19, 2021 at 07:05:41AM -0800, Florian Fainelli wrote:
> >>
> >>
> >> On 2/19/2021 12:25 AM, Greg Kroah-Hartman wrote:
> >>> On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote:
> >>>> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote:
> >>>>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote:
> >>>>>> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote:
> >>>>>>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote:
> >>>>>>>> As a company, we are most likely shooting ourselves in the foot by not
> >>>>>>>> having a point of coordination with the Linux Foundation and key people
> >>>>>>>> like you, Greg and other participants in the stable kernel.
> >>>>>>>
> >>>>>>> What does the LF have to do with this?
> >>>>>>>
> >>>>>>> We are here, on the mailing lists, working with everyone. Just test the
> >>>>>>> -rc releases we make and let us know if they work or not for you, it's
> >>>>>>> not a lot of "coordination" needed at all.
> >>>>>>>
> >>>>>>> Otherwise, if no one is saying that they are going to need these for 6
> >>>>>>> years and are willing to use it in their project (i.e. and test it),
> >>>>>>> there's no need for us to maintain it for that long, right?
> >>>>>>
> >>>>>> Greg, please remember I expressed I really need them for slightly more than
> >>>>>> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if
> >>>>>> this saves me from having to take over these kernels after you, like in the
> >>>>>> past, but I cannot engage on the regularity of my availability.
> >>>>>
> >>>>> Ok, great!
> >>>>>
> >>>>> That's one person/company saying they can help out (along with what CIP
> >>>>> has been stating.)
> >>>>>
> >>>>> What about others? Broadcom started this conversation, odd that they
> >>>>> don't seem to want to help out :)
> >>>> Greg, I'm sorry but I'm not in a position to provide such a commitment.
> >>>
> >>> Ok, who at Broadcom do I need to talk to to get that type of commitment?
> >>
> >> I am not sure if I was too subtle before, we (Broadcom) cannot give you
> >> an unified voice to speak with because we are divided in silos/business
> >> units that make their independent decisions.
> >
> > That's fine, I'm totally used to that, large (and even small) companies
> > always have different groups with different roadmaps and policies.
> >
> >> The group I work in (STB/CM, different from Scott's) is committed to
> >> using the 5.10 kernel for 6 years and that is a decision that has been
> >> taken.
> >
> > Great! Will you all be testing the -rc releases and letting me know how
> > they work for your systems?
>
> Yes I will, can you add me to your CC list for the stable candidates?
> That helps me not having to dig for those announcements specifically.
Now added. Of course _after_ I just sent out a round of stable -rc
releases, sorry about that :(
greg k-h
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-17 19:48 ` Scott Branden
@ 2021-02-18 17:42 ` Florian Fainelli
-1 siblings, 0 replies; 133+ messages in thread
From: Florian Fainelli @ 2021-02-18 17:42 UTC (permalink / raw)
To: Scott Branden, Greg Kroah-Hartman; +Cc: Linux ARM, LKML, BCM Kernel Feedback
On 2/17/2021 11:48 AM, Scott Branden wrote:
> Hi Greg,
>
> On 2021-02-17 1:40 a.m., Greg Kroah-Hartman wrote:
>> On Tue, Jan 26, 2021 at 07:51:18PM +0100, Greg Kroah-Hartman wrote:
>>> On Tue, Jan 26, 2021 at 10:30:16AM -0800, Scott Branden wrote:
>>>> Hi Greg,
>>>>
>>>>
>>>> On 2021-01-25 11:29 p.m., Greg Kroah-Hartman wrote:
>>>>> On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote:
>>>>>> Hi All,
>>>>>>
>>>>>> The 5.10 LTS kernel being officially LTS supported for 2 years presents a problem:
>>>>>> why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel has a 6 year LTS.
>>>>> Because they want to use all of the latest stuff that 5.10 provides
>>>>> them. Don't you want faster and more secure kernels for your devices?
>>>> Yes, 5.10 is a more secure and less buggy kernel than 5.4.
>>>
>>> Great, use it, ship it to your customers and we are all happy. What do
>>> you need me for any of this? :)
>>>
>>>>>> And AOSP has already declared the use
>>>>>> of 5.10 kernel in their Android S and T releases.
>>>>> Publically? Where? And is that really the name of the new Android
>>>>> releases, I thought they switched to numbers now (hence the naming of
>>>>> the current android-common kernel branches, marketing is fun...)
>>>> https://source.android.com/devices/architecture/kernel/android-common
>>>> Feature and launch kernels provides kernels supported per version.
>>>
>>> Oh nice, didn't know that.
>>>
>>> But note, Android kernels do not reflect the lifespan of LTS kernels.
>>> If that were the case, I would still be supporting 3.18 as they are
>>> doing that at the moment for their devices and customers, and will be
>>> doing so for I think another full year.
>>>
>>> So while Android is nice to see here, remember that is what Google is
>>> promising to support for their users. You can do the same thing for
>>> your users, what do you need me here for this? You can do the same
>>> thing that Google is doing for 3.18 right now, pick the stable fixes
>>> from upstream, backport them, test them, and push them out to their
>>> users.
>>>
>>> While Google is a great help to me in the LTS effort, providing huge
>>> amounts of resources to enable my life easier with this (i.e. funding
>>> Linaro's testing efforts), their promise to their customers/users does
>>> not depend on me keeping LTS kernels alive, if I stopped tomorrow their
>>> contracts are still in place and they know how to do this work
>>> themselves (as is proof with 3.18).
>>>
>>> So you can provide the same kind of guarantee to support any kernel
>>> version for any amount of time to any customer you like, it shouldn't
>>> require me to do that work for you, right?
>>>
>>>>>> Is there some way we could make the LTS support more clear.
>>>>>> A 2 year declaration is not LTS any more.
>>>>> Not true at all, a "normal" stable kernel is dropped after the next
>>>>> release happens, making their lifespan about 4 months long. 2 years is
>>>>> much longer than 4 months, so it still is a "long term supported" kernel
>>>>> in contrast, correct?
>>>> Perhaps a new name needs to be made for "LTS" for 6 years to distinguish it from 2 years.
>>>> The timeframes are very different.
>>>
>>> At this point in time, anyone wanting a kernel longer than 2 years
>>> should know how this all works.
>>>
>>> If not, please do some basic research, I have written whitepapers on
>>> this and given numerous talks. The information is out there...
>>>
>>>>>> If 5.10 is "actually" going to be supported for 6 years it would be quite valuable to make such a declaration.
>>>>>> https://www.kernel.org/category/releases.html
>>>>> Why? What would that change?
>>>>>
>>>>> Ok, seriously, this happens every year, and every year we go through the
>>>>> same thing, it's not like this is somehow new, right?
>>>> No, but why do we need to keep playing the same game every year now.
>>>
>>> Because, 5.4 almost did not become "6 years" of support from me. That
>>> was because in the beginning, no one said they were going to use it in
>>> their devices and offer me help in testing and backporting. Only when I
>>> knew for sure that we had people helping this out did I change the date
>>> on kernel.org.
>>>
>>> So far the jury is still out for 5.10, are you willing to help with
>>> this? If not, why are you willing to hope that others are going to do
>>> your work for you? I am talking to some companies, but am not willing
>>> to commit to anything in public just yet, because no one has committed
>>> to me yet.
>>
>> Following up on this as I did not hear back from you. Are you and/or
>> your company willing to help out with the testing of 5.10 to ensure that
>> it is a LTS kernel? So far I have not had any companies agree to help
>> out with this effort, which is sad to see as it seems that companies
>> want 6 years of stable kernels, yet do not seem to be able to at the
>> least, do a test-build/run of those kernels, which is quite odd...
> I personally cannot commit to supporting this kernel for 6 years
> (and personally do not want to backport new features to a 6 year old kernel).
> And customers are finicky and ask for one thing and then change their mind later.
> We'll have to see what decisions are made at a company level for this as there
> are added costs to run tests on LTS kernel branches. We already run extensive QA on
> whatever active development branches are in use and a subset on the mainline
> branch as well. QA resources are finite and committing those for 6 years is
> not something that makes sense if customers drop that kernel version.
> Testing of the LTS kernel changes really moves out of our hands and into the
> customer's testing after our major releases to them.
There are many different things, and I am not sure I follow you on all
counts.
If your customers are asking you for a 6 year LTS kernel, does not that
mean they themselves plan to commit to that time frame?
Assuming you are delivering a downstream kernel with all that is needed
to run on the hardware you support plus all the extra fancy stuff your
customers pay you for, what more do the customers do on top? Do they
happen to merge stable updates with your downstream tree, and why do
they do that, should not you be their upstream?
>
> Other difficulty with the LTS version is the frequency it is updated. We would not
> pickup the changes that frequently to test. A quarterly, bi-annually, or when a critical fix
> is identified would be when we update and perform any meaningful testing when in maintainence.
Well you really have the best of both worlds here, you are not forced to
update your downstream fork of the stable kernel every week, though
bonus points if you do.
For instance I try to test all the stable release candidates for 4.9,
5.4 and 5.10, and merge the stable tags every week when they show up. We
do release to our customers every 6 weeks though, so usually they will
jump several stable releases, and they are fine with that.
Yes it takes time, but I would rather do that than have to continuously
respond to questions about is this CVE fixed in kernel X.Y.Z like it
used to be before we did that. It also helps catch problem faster before
customers do, the usual benefits of continuous integration/delivery.
--
Florian
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 17:42 ` Florian Fainelli
0 siblings, 0 replies; 133+ messages in thread
From: Florian Fainelli @ 2021-02-18 17:42 UTC (permalink / raw)
To: Scott Branden, Greg Kroah-Hartman; +Cc: BCM Kernel Feedback, LKML, Linux ARM
On 2/17/2021 11:48 AM, Scott Branden wrote:
> Hi Greg,
>
> On 2021-02-17 1:40 a.m., Greg Kroah-Hartman wrote:
>> On Tue, Jan 26, 2021 at 07:51:18PM +0100, Greg Kroah-Hartman wrote:
>>> On Tue, Jan 26, 2021 at 10:30:16AM -0800, Scott Branden wrote:
>>>> Hi Greg,
>>>>
>>>>
>>>> On 2021-01-25 11:29 p.m., Greg Kroah-Hartman wrote:
>>>>> On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote:
>>>>>> Hi All,
>>>>>>
>>>>>> The 5.10 LTS kernel being officially LTS supported for 2 years presents a problem:
>>>>>> why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel has a 6 year LTS.
>>>>> Because they want to use all of the latest stuff that 5.10 provides
>>>>> them. Don't you want faster and more secure kernels for your devices?
>>>> Yes, 5.10 is a more secure and less buggy kernel than 5.4.
>>>
>>> Great, use it, ship it to your customers and we are all happy. What do
>>> you need me for any of this? :)
>>>
>>>>>> And AOSP has already declared the use
>>>>>> of 5.10 kernel in their Android S and T releases.
>>>>> Publically? Where? And is that really the name of the new Android
>>>>> releases, I thought they switched to numbers now (hence the naming of
>>>>> the current android-common kernel branches, marketing is fun...)
>>>> https://source.android.com/devices/architecture/kernel/android-common
>>>> Feature and launch kernels provides kernels supported per version.
>>>
>>> Oh nice, didn't know that.
>>>
>>> But note, Android kernels do not reflect the lifespan of LTS kernels.
>>> If that were the case, I would still be supporting 3.18 as they are
>>> doing that at the moment for their devices and customers, and will be
>>> doing so for I think another full year.
>>>
>>> So while Android is nice to see here, remember that is what Google is
>>> promising to support for their users. You can do the same thing for
>>> your users, what do you need me here for this? You can do the same
>>> thing that Google is doing for 3.18 right now, pick the stable fixes
>>> from upstream, backport them, test them, and push them out to their
>>> users.
>>>
>>> While Google is a great help to me in the LTS effort, providing huge
>>> amounts of resources to enable my life easier with this (i.e. funding
>>> Linaro's testing efforts), their promise to their customers/users does
>>> not depend on me keeping LTS kernels alive, if I stopped tomorrow their
>>> contracts are still in place and they know how to do this work
>>> themselves (as is proof with 3.18).
>>>
>>> So you can provide the same kind of guarantee to support any kernel
>>> version for any amount of time to any customer you like, it shouldn't
>>> require me to do that work for you, right?
>>>
>>>>>> Is there some way we could make the LTS support more clear.
>>>>>> A 2 year declaration is not LTS any more.
>>>>> Not true at all, a "normal" stable kernel is dropped after the next
>>>>> release happens, making their lifespan about 4 months long. 2 years is
>>>>> much longer than 4 months, so it still is a "long term supported" kernel
>>>>> in contrast, correct?
>>>> Perhaps a new name needs to be made for "LTS" for 6 years to distinguish it from 2 years.
>>>> The timeframes are very different.
>>>
>>> At this point in time, anyone wanting a kernel longer than 2 years
>>> should know how this all works.
>>>
>>> If not, please do some basic research, I have written whitepapers on
>>> this and given numerous talks. The information is out there...
>>>
>>>>>> If 5.10 is "actually" going to be supported for 6 years it would be quite valuable to make such a declaration.
>>>>>> https://www.kernel.org/category/releases.html
>>>>> Why? What would that change?
>>>>>
>>>>> Ok, seriously, this happens every year, and every year we go through the
>>>>> same thing, it's not like this is somehow new, right?
>>>> No, but why do we need to keep playing the same game every year now.
>>>
>>> Because, 5.4 almost did not become "6 years" of support from me. That
>>> was because in the beginning, no one said they were going to use it in
>>> their devices and offer me help in testing and backporting. Only when I
>>> knew for sure that we had people helping this out did I change the date
>>> on kernel.org.
>>>
>>> So far the jury is still out for 5.10, are you willing to help with
>>> this? If not, why are you willing to hope that others are going to do
>>> your work for you? I am talking to some companies, but am not willing
>>> to commit to anything in public just yet, because no one has committed
>>> to me yet.
>>
>> Following up on this as I did not hear back from you. Are you and/or
>> your company willing to help out with the testing of 5.10 to ensure that
>> it is a LTS kernel? So far I have not had any companies agree to help
>> out with this effort, which is sad to see as it seems that companies
>> want 6 years of stable kernels, yet do not seem to be able to at the
>> least, do a test-build/run of those kernels, which is quite odd...
> I personally cannot commit to supporting this kernel for 6 years
> (and personally do not want to backport new features to a 6 year old kernel).
> And customers are finicky and ask for one thing and then change their mind later.
> We'll have to see what decisions are made at a company level for this as there
> are added costs to run tests on LTS kernel branches. We already run extensive QA on
> whatever active development branches are in use and a subset on the mainline
> branch as well. QA resources are finite and committing those for 6 years is
> not something that makes sense if customers drop that kernel version.
> Testing of the LTS kernel changes really moves out of our hands and into the
> customer's testing after our major releases to them.
There are many different things, and I am not sure I follow you on all
counts.
If your customers are asking you for a 6 year LTS kernel, does not that
mean they themselves plan to commit to that time frame?
Assuming you are delivering a downstream kernel with all that is needed
to run on the hardware you support plus all the extra fancy stuff your
customers pay you for, what more do the customers do on top? Do they
happen to merge stable updates with your downstream tree, and why do
they do that, should not you be their upstream?
>
> Other difficulty with the LTS version is the frequency it is updated. We would not
> pickup the changes that frequently to test. A quarterly, bi-annually, or when a critical fix
> is identified would be when we update and perform any meaningful testing when in maintainence.
Well you really have the best of both worlds here, you are not forced to
update your downstream fork of the stable kernel every week, though
bonus points if you do.
For instance I try to test all the stable release candidates for 4.9,
5.4 and 5.10, and merge the stable tags every week when they show up. We
do release to our customers every 6 weeks though, so usually they will
jump several stable releases, and they are fine with that.
Yes it takes time, but I would rather do that than have to continuously
respond to questions about is this CVE fixed in kernel X.Y.Z like it
used to be before we did that. It also helps catch problem faster before
customers do, the usual benefits of continuous integration/delivery.
--
Florian
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
2021-02-18 17:42 ` Florian Fainelli
@ 2021-02-18 18:13 ` Willy Tarreau
-1 siblings, 0 replies; 133+ messages in thread
From: Willy Tarreau @ 2021-02-18 18:13 UTC (permalink / raw)
To: Florian Fainelli
Cc: Scott Branden, Greg Kroah-Hartman, BCM Kernel Feedback, LKML, Linux ARM
Hi Florian!
On Thu, Feb 18, 2021 at 09:42:00AM -0800, Florian Fainelli wrote:
> > Other difficulty with the LTS version is the frequency it is updated. We would not
> > pickup the changes that frequently to test. A quarterly, bi-annually, or when a critical fix
> > is identified would be when we update and perform any meaningful testing when in maintainence.
>
> Well you really have the best of both worlds here, you are not forced to
> update your downstream fork of the stable kernel every week, though
> bonus points if you do.
>
> For instance I try to test all the stable release candidates for 4.9,
> 5.4 and 5.10, and merge the stable tags every week when they show up. We
> do release to our customers every 6 weeks though, so usually they will
> jump several stable releases, and they are fine with that.
>
> Yes it takes time, but I would rather do that than have to continuously
> respond to questions about is this CVE fixed in kernel X.Y.Z like it
> used to be before we did that. It also helps catch problem faster before
> customers do, the usual benefits of continuous integration/delivery.
Totally agreed! In our use case at haproxy, it's slightly different but
not that much. We're much less sensitive to kernel bugs except for the
network parts and any long-term stability issue. However we're extremely
sensitive to openssl bugs and haproxy bugs. Thus we use them as triggers
to emit a new release and we systematically update the kernel to the
latest one. Our local patches usually apply pretty well on top of that.
We face maybe 1-3 rejects a year which take half an hour of extra manual
work, and roughly one regression every 3 years, essentially caused by
one of our local patches applying to the wrong place due to changes and
not caught by QA tests before being put online. I think that in ~15 years
(we started with 2.4), only a single customer was ever affected by a
regression caused by the kernel, it is so low we could almost laugh about
it. Quite frankly this is unrivaled, and it illustrates the huge benefit
in almost blindly following LTS this way! More quality, less work, and
faster response to critical bugs! For sure there's no "kernel hero" in
our dev team, but who really wants that anyway ?
Cheers,
Willy
^ permalink raw reply [flat|nested] 133+ messages in thread
* Re: 5.10 LTS Kernel: 2 or 6 years?
@ 2021-02-18 18:13 ` Willy Tarreau
0 siblings, 0 replies; 133+ messages in thread
From: Willy Tarreau @ 2021-02-18 18:13 UTC (permalink / raw)
To: Florian Fainelli
Cc: Greg Kroah-Hartman, BCM Kernel Feedback, LKML, Linux ARM, Scott Branden
Hi Florian!
On Thu, Feb 18, 2021 at 09:42:00AM -0800, Florian Fainelli wrote:
> > Other difficulty with the LTS version is the frequency it is updated. We would not
> > pickup the changes that frequently to test. A quarterly, bi-annually, or when a critical fix
> > is identified would be when we update and perform any meaningful testing when in maintainence.
>
> Well you really have the best of both worlds here, you are not forced to
> update your downstream fork of the stable kernel every week, though
> bonus points if you do.
>
> For instance I try to test all the stable release candidates for 4.9,
> 5.4 and 5.10, and merge the stable tags every week when they show up. We
> do release to our customers every 6 weeks though, so usually they will
> jump several stable releases, and they are fine with that.
>
> Yes it takes time, but I would rather do that than have to continuously
> respond to questions about is this CVE fixed in kernel X.Y.Z like it
> used to be before we did that. It also helps catch problem faster before
> customers do, the usual benefits of continuous integration/delivery.
Totally agreed! In our use case at haproxy, it's slightly different but
not that much. We're much less sensitive to kernel bugs except for the
network parts and any long-term stability issue. However we're extremely
sensitive to openssl bugs and haproxy bugs. Thus we use them as triggers
to emit a new release and we systematically update the kernel to the
latest one. Our local patches usually apply pretty well on top of that.
We face maybe 1-3 rejects a year which take half an hour of extra manual
work, and roughly one regression every 3 years, essentially caused by
one of our local patches applying to the wrong place due to changes and
not caught by QA tests before being put online. I think that in ~15 years
(we started with 2.4), only a single customer was ever affected by a
regression caused by the kernel, it is so low we could almost laugh about
it. Quite frankly this is unrivaled, and it illustrates the huge benefit
in almost blindly following LTS this way! More quality, less work, and
faster response to critical bugs! For sure there's no "kernel hero" in
our dev team, but who really wants that anyway ?
Cheers,
Willy
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 133+ messages in thread