Suggestion: Lengthen the review period for stable releases from 48 hours to 7 days.

Suggestion: Lengthen the review period for stable releases from 48 hours to 7 days.

[Hussam Al-Tayeb]

Hello. I would like to suggest lengthening the review period for stable
releases from 48 hours to 7 days.
The rationale is that 48 hours is not enough for people to test those
stable releases and make sure there are no regressions for particular
workflows.
This is especially important for companies deploying those kernels in
production machines. Often those releases are on weekends as well
further limiting the ability to test.
It is of course possible to skip stable updates that have large numbers
of patches and only update once a month but I feel a longer testing
period will work best for everyone.
It is, of course, always possible to exempt urgent security releases
from the waiting period.

Thank you.
Hussam.

Re: Suggestion: Lengthen the review period for stable releases from 48 hours to 7 days.

[Hussam Al-Tayeb]

On Sat, 2020-11-14 at 17:40 +0200, Hussam Al-Tayeb wrote:
> Hello. I would like to suggest lengthening the review period for
> stable
> releases from 48 hours to 7 days.
> The rationale is that 48 hours is not enough for people to test those
> stable releases and make sure there are no regressions for particular
> workflows.
> This is especially important for companies deploying those kernels in
> production machines. Often those releases are on weekends as well
> further limiting the ability to test.
> It is of course possible to skip stable updates that have large
> numbers
> of patches and only update once a month but I feel a longer testing
> period will work best for everyone.
> It is, of course, always possible to exempt urgent security releases
> from the waiting period.
>
> Thank you.
> Hussam.
>

One more note. This is not a random internet rant. I actually do need
more than 48 hours to test new stable kernels and I believe others do
too.
Regards,
Hussam.

Re: Suggestion: Lengthen the review period for stable releases from 48 hours to 7 days.

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 601 bytes --]

On Sat 2020-11-14 17:40:36, Hussam Al-Tayeb wrote:
> Hello. I would like to suggest lengthening the review period for stable
> releases from 48 hours to 7 days.
> The rationale is that 48 hours is not enough for people to test those
> stable releases and make sure there are no regressions for particular
> workflows.

You should probably cc stable list and Greg with this.

And yes, I believe that would be good idea.

Plus the period is very often shorter than advertised, which might be
also good to fix.

Best regards,
								pavel

-- 
http://www.livejournal.com/~pavelmachek

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Suggestion: Lengthen the review period for stable releases from 48 hours to 7 days.

[Hussam Al-Tayeb]

Please CC me in any replies as I am not subscribed to the list.
This is a legitimate request as I often need more than two days
especially on busy work days or weekends.
On Tue, 2020-11-17 at 09:01 +0100, Pavel Machek wrote:
> On Sat 2020-11-14 17:40:36, Hussam Al-Tayeb wrote:
> > Hello. I would like to suggest lengthening the review period for
> > stable
> > releases from 48 hours to 7 days.
> > The rationale is that 48 hours is not enough for people to test
> > those
> > stable releases and make sure there are no regressions for
> > particular
> > workflows.
>
> You should probably cc stable list and Greg with this.
>
> And yes, I believe that would be good idea.
>
> Plus the period is very often shorter than advertised, which might be
> also good to fix.
>
> Best regards,
>                                                                 pavel
>

Re: Suggestion: Lengthen the review period for stable releases from 48 hours to 7 days.

[Christoph Biedl]

[-- Attachment #1: Type: text/plain, Size: 1602 bytes --]

>> On Sat 2020-11-14 17:40:36, Hussam Al-Tayeb wrote:

>>> Hello. I would like to suggest lengthening the review period for stable
>>> releases from 48 hours to 7 days.
>>> The rationale is that 48 hours is not enough for people to test those
>>> stable releases and make sure there are no regressions for
>>> particular workflows.

Disclaimer: I am mostly a user of stable

It's hard to make a good decision here. I share your position the 48-ish
hours are a fairly short amound of time, and increasing it would grant
more time for tests. As for me, I might resume testing -rc on a regular
base as I used to in the past - which is a time-consuming procedure, and
since I do that as a hobby, sometimes more important things are in the
way. But I have to concede the number of issues that occured only here
was never high, and I don't expect it would grow significantly.

On the other hand the pace of the stable patches became fairly high¹, so
during a week of -rc review a *lot* of them will queue up and I predict
we'll see requests for fast-laning some of them. Also, a release would
immediately be followed by the next -rc review period, a procedure that
gives me a bad feeling.

So for me, I'd appreciate an extension of the review period, even if
it's just four days. But I understand if people prefer to keep the
procedures simple, and get fixes out of the door as soon as possible.

My 2¢

    Christoph

¹ If somebody made statistics on the development of the number of
  patches for stable kernels (in count/second), I'd be curious to see
  the numbers.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Suggestion: Lengthen the review period for stable releases from 48 hours to 7 days.

[Greg KH]

On Tue, Nov 17, 2020 at 11:29:16PM +0100, Christoph Biedl wrote:
> >> On Sat 2020-11-14 17:40:36, Hussam Al-Tayeb wrote:
> 
> >>> Hello. I would like to suggest lengthening the review period for stable
> >>> releases from 48 hours to 7 days.
> >>> The rationale is that 48 hours is not enough for people to test those
> >>> stable releases and make sure there are no regressions for
> >>> particular workflows.
> 
> Disclaimer: I am mostly a user of stable
> 
> It's hard to make a good decision here. I share your position the 48-ish
> hours are a fairly short amound of time, and increasing it would grant
> more time for tests. As for me, I might resume testing -rc on a regular
> base as I used to in the past - which is a time-consuming procedure, and
> since I do that as a hobby, sometimes more important things are in the
> way. But I have to concede the number of issues that occured only here
> was never high, and I don't expect it would grow significantly.
> 
> On the other hand the pace of the stable patches became fairly high¹, so
> during a week of -rc review a *lot* of them will queue up and I predict
> we'll see requests for fast-laning some of them. Also, a release would
> immediately be followed by the next -rc review period, a procedure that
> gives me a bad feeling.
> 
> So for me, I'd appreciate an extension of the review period, even if
> it's just four days. But I understand if people prefer to keep the
> procedures simple, and get fixes out of the door as soon as possible.

That's the thing, these releases almost always contain fixes that we
know people are having in the real world, or they fix reported security
issues, so we need to get them out to everyone as soon as possible.

If you are only a week or so behind (because your testing framework
takes a week), that's fine, let us know if we broke something last week
and we will be glad to revert it or find the fixup patch for it that is
in Linus's tree.  I work with almost all of the major SoC vendors and
they get back to me on this type of delayed cycle because their tests do
take longer, and it works fine.

Also, note that I do a release when the testers that I have come to rely
on tell me that all is good.  I think they are running some 50k+ tests
on each release at the moment, so while quantity isn't a substitute for
quality, it is a good indication that nothing regressed here which is
what I am looking for.  Recently these tests are coming back sooner than
2 days, which is great and why I do a release quicker at times
(sometimes it is because of just logistics reasons).

So slowing down releases is not the answer.  Getting back to me when you
have issues is the solution.  1-2-4 weeks is fine, just let us know if
you have regressions when you find them, otherwise we don't know that
there is an issue that needs to be resolved.

> ¹ If somebody made statistics on the development of the number of
>   patches for stable kernels (in count/second), I'd be curious to see
>   the numbers.

Yes, I have those numbers, we run about 30-35 patches/day in the stable
releases at the moment.  You can see this in the spreadsheet I keep at:
	https://github.com/gregkh/kernel-history
in the kernel_stats.ods file, look at the tabs on the bottom to see the
rate of change for the different stable releases.  It usually is a few
releases old, but I try to update it monthly.

thanks,

greg k-h

Re: Suggestion: Lengthen the review period for stable releases from 48 hours to 7 days.

[Sasha Levin]

On Tue, Nov 17, 2020 at 11:29:16PM +0100, Christoph Biedl wrote:
>On the other hand the pace of the stable patches became fairly high¹, so
>during a week of -rc review a *lot* of them will queue up and I predict
>we'll see requests for fast-laning some of them. Also, a release would
>immediately be followed by the next -rc review period, a procedure that
>gives me a bad feeling.

Keep in mind that the stable tree derives itself from Linus's tree -
it's not a development tree on it's own and we don't control how many
fixes flow into Linus's tree (and as a result into the stable tree).

This means that it doesn't matter how long the review window is open
for, you'll be getting the same time to review a single patch - whether
we do 200 patches twice a week or 400 patches once a week. We can't
create time by moving review windows around.

-- 
Thanks,
Sasha

Re: Suggestion: Lengthen the review period for stable releases from 48 hours to 7 days.

[Hussam Al-Tayeb]

On Wed, 2020-11-18 at 09:09 -0500, Sasha Levin wrote:
> On Tue, Nov 17, 2020 at 11:29:16PM +0100, Christoph Biedl wrote:
> > On the other hand the pace of the stable patches became fairly
> > high¹, so
> > during a week of -rc review a *lot* of them will queue up and I
> > predict
> > we'll see requests for fast-laning some of them. Also, a release
> > would
> > immediately be followed by the next -rc review period, a procedure
> > that
> > gives me a bad feeling.
>
> Keep in mind that the stable tree derives itself from Linus's tree -
> it's not a development tree on it's own and we don't control how many
> fixes flow into Linus's tree (and as a result into the stable tree).
>
> This means that it doesn't matter how long the review window is open
> for, you'll be getting the same time to review a single patch -
> whether
> we do 200 patches twice a week or 400 patches once a week. We can't
> create time by moving review windows around.
>

How long does it take for patches reaching Linux's tree to propagate
down to the stable trees and is there is mechanism for identifying
followup patches?
For instance, patch A fixes bug X but we eventually find out that this
patch did not fix all occurrences of the bug or caused a regression and
hence the author immediately sent patch B for inclusion in mainline
(Linux's tree).
Is patch B automatically identified for inclusion in stable as well?

In short, is there a guarantee that stable trees are as stable or
better than mainline through the current SOP?

Regards,
Hussam.

Re: Suggestion: Lengthen the review period for stable releases from 48 hours to 7 days.

[Greg KH]

On Wed, Nov 18, 2020 at 08:02:16PM +0200, Hussam Al-Tayeb wrote:
> On Wed, 2020-11-18 at 09:09 -0500, Sasha Levin wrote:
> > On Tue, Nov 17, 2020 at 11:29:16PM +0100, Christoph Biedl wrote:
> > > On the other hand the pace of the stable patches became fairly
> > > high¹, so
> > > during a week of -rc review a *lot* of them will queue up and I
> > > predict
> > > we'll see requests for fast-laning some of them. Also, a release
> > > would
> > > immediately be followed by the next -rc review period, a procedure
> > > that
> > > gives me a bad feeling.
> >
> > Keep in mind that the stable tree derives itself from Linus's tree -
> > it's not a development tree on it's own and we don't control how many
> > fixes flow into Linus's tree (and as a result into the stable tree).
> >
> > This means that it doesn't matter how long the review window is open
> > for, you'll be getting the same time to review a single patch -
> > whether
> > we do 200 patches twice a week or 400 patches once a week. We can't
> > create time by moving review windows around.
> >
> 
> How long does it take for patches reaching Linux's tree to propagate
> down to the stable trees

It depends from a week, to a day, sometimes if it's important, an hour,
and sometimes months.

> and is there is mechanism for identifying
> followup patches?

Yes, happens all the time, don't you see this?  If we have missed any
fixes for fixes, please let us know, but our tools usually catch these
pretty well these days.

> In short, is there a guarantee that stable trees are as stable or
> better than mainline through the current SOP?

There's no guarantees in life, especially for free software :)

thanks,

greg k-h