Re: Regressions in stable releases

[Willy Tarreau <w@1wt.eu>]

Hi Sasha,

On Fri, Aug 06, 2021 at 10:37:47AM -0400, Sasha Levin wrote:
> > Then in my opinion we should encourage *not* to use "Fixes:" on untested
> > patches (untested patches will always happen due to hardware availability
> > or lack of a reliable reproducer).
> > 
> > What about this to try to improve the situation in this specific case ?
> 
> No, please let's not. If there is no testing story behind a buggy patch
> then it'll explode either when we go to the next version, or when we
> pull it into -stable.
> 
> If we avoid taking groups of patches into -stable it'll just mean that
> we end up with a huge amount of issues waiting for us during a version
> upgrade.

I agree with this and that was the point I was explaining as well: someone
has to detect those bugs, and unfortunately if they're so hard to see that
they can't be detected before the users, it has to hit a user.

> Yes, we may be taking bugs in now, but the regression rate (according to
> LWN) is pretty low, and the somewhat linear distribution of those bugs
> throughout our releases makes them managable (to review when they're
> sent out, to find during testing, to bisect if we hit the bug).

I totally agree that they're extremely low. I only faced one in production
in 4 or 5 years now, and even then it was not a true one, it was caused by
a context change that made one local patch silently apply at the wrong place.

> As Guenter points out, the path forward should be to improve our testing
> story.

Yes but we also have to make people understand that it only improves the
situation a little bit and doesn't magically result in zero regression.
Most whiners seem to say "I've met a regression once, that's unacceptable".
This will not change their experience, it will just reduce the number of
whiners who complain about their first bug ever. The amount of effort to
invest in testing to reduce regressions by just 1% can be huge, and at
some point one has to wonder where resources are better assigned.

Again, I tend to think that releasing older releases less often (and with
more patches each time) could reassure some users. It used to happen in
the past when Paul, Ben and I were in charge of older & slower branches,
and it sometimes allowed us to drop one patch and its subsequent revert
from a series. That's still one regression saved whenever it happens.
And this maintains the principle of "older==extremely stable with slower
fixes, newer==very stable with faster fixes".

Cheers,
Willy