LKML Archive on lore.kernel.org
 help / color / Atom feed
From: Willy Tarreau <w@1wt.eu>
To: Florian Fainelli <f.fainelli@gmail.com>
Cc: Scott Branden <scott.branden@broadcom.com>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	BCM Kernel Feedback <bcm-kernel-feedback-list@broadcom.com>,
	LKML <linux-kernel@vger.kernel.org>,
	Linux ARM <linux-arm-kernel@lists.infradead.org>
Subject: Re: 5.10 LTS Kernel: 2 or 6 years?
Date: Thu, 18 Feb 2021 19:13:30 +0100
Message-ID: <20210218181330.GA15217@1wt.eu> (raw)
In-Reply-To: <f2a03516-0247-522e-184a-7df4840ed345@gmail.com>

Hi Florian!

On Thu, Feb 18, 2021 at 09:42:00AM -0800, Florian Fainelli wrote:
> > Other difficulty with the LTS version is the frequency it is updated.  We would not
> > pickup the changes that frequently to test.  A quarterly, bi-annually, or when a critical fix
> > is identified would be when we update and perform any meaningful testing when in maintainence.
> 
> Well you really have the best of both worlds here, you are not forced to
> update your downstream fork of the stable kernel every week, though
> bonus points if you do.
> 
> For instance I try to test all the stable release candidates for 4.9,
> 5.4 and 5.10, and merge the stable tags every week when they show up. We
> do release to our customers every 6 weeks though, so usually they will
> jump several stable releases, and they are fine with that.
> 
> Yes it takes time, but I would rather do that than have to continuously
> respond to questions about is this CVE fixed in kernel X.Y.Z like it
> used to be before we did that. It also helps catch problem faster before
> customers do, the usual benefits of continuous integration/delivery.

Totally agreed! In our use case at haproxy, it's slightly different but
not that much. We're much less sensitive to kernel bugs except for the
network parts and any long-term stability issue. However we're extremely
sensitive to openssl bugs and haproxy bugs. Thus we use them as triggers
to emit a new release and we systematically update the kernel to the
latest one. Our local patches usually apply pretty well on top of that.

We face maybe 1-3 rejects a year which take half an hour of extra manual
work, and roughly one regression every 3 years, essentially caused by
one of our local patches applying to the wrong place due to changes and
not caught by QA tests before being put online. I think that in ~15 years
(we started with 2.4), only a single customer was ever affected by a
regression caused by the kernel, it is so low we could almost laugh about
it. Quite frankly this is unrivaled, and it illustrates the huge benefit
in almost blindly following LTS this way! More quality, less work, and
faster response to critical bugs! For sure there's no "kernel hero" in
our dev team, but who really wants that anyway ?

Cheers,
Willy

  reply index

Thread overview: 65+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-01-25 19:55 Scott Branden
2021-01-26  2:50 ` Adam Borowski
2021-01-26  7:29 ` Greg Kroah-Hartman
2021-01-26 17:35   ` Florian Fainelli
2021-01-26 18:30   ` Scott Branden
2021-01-26 18:51     ` Greg Kroah-Hartman
2021-01-26 20:15       ` Willy Tarreau
2021-02-17  9:40       ` Greg Kroah-Hartman
2021-02-17 19:48         ` Scott Branden
2021-02-18  7:43           ` Greg Kroah-Hartman
2021-02-18 11:31             ` Willy Tarreau
2021-02-18 14:15               ` Jari Ruusu
2021-02-18 14:29                 ` Greg Kroah-Hartman
2021-02-18 20:55                   ` Pavel Machek
2021-02-18 22:43                     ` Ondrej Zary
2021-02-19  8:00                       ` Pavel Machek
2021-02-19  8:30                         ` Greg Kroah-Hartman
2021-02-18 14:33                 ` Willy Tarreau
2021-02-18 17:19                   ` Jari Ruusu
2021-02-18 17:22                     ` Russell King - ARM Linux admin
2021-02-18 17:44                     ` Greg Kroah-Hartman
2021-02-19  7:10                       ` Jari Ruusu
2021-02-19  8:22                         ` Greg Kroah-Hartman
2021-02-19 10:31                           ` Jari Ruusu
2021-02-19 10:37                             ` Greg Kroah-Hartman
2021-02-19 10:57                               ` Jari Ruusu
2021-02-19 11:16                                 ` Greg Kroah-Hartman
2021-02-19 15:23                                   ` Jari Ruusu
2021-02-20 13:29                                     ` Jari Ruusu
2021-02-20 16:05                                       ` Greg Kroah-Hartman
2021-02-20 17:06                                         ` Willy Tarreau
2021-02-21 11:38                                         ` Jari Ruusu
2021-02-19 16:50                                   ` Theodore Ts'o
2021-02-18 17:16               ` Florian Fainelli
2021-02-18 12:51             ` Pavel Machek
2021-02-18 16:51           ` Sasha Levin
2021-02-18 17:21             ` Florian Fainelli
2021-02-18 17:53               ` Greg Kroah-Hartman
2021-02-18 17:57                 ` Florian Fainelli
2021-02-18 18:20                 ` Willy Tarreau
2021-02-18 18:36                   ` Greg Kroah-Hartman
2021-02-18 20:16                     ` Scott Branden
2021-02-18 21:00                       ` Willy Tarreau
2021-02-18 22:38                         ` Scott Branden
2021-02-18 21:39                       ` Sasha Levin
2021-02-18 22:00                         ` Florian Fainelli
2021-02-18 22:26                         ` Scott Branden
2021-02-19  8:25                       ` Greg Kroah-Hartman
2021-02-19 15:05                         ` Florian Fainelli
2021-02-19 15:53                           ` Greg Kroah-Hartman
2021-02-19 17:44                             ` Florian Fainelli
2021-02-22 14:17                               ` Greg Kroah-Hartman
2021-02-18 17:42           ` Florian Fainelli
2021-02-18 18:13             ` Willy Tarreau [this message]
2021-02-18 10:04         ` Pavel Machek
2021-02-19  8:54   ` Hanjun Guo
2021-02-19  9:08     ` Greg Kroah-Hartman
2021-02-20  7:02       ` Hanjun Guo
2021-02-20  9:53         ` Greg Kroah-Hartman
2021-02-23  2:14           ` Hanjun Guo
2021-02-19 14:45     ` Nikolai Kondrashov
2021-02-26  8:03       ` Hanjun Guo
2021-02-26 11:21         ` Nikolai Kondrashov
2021-02-22 14:00   ` Nishanth Aravamudan
2021-02-22 14:24     ` Greg Kroah-Hartman

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210218181330.GA15217@1wt.eu \
    --to=w@1wt.eu \
    --cc=bcm-kernel-feedback-list@broadcom.com \
    --cc=f.fainelli@gmail.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=scott.branden@broadcom.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git
	git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git
	git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git
	git clone --mirror https://lore.kernel.org/lkml/9 lkml/git/9.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org
	public-inbox-index lkml

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git