Re: [LSF/MM TOPIC] FS, MM, and stable trees

From: Dave Chinner <david@fromorbit.com>
To: Amir Goldstein <amir73il@gmail.com>
Cc: Sasha Levin <sashal@kernel.org>,
	lsf-pc <lsf-pc@lists.linux-foundation.org>,
	linux-fsdevel <linux-fsdevel@vger.kernel.org>,
	Jan Kara <jack@suse.cz>, Theodore Tso <tytso@mit.edu>,
	"Darrick J. Wong" <darrick.wong@oracle.com>,
	Josef Bacik <josef@toxicpanda.com>,
	"Luis R. Rodriguez" <mcgrof@kernel.org>,
	Matthew Wilcox <willy@infradead.org>,
	Greg KH <gregkh@linuxfoundation.org>
Subject: Re: [LSF/MM TOPIC] FS, MM, and stable trees
Date: Wed, 9 Mar 2022 11:02:05 +1100	[thread overview]
Message-ID: <20220309000205.GV3927073@dread.disaster.area> (raw)
In-Reply-To: <CAOQ4uxjysufPUtwepPGNZDhoC_HdsnkHx7--kso_OXWPyPkw_A@mail.gmail.com>

On Tue, Mar 08, 2022 at 11:32:43AM +0200, Amir Goldstein wrote:
> On Tue, Feb 12, 2019 at 7:31 PM Sasha Levin <sashal@kernel.org> wrote:
> >
> > Hi all,
> >
> > I'd like to propose a discussion about the workflow of the stable trees
> > when it comes to fs/ and mm/. In the past year we had some friction with
> > regards to the policies and the procedures around picking patches for
> > stable tree, and I feel it would be very useful to establish better flow
> > with the folks who might be attending LSF/MM.
> >
> > I feel that fs/ and mm/ are in very different places with regards to
> > which patches go in -stable, what tests are expected, and the timeline
> > of patches from the point they are proposed on a mailing list to the
> > point they are released in a stable tree. Therefore, I'd like to propose
> > two different sessions on this (one for fs/ and one for mm/), as a
> > common session might be less conductive to agreeing on a path forward as
> > the starting point for both subsystems are somewhat different.
> >
> > We can go through the existing processes, automation, and testing
> > mechanisms we employ when building stable trees, and see how we can
> > improve these to address the concerns of fs/ and mm/ folks.
> >
> 
> Hi Sasha,
> 
> I think it would be interesting to have another discussion on the state of fs/
> in -stable and see if things have changed over the past couple of years.
> If you do not plan to attend LSF/MM in person, perhaps you will be able to
> join this discussion remotely?
> 
> From what I can see, the flow of ext4/btrfs patches into -stable still looks
> a lot healthier than the flow of xfs patches into -stable.
> 
> In 2019, Luis started an effort to improve this situation (with some
> assistance from me and you) that ended up with several submissions
> of stable patches for v4.19.y, but did not continue beyond 2019.
> 
> When one looks at xfstest bug reports on the list for xfs on kernels > v4.19
> one has to wonder if using xfs on kernels v5.x.y is a wise choice.
> 
> Which makes me wonder: how do the distro kernel maintainers keep up
> with xfs fixes?

For RHEL, we actively backport whole upstream releases with a few
cycle's delay. That means, for example, A RHEL 8 kernel might have a
5.10 XFS + random critical fixes from 5.11-16 in it. We monitor for
relevant "Fixes" tags, manage all the QE of those backports
ourselves, handle all the regressions and end user bug reports
ourselves, etc. 

There is almost zero impact on upstream from the RHEL stable
kernel process - they only intersect when a bug that also affects
upstream kernels is found. At which point, the "upstream first"
policy kicks in, and then we backport the upstream fix to the RHEL
stable kernels that need it as per eveything else that is done.

IOWs, there's a whole team of ppl at RH across FS, QE and SE who are
pretty much entirely dedicated to enabling, testing and supporting
the RHEL backports. This work is largely invisible to upstream
development and developers, except for the fact we tag bug fixes
with "fixes" tags so that distro kernel maintainers know to consider
that they need to consider backporting them sooner rather than later.

Keep in mind that an LTS kernel is no different to a SLES or RHEL
kernel in terms of the number or significance of changes it
accumulates over it's life time. However, those LTS kernels they
don't have anywhere near the same level of quality control as even
.0 upstream releases, nor do LTS kernels have a dedicated QE or
support organisations that maintaining and supporting a reliable
product that has millions of end users really requires.

It sounds like there are some things the LTS maintainers have
underway that substantially change the QE equation for LTS kernels.
WE've been asking for that for a long time with limited short term
success (e.g. Luis' effort), so I'm hopeful that things coming down
the pipeline will create a sustainable long term solution that will
enable us to have confidence that LTS backports (automated or
manual) are robust and regression free.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com