XFS LTS backport cabal

XFS LTS backport cabal

[Darrick J. Wong]

Hi everyone,

As most of the people cc'd on this message are aware, there's been a
recent groundswell of interest in increasing the amount of staff time
assigned to backporting bug fixes to LTS kernels.  As part of preparing
to resume maintainership on June 5th, I thought it would be a good idea
to introduce all the participants (that I know of!) and to capture a
summary of everyone's thoughts w.r.t. how to make stable backports
happen.

First, some introductions: Leah and Ted work at Google, and they've
expressed interest in 5.15 LTS backports.  Dave and Eric are the other
upstream XFS maintainers, so I've included them.  Amir seems to be
working on 5.10 LTS backports for his employer(?).  Chandan and I work
at Oracle (obviously) and he's also been working on a few 5.15 LTS
backports.

I won't speak for other organizations, but we (Oracle) are also
interested in stable backports for the 5.4 and 4.14 LTS kernels, since
we have customers running <cough> derivatives of those kernels.  Given
what I've heard from others, many kernel distributors lean on the LTS
kernels.

The goal of this thread, then, is to shed some light on who's currently
doing what to reduce duplication of LTS work, and to make sure that
we're all more or less on the same page with regards to what we will and
won't try to push to stable.  (A side goal of mine is to help everyone
working on the stable branches to avoid the wrath and unhelpful form
letters of the stable maintainers.)

Briefly, I think the patches that flow into XFS could be put into three
rough categories:

(a) Straightforward fixes.  These are usually pretty simple fixes (e.g.
omitted errno checking, insufficient validation, etc.) sometimes get
proper Fixes tags, which means that AUTOSEL can be of some benefit.

(b) Probable fixes.  Often these aren't all that obvious -- for example,
the author may be convinced that they correct a mis-interaction between
subsystems, but we would like the changes to soak in upstream for a few
months to build confidence that they solve the problem and without
causing more problems.

(c) Everything else.  New features, giant refactorings, etc.  These
generally should not be backported, unless someone has a /really/ good
reason.

Here are a few principles I'd like to see guiding stable backport
efforts:

1. AUTOSEL is a good tool to _start_ the process of identifying low
hanging fruit to backport.  Automation is our friend, but XFS is complex
so we still need people who have kept up with linux-xfs to know what's
appropriate (and what compile tests can't find) to finish the process.

2. Some other tag for patches that could be a fix, but need a few months
to soak.  This is targetted at (b), since I'm terrible at remembering
that there are patches that are reaching ripeness.

3. fstesting -- new patches proposed for stable branches shouldn't
introduce new regressions, and ideally there would also be a regression
test that would now pass.  As Dave and I have stated in the past,
fstests is a big umbrella of a test suite, which implies that A/B
testing is the way to go.  I think at least Zorro and I would like to
improve the tagging in fstests to make it more obvious which tests
contain enough randomness that they cannot be expected to behave 100%
reliably.

Here's a couple of antipatterns from the past:

i. Robots shovelling patches into stable kernels with no testing.

ii. Massively large backports.  New features don't go to stable kernels,
and I doubt the stable kernel maintainers will accept that anyway.  I
grok the temptation to backport more so that it's easier to land future
fixes via AUTOSEL, but I personally wouldn't endorse frontloading a
bunch of work to chase a promise of less future work.

And a question or two:

a> I've been following the recent fstests threads, and it seems to me
that there are really two classes of users -- sustaining people who want
fstests to run reliably so they can tell if their backports have broken
anything; and developers, who want the randomness to try to poke into
dusty corners of the filesystem.  Can we make it easier to associate
random bits of data (reliability rates, etc.) with a given fstests
configuration?  And create a test group^Wtag for the tests that rely on
RNGs to shake things up?

b> Testing relies very heavily on being able to spin up a lot of testing
resources.  Can/should we make it easier for people with a kernel.org
account to get free(ish) cloud accounts with the LF members who are also
cloud vendors?

Thoughts? Flames?

--D

Re: XFS LTS backport cabal

[Amir Goldstein]

On Thu, May 26, 2022 at 12:23 AM Darrick J. Wong <djwong@kernel.org> wrote:
>
> Hi everyone,
>

Hi Darrick!

Thanks for making this introduction.

I've added Tyler (Microsoft) for his interest in 5.10 LTS backports
and Luis (Samsung) who has worked with the stable maintainers on xfs
backports in the past and is collaborating with me on 5.10 LTS testing
these days.

> As most of the people cc'd on this message are aware, there's been a
> recent groundswell of interest in increasing the amount of staff time
> assigned to backporting bug fixes to LTS kernels.  As part of preparing
> to resume maintainership on June 5th, I thought it would be a good idea
> to introduce all the participants (that I know of!) and to capture a
> summary of everyone's thoughts w.r.t. how to make stable backports
> happen.
>
> First, some introductions: Leah and Ted work at Google, and they've
> expressed interest in 5.15 LTS backports.  Dave and Eric are the other
> upstream XFS maintainers, so I've included them.  Amir seems to be
> working on 5.10 LTS backports for his employer(?).

Correct, my goals are twofold:

1. Consolidate duplicate efforts on stable kernel -
For CTERA products that are shippened as a VM image, I do not *need*
my backports to CTERA kernel to go to LTS. I *want* them to go to LTS.
As a community of developers who work for different companies,
we work so well together on upstream and so poorly apart on the
stable kernels that we each maintain.
I would like to see that change.

2. For CTERA products that run in containers in the cloud, we rely on the flow
of fixes to LTS kernels that will someday flow into the container host OS
of the cloud provider. Unless I start sending patches to cloud
providers directly,
LTS is my only path to get things fixed in the future.
For this use case, we do not even have the luxury of assuming a recent LTS
kernel. The major cloud providers released host OS images based on 5.10
only late 2021.

> Chandan and I work
> at Oracle (obviously) and he's also been working on a few 5.15 LTS
> backports.
>
> I won't speak for other organizations, but we (Oracle) are also
> interested in stable backports for the 5.4 and 4.14 LTS kernels, since
> we have customers running <cough> derivatives of those kernels.  Given
> what I've heard from others, many kernel distributors lean on the LTS
> kernels.
>
> The goal of this thread, then, is to shed some light on who's currently
> doing what to reduce duplication of LTS work, and to make sure that
> we're all more or less on the same page with regards to what we will and
> won't try to push to stable.  (A side goal of mine is to help everyone
> working on the stable branches to avoid the wrath and unhelpful form
> letters of the stable maintainers.)
>
> Briefly, I think the patches that flow into XFS could be put into three
> rough categories:
>
> (a) Straightforward fixes.  These are usually pretty simple fixes (e.g.
> omitted errno checking, insufficient validation, etc.) sometimes get
> proper Fixes tags, which means that AUTOSEL can be of some benefit.
>
> (b) Probable fixes.  Often these aren't all that obvious -- for example,
> the author may be convinced that they correct a mis-interaction between
> subsystems, but we would like the changes to soak in upstream for a few
> months to build confidence that they solve the problem and without
> causing more problems.
>
> (c) Everything else.  New features, giant refactorings, etc.  These
> generally should not be backported, unless someone has a /really/ good
> reason.
>
> Here are a few principles I'd like to see guiding stable backport
> efforts:
>
> 1. AUTOSEL is a good tool to _start_ the process of identifying low
> hanging fruit to backport.  Automation is our friend, but XFS is complex
> so we still need people who have kept up with linux-xfs to know what's
> appropriate (and what compile tests can't find) to finish the process.
>

I am very happy that you stepped up to write these expectations.
Please spell out the process you want to see, because it was not
clear to Ted and to me how we should post the xfs stable candidates.

Should we post to xfs list and wait for explicit ACK/RVB on every patch?
Should we post to xfs list and if no objections are raised post to stable?
For those of you not subscribed to xfs list, I have just sent my first crop
of 5.10 candidates yesterday [1]. Still waiting to see how that plays out.

[1] https://lore.kernel.org/linux-xfs/20220525111715.2769700-1-amir73il@gmail.com/

> 2. Some other tag for patches that could be a fix, but need a few months
> to soak.  This is targetted at (b), since I'm terrible at remembering
> that there are patches that are reaching ripeness.

The question is, to soak in 'master' or to soak in a release?
which means to soak in the stable kernel.

My opinion is that if we want exposure to real users that use diverse
userspace tools, then soaking in stable is the right answer, so typically
(b) patches would be soaking ~2 months in -rc and then ~2 more months
in .y and at that point they can ONLY be considered for LTS, because that
.y release (which is not LTS) is EOL.

>
> 3. fstesting -- new patches proposed for stable branches shouldn't
> introduce new regressions, and ideally there would also be a regression
> test that would now pass.  As Dave and I have stated in the past,
> fstests is a big umbrella of a test suite, which implies that A/B
> testing is the way to go.  I think at least Zorro and I would like to
> improve the tagging in fstests to make it more obvious which tests
> contain enough randomness that they cannot be expected to behave 100%
> reliably.
>
> Here's a couple of antipatterns from the past:
>
> i. Robots shovelling patches into stable kernels with no testing.
>
> ii. Massively large backports.  New features don't go to stable kernels,
> and I doubt the stable kernel maintainers will accept that anyway.  I
> grok the temptation to backport more so that it's easier to land future
> fixes via AUTOSEL, but I personally wouldn't endorse frontloading a
> bunch of work to chase a promise of less future work.
>
> And a question or two:
>
> a> I've been following the recent fstests threads, and it seems to me
> that there are really two classes of users -- sustaining people who want
> fstests to run reliably so they can tell if their backports have broken
> anything; and developers, who want the randomness to try to poke into
> dusty corners of the filesystem.  Can we make it easier to associate
> random bits of data (reliability rates, etc.) with a given fstests
> configuration?  And create a test group^Wtag for the tests that rely on
> RNGs to shake things up?
>

I'm a big fan of that idea :)

> b> Testing relies very heavily on being able to spin up a lot of testing
> resources.  Can/should we make it easier for people with a kernel.org
> account to get free(ish) cloud accounts with the LF members who are also
> cloud vendors?
>
> Thoughts? Flames?
>

Here is one thought - the stable candidate review process is bound to add
more work to the overloaded xfs maintainer.

I think it is a classic role to delegate to xfs stable tree maintainer(s).
The xfs stable maintainer(s) job would be to ping reviewers for acks
make sure patches are tested according to the standard and then
curate a tree and communicate with the stable maintainers.

Ideally, there will be a maintainer per LTS tree, which could naturally
align with the interest of employers and easier for developers to justify
spent time, but the load can be shared among maintainers, as nagging
for review is often not LTS specific.

If xfs maintainers are willing to put their trust in me, I volunteer for the
job of xfs stable patch hurding in general and/or 5.10 LTS specific,
which I am also contributing to.

Thanks,
Amir.

Re: XFS LTS backport cabal

[Leah Rumancik]

On Wed, May 25, 2022 at 02:23:10PM -0700, Darrick J. Wong wrote:
> Hi everyone,
> 
> 3. fstesting -- new patches proposed for stable branches shouldn't
> introduce new regressions, and ideally there would also be a regression
> test that would now pass.  As Dave and I have stated in the past,
> fstests is a big umbrella of a test suite, which implies that A/B
> testing is the way to go.  I think at least Zorro and I would like to
> improve the tagging in fstests to make it more obvious which tests
> contain enough randomness that they cannot be expected to behave 100%
> reliably.
It would be nice to find an agreement on testing requirements. I have
attached some ideas on configs/number of tests/etc as well as the status
of my work on 5.15 below.


> a> I've been following the recent fstests threads, and it seems to me
> that there are really two classes of users -- sustaining people who want
> fstests to run reliably so they can tell if their backports have broken
> anything; and developers, who want the randomness to try to poke into
> dusty corners of the filesystem.  Can we make it easier to associate
> random bits of data (reliability rates, etc.) with a given fstests
> configuration?  And create a test group^Wtag for the tests that rely on
> RNGs to shake things up?
This would be great!

> 
> 
> Thoughts? Flames?
> 
> --D
This thread had good timing :) I have been working on setting up 
some automated testing. Currently, 5.15.y is our priority so I have 
started working on this branch.

Patches are being selected by simply searching for the “Fixes” 
tag and applying if the commit-to-be-fixed is in the stable branch, 
but AUTOSEL would be nice, so I’ll start playing around with that. 
Amir, it would be nice to sync up the patch selection process. I can 
help share the load, especially for 5.15.

Selecting just the tagged “Fixes” for 5.15.y for patches through 
5.17.2, 15 patches were found and applied - if there are no 
complaints about the testing setup, I can go ahead and send out this 
batch:

c30a0cbd07ec xfs: use kmem_cache_free() for kmem_cache objects
5ca5916b6bc9 xfs: punch out data fork delalloc blocks on COW writeback failure
a1de97fe296c xfs: Fix the free logic of state in xfs_attr_node_hasname
1090427bf18f xfs: remove xfs_inew_wait
089558bc7ba7 xfs: remove all COW fork extents when remounting readonly
7993f1a431bc xfs: only run COW extent recovery when there are no live extents
09654ed8a18c xfs: check sb_meta_uuid for dabuf buffer recovery
f8d92a66e810 xfs: prevent UAF in xfs_log_item_in_current_chkpt
b97cca3ba909 xfs: only bother with sync_filesystem during readonly remount
eba0549bc7d1 xfs: don't generate selinux audit messages for capability testing
e014f37db1a2 xfs: use setattr_copy to set vfs inode attributes
70447e0ad978 xfs: async CIL flushes need pending pushes to be made stable
c8c568259772 xfs: don't include bnobt blocks when reserving free block pool
cd6f79d1fb32 xfs: run callbacks before waking waiters in xlog_state_shutdown_callbacks
919edbadebe1 xfs: drop async cache flushes from CIL commits.

Tests are being run through gce-xfstests with the 5.15 kernel config 
from xfstests-bld 
(https://github.com/tytso/xfstests-bld/blob/master/kernel-configs/x86_
64-config-5.15). The configs being tested are the following:

xfs defaults
quota
quota 1k
v4
pmem and fsdax
realtime
8k directory blocks
external log
realtime and external log devices
realtime with 28k extents, external log devices
overlayfs atop xfs
overlayfs atop ext4
ext4 defaults

The test set will be run for each batch of backports, running each 
test 3 times, and if no new failures are seen compared to the same 
branch without the backports, the batch of patches will be deemed 
good. No regressions were seen for the first set of patches listed 
above when applied to 5.15.33. If new failures are seen during 
testing, a bisect can be run to find the offending commits, remove 
these from the batch, and confirm there are no remaining new 
failures. A bug report can be sent to indicate which commits would 
cause new test failures. The test results can be posted publicly 
after each run. The easiest option would be to send the test results 
to a mailing list, such as a google groups mailing list, similar to 
what syzkaller does, or directly to linux-xfs if it isn’t too 
spammy.


- Leah

Re: XFS LTS backport cabal

[Theodore Ts'o]

On Wed, May 25, 2022 at 02:23:10PM -0700, Darrick J. Wong wrote:
> 
> 2. Some other tag for patches that could be a fix, but need a few months
> to soak.  This is targetted at (b), since I'm terrible at remembering
> that there are patches that are reaching ripeness.

What I'd suggest here is a simple "Stable-Soak: <date>|<release>" tag.
It wouldn't need to be official, and so we don't need to get the
blessing of the Stable Tree maintainers; it would just be something
that would be honored by the "XFS LTS backport cabal".

> a> I've been following the recent fstests threads, and it seems to me
> that there are really two classes of users -- sustaining people who want
> fstests to run reliably so they can tell if their backports have broken
> anything; and developers, who want the randomness to try to poke into
> dusty corners of the filesystem.  Can we make it easier to associate
> random bits of data (reliability rates, etc.) with a given fstests
> configuration?  And create a test group^Wtag for the tests that rely on
> RNGs to shake things up?

In my experience, tests that have flaky results fall into two
categories; ones that are trying to deal traditional fuzzing, and
those that are running stress tests either by themselves, or as
antagonists against some other operation --- e.g., running fstress
while trying to do an online resize, or why trying to shut down the
file system, etc.

Some of these stress tests do use a PRNG, but even if we fixed the
seed to some value (such as 0), many of the test results would stlil
be potentially flaky.  These test results also tend to be very timing
dependant; so these are the tests that whose failure rate varies
depending on whether the test devices are on a loop device, eMMC flash
device, HDD, SSD, or various cloud virtual block devices, such as
AWS's EBS or or GCE's PD devices.

These tests are often very useful, since if we are missing a lock when
accessing some data structure, these tests which use stress test
programs are the most likely way noticing such problems.  So I don't
think we would want to exclude them; and if we're only excluding those
tests which are doing fuzz testing, I'm not sure it's really move the
needle.

> b> Testing relies very heavily on being able to spin up a lot of testing
> resources.  Can/should we make it easier for people with a kernel.org
> account to get free(ish) cloud accounts with the LF members who are also
> cloud vendors?

If anyone wants to use gce-xfstests, I'm happy to work on sponsoring
some GCE credits for that purpose.  One of the nice things about
gce-xfstests is that Test VM's only are left running when actually
running a test.  Once a test is finished, the VM shuts itself down.
And if we want to run a number of file system configs, we can spawn a
dozen VM's, one for each fsconfig, and when they are done, each VM
shuts itself down except for a small test test manager which collates
the results into a single report.  This makes gce-xfstests much more
cost efficient that those schemes which keeps a VM up and running at
all times, whether it is running tests or not.

Cheers,

						- Ted

Re: XFS LTS backport cabal

[Holger Hoffstätte]

On 2022-05-26 17:01, Leah Rumancik wrote:
> This thread had good timing :) I have been working on setting up
> some automated testing. Currently, 5.15.y is our priority so I have
> started working on this branch.
> 
> Patches are being selected by simply searching for the “Fixes”
> tag and applying if the commit-to-be-fixed is in the stable branch,
> but AUTOSEL would be nice, so I’ll start playing around with that.
> Amir, it would be nice to sync up the patch selection process. I can
> help share the load, especially for 5.15.
> 
> Selecting just the tagged “Fixes” for 5.15.y for patches through
> 5.17.2, 15 patches were found and applied - if there are no
> complaints about the testing setup, I can go ahead and send out this
> batch:
> 
> c30a0cbd07ec xfs: use kmem_cache_free() for kmem_cache objects
> 5ca5916b6bc9 xfs: punch out data fork delalloc blocks on COW writeback failure
> a1de97fe296c xfs: Fix the free logic of state in xfs_attr_node_hasname
> 1090427bf18f xfs: remove xfs_inew_wait
> 089558bc7ba7 xfs: remove all COW fork extents when remounting readonly
> 7993f1a431bc xfs: only run COW extent recovery when there are no live extents
> 09654ed8a18c xfs: check sb_meta_uuid for dabuf buffer recovery
> f8d92a66e810 xfs: prevent UAF in xfs_log_item_in_current_chkpt
> b97cca3ba909 xfs: only bother with sync_filesystem during readonly remount
> eba0549bc7d1 xfs: don't generate selinux audit messages for capability testing
> e014f37db1a2 xfs: use setattr_copy to set vfs inode attributes
> 70447e0ad978 xfs: async CIL flushes need pending pushes to be made stable
> c8c568259772 xfs: don't include bnobt blocks when reserving free block pool
> cd6f79d1fb32 xfs: run callbacks before waking waiters in xlog_state_shutdown_callbacks
> 919edbadebe1 xfs: drop async cache flushes from CIL commits.

Please include:
9a5280b312e2 xfs: reorder iunlink remove operation in xfs_ifree

Thanks!

Re: XFS LTS backport cabal

[Amir Goldstein]

On Thu, May 26, 2022 at 6:01 PM Leah Rumancik <leah.rumancik@gmail.com> wrote:
>
> On Wed, May 25, 2022 at 02:23:10PM -0700, Darrick J. Wong wrote:
> > Hi everyone,
> >
> > 3. fstesting -- new patches proposed for stable branches shouldn't
> > introduce new regressions, and ideally there would also be a regression
> > test that would now pass.  As Dave and I have stated in the past,
> > fstests is a big umbrella of a test suite, which implies that A/B
> > testing is the way to go.  I think at least Zorro and I would like to
> > improve the tagging in fstests to make it more obvious which tests
> > contain enough randomness that they cannot be expected to behave 100%
> > reliably.
> It would be nice to find an agreement on testing requirements. I have
> attached some ideas on configs/number of tests/etc as well as the status
> of my work on 5.15 below.
>
>
> > a> I've been following the recent fstests threads, and it seems to me
> > that there are really two classes of users -- sustaining people who want
> > fstests to run reliably so they can tell if their backports have broken
> > anything; and developers, who want the randomness to try to poke into
> > dusty corners of the filesystem.  Can we make it easier to associate
> > random bits of data (reliability rates, etc.) with a given fstests
> > configuration?  And create a test group^Wtag for the tests that rely on
> > RNGs to shake things up?
> This would be great!
>
> >
> >
> > Thoughts? Flames?
> >
> > --D
> This thread had good timing :) I have been working on setting up
> some automated testing. Currently, 5.15.y is our priority so I have
> started working on this branch.
>
> Patches are being selected by simply searching for the “Fixes”
> tag and applying if the commit-to-be-fixed is in the stable branch,
> but AUTOSEL would be nice, so I’ll start playing around with that.
> Amir, it would be nice to sync up the patch selection process. I can
> help share the load, especially for 5.15.
>

I would like that :)

> Selecting just the tagged “Fixes” for 5.15.y for patches through
> 5.17.2, 15 patches were found and applied - if there are no
> complaints about the testing setup, I can go ahead and send out this
> batch:
>
> c30a0cbd07ec xfs: use kmem_cache_free() for kmem_cache objects
> 5ca5916b6bc9 xfs: punch out data fork delalloc blocks on COW writeback failure
> a1de97fe296c xfs: Fix the free logic of state in xfs_attr_node_hasname
> 1090427bf18f xfs: remove xfs_inew_wait
> 089558bc7ba7 xfs: remove all COW fork extents when remounting readonly
> 7993f1a431bc xfs: only run COW extent recovery when there are no live extents
> 09654ed8a18c xfs: check sb_meta_uuid for dabuf buffer recovery
> f8d92a66e810 xfs: prevent UAF in xfs_log_item_in_current_chkpt
> b97cca3ba909 xfs: only bother with sync_filesystem during readonly remount
> eba0549bc7d1 xfs: don't generate selinux audit messages for capability testing
> e014f37db1a2 xfs: use setattr_copy to set vfs inode attributes
> 70447e0ad978 xfs: async CIL flushes need pending pushes to be made stable
> c8c568259772 xfs: don't include bnobt blocks when reserving free block pool
> cd6f79d1fb32 xfs: run callbacks before waking waiters in xlog_state_shutdown_callbacks
> 919edbadebe1 xfs: drop async cache flushes from CIL commits.
>

Here are my selection for v5.15..v5.17:

* 1cd231d9fdb1 - (tag: xfs-5.10.y-7) xfs: use setattr_copy to set vfs
inode attributes
* af09d052db41 - xfs: fallocate() should call file_modified()
* 0daebb90e096 - xfs: reject crazy array sizes being fed to XFS_IOC_GETBMAP*
* 35d876873c28 - xfs: prevent UAF in xfs_log_item_in_current_chkpt
* 796e9e00071d - xfs: xfs_log_force_lsn isn't passed a LSN
* fa33747dd25b - xfs: refactor xfs_file_fsync
* 374a05b9a2de - xfs: check sb_meta_uuid for dabuf buffer recovery
* 0b66f78d6af1 - (tag: xfs-5.10.y-6) xfs: remove all COW fork extents
when remounting readonly
* 44caa4c7aaf4 - xfs: remove incorrect ASSERT in xfs_rename
* 4133fc82c95d - xfs: punch out data fork delalloc blocks on COW
writeback failure

The branch of the moment is at:
https://github.com/amir73il/linux/commits/xfs-5.10.y
But I keep force pushing the branch and tags when dropping patches
from the queue.

Note that only half of those commits have Fixes: tag.
As I explained, I got to them by removing all the non-fixes and non-relevant
commits and then tried to evaluate the remaining commits individually.
This was only made scalable because I was working at the patch series
level and not at commit level, although at many times, a single fix patch
was selected from within a non-fix series.

Note that many of the fixes that you selected are impact waves of
big performance improvements that got merged after 5.10, so
were not relevant for my 5.10.y selection.

Thanks,
Amir.

Re: XFS LTS backport cabal

[Chandan Babu R]

On Thu, May 26, 2022 at 08:01:22 AM -0700, Leah Rumancik wrote:
> On Wed, May 25, 2022 at 02:23:10PM -0700, Darrick J. Wong wrote:
>> Hi everyone,
>> 
>> 3. fstesting -- new patches proposed for stable branches shouldn't
>> introduce new regressions, and ideally there would also be a regression
>> test that would now pass.  As Dave and I have stated in the past,
>> fstests is a big umbrella of a test suite, which implies that A/B
>> testing is the way to go.  I think at least Zorro and I would like to
>> improve the tagging in fstests to make it more obvious which tests
>> contain enough randomness that they cannot be expected to behave 100%
>> reliably.
> It would be nice to find an agreement on testing requirements. I have
> attached some ideas on configs/number of tests/etc as well as the status
> of my work on 5.15 below.
>
>
>> a> I've been following the recent fstests threads, and it seems to me
>> that there are really two classes of users -- sustaining people who want
>> fstests to run reliably so they can tell if their backports have broken
>> anything; and developers, who want the randomness to try to poke into
>> dusty corners of the filesystem.  Can we make it easier to associate
>> random bits of data (reliability rates, etc.) with a given fstests
>> configuration?  And create a test group^Wtag for the tests that rely on
>> RNGs to shake things up?
> This would be great!
>
>> 
>> 
>> Thoughts? Flames?
>> 
>> --D
> This thread had good timing :) I have been working on setting up 
> some automated testing. Currently, 5.15.y is our priority so I have 
> started working on this branch.
>
> Patches are being selected by simply searching for the “Fixes” 
> tag and applying if the commit-to-be-fixed is in the stable branch, 
> but AUTOSEL would be nice, so I’ll start playing around with that. 
> Amir, it would be nice to sync up the patch selection process. I can 
> help share the load, especially for 5.15.
>
> Selecting just the tagged “Fixes” for 5.15.y for patches through 
> 5.17.2, 15 patches were found and applied - if there are no 
> complaints about the testing setup, I can go ahead and send out this 
> batch:
>
> c30a0cbd07ec xfs: use kmem_cache_free() for kmem_cache objects
> 5ca5916b6bc9 xfs: punch out data fork delalloc blocks on COW writeback failure
> a1de97fe296c xfs: Fix the free logic of state in xfs_attr_node_hasname
> 1090427bf18f xfs: remove xfs_inew_wait
> 089558bc7ba7 xfs: remove all COW fork extents when remounting readonly
> 7993f1a431bc xfs: only run COW extent recovery when there are no live extents
> 09654ed8a18c xfs: check sb_meta_uuid for dabuf buffer recovery
> f8d92a66e810 xfs: prevent UAF in xfs_log_item_in_current_chkpt
> b97cca3ba909 xfs: only bother with sync_filesystem during readonly remount
> eba0549bc7d1 xfs: don't generate selinux audit messages for capability testing
> e014f37db1a2 xfs: use setattr_copy to set vfs inode attributes
> 70447e0ad978 xfs: async CIL flushes need pending pushes to be made stable
> c8c568259772 xfs: don't include bnobt blocks when reserving free block pool
> cd6f79d1fb32 xfs: run callbacks before waking waiters in xlog_state_shutdown_callbacks
> 919edbadebe1 xfs: drop async cache flushes from CIL commits.
>

In our experience, we found that some of the patches which fix bugs would not
have the associated "Fixes" tag. Hence I am currently using the script
https://gist.github.com/chandanr/c1e3affdb06eb2e025f955e7a77b2338 to identify
such commits along with the commits which have the "Fixes" tag.

The following command line obtains the list of commits from v5.18,

# list-xfs-fix-commits.sh v5.17 v5.18
--- Actual fixes ---
1:  eba0549bc7d10
2:  e014f37db1a2d
3:  70447e0ad9781
4:  c8c5682597727
5:  cd6f79d1fb324
6:  919edbadebe17
7:  9a5280b312e2e

---- Possible fixes; Along with matching regex  ----
1:  871b9316e7a77: bug
2:  41667260bc84d: bug
3:  83a44a4f47ad2: fix
4:  a9a4bc8c76d74: rac.+
5:  dbd0f5299302f: rac.+
6:  941fbdfd6dd0f: rac.+
7:  01728b44ef1b7: bug
8:  b9b1335e64030: fix
9:  82be38bcf8a2e: fix
10:  d2d7c0473586d: fix
11:  ab9c81ef321f9: assert
12:  b5f17bec1213a: rac.+
13:  41e6362183589: fix
14:  3c4cb76bce438: rac.+
15:  5652ef31705f2: fail

I go through each commit in the "Possible fixes" section and determine if any
of those need to be backported.

-- 
chandan