linux-xfs.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Amir Goldstein <amir73il@gmail.com>
To: "Darrick J. Wong" <djwong@kernel.org>
Cc: Leah Rumancik <leah.rumancik@gmail.com>,
	Chandan Babu R <chandan.babu@oracle.com>,
	linux-xfs <linux-xfs@vger.kernel.org>,
	fstests <fstests@vger.kernel.org>
Subject: Re: [PATCH 5.10 CANDIDATE 0/9] xfs stable candidate patches for 5.10.y (from v5.13+)
Date: Fri, 29 Jul 2022 18:15:38 +0200	[thread overview]
Message-ID: <CAOQ4uxgbWOEZTCX_S43DCT-0KihU6By3TtwvYjZA6DS=y20FHQ@mail.gmail.com> (raw)
In-Reply-To: <CAOQ4uxiSKNYAT_oUXAwGADt4KQvVH8s12nFSrRUoKYi8DOzQig@mail.gmail.com>

On Thu, Jul 28, 2022 at 11:39 AM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Thu, Jul 28, 2022 at 4:07 AM Darrick J. Wong <djwong@kernel.org> wrote:
> >
> > On Wed, Jul 27, 2022 at 07:01:15PM -0700, Darrick J. Wong wrote:
> > > On Wed, Jul 27, 2022 at 09:17:47PM +0200, Amir Goldstein wrote:
> > > > On Tue, Jul 26, 2022 at 11:21 AM Amir Goldstein <amir73il@gmail.com> wrote:
> > > > >
> > > > > Darrick,
> > > > >
> > > > > This backport series contains mostly fixes from v5.14 release along
> > > > > with three deferred patches from the joint 5.10/5.15 series [1].
> > > > >
> > > > > I ran the auto group 10 times on baseline (v5.10.131) and this series
> > > > > with no observed regressions.
> > > > >
> > > > > I ran the recoveryloop group 100 times with no observed regressions.
> > > > > The soak group run is in progress (10+) with no observed regressions
> > > > > so far.
> > > > >
> > > > > I am somewhat disappointed from not seeing any improvement in the
> > > > > results of the recoveryloop tests comapred to baseline.
> > > > >
> > > > > This is the summary of the recoveryloop test results on both baseline
> > > > > and backport branch:
> > > > >
> > > > > generic,455, generic/457, generic/646: pass
> > > > > generic/019, generic/475, generic/648: failing often in all config
> > >
> > > <nod> I posted a couple of patchsets to fstests@ yesterday that might
> > > help with these recoveryloop tests failing.
> > >
> > > https://lore.kernel.org/fstests/165886493457.1585218.32410114728132213.stgit@magnolia/T/#t
> > > https://lore.kernel.org/fstests/165886492580.1585149.760428651537119015.stgit@magnolia/T/#t
> > > https://lore.kernel.org/fstests/165886491119.1585061.14285332087646848837.stgit@magnolia/T/#t
> > >
> > > > > generic/388: failing often with reflink_1024
> > > > > generic/388: failing at ~1/50 rate for any config
> > > > > generic/482: failing often on V4 configs
> > > > > generic/482: failing at ~1/100 rate for V5 configs
> > > > > xfs/057: failing at ~1/200 rate for any config
> > > > >
> > > > > I observed no failures in soak group so far neither on baseline nor
> > > > > on backport branch. I will update when I have more results.
> > > > >
> > > >
> > > > Some more results after 1.5 days of spinning:
> > > > 1. soak group reached 100 runs (x5 configs) with no failures
> > > > 2. Ran all the tests also on debian/testing with xfsprogs 5.18 and
> > > >     observed a very similar fail/pass pattern as with xfsprogs 5.10
> > > > 3. Started to run the 3 passing recoveryloop tests 1000 times and
> > > >     an interesting pattern emerged -
> > > >
> > > > generic/455 failed 3 times on baseline (out of 250 runs x 5 configs),
> > > > but if has not failed on backport branch yet (700 runs x 5 configs).
> > > >
> > > > And it's not just failures, it's proper data corruptions, e.g.
> > > > "testfile2.mark1 md5sum mismatched" (and not always on mark1)
> > >
> > > Oh good!
> > >
> > >
> > > >
> > > > I will keep this loop spinning, but I am cautiously optimistic about
> > > > this being an actual proof of bug fix.
>
> That was wishful thinking - after 1500 x 5 runs there are 2 failures also
> on the backport branch.
>
> It is still better than 4 failures out of 800 x 5 runs on baseline, but I can
> not call that a proof for bug fix - only no regression.
>
> > > >
> > > > If these results don't change, I would be happy to get an ACK for the
> > > > series so I can post it after the long soaking.
> > >
> > > Patches 4-9 are an easy
> > > Acked-by: Darrick J. Wong <djwong@kernel.org>
> >
> > I hit send too fast.
> >
> > I think patches 1-3 look correct.  I still think it's sort of risky,
> > but your testing shows that things at least get better and don't
> > immediately explode or anything. :)
> >
> > By my recollection of the log changes between 5.10 and 5.17 I think the
> > lsn/cil split didn't change all that much, so if you get to the end of
> > the week with no further problems then I say Acked-by for them too.
> >
>
> Great. I'll keep it spinning.
>

Well, it's the end of my week now and loop has passed over 4,000
runs x 5 configs on the backport branch with only 4 failures, so it's
going to stable.

Thanks,
Amir.

      reply	other threads:[~2022-07-29 16:15 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-07-26  9:21 [PATCH 5.10 CANDIDATE 0/9] xfs stable candidate patches for 5.10.y (from v5.13+) Amir Goldstein
2022-07-26  9:21 ` [PATCH 5.10 CANDIDATE 1/9] xfs: refactor xfs_file_fsync Amir Goldstein
2022-07-26  9:21 ` [PATCH 5.10 CANDIDATE 2/9] xfs: xfs_log_force_lsn isn't passed a LSN Amir Goldstein
2022-07-26  9:21 ` [PATCH 5.10 CANDIDATE 3/9] xfs: prevent UAF in xfs_log_item_in_current_chkpt Amir Goldstein
2022-07-26  9:21 ` [PATCH 5.10 CANDIDATE 4/9] xfs: fix log intent recovery ENOSPC shutdowns when inactivating inodes Amir Goldstein
2022-07-26  9:21 ` [PATCH 5.10 CANDIDATE 5/9] xfs: force the log offline when log intent item recovery fails Amir Goldstein
2022-07-26  9:21 ` [PATCH 5.10 CANDIDATE 6/9] xfs: hold buffer across unpin and potential shutdown processing Amir Goldstein
2022-07-26  9:21 ` [PATCH 5.10 CANDIDATE 7/9] xfs: remove dead stale buf unpin handling code Amir Goldstein
2022-07-26  9:21 ` [PATCH 5.10 CANDIDATE 8/9] xfs: logging the on disk inode LSN can make it go backwards Amir Goldstein
2022-07-26  9:21 ` [PATCH 5.10 CANDIDATE 9/9] xfs: Enforce attr3 buffer recovery order Amir Goldstein
2022-07-27 19:17 ` [PATCH 5.10 CANDIDATE 0/9] xfs stable candidate patches for 5.10.y (from v5.13+) Amir Goldstein
2022-07-28  2:01   ` Darrick J. Wong
2022-07-28  2:07     ` Darrick J. Wong
2022-07-28  9:39       ` Amir Goldstein
2022-07-29 16:15         ` Amir Goldstein [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAOQ4uxgbWOEZTCX_S43DCT-0KihU6By3TtwvYjZA6DS=y20FHQ@mail.gmail.com' \
    --to=amir73il@gmail.com \
    --cc=chandan.babu@oracle.com \
    --cc=djwong@kernel.org \
    --cc=fstests@vger.kernel.org \
    --cc=leah.rumancik@gmail.com \
    --cc=linux-xfs@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).