fstests.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Amir Goldstein <amir73il@gmail.com>
To: Matthew Wilcox <willy@infradead.org>
Cc: fstests <fstests@vger.kernel.org>, Sasha Levin <sashal@kernel.org>
Subject: Re: irreproducibility
Date: Sun, 23 Aug 2020 16:22:48 +0300	[thread overview]
Message-ID: <CAOQ4uxjzfSc1XfTbbBgcxTL-EfO67e_JUc8XPa9nUBb0iJ3bDg@mail.gmail.com> (raw)
In-Reply-To: <20200823113049.GZ17456@casper.infradead.org>

On Sun, Aug 23, 2020 at 3:35 PM Matthew Wilcox <willy@infradead.org> wrote:
>
> I have a particular dislike of random tests.  They make it harder to
> figure out when a bug has been fixed.  What's worse is when the test
> gives you enough information to make it reproducible, and we throw that
> away.
>
> I made the following change to find out the random seed that fsx was
> using for this test:
>
> +++ b/tests/generic/112
> @@ -57,7 +57,7 @@ _do_test()
>
>      # This cd and use of -P gets full debug on "$RESULT_DIR" (not TEST_DEV)
>      cd $out
> -    if ! $here/ltp/fsx $_param -P "$RESULT_DIR" $FSX_AVOID $seq.$_n &>/dev/null
> +    if ! $here/ltp/fsx $_param -P "$RESULT_DIR" $FSX_AVOID $seq.$_n &>"$RESULT_DIR"/fsx.stdouterr
>      then
>         echo "    fsx ($_param) returned $? - see $seq.$_n.full"
>         mv "$RESULT_DIR"/$seq.$_n.fsxlog $seqres.$_n.full
>
> I'm sure somebody can do something better than that which would actually
> result in it being automatically preserved-on-failure/deleted-on-success.
>

These files appear to be cleaned only on success:

rm -f $seq.*.fsx{good,log}

Not sure if intentionally...
You may as well use the same pattern.

But while we are on the subject of "random" tests, recently fsstress
gained a few
more random ops, one of them is RENAME_WHITEOUT, which can hang stable
kernels on xfs (at least on 5.7.10).

I admit that this has value - you pull latest xfstests, run the tests,
find a bug and
look for a patch to fix it. But this can also be pretty annoying if
all you are interested
is checking for regressions in stable kernels.

What we could do is mark all those tests that are a moving target with some
label such as "unstable", meaning that getting updates from upstream xfstest
may change the test, so that for regression testing, it would be easy to
exclude the "unstable" group.

Thanks,
Amir.

      reply	other threads:[~2020-08-23 13:23 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-08-23 11:30 irreproducibility Matthew Wilcox
2020-08-23 13:22 ` Amir Goldstein [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAOQ4uxjzfSc1XfTbbBgcxTL-EfO67e_JUc8XPa9nUBb0iJ3bDg@mail.gmail.com \
    --to=amir73il@gmail.com \
    --cc=fstests@vger.kernel.org \
    --cc=sashal@kernel.org \
    --cc=willy@infradead.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).