irreproducibility

irreproducibility

[Matthew Wilcox]

I have a particular dislike of random tests.  They make it harder to
figure out when a bug has been fixed.  What's worse is when the test
gives you enough information to make it reproducible, and we throw that
away.

I made the following change to find out the random seed that fsx was
using for this test:

+++ b/tests/generic/112
@@ -57,7 +57,7 @@ _do_test()
 
     # This cd and use of -P gets full debug on "$RESULT_DIR" (not TEST_DEV)
     cd $out
-    if ! $here/ltp/fsx $_param -P "$RESULT_DIR" $FSX_AVOID $seq.$_n &>/dev/null
+    if ! $here/ltp/fsx $_param -P "$RESULT_DIR" $FSX_AVOID $seq.$_n &>"$RESULT_DIR"/fsx.stdouterr
     then
        echo "    fsx ($_param) returned $? - see $seq.$_n.full"
        mv "$RESULT_DIR"/$seq.$_n.fsxlog $seqres.$_n.full

I'm sure somebody can do something better than that which would actually
result in it being automatically preserved-on-failure/deleted-on-success.

Re: irreproducibility

[Amir Goldstein]

On Sun, Aug 23, 2020 at 3:35 PM Matthew Wilcox <willy@infradead.org> wrote:
>
> I have a particular dislike of random tests.  They make it harder to
> figure out when a bug has been fixed.  What's worse is when the test
> gives you enough information to make it reproducible, and we throw that
> away.
>
> I made the following change to find out the random seed that fsx was
> using for this test:
>
> +++ b/tests/generic/112
> @@ -57,7 +57,7 @@ _do_test()
>
>      # This cd and use of -P gets full debug on "$RESULT_DIR" (not TEST_DEV)
>      cd $out
> -    if ! $here/ltp/fsx $_param -P "$RESULT_DIR" $FSX_AVOID $seq.$_n &>/dev/null
> +    if ! $here/ltp/fsx $_param -P "$RESULT_DIR" $FSX_AVOID $seq.$_n &>"$RESULT_DIR"/fsx.stdouterr
>      then
>         echo "    fsx ($_param) returned $? - see $seq.$_n.full"
>         mv "$RESULT_DIR"/$seq.$_n.fsxlog $seqres.$_n.full
>
> I'm sure somebody can do something better than that which would actually
> result in it being automatically preserved-on-failure/deleted-on-success.
>

These files appear to be cleaned only on success:

rm -f $seq.*.fsx{good,log}

Not sure if intentionally...
You may as well use the same pattern.

But while we are on the subject of "random" tests, recently fsstress
gained a few
more random ops, one of them is RENAME_WHITEOUT, which can hang stable
kernels on xfs (at least on 5.7.10).

I admit that this has value - you pull latest xfstests, run the tests,
find a bug and
look for a patch to fix it. But this can also be pretty annoying if
all you are interested
is checking for regressions in stable kernels.

What we could do is mark all those tests that are a moving target with some
label such as "unstable", meaning that getting updates from upstream xfstest
may change the test, so that for regression testing, it would be easy to
exclude the "unstable" group.

Thanks,
Amir.