run more tests in the auto group

run more tests in the auto group

[Christoph Hellwig]

Hi all,

this series adds a more tests to the auto and quick groups so that they
are run as part of the usual regressions tests.

[PATCH 1/7] btrfs: add 198 and 219 to the auto group

[Christoph Hellwig]

The quick group should be a strict subset of the auto group, so add these
two tests that are in the quick group to the auto group as well.

Note: btrfs/219 fails for me in current mainline.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 tests/btrfs/198 | 2 +-
 tests/btrfs/219 | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/btrfs/198 b/tests/btrfs/198
index 1edc8330..2b68754a 100755
--- a/tests/btrfs/198
+++ b/tests/btrfs/198
@@ -9,7 +9,7 @@
 #    btrfs: remove identified alien device in open_fs_devices
 #
 . ./common/preamble
-_begin_fstest quick volume
+_begin_fstest auto quick volume
 
 # Import common functions.
 . ./common/filter
diff --git a/tests/btrfs/219 b/tests/btrfs/219
index d69e6ac9..528175b8 100755
--- a/tests/btrfs/219
+++ b/tests/btrfs/219
@@ -15,7 +15,7 @@
 #
 
 . ./common/preamble
-_begin_fstest quick volume
+_begin_fstest auto quick volume
 
 # Override the default cleanup function.
 _cleanup()
-- 
2.39.1

[PATCH 2/7] btrfs: add 125 to the auto and quick groups

[Christoph Hellwig]

btrfs/125 runs in 5 seconds on my VM setup, and found a regression in a
recent series.  Add it to the auto and quick groups.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 tests/btrfs/125 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/btrfs/125 b/tests/btrfs/125
index b58f2aa2..1b6e5d78 100755
--- a/tests/btrfs/125
+++ b/tests/btrfs/125
@@ -22,7 +22,7 @@
 # Verify if all three checkpoints match
 #
 . ./common/preamble
-_begin_fstest replace volume balance
+_begin_fstest replace volume balance auto quick
 
 # Override the default cleanup function.
 _cleanup()
-- 
2.39.1

[PATCH 3/7] btrfs: add 185 to the auto and quick groups

[Christoph Hellwig]

btrfs/185 runs in a second, add it to the auto and quick group.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 tests/btrfs/185 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/btrfs/185 b/tests/btrfs/185
index ee97b626..efb10ac7 100755
--- a/tests/btrfs/185
+++ b/tests/btrfs/185
@@ -9,7 +9,7 @@
 #  a9261d4125c9 ("btrfs: harden agaist duplicate fsid on scanned devices")
 #
 . ./common/preamble
-_begin_fstest volume
+_begin_fstest volume auto quick
 
 mnt=$TEST_DIR/$seq.mnt
 # Override the default cleanup function.
-- 
2.39.1

[PATCH 4/7] generic: add 042 to the auto and quick groups

[Christoph Hellwig]

generic/042 was removed from the auto group in 2015 by commit
7721b8501608 ("generic/042: remove from the 'auto' group") because it
always failed on XFS and wasn't run for other file systems back then.
Since then XFS fixed the problem it reproduces, and ext4 and f2fs
have grown shutdown support and also pass it reliably.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 tests/generic/042 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/generic/042 b/tests/generic/042
index 38e0a488..5116183f 100755
--- a/tests/generic/042
+++ b/tests/generic/042
@@ -13,7 +13,7 @@
 # stale data exposure can occur.
 #
 . ./common/preamble
-_begin_fstest shutdown rw punch zero prealloc
+_begin_fstest shutdown rw punch zero prealloc auto quick
 
 # Import common functions.
 . ./common/filter
-- 
2.39.1

[PATCH 5/7] generic: add 125 to the auto group

[Christoph Hellwig]

This test is not dangerous and passes reliably.  Add it to the auto
group.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 tests/generic/125 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/generic/125 b/tests/generic/125
index 077895d4..0d8e61a1 100755
--- a/tests/generic/125
+++ b/tests/generic/125
@@ -7,7 +7,7 @@
 # ftruncate test, modified from CXFSQA tests cxfs_ftrunc and cxfs_trunc
 #
 . ./common/preamble
-_begin_fstest other pnfs
+_begin_fstest other pnfs auto
 
 # Import common functions.
 . ./common/filter
-- 
2.39.1

[PATCH 6/7] generic: add 251 to the auto group

[Christoph Hellwig]

generic/251 isn't dangerous, doesn't takes overly long to run and doesn't
produce spurious failures, so add it to the auto group.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 tests/generic/251 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/generic/251 b/tests/generic/251
index 192ab5cc..2a271cd1 100755
--- a/tests/generic/251
+++ b/tests/generic/251
@@ -10,7 +10,7 @@
 # corrupts the filesystem (data/metadata).
 #
 . ./common/preamble
-_begin_fstest ioctl trim
+_begin_fstest ioctl trim auto
 
 tmp=`mktemp -d`
 trap "_cleanup; exit \$status" 0 1 3
-- 
2.39.1

[PATCH 7/7] xfs: add 080 to the auto and quick groups

[Christoph Hellwig]

xfs/080 is not dangerous, isn't a known fail and runs very quickly.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 tests/xfs/080 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/xfs/080 b/tests/xfs/080
index abcfa004..2d90b6c4 100755
--- a/tests/xfs/080
+++ b/tests/xfs/080
@@ -7,7 +7,7 @@
 # rwtest (iogen|doio)
 #
 . ./common/preamble
-_begin_fstest rw ioctl
+_begin_fstest rw ioctl auto quick
 
 # Import common functions.
 . ./common/filter
-- 
2.39.1

Re: run more tests in the auto group

[Zorro Lang]

On Thu, Feb 09, 2023 at 06:13:48AM +0100, Christoph Hellwig wrote:
> Hi all,
> 
> this series adds a more tests to the auto and quick groups so that they
> are run as part of the usual regressions tests.

Thanks, I'm glad to see more stable cases be added into *auto* group.

But as you said btrfs/219 still fails[1] now, so I hope we can use
another patch to fix btrfs/219 singly, even if use _fixed_by_kernel_commit
or _wants_kernel_commit to specify clarify why it's failed now.
Others looks good to me. If you don't mind, I'll merge others, exclude
btrfs/219 (before another patch improve it:)

Thanks,
Zorro

[1]
btrfs/219       [failed, exit status 1]- output mismatch (see /root/git/xfstests/results//simpledev/btrfs/219.out.bad)
    --- tests/btrfs/219.out     2022-04-29 23:07:22.997495284 +0800
    +++ /root/git/xfstests/results//simpledev/btrfs/219.out.bad 2023-02-09 22:22:43.447510026 +0800
    @@ -1,2 +1,4 @@
     QA output created by 219
    -Silence is golden
    +We couldn't mount the old generation
    +(see /root/git/xfstests/results//simpledev/btrfs/219.full for details)
    +umount: /mnt/test/219.mnt: not mounted.
    ...
    (Run 'diff -u /root/git/xfstests/tests/btrfs/219.out /root/git/xfstests/results//simpledev/btrfs/219.out.bad'  to see the entire diff)

> 

Re: [PATCH 6/7] generic: add 251 to the auto group

[Darrick J. Wong]

On Thu, Feb 09, 2023 at 06:13:54AM +0100, Christoph Hellwig wrote:
> generic/251 isn't dangerous, doesn't takes overly long to run and doesn't
> produce spurious failures, so add it to the auto group.

How long does it take for you?  It generally takes 300-700s to run on my
testing cloud.  That's not a reason to keep it out of the auto group,
but I've been surveying the long running tests to find the ones that run
stuff in a loop and subject them to TIME_FACTOR (or the new
SOAK_DURATION knob that I'm working on to allow direct control of loop
runtime) constraints.

--D

> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
>  tests/generic/251 | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tests/generic/251 b/tests/generic/251
> index 192ab5cc..2a271cd1 100755
> --- a/tests/generic/251
> +++ b/tests/generic/251
> @@ -10,7 +10,7 @@
>  # corrupts the filesystem (data/metadata).
>  #
>  . ./common/preamble
> -_begin_fstest ioctl trim
> +_begin_fstest ioctl trim auto
>  
>  tmp=`mktemp -d`
>  trap "_cleanup; exit \$status" 0 1 3
> -- 
> 2.39.1
> 

Re: run more tests in the auto group

[Darrick J. Wong]

On Thu, Feb 09, 2023 at 06:13:48AM +0100, Christoph Hellwig wrote:
> Hi all,
> 
> this series adds a more tests to the auto and quick groups so that they
> are run as part of the usual regressions tests.

For patches 4-7,

Reviewed-by: Darrick J. Wong <djwong@kernel.org>

--D

Re: [PATCH 6/7] generic: add 251 to the auto group

[Christoph Hellwig]

On Thu, Feb 09, 2023 at 08:13:11AM -0800, Darrick J. Wong wrote:
> On Thu, Feb 09, 2023 at 06:13:54AM +0100, Christoph Hellwig wrote:
> > generic/251 isn't dangerous, doesn't takes overly long to run and doesn't
> > produce spurious failures, so add it to the auto group.
> 
> How long does it take for you?

Between 300 and 400 seconds usually.

Re: run more tests in the auto group

[Anand Jain]

On 09/02/2023 13:13, Christoph Hellwig wrote:
> Hi all,
> 
> this series adds a more tests to the auto and quick groups so that they
> are run as part of the usual regressions tests.

For patches 1-4

      Reviewed-by: Anand Jain <anand.jain@oracle.com>


They also need the _fixed_by_kernel_commit attribute, which the 
(upcoming) series [1] fixes.

     [1] [PATCH 0/3] fstests: btrfs- add _fixed_by for new tests in the 
auto group

However, for btrfs/125, it is most likely that there isn't a single
patch that made it work.

Thanks, Anand