[PATCH] check: add CLI option to repeat tests

[PATCH] check: add CLI option to repeat tests

[Dave Chinner]

From: Dave Chinner <dchinner@redhat.com>

Frequently when trying to reproduce a problem I want to run a set of
specific tests in a loop, over and over again. I run fstests from a
set of run scripts that have non-trivial overhead (e.g. patterning
block devices before the runs start), so if all I want to do is run
the same test 100x, using a shell loop over the entire run
scripts reduces the iteration rate substantially.

Hence add an option to check to allow fstests to loop a number of
times over the configured test set without stopping.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 check | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)

diff --git a/check b/check
index 2e148e57..2676b619 100755
--- a/check
+++ b/check
@@ -26,6 +26,7 @@ subdir_xfile=""
 brief_test_summary=false
 do_report=false
 DUMP_OUTPUT=false
+iterations=1
 
 # This is a global variable used to pass test failure text to reporting gunk
 _err_msg=""
@@ -66,6 +67,7 @@ check options
     -n			show me, do not run tests
     -T			output timestamps
     -r			randomize test order
+    -i <n>		iterate the test list <n> times
     -d			dump test output to stdout
     -b			brief test summary
     -R fmt[,fmt]	generate report in formats specified. Supported format: [xunit]
@@ -297,7 +299,7 @@ while [ $# -gt 0 ]; do
 
 	-n)	showme=true ;;
         -r)	randomize=true ;;
-
+	-i)	iterations=$2; shift ;;
 	-T)	timestamp=true ;;
 	-d)	DUMP_OUTPUT=true ;;
 	-b)	brief_test_summary=true;;
@@ -525,7 +527,10 @@ else
 	trap "_wrapup; exit \$status" 0 1 2 3 15
 fi
 
-for section in $HOST_OPTIONS_SECTIONS; do
+function run_section()
+{
+	local section=$1
+
 	OLD_FSTYP=$FSTYP
 	OLD_TEST_FS_MOUNT_OPTS=$TEST_FS_MOUNT_OPTS
 	get_next_config $section
@@ -540,7 +545,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
 			fi
 		done
 		if $skip; then
-			continue
+			return
 		fi
 	fi
 
@@ -554,7 +559,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
 			fi
 		done
 		if $skip; then
-			continue
+			return
 		fi
 	fi
 
@@ -700,6 +705,8 @@ for section in $HOST_OPTIONS_SECTIONS; do
 		seqres="$REPORT_DIR/$seqnum"
 
 		mkdir -p $RESULT_DIR
+		rm -f ${RESULT_DIR}/require_scratch*
+		rm -f ${RESULT_DIR}/require_test*
 		echo -n "$seqnum"
 
 		if $showme; then
@@ -858,6 +865,12 @@ for section in $HOST_OPTIONS_SECTIONS; do
 
 	_test_unmount 2> /dev/null
 	_scratch_unmount 2> /dev/null
+}
+
+for ((iters = 0; iters < $iterations; iters++)) do
+	for section in $HOST_OPTIONS_SECTIONS; do
+		run_section $section
+	done
 done
 
 interrupt=false
-- 
2.26.2.761.g0e0b3e54be

Re: [PATCH] check: add CLI option to repeat tests

[Lukas Czerner]

On Wed, Jul 08, 2020 at 03:35:13PM +1000, Dave Chinner wrote:
> From: Dave Chinner <dchinner@redhat.com>
> 
> Frequently when trying to reproduce a problem I want to run a set of
> specific tests in a loop, over and over again. I run fstests from a
> set of run scripts that have non-trivial overhead (e.g. patterning
> block devices before the runs start), so if all I want to do is run
> the same test 100x, using a shell loop over the entire run
> scripts reduces the iteration rate substantially.
> 
> Hence add an option to check to allow fstests to loop a number of
> times over the configured test set without stopping.

Hi Dave,

this is useful as I am often doing the same. However what I am really
missing is the ability to end the loop when a test fails and the ability
to run tests in infinite loop (or until the test fails).

It should be easy modification to make, what do you think ?

-Lukas

> 
> Signed-off-by: Dave Chinner <dchinner@redhat.com>
> ---
>  check | 21 +++++++++++++++++----
>  1 file changed, 17 insertions(+), 4 deletions(-)
> 
> diff --git a/check b/check
> index 2e148e57..2676b619 100755
> --- a/check
> +++ b/check
> @@ -26,6 +26,7 @@ subdir_xfile=""
>  brief_test_summary=false
>  do_report=false
>  DUMP_OUTPUT=false
> +iterations=1
>  
>  # This is a global variable used to pass test failure text to reporting gunk
>  _err_msg=""
> @@ -66,6 +67,7 @@ check options
>      -n			show me, do not run tests
>      -T			output timestamps
>      -r			randomize test order
> +    -i <n>		iterate the test list <n> times
>      -d			dump test output to stdout
>      -b			brief test summary
>      -R fmt[,fmt]	generate report in formats specified. Supported format: [xunit]
> @@ -297,7 +299,7 @@ while [ $# -gt 0 ]; do
>  
>  	-n)	showme=true ;;
>          -r)	randomize=true ;;
> -
> +	-i)	iterations=$2; shift ;;
>  	-T)	timestamp=true ;;
>  	-d)	DUMP_OUTPUT=true ;;
>  	-b)	brief_test_summary=true;;
> @@ -525,7 +527,10 @@ else
>  	trap "_wrapup; exit \$status" 0 1 2 3 15
>  fi
>  
> -for section in $HOST_OPTIONS_SECTIONS; do
> +function run_section()
> +{
> +	local section=$1
> +
>  	OLD_FSTYP=$FSTYP
>  	OLD_TEST_FS_MOUNT_OPTS=$TEST_FS_MOUNT_OPTS
>  	get_next_config $section
> @@ -540,7 +545,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
>  			fi
>  		done
>  		if $skip; then
> -			continue
> +			return
>  		fi
>  	fi
>  
> @@ -554,7 +559,7 @@ for section in $HOST_OPTIONS_SECTIONS; do
>  			fi
>  		done
>  		if $skip; then
> -			continue
> +			return
>  		fi
>  	fi
>  
> @@ -700,6 +705,8 @@ for section in $HOST_OPTIONS_SECTIONS; do
>  		seqres="$REPORT_DIR/$seqnum"
>  
>  		mkdir -p $RESULT_DIR
> +		rm -f ${RESULT_DIR}/require_scratch*
> +		rm -f ${RESULT_DIR}/require_test*
>  		echo -n "$seqnum"
>  
>  		if $showme; then
> @@ -858,6 +865,12 @@ for section in $HOST_OPTIONS_SECTIONS; do
>  
>  	_test_unmount 2> /dev/null
>  	_scratch_unmount 2> /dev/null
> +}
> +
> +for ((iters = 0; iters < $iterations; iters++)) do
> +	for section in $HOST_OPTIONS_SECTIONS; do
> +		run_section $section
> +	done
>  done
>  
>  interrupt=false
> -- 
> 2.26.2.761.g0e0b3e54be
> 

Re: [PATCH] check: add CLI option to repeat tests

[Dave Chinner]

On Wed, Jul 08, 2020 at 10:15:18AM +0200, Lukas Czerner wrote:
> On Wed, Jul 08, 2020 at 03:35:13PM +1000, Dave Chinner wrote:
> > From: Dave Chinner <dchinner@redhat.com>
> > 
> > Frequently when trying to reproduce a problem I want to run a set of
> > specific tests in a loop, over and over again. I run fstests from a
> > set of run scripts that have non-trivial overhead (e.g. patterning
> > block devices before the runs start), so if all I want to do is run
> > the same test 100x, using a shell loop over the entire run
> > scripts reduces the iteration rate substantially.
> > 
> > Hence add an option to check to allow fstests to loop a number of
> > times over the configured test set without stopping.
> 
> Hi Dave,
> 
> this is useful as I am often doing the same. However what I am really
> missing is the ability to end the loop when a test fails and the ability
> to run tests in infinite loop (or until the test fails).

To run until a test set fails, I have just been using "-i 10000"
because if it passes 10000 loops, it isn't going to fail.. :)

Hence I didn't see a need to complicate things with a specific
"loop forever" options.

> It should be easy modification to make, what do you think ?

Feel free to add send a patch on top of this one to support that -
all of the failures I've been looking at with this have been "test
does not complete" failures, so it wasn't important to me.

As it is, right now I don't have time to feature creep simple
modifications like this into more complex functionality - I really
only sent this out because Willy mentioned on #xfs that he wished
there was a way to loop a test and I said "I've got a patch on my
local tree for that"....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com