[LTP] [PATCH] runltp: add -R option for randomize test order

[LTP] [PATCH] runltp: add -R option for randomize test order

[Zorro Lang]

Generally we write test entries into runtests/${testfile}, then
runltp reads and runs lines sequentially everytime, nearly no one
will change the test order.

Recently we find bugs by change the test order，different test
combination maybe find more bugs. So I add this new feature to
randomize test order.

Signed-off-by: Zorro Lang <zlang@redhat.com>
---
 Makefile      |  2 +-
 randlines.awk | 43 +++++++++++++++++++++++++++++++++++++++++++
 runltp        | 14 +++++++++++++-
 3 files changed, 57 insertions(+), 2 deletions(-)
 create mode 100644 randlines.awk

diff --git a/Makefile b/Makefile
index 297f8e7..f6a56a7 100644
--- a/Makefile
+++ b/Makefile
@@ -163,7 +163,7 @@ clean:: $(CLEAN_TARGETS)
 $(foreach tgt,$(MAKE_TARGETS) include-all lib-all $(filter-out clean_install_dir,$(CLEAN_TARGETS)) $(INSTALL_TARGETS) include-install lib-install,$(eval $(call target_to_dir_dep_mapping,$(tgt))))
 
 BINDIR_INSTALL_SCRIPTS	:= execltp
-SRCDIR_INSTALL_SCRIPTS	:= IDcheck.sh runalltests.sh runltp runltplite.sh ver_linux
+SRCDIR_INSTALL_SCRIPTS	:= IDcheck.sh runalltests.sh runltp runltplite.sh ver_linux randlines.awk
 SRCDIR_INSTALL_READONLY	:= Version
 SRCDIR_INSTALL_TARGETS	:= $(SRCDIR_INSTALL_SCRIPTS) $(SRCDIR_INSTALL_READONLY)
 
diff --git a/randlines.awk b/randlines.awk
new file mode 100644
index 0000000..39f34cd
--- /dev/null
+++ b/randlines.awk
@@ -0,0 +1,43 @@
+#######################################################################
+# Copyright (c) 2016 Red Hat Inc.,  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation; either version 2 of
+# the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, see <http://www.gnu.org/licenses/>.
+#
+# Author: Zorro Lang <zlang@redhat.com>
+#
+#######################################################################
+
+# randomize all lines of stdin
+
+function randomize(array, N) {
+	for(i = 0; i < N; i++) {
+		j = int(rand()*N)
+		if ( i != j) {
+			tmp = array[i]
+			array[i] = array[j]
+			array[j] = tmp
+		}
+	}
+	return
+}
+
+{
+	array[NR] = $0
+}
+
+END {
+	srand()
+	randomize(array, NR)
+	for(i = 0; i < NR; i++) printf("%s\n", array[i])
+}
diff --git a/runltp b/runltp
index bb2e5d8..1f44b1f 100755
--- a/runltp
+++ b/runltp
@@ -64,6 +64,8 @@
 #               - Added capability for default Log file generation
 #		Aug 17 2009 - Modified - Subrata Modak
 #		- Added Fault Injection generation Capability through -F Option
+#		Sep 08 2016 - Modified - Zorro Lang
+#		- Added randomize test order capability through -R Option
 #
 #################################################################################
 
@@ -153,6 +155,7 @@ usage()
     -p              Human readable format logfiles.
     -q              Print less verbose output to screen.
     -r LTPROOT      Fully qualified path where testsuite is installed.
+    -R              Randomize test order.
     -s PATTERN      Only run test cases which match PATTERN.
     -S SKIPFILE     Skip tests specified in SKIPFILE
     -t DURATION     Execute the testsuite for given duration. Examples:
@@ -213,12 +216,13 @@ main()
     local EMAIL_TO=""
     local TAG_RESTRICT_STRING=""
     local PAN_COMMAND=""
+    local RANDOMRUN=0
     local DEFAULT_FILE_NAME_GENERATION_TIME=`date +"%Y_%m_%d-%Hh_%Mm_%Ss"`
     local scenfile=
 
     version_date=$(cat "$LTPROOT/Version")
 
-    while getopts a:c:C:T:d:D:f:F:ehi:I:K:g:l:m:M:Nno:pqr:s:S:t:T:w:x:b:B:z:Z: arg
+    while getopts a:c:C:T:d:D:f:F:ehi:I:K:g:l:m:M:Nno:pqr:Rs:S:t:T:w:x:b:B:z:Z: arg
     do  case $arg in
         a)  EMAIL_TO=$OPTARG
             ALT_EMAIL_OUT=1;;
@@ -410,6 +414,8 @@ main()
 
         r)  LTPROOT=$OPTARG;;
 
+        R)  RANDOMRUN=1;;
+
         s)  TAG_RESTRICT_STRING=$OPTARG;;
 
 	S)  case $OPTARG in
@@ -740,6 +746,12 @@ main()
          done
     fi
 
+    # randomize alltests
+    if [ "$RANDOMRUN" != "0" ]; then
+       awk -f ${LTPROOT}/randlines.awk ${TMP}/alltests > ${TMP}/alltests.temp
+       cat ${TMP}/alltests.temp > ${TMP}/alltests
+    fi
+
     [ ! -z "$QUIET_MODE" ] && { echo "INFO: Test start time: $(date)" ; }
     PAN_COMMAND="${LTPROOT}/bin/ltp-pan $QUIET_MODE -e -S $INSTANCES $DURATION -a $$ \
     -n $$ $PRETTY_PRT -f ${TMP}/alltests $LOGFILE $OUTPUTFILE $FAILCMDFILE $TCONFCMDFILE"
-- 
2.7.4

[LTP] [PATCH] runltp: add -R option for randomize test order

[Cyril Hrubis]

Hi!
> +#######################################################################
> +# Copyright (c) 2016 Red Hat Inc.,  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation; either version 2 of
> +# the License, or (at your option) any later version.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, see <http://www.gnu.org/licenses/>.
> +#
> +# Author: Zorro Lang <zlang@redhat.com>
> +#
> +#######################################################################
> +
> +# randomize all lines of stdin
> +
> +function randomize(array, N) {
> +	for(i = 0; i < N; i++) {
> +		j = int(rand()*N)
> +		if ( i != j) {
> +			tmp = array[i]
> +			array[i] = array[j]
> +			array[j] = tmp
> +		}
> +	}
> +	return
> +}
> +
> +{
> +	array[NR] = $0
> +}
> +
> +END {
> +	srand()
> +	randomize(array, NR)
> +	for(i = 0; i < NR; i++) printf("%s\n", array[i])
> +}

This is absurdly complicated for the task it does.

Newer GNU coreutils has shuf that does exactly what you want and there
is short -R as well, that more or less does what you want (the sort is
random unless there are duplicated lines in the input).

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [PATCH] runltp: add -R option for randomize test order

[Zorro Lang]

On Thu, Sep 08, 2016 at 03:17:06PM +0200, Cyril Hrubis wrote:
> Hi!
> > +#######################################################################
> > +# Copyright (c) 2016 Red Hat Inc.,  All Rights Reserved.
> > +#
> > +# This program is free software; you can redistribute it and/or
> > +# modify it under the terms of the GNU General Public License as
> > +# published by the Free Software Foundation; either version 2 of
> > +# the License, or (at your option) any later version.
> > +#
> > +# This program is distributed in the hope that it would be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License
> > +# along with this program; if not, see <http://www.gnu.org/licenses/>.
> > +#
> > +# Author: Zorro Lang <zlang@redhat.com>
> > +#
> > +#######################################################################
> > +
> > +# randomize all lines of stdin
> > +
> > +function randomize(array, N) {
> > +	for(i = 0; i < N; i++) {
> > +		j = int(rand()*N)
> > +		if ( i != j) {
> > +			tmp = array[i]
> > +			array[i] = array[j]
> > +			array[j] = tmp
> > +		}
> > +	}
> > +	return
> > +}
> > +
> > +{
> > +	array[NR] = $0
> > +}
> > +
> > +END {
> > +	srand()
> > +	randomize(array, NR)
> > +	for(i = 0; i < NR; i++) printf("%s\n", array[i])
> > +}
> 
> This is absurdly complicated for the task it does.
> 
> Newer GNU coreutils has shuf that does exactly what you want and there
> is short -R as well, that more or less does what you want (the sort is
> random unless there are duplicated lines in the input).

What stupid things I have done, I forgot 'sort' command o_O
Thanks for point that, I'll send a V2 patch and go to bed...

Thanks,
Zorro

> 
> -- 
> Cyril Hrubis
> chrubis@suse.cz