[kvm-unit-tests PATCH v4 0/2] run_tests: support concurrent test execution

[kvm-unit-tests PATCH v4 0/2] run_tests: support concurrent test execution

[Peter Xu]

v4:
- add .gitignore for logs/ [Drew]
- instead of create globals.bash, renaming function.bash into
  common.bash, put globals inside [Drew]
- instead of removing logs/ directly when start run_tests, move it
  into logs.old so we at least have the last time result cached [Drew]
- s/ut_/unittest_/ through the whole series [Drew]
- remove unittest_log_summary var [Drew]
- remove radim's s-b in patch 2 since it does not suite [Drew]
- tiny fix on the usage lines [Drew]
- use bash arithmetic where proper [Drew]
- remove ut_in_parallel since not used [Drew]

v3:
- better handling for ctrl-c during run_tests.sh [Radim]

v2:
- patch 1: do per-test logging in all cases
- patch 2: throw away task.bash, instead, take Radim's suggestion to
  use jobs

run_tests.sh is getting slower. Maybe it's time to let it run faster.
An obvious issue is that, we were running the tests sequentially in
the past.

This series provides another new "-j" parameter. "-j 8" means we run
the tests on 8 task queues. That'll fasten the script a lot. A very
quick test of mine shows 3x speed boost with 8 task queues.

Please review, thanks.

Peter Xu (2):
  run_tests: put logs into per-test file
  run_tests: allow run tests in parallel

 .gitignore                              |  3 ++-
 Makefile                                |  4 ++--
 run_tests.sh                            | 34 ++++++++++++++++++++++++---------
 scripts/{functions.bash => common.bash} | 27 ++++++++++++++++++++++++--
 scripts/mkstandalone.sh                 |  2 +-
 5 files changed, 55 insertions(+), 15 deletions(-)
 rename scripts/{functions.bash => common.bash} (63%)

-- 
2.7.4

[Qemu-devel] [kvm-unit-tests PATCH v4 0/2] run_tests: support concurrent test execution

[Peter Xu]

v4:
- add .gitignore for logs/ [Drew]
- instead of create globals.bash, renaming function.bash into
  common.bash, put globals inside [Drew]
- instead of removing logs/ directly when start run_tests, move it
  into logs.old so we at least have the last time result cached [Drew]
- s/ut_/unittest_/ through the whole series [Drew]
- remove unittest_log_summary var [Drew]
- remove radim's s-b in patch 2 since it does not suite [Drew]
- tiny fix on the usage lines [Drew]
- use bash arithmetic where proper [Drew]
- remove ut_in_parallel since not used [Drew]

v3:
- better handling for ctrl-c during run_tests.sh [Radim]

v2:
- patch 1: do per-test logging in all cases
- patch 2: throw away task.bash, instead, take Radim's suggestion to
  use jobs

run_tests.sh is getting slower. Maybe it's time to let it run faster.
An obvious issue is that, we were running the tests sequentially in
the past.

This series provides another new "-j" parameter. "-j 8" means we run
the tests on 8 task queues. That'll fasten the script a lot. A very
quick test of mine shows 3x speed boost with 8 task queues.

Please review, thanks.

Peter Xu (2):
  run_tests: put logs into per-test file
  run_tests: allow run tests in parallel

 .gitignore                              |  3 ++-
 Makefile                                |  4 ++--
 run_tests.sh                            | 34 ++++++++++++++++++++++++---------
 scripts/{functions.bash => common.bash} | 27 ++++++++++++++++++++++++--
 scripts/mkstandalone.sh                 |  2 +-
 5 files changed, 55 insertions(+), 15 deletions(-)
 rename scripts/{functions.bash => common.bash} (63%)

-- 
2.7.4

[kvm-unit-tests PATCH v4 1/2] run_tests: put logs into per-test file

[Peter Xu]

We were using test.log before to keep all the test logs. This patch
creates one log file per test case under logs/ directory with name
"TESTNAME.log". Meanwhile, we will keep the last time log into
logs.old/.

Renaming scripts/functions.bash into scripts/common.bash to store some
more global variables.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 .gitignore                              |  3 ++-
 Makefile                                |  4 ++--
 run_tests.sh                            | 22 +++++++++++++++-------
 scripts/{functions.bash => common.bash} | 13 +++++++++++--
 scripts/mkstandalone.sh                 |  2 +-
 5 files changed, 31 insertions(+), 13 deletions(-)
 rename scripts/{functions.bash => common.bash} (75%)

diff --git a/.gitignore b/.gitignore
index 3155418..0dc9a39 100644
--- a/.gitignore
+++ b/.gitignore
@@ -12,7 +12,8 @@ cscope.*
 /lib/asm
 /config.mak
 /*-run
-/test.log
 /msr.out
 /tests
 /build-head
+logs
+logs.old
diff --git a/Makefile b/Makefile
index a32333b..bd8843a 100644
--- a/Makefile
+++ b/Makefile
@@ -94,9 +94,9 @@ libfdt_clean:
 	$(LIBFDT_objdir)/.*.d
 
 distclean: clean libfdt_clean
-	$(RM) lib/asm config.mak $(TEST_DIR)-run test.log msr.out cscope.* \
+	$(RM) lib/asm config.mak $(TEST_DIR)-run msr.out cscope.* \
 	      build-head
-	$(RM) -r tests
+	$(RM) -r tests logs logs.old
 
 cscope: cscope_dirs = lib lib/libfdt lib/linux $(TEST_DIR) $(ARCH_LIBDIRS) lib/asm-generic
 cscope:
diff --git a/run_tests.sh b/run_tests.sh
index 254129d..1e36d66 100755
--- a/run_tests.sh
+++ b/run_tests.sh
@@ -7,7 +7,7 @@ if [ ! -f config.mak ]; then
     exit 1
 fi
 source config.mak
-source scripts/functions.bash
+source scripts/common.bash
 
 function usage()
 {
@@ -46,17 +46,25 @@ while getopts "g:hv" opt; do
     esac
 done
 
-RUNTIME_log_stderr () { cat >> test.log; }
+# RUNTIME_log_file will be configured later
+RUNTIME_log_stderr () { cat >> $RUNTIME_log_file; }
 RUNTIME_log_stdout () {
     if [ "$PRETTY_PRINT_STACKS" = "yes" ]; then
-        ./scripts/pretty_print_stacks.py $1 >> test.log
+        ./scripts/pretty_print_stacks.py $1 >> $RUNTIME_log_file
     else
-        cat >> test.log
+        cat >> $RUNTIME_log_file
     fi
 }
 
-
 config=$TEST_DIR/unittests.cfg
-rm -f test.log
-printf "BUILD_HEAD=$(cat build-head)\n\n" > test.log
+
+rm -rf $unittest_log_dir.old
+if ! mv $unittest_log_dir $unittest_log_dir.log; then
+    echo "Failed to backup $unittest_log_dir"
+    exit 2
+fi
+
+mkdir $unittest_log_dir
+echo "BUILD_HEAD=$(cat build-head)" > $unittest_log_dir/SUMMARY
+
 for_each_unittest $config run
diff --git a/scripts/functions.bash b/scripts/common.bash
similarity index 75%
rename from scripts/functions.bash
rename to scripts/common.bash
index ee9143c..2dd7360 100644
--- a/scripts/functions.bash
+++ b/scripts/common.bash
@@ -1,3 +1,12 @@
+: ${unittest_log_dir:=logs}
+
+function run_task()
+{
+	local testname="$2"
+
+	RUNTIME_log_file="${unittest_log_dir}/${testname}.log"
+	"$@"
+}
 
 function for_each_unittest()
 {
@@ -17,7 +26,7 @@ function for_each_unittest()
 
 	while read -u $fd line; do
 		if [[ "$line" =~ ^\[(.*)\]$ ]]; then
-			"$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
+			run_task "$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
 			testname=${BASH_REMATCH[1]}
 			smp=1
 			kernel=""
@@ -45,6 +54,6 @@ function for_each_unittest()
 			timeout=${BASH_REMATCH[1]}
 		fi
 	done
-	"$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
+	run_task "$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
 	exec {fd}<&-
 }
diff --git a/scripts/mkstandalone.sh b/scripts/mkstandalone.sh
index d2bae19..3c1938e 100755
--- a/scripts/mkstandalone.sh
+++ b/scripts/mkstandalone.sh
@@ -5,7 +5,7 @@ if [ ! -f config.mak ]; then
 	exit 1
 fi
 source config.mak
-source scripts/functions.bash
+source scripts/common.bash
 
 escape ()
 {
-- 
2.7.4

[Qemu-devel] [kvm-unit-tests PATCH v4 1/2] run_tests: put logs into per-test file

[Peter Xu]

We were using test.log before to keep all the test logs. This patch
creates one log file per test case under logs/ directory with name
"TESTNAME.log". Meanwhile, we will keep the last time log into
logs.old/.

Renaming scripts/functions.bash into scripts/common.bash to store some
more global variables.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 .gitignore                              |  3 ++-
 Makefile                                |  4 ++--
 run_tests.sh                            | 22 +++++++++++++++-------
 scripts/{functions.bash => common.bash} | 13 +++++++++++--
 scripts/mkstandalone.sh                 |  2 +-
 5 files changed, 31 insertions(+), 13 deletions(-)
 rename scripts/{functions.bash => common.bash} (75%)

diff --git a/.gitignore b/.gitignore
index 3155418..0dc9a39 100644
--- a/.gitignore
+++ b/.gitignore
@@ -12,7 +12,8 @@ cscope.*
 /lib/asm
 /config.mak
 /*-run
-/test.log
 /msr.out
 /tests
 /build-head
+logs
+logs.old
diff --git a/Makefile b/Makefile
index a32333b..bd8843a 100644
--- a/Makefile
+++ b/Makefile
@@ -94,9 +94,9 @@ libfdt_clean:
 	$(LIBFDT_objdir)/.*.d
 
 distclean: clean libfdt_clean
-	$(RM) lib/asm config.mak $(TEST_DIR)-run test.log msr.out cscope.* \
+	$(RM) lib/asm config.mak $(TEST_DIR)-run msr.out cscope.* \
 	      build-head
-	$(RM) -r tests
+	$(RM) -r tests logs logs.old
 
 cscope: cscope_dirs = lib lib/libfdt lib/linux $(TEST_DIR) $(ARCH_LIBDIRS) lib/asm-generic
 cscope:
diff --git a/run_tests.sh b/run_tests.sh
index 254129d..1e36d66 100755
--- a/run_tests.sh
+++ b/run_tests.sh
@@ -7,7 +7,7 @@ if [ ! -f config.mak ]; then
     exit 1
 fi
 source config.mak
-source scripts/functions.bash
+source scripts/common.bash
 
 function usage()
 {
@@ -46,17 +46,25 @@ while getopts "g:hv" opt; do
     esac
 done
 
-RUNTIME_log_stderr () { cat >> test.log; }
+# RUNTIME_log_file will be configured later
+RUNTIME_log_stderr () { cat >> $RUNTIME_log_file; }
 RUNTIME_log_stdout () {
     if [ "$PRETTY_PRINT_STACKS" = "yes" ]; then
-        ./scripts/pretty_print_stacks.py $1 >> test.log
+        ./scripts/pretty_print_stacks.py $1 >> $RUNTIME_log_file
     else
-        cat >> test.log
+        cat >> $RUNTIME_log_file
     fi
 }
 
-
 config=$TEST_DIR/unittests.cfg
-rm -f test.log
-printf "BUILD_HEAD=$(cat build-head)\n\n" > test.log
+
+rm -rf $unittest_log_dir.old
+if ! mv $unittest_log_dir $unittest_log_dir.log; then
+    echo "Failed to backup $unittest_log_dir"
+    exit 2
+fi
+
+mkdir $unittest_log_dir
+echo "BUILD_HEAD=$(cat build-head)" > $unittest_log_dir/SUMMARY
+
 for_each_unittest $config run
diff --git a/scripts/functions.bash b/scripts/common.bash
similarity index 75%
rename from scripts/functions.bash
rename to scripts/common.bash
index ee9143c..2dd7360 100644
--- a/scripts/functions.bash
+++ b/scripts/common.bash
@@ -1,3 +1,12 @@
+: ${unittest_log_dir:=logs}
+
+function run_task()
+{
+	local testname="$2"
+
+	RUNTIME_log_file="${unittest_log_dir}/${testname}.log"
+	"$@"
+}
 
 function for_each_unittest()
 {
@@ -17,7 +26,7 @@ function for_each_unittest()
 
 	while read -u $fd line; do
 		if [[ "$line" =~ ^\[(.*)\]$ ]]; then
-			"$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
+			run_task "$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
 			testname=${BASH_REMATCH[1]}
 			smp=1
 			kernel=""
@@ -45,6 +54,6 @@ function for_each_unittest()
 			timeout=${BASH_REMATCH[1]}
 		fi
 	done
-	"$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
+	run_task "$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
 	exec {fd}<&-
 }
diff --git a/scripts/mkstandalone.sh b/scripts/mkstandalone.sh
index d2bae19..3c1938e 100755
--- a/scripts/mkstandalone.sh
+++ b/scripts/mkstandalone.sh
@@ -5,7 +5,7 @@ if [ ! -f config.mak ]; then
 	exit 1
 fi
 source config.mak
-source scripts/functions.bash
+source scripts/common.bash
 
 escape ()
 {
-- 
2.7.4

[kvm-unit-tests PATCH v4 2/2] run_tests: allow run tests in parallel

[Peter Xu]

run_task.sh is getting slow. This patch is trying to make it faster by
running the tests concurrently.

We provide a new parameter "-j" for the run_tests.sh, which can be used
to specify how many run queues we want for the tests. Default queue
length is 1, which is the old behavior.

Quick test on my laptop (4 cores, 2 threads each) shows 3x speed boost:

   |-----------------+-----------|
   | command         | time used |
   |-----------------+-----------|
   | run_test.sh     | 75s       |
   | run_test.sh -j8 | 27s       |
   |-----------------+-----------|

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 run_tests.sh        | 12 ++++++++++--
 scripts/common.bash | 16 +++++++++++++++-
 2 files changed, 25 insertions(+), 3 deletions(-)

diff --git a/run_tests.sh b/run_tests.sh
index 1e36d66..795cf73 100755
--- a/run_tests.sh
+++ b/run_tests.sh
@@ -13,10 +13,11 @@ function usage()
 {
 cat <<EOF
 
-Usage: $0 [-g group] [-h] [-v]
+Usage: $0 [-g group] [-h] [-v] [-j num_run_queues]
 
     -g: Only execute tests in the given group
     -h: Output this help text
+    -j: Execute tests in parallel
     -v: Enables verbose mode
 
 Set the environment variable QEMU=/path/to/qemu-system-ARCH to
@@ -28,7 +29,7 @@ EOF
 RUNTIME_arch_run="./$TEST_DIR/run"
 source scripts/runtime.bash
 
-while getopts "g:hv" opt; do
+while getopts "g:hj:v" opt; do
     case $opt in
         g)
             only_group=$OPTARG
@@ -37,6 +38,13 @@ while getopts "g:hv" opt; do
             usage
             exit
             ;;
+        j)
+            unittest_run_queues=$OPTARG
+            if ! (( "$unittest_run_queues" > 0 )); then
+                echo "Invalid -j option: $unittest_run_queues"
+                exit 1
+            fi
+            ;;
         v)
             verbose="yes"
             ;;
diff --git a/scripts/common.bash b/scripts/common.bash
index 2dd7360..83aebf8 100644
--- a/scripts/common.bash
+++ b/scripts/common.bash
@@ -1,11 +1,19 @@
 : ${unittest_log_dir:=logs}
+: ${unittest_run_queues:=1}
 
 function run_task()
 {
 	local testname="$2"
 
+	while (( "$(jobs | wc -l)" == $unittest_run_queues )); do
+		# wait for any background test to finish
+		wait -n
+	done
+
 	RUNTIME_log_file="${unittest_log_dir}/${testname}.log"
-	"$@"
+
+	# start the testcase in the background
+	"$@" &
 }
 
 function for_each_unittest()
@@ -22,6 +30,8 @@ function for_each_unittest()
 	local accel
 	local timeout
 
+	trap "wait; exit 130" SIGINT
+
 	exec {fd}<"$unittests"
 
 	while read -u $fd line; do
@@ -55,5 +65,9 @@ function for_each_unittest()
 		fi
 	done
 	run_task "$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
+
+	# wait all task finish
+	wait
+
 	exec {fd}<&-
 }
-- 
2.7.4

[Qemu-devel] [kvm-unit-tests PATCH v4 2/2] run_tests: allow run tests in parallel

[Peter Xu]

run_task.sh is getting slow. This patch is trying to make it faster by
running the tests concurrently.

We provide a new parameter "-j" for the run_tests.sh, which can be used
to specify how many run queues we want for the tests. Default queue
length is 1, which is the old behavior.

Quick test on my laptop (4 cores, 2 threads each) shows 3x speed boost:

   |-----------------+-----------|
   | command         | time used |
   |-----------------+-----------|
   | run_test.sh     | 75s       |
   | run_test.sh -j8 | 27s       |
   |-----------------+-----------|

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 run_tests.sh        | 12 ++++++++++--
 scripts/common.bash | 16 +++++++++++++++-
 2 files changed, 25 insertions(+), 3 deletions(-)

diff --git a/run_tests.sh b/run_tests.sh
index 1e36d66..795cf73 100755
--- a/run_tests.sh
+++ b/run_tests.sh
@@ -13,10 +13,11 @@ function usage()
 {
 cat <<EOF
 
-Usage: $0 [-g group] [-h] [-v]
+Usage: $0 [-g group] [-h] [-v] [-j num_run_queues]
 
     -g: Only execute tests in the given group
     -h: Output this help text
+    -j: Execute tests in parallel
     -v: Enables verbose mode
 
 Set the environment variable QEMU=/path/to/qemu-system-ARCH to
@@ -28,7 +29,7 @@ EOF
 RUNTIME_arch_run="./$TEST_DIR/run"
 source scripts/runtime.bash
 
-while getopts "g:hv" opt; do
+while getopts "g:hj:v" opt; do
     case $opt in
         g)
             only_group=$OPTARG
@@ -37,6 +38,13 @@ while getopts "g:hv" opt; do
             usage
             exit
             ;;
+        j)
+            unittest_run_queues=$OPTARG
+            if ! (( "$unittest_run_queues" > 0 )); then
+                echo "Invalid -j option: $unittest_run_queues"
+                exit 1
+            fi
+            ;;
         v)
             verbose="yes"
             ;;
diff --git a/scripts/common.bash b/scripts/common.bash
index 2dd7360..83aebf8 100644
--- a/scripts/common.bash
+++ b/scripts/common.bash
@@ -1,11 +1,19 @@
 : ${unittest_log_dir:=logs}
+: ${unittest_run_queues:=1}
 
 function run_task()
 {
 	local testname="$2"
 
+	while (( "$(jobs | wc -l)" == $unittest_run_queues )); do
+		# wait for any background test to finish
+		wait -n
+	done
+
 	RUNTIME_log_file="${unittest_log_dir}/${testname}.log"
-	"$@"
+
+	# start the testcase in the background
+	"$@" &
 }
 
 function for_each_unittest()
@@ -22,6 +30,8 @@ function for_each_unittest()
 	local accel
 	local timeout
 
+	trap "wait; exit 130" SIGINT
+
 	exec {fd}<"$unittests"
 
 	while read -u $fd line; do
@@ -55,5 +65,9 @@ function for_each_unittest()
 		fi
 	done
 	run_task "$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
+
+	# wait all task finish
+	wait
+
 	exec {fd}<&-
 }
-- 
2.7.4

Re: [Qemu-devel] [kvm-unit-tests PATCH v4 1/2] run_tests: put logs into per-test file

[Andrew Jones]

On Mon, Jan 09, 2017 at 12:04:53PM +0800, Peter Xu wrote:
> We were using test.log before to keep all the test logs. This patch
> creates one log file per test case under logs/ directory with name
> "TESTNAME.log". Meanwhile, we will keep the last time log into
> logs.old/.
> 
> Renaming scripts/functions.bash into scripts/common.bash to store some
> more global variables.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  .gitignore                              |  3 ++-
>  Makefile                                |  4 ++--
>  run_tests.sh                            | 22 +++++++++++++++-------
>  scripts/{functions.bash => common.bash} | 13 +++++++++++--
>  scripts/mkstandalone.sh                 |  2 +-
>  5 files changed, 31 insertions(+), 13 deletions(-)
>  rename scripts/{functions.bash => common.bash} (75%)
> 
> diff --git a/.gitignore b/.gitignore
> index 3155418..0dc9a39 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -12,7 +12,8 @@ cscope.*
>  /lib/asm
>  /config.mak
>  /*-run
> -/test.log
>  /msr.out
>  /tests
>  /build-head
> +logs
> +logs.old

Without the leading '/' logs[.old] files or dirs in subdirs will
also be ignored. You can add trailing '/' to all the dirs in
.gitignore too in order to ensure we only ignore dirs.

> diff --git a/Makefile b/Makefile
> index a32333b..bd8843a 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -94,9 +94,9 @@ libfdt_clean:
>  	$(LIBFDT_objdir)/.*.d
>  
>  distclean: clean libfdt_clean
> -	$(RM) lib/asm config.mak $(TEST_DIR)-run test.log msr.out cscope.* \
> +	$(RM) lib/asm config.mak $(TEST_DIR)-run msr.out cscope.* \
>  	      build-head

Can move build-head up now

> -	$(RM) -r tests
> +	$(RM) -r tests logs logs.old
>  
>  cscope: cscope_dirs = lib lib/libfdt lib/linux $(TEST_DIR) $(ARCH_LIBDIRS) lib/asm-generic
>  cscope:
> diff --git a/run_tests.sh b/run_tests.sh
> index 254129d..1e36d66 100755
> --- a/run_tests.sh
> +++ b/run_tests.sh
> @@ -7,7 +7,7 @@ if [ ! -f config.mak ]; then
>      exit 1
>  fi
>  source config.mak
> -source scripts/functions.bash
> +source scripts/common.bash
>  
>  function usage()
>  {
> @@ -46,17 +46,25 @@ while getopts "g:hv" opt; do
>      esac
>  done
>  
> -RUNTIME_log_stderr () { cat >> test.log; }
> +# RUNTIME_log_file will be configured later
> +RUNTIME_log_stderr () { cat >> $RUNTIME_log_file; }
>  RUNTIME_log_stdout () {
>      if [ "$PRETTY_PRINT_STACKS" = "yes" ]; then
> -        ./scripts/pretty_print_stacks.py $1 >> test.log
> +        ./scripts/pretty_print_stacks.py $1 >> $RUNTIME_log_file
>      else
> -        cat >> test.log
> +        cat >> $RUNTIME_log_file
>      fi
>  }
>  
> -
>  config=$TEST_DIR/unittests.cfg
> -rm -f test.log
> -printf "BUILD_HEAD=$(cat build-head)\n\n" > test.log
> +
> +rm -rf $unittest_log_dir.old
> +if ! mv $unittest_log_dir $unittest_log_dir.log; then

You remove the destination above, so this should never fail.

> +    echo "Failed to backup $unittest_log_dir"
> +    exit 2
> +fi
> +
> +mkdir $unittest_log_dir
> +echo "BUILD_HEAD=$(cat build-head)" > $unittest_log_dir/SUMMARY
> +
>  for_each_unittest $config run
> diff --git a/scripts/functions.bash b/scripts/common.bash
> similarity index 75%
> rename from scripts/functions.bash
> rename to scripts/common.bash
> index ee9143c..2dd7360 100644
> --- a/scripts/functions.bash
> +++ b/scripts/common.bash
> @@ -1,3 +1,12 @@
> +: ${unittest_log_dir:=logs}
> +
> +function run_task()
> +{
> +	local testname="$2"
> +
> +	RUNTIME_log_file="${unittest_log_dir}/${testname}.log"
> +	"$@"
> +}
>  
>  function for_each_unittest()
>  {
> @@ -17,7 +26,7 @@ function for_each_unittest()
>  
>  	while read -u $fd line; do
>  		if [[ "$line" =~ ^\[(.*)\]$ ]]; then
> -			"$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
> +			run_task "$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
>  			testname=${BASH_REMATCH[1]}
>  			smp=1
>  			kernel=""
> @@ -45,6 +54,6 @@ function for_each_unittest()
>  			timeout=${BASH_REMATCH[1]}
>  		fi
>  	done
> -	"$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
> +	run_task "$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
>  	exec {fd}<&-
>  }
> diff --git a/scripts/mkstandalone.sh b/scripts/mkstandalone.sh
> index d2bae19..3c1938e 100755
> --- a/scripts/mkstandalone.sh
> +++ b/scripts/mkstandalone.sh
> @@ -5,7 +5,7 @@ if [ ! -f config.mak ]; then
>  	exit 1
>  fi
>  source config.mak
> -source scripts/functions.bash
> +source scripts/common.bash
>  
>  escape ()
>  {
> -- 
> 2.7.4
> 
> 

thanks,
drew

Re: [Qemu-devel] [kvm-unit-tests PATCH v4 2/2] run_tests: allow run tests in parallel

[Andrew Jones]

On Mon, Jan 09, 2017 at 12:04:54PM +0800, Peter Xu wrote:
> run_task.sh is getting slow. This patch is trying to make it faster by
> running the tests concurrently.
> 
> We provide a new parameter "-j" for the run_tests.sh, which can be used
> to specify how many run queues we want for the tests. Default queue
> length is 1, which is the old behavior.
> 
> Quick test on my laptop (4 cores, 2 threads each) shows 3x speed boost:
> 
>    |-----------------+-----------|
>    | command         | time used |
>    |-----------------+-----------|
>    | run_test.sh     | 75s       |
>    | run_test.sh -j8 | 27s       |
>    |-----------------+-----------|
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  run_tests.sh        | 12 ++++++++++--
>  scripts/common.bash | 16 +++++++++++++++-
>  2 files changed, 25 insertions(+), 3 deletions(-)
> 
> diff --git a/run_tests.sh b/run_tests.sh
> index 1e36d66..795cf73 100755
> --- a/run_tests.sh
> +++ b/run_tests.sh
> @@ -13,10 +13,11 @@ function usage()
>  {
>  cat <<EOF
>  
> -Usage: $0 [-g group] [-h] [-v]
> +Usage: $0 [-g group] [-h] [-v] [-j num_run_queues]
>  
>      -g: Only execute tests in the given group
>      -h: Output this help text
> +    -j: Execute tests in parallel
>      -v: Enables verbose mode
>  
>  Set the environment variable QEMU=/path/to/qemu-system-ARCH to
> @@ -28,7 +29,7 @@ EOF
>  RUNTIME_arch_run="./$TEST_DIR/run"
>  source scripts/runtime.bash
>  
> -while getopts "g:hv" opt; do
> +while getopts "g:hj:v" opt; do
>      case $opt in
>          g)
>              only_group=$OPTARG
> @@ -37,6 +38,13 @@ while getopts "g:hv" opt; do
>              usage
>              exit
>              ;;
> +        j)
> +            unittest_run_queues=$OPTARG
> +            if ! (( "$unittest_run_queues" > 0 )); then

The equivalent expression without '!' is (( $var <= 0 )), and
no need for the "'s around the var.

> +                echo "Invalid -j option: $unittest_run_queues"
> +                exit 1
> +            fi
> +            ;;
>          v)
>              verbose="yes"
>              ;;
> diff --git a/scripts/common.bash b/scripts/common.bash
> index 2dd7360..83aebf8 100644
> --- a/scripts/common.bash
> +++ b/scripts/common.bash
> @@ -1,11 +1,19 @@
>  : ${unittest_log_dir:=logs}
> +: ${unittest_run_queues:=1}
>  
>  function run_task()
>  {
>  	local testname="$2"
>  
> +	while (( "$(jobs | wc -l)" == $unittest_run_queues )); do

Why the "'s? The result must resolve to an int, otherwise we don't
want ((...))

> +		# wait for any background test to finish
> +		wait -n
> +	done
> +
>  	RUNTIME_log_file="${unittest_log_dir}/${testname}.log"
> -	"$@"
> +
> +	# start the testcase in the background
> +	"$@" &
>  }
>  
>  function for_each_unittest()
> @@ -22,6 +30,8 @@ function for_each_unittest()
>  	local accel
>  	local timeout
>  
> +	trap "wait; exit 130" SIGINT
> +
>  	exec {fd}<"$unittests"
>  
>  	while read -u $fd line; do
> @@ -55,5 +65,9 @@ function for_each_unittest()
>  		fi
>  	done
>  	run_task "$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
> +
> +	# wait all task finish

wait until all tasks finish

> +	wait
> +
>  	exec {fd}<&-
>  }
> -- 
> 2.7.4
> 
> 

Re: [Qemu-devel] [kvm-unit-tests PATCH v4 1/2] run_tests: put logs into per-test file

[Peter Xu]

On Tue, Jan 10, 2017 at 06:28:41PM +0100, Andrew Jones wrote:
> On Mon, Jan 09, 2017 at 12:04:53PM +0800, Peter Xu wrote:
> > We were using test.log before to keep all the test logs. This patch
> > creates one log file per test case under logs/ directory with name
> > "TESTNAME.log". Meanwhile, we will keep the last time log into
> > logs.old/.
> > 
> > Renaming scripts/functions.bash into scripts/common.bash to store some
> > more global variables.
> > 
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >  .gitignore                              |  3 ++-
> >  Makefile                                |  4 ++--
> >  run_tests.sh                            | 22 +++++++++++++++-------
> >  scripts/{functions.bash => common.bash} | 13 +++++++++++--
> >  scripts/mkstandalone.sh                 |  2 +-
> >  5 files changed, 31 insertions(+), 13 deletions(-)
> >  rename scripts/{functions.bash => common.bash} (75%)
> > 
> > diff --git a/.gitignore b/.gitignore
> > index 3155418..0dc9a39 100644
> > --- a/.gitignore
> > +++ b/.gitignore
> > @@ -12,7 +12,8 @@ cscope.*
> >  /lib/asm
> >  /config.mak
> >  /*-run
> > -/test.log
> >  /msr.out
> >  /tests
> >  /build-head
> > +logs
> > +logs.old
> 
> Without the leading '/' logs[.old] files or dirs in subdirs will
> also be ignored. You can add trailing '/' to all the dirs in
> .gitignore too in order to ensure we only ignore dirs.

Will do.

> 
> > diff --git a/Makefile b/Makefile
> > index a32333b..bd8843a 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -94,9 +94,9 @@ libfdt_clean:
> >  	$(LIBFDT_objdir)/.*.d
> >  
> >  distclean: clean libfdt_clean
> > -	$(RM) lib/asm config.mak $(TEST_DIR)-run test.log msr.out cscope.* \
> > +	$(RM) lib/asm config.mak $(TEST_DIR)-run msr.out cscope.* \
> >  	      build-head
> 
> Can move build-head up now

Yep.

[...]

> > -
> >  config=$TEST_DIR/unittests.cfg
> > -rm -f test.log
> > -printf "BUILD_HEAD=$(cat build-head)\n\n" > test.log
> > +
> > +rm -rf $unittest_log_dir.old
> > +if ! mv $unittest_log_dir $unittest_log_dir.log; then
> 
> You remove the destination above, so this should never fail.

Hmm looks so... :) Let me fix.

(And I used rm -rf again, but with ".old" it looks much safer so I
 dare to use it)

Thanks,

-- peterx

Re: [Qemu-devel] [kvm-unit-tests PATCH v4 2/2] run_tests: allow run tests in parallel

[Peter Xu]

On Tue, Jan 10, 2017 at 06:39:59PM +0100, Andrew Jones wrote:
> On Mon, Jan 09, 2017 at 12:04:54PM +0800, Peter Xu wrote:
> > run_task.sh is getting slow. This patch is trying to make it faster by
> > running the tests concurrently.
> > 
> > We provide a new parameter "-j" for the run_tests.sh, which can be used
> > to specify how many run queues we want for the tests. Default queue
> > length is 1, which is the old behavior.
> > 
> > Quick test on my laptop (4 cores, 2 threads each) shows 3x speed boost:
> > 
> >    |-----------------+-----------|
> >    | command         | time used |
> >    |-----------------+-----------|
> >    | run_test.sh     | 75s       |
> >    | run_test.sh -j8 | 27s       |
> >    |-----------------+-----------|
> > 
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >  run_tests.sh        | 12 ++++++++++--
> >  scripts/common.bash | 16 +++++++++++++++-
> >  2 files changed, 25 insertions(+), 3 deletions(-)
> > 
> > diff --git a/run_tests.sh b/run_tests.sh
> > index 1e36d66..795cf73 100755
> > --- a/run_tests.sh
> > +++ b/run_tests.sh
> > @@ -13,10 +13,11 @@ function usage()
> >  {
> >  cat <<EOF
> >  
> > -Usage: $0 [-g group] [-h] [-v]
> > +Usage: $0 [-g group] [-h] [-v] [-j num_run_queues]
> >  
> >      -g: Only execute tests in the given group
> >      -h: Output this help text
> > +    -j: Execute tests in parallel
> >      -v: Enables verbose mode
> >  
> >  Set the environment variable QEMU=/path/to/qemu-system-ARCH to
> > @@ -28,7 +29,7 @@ EOF
> >  RUNTIME_arch_run="./$TEST_DIR/run"
> >  source scripts/runtime.bash
> >  
> > -while getopts "g:hv" opt; do
> > +while getopts "g:hj:v" opt; do
> >      case $opt in
> >          g)
> >              only_group=$OPTARG
> > @@ -37,6 +38,13 @@ while getopts "g:hv" opt; do
> >              usage
> >              exit
> >              ;;
> > +        j)
> > +            unittest_run_queues=$OPTARG
> > +            if ! (( "$unittest_run_queues" > 0 )); then
> 
> The equivalent expression without '!' is (( $var <= 0 )), and
> no need for the "'s around the var.
> 
> > +                echo "Invalid -j option: $unittest_run_queues"
> > +                exit 1
> > +            fi
> > +            ;;
> >          v)
> >              verbose="yes"
> >              ;;
> > diff --git a/scripts/common.bash b/scripts/common.bash
> > index 2dd7360..83aebf8 100644
> > --- a/scripts/common.bash
> > +++ b/scripts/common.bash
> > @@ -1,11 +1,19 @@
> >  : ${unittest_log_dir:=logs}
> > +: ${unittest_run_queues:=1}
> >  
> >  function run_task()
> >  {
> >  	local testname="$2"
> >  
> > +	while (( "$(jobs | wc -l)" == $unittest_run_queues )); do
> 
> Why the "'s? The result must resolve to an int, otherwise we don't
> want ((...))
> 
> > +		# wait for any background test to finish
> > +		wait -n
> > +	done
> > +
> >  	RUNTIME_log_file="${unittest_log_dir}/${testname}.log"
> > -	"$@"
> > +
> > +	# start the testcase in the background
> > +	"$@" &
> >  }
> >  
> >  function for_each_unittest()
> > @@ -22,6 +30,8 @@ function for_each_unittest()
> >  	local accel
> >  	local timeout
> >  
> > +	trap "wait; exit 130" SIGINT
> > +
> >  	exec {fd}<"$unittests"
> >  
> >  	while read -u $fd line; do
> > @@ -55,5 +65,9 @@ function for_each_unittest()
> >  		fi
> >  	done
> >  	run_task "$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout"
> > +
> > +	# wait all task finish
> 
> wait until all tasks finish

Fixing all three places, thanks!

-- peterx