From: Peter Xu <peterx@redhat.com> To: qemu-devel@nongnu.org, kvm@vger.kernel.org Cc: "Paolo Bonzini" <pbonzini@redhat.com>, "Andrew Jones" <drjones@redhat.com>, peterx@redhat.com, "Radim Krčmář" <rkrcmar@redhat.com> Subject: [kvm-unit-tests PATCH v6 3/3] run_tests: allow run tests in parallel Date: Thu, 12 Jan 2017 11:36:22 +0800 [thread overview] Message-ID: <1484192182-13760-4-git-send-email-peterx@redhat.com> (raw) In-Reply-To: <1484192182-13760-1-git-send-email-peterx@redhat.com> run_task.sh is getting slow. This patch is trying to make it faster by running the tests concurrently. We provide a new parameter "-j" for the run_tests.sh, which can be used to specify how many run queues we want for the tests. Default queue length is 1, which is the old behavior. Quick test on my laptop (4 cores, 2 threads each) shows 3x speed boost: |-----------------+-----------| | command | time used | |-----------------+-----------| | run_test.sh | 75s | | run_test.sh -j8 | 27s | |-----------------+-----------| Signed-off-by: Peter Xu <peterx@redhat.com> --- run_tests.sh | 12 ++++++++++-- scripts/common.bash | 16 +++++++++++++++- 2 files changed, 25 insertions(+), 3 deletions(-) diff --git a/run_tests.sh b/run_tests.sh index afd3d95..4d57ff9 100755 --- a/run_tests.sh +++ b/run_tests.sh @@ -13,10 +13,11 @@ function usage() { cat <<EOF -Usage: $0 [-g group] [-h] [-v] +Usage: $0 [-g group] [-h] [-v] [-j num_run_queues] -g: Only execute tests in the given group -h: Output this help text + -j: Execute tests in parallel -v: Enables verbose mode Set the environment variable QEMU=/path/to/qemu-system-ARCH to @@ -28,7 +29,7 @@ EOF RUNTIME_arch_run="./$TEST_DIR/run" source scripts/runtime.bash -while getopts "g:hv" opt; do +while getopts "g:hj:v" opt; do case $opt in g) only_group=$OPTARG @@ -37,6 +38,13 @@ while getopts "g:hv" opt; do usage exit ;; + j) + unittest_run_queues=$OPTARG + if (( $unittest_run_queues <= 0 )); then + echo "Invalid -j option: $unittest_run_queues" + exit 2 + fi + ;; v) verbose="yes" ;; diff --git a/scripts/common.bash b/scripts/common.bash index 2dd7360..9bd560f 100644 --- a/scripts/common.bash +++ b/scripts/common.bash @@ -1,11 +1,19 @@ : ${unittest_log_dir:=logs} +: ${unittest_run_queues:=1} function run_task() { local testname="$2" + while (( $(jobs | wc -l) == $unittest_run_queues )); do + # wait for any background test to finish + wait -n + done + RUNTIME_log_file="${unittest_log_dir}/${testname}.log" - "$@" + + # start the testcase in the background + "$@" & } function for_each_unittest() @@ -22,6 +30,8 @@ function for_each_unittest() local accel local timeout + trap "wait; exit 130" SIGINT + exec {fd}<"$unittests" while read -u $fd line; do @@ -55,5 +65,9 @@ function for_each_unittest() fi done run_task "$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout" + + # wait until all tasks finish + wait + exec {fd}<&- } -- 2.7.4
WARNING: multiple messages have this Message-ID (diff)
From: Peter Xu <peterx@redhat.com> To: qemu-devel@nongnu.org, kvm@vger.kernel.org Cc: "Paolo Bonzini" <pbonzini@redhat.com>, "Andrew Jones" <drjones@redhat.com>, peterx@redhat.com, "Radim Krčmář" <rkrcmar@redhat.com> Subject: [Qemu-devel] [kvm-unit-tests PATCH v6 3/3] run_tests: allow run tests in parallel Date: Thu, 12 Jan 2017 11:36:22 +0800 [thread overview] Message-ID: <1484192182-13760-4-git-send-email-peterx@redhat.com> (raw) In-Reply-To: <1484192182-13760-1-git-send-email-peterx@redhat.com> run_task.sh is getting slow. This patch is trying to make it faster by running the tests concurrently. We provide a new parameter "-j" for the run_tests.sh, which can be used to specify how many run queues we want for the tests. Default queue length is 1, which is the old behavior. Quick test on my laptop (4 cores, 2 threads each) shows 3x speed boost: |-----------------+-----------| | command | time used | |-----------------+-----------| | run_test.sh | 75s | | run_test.sh -j8 | 27s | |-----------------+-----------| Signed-off-by: Peter Xu <peterx@redhat.com> --- run_tests.sh | 12 ++++++++++-- scripts/common.bash | 16 +++++++++++++++- 2 files changed, 25 insertions(+), 3 deletions(-) diff --git a/run_tests.sh b/run_tests.sh index afd3d95..4d57ff9 100755 --- a/run_tests.sh +++ b/run_tests.sh @@ -13,10 +13,11 @@ function usage() { cat <<EOF -Usage: $0 [-g group] [-h] [-v] +Usage: $0 [-g group] [-h] [-v] [-j num_run_queues] -g: Only execute tests in the given group -h: Output this help text + -j: Execute tests in parallel -v: Enables verbose mode Set the environment variable QEMU=/path/to/qemu-system-ARCH to @@ -28,7 +29,7 @@ EOF RUNTIME_arch_run="./$TEST_DIR/run" source scripts/runtime.bash -while getopts "g:hv" opt; do +while getopts "g:hj:v" opt; do case $opt in g) only_group=$OPTARG @@ -37,6 +38,13 @@ while getopts "g:hv" opt; do usage exit ;; + j) + unittest_run_queues=$OPTARG + if (( $unittest_run_queues <= 0 )); then + echo "Invalid -j option: $unittest_run_queues" + exit 2 + fi + ;; v) verbose="yes" ;; diff --git a/scripts/common.bash b/scripts/common.bash index 2dd7360..9bd560f 100644 --- a/scripts/common.bash +++ b/scripts/common.bash @@ -1,11 +1,19 @@ : ${unittest_log_dir:=logs} +: ${unittest_run_queues:=1} function run_task() { local testname="$2" + while (( $(jobs | wc -l) == $unittest_run_queues )); do + # wait for any background test to finish + wait -n + done + RUNTIME_log_file="${unittest_log_dir}/${testname}.log" - "$@" + + # start the testcase in the background + "$@" & } function for_each_unittest() @@ -22,6 +30,8 @@ function for_each_unittest() local accel local timeout + trap "wait; exit 130" SIGINT + exec {fd}<"$unittests" while read -u $fd line; do @@ -55,5 +65,9 @@ function for_each_unittest() fi done run_task "$cmd" "$testname" "$groups" "$smp" "$kernel" "$opts" "$arch" "$check" "$accel" "$timeout" + + # wait until all tasks finish + wait + exec {fd}<&- } -- 2.7.4
next prev parent reply other threads:[~2017-01-12 3:36 UTC|newest] Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top 2017-01-12 3:36 [kvm-unit-tests PATCH v6 0/3] run_tests: support concurrent test execution Peter Xu 2017-01-12 3:36 ` [Qemu-devel] " Peter Xu 2017-01-12 3:36 ` [kvm-unit-tests PATCH v6 1/3] run_tests: fix errno for param parsing Peter Xu 2017-01-12 3:36 ` [Qemu-devel] " Peter Xu 2017-01-12 11:42 ` Andrew Jones 2017-01-12 3:36 ` [kvm-unit-tests PATCH v6 2/3] run_tests: put logs into per-test file Peter Xu 2017-01-12 3:36 ` [Qemu-devel] " Peter Xu 2017-01-12 10:04 ` Andrew Jones 2017-01-12 10:35 ` Peter Xu 2017-01-12 12:55 ` Paolo Bonzini 2017-01-12 12:55 ` [Qemu-devel] " Paolo Bonzini 2017-01-12 14:14 ` Peter Xu 2017-01-12 14:14 ` [Qemu-devel] " Peter Xu 2017-01-12 3:36 ` Peter Xu [this message] 2017-01-12 3:36 ` [Qemu-devel] [kvm-unit-tests PATCH v6 3/3] run_tests: allow run tests in parallel Peter Xu 2017-01-12 11:42 ` Andrew Jones 2017-01-12 13:31 ` Paolo Bonzini
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1484192182-13760-4-git-send-email-peterx@redhat.com \ --to=peterx@redhat.com \ --cc=drjones@redhat.com \ --cc=kvm@vger.kernel.org \ --cc=pbonzini@redhat.com \ --cc=qemu-devel@nongnu.org \ --cc=rkrcmar@redhat.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.