* [kvm-unit-tests PATCH 0/2] Use same test names in the default and the TAP13 output format
@ 2020-08-21 12:37 Marc Hartmayer
2020-08-21 12:37 ` [kvm-unit-tests PATCH 1/2] runtime.bash: remove outdated comment Marc Hartmayer
2020-08-21 12:37 ` [kvm-unit-tests PATCH 2/2] Use same test names in the default and the TAP13 output format Marc Hartmayer
0 siblings, 2 replies; 8+ messages in thread
From: Marc Hartmayer @ 2020-08-21 12:37 UTC (permalink / raw)
To: kvm
Cc: Thomas Huth, David Hildenbrand, Janosch Frank, Cornelia Huck,
Andrew Jones, Paolo Bonzini
See patches description.
For everybody's convenience there is a branch:
https://gitlab.com/mhartmay/kvm-unit-tests/-/tree/tap
Marc Hartmayer (2):
runtime.bash: remove outdated comment
Use same test names in the default and the TAP13 output format
run_tests.sh | 15 +++++++++------
scripts/runtime.bash | 9 +++------
2 files changed, 12 insertions(+), 12 deletions(-)
--
2.25.4
^ permalink raw reply [flat|nested] 8+ messages in thread
* [kvm-unit-tests PATCH 1/2] runtime.bash: remove outdated comment
2020-08-21 12:37 [kvm-unit-tests PATCH 0/2] Use same test names in the default and the TAP13 output format Marc Hartmayer
@ 2020-08-21 12:37 ` Marc Hartmayer
2020-08-21 13:16 ` Andrew Jones
2020-08-24 12:48 ` Cornelia Huck
2020-08-21 12:37 ` [kvm-unit-tests PATCH 2/2] Use same test names in the default and the TAP13 output format Marc Hartmayer
1 sibling, 2 replies; 8+ messages in thread
From: Marc Hartmayer @ 2020-08-21 12:37 UTC (permalink / raw)
To: kvm
Cc: Thomas Huth, David Hildenbrand, Janosch Frank, Cornelia Huck,
Andrew Jones, Paolo Bonzini
Since commit 6e1d3752d7ca ("tap13: list testcases individually") the
comment is no longer valid. Therefore let's remove it.
Signed-off-by: Marc Hartmayer <mhartmay@linux.ibm.com>
---
scripts/runtime.bash | 3 ---
1 file changed, 3 deletions(-)
diff --git a/scripts/runtime.bash b/scripts/runtime.bash
index c88e246245a6..caa4c5ba18cc 100644
--- a/scripts/runtime.bash
+++ b/scripts/runtime.bash
@@ -53,9 +53,6 @@ skip_nodefault()
function print_result()
{
- # output test results in a TAP format
- # https://testanything.org/tap-version-13-specification.html
-
local status="$1"
local testname="$2"
local summary="$3"
--
2.25.4
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [kvm-unit-tests PATCH 2/2] Use same test names in the default and the TAP13 output format
2020-08-21 12:37 [kvm-unit-tests PATCH 0/2] Use same test names in the default and the TAP13 output format Marc Hartmayer
2020-08-21 12:37 ` [kvm-unit-tests PATCH 1/2] runtime.bash: remove outdated comment Marc Hartmayer
@ 2020-08-21 12:37 ` Marc Hartmayer
2020-08-21 13:16 ` Andrew Jones
2020-08-21 14:05 ` Janosch Frank
1 sibling, 2 replies; 8+ messages in thread
From: Marc Hartmayer @ 2020-08-21 12:37 UTC (permalink / raw)
To: kvm
Cc: Thomas Huth, David Hildenbrand, Janosch Frank, Cornelia Huck,
Andrew Jones, Paolo Bonzini
Use the same test names in the TAP13 output as in the default output
format. This makes the output more consistent. To achieve this, we
need to pass the test name as an argument to the function
`process_test_output`.
Before this change:
$ ./run_tests.sh
PASS selftest-setup (14 tests)
...
vs.
$ ./run_tests.sh -t
TAP version 13
ok 1 - selftest: true
ok 2 - selftest: argc == 3
...
After this change:
$ ./run_tests.sh
PASS selftest-setup (14 tests)
...
vs.
$ ./run_tests.sh -t
TAP version 13
ok 1 - selftest-setup: true
ok 2 - selftest-setup: argc == 3
...
While at it, introduce a local variable `kernel` in
`RUNTIME_log_stdout` since this makes the function easier to read.
Signed-off-by: Marc Hartmayer <mhartmay@linux.ibm.com>
---
run_tests.sh | 15 +++++++++------
scripts/runtime.bash | 6 +++---
2 files changed, 12 insertions(+), 9 deletions(-)
diff --git a/run_tests.sh b/run_tests.sh
index 01e36dcfa06e..b5812336866f 100755
--- a/run_tests.sh
+++ b/run_tests.sh
@@ -81,18 +81,19 @@ if [[ $tap_output == "no" ]]; then
postprocess_suite_output() { cat; }
else
process_test_output() {
+ local testname="$1"
CR=$'\r'
while read -r line; do
line="${line%$CR}"
case "${line:0:4}" in
PASS)
- echo "ok TEST_NUMBER - ${line#??????}" >&3
+ echo "ok TEST_NUMBER - ${testname}:${line#*:*:}" >&3
;;
FAIL)
- echo "not ok TEST_NUMBER - ${line#??????}" >&3
+ echo "not ok TEST_NUMBER - ${testname}:${line#*:*:}" >&3
;;
SKIP)
- echo "ok TEST_NUMBER - ${line#??????} # skip" >&3
+ echo "ok TEST_NUMBER - ${testname}:${line#*:*:} # skip" >&3
;;
*)
;;
@@ -114,12 +115,14 @@ else
}
fi
-RUNTIME_log_stderr () { process_test_output; }
+RUNTIME_log_stderr () { process_test_output "$1"; }
RUNTIME_log_stdout () {
+ local testname="$1"
if [ "$PRETTY_PRINT_STACKS" = "yes" ]; then
- ./scripts/pretty_print_stacks.py $1 | process_test_output
+ local kernel="$2"
+ ./scripts/pretty_print_stacks.py "$kernel" | process_test_output "$testname"
else
- process_test_output
+ process_test_output "$testname"
fi
}
diff --git a/scripts/runtime.bash b/scripts/runtime.bash
index caa4c5ba18cc..294e6b15a5e2 100644
--- a/scripts/runtime.bash
+++ b/scripts/runtime.bash
@@ -140,10 +140,10 @@ function run()
# extra_params in the config file may contain backticks that need to be
# expanded, so use eval to start qemu. Use "> >(foo)" instead of a pipe to
# preserve the exit status.
- summary=$(eval $cmdline 2> >(RUNTIME_log_stderr) \
- > >(tee >(RUNTIME_log_stdout $kernel) | extract_summary))
+ summary=$(eval $cmdline 2> >(RUNTIME_log_stderr $testname) \
+ > >(tee >(RUNTIME_log_stdout $testname $kernel) | extract_summary))
ret=$?
- [ "$STANDALONE" != "yes" ] && echo > >(RUNTIME_log_stdout $kernel)
+ [ "$STANDALONE" != "yes" ] && echo > >(RUNTIME_log_stdout $testname $kernel)
if [ $ret -eq 0 ]; then
print_result "PASS" $testname "$summary"
--
2.25.4
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [kvm-unit-tests PATCH 1/2] runtime.bash: remove outdated comment
2020-08-21 12:37 ` [kvm-unit-tests PATCH 1/2] runtime.bash: remove outdated comment Marc Hartmayer
@ 2020-08-21 13:16 ` Andrew Jones
2020-08-24 12:48 ` Cornelia Huck
1 sibling, 0 replies; 8+ messages in thread
From: Andrew Jones @ 2020-08-21 13:16 UTC (permalink / raw)
To: Marc Hartmayer
Cc: kvm, Thomas Huth, David Hildenbrand, Janosch Frank,
Cornelia Huck, Paolo Bonzini
On Fri, Aug 21, 2020 at 02:37:43PM +0200, Marc Hartmayer wrote:
> Since commit 6e1d3752d7ca ("tap13: list testcases individually") the
> comment is no longer valid. Therefore let's remove it.
>
> Signed-off-by: Marc Hartmayer <mhartmay@linux.ibm.com>
> ---
> scripts/runtime.bash | 3 ---
> 1 file changed, 3 deletions(-)
>
> diff --git a/scripts/runtime.bash b/scripts/runtime.bash
> index c88e246245a6..caa4c5ba18cc 100644
> --- a/scripts/runtime.bash
> +++ b/scripts/runtime.bash
> @@ -53,9 +53,6 @@ skip_nodefault()
>
> function print_result()
> {
> - # output test results in a TAP format
> - # https://testanything.org/tap-version-13-specification.html
> -
> local status="$1"
> local testname="$2"
> local summary="$3"
> --
> 2.25.4
>
Reviewed-by: Andrew Jones <drjones@redhat.com>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [kvm-unit-tests PATCH 2/2] Use same test names in the default and the TAP13 output format
2020-08-21 12:37 ` [kvm-unit-tests PATCH 2/2] Use same test names in the default and the TAP13 output format Marc Hartmayer
@ 2020-08-21 13:16 ` Andrew Jones
2020-08-21 14:05 ` Janosch Frank
1 sibling, 0 replies; 8+ messages in thread
From: Andrew Jones @ 2020-08-21 13:16 UTC (permalink / raw)
To: Marc Hartmayer
Cc: kvm, Thomas Huth, David Hildenbrand, Janosch Frank,
Cornelia Huck, Paolo Bonzini
On Fri, Aug 21, 2020 at 02:37:44PM +0200, Marc Hartmayer wrote:
> Use the same test names in the TAP13 output as in the default output
> format. This makes the output more consistent. To achieve this, we
> need to pass the test name as an argument to the function
> `process_test_output`.
>
> Before this change:
> $ ./run_tests.sh
> PASS selftest-setup (14 tests)
> ...
>
> vs.
>
> $ ./run_tests.sh -t
> TAP version 13
> ok 1 - selftest: true
> ok 2 - selftest: argc == 3
> ...
>
> After this change:
> $ ./run_tests.sh
> PASS selftest-setup (14 tests)
> ...
>
> vs.
>
> $ ./run_tests.sh -t
> TAP version 13
> ok 1 - selftest-setup: true
> ok 2 - selftest-setup: argc == 3
> ...
>
> While at it, introduce a local variable `kernel` in
> `RUNTIME_log_stdout` since this makes the function easier to read.
>
> Signed-off-by: Marc Hartmayer <mhartmay@linux.ibm.com>
> ---
> run_tests.sh | 15 +++++++++------
> scripts/runtime.bash | 6 +++---
> 2 files changed, 12 insertions(+), 9 deletions(-)
>
> diff --git a/run_tests.sh b/run_tests.sh
> index 01e36dcfa06e..b5812336866f 100755
> --- a/run_tests.sh
> +++ b/run_tests.sh
> @@ -81,18 +81,19 @@ if [[ $tap_output == "no" ]]; then
> postprocess_suite_output() { cat; }
> else
> process_test_output() {
> + local testname="$1"
> CR=$'\r'
> while read -r line; do
> line="${line%$CR}"
> case "${line:0:4}" in
> PASS)
> - echo "ok TEST_NUMBER - ${line#??????}" >&3
> + echo "ok TEST_NUMBER - ${testname}:${line#*:*:}" >&3
> ;;
> FAIL)
> - echo "not ok TEST_NUMBER - ${line#??????}" >&3
> + echo "not ok TEST_NUMBER - ${testname}:${line#*:*:}" >&3
> ;;
> SKIP)
> - echo "ok TEST_NUMBER - ${line#??????} # skip" >&3
> + echo "ok TEST_NUMBER - ${testname}:${line#*:*:} # skip" >&3
> ;;
> *)
> ;;
> @@ -114,12 +115,14 @@ else
> }
> fi
>
> -RUNTIME_log_stderr () { process_test_output; }
> +RUNTIME_log_stderr () { process_test_output "$1"; }
> RUNTIME_log_stdout () {
> + local testname="$1"
> if [ "$PRETTY_PRINT_STACKS" = "yes" ]; then
> - ./scripts/pretty_print_stacks.py $1 | process_test_output
> + local kernel="$2"
> + ./scripts/pretty_print_stacks.py "$kernel" | process_test_output "$testname"
> else
> - process_test_output
> + process_test_output "$testname"
> fi
> }
>
> diff --git a/scripts/runtime.bash b/scripts/runtime.bash
> index caa4c5ba18cc..294e6b15a5e2 100644
> --- a/scripts/runtime.bash
> +++ b/scripts/runtime.bash
> @@ -140,10 +140,10 @@ function run()
> # extra_params in the config file may contain backticks that need to be
> # expanded, so use eval to start qemu. Use "> >(foo)" instead of a pipe to
> # preserve the exit status.
> - summary=$(eval $cmdline 2> >(RUNTIME_log_stderr) \
> - > >(tee >(RUNTIME_log_stdout $kernel) | extract_summary))
> + summary=$(eval $cmdline 2> >(RUNTIME_log_stderr $testname) \
> + > >(tee >(RUNTIME_log_stdout $testname $kernel) | extract_summary))
> ret=$?
> - [ "$STANDALONE" != "yes" ] && echo > >(RUNTIME_log_stdout $kernel)
> + [ "$STANDALONE" != "yes" ] && echo > >(RUNTIME_log_stdout $testname $kernel)
>
> if [ $ret -eq 0 ]; then
> print_result "PASS" $testname "$summary"
> --
> 2.25.4
>
Reviewed-by: Andrew Jones <drjones@redhat.com>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [kvm-unit-tests PATCH 2/2] Use same test names in the default and the TAP13 output format
2020-08-21 12:37 ` [kvm-unit-tests PATCH 2/2] Use same test names in the default and the TAP13 output format Marc Hartmayer
2020-08-21 13:16 ` Andrew Jones
@ 2020-08-21 14:05 ` Janosch Frank
2020-08-21 14:16 ` Marc Hartmayer
1 sibling, 1 reply; 8+ messages in thread
From: Janosch Frank @ 2020-08-21 14:05 UTC (permalink / raw)
To: Marc Hartmayer, kvm
Cc: Thomas Huth, David Hildenbrand, Cornelia Huck, Andrew Jones,
Paolo Bonzini
[-- Attachment #1.1: Type: text/plain, Size: 3861 bytes --]
On 8/21/20 2:37 PM, Marc Hartmayer wrote:
> Use the same test names in the TAP13 output as in the default output
> format. This makes the output more consistent. To achieve this, we
> need to pass the test name as an argument to the function
> `process_test_output`.
>
> Before this change:
> $ ./run_tests.sh
> PASS selftest-setup (14 tests)
> ...
>
> vs.
>
> $ ./run_tests.sh -t
> TAP version 13
> ok 1 - selftest: true
> ok 2 - selftest: argc == 3
> ...
>
> After this change:
> $ ./run_tests.sh
> PASS selftest-setup (14 tests)
> ...
>
> vs.
>
> $ ./run_tests.sh -t
> TAP version 13
> ok 1 - selftest-setup: true
> ok 2 - selftest-setup: argc == 3
This doesn't work for me, we can't just drop prefixes.
I.e. it needs to be "testname: prefix1: prefix2: prefixn: Test text"
selftest-setup: selftest: true
> ...
>
> While at it, introduce a local variable `kernel` in
> `RUNTIME_log_stdout` since this makes the function easier to read.
>
> Signed-off-by: Marc Hartmayer <mhartmay@linux.ibm.com>
> ---
> run_tests.sh | 15 +++++++++------
> scripts/runtime.bash | 6 +++---
> 2 files changed, 12 insertions(+), 9 deletions(-)
>
> diff --git a/run_tests.sh b/run_tests.sh
> index 01e36dcfa06e..b5812336866f 100755
> --- a/run_tests.sh
> +++ b/run_tests.sh
> @@ -81,18 +81,19 @@ if [[ $tap_output == "no" ]]; then
> postprocess_suite_output() { cat; }
> else
> process_test_output() {
> + local testname="$1"
> CR=$'\r'
> while read -r line; do
> line="${line%$CR}"
> case "${line:0:4}" in
> PASS)
> - echo "ok TEST_NUMBER - ${line#??????}" >&3
> + echo "ok TEST_NUMBER - ${testname}:${line#*:*:}" >&3
> ;;
> FAIL)
> - echo "not ok TEST_NUMBER - ${line#??????}" >&3
> + echo "not ok TEST_NUMBER - ${testname}:${line#*:*:}" >&3
> ;;
> SKIP)
> - echo "ok TEST_NUMBER - ${line#??????} # skip" >&3
> + echo "ok TEST_NUMBER - ${testname}:${line#*:*:} # skip" >&3
> ;;
> *)
> ;;
> @@ -114,12 +115,14 @@ else
> }
> fi
>
> -RUNTIME_log_stderr () { process_test_output; }
> +RUNTIME_log_stderr () { process_test_output "$1"; }
> RUNTIME_log_stdout () {
> + local testname="$1"
> if [ "$PRETTY_PRINT_STACKS" = "yes" ]; then
> - ./scripts/pretty_print_stacks.py $1 | process_test_output
> + local kernel="$2"
> + ./scripts/pretty_print_stacks.py "$kernel" | process_test_output "$testname"
> else
> - process_test_output
> + process_test_output "$testname"
> fi
> }
>
> diff --git a/scripts/runtime.bash b/scripts/runtime.bash
> index caa4c5ba18cc..294e6b15a5e2 100644
> --- a/scripts/runtime.bash
> +++ b/scripts/runtime.bash
> @@ -140,10 +140,10 @@ function run()
> # extra_params in the config file may contain backticks that need to be
> # expanded, so use eval to start qemu. Use "> >(foo)" instead of a pipe to
> # preserve the exit status.
> - summary=$(eval $cmdline 2> >(RUNTIME_log_stderr) \
> - > >(tee >(RUNTIME_log_stdout $kernel) | extract_summary))
> + summary=$(eval $cmdline 2> >(RUNTIME_log_stderr $testname) \
> + > >(tee >(RUNTIME_log_stdout $testname $kernel) | extract_summary))
> ret=$?
> - [ "$STANDALONE" != "yes" ] && echo > >(RUNTIME_log_stdout $kernel)
> + [ "$STANDALONE" != "yes" ] && echo > >(RUNTIME_log_stdout $testname $kernel)
>
> if [ $ret -eq 0 ]; then
> print_result "PASS" $testname "$summary"
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [kvm-unit-tests PATCH 2/2] Use same test names in the default and the TAP13 output format
2020-08-21 14:05 ` Janosch Frank
@ 2020-08-21 14:16 ` Marc Hartmayer
0 siblings, 0 replies; 8+ messages in thread
From: Marc Hartmayer @ 2020-08-21 14:16 UTC (permalink / raw)
To: Janosch Frank, Marc Hartmayer, kvm
Cc: Thomas Huth, David Hildenbrand, Cornelia Huck, Andrew Jones,
Paolo Bonzini
On Fri, Aug 21, 2020 at 04:05 PM +0200, Janosch Frank <frankja@linux.ibm.com> wrote:
> On 8/21/20 2:37 PM, Marc Hartmayer wrote:
>> Use the same test names in the TAP13 output as in the default output
>> format. This makes the output more consistent. To achieve this, we
>> need to pass the test name as an argument to the function
>> `process_test_output`.
>>
>> Before this change:
>> $ ./run_tests.sh
>> PASS selftest-setup (14 tests)
>> ...
>>
>> vs.
>>
>> $ ./run_tests.sh -t
>> TAP version 13
>> ok 1 - selftest: true
>> ok 2 - selftest: argc == 3
>> ...
>>
>> After this change:
>> $ ./run_tests.sh
>> PASS selftest-setup (14 tests)
>> ...
>>
>> vs.
>>
>> $ ./run_tests.sh -t
>> TAP version 13
>> ok 1 - selftest-setup: true
>> ok 2 - selftest-setup: argc == 3
>
> This doesn't work for me, we can't just drop prefixes.
>
> I.e. it needs to be "testname: prefix1: prefix2: prefixn: Test text"
> selftest-setup: selftest: true
Okay, I’ll fix that.
[…snip]
--
Kind regards / Beste Grüße
Marc Hartmayer
IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Gregor Pillen
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [kvm-unit-tests PATCH 1/2] runtime.bash: remove outdated comment
2020-08-21 12:37 ` [kvm-unit-tests PATCH 1/2] runtime.bash: remove outdated comment Marc Hartmayer
2020-08-21 13:16 ` Andrew Jones
@ 2020-08-24 12:48 ` Cornelia Huck
1 sibling, 0 replies; 8+ messages in thread
From: Cornelia Huck @ 2020-08-24 12:48 UTC (permalink / raw)
To: Marc Hartmayer
Cc: kvm, Thomas Huth, David Hildenbrand, Janosch Frank, Andrew Jones,
Paolo Bonzini
On Fri, 21 Aug 2020 14:37:43 +0200
Marc Hartmayer <mhartmay@linux.ibm.com> wrote:
> Since commit 6e1d3752d7ca ("tap13: list testcases individually") the
> comment is no longer valid. Therefore let's remove it.
>
> Signed-off-by: Marc Hartmayer <mhartmay@linux.ibm.com>
> ---
> scripts/runtime.bash | 3 ---
> 1 file changed, 3 deletions(-)
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2020-08-24 12:48 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-08-21 12:37 [kvm-unit-tests PATCH 0/2] Use same test names in the default and the TAP13 output format Marc Hartmayer
2020-08-21 12:37 ` [kvm-unit-tests PATCH 1/2] runtime.bash: remove outdated comment Marc Hartmayer
2020-08-21 13:16 ` Andrew Jones
2020-08-24 12:48 ` Cornelia Huck
2020-08-21 12:37 ` [kvm-unit-tests PATCH 2/2] Use same test names in the default and the TAP13 output format Marc Hartmayer
2020-08-21 13:16 ` Andrew Jones
2020-08-21 14:05 ` Janosch Frank
2020-08-21 14:16 ` Marc Hartmayer
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.