[XTF PATCH v2 0/4] Runner exit code clean up

[XTF PATCH v2 0/4] Runner exit code clean up

[Wei Liu]

Wei Liu (4):
  gitignore: ignore vim swp file
  xtf-runner: sync all test results
  xtf-runner: provide a set of exit codes for different results
  xtf-runner: regularise runner exit code

 .gitignore |  1 +
 xtf-runner | 37 ++++++++++++++++++++++++++++++++-----
 2 files changed, 33 insertions(+), 5 deletions(-)

-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[XTF PATCH v2 1/4] gitignore: ignore vim swp file

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitignore b/.gitignore
index 01243f0..56e1884 100644
--- a/.gitignore
+++ b/.gitignore
@@ -3,6 +3,7 @@
 *.d
 *.pyc
 *.pyo
+*.swp
 /arch/x86/*.lds
 /cscope.*
 /dist/
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[XTF PATCH v2 2/4] xtf-runner: sync all test results

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 xtf-runner | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/xtf-runner b/xtf-runner
index 50a5e96..c938f28 100755
--- a/xtf-runner
+++ b/xtf-runner
@@ -17,6 +17,9 @@ try:
 except ImportError:
     import simplejson as json
 
+# All results of a test, keep in sync with C code report.h.
+# Note that warning is not a result on its own.
+all_results = ('SUCCESS', 'SKIP', 'ERROR', 'FAILURE')
 
 # All test categories and configurations
 all_categories   = ("special", "functional", "xsa", "utility")
@@ -161,7 +164,7 @@ def run_test(test):
     if not "Test result:" in test_result:
         return "ERROR"
 
-    for res in ("SUCCESS", "SKIP", "FAILURE"):
+    for res in all_results:
 
         if res in test_result:
             return res
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[XTF PATCH v2 3/4] xtf-runner: provide a set of exit codes for different results

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 xtf-runner | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/xtf-runner b/xtf-runner
index c938f28..1c96750 100755
--- a/xtf-runner
+++ b/xtf-runner
@@ -21,6 +21,16 @@ except ImportError:
 # Note that warning is not a result on its own.
 all_results = ('SUCCESS', 'SKIP', 'ERROR', 'FAILURE')
 
+# Return the exit code for different states.  Avoid using 1 and 2 because
+# python interpreter uses them -- see document for sys.exit.
+def exit_code(state):
+    """ Convert a test result to an xtf-runner exit code. """
+    return { "SUCCESS": 0,
+             "SKIP":    3,
+             "ERROR":   4,
+             "FAILURE": 5,
+    }.get(state, 4)
+
 # All test categories and configurations
 all_categories   = ("special", "functional", "xsa", "utility")
 pv_environments  = ("pv64", "pv32pae")
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[XTF PATCH v2 4/4] xtf-runner: regularise runner exit code

[Wei Liu]

The script now returns the most severe result. Document the exit code in
help string.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 xtf-runner | 22 ++++++++++++++++++----
 1 file changed, 18 insertions(+), 4 deletions(-)

diff --git a/xtf-runner b/xtf-runner
index 1c96750..15b98c6 100755
--- a/xtf-runner
+++ b/xtf-runner
@@ -251,23 +251,30 @@ def run_tests(args):
     if not len(tests):
         raise RunnerError("No tests to run")
 
-    rc = 0
+    rc = all_results.index('SUCCESS')
     results = []
 
     for test in tests:
 
         res = run_test(test)
-        if res != "SUCCESS":
-            rc = 1
+        res_idx = all_results.index(res);
+        if res_idx > rc:
+            rc = res_idx
 
         results.append(res)
 
+    if rc == exit_code('SUCCESS'):
+        for res in results:
+            if res == 'SKIP':
+                rc = exit_code('SKIP')
+                break
+
     print "\nCombined test results:"
 
     for test, res in zip(tests, results):
         print "%-40s %s" % (test, res)
 
-    return rc
+    return exit_code(all_results[rc])
 
 
 def main():
@@ -308,6 +315,13 @@ def main():
                   "       List all 'functional' or 'special' tests\n"
                   "    ./xtf-runner --list hvm64\n"
                   "       List all 'hvm64' tests\n"
+                  "\n"
+                  "  Exit code for this script:\n"
+                  "    0:    everythin is ok\n"
+                  "    1,2:  reserved for python interpreter\n"
+                  "    3:    test(s) are skipped\n"
+                  "    4:    test(s) report error\n"
+                  "    5:    test(s) report failure\n"
                   ),
     )
 
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [XTF PATCH v2 4/4] xtf-runner: regularise runner exit code

[Andrew Cooper]

On 22/07/16 10:43, Wei Liu wrote:

> The script now returns the most severe result. Document the exit code in
> help string.
>
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> ---
>   xtf-runner | 22 ++++++++++++++++++----
>   1 file changed, 18 insertions(+), 4 deletions(-)
>
> diff --git a/xtf-runner b/xtf-runner
> index 1c96750..15b98c6 100755
> --- a/xtf-runner
> +++ b/xtf-runner
> @@ -251,23 +251,30 @@ def run_tests(args):
>       if not len(tests):
>           raise RunnerError("No tests to run")
>   
> -    rc = 0
> +    rc = all_results.index('SUCCESS')
>       results = []
>   
>       for test in tests:
>   
>           res = run_test(test)
> -        if res != "SUCCESS":
> -            rc = 1
> +        res_idx = all_results.index(res);
> +        if res_idx > rc:
> +            rc = res_idx
>   
>           results.append(res)
>   
> +    if rc == exit_code('SUCCESS'):
> +        for res in results:
> +            if res == 'SKIP':
> +                rc = exit_code('SKIP')
> +                break

Why is this conditional needed? SKIP has index 1 so will automatically 
displace SUCCESS in the change above.

> +
>       print "\nCombined test results:"
>   
>       for test, res in zip(tests, results):
>           print "%-40s %s" % (test, res)
>   
> -    return rc
> +    return exit_code(all_results[rc])
>   
>   
>   def main():
> @@ -308,6 +315,13 @@ def main():
>                     "       List all 'functional' or 'special' tests\n"
>                     "    ./xtf-runner --list hvm64\n"
>                     "       List all 'hvm64' tests\n"
> +                  "\n"
> +                  "  Exit code for this script:\n"
> +                  "    0:    everythin is ok\n"

everything

~Andrew

> +                  "    1,2:  reserved for python interpreter\n"
> +                  "    3:    test(s) are skipped\n"
> +                  "    4:    test(s) report error\n"
> +                  "    5:    test(s) report failure\n"
>                     ),
>       )
>   


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [XTF PATCH v2 4/4] xtf-runner: regularise runner exit code

[Wei Liu]

On Fri, Jul 22, 2016 at 10:49:18AM +0100, Andrew Cooper wrote:
> On 22/07/16 10:43, Wei Liu wrote:
> 
> >The script now returns the most severe result. Document the exit code in
> >help string.
> >
> >Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> >---
> >  xtf-runner | 22 ++++++++++++++++++----
> >  1 file changed, 18 insertions(+), 4 deletions(-)
> >
> >diff --git a/xtf-runner b/xtf-runner
> >index 1c96750..15b98c6 100755
> >--- a/xtf-runner
> >+++ b/xtf-runner
> >@@ -251,23 +251,30 @@ def run_tests(args):
> >      if not len(tests):
> >          raise RunnerError("No tests to run")
> >-    rc = 0
> >+    rc = all_results.index('SUCCESS')
> >      results = []
> >      for test in tests:
> >          res = run_test(test)
> >-        if res != "SUCCESS":
> >-            rc = 1
> >+        res_idx = all_results.index(res);
> >+        if res_idx > rc:
> >+            rc = res_idx
> >          results.append(res)
> >+    if rc == exit_code('SUCCESS'):
> >+        for res in results:
> >+            if res == 'SKIP':
> >+                rc = exit_code('SKIP')
> >+                break
> 
> Why is this conditional needed? SKIP has index 1 so will automatically
> displace SUCCESS in the change above.
> 

Forgot to delete that hunk. I will resend.

> >+
> >      print "\nCombined test results:"
> >      for test, res in zip(tests, results):
> >          print "%-40s %s" % (test, res)
> >-    return rc
> >+    return exit_code(all_results[rc])
> >  def main():
> >@@ -308,6 +315,13 @@ def main():
> >                    "       List all 'functional' or 'special' tests\n"
> >                    "    ./xtf-runner --list hvm64\n"
> >                    "       List all 'hvm64' tests\n"
> >+                  "\n"
> >+                  "  Exit code for this script:\n"
> >+                  "    0:    everythin is ok\n"
> 
> everything

Fixed.

Wei.

> 
> ~Andrew
> 
> >+                  "    1,2:  reserved for python interpreter\n"
> >+                  "    3:    test(s) are skipped\n"
> >+                  "    4:    test(s) report error\n"
> >+                  "    5:    test(s) report failure\n"
> >                    ),
> >      )
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [XTF PATCH v2 4/4] xtf-runner: regularise runner exit code

[Wei Liu]

Updated this patch

---8<---
From ef1f9ddfa797bb4095d48b90efc3c92a0a8fd1b6 Mon Sep 17 00:00:00 2001
From: Wei Liu <wei.liu2@citrix.com>
Date: Mon, 13 Jun 2016 15:06:48 +0100
Subject: [XTF PATCH] xtf-runner: regularise runner exit code
Cc: andrew.cooper3@citrix.com

The script now returns the most severe result. Document the exit code in
help string.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 xtf-runner | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/xtf-runner b/xtf-runner
index 1c96750..7743316 100755
--- a/xtf-runner
+++ b/xtf-runner
@@ -251,14 +251,15 @@ def run_tests(args):
     if not len(tests):
         raise RunnerError("No tests to run")
 
-    rc = 0
+    rc = all_results.index('SUCCESS')
     results = []
 
     for test in tests:
 
         res = run_test(test)
-        if res != "SUCCESS":
-            rc = 1
+        res_idx = all_results.index(res);
+        if res_idx > rc:
+            rc = res_idx
 
         results.append(res)
 
@@ -267,7 +268,7 @@ def run_tests(args):
     for test, res in zip(tests, results):
         print "%-40s %s" % (test, res)
 
-    return rc
+    return exit_code(all_results[rc])
 
 
 def main():
@@ -308,6 +309,13 @@ def main():
                   "       List all 'functional' or 'special' tests\n"
                   "    ./xtf-runner --list hvm64\n"
                   "       List all 'hvm64' tests\n"
+                  "\n"
+                  "  Exit code for this script:\n"
+                  "    0:    everything is ok\n"
+                  "    1,2:  reserved for python interpreter\n"
+                  "    3:    test(s) are skipped\n"
+                  "    4:    test(s) report error\n"
+                  "    5:    test(s) report failure\n"
                   ),
     )
 
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [XTF PATCH v2 4/4] xtf-runner: regularise runner exit code

[Andrew Cooper]

On 22/07/16 10:58, Wei Liu wrote:

> Updated this patch

Thanks.

Pylint warned of an unnecessary semicolon from "res_idx = 
all_results.index(res);" which I dropped.

Whole series Reviewed and committed.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel