[iptables PATCH 1/7] tests: iptables-test: Fix missing chain case

[iptables PATCH 1/7] tests: iptables-test: Fix missing chain case

[Phil Sutter]

If a chain line was really missing, Python complained about reference
before assignment of 'chain_array' variable. While being at it, reuse
print_error() function for reporting and allow to continue with the next
input file instead of exiting.

Signed-off-by: Phil Sutter <phil@nwl.cc>
---
 iptables-test.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/iptables-test.py b/iptables-test.py
index 90e07feed3658..01966f916957b 100755
--- a/iptables-test.py
+++ b/iptables-test.py
@@ -215,6 +215,7 @@ def run_test_file(filename, netns):
     tests = 0
     passed = 0
     table = ""
+    chain_array = []
     total_test_passed = True
 
     if netns:
@@ -249,8 +250,10 @@ def run_test_file(filename, netns):
             continue
 
         if len(chain_array) == 0:
-            print("broken test, missing chain, leaving")
-            sys.exit()
+            print_error("broken test, missing chain",
+                        filename = filename, lineno = lineno)
+            total_test_passed = False
+            break
 
         test_passed = True
         tests += 1
-- 
2.33.0

[iptables PATCH 2/7] tests: xlate-test: Don't skip any input after the first empty line

[Phil Sutter]

In conditionals, testing the empty string evaluates to false. This is
dumb but seems intentional, as readline() method returns an empty string
at EOF. This is distinct from reading an empty line as the latter
contains the newline character - unless it is stripped in between
readline() and conditional. The fixed commit introduced just that by
accident, effectively reducing any test file to the first contained
test:

| $ ./xlate-test.py
| [...]
| 81 test files, 84 tests, 84 tests passed, 0 tests failed, 0 errors

With this change in place, the summary looks much better:

| 81 test files, 368 tests, 368 tests passed, 0 tests failed, 0 errors

Fixes: 62828a6aff231 ("tests: xlate-test: support multiline expectation")
Signed-off-by: Phil Sutter <phil@nwl.cc>
---
 xlate-test.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/xlate-test.py b/xlate-test.py
index cba98b6e8e491..1fa5eca3e0764 100755
--- a/xlate-test.py
+++ b/xlate-test.py
@@ -48,9 +48,9 @@ def run_test(name, payload):
             if process.returncode == 0:
                 translation = output.decode("utf-8").rstrip(" \n")
                 expected = payload.readline().rstrip(" \n")
-                next_expected = payload.readline().rstrip(" \n")
+                next_expected = payload.readline()
                 if next_expected.startswith("nft"):
-                    expected += "\n" + next_expected
+                    expected += "\n" + next_expected.rstrip(" \n")
                     line = payload.readline()
                 else:
                     line = next_expected
-- 
2.33.0

[iptables PATCH 3/7] tests: xlate-test: Print errors to stderr

[Phil Sutter]

Return code is always zero, so grepping for output on stderr is a
simple way to detect testsuite failures.

Signed-off-by: Phil Sutter <phil@nwl.cc>
---
 xlate-test.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/xlate-test.py b/xlate-test.py
index 1fa5eca3e0764..bb7a447dc799e 100755
--- a/xlate-test.py
+++ b/xlate-test.py
@@ -75,7 +75,7 @@ def run_test(name, payload):
     if (passed == tests) and not args.test:
         print(name + ": " + green("OK"))
     if not test_passed:
-        print("\n".join(result))
+        print("\n".join(result), file=sys.stderr)
     if args.test:
         print("1 test file, %d tests, %d tests passed, %d tests failed, %d errors" % (tests, passed, failed, errors))
     else:
@@ -111,7 +111,7 @@ def main():
             with open(args.test, "r") as payload:
                 run_test(args.test, payload)
         except IOError:
-            print(red("Error: ") + "test file does not exist")
+            print(red("Error: ") + "test file does not exist", file=sys.stderr)
     else:
         load_test_files()
 
-- 
2.33.0

[iptables PATCH 4/7] tests: iptables-test: Print errors to stderr

[Phil Sutter]

No big deal, just pass the extra parameter to the four error print
calls.

Signed-off-by: Phil Sutter <phil@nwl.cc>
---
 iptables-test.py | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/iptables-test.py b/iptables-test.py
index 01966f916957b..1790da3d0b074 100755
--- a/iptables-test.py
+++ b/iptables-test.py
@@ -47,7 +47,7 @@ def print_error(reason, filename=None, lineno=None):
     Prints an error with nice colors, indicating file and line number.
     '''
     print(filename + ": " + Colors.RED + "ERROR" +
-        Colors.ENDC + ": line %d (%s)" % (lineno, reason))
+        Colors.ENDC + ": line %d (%s)" % (lineno, reason), file=sys.stderr)
 
 
 def delete_rule(iptables, rule, filename, lineno):
@@ -368,11 +368,12 @@ def main():
         EXECUTEABLE = "xtables-nft-multi"
 
     if os.getuid() != 0:
-        print("You need to be root to run this, sorry")
+        print("You need to be root to run this, sorry", file=sys.stderr)
         return
 
     if not args.netns and not args.no_netns and not spawn_netns():
-        print("Cannot run in own namespace, connectivity might break")
+        print("Cannot run in own namespace, connectivity might break",
+              file=sys.stderr)
 
     if not args.host:
         os.putenv("XTABLES_LIBDIR", os.path.abspath(EXTENSIONS_PATH))
@@ -388,7 +389,7 @@ def main():
     try:
         log_file = open(LOGFILE, 'w')
     except IOError:
-        print("Couldn't open log file %s" % LOGFILE)
+        print("Couldn't open log file %s" % LOGFILE, file=sys.stderr)
         return
 
     if args.filename:
-- 
2.33.0

[iptables PATCH 5/7] tests: xlate-test: Exit non-zero on error

[Phil Sutter]

If a test fails, return a non-zero exit code. To do so, propagate the
pass/fail statistics up to main() for evaluation. While being at it,
move the statistics printing into there as well and get rid of that
redundant assignment to 'test_passed'.

Signed-off-by: Phil Sutter <phil@nwl.cc>
---
 xlate-test.py | 26 ++++++++++++++++----------
 1 file changed, 16 insertions(+), 10 deletions(-)

diff --git a/xlate-test.py b/xlate-test.py
index bb7a447dc799e..4a56e798b9587 100755
--- a/xlate-test.py
+++ b/xlate-test.py
@@ -61,7 +61,6 @@ def run_test(name, payload):
                     result.append(magenta("src: ") + line.rstrip(" \n"))
                     result.append(magenta("exp: ") + expected)
                     result.append(magenta("res: ") + translation + "\n")
-                    test_passed = False
                 else:
                     passed += 1
             else:
@@ -76,10 +75,7 @@ def run_test(name, payload):
         print(name + ": " + green("OK"))
     if not test_passed:
         print("\n".join(result), file=sys.stderr)
-    if args.test:
-        print("1 test file, %d tests, %d tests passed, %d tests failed, %d errors" % (tests, passed, failed, errors))
-    else:
-        return tests, passed, failed, errors
+    return tests, passed, failed, errors
 
 
 def load_test_files():
@@ -93,10 +89,9 @@ def load_test_files():
                 total_passed += passed
                 total_failed += failed
                 total_error += errors
+    return (test_files, total_tests, total_passed, total_failed, total_error)
 
 
-    print("%d test files, %d tests, %d tests passed, %d tests failed, %d errors" % (test_files, total_tests, total_passed, total_failed, total_error))
-
 def main():
     global xtables_nft_multi
     if not args.host:
@@ -104,16 +99,27 @@ def main():
         xtables_nft_multi = os.path.abspath(os.path.curdir) \
                             + '/iptables/' + xtables_nft_multi
 
+    files = tests = passed = failed = errors = 0
     if args.test:
         if not args.test.endswith(".txlate"):
             args.test += ".txlate"
         try:
             with open(args.test, "r") as payload:
-                run_test(args.test, payload)
+                files = 1
+                tests, passed, failed, errors = run_test(args.test, payload)
         except IOError:
             print(red("Error: ") + "test file does not exist", file=sys.stderr)
+            return -1
+    else:
+        files, tests, passed, failed, errors = load_test_files()
+
+    if files > 1:
+        file_word = "files"
     else:
-        load_test_files()
+        file_word = "file"
+    print("%d test %s, %d tests, %d tests passed, %d tests failed, %d errors"
+            % (files, file_word, tests, passed, failed, errors))
+    return passed - tests
 
 
 parser = argparse.ArgumentParser()
@@ -121,4 +127,4 @@ parser.add_argument('-H', '--host', action='store_true',
                     help='Run tests against installed binaries')
 parser.add_argument("test", nargs="?", help="run only the specified test file")
 args = parser.parse_args()
-main()
+sys.exit(main())
-- 
2.33.0

[iptables PATCH 6/7] tests: iptables-test: Exit non-zero on error

[Phil Sutter]

If any test fails, return a non-zero exit code.

Signed-off-by: Phil Sutter <phil@nwl.cc>
---
 iptables-test.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/iptables-test.py b/iptables-test.py
index 1790da3d0b074..5eafe5896414b 100755
--- a/iptables-test.py
+++ b/iptables-test.py
@@ -408,7 +408,8 @@ def main():
             test_files += 1
 
     print("%d test files, %d unit tests, %d passed" % (test_files, tests, passed))
+    return passed - tests
 
 
 if __name__ == '__main__':
-    main()
+    sys.exit(main())
-- 
2.33.0

[iptables PATCH 7/7] tests: shell: Return non-zero on error

[Phil Sutter]

If any test fails, return a non-zero exit code.

Signed-off-by: Phil Sutter <phil@nwl.cc>
---
 iptables/tests/shell/run-tests.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/iptables/tests/shell/run-tests.sh b/iptables/tests/shell/run-tests.sh
index 65c37adb75f2a..7878760fdcc4d 100755
--- a/iptables/tests/shell/run-tests.sh
+++ b/iptables/tests/shell/run-tests.sh
@@ -195,4 +195,4 @@ failed=$((legacy_fail+failed))
 
 msg_info "combined results: [OK] $ok [FAILED] $failed [TOTAL] $((ok+failed))"
 
-exit 0
+exit -$failed
-- 
2.33.0