* [iptables PATCH 1/7] tests: iptables-test: Fix missing chain case
@ 2021-09-06 16:30 Phil Sutter
2021-09-06 16:30 ` [iptables PATCH 2/7] tests: xlate-test: Don't skip any input after the first empty line Phil Sutter
` (5 more replies)
0 siblings, 6 replies; 7+ messages in thread
From: Phil Sutter @ 2021-09-06 16:30 UTC (permalink / raw)
To: Pablo Neira Ayuso; +Cc: netfilter-devel
If a chain line was really missing, Python complained about reference
before assignment of 'chain_array' variable. While being at it, reuse
print_error() function for reporting and allow to continue with the next
input file instead of exiting.
Signed-off-by: Phil Sutter <phil@nwl.cc>
---
iptables-test.py | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/iptables-test.py b/iptables-test.py
index 90e07feed3658..01966f916957b 100755
--- a/iptables-test.py
+++ b/iptables-test.py
@@ -215,6 +215,7 @@ def run_test_file(filename, netns):
tests = 0
passed = 0
table = ""
+ chain_array = []
total_test_passed = True
if netns:
@@ -249,8 +250,10 @@ def run_test_file(filename, netns):
continue
if len(chain_array) == 0:
- print("broken test, missing chain, leaving")
- sys.exit()
+ print_error("broken test, missing chain",
+ filename = filename, lineno = lineno)
+ total_test_passed = False
+ break
test_passed = True
tests += 1
--
2.33.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [iptables PATCH 2/7] tests: xlate-test: Don't skip any input after the first empty line
2021-09-06 16:30 [iptables PATCH 1/7] tests: iptables-test: Fix missing chain case Phil Sutter
@ 2021-09-06 16:30 ` Phil Sutter
2021-09-06 16:30 ` [iptables PATCH 3/7] tests: xlate-test: Print errors to stderr Phil Sutter
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Phil Sutter @ 2021-09-06 16:30 UTC (permalink / raw)
To: Pablo Neira Ayuso; +Cc: netfilter-devel
In conditionals, testing the empty string evaluates to false. This is
dumb but seems intentional, as readline() method returns an empty string
at EOF. This is distinct from reading an empty line as the latter
contains the newline character - unless it is stripped in between
readline() and conditional. The fixed commit introduced just that by
accident, effectively reducing any test file to the first contained
test:
| $ ./xlate-test.py
| [...]
| 81 test files, 84 tests, 84 tests passed, 0 tests failed, 0 errors
With this change in place, the summary looks much better:
| 81 test files, 368 tests, 368 tests passed, 0 tests failed, 0 errors
Fixes: 62828a6aff231 ("tests: xlate-test: support multiline expectation")
Signed-off-by: Phil Sutter <phil@nwl.cc>
---
xlate-test.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/xlate-test.py b/xlate-test.py
index cba98b6e8e491..1fa5eca3e0764 100755
--- a/xlate-test.py
+++ b/xlate-test.py
@@ -48,9 +48,9 @@ def run_test(name, payload):
if process.returncode == 0:
translation = output.decode("utf-8").rstrip(" \n")
expected = payload.readline().rstrip(" \n")
- next_expected = payload.readline().rstrip(" \n")
+ next_expected = payload.readline()
if next_expected.startswith("nft"):
- expected += "\n" + next_expected
+ expected += "\n" + next_expected.rstrip(" \n")
line = payload.readline()
else:
line = next_expected
--
2.33.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [iptables PATCH 3/7] tests: xlate-test: Print errors to stderr
2021-09-06 16:30 [iptables PATCH 1/7] tests: iptables-test: Fix missing chain case Phil Sutter
2021-09-06 16:30 ` [iptables PATCH 2/7] tests: xlate-test: Don't skip any input after the first empty line Phil Sutter
@ 2021-09-06 16:30 ` Phil Sutter
2021-09-06 16:30 ` [iptables PATCH 4/7] tests: iptables-test: " Phil Sutter
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Phil Sutter @ 2021-09-06 16:30 UTC (permalink / raw)
To: Pablo Neira Ayuso; +Cc: netfilter-devel
Return code is always zero, so grepping for output on stderr is a
simple way to detect testsuite failures.
Signed-off-by: Phil Sutter <phil@nwl.cc>
---
xlate-test.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/xlate-test.py b/xlate-test.py
index 1fa5eca3e0764..bb7a447dc799e 100755
--- a/xlate-test.py
+++ b/xlate-test.py
@@ -75,7 +75,7 @@ def run_test(name, payload):
if (passed == tests) and not args.test:
print(name + ": " + green("OK"))
if not test_passed:
- print("\n".join(result))
+ print("\n".join(result), file=sys.stderr)
if args.test:
print("1 test file, %d tests, %d tests passed, %d tests failed, %d errors" % (tests, passed, failed, errors))
else:
@@ -111,7 +111,7 @@ def main():
with open(args.test, "r") as payload:
run_test(args.test, payload)
except IOError:
- print(red("Error: ") + "test file does not exist")
+ print(red("Error: ") + "test file does not exist", file=sys.stderr)
else:
load_test_files()
--
2.33.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [iptables PATCH 4/7] tests: iptables-test: Print errors to stderr
2021-09-06 16:30 [iptables PATCH 1/7] tests: iptables-test: Fix missing chain case Phil Sutter
2021-09-06 16:30 ` [iptables PATCH 2/7] tests: xlate-test: Don't skip any input after the first empty line Phil Sutter
2021-09-06 16:30 ` [iptables PATCH 3/7] tests: xlate-test: Print errors to stderr Phil Sutter
@ 2021-09-06 16:30 ` Phil Sutter
2021-09-06 16:30 ` [iptables PATCH 5/7] tests: xlate-test: Exit non-zero on error Phil Sutter
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Phil Sutter @ 2021-09-06 16:30 UTC (permalink / raw)
To: Pablo Neira Ayuso; +Cc: netfilter-devel
No big deal, just pass the extra parameter to the four error print
calls.
Signed-off-by: Phil Sutter <phil@nwl.cc>
---
iptables-test.py | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/iptables-test.py b/iptables-test.py
index 01966f916957b..1790da3d0b074 100755
--- a/iptables-test.py
+++ b/iptables-test.py
@@ -47,7 +47,7 @@ def print_error(reason, filename=None, lineno=None):
Prints an error with nice colors, indicating file and line number.
'''
print(filename + ": " + Colors.RED + "ERROR" +
- Colors.ENDC + ": line %d (%s)" % (lineno, reason))
+ Colors.ENDC + ": line %d (%s)" % (lineno, reason), file=sys.stderr)
def delete_rule(iptables, rule, filename, lineno):
@@ -368,11 +368,12 @@ def main():
EXECUTEABLE = "xtables-nft-multi"
if os.getuid() != 0:
- print("You need to be root to run this, sorry")
+ print("You need to be root to run this, sorry", file=sys.stderr)
return
if not args.netns and not args.no_netns and not spawn_netns():
- print("Cannot run in own namespace, connectivity might break")
+ print("Cannot run in own namespace, connectivity might break",
+ file=sys.stderr)
if not args.host:
os.putenv("XTABLES_LIBDIR", os.path.abspath(EXTENSIONS_PATH))
@@ -388,7 +389,7 @@ def main():
try:
log_file = open(LOGFILE, 'w')
except IOError:
- print("Couldn't open log file %s" % LOGFILE)
+ print("Couldn't open log file %s" % LOGFILE, file=sys.stderr)
return
if args.filename:
--
2.33.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [iptables PATCH 5/7] tests: xlate-test: Exit non-zero on error
2021-09-06 16:30 [iptables PATCH 1/7] tests: iptables-test: Fix missing chain case Phil Sutter
` (2 preceding siblings ...)
2021-09-06 16:30 ` [iptables PATCH 4/7] tests: iptables-test: " Phil Sutter
@ 2021-09-06 16:30 ` Phil Sutter
2021-09-06 16:30 ` [iptables PATCH 6/7] tests: iptables-test: " Phil Sutter
2021-09-06 16:30 ` [iptables PATCH 7/7] tests: shell: Return " Phil Sutter
5 siblings, 0 replies; 7+ messages in thread
From: Phil Sutter @ 2021-09-06 16:30 UTC (permalink / raw)
To: Pablo Neira Ayuso; +Cc: netfilter-devel
If a test fails, return a non-zero exit code. To do so, propagate the
pass/fail statistics up to main() for evaluation. While being at it,
move the statistics printing into there as well and get rid of that
redundant assignment to 'test_passed'.
Signed-off-by: Phil Sutter <phil@nwl.cc>
---
xlate-test.py | 26 ++++++++++++++++----------
1 file changed, 16 insertions(+), 10 deletions(-)
diff --git a/xlate-test.py b/xlate-test.py
index bb7a447dc799e..4a56e798b9587 100755
--- a/xlate-test.py
+++ b/xlate-test.py
@@ -61,7 +61,6 @@ def run_test(name, payload):
result.append(magenta("src: ") + line.rstrip(" \n"))
result.append(magenta("exp: ") + expected)
result.append(magenta("res: ") + translation + "\n")
- test_passed = False
else:
passed += 1
else:
@@ -76,10 +75,7 @@ def run_test(name, payload):
print(name + ": " + green("OK"))
if not test_passed:
print("\n".join(result), file=sys.stderr)
- if args.test:
- print("1 test file, %d tests, %d tests passed, %d tests failed, %d errors" % (tests, passed, failed, errors))
- else:
- return tests, passed, failed, errors
+ return tests, passed, failed, errors
def load_test_files():
@@ -93,10 +89,9 @@ def load_test_files():
total_passed += passed
total_failed += failed
total_error += errors
+ return (test_files, total_tests, total_passed, total_failed, total_error)
- print("%d test files, %d tests, %d tests passed, %d tests failed, %d errors" % (test_files, total_tests, total_passed, total_failed, total_error))
-
def main():
global xtables_nft_multi
if not args.host:
@@ -104,16 +99,27 @@ def main():
xtables_nft_multi = os.path.abspath(os.path.curdir) \
+ '/iptables/' + xtables_nft_multi
+ files = tests = passed = failed = errors = 0
if args.test:
if not args.test.endswith(".txlate"):
args.test += ".txlate"
try:
with open(args.test, "r") as payload:
- run_test(args.test, payload)
+ files = 1
+ tests, passed, failed, errors = run_test(args.test, payload)
except IOError:
print(red("Error: ") + "test file does not exist", file=sys.stderr)
+ return -1
+ else:
+ files, tests, passed, failed, errors = load_test_files()
+
+ if files > 1:
+ file_word = "files"
else:
- load_test_files()
+ file_word = "file"
+ print("%d test %s, %d tests, %d tests passed, %d tests failed, %d errors"
+ % (files, file_word, tests, passed, failed, errors))
+ return passed - tests
parser = argparse.ArgumentParser()
@@ -121,4 +127,4 @@ parser.add_argument('-H', '--host', action='store_true',
help='Run tests against installed binaries')
parser.add_argument("test", nargs="?", help="run only the specified test file")
args = parser.parse_args()
-main()
+sys.exit(main())
--
2.33.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [iptables PATCH 6/7] tests: iptables-test: Exit non-zero on error
2021-09-06 16:30 [iptables PATCH 1/7] tests: iptables-test: Fix missing chain case Phil Sutter
` (3 preceding siblings ...)
2021-09-06 16:30 ` [iptables PATCH 5/7] tests: xlate-test: Exit non-zero on error Phil Sutter
@ 2021-09-06 16:30 ` Phil Sutter
2021-09-06 16:30 ` [iptables PATCH 7/7] tests: shell: Return " Phil Sutter
5 siblings, 0 replies; 7+ messages in thread
From: Phil Sutter @ 2021-09-06 16:30 UTC (permalink / raw)
To: Pablo Neira Ayuso; +Cc: netfilter-devel
If any test fails, return a non-zero exit code.
Signed-off-by: Phil Sutter <phil@nwl.cc>
---
iptables-test.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/iptables-test.py b/iptables-test.py
index 1790da3d0b074..5eafe5896414b 100755
--- a/iptables-test.py
+++ b/iptables-test.py
@@ -408,7 +408,8 @@ def main():
test_files += 1
print("%d test files, %d unit tests, %d passed" % (test_files, tests, passed))
+ return passed - tests
if __name__ == '__main__':
- main()
+ sys.exit(main())
--
2.33.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [iptables PATCH 7/7] tests: shell: Return non-zero on error
2021-09-06 16:30 [iptables PATCH 1/7] tests: iptables-test: Fix missing chain case Phil Sutter
` (4 preceding siblings ...)
2021-09-06 16:30 ` [iptables PATCH 6/7] tests: iptables-test: " Phil Sutter
@ 2021-09-06 16:30 ` Phil Sutter
5 siblings, 0 replies; 7+ messages in thread
From: Phil Sutter @ 2021-09-06 16:30 UTC (permalink / raw)
To: Pablo Neira Ayuso; +Cc: netfilter-devel
If any test fails, return a non-zero exit code.
Signed-off-by: Phil Sutter <phil@nwl.cc>
---
iptables/tests/shell/run-tests.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/iptables/tests/shell/run-tests.sh b/iptables/tests/shell/run-tests.sh
index 65c37adb75f2a..7878760fdcc4d 100755
--- a/iptables/tests/shell/run-tests.sh
+++ b/iptables/tests/shell/run-tests.sh
@@ -195,4 +195,4 @@ failed=$((legacy_fail+failed))
msg_info "combined results: [OK] $ok [FAILED] $failed [TOTAL] $((ok+failed))"
-exit 0
+exit -$failed
--
2.33.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-09-06 16:30 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-06 16:30 [iptables PATCH 1/7] tests: iptables-test: Fix missing chain case Phil Sutter
2021-09-06 16:30 ` [iptables PATCH 2/7] tests: xlate-test: Don't skip any input after the first empty line Phil Sutter
2021-09-06 16:30 ` [iptables PATCH 3/7] tests: xlate-test: Print errors to stderr Phil Sutter
2021-09-06 16:30 ` [iptables PATCH 4/7] tests: iptables-test: " Phil Sutter
2021-09-06 16:30 ` [iptables PATCH 5/7] tests: xlate-test: Exit non-zero on error Phil Sutter
2021-09-06 16:30 ` [iptables PATCH 6/7] tests: iptables-test: " Phil Sutter
2021-09-06 16:30 ` [iptables PATCH 7/7] tests: shell: Return " Phil Sutter
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.