[iptables PATCH v2] tests: iptables-test: Support variant deviation

[iptables PATCH v2] tests: iptables-test: Support variant deviation

[Phil Sutter]

Some test results are not consistent between variants:

* CLUSTERIP is not supported with nft_compat, so all related tests fail
  with iptables-nft.
* iptables-legacy mandates TCPMSS be combined with SYN flag match,
  iptables-nft does not care. (Or precisely, xt_TCPMSS.ko can't validate
  match presence.)

Introduce an optional fourth test spec field to specify the variant it
applies to. Consequently, the opposite result is expected with the other
variant.

Signed-off-by: Phil Sutter <phil@nwl.cc>
---
Changes since v1:
- Do not replace "OK" result but add a fourth field as requested in v1
  review.
---
 extensions/libipt_CLUSTERIP.t |  4 ++--
 extensions/libxt_TCPMSS.t     |  2 +-
 iptables-test.py              | 25 +++++++++++++++++++++++++
 3 files changed, 28 insertions(+), 3 deletions(-)

diff --git a/extensions/libipt_CLUSTERIP.t b/extensions/libipt_CLUSTERIP.t
index 5af555e005c1d..30b80167a6223 100644
--- a/extensions/libipt_CLUSTERIP.t
+++ b/extensions/libipt_CLUSTERIP.t
@@ -1,4 +1,4 @@
 :INPUT
 -d 10.31.3.236/32 -i lo -j CLUSTERIP --new --hashmode sourceip --clustermac 01:AA:7B:47:F7:D7 --total-nodes 2 --local-node 0 --hash-init 1;=;FAIL
--d 10.31.3.236/32 -i lo -j CLUSTERIP --new --hashmode sourceip --clustermac 01:AA:7B:47:F7:D7 --total-nodes 2 --local-node 1 --hash-init 1;=;OK
--d 10.31.3.236/32 -i lo -j CLUSTERIP --new --hashmode sourceip --clustermac 01:AA:7B:47:F7:D7 --total-nodes 2 --local-node 2 --hash-init 1;=;OK
+-d 10.31.3.236/32 -i lo -j CLUSTERIP --new --hashmode sourceip --clustermac 01:AA:7B:47:F7:D7 --total-nodes 2 --local-node 1 --hash-init 1;=;OK;LEGACY
+-d 10.31.3.236/32 -i lo -j CLUSTERIP --new --hashmode sourceip --clustermac 01:AA:7B:47:F7:D7 --total-nodes 2 --local-node 2 --hash-init 1;=;OK;LEGACY
diff --git a/extensions/libxt_TCPMSS.t b/extensions/libxt_TCPMSS.t
index 553a3452e4876..fbfbfcf88d81a 100644
--- a/extensions/libxt_TCPMSS.t
+++ b/extensions/libxt_TCPMSS.t
@@ -1,6 +1,6 @@
 :FORWARD,OUTPUT,POSTROUTING
 *mangle
 -j TCPMSS;;FAIL
--p tcp -j TCPMSS --set-mss 42;;FAIL
+-p tcp -j TCPMSS --set-mss 42;;FAIL;LEGACY
 -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -j TCPMSS --set-mss 42;=;OK
 -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -j TCPMSS --clamp-mss-to-pmtu;=;OK
diff --git a/iptables-test.py b/iptables-test.py
index 95fa11b1475ca..b49afcab3c40c 100755
--- a/iptables-test.py
+++ b/iptables-test.py
@@ -182,6 +182,28 @@ def execute_cmd(cmd, filename, lineno):
     return ret
 
 
+def variant_res(res, variant):
+    '''
+    Adjust expected result with given variant
+
+    If expected result is scoped to a variant, the other one yields a different
+    result. Therefore map @res to itself if given variant is current, invert it
+    otherwise.
+
+    :param res: expected result from test spec ("OK" or "FAIL")
+    :param variant: variant @res is scoped to by test spec ("NFT" or "LEGACY")
+    '''
+    variant_executable = {
+            "NFT": "xtables-nft-multi",
+            "LEGACY": "xtables-legacy-multi"
+    }
+    res_inverse = { "OK": "FAIL", "FAIL": "OK" }
+
+    if variant_executable[variant] == EXECUTABLE:
+        return res
+    return res_inverse[res]
+
+
 def run_test_file(filename, netns):
     '''
     Runs a test file
@@ -275,6 +297,9 @@ def run_test_file(filename, netns):
                 rule_save = chain + " " + item[1]
 
             res = item[2].rstrip()
+            if len(item) > 3:
+                res = variant_res(res, item[3].rstrip())
+
             ret = run_test(iptables, rule, rule_save,
                            res, filename, lineno + 1, netns)
 
-- 
2.34.1

Re: [iptables PATCH v2] tests: iptables-test: Support variant deviation

[Pablo Neira Ayuso]

On Wed, Feb 09, 2022 at 04:41:07PM +0100, Phil Sutter wrote:
> Some test results are not consistent between variants:
> 
> * CLUSTERIP is not supported with nft_compat, so all related tests fail
>   with iptables-nft.
> * iptables-legacy mandates TCPMSS be combined with SYN flag match,
>   iptables-nft does not care. (Or precisely, xt_TCPMSS.ko can't validate
>   match presence.)
> 
> Introduce an optional fourth test spec field to specify the variant it
> applies to. Consequently, the opposite result is expected with the other
> variant.

LGTM, thanks