[PATCH v2 net-next 0/3] tc-testing: Add JSON verification and simple traffic generation

[PATCH v2 net-next 0/3] tc-testing: Add JSON verification and simple traffic generation

[Lucas Bates]

This patchset introduces JSON as a verification method in tdc and adds a new
plugin, scapyPlugin, as a way to send traffic to test tc filters and actions.
This version includes the patch signoffs missing in the previous submission.

The first patch adds the JSON verification to the core tdc script.

The second patch makes a change to the TdcPlugin module that will allow tdc
plugins to examine the test case currently being executed, such that plugins
can play a more active role in testing. This feature is needed for the
new plugin.

The third patch adds the scapyPlugin itself, and an example test case file to
demonstrate how the scapy block works.


Lucas Bates (3):
  tc-testing: Add JSON verification to tdc
  tc-testing: Allow tdc plugins to see test case data
  tc-testing: introduce scapyPlugin for basic traffic

 tools/testing/selftests/tc-testing/TdcPlugin.py    |   5 +-
 .../creating-testcases/scapy-example.json          |  98 ++++++++++++++
 .../selftests/tc-testing/plugin-lib/scapyPlugin.py |  51 +++++++
 tools/testing/selftests/tc-testing/tdc.py          | 146 ++++++++++++++++++---
 4 files changed, 279 insertions(+), 21 deletions(-)
 create mode 100644 tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json
 create mode 100644 tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py

--
2.7.4

[PATCH v2 net-next 1/3] tc-testing: Add JSON verification to tdc

[Lucas Bates]

This patch allows tdc to process JSON output to perform secondary
verification of the command under test. If the verifyCmd generates
JSON, one can provide the 'matchJSON' key to process it
instead of a regex.

matchJSON has two elements: 'path' and 'value'. The 'path' key is a
list of integers and strings that provide the key values for tdc to
navigate the JSON information. The value is an integer or string
that tdc will compare against what it finds in the provided path.

If the numerical position of an element can vary, it's possible to
substitute an asterisk as a wildcard. tdc will search all possible
entries in the array.

Multiple matches are possible, but everything specified must
match for the test to pass.

If both matchPattern and matchJSON are present, tdc will only
operate on matchPattern. If neither are present, verification
is skipped.

Example:

  "cmdUnderTest": "$TC actions add action pass index 8",
  "verifyCmd": "$TC actions list action gact",
  "matchJSON": [
      {
          "path": [
              0,
              "actions",
              0,
              "control action",
              "type"
          ],
          "value": "gact"
      },
      {
          "path": [
              0,
              "actions",
              0,
              "index"
          ],
          "value": 8
      }
  ]

Signed-off-by: Lucas Bates <lucasb@mojatatu.com>
---
 tools/testing/selftests/tc-testing/tdc.py | 136 +++++++++++++++++++++++++++---
 1 file changed, 123 insertions(+), 13 deletions(-)

diff --git a/tools/testing/selftests/tc-testing/tdc.py b/tools/testing/selftests/tc-testing/tdc.py
index 678182a..1afa803 100755
--- a/tools/testing/selftests/tc-testing/tdc.py
+++ b/tools/testing/selftests/tc-testing/tdc.py
@@ -35,6 +35,10 @@ class PluginMgrTestFail(Exception):
         self.output = output
         self.message = message
 
+class ElemNotFound(Exception):
+    def __init__(self, path_element):
+        self.path_element = path_element
+
 class PluginMgr:
     def __init__(self, argparser):
         super().__init__()
@@ -167,6 +171,40 @@ class PluginMgr:
         self.argparser = argparse.ArgumentParser(
             description='Linux TC unit tests')
 
+def find_in_json(jsonobj, path):
+    if type(jsonobj) == list:
+        if type(path[0]) == int:
+            if len(jsonobj) > path[0]:
+                return find_in_json(jsonobj[path[0]], path[1:])
+            else:
+                raise ElemNotFound(path[0])
+        elif path[0] == '*':
+            res = []
+            for index in jsonobj:
+                try:
+                    res.append(find_in_json(index, path[1:]))
+                except ElemNotFound:
+                    continue
+            if len(res) == 0:
+                raise ElemNotFound(path[0])
+            else:
+                return res
+    elif type(jsonobj) == dict:
+        if path[0] in jsonobj:
+            if len(path) > 1:
+                return find_in_json(jsonobj[path[0]], path[1:])
+            return jsonobj[path[0]]
+        else:
+            raise ElemNotFound(path[0])
+    else:
+        # Assume we have found the correct depth in the object
+        if len(path) >= 1:
+            print('The remainder of the specified path cannot be found!')
+            print('Path values: {}'.format(path))
+            raise ElemNotFound(path[0])
+        return jsonobj
+
+
 def replace_keywords(cmd):
     """
     For a given executable command, substitute any known
@@ -246,6 +284,86 @@ def prepare_env(args, pm, stage, prefix, cmdlist, output = None):
                 stage, output,
                 '"{}" did not complete successfully'.format(prefix))
 
+def verify_by_regex(res, tidx, args, pm):
+    if 'matchCount' not in tidx:
+        res.set_result(ResultState.skip)
+        fmsg = 'matchCount was not provided in the test case. '
+        fmsg += 'Unable to complete pattern match.'
+        res.set_failmsg(fmsg)
+        print(fmsg)
+        return res
+    (p, procout) = exec_cmd(args, pm, 'verify', tidx["verifyCmd"])
+    match_pattern = re.compile(
+        str(tidx["matchPattern"]), re.DOTALL | re.MULTILINE)
+    if procout:
+        match_index = re.findall(match_pattern, procout)
+        if len(match_index) != int(tidx["matchCount"]):
+            res.set_result(ResultState.fail)
+            fmsg = 'Verify stage failed because the output did not match '
+            fmsg += 'the pattern in the test case.\nMatch pattern is:\n'
+            fmsg += '\t{}\n'.format(tidx["matchPattern"])
+            fmsg += 'Output generated by the verify command:\n'
+            fmsg += '{}\n'.format(procout)
+            res.set_failmsg(fmsg)
+        else:
+            res.set_result(ResultState.success)
+    elif int(tidx["matchCount"]) != 0:
+        res.set_result(ResultState.fail)
+        res.set_failmsg('No output generated by verify command.')
+    else:
+        res.set_result(ResultState.success)
+    return res
+
+def verify_by_json(res, tidx, args, pm):
+    # Validate the matchJSON struct
+    for match in tidx['matchJSON']:
+        if 'path' in match and 'value' in match:
+            pass
+        else:
+            res.set_result(ResultState.skip)
+            res.set_failmsg('matchJSON missing required keys for this case.')
+            return res
+    (p, procout) = exec_cmd(args, pm, 'verify', tidx["verifyCmd"])
+    # Run procout through the JSON decoder
+    try:
+        jsonobj = json.loads(procout)
+    except json.JSONDecodeError:
+        if len(tidx['matchJSON']) > 0:
+            res.set_result(ResultState.fail)
+            res.set_failmsg('Cannot decode verify command\'s output. Is it JSON?')
+            return res
+    # Then recurse through the object
+    valuesmatch = True
+    for match in tidx['matchJSON']:
+        try:
+            value = find_in_json(jsonobj, match['path'])
+        except ElemNotFound as ENF:
+            fmsg = 'Could not find the element {} specified in the path.'.format(ENF.path_element)
+            valuesmatch = False
+            break
+        if type(value) == list:
+            if match['value'] not in value:
+                valuesmatch = False
+                fmsg = 'Verify stage failed because the value specified in the path\n'
+                fmsg += '{}\n'.format(match['path'])
+                fmsg += 'Expected value: {}\nReceived value: {}'.format(
+                    match['value'], value)
+                break
+        elif match['value'] != value:
+            valuesmatch = False
+            fmsg = 'Verify stage failed because the value specified in the path\n'
+            fmsg += '{}\n'.format(match['path'])
+            fmsg += 'Expected value: {}\nReceived value: {}'.format(
+                match['value'], value)
+            break
+    if valuesmatch:
+        res.set_result(ResultState.success)
+    else:
+        res.set_result(ResultState.fail)
+        res.set_failmsg(fmsg)
+        print(fmsg)
+    return res
+
 def run_one_test(pm, args, index, tidx):
     global NAMES
     result = True
@@ -292,21 +410,13 @@ def run_one_test(pm, args, index, tidx):
     else:
         if args.verbose > 0:
             print('-----> verify stage')
-        match_pattern = re.compile(
-            str(tidx["matchPattern"]), re.DOTALL | re.MULTILINE)
-        (p, procout) = exec_cmd(args, pm, 'verify', tidx["verifyCmd"])
-        if procout:
-            match_index = re.findall(match_pattern, procout)
-            if len(match_index) != int(tidx["matchCount"]):
-                res.set_result(ResultState.fail)
-                res.set_failmsg('Could not match regex pattern. Verify command output:\n{}'.format(procout))
-            else:
-                res.set_result(ResultState.success)
-        elif int(tidx["matchCount"]) != 0:
-            res.set_result(ResultState.fail)
-            res.set_failmsg('No output generated by verify command.')
+        if 'matchPattern' in tidx:
+            res = verify_by_regex(res, tidx, args, pm)
+        elif 'matchJSON' in tidx:
+            res = verify_by_json(res, tidx, args, pm)
         else:
             res.set_result(ResultState.success)
+            print('No match method defined in current test case, skipping verify')
 
     prepare_env(args, pm, 'teardown', '-----> teardown stage', tidx['teardown'], procout)
     pm.call_post_case()
-- 
2.7.4

[PATCH v2 net-next 2/3] tc-testing: Allow tdc plugins to see test case data

[Lucas Bates]

Instead of only passing the test case name and ID, pass the
entire current test case down to the plugins. This change
allows plugins to start accepting commands and directives
from the test cases themselves, for greater flexibility
in testing.

Signed-off-by: Lucas Bates <lucasb@mojatatu.com>
---
 tools/testing/selftests/tc-testing/TdcPlugin.py |  5 ++---
 tools/testing/selftests/tc-testing/tdc.py       | 10 +++++-----
 2 files changed, 7 insertions(+), 8 deletions(-)

diff --git a/tools/testing/selftests/tc-testing/TdcPlugin.py b/tools/testing/selftests/tc-testing/TdcPlugin.py
index b980a56..79f3ca8 100644
--- a/tools/testing/selftests/tc-testing/TdcPlugin.py
+++ b/tools/testing/selftests/tc-testing/TdcPlugin.py
@@ -18,12 +18,11 @@ class TdcPlugin:
         if self.args.verbose > 1:
             print(' -- {}.post_suite'.format(self.sub_class))
 
-    def pre_case(self, testid, test_name, test_skip):
+    def pre_case(self, caseinfo, test_skip):
         '''run commands before test_runner does one test'''
         if self.args.verbose > 1:
             print(' -- {}.pre_case'.format(self.sub_class))
-        self.args.testid = testid
-        self.args.test_name = test_name
+        self.args.caseinfo = caseinfo
         self.args.test_skip = test_skip
 
     def post_case(self):
diff --git a/tools/testing/selftests/tc-testing/tdc.py b/tools/testing/selftests/tc-testing/tdc.py
index 1afa803..de7da9a 100755
--- a/tools/testing/selftests/tc-testing/tdc.py
+++ b/tools/testing/selftests/tc-testing/tdc.py
@@ -126,15 +126,15 @@ class PluginMgr:
         for pgn_inst in reversed(self.plugin_instances):
             pgn_inst.post_suite(index)
 
-    def call_pre_case(self, testid, test_name, *, test_skip=False):
+    def call_pre_case(self, caseinfo, *, test_skip=False):
         for pgn_inst in self.plugin_instances:
             try:
-                pgn_inst.pre_case(testid, test_name, test_skip)
+                pgn_inst.pre_case(caseinfo, test_skip)
             except Exception as ee:
                 print('exception {} in call to pre_case for {} plugin'.
                       format(ee, pgn_inst.__class__))
                 print('test_ordinal is {}'.format(test_ordinal))
-                print('testid is {}'.format(testid))
+                print('testid is {}'.format(caseinfo['id']))
                 raise
 
     def call_post_case(self):
@@ -379,14 +379,14 @@ def run_one_test(pm, args, index, tidx):
             res = TestResult(tidx['id'], tidx['name'])
             res.set_result(ResultState.skip)
             res.set_errormsg('Test case designated as skipped.')
-            pm.call_pre_case(tidx['id'], tidx['name'], test_skip=True)
+            pm.call_pre_case(tidx, test_skip=True)
             pm.call_post_execute()
             return res
 
     # populate NAMES with TESTID for this test
     NAMES['TESTID'] = tidx['id']
 
-    pm.call_pre_case(tidx['id'], tidx['name'])
+    pm.call_pre_case(tidx)
     prepare_env(args, pm, 'setup', "-----> prepare stage", tidx["setup"])
 
     if (args.verbose > 0):
-- 
2.7.4

[PATCH v2 net-next 3/3] tc-testing: introduce scapyPlugin for basic traffic

[Lucas Bates]

The scapyPlugin allows for simple traffic generation in tdc to
test various tc features. It was tested with scapy v2.4.2, but
should work with any successive version.

In order to use the plugin's functionality, scapy must be
installed. This can be done with:
   pip3 install scapy

or to install 2.4.2:
   pip3 install scapy==2.4.2

If the plugin is unable to import the scapy module, it will
terminate the tdc run.

The plugin makes use of a new key in the test case data, 'scapy'.
This block contains three other elements: 'iface', 'count', and
'packet':

        "scapy": {
            "iface": "$DEV0",
            "count": 1,
            "packet": "Ether(type=0x800)/IP(src='16.61.16.61')/ICMP()"
        },

* iface is the name of the device on the host machine from which
  the packet(s) will be sent. Values contained within tdc_config.py's
  NAMES dict can be used here - this is useful if paired with
  nsPlugin
* count is the number of copies of this packet to be sent
* packet is a string detailing the different layers of the packet
  to be sent. If a property isn't explicitly set, scapy will set
  default values for you.

Layers in the packet info are separated by slashes. For info about
common TCP and IP properties, see:
https://blogs.sans.org/pen-testing/files/2016/04/ScapyCheatSheet_v0.2.pdf

Caution is advised when running tests using the scapy functionality,
since the plugin blindly sends the packet as defined in the test case
data.

See creating-testcases/scapy-example.json for sample test cases;
the first test is intended to pass while the second is intended to
fail. Consider using the matchJSON functionality for verification
when using scapy.

Signed-off-by: Lucas Bates <lucasb@mojatatu.com>
---
 .../creating-testcases/scapy-example.json          | 98 ++++++++++++++++++++++
 .../selftests/tc-testing/plugin-lib/scapyPlugin.py | 51 +++++++++++
 2 files changed, 149 insertions(+)
 create mode 100644 tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json
 create mode 100644 tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py

diff --git a/tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json b/tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json
new file mode 100644
index 0000000..5a9377b
--- /dev/null
+++ b/tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json
@@ -0,0 +1,98 @@
+[
+    {
+        "id": "b1e9",
+        "name": "Test matching of source IP",
+        "category": [
+            "actions",
+            "scapy"
+        ],
+        "plugins": {
+            "requires": [
+                "nsPlugin",
+                "scapyPlugin"
+            ]
+        },
+        "setup": [
+            [
+                "$TC qdisc del dev $DEV1 ingress",
+                0,
+                1,
+                2,
+                255
+            ],
+            "$TC qdisc add dev $DEV1 ingress"
+        ],
+        "cmdUnderTest": "$TC filter add dev $DEV1 parent ffff: prio 3 protocol ip flower src_ip 16.61.16.61 flowid 1:1 action ok",
+        "scapy": {
+            "iface": "$DEV0",
+            "count": 1,
+            "packet": "Ether(type=0x800)/IP(src='16.61.16.61')/ICMP()"
+        },
+        "expExitCode": "0",
+        "verifyCmd": "$TC -s -j filter ls dev $DEV1 ingress prio 3",
+        "matchJSON": [
+            {
+                "path": [
+                    1,
+                    "options",
+                    "actions",
+                    0,
+                    "stats",
+                    "packets"
+                ],
+                "value": 1
+            }
+        ],
+        "teardown": [
+            "$TC qdisc del dev $DEV1 ingress"
+        ]
+    },
+    {
+        "id": "e9c4",
+        "name": "Test matching of source IP with wrong count",
+        "category": [
+            "actions",
+            "scapy"
+        ],
+        "plugins": {
+            "requires": [
+                "nsPlugin",
+                "scapyPlugin"
+            ]
+        },
+        "setup": [
+            [
+                "$TC qdisc del dev $DEV1 ingress",
+                0,
+                1,
+                2,
+                255
+            ],
+            "$TC qdisc add dev $DEV1 ingress"
+        ],
+        "cmdUnderTest": "$TC filter add dev $DEV1 parent ffff: prio 3 protocol ip flower src_ip 16.61.16.61 flowid 1:1 action ok",
+        "scapy": {
+            "iface": "$DEV0",
+            "count": 3,
+            "packet": "Ether(type=0x800)/IP(src='16.61.16.61')/ICMP()"
+        },
+        "expExitCode": "0",
+        "verifyCmd": "$TC -s -j filter ls dev $DEV1 parent ffff:",
+        "matchJSON": [
+            {
+                "path": [
+                    1,
+                    "options",
+                    "actions",
+                    0,
+                    "stats",
+                    "packets"
+                ],
+                "value": 1
+            }
+        ],
+        "teardown": [
+            "$TC qdisc del dev $DEV1 ingress"
+        ]
+    }
+]
diff --git a/tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py b/tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py
new file mode 100644
index 0000000..db57916
--- /dev/null
+++ b/tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py
@@ -0,0 +1,51 @@
+#!/usr/bin/env python3
+
+import os
+import signal
+from string import Template
+import subprocess
+import time
+from TdcPlugin import TdcPlugin
+
+from tdc_config import *
+
+try:
+    from scapy.all import *
+except ImportError:
+    print("Unable to import the scapy python module.")
+    print("\nIf not already installed, you may do so with:")
+    print("\t\tpip3 install scapy==2.4.2")
+    exit(1)
+
+class SubPlugin(TdcPlugin):
+    def __init__(self):
+        self.sub_class = 'scapy/SubPlugin'
+        super().__init__()
+
+    def post_execute(self):
+        if 'scapy' not in self.args.caseinfo:
+            if self.args.verbose:
+                print('{}.post_execute: no scapy info in test case'.format(self.sub_class))
+            return
+
+        # Check for required fields
+        scapyinfo = self.args.caseinfo['scapy']
+        scapy_keys = ['iface', 'count', 'packet']
+        missing_keys = []
+        keyfail = False
+        for k in scapy_keys:
+            if k not in scapyinfo:
+                keyfail = True
+                missing_keys.add(k)
+        if keyfail:
+            print('{}: Scapy block present in the test, but is missing info:'
+                .format(self.sub_class))
+            print('{}'.format(missing_keys))
+
+        pkt = eval(scapyinfo['packet'])
+        if '$' in scapyinfo['iface']:
+            tpl = Template(scapyinfo['iface'])
+            scapyinfo['iface'] = tpl.safe_substitute(NAMES)
+        for count in range(scapyinfo['count']):
+            sendp(pkt, iface=scapyinfo['iface'])
+
-- 
2.7.4

Re: [PATCH v2 net-next 1/3] tc-testing: Add JSON verification to tdc

[Alexander Aring]

Hi,

On Wed, Jul 03, 2019 at 08:45:00PM -0400, Lucas Bates wrote:
> This patch allows tdc to process JSON output to perform secondary
> verification of the command under test. If the verifyCmd generates
> JSON, one can provide the 'matchJSON' key to process it
> instead of a regex.
> 
> matchJSON has two elements: 'path' and 'value'. The 'path' key is a
> list of integers and strings that provide the key values for tdc to
> navigate the JSON information. The value is an integer or string
> that tdc will compare against what it finds in the provided path.
> 
> If the numerical position of an element can vary, it's possible to
> substitute an asterisk as a wildcard. tdc will search all possible
> entries in the array.
> 
> Multiple matches are possible, but everything specified must
> match for the test to pass.
> 
> If both matchPattern and matchJSON are present, tdc will only
> operate on matchPattern. If neither are present, verification
> is skipped.
> 
> Example:
> 
>   "cmdUnderTest": "$TC actions add action pass index 8",
>   "verifyCmd": "$TC actions list action gact",
>   "matchJSON": [
>       {
>           "path": [
>               0,
>               "actions",
>               0,
>               "control action",
>               "type"
>           ],
>           "value": "gact"
>       },
>       {
>           "path": [
>               0,
>               "actions",
>               0,
>               "index"
>           ],
>           "value": 8
>       }
>   ]

why you just use eval() as pattern matching operation and let the user
define how to declare a matching mechanism instead you introduce another
static matching scheme based on a json description?

Whereas in eval() you could directly use the python bool expression
parser to make whatever you want.

I don't know, I see at some points you will hit limitations what you can
express with this matchFOO and we need to introduce another matchBAR,
whereas in providing the code it should be no problem expression
anything. If you want smaller shortcuts writing matching patterns you
can implement them and using in your eval() operation.

- Alex

Re: [PATCH v2 net-next 3/3] tc-testing: introduce scapyPlugin for basic traffic

[Alexander Aring]

Hi,

On Wed, Jul 03, 2019 at 08:45:02PM -0400, Lucas Bates wrote:
> The scapyPlugin allows for simple traffic generation in tdc to
> test various tc features. It was tested with scapy v2.4.2, but
> should work with any successive version.
> 
> In order to use the plugin's functionality, scapy must be
> installed. This can be done with:
>    pip3 install scapy
> 
> or to install 2.4.2:
>    pip3 install scapy==2.4.2
> 
> If the plugin is unable to import the scapy module, it will
> terminate the tdc run.
> 
> The plugin makes use of a new key in the test case data, 'scapy'.
> This block contains three other elements: 'iface', 'count', and
> 'packet':
> 
>         "scapy": {
>             "iface": "$DEV0",
>             "count": 1,
>             "packet": "Ether(type=0x800)/IP(src='16.61.16.61')/ICMP()"
>         },
> 
> * iface is the name of the device on the host machine from which
>   the packet(s) will be sent. Values contained within tdc_config.py's
>   NAMES dict can be used here - this is useful if paired with
>   nsPlugin
> * count is the number of copies of this packet to be sent
> * packet is a string detailing the different layers of the packet
>   to be sent. If a property isn't explicitly set, scapy will set
>   default values for you.
> 
> Layers in the packet info are separated by slashes. For info about
> common TCP and IP properties, see:
> https://blogs.sans.org/pen-testing/files/2016/04/ScapyCheatSheet_v0.2.pdf
> 
> Caution is advised when running tests using the scapy functionality,
> since the plugin blindly sends the packet as defined in the test case
> data.
> 
> See creating-testcases/scapy-example.json for sample test cases;
> the first test is intended to pass while the second is intended to
> fail. Consider using the matchJSON functionality for verification
> when using scapy.
> 

Is there a way to introduce thrid party scapy level descriptions which
are not upstream yet?

- Alex

Re: [PATCH v2 net-next 0/3] tc-testing: Add JSON verification and simple traffic generation

[David Miller]

From: Lucas Bates <lucasb@mojatatu.com>
Date: Wed,  3 Jul 2019 20:44:59 -0400

> This patchset introduces JSON as a verification method in tdc and adds a new
> plugin, scapyPlugin, as a way to send traffic to test tc filters and actions.
> This version includes the patch signoffs missing in the previous submission.
> 
> The first patch adds the JSON verification to the core tdc script.
> 
> The second patch makes a change to the TdcPlugin module that will allow tdc
> plugins to examine the test case currently being executed, such that plugins
> can play a more active role in testing. This feature is needed for the
> new plugin.
> 
> The third patch adds the scapyPlugin itself, and an example test case file to
> demonstrate how the scapy block works.

Lucas, please address the feedback about using eval().

Thank you.

Re: [PATCH v2 net-next 1/3] tc-testing: Add JSON verification to tdc

[Lucas Bates]

On Thu, Jul 4, 2019 at 4:21 PM Alexander Aring <aring@mojatatu.com> wrote:

> why you just use eval() as pattern matching operation and let the user
> define how to declare a matching mechanism instead you introduce another
> static matching scheme based on a json description?
>
> Whereas in eval() you could directly use the python bool expression
> parser to make whatever you want.
>
> I don't know, I see at some points you will hit limitations what you can
> express with this matchFOO and we need to introduce another matchBAR,
> whereas in providing the code it should be no problem expression
> anything. If you want smaller shortcuts writing matching patterns you
> can implement them and using in your eval() operation.

Regarding hitting limitations: quite possibly, yes.

Using eval() to provide code for matching is going to put more of a
dependency on the test writer knowing Python.  I know it's not a
terribly difficult language to pick up, but it's still setting a
higher barrier to entry.  This is the primary reason I scrapped the
work I had presented at Netdev 1.2 in Tokyo, where all the tests were
coded using Python's unittest framework - I want to be sure it's as
easy as possible for people to use tdc and write tests for it.

Unless I'm off-base here?

Lucas

Re: [PATCH v2 net-next 1/3] tc-testing: Add JSON verification to tdc

[Alexander Aring]

Hi,

On Mon, Jul 08, 2019 at 12:48:12PM -0400, Lucas Bates wrote:
> On Thu, Jul 4, 2019 at 4:21 PM Alexander Aring <aring@mojatatu.com> wrote:
> 
> > why you just use eval() as pattern matching operation and let the user
> > define how to declare a matching mechanism instead you introduce another
> > static matching scheme based on a json description?
> >
> > Whereas in eval() you could directly use the python bool expression
> > parser to make whatever you want.
> >
> > I don't know, I see at some points you will hit limitations what you can
> > express with this matchFOO and we need to introduce another matchBAR,
> > whereas in providing the code it should be no problem expression
> > anything. If you want smaller shortcuts writing matching patterns you
> > can implement them and using in your eval() operation.
> 
> Regarding hitting limitations: quite possibly, yes.
> 
> Using eval() to provide code for matching is going to put more of a
> dependency on the test writer knowing Python.  I know it's not a
> terribly difficult language to pick up, but it's still setting a
> higher barrier to entry.  This is the primary reason I scrapped the
> work I had presented at Netdev 1.2 in Tokyo, where all the tests were
> coded using Python's unittest framework - I want to be sure it's as
> easy as possible for people to use tdc and write tests for it.
> 
> Unless I'm off-base here?

yes you need to know some python, complex code can be hidden by some
helper functionality I guess.

I have no problem to let this patch in, it will not harm anything...

Maybe I work on a matchEval and show some examples... in a human
readable way you can even concatenate bool expressions in combinations
with helpers.

I just was curious, so I might add the matchEval or something to show
this approach.

add the and it shows like:

"x == 5 or x == '5'"

Whereas you could introduce helpers to do:

"str_or_num(x, 5)"

even

"str_or_num_any_base(x, 5)"

to also catch if somebody change the base.
In this case "or" could be also concatenate with python bool
expression... depends on how lowlvl your helpers be.

Pretty sure the x as inputstring to match can also be hidden by user or
transformed with split, regex, etc before. At the end it will work like
TC with actions just provide the code to run... or is it more like
"act_bpf"?, where act is the hook and bpf the eval(). :-)

- Alex

Re: [PATCH v2 net-next 1/3] tc-testing: Add JSON verification to tdc

[Lucas Bates]

On Mon, Jul 8, 2019 at 1:25 PM Alexander Aring <aring@mojatatu.com> wrote:
> > Unless I'm off-base here?
>
> yes you need to know some python, complex code can be hidden by some
> helper functionality I guess.
>
> I have no problem to let this patch in, it will not harm anything...
I think I'm going to pull it for the moment - I started thinking about
the patch today and I think it needs more testing against larger
amounts of data.

> Maybe I work on a matchEval and show some examples... in a human
> readable way you can even concatenate bool expressions in combinations
> with helpers.
>
> I just was curious, so I might add the matchEval or something to show
> this approach.

Go for it, I think you have a much better grasp on the use of eval
than I do - and it could be very useful for test cases.

Re: [PATCH v2 net-next 3/3] tc-testing: introduce scapyPlugin for basic traffic

[Lucas Bates]

Sorry Alex, I completely forgot about this email.
On Thu, Jul 4, 2019 at 4:29 PM Alexander Aring <aring@mojatatu.com> wrote:
>
> Hi,
>
> On Wed, Jul 03, 2019 at 08:45:02PM -0400, Lucas Bates wrote:
> > The scapyPlugin allows for simple traffic generation in tdc to
> > test various tc features. It was tested with scapy v2.4.2, but
> > should work with any successive version.
> Is there a way to introduce thrid party scapy level descriptions which
> are not upstream yet?

Upstream to scapy? Not yet.  This version of the plugin is extremely
simple, and good for basic traffic.  I'll add features to it so we can
get more creative with the packets that can be sent, though.

Lucas

Re: [PATCH v2 net-next 3/3] tc-testing: introduce scapyPlugin for basic traffic

[Alexander Aring]

On Mon, Jul 08, 2019 at 09:28:09PM -0400, Lucas Bates wrote:
> Sorry Alex, I completely forgot about this email.
> On Thu, Jul 4, 2019 at 4:29 PM Alexander Aring <aring@mojatatu.com> wrote:
> >
> > Hi,
> >
> > On Wed, Jul 03, 2019 at 08:45:02PM -0400, Lucas Bates wrote:
> > > The scapyPlugin allows for simple traffic generation in tdc to
> > > test various tc features. It was tested with scapy v2.4.2, but
> > > should work with any successive version.
> > Is there a way to introduce thrid party scapy level descriptions which
> > are not upstream yet?
> 
> Upstream to scapy? Not yet.  This version of the plugin is extremely
> simple, and good for basic traffic.  I'll add features to it so we can
> get more creative with the packets that can be sent, though.
> 

Can you add this now? I have some tests here for ife and I am on the way
to send it upstream to scapy.

So far this isn't done yet, I like to provide them via a external
directory in the tctesting directory.

Thanks.

- Alex