[nft PATCH v3 0/2] JSON schema for nftables.py

[nft PATCH v3 0/2] JSON schema for nftables.py

[Phil Sutter]

Round three of JSON validation enhancement.

Changes since v2:
- Make enhancement to nftables module Python3 compliant.
- Complain in nft-test.py if --schema was given without --json.

Changes since v1:
- Fix patch 2 commit message, thanks to Jones Desougi who reported the
  inconsistency.

Changes since RFC:
- Import builtin traceback module unconditionally.

Phil Sutter (2):
  py: Implement JSON validation in nftables module
  tests/py: Support JSON validation

 py/Makefile.am       |  2 +-
 py/nftables.py       | 29 +++++++++++++++++++++++++++++
 py/schema.json       | 17 +++++++++++++++++
 py/setup.py          |  1 +
 tests/py/nft-test.py | 25 ++++++++++++++++++++++++-
 5 files changed, 72 insertions(+), 2 deletions(-)
 create mode 100644 py/schema.json

-- 
2.21.0

[nft PATCH v3 1/2] py: Implement JSON validation in nftables module

[Phil Sutter]

Using jsonschema it is possible to validate any JSON input to make sure
it formally conforms with libnftables JSON API requirements.

Implement a simple validator class for use within a new Nftables class
method 'json_validate' and ship a minimal schema definition along with
the package.

Signed-off-by: Phil Sutter <phil@nwl.cc>
---
Changes since v2:
- Replace file() as that is not supported by python3, instead use open()
  and that fancy 'with' statement.
---
 py/Makefile.am |  2 +-
 py/nftables.py | 29 +++++++++++++++++++++++++++++
 py/schema.json | 17 +++++++++++++++++
 py/setup.py    |  1 +
 4 files changed, 48 insertions(+), 1 deletion(-)
 create mode 100644 py/schema.json

diff --git a/py/Makefile.am b/py/Makefile.am
index 0963535d068dc..9fce7c9e54c38 100644
--- a/py/Makefile.am
+++ b/py/Makefile.am
@@ -1,4 +1,4 @@
-EXTRA_DIST = setup.py __init__.py nftables.py
+EXTRA_DIST = setup.py __init__.py nftables.py schema.json
 
 if HAVE_PYTHON
 
diff --git a/py/nftables.py b/py/nftables.py
index 33cd2dfd736d4..81e57567c8024 100644
--- a/py/nftables.py
+++ b/py/nftables.py
@@ -17,9 +17,23 @@
 import json
 from ctypes import *
 import sys
+import os
 
 NFTABLES_VERSION = "0.1"
 
+class SchemaValidator:
+    """Libnftables JSON validator using jsonschema"""
+
+    def __init__(self):
+        schema_path = os.path.join(os.path.dirname(__file__), "schema.json")
+        with open(schema_path, 'r') as schema_file:
+            self.schema = json.load(schema_file)
+        import jsonschema
+        self.jsonschema = jsonschema
+
+    def validate(self, json):
+        self.jsonschema.validate(instance=json, schema=self.schema)
+
 class Nftables:
     """A class representing libnftables interface"""
 
@@ -46,6 +60,8 @@ class Nftables:
         "numeric_symbol": (1 << 9),
     }
 
+    validator = None
+
     def __init__(self, sofile="libnftables.so"):
         """Instantiate a new Nftables class object.
 
@@ -382,3 +398,16 @@ class Nftables:
         if len(output):
             output = json.loads(output)
         return (rc, output, error)
+
+    def json_validate(self, json_root):
+        """Validate JSON object against libnftables schema.
+
+        Accepts a hash object as input.
+
+        Returns True if JSON is valid, raises an exception otherwise.
+        """
+        if not self.validator:
+            self.validator = SchemaValidator()
+
+        self.validator.validate(json_root)
+        return True
diff --git a/py/schema.json b/py/schema.json
new file mode 100644
index 0000000000000..6cb731a228bf4
--- /dev/null
+++ b/py/schema.json
@@ -0,0 +1,17 @@
+{
+	"$schema": "http://json-schema.org/schema#",
+	"id": "http://netfilter.org/nftables/ruleset-schema.json",
+	"description": "libnftables JSON API schema",
+
+	"type": "object",
+        "properties": {
+		"nftables": {
+			"type": "array",
+			"minitems": 0,
+			"items": {
+				"type": "object"
+			}
+		}
+	},
+	"required": [ "nftables" ]
+}
diff --git a/py/setup.py b/py/setup.py
index ef143c42a21b0..72fc8fd98b269 100755
--- a/py/setup.py
+++ b/py/setup.py
@@ -11,6 +11,7 @@ setup(name='nftables',
       packages=['nftables'],
       provides=['nftables'],
       package_dir={'nftables':'.'},
+      package_data={'nftables':['schema.json']},
       classifiers=[
           'Development Status :: 4 - Beta',
           'Environment :: Console',
-- 
2.21.0

[nft PATCH v3 2/2] tests/py: Support JSON validation

[Phil Sutter]

Introduce a new flag -s/--schema to nft-test.py which enables validation
of any JSON input and output against our schema.

Make use of traceback module to get more details if validation fails.

Signed-off-by: Phil Sutter <phil@nwl.cc>
---
Changes since v2:
- Complain if --schema was given but not --json.

Changes since v1:
- Adjust commit message to changes from RFC.

Changes since RFC:
- Import builtin traceback module unconditionally
---
 tests/py/nft-test.py | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/tests/py/nft-test.py b/tests/py/nft-test.py
index 1c0afd0ec0eb3..09d00dba1510a 100755
--- a/tests/py/nft-test.py
+++ b/tests/py/nft-test.py
@@ -18,6 +18,7 @@ import os
 import argparse
 import signal
 import json
+import traceback
 
 TESTS_PATH = os.path.dirname(os.path.abspath(__file__))
 sys.path.insert(0, os.path.join(TESTS_PATH, '../../py/'))
@@ -687,6 +688,13 @@ def json_dump_normalize(json_string, human_readable = False):
     else:
         return json.dumps(json_obj, sort_keys = True)
 
+def json_validate(json_string):
+    json_obj = json.loads(json_string)
+    try:
+        nftables.json_validate(json_obj)
+    except Exception:
+        print_error("schema validation failed for input '%s'" % json_string)
+        print_error(traceback.format_exc())
 
 def rule_add(rule, filename, lineno, force_all_family_option, filename_path):
     '''
@@ -912,6 +920,9 @@ def rule_add(rule, filename, lineno, force_all_family_option, filename_path):
                     "expr": json.loads(json_input),
             }}}]})
 
+            if enable_json_schema:
+                json_validate(cmd)
+
             json_old = nftables.set_json_output(True)
             ret = execute_cmd(cmd, filename, lineno, payload_log, debug="netlink")
             nftables.set_json_output(json_old)
@@ -945,6 +956,9 @@ def rule_add(rule, filename, lineno, force_all_family_option, filename_path):
             nftables.set_numeric_proto_output(numeric_proto_old)
             nftables.set_stateless_output(stateless_old)
 
+            if enable_json_schema:
+                json_validate(json_output)
+
             json_output = json.loads(json_output)
             for item in json_output["nftables"]:
                 if "rule" in item:
@@ -1341,12 +1355,17 @@ def main():
                         dest='enable_json',
                         help='test JSON functionality as well')
 
+    parser.add_argument('-s', '--schema', action='store_true',
+                        dest='enable_schema',
+                        help='verify json input/output against schema')
+
     args = parser.parse_args()
-    global debug_option, need_fix_option, enable_json_option
+    global debug_option, need_fix_option, enable_json_option, enable_json_schema
     debug_option = args.debug
     need_fix_option = args.need_fix_line
     force_all_family_option = args.force_all_family
     enable_json_option = args.enable_json
+    enable_json_schema = args.enable_schema
     specific_file = False
 
     signal.signal(signal.SIGINT, signal_handler)
@@ -1364,6 +1383,10 @@ def main():
               "You need to build the project."
         return
 
+    if args.enable_schema and not args.enable_json:
+        print_error("Option --schema requires option --json")
+        return
+
     global nftables
     nftables = Nftables(sofile = 'src/.libs/libnftables.so')
 
-- 
2.21.0

Re: [nft PATCH v3 1/2] py: Implement JSON validation in nftables module

[Eric Garver]

On Wed, May 22, 2019 at 06:14:52PM +0200, Phil Sutter wrote:
> Using jsonschema it is possible to validate any JSON input to make sure
> it formally conforms with libnftables JSON API requirements.
> 
> Implement a simple validator class for use within a new Nftables class
> method 'json_validate' and ship a minimal schema definition along with
> the package.
> 
> Signed-off-by: Phil Sutter <phil@nwl.cc>
> ---
> Changes since v2:
> - Replace file() as that is not supported by python3, instead use open()
>   and that fancy 'with' statement.
> ---

Thanks Phil!

Acked-by: Eric Garver <eric@garver.life>

Re: [nft PATCH v3 2/2] tests/py: Support JSON validation

[Eric Garver]

On Wed, May 22, 2019 at 06:14:53PM +0200, Phil Sutter wrote:
> Introduce a new flag -s/--schema to nft-test.py which enables validation
> of any JSON input and output against our schema.
> 
> Make use of traceback module to get more details if validation fails.
> 
> Signed-off-by: Phil Sutter <phil@nwl.cc>
> ---
> Changes since v2:
> - Complain if --schema was given but not --json.
> 
> Changes since v1:
> - Adjust commit message to changes from RFC.
> 
> Changes since RFC:
> - Import builtin traceback module unconditionally
> ---

Acked-by: Eric Garver <eric@garver.life>

Re: [nft PATCH v3 0/2] JSON schema for nftables.py

[Pablo Neira Ayuso]

On Wed, May 22, 2019 at 06:14:51PM +0200, Phil Sutter wrote:
> Round three of JSON validation enhancement.
> 
> Changes since v2:
> - Make enhancement to nftables module Python3 compliant.
> - Complain in nft-test.py if --schema was given without --json.
> 
> Changes since v1:
> - Fix patch 2 commit message, thanks to Jones Desougi who reported the
>   inconsistency.
> 
> Changes since RFC:
> - Import builtin traceback module unconditionally.
> 
> Phil Sutter (2):
>   py: Implement JSON validation in nftables module
>   tests/py: Support JSON validation
> 
>  py/Makefile.am       |  2 +-
>  py/nftables.py       | 29 +++++++++++++++++++++++++++++
>  py/schema.json       | 17 +++++++++++++++++
>  py/setup.py          |  1 +
>  tests/py/nft-test.py | 25 ++++++++++++++++++++++++-

Where is ruleset-schema.json?

+       "id": "http://netfilter.org/nftables/ruleset-schema.json",
+       "description": "libnftables JSON API schema",

Re: [nft PATCH v3 0/2] JSON schema for nftables.py

[Phil Sutter]

Hi Pablo,

On Fri, May 24, 2019 at 10:45:14PM +0200, Pablo Neira Ayuso wrote:
> On Wed, May 22, 2019 at 06:14:51PM +0200, Phil Sutter wrote:
> > Round three of JSON validation enhancement.
> > 
> > Changes since v2:
> > - Make enhancement to nftables module Python3 compliant.
> > - Complain in nft-test.py if --schema was given without --json.
> > 
> > Changes since v1:
> > - Fix patch 2 commit message, thanks to Jones Desougi who reported the
> >   inconsistency.
> > 
> > Changes since RFC:
> > - Import builtin traceback module unconditionally.
> > 
> > Phil Sutter (2):
> >   py: Implement JSON validation in nftables module
> >   tests/py: Support JSON validation
> > 
> >  py/Makefile.am       |  2 +-
> >  py/nftables.py       | 29 +++++++++++++++++++++++++++++
> >  py/schema.json       | 17 +++++++++++++++++
> >  py/setup.py          |  1 +
> >  tests/py/nft-test.py | 25 ++++++++++++++++++++++++-
> 
> Where is ruleset-schema.json?
> 
> +       "id": "http://netfilter.org/nftables/ruleset-schema.json",
> +       "description": "libnftables JSON API schema",

Oh, I forgot about that. There are actually two problems with it: On one
hand, current draft version suggests to use "$id" instead of "id" for
the property name. On the other, the URL should point to an online
location of the document itself, which is obviously not correct.

Given that it is optional according to the draft, I would just drop it
for now. What do you think?

Thanks, Phil

Re: [nft PATCH v3 0/2] JSON schema for nftables.py

[Pablo Neira Ayuso]

On Mon, May 27, 2019 at 11:57:20AM +0200, Phil Sutter wrote:
> Hi Pablo,
> 
> On Fri, May 24, 2019 at 10:45:14PM +0200, Pablo Neira Ayuso wrote:
> > On Wed, May 22, 2019 at 06:14:51PM +0200, Phil Sutter wrote:
> > > Round three of JSON validation enhancement.
> > > 
> > > Changes since v2:
> > > - Make enhancement to nftables module Python3 compliant.
> > > - Complain in nft-test.py if --schema was given without --json.
> > > 
> > > Changes since v1:
> > > - Fix patch 2 commit message, thanks to Jones Desougi who reported the
> > >   inconsistency.
> > > 
> > > Changes since RFC:
> > > - Import builtin traceback module unconditionally.
> > > 
> > > Phil Sutter (2):
> > >   py: Implement JSON validation in nftables module
> > >   tests/py: Support JSON validation
> > > 
> > >  py/Makefile.am       |  2 +-
> > >  py/nftables.py       | 29 +++++++++++++++++++++++++++++
> > >  py/schema.json       | 17 +++++++++++++++++
> > >  py/setup.py          |  1 +
> > >  tests/py/nft-test.py | 25 ++++++++++++++++++++++++-
> > 
> > Where is ruleset-schema.json?
> > 
> > +       "id": "http://netfilter.org/nftables/ruleset-schema.json",
> > +       "description": "libnftables JSON API schema",
> 
> Oh, I forgot about that. There are actually two problems with it: On one
> hand, current draft version suggests to use "$id" instead of "id" for
> the property name. On the other, the URL should point to an online
> location of the document itself, which is obviously not correct.

We can upload it to exactly the location you specify above, that won't
be a problem.

> Given that it is optional according to the draft, I would just drop it
> for now. What do you think?

Drop for now is fine with fine.

Thanks.

Re: [nft PATCH v3 0/2] JSON schema for nftables.py

[Phil Sutter]

Hi,

On Mon, May 27, 2019 at 12:02:38PM +0200, Pablo Neira Ayuso wrote:
> On Mon, May 27, 2019 at 11:57:20AM +0200, Phil Sutter wrote:
> > Hi Pablo,
> > 
> > On Fri, May 24, 2019 at 10:45:14PM +0200, Pablo Neira Ayuso wrote:
> > > On Wed, May 22, 2019 at 06:14:51PM +0200, Phil Sutter wrote:
> > > > Round three of JSON validation enhancement.
> > > > 
> > > > Changes since v2:
> > > > - Make enhancement to nftables module Python3 compliant.
> > > > - Complain in nft-test.py if --schema was given without --json.
> > > > 
> > > > Changes since v1:
> > > > - Fix patch 2 commit message, thanks to Jones Desougi who reported the
> > > >   inconsistency.
> > > > 
> > > > Changes since RFC:
> > > > - Import builtin traceback module unconditionally.
> > > > 
> > > > Phil Sutter (2):
> > > >   py: Implement JSON validation in nftables module
> > > >   tests/py: Support JSON validation
> > > > 
> > > >  py/Makefile.am       |  2 +-
> > > >  py/nftables.py       | 29 +++++++++++++++++++++++++++++
> > > >  py/schema.json       | 17 +++++++++++++++++
> > > >  py/setup.py          |  1 +
> > > >  tests/py/nft-test.py | 25 ++++++++++++++++++++++++-
> > > 
> > > Where is ruleset-schema.json?
> > > 
> > > +       "id": "http://netfilter.org/nftables/ruleset-schema.json",
> > > +       "description": "libnftables JSON API schema",
> > 
> > Oh, I forgot about that. There are actually two problems with it: On one
> > hand, current draft version suggests to use "$id" instead of "id" for
> > the property name. On the other, the URL should point to an online
> > location of the document itself, which is obviously not correct.
> 
> We can upload it to exactly the location you specify above, that won't
> be a problem.

It is far from complete, so I guess that would be unnecessary work.
> 
> > Given that it is optional according to the draft, I would just drop it
> > for now. What do you think?
> 
> Drop for now is fine with fine.

If it's fine with fine it's finest with me, too! :D

I'll send a v4.

Thanks, Phil