[iptables PATCH v3 00/12] Implement among match support

[iptables PATCH v3 00/12] Implement among match support

[Phil Sutter]

Changes since v2:
- Rebased onto current master branch. With cache levels being upstream,
  no dependencies on unfinished work remains.
- Integrate sets caching into cache level infrastructure.
- Fixed temporary build breakage within series.
- Larger review of last patch containing the actual among match
  extension.

Changes since v1:
- Rebased onto my performance improvements patch series.
- Aligned set caching routines with changes in above series.
- Fixed patch ordering so builds are not broken intermittently.
- Replaced magic numbers by defines or offsetof() statements. Note that
  I did not move any defines into libnftnl; the remaining ones are for
  values in sets' key_type attribute which neither libnftnl nor kernel
  care about. Setting that is merely for compatibility with nft tool.

This series ultimately adds among match support to ebtables-nft. The
implementation merely shares the user interface with legacy one,
internally the code is distinct: libebt_among.c does not make use of the
wormhash data structure but a much simpler one for "temporary" storage
of data until being converted into an anonymous set and associated
lookup expression.

Patches 1 to 5 implement required changes and are rather boring by
themselves: When converting an nftnl rule to iptables command state,
cache access is required (to lookup set references).

Patch 6 simplifies things a bit with the above in place.

Patches 7 to 11 implement anonymous set support.

Patch 12 then adds the actual among match implementation for
ebtables-nft.

Phil Sutter (12):
  nft: family_ops: Pass nft_handle to 'add' callback
  nft: family_ops: Pass nft_handle to 'rule_find' callback
  nft: family_ops: Pass nft_handle to 'print_rule' callback
  nft: family_ops: Pass nft_handle to 'rule_to_cs' callback
  nft: Keep nft_handle pointer in nft_xt_ctx
  nft: Eliminate pointless calls to nft_family_ops_lookup()
  nft: Introduce NFT_CL_SETS cache level
  nft: Support NFT_COMPAT_SET_ADD
  nft: Bore up nft_parse_payload()
  nft: Embed rule's table name in nft_xt_ctx
  nft: Support parsing lookup expression
  nft: bridge: Rudimental among extension support

 extensions/libebt_among.c  | 243 +++++++++++++++++++++++++++++++++
 extensions/libebt_among.t  |  16 +++
 iptables/ebtables-nft.8    |  66 ++++-----
 iptables/nft-arp.c         |  13 +-
 iptables/nft-bridge.c      | 232 ++++++++++++++++++++++++++++++--
 iptables/nft-bridge.h      |  56 ++++++++
 iptables/nft-cache.c       | 205 ++++++++++++++++++++++++++--
 iptables/nft-cache.h       |   2 +
 iptables/nft-ipv4.c        |  10 +-
 iptables/nft-ipv6.c        |  10 +-
 iptables/nft-shared.c      |  70 +++++-----
 iptables/nft-shared.h      |  26 ++--
 iptables/nft.c             | 269 ++++++++++++++++++++++++++++++++-----
 iptables/nft.h             |   8 +-
 iptables/xtables-eb.c      |   1 +
 iptables/xtables-monitor.c |  17 ++-
 iptables/xtables-save.c    |   3 +
 17 files changed, 1101 insertions(+), 146 deletions(-)
 create mode 100644 extensions/libebt_among.c
 create mode 100644 extensions/libebt_among.t

-- 
2.23.0

[iptables PATCH v3 01/12] nft: family_ops: Pass nft_handle to 'add' callback

[Phil Sutter]

In order for add_match() to create anonymous sets when converting
xtables matches it needs access to nft handle. So pass it along from
callers of family ops' add callback.

Signed-off-by: Phil Sutter <phil@nwl.cc>
---
 iptables/nft-arp.c    | 2 +-
 iptables/nft-bridge.c | 5 +++--
 iptables/nft-ipv4.c   | 4 ++--
 iptables/nft-ipv6.c   | 4 ++--
 iptables/nft-shared.h | 4 ++--
 iptables/nft.c        | 5 +++--
 iptables/nft.h        | 2 +-
 7 files changed, 14 insertions(+), 12 deletions(-)

diff --git a/iptables/nft-arp.c b/iptables/nft-arp.c
index 7068f82c5495a..de7743392afa1 100644
--- a/iptables/nft-arp.c
+++ b/iptables/nft-arp.c
@@ -126,7 +126,7 @@ static bool need_devaddr(struct arpt_devaddr_info *info)
 	return false;
 }
 
-static int nft_arp_add(struct nftnl_rule *r, void *data)
+static int nft_arp_add(struct nft_handle *h, struct nftnl_rule *r, void *data)
 {
 	struct iptables_command_state *cs = data;
 	struct arpt_entry *fw = &cs->arp;
diff --git a/iptables/nft-bridge.c b/iptables/nft-bridge.c
index 2e4b309b86135..0fc21b3a3c0d6 100644
--- a/iptables/nft-bridge.c
+++ b/iptables/nft-bridge.c
@@ -126,7 +126,8 @@ static int _add_action(struct nftnl_rule *r, struct iptables_command_state *cs)
 	return add_action(r, cs, false);
 }
 
-static int nft_bridge_add(struct nftnl_rule *r, void *data)
+static int nft_bridge_add(struct nft_handle *h,
+			  struct nftnl_rule *r, void *data)
 {
 	struct iptables_command_state *cs = data;
 	struct ebt_match *iter;
@@ -182,7 +183,7 @@ static int nft_bridge_add(struct nftnl_rule *r, void *data)
 
 	for (iter = cs->match_list; iter; iter = iter->next) {
 		if (iter->ismatch) {
-			if (add_match(r, iter->u.match->m))
+			if (add_match(h, r, iter->u.match->m))
 				break;
 		} else {
 			if (add_target(r, iter->u.watcher->t))
diff --git a/iptables/nft-ipv4.c b/iptables/nft-ipv4.c
index 4497eb9b9347c..57d1b3c6d2d0c 100644
--- a/iptables/nft-ipv4.c
+++ b/iptables/nft-ipv4.c
@@ -26,7 +26,7 @@
 #include "nft.h"
 #include "nft-shared.h"
 
-static int nft_ipv4_add(struct nftnl_rule *r, void *data)
+static int nft_ipv4_add(struct nft_handle *h, struct nftnl_rule *r, void *data)
 {
 	struct iptables_command_state *cs = data;
 	struct xtables_rule_match *matchp;
@@ -77,7 +77,7 @@ static int nft_ipv4_add(struct nftnl_rule *r, void *data)
 	add_compat(r, cs->fw.ip.proto, cs->fw.ip.invflags & XT_INV_PROTO);
 
 	for (matchp = cs->matches; matchp; matchp = matchp->next) {
-		ret = add_match(r, matchp->match->m);
+		ret = add_match(h, r, matchp->match->m);
 		if (ret < 0)
 			return ret;
 	}
diff --git a/iptables/nft-ipv6.c b/iptables/nft-ipv6.c
index cacb1c9e141f2..0e2c4a2946e25 100644
--- a/iptables/nft-ipv6.c
+++ b/iptables/nft-ipv6.c
@@ -25,7 +25,7 @@
 #include "nft.h"
 #include "nft-shared.h"
 
-static int nft_ipv6_add(struct nftnl_rule *r, void *data)
+static int nft_ipv6_add(struct nft_handle *h, struct nftnl_rule *r, void *data)
 {
 	struct iptables_command_state *cs = data;
 	struct xtables_rule_match *matchp;
@@ -66,7 +66,7 @@ static int nft_ipv6_add(struct nftnl_rule *r, void *data)
 	add_compat(r, cs->fw6.ipv6.proto, cs->fw6.ipv6.invflags & XT_INV_PROTO);
 
 	for (matchp = cs->matches; matchp; matchp = matchp->next) {
-		ret = add_match(r, matchp->match->m);
+		ret = add_match(h, r, matchp->match->m);
 		if (ret < 0)
 			return ret;
 	}
diff --git a/iptables/nft-shared.h b/iptables/nft-shared.h
index e236a981119ac..b4a62619c09b6 100644
--- a/iptables/nft-shared.h
+++ b/iptables/nft-shared.h
@@ -35,6 +35,7 @@
 #define FMT(tab,notab) ((format) & FMT_NOTABLE ? (notab) : (tab))
 
 struct xtables_args;
+struct nft_handle;
 struct xt_xlate;
 
 enum {
@@ -69,7 +70,7 @@ struct nft_xt_ctx {
 };
 
 struct nft_family_ops {
-	int (*add)(struct nftnl_rule *r, void *data);
+	int (*add)(struct nft_handle *h, struct nftnl_rule *r, void *data);
 	bool (*is_same)(const void *data_a,
 			const void *data_b);
 	void (*print_payload)(struct nftnl_expr *e,
@@ -163,7 +164,6 @@ void save_matches_and_target(const struct iptables_command_state *cs,
 
 struct nft_family_ops *nft_family_ops_lookup(int family);
 
-struct nft_handle;
 void nft_ipv46_parse_target(struct xtables_target *t, void *data);
 bool nft_ipv46_rule_find(struct nft_family_ops *ops, struct nftnl_rule *r,
 			 void *data);
diff --git a/iptables/nft.c b/iptables/nft.c
index 3c230c121f8b9..bb0f9dfcd558f 100644
--- a/iptables/nft.c
+++ b/iptables/nft.c
@@ -930,7 +930,8 @@ static int add_nft_limit(struct nftnl_rule *r, struct xt_entry_match *m)
 	return 0;
 }
 
-int add_match(struct nftnl_rule *r, struct xt_entry_match *m)
+int add_match(struct nft_handle *h,
+	      struct nftnl_rule *r, struct xt_entry_match *m)
 {
 	struct nftnl_expr *expr;
 	int ret;
@@ -1152,7 +1153,7 @@ nft_rule_new(struct nft_handle *h, const char *chain, const char *table,
 	nftnl_rule_set_str(r, NFTNL_RULE_TABLE, table);
 	nftnl_rule_set_str(r, NFTNL_RULE_CHAIN, chain);
 
-	if (h->ops->add(r, data) < 0)
+	if (h->ops->add(h, r, data) < 0)
 		goto err;
 
 	return r;
diff --git a/iptables/nft.h b/iptables/nft.h
index 4b8b3033a56c0..94dc72d88b7db 100644
--- a/iptables/nft.h
+++ b/iptables/nft.h
@@ -135,7 +135,7 @@ int nft_rule_zero_counters(struct nft_handle *h, const char *chain, const char *
  */
 int add_counters(struct nftnl_rule *r, uint64_t packets, uint64_t bytes);
 int add_verdict(struct nftnl_rule *r, int verdict);
-int add_match(struct nftnl_rule *r, struct xt_entry_match *m);
+int add_match(struct nft_handle *h, struct nftnl_rule *r, struct xt_entry_match *m);
 int add_target(struct nftnl_rule *r, struct xt_entry_target *t);
 int add_jumpto(struct nftnl_rule *r, const char *name, int verdict);
 int add_action(struct nftnl_rule *r, struct iptables_command_state *cs, bool goto_set);
-- 
2.23.0

[iptables PATCH v3 02/12] nft: family_ops: Pass nft_handle to 'rule_find' callback

[Phil Sutter]

In order to prepare for rules containing set references, nft handle has
to be passed to nft_rule_to_iptables_command_state() in order to let it
access the set in cache.

Signed-off-by: Phil Sutter <phil@nwl.cc>
---
 iptables/nft-arp.c    | 4 ++--
 iptables/nft-bridge.c | 4 ++--
 iptables/nft-shared.c | 7 +++----
 iptables/nft-shared.h | 4 ++--
 iptables/nft.c        | 2 +-
 5 files changed, 10 insertions(+), 11 deletions(-)

diff --git a/iptables/nft-arp.c b/iptables/nft-arp.c
index de7743392afa1..5ad7556c637b8 100644
--- a/iptables/nft-arp.c
+++ b/iptables/nft-arp.c
@@ -632,7 +632,7 @@ static bool nft_arp_is_same(const void *data_a,
 				  (unsigned char *)b->arp.outiface_mask);
 }
 
-static bool nft_arp_rule_find(struct nft_family_ops *ops, struct nftnl_rule *r,
+static bool nft_arp_rule_find(struct nft_handle *h, struct nftnl_rule *r,
 			      void *data)
 {
 	const struct iptables_command_state *cs = data;
@@ -653,7 +653,7 @@ static bool nft_arp_rule_find(struct nft_family_ops *ops, struct nftnl_rule *r,
 
 	ret = true;
 out:
-	ops->clear_cs(&this);
+	h->ops->clear_cs(&this);
 	return ret;
 }
 
diff --git a/iptables/nft-bridge.c b/iptables/nft-bridge.c
index 0fc21b3a3c0d6..73bca2f38101e 100644
--- a/iptables/nft-bridge.c
+++ b/iptables/nft-bridge.c
@@ -537,7 +537,7 @@ static bool nft_bridge_is_same(const void *data_a, const void *data_b)
 	return strcmp(a->in, b->in) == 0 && strcmp(a->out, b->out) == 0;
 }
 
-static bool nft_bridge_rule_find(struct nft_family_ops *ops, struct nftnl_rule *r,
+static bool nft_bridge_rule_find(struct nft_handle *h, struct nftnl_rule *r,
 				 void *data)
 {
 	struct iptables_command_state *cs = data;
@@ -568,7 +568,7 @@ static bool nft_bridge_rule_find(struct nft_family_ops *ops, struct nftnl_rule *
 
 	ret = true;
 out:
-	ops->clear_cs(&this);
+	h->ops->clear_cs(&this);
 	return ret;
 }
 
diff --git a/iptables/nft-shared.c b/iptables/nft-shared.c
index 6fd8ade5a1e59..b6d85f1af1da9 100644
--- a/iptables/nft-shared.c
+++ b/iptables/nft-shared.c
@@ -982,8 +982,7 @@ void nft_ipv46_parse_target(struct xtables_target *t, void *data)
 	cs->target = t;
 }
 
-bool nft_ipv46_rule_find(struct nft_family_ops *ops,
-			 struct nftnl_rule *r, void *data)
+bool nft_ipv46_rule_find(struct nft_handle *h, struct nftnl_rule *r, void *data)
 {
 	struct iptables_command_state *cs = data, this = {};
 	bool ret = false;
@@ -994,7 +993,7 @@ bool nft_ipv46_rule_find(struct nft_family_ops *ops,
 #ifdef DEBUG_DEL
 	nft_rule_print_save(r, NFT_RULE_APPEND, 0);
 #endif
-	if (!ops->is_same(cs, &this))
+	if (!h->ops->is_same(cs, &this))
 		goto out;
 
 	if (!compare_matches(cs->matches, this.matches)) {
@@ -1014,7 +1013,7 @@ bool nft_ipv46_rule_find(struct nft_family_ops *ops,
 
 	ret = true;
 out:
-	ops->clear_cs(&this);
+	h->ops->clear_cs(&this);
 	return ret;
 }
 
diff --git a/iptables/nft-shared.h b/iptables/nft-shared.h
index b4a62619c09b6..e98ce11a4b1a2 100644
--- a/iptables/nft-shared.h
+++ b/iptables/nft-shared.h
@@ -104,7 +104,7 @@ struct nft_family_ops {
 	void (*rule_to_cs)(const struct nftnl_rule *r,
 			   struct iptables_command_state *cs);
 	void (*clear_cs)(struct iptables_command_state *cs);
-	bool (*rule_find)(struct nft_family_ops *ops, struct nftnl_rule *r,
+	bool (*rule_find)(struct nft_handle *h, struct nftnl_rule *r,
 			  void *data);
 	int (*xlate)(const void *data, struct xt_xlate *xl);
 };
@@ -165,7 +165,7 @@ void save_matches_and_target(const struct iptables_command_state *cs,
 struct nft_family_ops *nft_family_ops_lookup(int family);
 
 void nft_ipv46_parse_target(struct xtables_target *t, void *data);
-bool nft_ipv46_rule_find(struct nft_family_ops *ops, struct nftnl_rule *r,
+bool nft_ipv46_rule_find(struct nft_handle *h, struct nftnl_rule *r,
 			 void *data);
 
 bool compare_matches(struct xtables_rule_match *mt1, struct xtables_rule_match *mt2);
diff --git a/iptables/nft.c b/iptables/nft.c
index bb0f9dfcd558f..e03ed59c71953 100644
--- a/iptables/nft.c
+++ b/iptables/nft.c
@@ -1907,7 +1907,7 @@ nft_rule_find(struct nft_handle *h, struct nftnl_chain *c, void *data, int rulen
 
 	r = nftnl_rule_iter_next(iter);
 	while (r != NULL) {
-		found = h->ops->rule_find(h->ops, r, data);
+		found = h->ops->rule_find(h, r, data);
 		if (found)
 			break;
 		r = nftnl_rule_iter_next(iter);
-- 
2.23.0

[iptables PATCH v3 03/12] nft: family_ops: Pass nft_handle to 'print_rule' callback

[Phil Sutter]

Prepare for 'rule_to_cs' callback to receive nft_handle pointer so it is
able to access cache for set lookups.

Signed-off-by: Phil Sutter <phil@nwl.cc>
---
 iptables/nft-arp.c    |  3 ++-
 iptables/nft-bridge.c |  4 ++--
 iptables/nft-ipv4.c   |  4 ++--
 iptables/nft-ipv6.c   |  4 ++--
 iptables/nft-shared.h |  4 ++--
 iptables/nft.c        | 19 ++++++++++---------
 6 files changed, 20 insertions(+), 18 deletions(-)

diff --git a/iptables/nft-arp.c b/iptables/nft-arp.c
index 5ad7556c637b8..da22c12d34a7b 100644
--- a/iptables/nft-arp.c
+++ b/iptables/nft-arp.c
@@ -582,7 +582,8 @@ nft_arp_save_rule(const void *data, unsigned int format)
 }
 
 static void
-nft_arp_print_rule(struct nftnl_rule *r, unsigned int num, unsigned int format)
+nft_arp_print_rule(struct nft_handle *h, struct nftnl_rule *r,
+		   unsigned int num, unsigned int format)
 {
 	struct iptables_command_state cs = {};
 
diff --git a/iptables/nft-bridge.c b/iptables/nft-bridge.c
index 73bca2f38101e..b0c6c5a4db3cd 100644
--- a/iptables/nft-bridge.c
+++ b/iptables/nft-bridge.c
@@ -472,8 +472,8 @@ static void nft_bridge_save_rule(const void *data, unsigned int format)
 		fputc('\n', stdout);
 }
 
-static void nft_bridge_print_rule(struct nftnl_rule *r, unsigned int num,
-				  unsigned int format)
+static void nft_bridge_print_rule(struct nft_handle *h, struct nftnl_rule *r,
+				  unsigned int num, unsigned int format)
 {
 	struct iptables_command_state cs = {};
 
diff --git a/iptables/nft-ipv4.c b/iptables/nft-ipv4.c
index 57d1b3c6d2d0c..98d7f966e3694 100644
--- a/iptables/nft-ipv4.c
+++ b/iptables/nft-ipv4.c
@@ -261,8 +261,8 @@ static void print_fragment(unsigned int flags, unsigned int invflags,
 	fputc(' ', stdout);
 }
 
-static void nft_ipv4_print_rule(struct nftnl_rule *r, unsigned int num,
-				unsigned int format)
+static void nft_ipv4_print_rule(struct nft_handle *h, struct nftnl_rule *r,
+				unsigned int num, unsigned int format)
 {
 	struct iptables_command_state cs = {};
 
diff --git a/iptables/nft-ipv6.c b/iptables/nft-ipv6.c
index 0e2c4a2946e25..56236bff03c2b 100644
--- a/iptables/nft-ipv6.c
+++ b/iptables/nft-ipv6.c
@@ -187,8 +187,8 @@ static void nft_ipv6_parse_immediate(const char *jumpto, bool nft_goto,
 		cs->fw6.ipv6.flags |= IP6T_F_GOTO;
 }
 
-static void nft_ipv6_print_rule(struct nftnl_rule *r, unsigned int num,
-				unsigned int format)
+static void nft_ipv6_print_rule(struct nft_handle *h, struct nftnl_rule *r,
+				unsigned int num, unsigned int format)
 {
 	struct iptables_command_state cs = {};
 
diff --git a/iptables/nft-shared.h b/iptables/nft-shared.h
index e98ce11a4b1a2..65d4997b2539d 100644
--- a/iptables/nft-shared.h
+++ b/iptables/nft-shared.h
@@ -90,8 +90,8 @@ struct nft_family_ops {
 			     const char *pol,
 			     const struct xt_counters *counters, bool basechain,
 			     uint32_t refs, uint32_t entries);
-	void (*print_rule)(struct nftnl_rule *r, unsigned int num,
-			   unsigned int format);
+	void (*print_rule)(struct nft_handle *h, struct nftnl_rule *r,
+			   unsigned int num, unsigned int format);
 	void (*save_rule)(const void *data, unsigned int format);
 	void (*save_counters)(const void *data);
 	void (*save_chain)(const struct nftnl_chain *c, const char *policy);
diff --git a/iptables/nft.c b/iptables/nft.c
index e03ed59c71953..0c6ce12f54582 100644
--- a/iptables/nft.c
+++ b/iptables/nft.c
@@ -1202,7 +1202,7 @@ nft_rule_append(struct nft_handle *h, const char *chain, const char *table,
 	}
 
 	if (verbose)
-		h->ops->print_rule(r, 0, FMT_PRINT_RULE);
+		h->ops->print_rule(h, r, 0, FMT_PRINT_RULE);
 
 	if (ref) {
 		nftnl_chain_rule_insert_at(r, ref);
@@ -1935,7 +1935,7 @@ int nft_rule_check(struct nft_handle *h, const char *chain,
 		goto fail_enoent;
 
 	if (verbose)
-		h->ops->print_rule(r, 0, FMT_PRINT_RULE);
+		h->ops->print_rule(h, r, 0, FMT_PRINT_RULE);
 
 	return 1;
 fail_enoent:
@@ -1964,7 +1964,7 @@ int nft_rule_delete(struct nft_handle *h, const char *chain,
 		if (ret < 0)
 			errno = ENOMEM;
 		if (verbose)
-			h->ops->print_rule(r, 0, FMT_PRINT_RULE);
+			h->ops->print_rule(h, r, 0, FMT_PRINT_RULE);
 	} else
 		errno = ENOENT;
 
@@ -2005,7 +2005,7 @@ nft_rule_add(struct nft_handle *h, const char *chain,
 	}
 
 	if (verbose)
-		h->ops->print_rule(r, 0, FMT_PRINT_RULE);
+		h->ops->print_rule(h, r, 0, FMT_PRINT_RULE);
 
 	return r;
 }
@@ -2114,8 +2114,8 @@ int nft_rule_replace(struct nft_handle *h, const char *chain,
 static int
 __nft_rule_list(struct nft_handle *h, struct nftnl_chain *c,
 		int rulenum, unsigned int format,
-		void (*cb)(struct nftnl_rule *r, unsigned int num,
-			   unsigned int format))
+		void (*cb)(struct nft_handle *h, struct nftnl_rule *r,
+			   unsigned int num, unsigned int format))
 {
 	struct nftnl_rule_iter *iter;
 	struct nftnl_rule *r;
@@ -2128,7 +2128,7 @@ __nft_rule_list(struct nft_handle *h, struct nftnl_chain *c,
 			 * valid chain but invalid rule number
 			 */
 			return 1;
-		cb(r, rulenum, format);
+		cb(h, r, rulenum, format);
 		return 1;
 	}
 
@@ -2138,7 +2138,7 @@ __nft_rule_list(struct nft_handle *h, struct nftnl_chain *c,
 
 	r = nftnl_rule_iter_next(iter);
 	while (r != NULL) {
-		cb(r, ++rule_ctr, format);
+		cb(h, r, ++rule_ctr, format);
 		r = nftnl_rule_iter_next(iter);
 	}
 
@@ -2242,7 +2242,8 @@ int nft_rule_list(struct nft_handle *h, const char *chain, const char *table,
 }
 
 static void
-list_save(struct nftnl_rule *r, unsigned int num, unsigned int format)
+list_save(struct nft_handle *h, struct nftnl_rule *r,
+	  unsigned int num, unsigned int format)
 {
 	nft_rule_print_save(r, NFT_RULE_APPEND, format);
 }
-- 
2.23.0

[iptables PATCH v3 04/12] nft: family_ops: Pass nft_handle to 'rule_to_cs' callback

[Phil Sutter]

This is the actual callback used to parse nftables rules. Pass
nft_handle to it so it can access the cache (and possible sets therein).

Having to pass nft_handle to nft_rule_print_save() allows to simplify it
a bit since no family ops lookup has to be done anymore.

Signed-off-by: Phil Sutter <phil@nwl.cc>
---
 iptables/nft-arp.c         |  4 ++--
 iptables/nft-bridge.c      |  9 +++++----
 iptables/nft-ipv4.c        |  2 +-
 iptables/nft-ipv6.c        |  2 +-
 iptables/nft-shared.c      |  5 +++--
 iptables/nft-shared.h      |  5 +++--
 iptables/nft.c             | 18 ++++++++----------
 iptables/nft.h             |  4 ++--
 iptables/xtables-monitor.c | 17 +++++++++++++++--
 iptables/xtables-save.c    |  3 +++
 10 files changed, 43 insertions(+), 26 deletions(-)

diff --git a/iptables/nft-arp.c b/iptables/nft-arp.c
index da22c12d34a7b..d4a86610ec217 100644
--- a/iptables/nft-arp.c
+++ b/iptables/nft-arp.c
@@ -590,7 +590,7 @@ nft_arp_print_rule(struct nft_handle *h, struct nftnl_rule *r,
 	if (format & FMT_LINENUMBERS)
 		printf("%u ", num);
 
-	nft_rule_to_iptables_command_state(r, &cs);
+	nft_rule_to_iptables_command_state(h, r, &cs);
 
 	nft_arp_print_rule_details(&cs, format);
 	print_matches_and_target(&cs, format);
@@ -641,7 +641,7 @@ static bool nft_arp_rule_find(struct nft_handle *h, struct nftnl_rule *r,
 	bool ret = false;
 
 	/* Delete by matching rule case */
-	nft_rule_to_iptables_command_state(r, &this);
+	nft_rule_to_iptables_command_state(h, r, &this);
 
 	if (!nft_arp_is_same(&cs->arp, &this.arp))
 		goto out;
diff --git a/iptables/nft-bridge.c b/iptables/nft-bridge.c
index b0c6c5a4db3cd..20ce92a6d5242 100644
--- a/iptables/nft-bridge.c
+++ b/iptables/nft-bridge.c
@@ -333,11 +333,12 @@ static void nft_bridge_parse_target(struct xtables_target *t, void *data)
 	cs->target = t;
 }
 
-static void nft_rule_to_ebtables_command_state(const struct nftnl_rule *r,
+static void nft_rule_to_ebtables_command_state(struct nft_handle *h,
+					       const struct nftnl_rule *r,
 					       struct iptables_command_state *cs)
 {
 	cs->eb.bitmask = EBT_NOPROTO;
-	nft_rule_to_iptables_command_state(r, cs);
+	nft_rule_to_iptables_command_state(h, r, cs);
 }
 
 static void print_iface(const char *option, const char *name, bool invert)
@@ -480,7 +481,7 @@ static void nft_bridge_print_rule(struct nft_handle *h, struct nftnl_rule *r,
 	if (format & FMT_LINENUMBERS)
 		printf("%d ", num);
 
-	nft_rule_to_ebtables_command_state(r, &cs);
+	nft_rule_to_ebtables_command_state(h, r, &cs);
 	nft_bridge_save_rule(&cs, format);
 	ebt_cs_clean(&cs);
 }
@@ -544,7 +545,7 @@ static bool nft_bridge_rule_find(struct nft_handle *h, struct nftnl_rule *r,
 	struct iptables_command_state this = {};
 	bool ret = false;
 
-	nft_rule_to_ebtables_command_state(r, &this);
+	nft_rule_to_ebtables_command_state(h, r, &this);
 
 	DEBUGP("comparing with... ");
 
diff --git a/iptables/nft-ipv4.c b/iptables/nft-ipv4.c
index 98d7f966e3694..70634f8fad84d 100644
--- a/iptables/nft-ipv4.c
+++ b/iptables/nft-ipv4.c
@@ -266,7 +266,7 @@ static void nft_ipv4_print_rule(struct nft_handle *h, struct nftnl_rule *r,
 {
 	struct iptables_command_state cs = {};
 
-	nft_rule_to_iptables_command_state(r, &cs);
+	nft_rule_to_iptables_command_state(h, r, &cs);
 
 	print_rule_details(&cs, cs.jumpto, cs.fw.ip.flags,
 			   cs.fw.ip.invflags, cs.fw.ip.proto, num, format);
diff --git a/iptables/nft-ipv6.c b/iptables/nft-ipv6.c
index 56236bff03c2b..d01491bfdb689 100644
--- a/iptables/nft-ipv6.c
+++ b/iptables/nft-ipv6.c
@@ -192,7 +192,7 @@ static void nft_ipv6_print_rule(struct nft_handle *h, struct nftnl_rule *r,
 {
 	struct iptables_command_state cs = {};
 
-	nft_rule_to_iptables_command_state(r, &cs);
+	nft_rule_to_iptables_command_state(h, r, &cs);
 
 	print_rule_details(&cs, cs.jumpto, cs.fw6.ipv6.flags,
 			   cs.fw6.ipv6.invflags, cs.fw6.ipv6.proto,
diff --git a/iptables/nft-shared.c b/iptables/nft-shared.c
index b6d85f1af1da9..bdbd3238b2890 100644
--- a/iptables/nft-shared.c
+++ b/iptables/nft-shared.c
@@ -591,7 +591,8 @@ static void nft_parse_limit(struct nft_xt_ctx *ctx, struct nftnl_expr *e)
 		ops->parse_match(match, ctx->cs);
 }
 
-void nft_rule_to_iptables_command_state(const struct nftnl_rule *r,
+void nft_rule_to_iptables_command_state(struct nft_handle *h,
+					const struct nftnl_rule *r,
 					struct iptables_command_state *cs)
 {
 	struct nftnl_expr_iter *iter;
@@ -987,7 +988,7 @@ bool nft_ipv46_rule_find(struct nft_handle *h, struct nftnl_rule *r, void *data)
 	struct iptables_command_state *cs = data, this = {};
 	bool ret = false;
 
-	nft_rule_to_iptables_command_state(r, &this);
+	nft_rule_to_iptables_command_state(h, r, &this);
 
 	DEBUGP("comparing with... ");
 #ifdef DEBUG_DEL
diff --git a/iptables/nft-shared.h b/iptables/nft-shared.h
index 65d4997b2539d..ee0857b6f187a 100644
--- a/iptables/nft-shared.h
+++ b/iptables/nft-shared.h
@@ -101,7 +101,7 @@ struct nft_family_ops {
 			   struct xtables_args *args);
 	void (*parse_match)(struct xtables_match *m, void *data);
 	void (*parse_target)(struct xtables_target *t, void *data);
-	void (*rule_to_cs)(const struct nftnl_rule *r,
+	void (*rule_to_cs)(struct nft_handle *h, const struct nftnl_rule *r,
 			   struct iptables_command_state *cs);
 	void (*clear_cs)(struct iptables_command_state *cs);
 	bool (*rule_find)(struct nft_handle *h, struct nftnl_rule *r,
@@ -138,7 +138,8 @@ int parse_meta(struct nftnl_expr *e, uint8_t key, char *iniface,
 		unsigned char *outiface_mask, uint8_t *invflags);
 void print_proto(uint16_t proto, int invert);
 void get_cmp_data(struct nftnl_expr *e, void *data, size_t dlen, bool *inv);
-void nft_rule_to_iptables_command_state(const struct nftnl_rule *r,
+void nft_rule_to_iptables_command_state(struct nft_handle *h,
+					const struct nftnl_rule *r,
 					struct iptables_command_state *cs);
 void nft_clear_iptables_command_state(struct iptables_command_state *cs);
 void print_header(unsigned int format, const char *chain, const char *pol,
diff --git a/iptables/nft.c b/iptables/nft.c
index 0c6ce12f54582..6a8eafd1938ca 100644
--- a/iptables/nft.c
+++ b/iptables/nft.c
@@ -351,7 +351,7 @@ static int mnl_append_error(const struct nft_handle *h,
 			 nftnl_rule_get_str(o->rule, NFTNL_RULE_CHAIN));
 #if 0
 		{
-			nft_rule_print_save(o->rule, NFT_RULE_APPEND, FMT_NOCOUNTS);
+			nft_rule_print_save(h, o->rule, NFT_RULE_APPEND, FMT_NOCOUNTS);
 		}
 #endif
 		break;
@@ -1220,16 +1220,14 @@ nft_rule_append(struct nft_handle *h, const char *chain, const char *table,
 }
 
 void
-nft_rule_print_save(const struct nftnl_rule *r, enum nft_rule_print type,
-		    unsigned int format)
+nft_rule_print_save(struct nft_handle *h, const struct nftnl_rule *r,
+		    enum nft_rule_print type, unsigned int format)
 {
 	const char *chain = nftnl_rule_get_str(r, NFTNL_RULE_CHAIN);
-	int family = nftnl_rule_get_u32(r, NFTNL_RULE_FAMILY);
 	struct iptables_command_state cs = {};
-	struct nft_family_ops *ops;
+	struct nft_family_ops *ops = h->ops;
 
-	ops = nft_family_ops_lookup(family);
-	ops->rule_to_cs(r, &cs);
+	ops->rule_to_cs(h, r, &cs);
 
 	if (!(format & (FMT_NOCOUNTS | FMT_C_COUNTS)) && ops->save_counters)
 		ops->save_counters(&cs);
@@ -1392,7 +1390,7 @@ static int nft_chain_save_rules(struct nft_handle *h,
 
 	r = nftnl_rule_iter_next(iter);
 	while (r != NULL) {
-		nft_rule_print_save(r, NFT_RULE_APPEND, format);
+		nft_rule_print_save(h, r, NFT_RULE_APPEND, format);
 		r = nftnl_rule_iter_next(iter);
 	}
 
@@ -2245,7 +2243,7 @@ static void
 list_save(struct nft_handle *h, struct nftnl_rule *r,
 	  unsigned int num, unsigned int format)
 {
-	nft_rule_print_save(r, NFT_RULE_APPEND, format);
+	nft_rule_print_save(h, r, NFT_RULE_APPEND, format);
 }
 
 static int __nftnl_rule_list_chain_save(struct nftnl_chain *c, void *data)
@@ -2357,7 +2355,7 @@ int nft_rule_zero_counters(struct nft_handle *h, const char *chain,
 		goto error;
 	}
 
-	nft_rule_to_iptables_command_state(r, &cs);
+	nft_rule_to_iptables_command_state(h, r, &cs);
 
 	cs.counters.pcnt = cs.counters.bcnt = 0;
 
diff --git a/iptables/nft.h b/iptables/nft.h
index 94dc72d88b7db..e157b525cffdc 100644
--- a/iptables/nft.h
+++ b/iptables/nft.h
@@ -146,8 +146,8 @@ enum nft_rule_print {
 	NFT_RULE_DEL,
 };
 
-void nft_rule_print_save(const struct nftnl_rule *r, enum nft_rule_print type,
-			 unsigned int format);
+void nft_rule_print_save(struct nft_handle *h, const struct nftnl_rule *r,
+			 enum nft_rule_print type, unsigned int format);
 
 uint32_t nft_invflags2cmp(uint32_t invflags, uint32_t flag);
 
diff --git a/iptables/xtables-monitor.c b/iptables/xtables-monitor.c
index eb80bac81c645..a5245d1422af9 100644
--- a/iptables/xtables-monitor.c
+++ b/iptables/xtables-monitor.c
@@ -11,6 +11,7 @@
 
 #define _GNU_SOURCE
 #include "config.h"
+#include <errno.h>
 #include <stdlib.h>
 #include <time.h>
 #include <string.h>
@@ -41,6 +42,7 @@
 struct cb_arg {
 	uint32_t nfproto;
 	bool is_event;
+	struct nft_handle *h;
 };
 
 static int table_cb(const struct nlmsghdr *nlh, void *data)
@@ -106,7 +108,7 @@ static int rule_cb(const struct nlmsghdr *nlh, void *data)
 	}
 
 	printf("-t %s ", nftnl_rule_get_str(r, NFTNL_RULE_TABLE));
-	nft_rule_print_save(r, type == NFT_MSG_NEWRULE ? NFT_RULE_APPEND :
+	nft_rule_print_save(arg->h, r, type == NFT_MSG_NEWRULE ? NFT_RULE_APPEND :
 							   NFT_RULE_DEL,
 			    counters ? 0 : FMT_NOCOUNTS);
 err_free:
@@ -593,7 +595,10 @@ int xtables_monitor_main(int argc, char *argv[])
 	struct mnl_socket *nl;
 	char buf[MNL_SOCKET_BUFFER_SIZE];
 	uint32_t nfgroup = 0;
-	struct cb_arg cb_arg = {};
+	struct nft_handle h = {};
+	struct cb_arg cb_arg = {
+		.h = &h,
+	};
 	int ret, c;
 
 	xtables_globals.program_name = "xtables-monitor";
@@ -610,6 +615,14 @@ int xtables_monitor_main(int argc, char *argv[])
 	init_extensions4();
 #endif
 
+	if (nft_init(&h, xtables_ipv4)) {
+		fprintf(stderr, "%s/%s Failed to initialize nft: %s\n",
+			xtables_globals.program_name,
+			xtables_globals.program_version,
+			strerror(errno));
+		exit(EXIT_FAILURE);
+	}
+
 	opterr = 0;
 	while ((c = getopt_long(argc, argv, "ceht46V", options, NULL)) != -1) {
 		switch (c) {
diff --git a/iptables/xtables-save.c b/iptables/xtables-save.c
index 44687f998c91a..3a52f8c3d8209 100644
--- a/iptables/xtables-save.c
+++ b/iptables/xtables-save.c
@@ -249,6 +249,9 @@ xtables_save_main(int family, int argc, char *argv[],
 				strerror(errno));
 		exit(EXIT_FAILURE);
 	}
+	h.ops = nft_family_ops_lookup(h.family);
+	if (!h.ops)
+		xtables_error(PARAMETER_PROBLEM, "Unknown family");
 
 	ret = do_output(&h, tablename, &d);
 	nft_fini(&h);
-- 
2.23.0

[iptables PATCH v3 05/12] nft: Keep nft_handle pointer in nft_xt_ctx

[Phil Sutter]

Instead of carrying the family value, carry the handle (which contains
the family value) and relieve expression parsers from having to call
nft_family_ops_lookup().

Signed-off-by: Phil Sutter <phil@nwl.cc>
---
 iptables/nft-shared.c | 40 ++++++++++++++--------------------------
 iptables/nft-shared.h |  2 +-
 2 files changed, 15 insertions(+), 27 deletions(-)

diff --git a/iptables/nft-shared.c b/iptables/nft-shared.c
index bdbd3238b2890..80d4e1fcdcea1 100644
--- a/iptables/nft-shared.c
+++ b/iptables/nft-shared.c
@@ -310,7 +310,6 @@ static void nft_parse_target(struct nft_xt_ctx *ctx, struct nftnl_expr *e)
 	struct xtables_target *target;
 	struct xt_entry_target *t;
 	size_t size;
-	struct nft_family_ops *ops;
 	void *data = ctx->cs;
 
 	target = xtables_find_target(targname, XTF_TRY_LOAD);
@@ -327,8 +326,7 @@ static void nft_parse_target(struct nft_xt_ctx *ctx, struct nftnl_expr *e)
 
 	target->t = t;
 
-	ops = nft_family_ops_lookup(ctx->family);
-	ops->parse_target(target, data);
+	ctx->h->ops->parse_target(target, data);
 }
 
 static void nft_parse_match(struct nft_xt_ctx *ctx, struct nftnl_expr *e)
@@ -339,9 +337,8 @@ static void nft_parse_match(struct nft_xt_ctx *ctx, struct nftnl_expr *e)
 	struct xtables_match *match;
 	struct xtables_rule_match **matches;
 	struct xt_entry_match *m;
-	struct nft_family_ops *ops;
 
-	switch (ctx->family) {
+	switch (ctx->h->family) {
 	case NFPROTO_IPV4:
 	case NFPROTO_IPV6:
 	case NFPROTO_BRIDGE:
@@ -349,7 +346,7 @@ static void nft_parse_match(struct nft_xt_ctx *ctx, struct nftnl_expr *e)
 		break;
 	default:
 		fprintf(stderr, "BUG: nft_parse_match() unknown family %d\n",
-			ctx->family);
+			ctx->h->family);
 		exit(EXIT_FAILURE);
 	}
 
@@ -365,9 +362,8 @@ static void nft_parse_match(struct nft_xt_ctx *ctx, struct nftnl_expr *e)
 
 	match->m = m;
 
-	ops = nft_family_ops_lookup(ctx->family);
-	if (ops->parse_match != NULL)
-		ops->parse_match(match, ctx->cs);
+	if (ctx->h->ops->parse_match != NULL)
+		ctx->h->ops->parse_match(match, ctx->cs);
 }
 
 void print_proto(uint16_t proto, int invert)
@@ -400,7 +396,6 @@ void get_cmp_data(struct nftnl_expr *e, void *data, size_t dlen, bool *inv)
 
 static void nft_meta_set_to_target(struct nft_xt_ctx *ctx)
 {
-	const struct nft_family_ops *ops;
 	struct xtables_target *target;
 	struct xt_entry_target *t;
 	unsigned int size;
@@ -429,8 +424,7 @@ static void nft_meta_set_to_target(struct nft_xt_ctx *ctx)
 
 	target->t = t;
 
-	ops = nft_family_ops_lookup(ctx->family);
-	ops->parse_target(target, ctx->cs);
+	ctx->h->ops->parse_target(target, ctx->cs);
 }
 
 static void nft_parse_meta(struct nft_xt_ctx *ctx, struct nftnl_expr *e)
@@ -474,7 +468,6 @@ static void nft_parse_bitwise(struct nft_xt_ctx *ctx, struct nftnl_expr *e)
 
 static void nft_parse_cmp(struct nft_xt_ctx *ctx, struct nftnl_expr *e)
 {
-	struct nft_family_ops *ops = nft_family_ops_lookup(ctx->family);
 	void *data = ctx->cs;
 	uint32_t reg;
 
@@ -483,12 +476,12 @@ static void nft_parse_cmp(struct nft_xt_ctx *ctx, struct nftnl_expr *e)
 		return;
 
 	if (ctx->flags & NFT_XT_CTX_META) {
-		ops->parse_meta(ctx, e, data);
+		ctx->h->ops->parse_meta(ctx, e, data);
 		ctx->flags &= ~NFT_XT_CTX_META;
 	}
 	/* bitwise context is interpreted from payload */
 	if (ctx->flags & NFT_XT_CTX_PAYLOAD) {
-		ops->parse_payload(ctx, e, data);
+		ctx->h->ops->parse_payload(ctx, e, data);
 		ctx->flags &= ~NFT_XT_CTX_PAYLOAD;
 	}
 }
@@ -502,7 +495,6 @@ static void nft_parse_counter(struct nftnl_expr *e, struct xt_counters *counters
 static void nft_parse_immediate(struct nft_xt_ctx *ctx, struct nftnl_expr *e)
 {
 	const char *chain = nftnl_expr_get_str(e, NFTNL_EXPR_IMM_CHAIN);
-	struct nft_family_ops *ops;
 	const char *jumpto = NULL;
 	bool nft_goto = false;
 	void *data = ctx->cs;
@@ -544,8 +536,7 @@ static void nft_parse_immediate(struct nft_xt_ctx *ctx, struct nftnl_expr *e)
 		break;
 	}
 
-	ops = nft_family_ops_lookup(ctx->family);
-	ops->parse_immediate(jumpto, nft_goto, data);
+	ctx->h->ops->parse_immediate(jumpto, nft_goto, data);
 }
 
 static void nft_parse_limit(struct nft_xt_ctx *ctx, struct nftnl_expr *e)
@@ -555,11 +546,10 @@ static void nft_parse_limit(struct nft_xt_ctx *ctx, struct nftnl_expr *e)
 	__u64 rate = nftnl_expr_get_u64(e, NFTNL_EXPR_LIMIT_RATE);
 	struct xtables_rule_match **matches;
 	struct xtables_match *match;
-	struct nft_family_ops *ops;
 	struct xt_rateinfo *rinfo;
 	size_t size;
 
-	switch (ctx->family) {
+	switch (ctx->h->family) {
 	case NFPROTO_IPV4:
 	case NFPROTO_IPV6:
 	case NFPROTO_BRIDGE:
@@ -567,7 +557,7 @@ static void nft_parse_limit(struct nft_xt_ctx *ctx, struct nftnl_expr *e)
 		break;
 	default:
 		fprintf(stderr, "BUG: nft_parse_limit() unknown family %d\n",
-			ctx->family);
+			ctx->h->family);
 		exit(EXIT_FAILURE);
 	}
 
@@ -586,9 +576,8 @@ static void nft_parse_limit(struct nft_xt_ctx *ctx, struct nftnl_expr *e)
 	rinfo->avg = XT_LIMIT_SCALE * unit / rate;
 	rinfo->burst = burst;
 
-	ops = nft_family_ops_lookup(ctx->family);
-	if (ops->parse_match != NULL)
-		ops->parse_match(match, ctx->cs);
+	if (ctx->h->ops->parse_match != NULL)
+		ctx->h->ops->parse_match(match, ctx->cs);
 }
 
 void nft_rule_to_iptables_command_state(struct nft_handle *h,
@@ -597,10 +586,9 @@ void nft_rule_to_iptables_command_state(struct nft_handle *h,
 {
 	struct nftnl_expr_iter *iter;
 	struct nftnl_expr *expr;
-	int family = nftnl_rule_get_u32(r, NFTNL_RULE_FAMILY);
 	struct nft_xt_ctx ctx = {
 		.cs = cs,
-		.family = family,
+		.h = h,
 	};
 
 	iter = nftnl_expr_iter_create(r);
diff --git a/iptables/nft-shared.h b/iptables/nft-shared.h
index ee0857b6f187a..d105a1eb1266b 100644
--- a/iptables/nft-shared.h
+++ b/iptables/nft-shared.h
@@ -48,7 +48,7 @@ enum {
 struct nft_xt_ctx {
 	struct iptables_command_state *cs;
 	struct nftnl_expr_iter *iter;
-	int family;
+	struct nft_handle *h;
 	uint32_t flags;
 
 	uint32_t reg;
-- 
2.23.0

[iptables PATCH v3 06/12] nft: Eliminate pointless calls to nft_family_ops_lookup()

[Phil Sutter]

If nft_handle is available, use its 'ops' field instead of performing a
new lookup. For the same reason, there is no need to pass ops pointer to
__nft_print_header().

Signed-off-by: Phil Sutter <phil@nwl.cc>
---
 iptables/nft.c | 15 +++++----------
 1 file changed, 5 insertions(+), 10 deletions(-)

diff --git a/iptables/nft.c b/iptables/nft.c
index 6a8eafd1938ca..547bc5f0aaad0 100644
--- a/iptables/nft.c
+++ b/iptables/nft.c
@@ -1336,12 +1336,10 @@ static const char *policy_name[NF_ACCEPT+1] = {
 
 int nft_chain_save(struct nft_handle *h, struct nftnl_chain_list *list)
 {
+	struct nft_family_ops *ops = h->ops;
 	struct nftnl_chain_list_iter *iter;
-	struct nft_family_ops *ops;
 	struct nftnl_chain *c;
 
-	ops = nft_family_ops_lookup(h->family);
-
 	iter = nftnl_chain_list_iter_create(list);
 	if (iter == NULL)
 		return 0;
@@ -2165,7 +2163,6 @@ static int nft_rule_count(struct nft_handle *h, struct nftnl_chain *c)
 }
 
 static void __nft_print_header(struct nft_handle *h,
-			       const struct nft_family_ops *ops,
 			       struct nftnl_chain *c, unsigned int format)
 {
 	const char *chain_name = nftnl_chain_get_str(c, NFTNL_CHAIN_NAME);
@@ -2181,14 +2178,14 @@ static void __nft_print_header(struct nft_handle *h,
 	if (nftnl_chain_is_set(c, NFTNL_CHAIN_POLICY))
 		pname = policy_name[nftnl_chain_get_u32(c, NFTNL_CHAIN_POLICY)];
 
-	ops->print_header(format, chain_name, pname,
+	h->ops->print_header(format, chain_name, pname,
 			&ctrs, basechain, refs - entries, entries);
 }
 
 int nft_rule_list(struct nft_handle *h, const char *chain, const char *table,
 		  int rulenum, unsigned int format)
 {
-	const struct nft_family_ops *ops;
+	const struct nft_family_ops *ops = h->ops;
 	struct nftnl_chain_list *list;
 	struct nftnl_chain_list_iter *iter;
 	struct nftnl_chain *c;
@@ -2197,8 +2194,6 @@ int nft_rule_list(struct nft_handle *h, const char *chain, const char *table,
 	nft_xt_builtin_init(h, table);
 	nft_assert_table_compatible(h, table, chain);
 
-	ops = nft_family_ops_lookup(h->family);
-
 	list = nft_chain_list_get(h, table, chain);
 	if (!list)
 		return 0;
@@ -2211,7 +2206,7 @@ int nft_rule_list(struct nft_handle *h, const char *chain, const char *table,
 		if (!rulenum) {
 			if (ops->print_table_header)
 				ops->print_table_header(table);
-			__nft_print_header(h, ops, c, format);
+			__nft_print_header(h, c, format);
 		}
 		__nft_rule_list(h, c, rulenum, format, ops->print_rule);
 		return 1;
@@ -2229,7 +2224,7 @@ int nft_rule_list(struct nft_handle *h, const char *chain, const char *table,
 		if (found)
 			printf("\n");
 
-		__nft_print_header(h, ops, c, format);
+		__nft_print_header(h, c, format);
 		__nft_rule_list(h, c, rulenum, format, ops->print_rule);
 
 		found = true;
-- 
2.23.0

[iptables PATCH v3 07/12] nft: Introduce NFT_CL_SETS cache level

[Phil Sutter]

In order to support anonymous sets, introduce an intermediate cache
level between NFT_CL_CHAINS and NFT_CL_RULES. Actually chains are not
needed to fetch sets, but given that sets are only needed for rules, put
it late to not slow down fetching chains.

Signed-off-by: Phil Sutter <phil@nwl.cc>
---
Changes since v2:
- Introduce NFT_CL_SETS and integrate fetch_set_cache() into
  __nft_build_cache() instead of calling from fetch_rule_cache().
- Extend __nft_build_cache() parameters to optionally take a set name to
  fetch.
- Don't fetch tables from fetch_set_cache(), cache leveling takes care
  of that.

Changes since v1:
- Fix cache flushing: Since set fetching happens only if rules are
  fetched, too, set list may be NULL so call nftnl_set_list_free() and
  nftnl_set_list_foreach() conditionally.
- Support fetching only a specific set from kernel. Make sure no
  duplicate element fetches happen by checking set element count. (If
  non-zero, don't fetch elements.)
- When fetching rules for a single chain, fetch only that table's sets.
---
 iptables/nft-cache.c | 205 +++++++++++++++++++++++++++++++++++++++++--
 iptables/nft-cache.h |   2 +
 iptables/nft.h       |   2 +
 3 files changed, 200 insertions(+), 9 deletions(-)

diff --git a/iptables/nft-cache.c b/iptables/nft-cache.c
index c55970d074a76..34c36b9031529 100644
--- a/iptables/nft-cache.c
+++ b/iptables/nft-cache.c
@@ -18,6 +18,7 @@
 
 #include <libmnl/libmnl.h>
 #include <libnftnl/gen.h>
+#include <libnftnl/set.h>
 #include <libnftnl/table.h>
 
 #include "nft.h"
@@ -152,6 +153,157 @@ err:
 	return MNL_CB_OK;
 }
 
+struct nftnl_set_list_cb_data {
+	struct nft_handle *h;
+	const struct builtin_table *t;
+};
+
+static int nftnl_set_list_cb(const struct nlmsghdr *nlh, void *data)
+{
+	struct nftnl_set_list_cb_data *d = data;
+	const struct builtin_table *t = d->t;
+	struct nftnl_set_list *list;
+	struct nft_handle *h = d->h;
+	const char *tname, *sname;
+	struct nftnl_set *s;
+
+	s = nftnl_set_alloc();
+	if (s == NULL)
+		return MNL_CB_OK;
+
+	if (nftnl_set_nlmsg_parse(nlh, s) < 0)
+		goto out_free;
+
+	tname = nftnl_set_get_str(s, NFTNL_SET_TABLE);
+
+	if (!t)
+		t = nft_table_builtin_find(h, tname);
+	else if (strcmp(t->name, tname))
+		goto out_free;
+
+	if (!t)
+		goto out_free;
+
+	list = h->cache->table[t->type].sets;
+	sname = nftnl_set_get_str(s, NFTNL_SET_NAME);
+
+	if (nftnl_set_list_lookup_byname(list, sname))
+		goto out_free;
+
+	nftnl_set_list_add_tail(s, list);
+
+	return MNL_CB_OK;
+out_free:
+	nftnl_set_free(s);
+	return MNL_CB_OK;
+}
+
+static int set_elem_cb(const struct nlmsghdr *nlh, void *data)
+{
+	return nftnl_set_elems_nlmsg_parse(nlh, data) ? -1 : MNL_CB_OK;
+}
+
+static bool set_has_elements(struct nftnl_set *s)
+{
+	struct nftnl_set_elems_iter *iter;
+	bool ret = false;
+
+	iter = nftnl_set_elems_iter_create(s);
+	if (iter) {
+		ret = !!nftnl_set_elems_iter_cur(iter);
+		nftnl_set_elems_iter_destroy(iter);
+	}
+	return ret;
+}
+
+static int set_fetch_elem_cb(struct nftnl_set *s, void *data)
+{
+        char buf[MNL_SOCKET_BUFFER_SIZE];
+	struct nft_handle *h = data;
+        struct nlmsghdr *nlh;
+
+	if (set_has_elements(s))
+		return 0;
+
+	nlh = nftnl_nlmsg_build_hdr(buf, NFT_MSG_GETSETELEM, h->family,
+				    NLM_F_DUMP, h->seq);
+        nftnl_set_elems_nlmsg_build_payload(nlh, s);
+
+	return mnl_talk(h, nlh, set_elem_cb, s);
+}
+
+static int fetch_set_cache(struct nft_handle *h,
+			   const struct builtin_table *t, const char *set)
+{
+	struct nftnl_set_list_cb_data d = {
+		.h = h,
+		.t = t,
+	};
+	struct nlmsghdr *nlh;
+	char buf[16536];
+	int i, ret;
+
+	if (!t) {
+		for (i = 0; i < NFT_TABLE_MAX; i++) {
+			enum nft_table_type type = h->tables[i].type;
+
+			if (!h->tables[i].name)
+				continue;
+
+			h->cache->table[type].sets = nftnl_set_list_alloc();
+			if (!h->cache->table[type].sets)
+				return -1;
+		}
+	} else if (!h->cache->table[t->type].sets) {
+		h->cache->table[t->type].sets = nftnl_set_list_alloc();
+	}
+
+	if (t && set) {
+		struct nftnl_set *s = nftnl_set_alloc();
+
+		if (!s)
+			return -1;
+
+		nlh = nftnl_set_nlmsg_build_hdr(buf, NFT_MSG_GETSET, h->family,
+						NLM_F_ACK, h->seq);
+		nftnl_set_set_str(s, NFTNL_SET_TABLE, t->name);
+		nftnl_set_set_str(s, NFTNL_SET_NAME, set);
+		nftnl_set_nlmsg_build_payload(nlh, s);
+		nftnl_set_free(s);
+	} else {
+		nlh = nftnl_set_nlmsg_build_hdr(buf, NFT_MSG_GETSET, h->family,
+						NLM_F_DUMP, h->seq);
+	}
+
+	ret = mnl_talk(h, nlh, nftnl_set_list_cb, &d);
+	if (ret < 0 && errno == EINTR) {
+		assert(nft_restart(h) >= 0);
+		return ret;
+	}
+
+	if (t && set) {
+		struct nftnl_set *s;
+
+		s = nftnl_set_list_lookup_byname(h->cache->table[t->type].sets,
+						 set);
+		set_fetch_elem_cb(s, h);
+	} else if (t) {
+		nftnl_set_list_foreach(h->cache->table[t->type].sets,
+				       set_fetch_elem_cb, h);
+	} else {
+		for (i = 0; i < NFT_TABLE_MAX; i++) {
+			enum nft_table_type type = h->tables[i].type;
+
+			if (!h->tables[i].name)
+				continue;
+
+			nftnl_set_list_foreach(h->cache->table[type].sets,
+					       set_fetch_elem_cb, h);
+		}
+	}
+	return ret;
+}
+
 static int fetch_chain_cache(struct nft_handle *h,
 			     const struct builtin_table *t,
 			     const char *chain)
@@ -297,7 +449,8 @@ static int fetch_rule_cache(struct nft_handle *h,
 
 static void
 __nft_build_cache(struct nft_handle *h, enum nft_cache_level level,
-		  const struct builtin_table *t, const char *chain)
+		  const struct builtin_table *t, const char *set,
+		  const char *chain)
 {
 	uint32_t genid_start, genid_stop;
 
@@ -321,6 +474,11 @@ retry:
 			break;
 		/* fall through */
 	case NFT_CL_CHAINS:
+		fetch_set_cache(h, t, set);
+		if (level == NFT_CL_SETS)
+			break;
+		/* fall through */
+	case NFT_CL_SETS:
 		fetch_rule_cache(h, t, chain);
 		if (level == NFT_CL_RULES)
 			break;
@@ -349,12 +507,12 @@ void nft_build_cache(struct nft_handle *h, struct nftnl_chain *c)
 	const char *table, *chain;
 
 	if (!c)
-		return __nft_build_cache(h, NFT_CL_RULES, NULL, NULL);
+		return __nft_build_cache(h, NFT_CL_RULES, NULL, NULL, NULL);
 
 	table = nftnl_chain_get_str(c, NFTNL_CHAIN_TABLE);
 	chain = nftnl_chain_get_str(c, NFTNL_CHAIN_NAME);
 	t = nft_table_builtin_find(h, table);
-	__nft_build_cache(h, NFT_CL_RULES, t, chain);
+	__nft_build_cache(h, NFT_CL_RULES, t, NULL, chain);
 }
 
 void nft_fake_cache(struct nft_handle *h)
@@ -421,6 +579,14 @@ static int __flush_chain_cache(struct nftnl_chain *c, void *data)
 	return 0;
 }
 
+static int __flush_set_cache(struct nftnl_set *s, void *data)
+{
+	nftnl_set_list_del(s);
+	nftnl_set_free(s);
+
+	return 0;
+}
+
 static int flush_cache(struct nft_handle *h, struct nft_cache *c,
 		       const char *tablename)
 {
@@ -429,10 +595,14 @@ static int flush_cache(struct nft_handle *h, struct nft_cache *c,
 
 	if (tablename) {
 		table = nft_table_builtin_find(h, tablename);
-		if (!table || !c->table[table->type].chains)
+		if (!table)
 			return 0;
-		nftnl_chain_list_foreach(c->table[table->type].chains,
-					 __flush_chain_cache, NULL);
+		if (c->table[table->type].chains)
+			nftnl_chain_list_foreach(c->table[table->type].chains,
+						 __flush_chain_cache, NULL);
+		if (c->table[table->type].sets)
+			nftnl_set_list_foreach(c->table[table->type].sets,
+					       __flush_set_cache, NULL);
 		return 0;
 	}
 
@@ -445,6 +615,9 @@ static int flush_cache(struct nft_handle *h, struct nft_cache *c,
 
 		nftnl_chain_list_free(c->table[i].chains);
 		c->table[i].chains = NULL;
+		if (c->table[i].sets)
+			nftnl_set_list_free(c->table[i].sets);
+		c->table[i].sets = NULL;
 	}
 	nftnl_table_list_free(c->tables);
 	c->tables = NULL;
@@ -469,7 +642,7 @@ void nft_rebuild_cache(struct nft_handle *h)
 		__nft_flush_cache(h);
 
 	h->cache_level = NFT_CL_NONE;
-	__nft_build_cache(h, level, NULL, NULL);
+	__nft_build_cache(h, level, NULL, NULL, NULL);
 }
 
 void nft_release_cache(struct nft_handle *h)
@@ -480,11 +653,25 @@ void nft_release_cache(struct nft_handle *h)
 
 struct nftnl_table_list *nftnl_table_list_get(struct nft_handle *h)
 {
-	__nft_build_cache(h, NFT_CL_TABLES, NULL, NULL);
+	__nft_build_cache(h, NFT_CL_TABLES, NULL, NULL, NULL);
 
 	return h->cache->tables;
 }
 
+struct nftnl_set_list *
+nft_set_list_get(struct nft_handle *h, const char *table, const char *set)
+{
+	const struct builtin_table *t;
+
+	t = nft_table_builtin_find(h, table);
+	if (!t)
+		return NULL;
+
+	__nft_build_cache(h, NFT_CL_RULES, t, set, NULL);
+
+	return h->cache->table[t->type].sets;
+}
+
 struct nftnl_chain_list *
 nft_chain_list_get(struct nft_handle *h, const char *table, const char *chain)
 {
@@ -494,7 +681,7 @@ nft_chain_list_get(struct nft_handle *h, const char *table, const char *chain)
 	if (!t)
 		return NULL;
 
-	__nft_build_cache(h, NFT_CL_CHAINS, t, chain);
+	__nft_build_cache(h, NFT_CL_CHAINS, t, NULL, chain);
 
 	return h->cache->table[t->type].chains;
 }
diff --git a/iptables/nft-cache.h b/iptables/nft-cache.h
index cb7a7688be37a..ed498835676e2 100644
--- a/iptables/nft-cache.h
+++ b/iptables/nft-cache.h
@@ -13,6 +13,8 @@ int flush_rule_cache(struct nft_handle *h, const char *table,
 
 struct nftnl_chain_list *
 nft_chain_list_get(struct nft_handle *h, const char *table, const char *chain);
+struct nftnl_set_list *
+nft_set_list_get(struct nft_handle *h, const char *table, const char *set);
 struct nftnl_table_list *nftnl_table_list_get(struct nft_handle *h);
 
 #endif /* _NFT_CACHE_H_ */
diff --git a/iptables/nft.h b/iptables/nft.h
index e157b525cffdc..51b5660314c0c 100644
--- a/iptables/nft.h
+++ b/iptables/nft.h
@@ -31,6 +31,7 @@ enum nft_cache_level {
 	NFT_CL_NONE,
 	NFT_CL_TABLES,
 	NFT_CL_CHAINS,
+	NFT_CL_SETS,
 	NFT_CL_RULES
 };
 
@@ -38,6 +39,7 @@ struct nft_cache {
 	struct nftnl_table_list		*tables;
 	struct {
 		struct nftnl_chain_list *chains;
+		struct nftnl_set_list	*sets;
 		bool			initialized;
 	} table[NFT_TABLE_MAX];
 };
-- 
2.23.0

[iptables PATCH v3 08/12] nft: Support NFT_COMPAT_SET_ADD

[Phil Sutter]

Implement the required infrastructure to create sets as part of a batch
job commit.

Signed-off-by: Phil Sutter <phil@nwl.cc>
---
 iptables/nft.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)

diff --git a/iptables/nft.c b/iptables/nft.c
index 547bc5f0aaad0..fbb3e2f2a0f4a 100644
--- a/iptables/nft.c
+++ b/iptables/nft.c
@@ -271,6 +271,7 @@ enum obj_update_type {
 	NFT_COMPAT_RULE_REPLACE,
 	NFT_COMPAT_RULE_DELETE,
 	NFT_COMPAT_RULE_FLUSH,
+	NFT_COMPAT_SET_ADD,
 };
 
 enum obj_action {
@@ -288,6 +289,7 @@ struct obj_update {
 		struct nftnl_table	*table;
 		struct nftnl_chain	*chain;
 		struct nftnl_rule	*rule;
+		struct nftnl_set	*set;
 		void			*ptr;
 	};
 	struct {
@@ -315,6 +317,7 @@ static int mnl_append_error(const struct nft_handle *h,
 		[NFT_COMPAT_RULE_REPLACE] = "RULE_REPLACE",
 		[NFT_COMPAT_RULE_DELETE] = "RULE_DELETE",
 		[NFT_COMPAT_RULE_FLUSH] = "RULE_FLUSH",
+		[NFT_COMPAT_SET_ADD] = "SET_ADD",
 	};
 	char errmsg[256];
 	char tcr[128];
@@ -355,6 +358,10 @@ static int mnl_append_error(const struct nft_handle *h,
 		}
 #endif
 		break;
+	case NFT_COMPAT_SET_ADD:
+		snprintf(tcr, sizeof(tcr), "set %s",
+			 nftnl_set_get_str(o->set, NFTNL_SET_NAME));
+		break;
 	}
 
 	return snprintf(buf, len, "%s: %s", errmsg, tcr);
@@ -384,6 +391,13 @@ batch_table_add(struct nft_handle *h, enum obj_update_type type,
 	return batch_add(h, type, t);
 }
 
+static struct obj_update *
+batch_set_add(struct nft_handle *h, enum obj_update_type type,
+	      struct nftnl_set *s)
+{
+	return batch_add(h, type, s);
+}
+
 static int batch_chain_add(struct nft_handle *h, enum obj_update_type type,
 			   struct nftnl_chain *c)
 {
@@ -2371,6 +2385,39 @@ static void nft_compat_table_batch_add(struct nft_handle *h, uint16_t type,
 	nftnl_table_nlmsg_build_payload(nlh, table);
 }
 
+static void nft_compat_set_batch_add(struct nft_handle *h, uint16_t type,
+				     uint16_t flags, uint32_t seq,
+				     struct nftnl_set *set)
+{
+	struct nlmsghdr *nlh;
+
+	nlh = nftnl_nlmsg_build_hdr(nftnl_batch_buffer(h->batch),
+					type, h->family, flags, seq);
+	nftnl_set_nlmsg_build_payload(nlh, set);
+}
+
+static void nft_compat_setelem_batch_add(struct nft_handle *h, uint16_t type,
+					 uint16_t flags, uint32_t *seq,
+					 struct nftnl_set *set)
+{
+	struct nftnl_set_elems_iter *iter;
+	struct nlmsghdr *nlh;
+
+	iter = nftnl_set_elems_iter_create(set);
+	if (!iter)
+		return;
+
+	while (nftnl_set_elems_iter_cur(iter)) {
+		(*seq)++;
+		mnl_nft_batch_continue(h->batch);
+		nlh = nftnl_nlmsg_build_hdr(nftnl_batch_buffer(h->batch),
+					    type, h->family, flags, *seq);
+		if (nftnl_set_elems_nlmsg_build_payload_iter(nlh, iter) <= 0)
+			break;
+	}
+	nftnl_set_elems_iter_destroy(iter);
+}
+
 static void nft_compat_chain_batch_add(struct nft_handle *h, uint16_t type,
 				       uint16_t flags, uint32_t seq,
 				       struct nftnl_chain *chain)
@@ -2420,6 +2467,9 @@ static void batch_obj_del(struct nft_handle *h, struct obj_update *o)
 	case NFT_COMPAT_RULE_FLUSH:
 		nftnl_rule_free(o->rule);
 		break;
+	case NFT_COMPAT_SET_ADD:
+		nftnl_set_free(o->set);
+		break;
 	}
 	h->obj_list_num--;
 	list_del(&o->head);
@@ -2486,6 +2536,7 @@ static void nft_refresh_transaction(struct nft_handle *h)
 		case NFT_COMPAT_RULE_REPLACE:
 		case NFT_COMPAT_RULE_DELETE:
 		case NFT_COMPAT_RULE_FLUSH:
+		case NFT_COMPAT_SET_ADD:
 			break;
 		}
 	}
@@ -2576,6 +2627,13 @@ retry:
 			nft_compat_rule_batch_add(h, NFT_MSG_DELRULE, 0,
 						  n->seq, n->rule);
 			break;
+		case NFT_COMPAT_SET_ADD:
+			nft_compat_set_batch_add(h, NFT_MSG_NEWSET,
+						 NLM_F_CREATE, n->seq, n->set);
+			nft_compat_setelem_batch_add(h, NFT_MSG_NEWSETELEM,
+						     NLM_F_CREATE, &n->seq, n->set);
+			seq = n->seq;
+			break;
 		}
 
 		mnl_nft_batch_continue(h->batch);
-- 
2.23.0

[iptables PATCH v3 09/12] nft: Bore up nft_parse_payload()

[Phil Sutter]

Allow for closer inspection by storing payload expression's base and
length values. Also facilitate for two consecutive payload expressions
as LHS of a (cmp/lookup) statement as used with concatenations.

Signed-off-by: Phil Sutter <phil@nwl.cc>
---
 iptables/nft-shared.c | 8 ++++++++
 iptables/nft-shared.h | 4 +++-
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/iptables/nft-shared.c b/iptables/nft-shared.c
index 80d4e1fcdcea1..a67302ee621ae 100644
--- a/iptables/nft-shared.c
+++ b/iptables/nft-shared.c
@@ -445,8 +445,16 @@ static void nft_parse_meta(struct nft_xt_ctx *ctx, struct nftnl_expr *e)
 
 static void nft_parse_payload(struct nft_xt_ctx *ctx, struct nftnl_expr *e)
 {
+	if (ctx->flags & NFT_XT_CTX_PAYLOAD) {
+		memcpy(&ctx->prev_payload, &ctx->payload,
+		       sizeof(ctx->prev_payload));
+		ctx->flags |= NFT_XT_CTX_PREV_PAYLOAD;
+	}
+
 	ctx->reg = nftnl_expr_get_u32(e, NFTNL_EXPR_META_DREG);
+	ctx->payload.base = nftnl_expr_get_u32(e, NFTNL_EXPR_PAYLOAD_BASE);
 	ctx->payload.offset = nftnl_expr_get_u32(e, NFTNL_EXPR_PAYLOAD_OFFSET);
+	ctx->payload.len = nftnl_expr_get_u32(e, NFTNL_EXPR_PAYLOAD_LEN);
 	ctx->flags |= NFT_XT_CTX_PAYLOAD;
 }
 
diff --git a/iptables/nft-shared.h b/iptables/nft-shared.h
index d105a1eb1266b..6b2ff630b2257 100644
--- a/iptables/nft-shared.h
+++ b/iptables/nft-shared.h
@@ -43,6 +43,7 @@ enum {
 	NFT_XT_CTX_META		= (1 << 1),
 	NFT_XT_CTX_BITWISE	= (1 << 2),
 	NFT_XT_CTX_IMMEDIATE	= (1 << 3),
+	NFT_XT_CTX_PREV_PAYLOAD	= (1 << 4),
 };
 
 struct nft_xt_ctx {
@@ -53,9 +54,10 @@ struct nft_xt_ctx {
 
 	uint32_t reg;
 	struct {
+		uint32_t base;
 		uint32_t offset;
 		uint32_t len;
-	} payload;
+	} payload, prev_payload;
 	struct {
 		uint32_t key;
 	} meta;
-- 
2.23.0

[iptables PATCH v3 10/12] nft: Embed rule's table name in nft_xt_ctx

[Phil Sutter]

Down to the point where expression parsing happens, the rule's table is
not known anymore but relevant if set lookups are required.

Signed-off-by: Phil Sutter <phil@nwl.cc>
---
 iptables/nft-shared.c | 1 +
 iptables/nft-shared.h | 1 +
 2 files changed, 2 insertions(+)

diff --git a/iptables/nft-shared.c b/iptables/nft-shared.c
index a67302ee621ae..19630c1e2990c 100644
--- a/iptables/nft-shared.c
+++ b/iptables/nft-shared.c
@@ -597,6 +597,7 @@ void nft_rule_to_iptables_command_state(struct nft_handle *h,
 	struct nft_xt_ctx ctx = {
 		.cs = cs,
 		.h = h,
+		.table = nftnl_rule_get_str(r, NFTNL_RULE_TABLE),
 	};
 
 	iter = nftnl_expr_iter_create(r);
diff --git a/iptables/nft-shared.h b/iptables/nft-shared.h
index 6b2ff630b2257..0519749f43cd8 100644
--- a/iptables/nft-shared.h
+++ b/iptables/nft-shared.h
@@ -51,6 +51,7 @@ struct nft_xt_ctx {
 	struct nftnl_expr_iter *iter;
 	struct nft_handle *h;
 	uint32_t flags;
+	const char *table;
 
 	uint32_t reg;
 	struct {
-- 
2.23.0

[iptables PATCH v3 11/12] nft: Support parsing lookup expression

[Phil Sutter]

Add required glue code to support family specific lookup expression
parsers implemented as family_ops callback.

Signed-off-by: Phil Sutter <phil@nwl.cc>
---
Changes since v2:
- Remove call to add_nft_among() which is implemented in next patch to
  not temporarily break builds.
---
 iptables/nft-shared.c | 9 +++++++++
 iptables/nft-shared.h | 2 ++
 iptables/nft.c        | 3 ++-
 3 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/iptables/nft-shared.c b/iptables/nft-shared.c
index 19630c1e2990c..78e422781723f 100644
--- a/iptables/nft-shared.c
+++ b/iptables/nft-shared.c
@@ -588,6 +588,13 @@ static void nft_parse_limit(struct nft_xt_ctx *ctx, struct nftnl_expr *e)
 		ctx->h->ops->parse_match(match, ctx->cs);
 }
 
+static void nft_parse_lookup(struct nft_xt_ctx *ctx, struct nft_handle *h,
+			     struct nftnl_expr *e)
+{
+	if (ctx->h->ops->parse_lookup)
+		ctx->h->ops->parse_lookup(ctx, e, NULL);
+}
+
 void nft_rule_to_iptables_command_state(struct nft_handle *h,
 					const struct nftnl_rule *r,
 					struct iptables_command_state *cs)
@@ -628,6 +635,8 @@ void nft_rule_to_iptables_command_state(struct nft_handle *h,
 			nft_parse_target(&ctx, expr);
 		else if (strcmp(name, "limit") == 0)
 			nft_parse_limit(&ctx, expr);
+		else if (strcmp(name, "lookup") == 0)
+			nft_parse_lookup(&ctx, h, expr);
 
 		expr = nftnl_expr_iter_next(iter);
 	}
diff --git a/iptables/nft-shared.h b/iptables/nft-shared.h
index 0519749f43cd8..b83d148a14dc2 100644
--- a/iptables/nft-shared.h
+++ b/iptables/nft-shared.h
@@ -86,6 +86,8 @@ struct nft_family_ops {
 			      void *data);
 	void (*parse_cmp)(struct nft_xt_ctx *ctx, struct nftnl_expr *e,
 			  void *data);
+	void (*parse_lookup)(struct nft_xt_ctx *ctx, struct nftnl_expr *e,
+			     void *data);
 	void (*parse_immediate)(const char *jumpto, bool nft_goto, void *data);
 
 	void (*print_table_header)(const char *tablename);
diff --git a/iptables/nft.c b/iptables/nft.c
index fbb3e2f2a0f4a..111d4982b1181 100644
--- a/iptables/nft.c
+++ b/iptables/nft.c
@@ -3073,7 +3073,8 @@ static const char *supported_exprs[] = {
 	"cmp",
 	"bitwise",
 	"counter",
-	"immediate"
+	"immediate",
+	"lookup",
 };
 
 
-- 
2.23.0

[iptables PATCH v3 12/12] nft: bridge: Rudimental among extension support

[Phil Sutter]

Support among match as far as possible given the limitations of nftables
sets, namely limited to homogeneous MAC address only or MAC and IP
address only matches.

Signed-off-by: Phil Sutter <phil@nwl.cc>
---
Changes since v2:
- Move call to add_nft_among() here from previous patch.
- Large code review, simplify parsing code and share common bits between
  extension and nft-bridge.c.

Changes since v1:
- Fix for overlong lines.
- Use nftnl_set_list_lookup_byname() from libnftnl.
---
 extensions/libebt_among.c | 243 ++++++++++++++++++++++++++++++++++++++
 extensions/libebt_among.t |  16 +++
 iptables/ebtables-nft.8   |  66 ++++++-----
 iptables/nft-bridge.c     | 210 ++++++++++++++++++++++++++++++++
 iptables/nft-bridge.h     |  56 +++++++++
 iptables/nft.c            | 149 +++++++++++++++++++++++
 iptables/xtables-eb.c     |   1 +
 7 files changed, 710 insertions(+), 31 deletions(-)
 create mode 100644 extensions/libebt_among.c
 create mode 100644 extensions/libebt_among.t

diff --git a/extensions/libebt_among.c b/extensions/libebt_among.c
new file mode 100644
index 0000000000000..2e87db3bc06fa
--- /dev/null
+++ b/extensions/libebt_among.c
@@ -0,0 +1,243 @@
+/* ebt_among
+ *
+ * Authors:
+ * Grzegorz Borowiak <grzes@gnu.univ.gda.pl>
+ *
+ * August, 2003
+ */
+
+#include <ctype.h>
+#include <fcntl.h>
+#include <getopt.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <xtables.h>
+#include <arpa/inet.h>
+#include <netinet/ether.h>
+#include <netinet/in.h>
+#include <linux/if_ether.h>
+#include <linux/netfilter_bridge/ebt_among.h>
+#include <sys/mman.h>
+#include <sys/socket.h>
+#include <sys/stat.h>
+#include "iptables/nft.h"
+#include "iptables/nft-bridge.h"
+
+#define AMONG_DST '1'
+#define AMONG_SRC '2'
+#define AMONG_DST_F '3'
+#define AMONG_SRC_F '4'
+
+static const struct option bramong_opts[] = {
+	{"among-dst", required_argument, 0, AMONG_DST},
+	{"among-src", required_argument, 0, AMONG_SRC},
+	{"among-dst-file", required_argument, 0, AMONG_DST_F},
+	{"among-src-file", required_argument, 0, AMONG_SRC_F},
+	{0}
+};
+
+static void bramong_print_help(void)
+{
+	printf(
+"`among' options:\n"
+"--among-dst      [!] list      : matches if ether dst is in list\n"
+"--among-src      [!] list      : matches if ether src is in list\n"
+"--among-dst-file [!] file      : obtain dst list from file\n"
+"--among-src-file [!] file      : obtain src list from file\n"
+"list has form:\n"
+" xx:xx:xx:xx:xx:xx[=ip.ip.ip.ip],yy:yy:yy:yy:yy:yy[=ip.ip.ip.ip]"
+",...,zz:zz:zz:zz:zz:zz[=ip.ip.ip.ip][,]\n"
+"Things in brackets are optional.\n"
+"If you want to allow two (or more) IP addresses to one MAC address, you\n"
+"can specify two (or more) pairs with the same MAC, e.g.\n"
+" 00:00:00:fa:eb:fe=153.19.120.250,00:00:00:fa:eb:fe=192.168.0.1\n"
+	);
+}
+
+static void
+parse_nft_among_pair(char *buf, struct nft_among_pair *pair, bool have_ip)
+{
+	char *sep = index(buf, '=');
+	struct ether_addr *ether;
+
+	if (have_ip ^ !!sep)
+		xtables_error(PARAMETER_PROBLEM,
+			      "among: Mixed MAC and MAC=IP not allowed.");
+
+	if (sep) {
+		*sep = '\0';
+
+		if (!inet_aton(sep + 1, &pair->in))
+			xtables_error(PARAMETER_PROBLEM,
+				      "Invalid IP address '%s'\n", sep + 1);
+	}
+	ether = ether_aton(buf);
+	if (!ether)
+		xtables_error(PARAMETER_PROBLEM,
+			      "Invalid MAC address '%s'\n", buf);
+	memcpy(&pair->ether, ether, sizeof(*ether));
+}
+
+static void
+parse_nft_among_pairs(struct nft_among_pair *pairs, char *buf,
+		      size_t cnt, bool have_ip)
+{
+	size_t tmpcnt = 0;
+
+	buf = strtok(buf, ",");
+	while (buf) {
+		struct nft_among_pair pair = {};
+
+		parse_nft_among_pair(buf, &pair, have_ip);
+		nft_among_insert_pair(pairs, &tmpcnt, &pair);
+		buf = strtok(NULL, ",");
+	}
+}
+
+static size_t count_nft_among_pairs(char *buf)
+{
+	size_t cnt = 0;
+	char *p = buf;
+
+	if (!*buf)
+		return 0;
+
+	do {
+		cnt++;
+		p = index(++p, ',');
+	} while (p);
+
+	return cnt;
+}
+
+static bool nft_among_pairs_have_ip(char *buf)
+{
+	return !!index(buf, '=');
+}
+
+static int bramong_parse(int c, char **argv, int invert,
+		 unsigned int *flags, const void *entry,
+		 struct xt_entry_match **match)
+{
+	struct nft_among_data *data = (struct nft_among_data *)(*match)->data;
+	struct xt_entry_match *new_match;
+	bool have_ip, dst = false;
+	size_t new_size, cnt;
+	struct stat stats;
+	int fd = -1, poff;
+	long flen = 0;
+
+	switch (c) {
+	case AMONG_DST_F:
+		dst = true;
+		/* fall through */
+	case AMONG_SRC_F:
+		if ((fd = open(optarg, O_RDONLY)) == -1)
+			xtables_error(PARAMETER_PROBLEM,
+				      "Couldn't open file '%s'", optarg);
+		fstat(fd, &stats);
+		flen = stats.st_size;
+		/* use mmap because the file will probably be big */
+		optarg = mmap(0, flen, PROT_READ | PROT_WRITE,
+			      MAP_PRIVATE, fd, 0);
+		if (optarg == MAP_FAILED)
+			xtables_error(PARAMETER_PROBLEM,
+				      "Couldn't map file to memory");
+		if (optarg[flen-1] != '\n')
+			xtables_error(PARAMETER_PROBLEM,
+				      "File should end with a newline");
+		if (strchr(optarg, '\n') != optarg+flen-1)
+			xtables_error(PARAMETER_PROBLEM,
+				      "File should only contain one line");
+		optarg[flen-1] = '\0';
+		/* fall through */
+	case AMONG_DST:
+		if (c == AMONG_DST)
+			dst = true;
+		/* fall through */
+	case AMONG_SRC:
+		break;
+	default:
+		return 0;
+	}
+
+	cnt = count_nft_among_pairs(optarg);
+	if (cnt == 0)
+		return 0;
+
+	new_size = data->src.cnt + data->dst.cnt + cnt;
+	new_size *= sizeof(struct nft_among_pair);
+	new_size += XT_ALIGN(sizeof(struct xt_entry_match)) +
+			sizeof(struct nft_among_data);
+	new_match = xtables_calloc(1, new_size);
+	memcpy(new_match, *match, (*match)->u.match_size);
+	new_match->u.match_size = new_size;
+
+	data = (struct nft_among_data *)new_match->data;
+	have_ip = nft_among_pairs_have_ip(optarg);
+	poff = nft_among_prepare_data(data, dst, cnt, invert, have_ip);
+	parse_nft_among_pairs(data->pairs + poff, optarg, cnt, have_ip);
+
+	free(*match);
+	*match = new_match;
+
+	if (c == AMONG_DST_F || c == AMONG_SRC_F) {
+		munmap(argv, flen);
+		close(fd);
+	}
+	return 1;
+}
+
+static void __bramong_print(struct nft_among_pair *pairs,
+			    int cnt, bool inv, bool have_ip)
+{
+	const char *isep = inv ? "! " : "";
+	int i;
+
+	for (i = 0; i < cnt; i++) {
+		printf("%s", isep);
+		isep = ",";
+
+		printf("%s", ether_ntoa(&pairs[i].ether));
+		if (have_ip)
+			printf("=%s", inet_ntoa(pairs[i].in));
+	}
+	printf(" ");
+}
+
+static void bramong_print(const void *ip, const struct xt_entry_match *match,
+			  int numeric)
+{
+	struct nft_among_data *data = (struct nft_among_data *)match->data;
+
+	if (data->src.cnt) {
+		printf("--among-src ");
+		__bramong_print(data->pairs,
+				data->src.cnt, data->src.inv, data->src.ip);
+	}
+	if (data->dst.cnt) {
+		printf("--among-dst ");
+		__bramong_print(data->pairs + data->src.cnt,
+				data->dst.cnt, data->dst.inv, data->dst.ip);
+	}
+}
+
+static struct xtables_match bramong_match = {
+	.name		= "among",
+	.revision	= 0,
+	.version	= XTABLES_VERSION,
+	.family		= NFPROTO_BRIDGE,
+	.size		= XT_ALIGN(sizeof(struct nft_among_data)),
+	.userspacesize	= XT_ALIGN(sizeof(struct nft_among_data)),
+	.help		= bramong_print_help,
+	.parse		= bramong_parse,
+	.print		= bramong_print,
+	.extra_opts	= bramong_opts,
+};
+
+void _init(void)
+{
+	xtables_register_match(&bramong_match);
+}
diff --git a/extensions/libebt_among.t b/extensions/libebt_among.t
new file mode 100644
index 0000000000000..56b299161ff31
--- /dev/null
+++ b/extensions/libebt_among.t
@@ -0,0 +1,16 @@
+:INPUT,FORWARD,OUTPUT
+--among-dst de:ad:0:be:ee:ff,c0:ff:ee:0:ba:be;--among-dst c0:ff:ee:0:ba:be,de:ad:0:be:ee:ff;OK
+--among-dst ! c0:ff:ee:0:ba:be,de:ad:0:be:ee:ff;=;OK
+--among-src be:ef:0:c0:ff:ee,c0:ff:ee:0:ba:be,de:ad:0:be:ee:ff;=;OK
+--among-src de:ad:0:be:ee:ff=10.0.0.1,c0:ff:ee:0:ba:be=192.168.1.1;--among-src c0:ff:ee:0:ba:be=192.168.1.1,de:ad:0:be:ee:ff=10.0.0.1;OK
+--among-src ! c0:ff:ee:0:ba:be=192.168.1.1,de:ad:0:be:ee:ff=10.0.0.1;=;OK
+--among-src de:ad:0:be:ee:ff --among-dst c0:ff:ee:0:ba:be;=;OK
+--among-src de:ad:0:be:ee:ff=10.0.0.1 --among-dst c0:ff:ee:0:ba:be=192.168.1.1;=;OK
+--among-src ! de:ad:0:be:ee:ff --among-dst c0:ff:ee:0:ba:be;=;OK
+--among-src de:ad:0:be:ee:ff=10.0.0.1 --among-dst ! c0:ff:ee:0:ba:be=192.168.1.1;=;OK
+--among-src ! de:ad:0:be:ee:ff --among-dst c0:ff:ee:0:ba:be=192.168.1.1;=;OK
+--among-src de:ad:0:be:ee:ff=10.0.0.1 --among-dst ! c0:ff:ee:0:ba:be=192.168.1.1;=;OK
+--among-src;=;FAIL
+--among-src 00:11=10.0.0.1;=;FAIL
+--among-src de:ad:0:be:ee:ff=10.256.0.1;=;FAIL
+--among-src de:ad:0:be:ee:ff,c0:ff:ee:0:ba:be=192.168.1.1;=;FAIL
diff --git a/iptables/ebtables-nft.8 b/iptables/ebtables-nft.8
index db8b2ab28cca5..a91f0c1aacb0f 100644
--- a/iptables/ebtables-nft.8
+++ b/iptables/ebtables-nft.8
@@ -522,35 +522,39 @@ If the 802.3 DSAP and SSAP values are 0xaa then the SNAP type field must
 be consulted to determine the payload protocol.  This is a two byte
 (hexadecimal) argument.  Only 802.3 frames with DSAP/SSAP 0xaa are
 checked for type.
-.\" .SS among
-.\" Match a MAC address or MAC/IP address pair versus a list of MAC addresses
-.\" and MAC/IP address pairs.
-.\" A list entry has the following format:
-.\" .IR xx:xx:xx:xx:xx:xx[=ip.ip.ip.ip][,] ". Multiple"
-.\" list entries are separated by a comma, specifying an IP address corresponding to
-.\" the MAC address is optional. Multiple MAC/IP address pairs with the same MAC address
-.\" but different IP address (and vice versa) can be specified. If the MAC address doesn't
-.\" match any entry from the list, the frame doesn't match the rule (unless "!" was used).
-.\" .TP
-.\" .BR "--among-dst " "[!] \fIlist\fP"
-.\" Compare the MAC destination to the given list. If the Ethernet frame has type
-.\" .IR IPv4 " or " ARP ,
-.\" then comparison with MAC/IP destination address pairs from the
-.\" list is possible.
-.\" .TP
-.\" .BR "--among-src " "[!] \fIlist\fP"
-.\" Compare the MAC source to the given list. If the Ethernet frame has type
-.\" .IR IPv4 " or " ARP ,
-.\" then comparison with MAC/IP source address pairs from the list
-.\" is possible.
-.\" .TP
-.\" .BR "--among-dst-file " "[!] \fIfile\fP"
-.\" Same as
-.\" .BR --among-dst " but the list is read in from the specified file."
-.\" .TP
-.\" .BR "--among-src-file " "[!] \fIfile\fP"
-.\" Same as
-.\" .BR --among-src " but the list is read in from the specified file."
+.SS among
+Match a MAC address or MAC/IP address pair versus a list of MAC addresses
+and MAC/IP address pairs.
+A list entry has the following format:
+.IR xx:xx:xx:xx:xx:xx[=ip.ip.ip.ip][,] ". Multiple"
+list entries are separated by a comma, specifying an IP address corresponding to
+the MAC address is optional. Multiple MAC/IP address pairs with the same MAC address
+but different IP address (and vice versa) can be specified. If the MAC address doesn't
+match any entry from the list, the frame doesn't match the rule (unless "!" was used).
+.TP
+.BR "--among-dst " "[!] \fIlist\fP"
+Compare the MAC destination to the given list. If the Ethernet frame has type
+.IR IPv4 " or " ARP ,
+then comparison with MAC/IP destination address pairs from the
+list is possible.
+.TP
+.BR "--among-src " "[!] \fIlist\fP"
+Compare the MAC source to the given list. If the Ethernet frame has type
+.IR IPv4 " or " ARP ,
+then comparison with MAC/IP source address pairs from the list
+is possible.
+.TP
+.BR "--among-dst-file " "[!] \fIfile\fP"
+Same as
+.BR --among-dst " but the list is read in from the specified file."
+.TP
+.BR "--among-src-file " "[!] \fIfile\fP"
+Same as
+.BR --among-src " but the list is read in from the specified file."
+.PP
+Note that in this implementation of ebtables, among lists uses must be
+internally homogeneous regarding whether IP addresses are present or not. Mixed
+use of MAC addresses and MAC/IP address pairs is not supported yet.
 .SS arp
 Specify (R)ARP fields. The protocol must be specified as
 .IR ARP " or " RARP .
@@ -1108,8 +1112,8 @@ arp message and the hardware address length in the arp header is 6 bytes.
 The version of ebtables this man page ships with does not support the
 .B broute
 table. Also there is no support for
-.BR among " and " string
-matches. And finally, this list is probably not complete.
+.B string
+match. And finally, this list is probably not complete.
 .SH SEE ALSO
 .BR xtables-nft "(8), " iptables "(8), " ip (8)
 .PP
diff --git a/iptables/nft-bridge.c b/iptables/nft-bridge.c
index 20ce92a6d5242..3f85cbbf5e4cf 100644
--- a/iptables/nft-bridge.c
+++ b/iptables/nft-bridge.c
@@ -17,8 +17,11 @@
 #include <libiptc/libxtc.h>
 #include <linux/netfilter/nf_tables.h>
 
+#include <libnftnl/set.h>
+
 #include "nft-shared.h"
 #include "nft-bridge.h"
+#include "nft-cache.h"
 #include "nft.h"
 
 void ebt_cs_clean(struct iptables_command_state *cs)
@@ -291,6 +294,212 @@ static void nft_bridge_parse_immediate(const char *jumpto, bool nft_goto,
 	cs->jumpto = jumpto;
 }
 
+/* return 0 if saddr, 1 if daddr, -1 on error */
+static int
+lookup_check_ether_payload(uint32_t base, uint32_t offset, uint32_t len)
+{
+	if (base != 0 || len != ETH_ALEN)
+		return -1;
+
+	switch (offset) {
+	case offsetof(struct ether_header, ether_dhost):
+		return 1;
+	case offsetof(struct ether_header, ether_shost):
+		return 0;
+	default:
+		return -1;
+	}
+}
+
+/* return 0 if saddr, 1 if daddr, -1 on error */
+static int
+lookup_check_iphdr_payload(uint32_t base, uint32_t offset, uint32_t len)
+{
+	if (base != 1 || len != 4)
+		return -1;
+
+	switch (offset) {
+	case offsetof(struct iphdr, daddr):
+		return 1;
+	case offsetof(struct iphdr, saddr):
+		return 0;
+	default:
+		return -1;
+	}
+}
+
+/* Make sure previous payload expression(s) is/are consistent and extract if
+ * matching on source or destination address and if matching on MAC and IP or
+ * only MAC address. */
+static int lookup_analyze_payloads(const struct nft_xt_ctx *ctx,
+				   bool *dst, bool *ip)
+{
+	int val, val2 = -1;
+
+	if (ctx->flags & NFT_XT_CTX_PREV_PAYLOAD) {
+		val = lookup_check_ether_payload(ctx->prev_payload.base,
+						 ctx->prev_payload.offset,
+						 ctx->prev_payload.len);
+		if (val < 0) {
+			DEBUGP("unknown payload base/offset/len %d/%d/%d\n",
+			       ctx->prev_payload.base, ctx->prev_payload.offset,
+			       ctx->prev_payload.len);
+			return -1;
+		}
+		if (!(ctx->flags & NFT_XT_CTX_PAYLOAD)) {
+			DEBUGP("Previous but no current payload?\n");
+			return -1;
+		}
+		val2 = lookup_check_iphdr_payload(ctx->payload.base,
+						  ctx->payload.offset,
+						  ctx->payload.len);
+		if (val2 < 0) {
+			DEBUGP("unknown payload base/offset/len %d/%d/%d\n",
+			       ctx->payload.base, ctx->payload.offset,
+			       ctx->payload.len);
+			return -1;
+		} else if (val != val2) {
+			DEBUGP("mismatching payload match offsets\n");
+			return -1;
+		}
+	} else if (ctx->flags & NFT_XT_CTX_PAYLOAD) {
+		val = lookup_check_ether_payload(ctx->payload.base,
+						 ctx->payload.offset,
+						 ctx->payload.len);
+		if (val < 0) {
+			DEBUGP("unknown payload base/offset/len %d/%d/%d\n",
+			       ctx->payload.base, ctx->payload.offset,
+			       ctx->payload.len);
+			return -1;
+		}
+	} else {
+		DEBUGP("unknown LHS of lookup expression\n");
+		return -1;
+	}
+
+	if (dst)
+		*dst = (val == 1);
+	if (ip)
+		*ip = (val2 != -1);
+	return 0;
+}
+
+static int set_elems_to_among_pairs(struct nft_among_pair *pairs,
+				    const struct nftnl_set *s, int cnt)
+{
+	struct nftnl_set_elems_iter *iter = nftnl_set_elems_iter_create(s);
+	struct nftnl_set_elem *elem;
+	size_t tmpcnt = 0;
+	const void *data;
+	uint32_t datalen;
+	int ret = -1;
+
+	if (!iter) {
+		fprintf(stderr, "BUG: set elems iter allocation failed\n");
+		return ret;
+	}
+
+	while ((elem = nftnl_set_elems_iter_next(iter))) {
+		data = nftnl_set_elem_get(elem, NFTNL_SET_ELEM_KEY, &datalen);
+		if (!data) {
+			fprintf(stderr, "BUG: set elem without key\n");
+			goto err;
+		}
+		if (datalen > sizeof(*pairs)) {
+			fprintf(stderr, "BUG: overlong set elem\n");
+			goto err;
+		}
+		nft_among_insert_pair(pairs, &tmpcnt, data);
+	}
+	ret = 0;
+err:
+	nftnl_set_elems_iter_destroy(iter);
+	return ret;
+}
+
+static struct nftnl_set *set_from_lookup_expr(struct nft_xt_ctx *ctx,
+					      const struct nftnl_expr *e)
+{
+	const char *set_name = nftnl_expr_get_str(e, NFTNL_EXPR_LOOKUP_SET);
+	struct nftnl_set_list *slist;
+
+	slist = nft_set_list_get(ctx->h, ctx->table, set_name);
+	if (slist)
+		return nftnl_set_list_lookup_byname(slist, set_name);
+
+	return NULL;
+}
+
+static void nft_bridge_parse_lookup(struct nft_xt_ctx *ctx,
+				    struct nftnl_expr *e, void *data)
+{
+	struct xtables_match *match = NULL;
+	struct nft_among_data *among_data;
+	bool is_dst, have_ip, inv;
+	struct ebt_match *ematch;
+	struct nftnl_set *s;
+	size_t poff, size;
+	uint32_t cnt;
+
+	if (lookup_analyze_payloads(ctx, &is_dst, &have_ip))
+		return;
+
+	s = set_from_lookup_expr(ctx, e);
+	if (!s)
+		xtables_error(OTHER_PROBLEM,
+			      "BUG: lookup expression references unknown set");
+
+	cnt = nftnl_set_get_u32(s, NFTNL_SET_DESC_SIZE);
+
+	for (ematch = ctx->cs->match_list; ematch; ematch = ematch->next) {
+		if (!ematch->ismatch || strcmp(ematch->u.match->name, "among"))
+			continue;
+
+		match = ematch->u.match;
+		among_data = (struct nft_among_data *)match->m->data;
+
+		size = cnt + among_data->src.cnt + among_data->dst.cnt;
+		size *= sizeof(struct nft_among_pair);
+
+		size += XT_ALIGN(sizeof(struct xt_entry_match)) +
+			sizeof(struct nft_among_data);
+
+		match->m = xtables_realloc(match->m, size);
+		break;
+	}
+	if (!match) {
+		match = xtables_find_match("among", XTF_TRY_LOAD,
+					   &ctx->cs->matches);
+
+		size = cnt * sizeof(struct nft_among_pair);
+		size += XT_ALIGN(sizeof(struct xt_entry_match)) +
+			sizeof(struct nft_among_data);
+
+		match->m = xtables_calloc(1, size);
+		strcpy(match->m->u.user.name, match->name);
+		match->m->u.user.revision = match->revision;
+		xs_init_match(match);
+
+		if (ctx->h->ops->parse_match != NULL)
+			ctx->h->ops->parse_match(match, ctx->cs);
+	}
+	if (!match)
+		return;
+
+	match->m->u.match_size = size;
+
+	inv = !!(nftnl_expr_get_u32(e, NFTNL_EXPR_LOOKUP_FLAGS) &
+				    NFT_LOOKUP_F_INV);
+
+	among_data = (struct nft_among_data *)match->m->data;
+	poff = nft_among_prepare_data(among_data, is_dst, cnt, inv, have_ip);
+	if (set_elems_to_among_pairs(among_data->pairs + poff, s, cnt))
+		xtables_error(OTHER_PROBLEM,
+			      "ebtables among pair parsing failed");
+
+	ctx->flags &= ~(NFT_XT_CTX_PAYLOAD | NFT_XT_CTX_PREV_PAYLOAD);
+}
+
 static void parse_watcher(void *object, struct ebt_match **match_list,
 			  bool ismatch)
 {
@@ -742,6 +951,7 @@ struct nft_family_ops nft_family_ops_bridge = {
 	.parse_meta		= nft_bridge_parse_meta,
 	.parse_payload		= nft_bridge_parse_payload,
 	.parse_immediate	= nft_bridge_parse_immediate,
+	.parse_lookup		= nft_bridge_parse_lookup,
 	.parse_match		= nft_bridge_parse_match,
 	.parse_target		= nft_bridge_parse_target,
 	.print_table_header	= nft_bridge_print_table_header,
diff --git a/iptables/nft-bridge.h b/iptables/nft-bridge.h
index d90066f1030a2..eb1b3928b6543 100644
--- a/iptables/nft-bridge.h
+++ b/iptables/nft-bridge.h
@@ -122,4 +122,60 @@ void ebt_add_watcher(struct xtables_target *watcher,
                      struct iptables_command_state *cs);
 int ebt_command_default(struct iptables_command_state *cs);
 
+struct nft_among_pair {
+	struct ether_addr ether;
+	struct in_addr in __attribute__((aligned (4)));
+};
+
+struct nft_among_data {
+	struct {
+		size_t cnt;
+		bool inv;
+		bool ip;
+	} src, dst;
+	/* first source, then dest pairs */
+	struct nft_among_pair pairs[0];
+};
+
+/* initialize fields, return offset into pairs array to write pairs to */
+static inline size_t
+nft_among_prepare_data(struct nft_among_data *data, bool dst,
+		       size_t cnt, bool inv, bool ip)
+{
+	size_t poff;
+
+	if (dst) {
+		data->dst.cnt = cnt;
+		data->dst.inv = inv;
+		data->dst.ip = ip;
+		poff = data->src.cnt;
+	} else {
+		data->src.cnt = cnt;
+		data->src.inv = inv;
+		data->src.ip = ip;
+		poff = 0;
+		memmove(data->pairs + cnt, data->pairs,
+			data->dst.cnt * sizeof(*data->pairs));
+	}
+	return poff;
+}
+
+static inline void
+nft_among_insert_pair(struct nft_among_pair *pairs,
+		      size_t *pcount, const struct nft_among_pair *new)
+{
+	int i;
+
+	/* nftables automatically sorts set elements from smallest to largest,
+	 * insert sorted so extension comparison works */
+
+	for (i = 0; i < *pcount; i++) {
+		if (memcmp(new, &pairs[i], sizeof(*new)) < 0)
+			break;
+	}
+	memmove(&pairs[i + 1], &pairs[i], sizeof(*pairs) * (*pcount - i));
+	memcpy(&pairs[i], new, sizeof(*new));
+	(*pcount)++;
+}
+
 #endif
diff --git a/iptables/nft.c b/iptables/nft.c
index 111d4982b1181..8c0088b46c45d 100644
--- a/iptables/nft.c
+++ b/iptables/nft.c
@@ -944,6 +944,153 @@ static int add_nft_limit(struct nftnl_rule *r, struct xt_entry_match *m)
 	return 0;
 }
 
+static struct nftnl_set *add_anon_set(struct nft_handle *h, const char *table,
+				      uint32_t flags, uint32_t key_type,
+				      uint32_t key_len, uint32_t size)
+{
+	static uint32_t set_id = 0;
+	struct nftnl_set *s;
+
+	s = nftnl_set_alloc();
+	if (!s)
+		return NULL;
+
+	nftnl_set_set_u32(s, NFTNL_SET_FAMILY, h->family);
+	nftnl_set_set_str(s, NFTNL_SET_TABLE, table);
+	nftnl_set_set_str(s, NFTNL_SET_NAME, "__set%d");
+	nftnl_set_set_u32(s, NFTNL_SET_ID, ++set_id);
+	nftnl_set_set_u32(s, NFTNL_SET_FLAGS,
+			  NFT_SET_ANONYMOUS | NFT_SET_CONSTANT | flags);
+	nftnl_set_set_u32(s, NFTNL_SET_KEY_TYPE, key_type);
+	nftnl_set_set_u32(s, NFTNL_SET_KEY_LEN, key_len);
+	nftnl_set_set_u32(s, NFTNL_SET_DESC_SIZE, size);
+
+	return batch_set_add(h, NFT_COMPAT_SET_ADD, s) ? s : NULL;
+}
+
+static struct nftnl_expr *
+gen_payload(uint32_t base, uint32_t offset, uint32_t len, uint32_t dreg)
+{
+	struct nftnl_expr *e = nftnl_expr_alloc("payload");
+
+	if (!e)
+		return NULL;
+	nftnl_expr_set_u32(e, NFTNL_EXPR_PAYLOAD_BASE, base);
+	nftnl_expr_set_u32(e, NFTNL_EXPR_PAYLOAD_OFFSET, offset);
+	nftnl_expr_set_u32(e, NFTNL_EXPR_PAYLOAD_LEN, len);
+	nftnl_expr_set_u32(e, NFTNL_EXPR_PAYLOAD_DREG, dreg);
+	return e;
+}
+
+static struct nftnl_expr *
+gen_lookup(uint32_t sreg, const char *set_name, uint32_t set_id, uint32_t flags)
+{
+	struct nftnl_expr *e = nftnl_expr_alloc("lookup");
+
+	if (!e)
+		return NULL;
+	nftnl_expr_set_u32(e, NFTNL_EXPR_LOOKUP_SREG, sreg);
+	nftnl_expr_set_str(e, NFTNL_EXPR_LOOKUP_SET, set_name);
+	nftnl_expr_set_u32(e, NFTNL_EXPR_LOOKUP_SET_ID, set_id);
+	nftnl_expr_set_u32(e, NFTNL_EXPR_LOOKUP_FLAGS, flags);
+	return e;
+}
+
+/* simplified nftables:include/netlink.h, netlink_padded_len() */
+#define NETLINK_ALIGN		4
+
+/* from nftables:include/datatype.h, TYPE_BITS */
+#define CONCAT_TYPE_BITS	6
+
+/* from nftables:include/datatype.h, enum datatypes */
+#define NFT_DATATYPE_IPADDR	7
+#define NFT_DATATYPE_ETHERADDR	9
+
+static int __add_nft_among(struct nft_handle *h, const char *table,
+			   struct nftnl_rule *r, struct nft_among_pair *pairs,
+			   int cnt, bool dst, bool inv, bool ip)
+{
+	uint32_t set_id, type = NFT_DATATYPE_ETHERADDR, len = ETH_ALEN;
+	/* { !dst, dst } */
+	static const int eth_addr_off[] = {
+		offsetof(struct ether_header, ether_shost),
+		offsetof(struct ether_header, ether_dhost)
+	};
+	static const int ip_addr_off[] = {
+		offsetof(struct iphdr, saddr),
+		offsetof(struct iphdr, daddr)
+	};
+	struct nftnl_expr *e;
+	struct nftnl_set *s;
+	int idx = 0;
+
+	if (ip) {
+		type = type << CONCAT_TYPE_BITS | NFT_DATATYPE_IPADDR;
+		len += sizeof(struct in_addr) + NETLINK_ALIGN - 1;
+		len &= ~(NETLINK_ALIGN - 1);
+	}
+
+	s = add_anon_set(h, table, 0, type, len, cnt);
+	if (!s)
+		return -ENOMEM;
+	set_id = nftnl_set_get_u32(s, NFTNL_SET_ID);
+
+	for (idx = 0; idx < cnt; idx++) {
+		struct nftnl_set_elem *elem = nftnl_set_elem_alloc();
+
+		if (!elem)
+			return -ENOMEM;
+		nftnl_set_elem_set(elem, NFTNL_SET_ELEM_KEY,
+				   &pairs[idx], len);
+		nftnl_set_elem_add(s, elem);
+	}
+
+	e = gen_payload(NFT_PAYLOAD_LL_HEADER,
+			eth_addr_off[dst], ETH_ALEN, NFT_REG_1);
+	if (!e)
+		return -ENOMEM;
+	nftnl_rule_add_expr(r, e);
+
+	if (ip) {
+		e = gen_payload(NFT_PAYLOAD_NETWORK_HEADER, ip_addr_off[dst],
+				sizeof(struct in_addr), NFT_REG32_02);
+		if (!e)
+			return -ENOMEM;
+		nftnl_rule_add_expr(r, e);
+	}
+
+	e = gen_lookup(NFT_REG_1, "__set%d", set_id, inv);
+	if (!e)
+		return -ENOMEM;
+	nftnl_rule_add_expr(r, e);
+
+	return 0;
+}
+
+static int add_nft_among(struct nft_handle *h,
+			 struct nftnl_rule *r, struct xt_entry_match *m)
+{
+	struct nft_among_data *data = (struct nft_among_data *)m->data;
+	const char *table = nftnl_rule_get(r, NFTNL_RULE_TABLE);
+
+	if ((data->src.cnt && data->src.ip) ||
+	    (data->dst.cnt && data->dst.ip)) {
+		uint16_t eth_p_ip = htons(ETH_P_IP);
+
+		add_meta(r, NFT_META_PROTOCOL);
+		add_cmp_ptr(r, NFT_CMP_EQ, &eth_p_ip, 2);
+	}
+
+	if (data->src.cnt)
+		__add_nft_among(h, table, r, data->pairs, data->src.cnt,
+				false, data->src.inv, data->src.ip);
+	if (data->dst.cnt)
+		__add_nft_among(h, table, r, data->pairs + data->src.cnt,
+				data->dst.cnt, true, data->dst.inv,
+				data->dst.ip);
+	return 0;
+}
+
 int add_match(struct nft_handle *h,
 	      struct nftnl_rule *r, struct xt_entry_match *m)
 {
@@ -952,6 +1099,8 @@ int add_match(struct nft_handle *h,
 
 	if (!strcmp(m->u.user.name, "limit"))
 		return add_nft_limit(r, m);
+	else if (!strcmp(m->u.user.name, "among"))
+		return add_nft_among(h, r, m);
 
 	expr = nftnl_expr_alloc("match");
 	if (expr == NULL)
diff --git a/iptables/xtables-eb.c b/iptables/xtables-eb.c
index fd7d601f6136a..15b971da3d425 100644
--- a/iptables/xtables-eb.c
+++ b/iptables/xtables-eb.c
@@ -594,6 +594,7 @@ void ebt_load_match_extensions(void)
 	ebt_load_match("pkttype");
 	ebt_load_match("vlan");
 	ebt_load_match("stp");
+	ebt_load_match("among");
 
 	ebt_load_watcher("log");
 	ebt_load_watcher("nflog");
-- 
2.23.0

Re: [iptables PATCH v3 07/12] nft: Introduce NFT_CL_SETS cache level

[Pablo Neira Ayuso]

On Wed, Oct 30, 2019 at 06:26:56PM +0100, Phil Sutter wrote:
> +static int set_fetch_elem_cb(struct nftnl_set *s, void *data)
> +{
> +        char buf[MNL_SOCKET_BUFFER_SIZE];
> +	struct nft_handle *h = data;
> +        struct nlmsghdr *nlh;
> +
> +	if (set_has_elements(s))
> +		return 0;
> +
> +	nlh = nftnl_nlmsg_build_hdr(buf, NFT_MSG_GETSETELEM, h->family,
> +				    NLM_F_DUMP, h->seq);
> +        nftnl_set_elems_nlmsg_build_payload(nlh, s);
> +
> +	return mnl_talk(h, nlh, set_elem_cb, s);
> +}

Please, mind coding style, irregular indentation.

Re: [iptables PATCH v3 01/12] nft: family_ops: Pass nft_handle to 'add' callback

[Pablo Neira Ayuso]

On Wed, Oct 30, 2019 at 06:26:50PM +0100, Phil Sutter wrote:
> In order for add_match() to create anonymous sets when converting
> xtables matches it needs access to nft handle. So pass it along from
> callers of family ops' add callback.
> 
> Signed-off-by: Phil Sutter <phil@nwl.cc>

Acked-by: Pablo Neira Ayuso <pablo@netfilter.org>

Re: [iptables PATCH v3 02/12] nft: family_ops: Pass nft_handle to 'rule_find' callback

[Pablo Neira Ayuso]

On Wed, Oct 30, 2019 at 06:26:51PM +0100, Phil Sutter wrote:
> In order to prepare for rules containing set references, nft handle has
> to be passed to nft_rule_to_iptables_command_state() in order to let it
> access the set in cache.
> 
> Signed-off-by: Phil Sutter <phil@nwl.cc>

Acked-by: Pablo Neira Ayuso <pablo@netfilter.org>

Re: [iptables PATCH v3 03/12] nft: family_ops: Pass nft_handle to 'print_rule' callback

[Pablo Neira Ayuso]

On Wed, Oct 30, 2019 at 06:26:52PM +0100, Phil Sutter wrote:
> Prepare for 'rule_to_cs' callback to receive nft_handle pointer so it is
> able to access cache for set lookups.
> 
> Signed-off-by: Phil Sutter <phil@nwl.cc>

Acked-by: Pablo Neira Ayuso <pablo@netfilter.org>

Re: [iptables PATCH v3 07/12] nft: Introduce NFT_CL_SETS cache level

[Phil Sutter]

On Thu, Oct 31, 2019 at 03:04:25PM +0100, Pablo Neira Ayuso wrote:
> On Wed, Oct 30, 2019 at 06:26:56PM +0100, Phil Sutter wrote:
> > +static int set_fetch_elem_cb(struct nftnl_set *s, void *data)
> > +{
> > +        char buf[MNL_SOCKET_BUFFER_SIZE];
> > +	struct nft_handle *h = data;
> > +        struct nlmsghdr *nlh;
> > +
> > +	if (set_has_elements(s))
> > +		return 0;
> > +
> > +	nlh = nftnl_nlmsg_build_hdr(buf, NFT_MSG_GETSETELEM, h->family,
> > +				    NLM_F_DUMP, h->seq);
> > +        nftnl_set_elems_nlmsg_build_payload(nlh, s);
> > +
> > +	return mnl_talk(h, nlh, set_elem_cb, s);
> > +}
> 
> Please, mind coding style, irregular indentation.

Oh, thanks for spotting that. Eight spaces are not noticeable in my
editor, maybe I should highlight those.

Thanks, Phil

Re: [iptables PATCH v3 00/12] Implement among match support

[Pablo Neira Ayuso]

On Wed, Oct 30, 2019 at 06:26:49PM +0100, Phil Sutter wrote:
[...]
> Patches 1 to 5 implement required changes and are rather boring by
> themselves: When converting an nftnl rule to iptables command state,
> cache access is required (to lookup set references).

nft_handle is passed now all over the place, this allows anyone to
access all of its content. This layering design was done on purpose,
to avoid giving access to all information to the callers, instead
force the developer to give a reason to show why it needs something
else from wherever he is. I'm not entirely convinced exposing the
handle everywhere just because you need to access the set cache is the
way to go.

Re: [iptables PATCH v3 00/12] Implement among match support

[Pablo Neira Ayuso]

On Thu, Oct 31, 2019 at 03:13:14PM +0100, Pablo Neira Ayuso wrote:
> On Wed, Oct 30, 2019 at 06:26:49PM +0100, Phil Sutter wrote:
> [...]
> > Patches 1 to 5 implement required changes and are rather boring by
> > themselves: When converting an nftnl rule to iptables command state,
> > cache access is required (to lookup set references).
> 
> nft_handle is passed now all over the place, this allows anyone to
> access all of its content. This layering design was done on purpose,
> to avoid giving access to all information to the callers, instead
> force the developer to give a reason to show why it needs something
> else from wherever he is. I'm not entirely convinced exposing the
> handle everywhere just because you need to access the set cache is the
> way to go.

In other words: You only need the cache, right? Why don't you just
expose cache to these functions which what you need?

Re: [iptables PATCH v3 00/12] Implement among match support

[Phil Sutter]

Hi,

On Thu, Oct 31, 2019 at 03:14:52PM +0100, Pablo Neira Ayuso wrote:
> On Thu, Oct 31, 2019 at 03:13:14PM +0100, Pablo Neira Ayuso wrote:
> > On Wed, Oct 30, 2019 at 06:26:49PM +0100, Phil Sutter wrote:
> > [...]
> > > Patches 1 to 5 implement required changes and are rather boring by
> > > themselves: When converting an nftnl rule to iptables command state,
> > > cache access is required (to lookup set references).
> > 
> > nft_handle is passed now all over the place, this allows anyone to
> > access all of its content. This layering design was done on purpose,
> > to avoid giving access to all information to the callers, instead
> > force the developer to give a reason to show why it needs something
> > else from wherever he is. I'm not entirely convinced exposing the
> > handle everywhere just because you need to access the set cache is the
> > way to go.
> 
> In other words: You only need the cache, right? Why don't you just
> expose cache to these functions which what you need?

When creating a new rule with among match, code needs to call
batch_add() to add the NFT_COMPAT_SET_ADD job. So in that direction I
don't see an alternative to passing nft_handle around.

When parsing a lookup expression, we may get by without having to call
__nft_build_cache() as cache might be present already (not sure if I
miss something). If not, nft_handle is mandatory - cache update
functions access many fields in nft_handle.

So when passing cache and builtin_table pointers to rule_to_cs, pure set
lookups should be possible without nft_handle access. We need
builtin_table pointer to identify the right table array item in cache.
With only table name, we need to call nft_table_builtin_find() and that
takes nft_handle as well.

I could give it a try if you still think it's feasible to keep
nft_handle away from nft_xt_ctx.

Thanks, Phil