dict: A netfilter expression for dictionary lookups

dict: A netfilter expression for dictionary lookups

[Brett Mastbergen]

My name is Brett Mastbergen.  My colleage Dirk Morris and I have been
working on some nftables functionality that we think is kind of cool so we
figured we'd post it to the list to see if anyone had any thoughts or
feedback on what we are doing.

Below is the summary of nft_dict from our documentation which can be found 
here:
https://github.com/untangle/nft_dict/blob/master/docs/dict.rst

The nft_dict, short for netfilter dictionary, module provides a nft rule 
mechanism do table lookups on environment metadata that is not present in the 
packet and not contained within the rule. In a firewall you often wish to 
block or manipulate packets based on things not immediately evident in the
packet, but things that can often be calculated via other mechanisms.

nft_dict provides the ability to create dictionaries (lookup tables) stored 
within kernel space for fast lookups. These tables can be maintained by 
userspace applications where it is more convenient to calculate various 
network metadata. In our case, we have a userspace daemon (packetd) which
listens to various packets with nfqueue and builds various dictionaries of
metadata based on the traffic it sees and the information it gathers.

nft_dict is a kernel module providing the kernel support for the "dict" 
expression.  Its source can be found at the link below:

https://github.com/untangle/nft_dict

Additionally, userspace patches are required in order to use the 
"dict" expression from within an nft rule. These currently live here:

https://github.com/untangle/mfw_openwrt/blob/openwrt-18.06/libnftnl/patches/999-libnftnl-Add-dict-support.patch 
https://github.com/untangle/mfw_openwrt/blob/openwrt-18.06/nftables/patches/999-nftables-Add-dict.patch

These patches, along with the nft_dict kernel module, also add support for an 
"id" key to the "ct" match expression. The "ct id" expression simply returns 
the conntrack id of the conntrack. This is the same conntrack id you see if 
you run 'conntrack -L --output=id'. While not strictly required, the "ct id" 
expression is extremely useful as a key expression into a dict table for 
matching entries to a particular conntrack. The correct place to implement 
the "id" ct key is in the existing nft_ct module, but for now its implemented
in the nft_dict module.

For a more in depth description of how things work I suggest reading the doc 
in the first link I posted, but below are a few simple examples of what is 
possible using dict expressions:

nft add rule ip filter forward dict sessions ct id application long_string 
NETFLIX reject 

If I were to describe that rule in plain English it would be:

For traffic passing through the ip filter table forward hook, use the 
conntrack id as a key to lookup an entry in the sessions table.  For that
entry check if it has an application field set to a string value of NETFLIX,
if so, reject the traffic.

How did that field get set to NETFLIX?  That is up to some other entity.  In 
our case it is a userspace daemon that is inspecting traffic via nfqueue.
All of the metadata generated for the stream of traffic can be written into
the sessions table such that these dict rules can act upon it.  The userspace
daemon doesn't make determinations about how traffic should be acted upon
(accept, reject, drop, route, count, etc), it just populates the information
in the table to let the nft rules act upon the traffic.  This means the
userspace daemon only has to listen to traffic of a particular stream for as
long as it needs to determine all of the target metadata.  That particular
traffic stream can then bypass the nfqueue hook but can continue to be acted
upon using the information in the sessions table.

Here is another example:

nft add rule ip filter forward tcp dport 80 dict hosts ip saddr \
captive-portal-authenticated bool false dnat to 127.0.0.1:80

Here we are using the packet source ip address as the key into a hosts table
that (among other things) lets us know if this host has been authenticated via
captive portal.  If it hasn't, we send its web traffic to the local webserver.

The next example uses a dict expresssion as the key to another dict expression:

nft add rule ip filter forward dict users dict sessions ct id username \
long_string quota-exceeded bool true reject

This rule uses the conntrack id to look up the username associated with the
session, and then uses that username as the key into the users table so it
can check if this user has exceeded its quota.

If any of that sounds interesting, have a look at the code and docs.  Let us
know if you have questions or thoughts.

Thanks,
Brett

Re: dict: A netfilter expression for dictionary lookups

[Florian Westphal]

Brett Mastbergen <bmastbergen@untangle.com> wrote:
> My name is Brett Mastbergen.  My colleage Dirk Morris and I have been
> working on some nftables functionality that we think is kind of cool so we
> figured we'd post it to the list to see if anyone had any thoughts or
> feedback on what we are doing.
> 
> Below is the summary of nft_dict from our documentation which can be found 
> here:
> https://github.com/untangle/nft_dict/blob/master/docs/dict.rst
> 
> The nft_dict, short for netfilter dictionary, module provides a nft rule 
> mechanism do table lookups on environment metadata that is not present in the 
> packet and not contained within the rule. In a firewall you often wish to 
> block or manipulate packets based on things not immediately evident in the
> packet, but things that can often be calculated via other mechanisms.
>
> nft_dict provides the ability to create dictionaries (lookup tables) stored 
> within kernel space for fast lookups. These tables can be maintained by 
> userspace applications where it is more convenient to calculate various 
> network metadata. In our case, we have a userspace daemon (packetd) which
> listens to various packets with nfqueue and builds various dictionaries of
> metadata based on the traffic it sees and the information it gathers.
> 
> nft_dict is a kernel module providing the kernel support for the "dict" 
> expression.  Its source can be found at the link below:
> 
> https://github.com/untangle/nft_dict
> 
> Additionally, userspace patches are required in order to use the 
> "dict" expression from within an nft rule. These currently live here:
> 
> https://github.com/untangle/mfw_openwrt/blob/openwrt-18.06/libnftnl/patches/999-libnftnl-Add-dict-support.patch 
> https://github.com/untangle/mfw_openwrt/blob/openwrt-18.06/nftables/patches/999-nftables-Add-dict.patch
> 
> These patches, along with the nft_dict kernel module, also add support for an 
> "id" key to the "ct" match expression. The "ct id" expression simply returns 
> the conntrack id of the conntrack. This is the same conntrack id you see if 
> you run 'conntrack -L --output=id'. While not strictly required, the "ct id" 
> expression is extremely useful as a key expression into a dict table for 
> matching entries to a particular conntrack.

> The correct place to implement
> the "id" ct key is in the existing nft_ct module, but for now its implemented
> in the nft_dict module.

I would welcome adding support for this to nft_ct.c directly (i.e., get
support for a unique per-ct id) once this patch is accepted:

 https://patchwork.ozlabs.org/patch/1072650/

I added nf_ct_get_id() to the conntrack core so it can be called from
nft_ct later on.

> For a more in depth description of how things work I suggest reading the doc 
> in the first link I posted, but below are a few simple examples of what is 
> possible using dict expressions:
> 
> nft add rule ip filter forward dict sessions ct id application long_string 
> NETFLIX reject 
> 
> If I were to describe that rule in plain English it would be:
> 
> For traffic passing through the ip filter table forward hook, use the 
> conntrack id as a key to lookup an entry in the sessions table.  For that
> entry check if it has an application field set to a string value of NETFLIX,
> if so, reject the traffic.
> 
> How did that field get set to NETFLIX?  That is up to some other entity.  In 

Why isn't it possible to use the existing nftables set infrastructure
for this?

The nft sets store arbitrary octets/bytes as keys, so we could at least
from kernel side store arbitrary identifiers (strings, integers etc).

> nft add rule ip filter forward tcp dport 80 dict hosts ip saddr \
> captive-portal-authenticated bool false dnat to 127.0.0.1:80

nft supports something like

nft add rule ip filter forward tcp dport 80 ip saddr != @captive_authenticated ...

and allows external entity to push ip addresses to the set
@captive_authenticated.

I'll have a look at doc and code later today.

Re: dict: A netfilter expression for dictionary lookups

[Brett Mastbergen]

On 11-04-19, Florian Westphal wrote:
> Brett Mastbergen <bmastbergen@untangle.com> wrote:
> > My name is Brett Mastbergen.  My colleage Dirk Morris and I have been
> > working on some nftables functionality that we think is kind of cool so we
> > figured we'd post it to the list to see if anyone had any thoughts or
> > feedback on what we are doing.
> > 
> > Below is the summary of nft_dict from our documentation which can be found 
> > here:
> > https://github.com/untangle/nft_dict/blob/master/docs/dict.rst
> > 
> > The nft_dict, short for netfilter dictionary, module provides a nft rule 
> > mechanism do table lookups on environment metadata that is not present in the 
> > packet and not contained within the rule. In a firewall you often wish to 
> > block or manipulate packets based on things not immediately evident in the
> > packet, but things that can often be calculated via other mechanisms.
> >
> > nft_dict provides the ability to create dictionaries (lookup tables) stored 
> > within kernel space for fast lookups. These tables can be maintained by 
> > userspace applications where it is more convenient to calculate various 
> > network metadata. In our case, we have a userspace daemon (packetd) which
> > listens to various packets with nfqueue and builds various dictionaries of
> > metadata based on the traffic it sees and the information it gathers.
> > 
> > nft_dict is a kernel module providing the kernel support for the "dict" 
> > expression.  Its source can be found at the link below:
> > 
> > https://github.com/untangle/nft_dict
> > 
> > Additionally, userspace patches are required in order to use the 
> > "dict" expression from within an nft rule. These currently live here:
> > 
> > https://github.com/untangle/mfw_openwrt/blob/openwrt-18.06/libnftnl/patches/999-libnftnl-Add-dict-support.patch 
> > https://github.com/untangle/mfw_openwrt/blob/openwrt-18.06/nftables/patches/999-nftables-Add-dict.patch
> > 
> > These patches, along with the nft_dict kernel module, also add support for an 
> > "id" key to the "ct" match expression. The "ct id" expression simply returns 
> > the conntrack id of the conntrack. This is the same conntrack id you see if 
> > you run 'conntrack -L --output=id'. While not strictly required, the "ct id" 
> > expression is extremely useful as a key expression into a dict table for 
> > matching entries to a particular conntrack.
> 
> > The correct place to implement
> > the "id" ct key is in the existing nft_ct module, but for now its implemented
> > in the nft_dict module.
> 
> I would welcome adding support for this to nft_ct.c directly (i.e., get
> support for a unique per-ct id) once this patch is accepted:
> 
>  https://patchwork.ozlabs.org/patch/1072650/
> 
> I added nf_ct_get_id() to the conntrack core so it can be called from
> nft_ct later on.

This patch looks great.  That would greatly improve the usefulness of keying
off of the conntrack id.  If it gets accepted I can send a kernel patch and
an nft patch to support the ct id key.
> 
> > For a more in depth description of how things work I suggest reading the doc 
> > in the first link I posted, but below are a few simple examples of what is 
> > possible using dict expressions:
> > 
> > nft add rule ip filter forward dict sessions ct id application long_string 
> > NETFLIX reject 
> > 
> > If I were to describe that rule in plain English it would be:
> > 
> > For traffic passing through the ip filter table forward hook, use the 
> > conntrack id as a key to lookup an entry in the sessions table.  For that
> > entry check if it has an application field set to a string value of NETFLIX,
> > if so, reject the traffic.
> > 
> > How did that field get set to NETFLIX?  That is up to some other entity.  In 
> 
> Why isn't it possible to use the existing nftables set infrastructure
> for this?
> 
> The nft sets store arbitrary octets/bytes as keys, so we could at least
> from kernel side store arbitrary identifiers (strings, integers etc).
> 

I wanted to use the existing set infrastructure, but I ran into two issues
wrt to what we were trying to acheive:

1.  As far as I can tell the current sets are only set up to hold elements
of a particular data type, where as we are storing elements of many different
types in a particular dictionary.

2.  sets are associated with a particular table and therefore can only be
accessed by rules in that table.  If you want to associate some piece of
information with a particular connection and use it from rules in multiple
different tables, you'd have to populate it in a set in each of those tables.
While thats certainly doable, the dictionaries we create are "global", that
is, rules from any table can access them and entries can be added and removed
from a single place.

> > nft add rule ip filter forward tcp dport 80 dict hosts ip saddr \
> > captive-portal-authenticated bool false dnat to 127.0.0.1:80
> 
> nft supports something like
> 
> nft add rule ip filter forward tcp dport 80 ip saddr != @captive_authenticated ...
> 
> and allows external entity to push ip addresses to the set
> @captive_authenticated.

For sure, sets can do that.  That maybe isn't the best example of the dict's
usefulness. :)
> 
> I'll have a look at doc and code later today.

Cool.  I really appreciate you taking the time.

Re: dict: A netfilter expression for dictionary lookups

[Florian Westphal]

Brett Mastbergen <bmastbergen@untangle.com> wrote:
> This patch looks great.  That would greatly improve the usefulness of keying
> off of the conntrack id.  If it gets accepted I can send a kernel patch and
> an nft patch to support the ct id key.

That would be really nice to have.

> > > For a more in depth description of how things work I suggest reading the doc 
> > > in the first link I posted, but below are a few simple examples of what is 
> > > possible using dict expressions:
> > > 
> > > nft add rule ip filter forward dict sessions ct id application long_string 
> > > NETFLIX reject 
> > > 
> > > If I were to describe that rule in plain English it would be:
> > > 
> > > For traffic passing through the ip filter table forward hook, use the 
> > > conntrack id as a key to lookup an entry in the sessions table.  For that
> > > entry check if it has an application field set to a string value of NETFLIX,
> > > if so, reject the traffic.
> > > 
> > > How did that field get set to NETFLIX?  That is up to some other entity.  In 
> > 
> > Why isn't it possible to use the existing nftables set infrastructure
> > for this?
> > 
> > The nft sets store arbitrary octets/bytes as keys, so we could at least
> > from kernel side store arbitrary identifiers (strings, integers etc).
> > 
> 
> I wanted to use the existing set infrastructure, but I ran into two issues
> wrt to what we were trying to acheive:
> 
> 1.  As far as I can tell the current sets are only set up to hold elements
> of a particular data type, where as we are storing elements of many different
> types in a particular dictionary.

Yes, a set is only one data type (They support combining types though).

> 2.  sets are associated with a particular table and therefore can only be
> accessed by rules in that table.  If you want to associate some piece of
> information with a particular connection and use it from rules in multiple
> different tables, you'd have to populate it in a set in each of those tables.

Yes, but thats intentional, a table forms a namespace; so
crossing it is not desireable.

The idea is that different entities each can manage their own tables
without clashing with chain or set names of another table.

Still looking at dicts code, I am trying to understand where normal nft
set infra can't be used (or is too cumbersome) to best understand how we
can fit dicts ideas into nftables' architecture.

Re: dict: A netfilter expression for dictionary lookups

[Brett Mastbergen]

On 16-04-19, Florian Westphal wrote:
> Brett Mastbergen <bmastbergen@untangle.com> wrote:
> > This patch looks great.  That would greatly improve the usefulness of keying
> > off of the conntrack id.  If it gets accepted I can send a kernel patch and
> > an nft patch to support the ct id key.
> 
> That would be really nice to have.
> 
> > > > For a more in depth description of how things work I suggest reading the doc 
> > > > in the first link I posted, but below are a few simple examples of what is 
> > > > possible using dict expressions:
> > > > 
> > > > nft add rule ip filter forward dict sessions ct id application long_string 
> > > > NETFLIX reject 
> > > > 
> > > > If I were to describe that rule in plain English it would be:
> > > > 
> > > > For traffic passing through the ip filter table forward hook, use the 
> > > > conntrack id as a key to lookup an entry in the sessions table.  For that
> > > > entry check if it has an application field set to a string value of NETFLIX,
> > > > if so, reject the traffic.
> > > > 
> > > > How did that field get set to NETFLIX?  That is up to some other entity.  In 
> > > 
> > > Why isn't it possible to use the existing nftables set infrastructure
> > > for this?
> > > 
> > > The nft sets store arbitrary octets/bytes as keys, so we could at least
> > > from kernel side store arbitrary identifiers (strings, integers etc).
> > > 
> > 
> > I wanted to use the existing set infrastructure, but I ran into two issues
> > wrt to what we were trying to acheive:
> > 
> > 1.  As far as I can tell the current sets are only set up to hold elements
> > of a particular data type, where as we are storing elements of many different
> > types in a particular dictionary.
> 
> Yes, a set is only one data type (They support combining types though).
> 
> > 2.  sets are associated with a particular table and therefore can only be
> > accessed by rules in that table.  If you want to associate some piece of
> > information with a particular connection and use it from rules in multiple
> > different tables, you'd have to populate it in a set in each of those tables.
> 
> Yes, but thats intentional, a table forms a namespace; so
> crossing it is not desireable.
> 
> The idea is that different entities each can manage their own tables
> without clashing with chain or set names of another table.
> 
> Still looking at dicts code, I am trying to understand where normal nft
> set infra can't be used (or is too cumbersome) to best understand how we
> can fit dicts ideas into nftables' architecture.

The idea of some entity managing a table and its sets,rules,maps,etc in its
own separate namespace makes total sense.  Thats one of the cool things about
nftables.  I think what inspired us to create data stores that are more
global, that can be accessed/updated from any table, is conntrack data.  If
you think about things like the conntrack state, mark, bytes, status, etc,
that is meta data about a connection that can be accessed from any table as
long as the conntrack has been created.  What we call the sessions table is
really just more meta data about the connection that is keyed off of the
conntrack id that can be accessed/updated from any table.  In our system we
have a single entity that is responsible for populating sessions data, which
then allows the entities managing tables to use it, or not.  Just the same
way any table may or may not use conntrack data.

As Dirk and I were discussing this yesterday we thought that maybe a
"global" map would be close to what we are doing.  Something like below:

Create a global map from conntrack id to an application string (obviously
these data types and syntax are made up)

nft add map global application_map { type ct_id: app_string ; }

and then some table could have a rule like this:

nft add rule inet foo bar ct id map @application_map "netflix" drop

And then something a bit more complex: map connections to users and users
to quota status:

nft add map global user_map { type ct_id: username ; }
nft add map global over_quota { type username: bool ; }
nft add rule inet foo bar ct id map @user_map map @over_quota true drop

Re: dict: A netfilter expression for dictionary lookups

[Jozsef Kadlecsik]

Hi,

On Wed, 17 Apr 2019, Brett Mastbergen wrote:

> On 16-04-19, Florian Westphal wrote:
> > Brett Mastbergen <bmastbergen@untangle.com> wrote:
> > > This patch looks great.  That would greatly improve the usefulness of keying
> > > off of the conntrack id.  If it gets accepted I can send a kernel patch and
> > > an nft patch to support the ct id key.
> > 
> > That would be really nice to have.
> > 
> > > > > For a more in depth description of how things work I suggest reading the doc 
> > > > > in the first link I posted, but below are a few simple examples of what is 
> > > > > possible using dict expressions:
> > > > > 
> > > > > nft add rule ip filter forward dict sessions ct id application long_string 
> > > > > NETFLIX reject 
> > > > > 
> > > > > If I were to describe that rule in plain English it would be:
> > > > > 
> > > > > For traffic passing through the ip filter table forward hook, use the 
> > > > > conntrack id as a key to lookup an entry in the sessions table.  For that
> > > > > entry check if it has an application field set to a string value of NETFLIX,
> > > > > if so, reject the traffic.
> > > > > 
> > > > > How did that field get set to NETFLIX?  That is up to some other entity.  In 
> > > > 
> > > > Why isn't it possible to use the existing nftables set infrastructure
> > > > for this?
> > > > 
> > > > The nft sets store arbitrary octets/bytes as keys, so we could at least
> > > > from kernel side store arbitrary identifiers (strings, integers etc).
> > > > 
> > > 
> > > I wanted to use the existing set infrastructure, but I ran into two issues
> > > wrt to what we were trying to acheive:
> > > 
> > > 1.  As far as I can tell the current sets are only set up to hold elements
> > > of a particular data type, where as we are storing elements of many different
> > > types in a particular dictionary.
> > 
> > Yes, a set is only one data type (They support combining types though).
> > 
> > > 2.  sets are associated with a particular table and therefore can only be
> > > accessed by rules in that table.  If you want to associate some piece of
> > > information with a particular connection and use it from rules in multiple
> > > different tables, you'd have to populate it in a set in each of those tables.
> > 
> > Yes, but thats intentional, a table forms a namespace; so
> > crossing it is not desireable.
> > 
> > The idea is that different entities each can manage their own tables
> > without clashing with chain or set names of another table.
> > 
> > Still looking at dicts code, I am trying to understand where normal nft
> > set infra can't be used (or is too cumbersome) to best understand how we
> > can fit dicts ideas into nftables' architecture.
> 
> The idea of some entity managing a table and its sets,rules,maps,etc in 
> its own separate namespace makes total sense.  Thats one of the cool 
> things about nftables.  I think what inspired us to create data stores 
> that are more global, that can be accessed/updated from any table, is 
> conntrack data.  If you think about things like the conntrack state, 
> mark, bytes, status, etc, that is meta data about a connection that can 
> be accessed from any table as long as the conntrack has been created.  
> What we call the sessions table is really just more meta data about the 
> connection that is keyed off of the conntrack id that can be 
> accessed/updated from any table.  In our system we have a single entity 
> that is responsible for populating sessions data, which then allows the 
> entities managing tables to use it, or not.  Just the same way any table 
> may or may not use conntrack data.

Yes, it's a valid argument. And you presented the two most natural 
examples: data connected to conntrack entries and quotas (maintained in 
the "filter" table and when the subject is over its quota, drop as early 
as possible, in the "raw" table).
 
> As Dirk and I were discussing this yesterday we thought that maybe a 
> "global" map would be close to what we are doing.  Something like below:

I think a "global" namespace could be the solution: sets and maps defined 
in it could be used from any table (i.e. namespace). That is the contents 
can be matched/updated/added/deleted from anywhere, but such sets/maps can 
be defined/destroyed in "global" only.

nftables dictionaries are tightly coupled with their namespace because of 
the verdict/jump part, so those would not be allowed in this global 
namespace.

What do you think, is that a good generalization and could properly cover 
your cases?

> Create a global map from conntrack id to an application string (obviously
> these data types and syntax are made up)
> 
> nft add map global application_map { type ct_id: app_string ; }
> 
> and then some table could have a rule like this:
> 
> nft add rule inet foo bar ct id map @application_map "netflix" drop
> 
> And then something a bit more complex: map connections to users and users
> to quota status:
> 
> nft add map global user_map { type ct_id: username ; }
> nft add map global over_quota { type username: bool ; }
> nft add rule inet foo bar ct id map @user_map map @over_quota true drop

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary

Re: dict: A netfilter expression for dictionary lookups

[Brett Mastbergen]

On 18-04-19, Jozsef Kadlecsik wrote:
> Hi,
> 
> On Wed, 17 Apr 2019, Brett Mastbergen wrote:
> 
> > On 16-04-19, Florian Westphal wrote:
> > > Brett Mastbergen <bmastbergen@untangle.com> wrote:
> > > > This patch looks great.  That would greatly improve the usefulness of keying
> > > > off of the conntrack id.  If it gets accepted I can send a kernel patch and
> > > > an nft patch to support the ct id key.
> > > 
> > > That would be really nice to have.
> > > 
> > > > > > For a more in depth description of how things work I suggest reading the doc 
> > > > > > in the first link I posted, but below are a few simple examples of what is 
> > > > > > possible using dict expressions:
> > > > > > 
> > > > > > nft add rule ip filter forward dict sessions ct id application long_string 
> > > > > > NETFLIX reject 
> > > > > > 
> > > > > > If I were to describe that rule in plain English it would be:
> > > > > > 
> > > > > > For traffic passing through the ip filter table forward hook, use the 
> > > > > > conntrack id as a key to lookup an entry in the sessions table.  For that
> > > > > > entry check if it has an application field set to a string value of NETFLIX,
> > > > > > if so, reject the traffic.
> > > > > > 
> > > > > > How did that field get set to NETFLIX?  That is up to some other entity.  In 
> > > > > 
> > > > > Why isn't it possible to use the existing nftables set infrastructure
> > > > > for this?
> > > > > 
> > > > > The nft sets store arbitrary octets/bytes as keys, so we could at least
> > > > > from kernel side store arbitrary identifiers (strings, integers etc).
> > > > > 
> > > > 
> > > > I wanted to use the existing set infrastructure, but I ran into two issues
> > > > wrt to what we were trying to acheive:
> > > > 
> > > > 1.  As far as I can tell the current sets are only set up to hold elements
> > > > of a particular data type, where as we are storing elements of many different
> > > > types in a particular dictionary.
> > > 
> > > Yes, a set is only one data type (They support combining types though).
> > > 
> > > > 2.  sets are associated with a particular table and therefore can only be
> > > > accessed by rules in that table.  If you want to associate some piece of
> > > > information with a particular connection and use it from rules in multiple
> > > > different tables, you'd have to populate it in a set in each of those tables.
> > > 
> > > Yes, but thats intentional, a table forms a namespace; so
> > > crossing it is not desireable.
> > > 
> > > The idea is that different entities each can manage their own tables
> > > without clashing with chain or set names of another table.
> > > 
> > > Still looking at dicts code, I am trying to understand where normal nft
> > > set infra can't be used (or is too cumbersome) to best understand how we
> > > can fit dicts ideas into nftables' architecture.
> > 
> > The idea of some entity managing a table and its sets,rules,maps,etc in 
> > its own separate namespace makes total sense.  Thats one of the cool 
> > things about nftables.  I think what inspired us to create data stores 
> > that are more global, that can be accessed/updated from any table, is 
> > conntrack data.  If you think about things like the conntrack state, 
> > mark, bytes, status, etc, that is meta data about a connection that can 
> > be accessed from any table as long as the conntrack has been created.  
> > What we call the sessions table is really just more meta data about the 
> > connection that is keyed off of the conntrack id that can be 
> > accessed/updated from any table.  In our system we have a single entity 
> > that is responsible for populating sessions data, which then allows the 
> > entities managing tables to use it, or not.  Just the same way any table 
> > may or may not use conntrack data.
> 
> Yes, it's a valid argument. And you presented the two most natural 
> examples: data connected to conntrack entries and quotas (maintained in 
> the "filter" table and when the subject is over its quota, drop as early 
> as possible, in the "raw" table).
>  
> > As Dirk and I were discussing this yesterday we thought that maybe a 
> > "global" map would be close to what we are doing.  Something like below:
> 
> I think a "global" namespace could be the solution: sets and maps defined 
> in it could be used from any table (i.e. namespace). That is the contents 
> can be matched/updated/added/deleted from anywhere, but such sets/maps can 
> be defined/destroyed in "global" only.
> 
> nftables dictionaries are tightly coupled with their namespace because of 
> the verdict/jump part, so those would not be allowed in this global 
> namespace.
> 
> What do you think, is that a good generalization and could properly cover 
> your cases?
> 

Yes, I think your description of a global namespace that sets and maps can
be defined in sounds right on and would cover our use cases as far as where
the data resides (globally) and where it is accessible from (any table).

There are a couple other considerations I'd like to throw out there:

1.  Today it seems that nftables assumes that the map expressions will only
be used as the right hand expression of things like dnat and snat rules.

# from the wiki
nft add rule ip nat postrouting snat tcp dport map @porttoip

But we'd really like to be able to use map lookups on the left side of
comparisons so we can look up the additional meta data we put in a global
map and make some decision based on whether it matches:

nft add rule inet foo bar ct id map @application_map "netflix" drop

I've just about got that working locally.  It seems there isn't any
architectural reason that isn't possible today, I think it was just coded
up with the assumption of being a right side expression all the time.

2.  If you take a peek at our nft_dict documentation you'll notice we have
defined some new generic data types: int_type, int64_type, long_string, and
bool (I think upstream has a boolean type now though).  Since we are storing
lots of artibrary meta data in our tables we didn't want to have to define
a new data type every time we decided to store some new piece of data in a
table.  So instead of defining a new fixed length string subtype for
application type, we'd just define it as a long_string.  And instead of
defining a new fixed length integer subtype for reputation, we'd just
define it as an int_type.  Obviously, there are times when you do want
to define a new type if only certain values are valid, but we've found the
flexibility of these generic types to be really handy.
> > Create a global map from conntrack id to an application string (obviously
> > these data types and syntax are made up)
> > 
> > nft add map global application_map { type ct_id: app_string ; }
> > 
> > and then some table could have a rule like this:
> > 
> > nft add rule inet foo bar ct id map @application_map "netflix" drop
> > 
> > And then something a bit more complex: map connections to users and users
> > to quota status:
> > 
> > nft add map global user_map { type ct_id: username ; }
> > nft add map global over_quota { type username: bool ; }
> > nft add rule inet foo bar ct id map @user_map map @over_quota true drop
> 
> Best regards,
> Jozsef
> -
> E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
> PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
> Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
>           H-1525 Budapest 114, POB. 49, Hungary

Re: dict: A netfilter expression for dictionary lookups

[Brett Mastbergen]

On 16-04-19, Florian Westphal wrote:
> Brett Mastbergen <bmastbergen@untangle.com> wrote:
> > This patch looks great.  That would greatly improve the usefulness of keying
> > off of the conntrack id.  If it gets accepted I can send a kernel patch and
> > an nft patch to support the ct id key.
> 
> That would be really nice to have.
> 

I've just sent the kernel, libnftnl, and nft patches that add support for the
ct id key to the list.  Take a look, see what you think.

> > > > For a more in depth description of how things work I suggest reading the doc 
> > > > in the first link I posted, but below are a few simple examples of what is 
> > > > possible using dict expressions:
> > > > 
> > > > nft add rule ip filter forward dict sessions ct id application long_string 
> > > > NETFLIX reject 
> > > > 
> > > > If I were to describe that rule in plain English it would be:
> > > > 
> > > > For traffic passing through the ip filter table forward hook, use the 
> > > > conntrack id as a key to lookup an entry in the sessions table.  For that
> > > > entry check if it has an application field set to a string value of NETFLIX,
> > > > if so, reject the traffic.
> > > > 
> > > > How did that field get set to NETFLIX?  That is up to some other entity.  In 
> > > 
> > > Why isn't it possible to use the existing nftables set infrastructure
> > > for this?
> > > 
> > > The nft sets store arbitrary octets/bytes as keys, so we could at least
> > > from kernel side store arbitrary identifiers (strings, integers etc).
> > > 
> > 
> > I wanted to use the existing set infrastructure, but I ran into two issues
> > wrt to what we were trying to acheive:
> > 
> > 1.  As far as I can tell the current sets are only set up to hold elements
> > of a particular data type, where as we are storing elements of many different
> > types in a particular dictionary.
> 
> Yes, a set is only one data type (They support combining types though).
> 
> > 2.  sets are associated with a particular table and therefore can only be
> > accessed by rules in that table.  If you want to associate some piece of
> > information with a particular connection and use it from rules in multiple
> > different tables, you'd have to populate it in a set in each of those tables.
> 
> Yes, but thats intentional, a table forms a namespace; so
> crossing it is not desireable.
> 
> The idea is that different entities each can manage their own tables
> without clashing with chain or set names of another table.
> 
> Still looking at dicts code, I am trying to understand where normal nft
> set infra can't be used (or is too cumbersome) to best understand how we
> can fit dicts ideas into nftables' architecture.