Re: dict: A netfilter expression for dictionary lookups

[Brett Mastbergen <bmastbergen@untangle.com>]

On 16-04-19, Florian Westphal wrote:
> Brett Mastbergen <bmastbergen@untangle.com> wrote:
> > This patch looks great.  That would greatly improve the usefulness of keying
> > off of the conntrack id.  If it gets accepted I can send a kernel patch and
> > an nft patch to support the ct id key.
> 
> That would be really nice to have.
> 
> > > > For a more in depth description of how things work I suggest reading the doc 
> > > > in the first link I posted, but below are a few simple examples of what is 
> > > > possible using dict expressions:
> > > > 
> > > > nft add rule ip filter forward dict sessions ct id application long_string 
> > > > NETFLIX reject 
> > > > 
> > > > If I were to describe that rule in plain English it would be:
> > > > 
> > > > For traffic passing through the ip filter table forward hook, use the 
> > > > conntrack id as a key to lookup an entry in the sessions table.  For that
> > > > entry check if it has an application field set to a string value of NETFLIX,
> > > > if so, reject the traffic.
> > > > 
> > > > How did that field get set to NETFLIX?  That is up to some other entity.  In 
> > > 
> > > Why isn't it possible to use the existing nftables set infrastructure
> > > for this?
> > > 
> > > The nft sets store arbitrary octets/bytes as keys, so we could at least
> > > from kernel side store arbitrary identifiers (strings, integers etc).
> > > 
> > 
> > I wanted to use the existing set infrastructure, but I ran into two issues
> > wrt to what we were trying to acheive:
> > 
> > 1.  As far as I can tell the current sets are only set up to hold elements
> > of a particular data type, where as we are storing elements of many different
> > types in a particular dictionary.
> 
> Yes, a set is only one data type (They support combining types though).
> 
> > 2.  sets are associated with a particular table and therefore can only be
> > accessed by rules in that table.  If you want to associate some piece of
> > information with a particular connection and use it from rules in multiple
> > different tables, you'd have to populate it in a set in each of those tables.
> 
> Yes, but thats intentional, a table forms a namespace; so
> crossing it is not desireable.
> 
> The idea is that different entities each can manage their own tables
> without clashing with chain or set names of another table.
> 
> Still looking at dicts code, I am trying to understand where normal nft
> set infra can't be used (or is too cumbersome) to best understand how we
> can fit dicts ideas into nftables' architecture.

The idea of some entity managing a table and its sets,rules,maps,etc in its
own separate namespace makes total sense.  Thats one of the cool things about
nftables.  I think what inspired us to create data stores that are more
global, that can be accessed/updated from any table, is conntrack data.  If
you think about things like the conntrack state, mark, bytes, status, etc,
that is meta data about a connection that can be accessed from any table as
long as the conntrack has been created.  What we call the sessions table is
really just more meta data about the connection that is keyed off of the
conntrack id that can be accessed/updated from any table.  In our system we
have a single entity that is responsible for populating sessions data, which
then allows the entities managing tables to use it, or not.  Just the same
way any table may or may not use conntrack data.

As Dirk and I were discussing this yesterday we thought that maybe a
"global" map would be close to what we are doing.  Something like below:

Create a global map from conntrack id to an application string (obviously
these data types and syntax are made up)

nft add map global application_map { type ct_id: app_string ; }

and then some table could have a rule like this:

nft add rule inet foo bar ct id map @application_map "netflix" drop

And then something a bit more complex: map connections to users and users
to quota status:

nft add map global user_map { type ct_id: username ; }
nft add map global over_quota { type username: bool ; }
nft add rule inet foo bar ct id map @user_map map @over_quota true drop