Re: dict: A netfilter expression for dictionary lookups

[Jozsef Kadlecsik <kadlec@blackhole.kfki.hu>]

Hi,

On Wed, 17 Apr 2019, Brett Mastbergen wrote:

> On 16-04-19, Florian Westphal wrote:
> > Brett Mastbergen <bmastbergen@untangle.com> wrote:
> > > This patch looks great.  That would greatly improve the usefulness of keying
> > > off of the conntrack id.  If it gets accepted I can send a kernel patch and
> > > an nft patch to support the ct id key.
> > 
> > That would be really nice to have.
> > 
> > > > > For a more in depth description of how things work I suggest reading the doc 
> > > > > in the first link I posted, but below are a few simple examples of what is 
> > > > > possible using dict expressions:
> > > > > 
> > > > > nft add rule ip filter forward dict sessions ct id application long_string 
> > > > > NETFLIX reject 
> > > > > 
> > > > > If I were to describe that rule in plain English it would be:
> > > > > 
> > > > > For traffic passing through the ip filter table forward hook, use the 
> > > > > conntrack id as a key to lookup an entry in the sessions table.  For that
> > > > > entry check if it has an application field set to a string value of NETFLIX,
> > > > > if so, reject the traffic.
> > > > > 
> > > > > How did that field get set to NETFLIX?  That is up to some other entity.  In 
> > > > 
> > > > Why isn't it possible to use the existing nftables set infrastructure
> > > > for this?
> > > > 
> > > > The nft sets store arbitrary octets/bytes as keys, so we could at least
> > > > from kernel side store arbitrary identifiers (strings, integers etc).
> > > > 
> > > 
> > > I wanted to use the existing set infrastructure, but I ran into two issues
> > > wrt to what we were trying to acheive:
> > > 
> > > 1.  As far as I can tell the current sets are only set up to hold elements
> > > of a particular data type, where as we are storing elements of many different
> > > types in a particular dictionary.
> > 
> > Yes, a set is only one data type (They support combining types though).
> > 
> > > 2.  sets are associated with a particular table and therefore can only be
> > > accessed by rules in that table.  If you want to associate some piece of
> > > information with a particular connection and use it from rules in multiple
> > > different tables, you'd have to populate it in a set in each of those tables.
> > 
> > Yes, but thats intentional, a table forms a namespace; so
> > crossing it is not desireable.
> > 
> > The idea is that different entities each can manage their own tables
> > without clashing with chain or set names of another table.
> > 
> > Still looking at dicts code, I am trying to understand where normal nft
> > set infra can't be used (or is too cumbersome) to best understand how we
> > can fit dicts ideas into nftables' architecture.
> 
> The idea of some entity managing a table and its sets,rules,maps,etc in 
> its own separate namespace makes total sense.  Thats one of the cool 
> things about nftables.  I think what inspired us to create data stores 
> that are more global, that can be accessed/updated from any table, is 
> conntrack data.  If you think about things like the conntrack state, 
> mark, bytes, status, etc, that is meta data about a connection that can 
> be accessed from any table as long as the conntrack has been created.  
> What we call the sessions table is really just more meta data about the 
> connection that is keyed off of the conntrack id that can be 
> accessed/updated from any table.  In our system we have a single entity 
> that is responsible for populating sessions data, which then allows the 
> entities managing tables to use it, or not.  Just the same way any table 
> may or may not use conntrack data.

Yes, it's a valid argument. And you presented the two most natural 
examples: data connected to conntrack entries and quotas (maintained in 
the "filter" table and when the subject is over its quota, drop as early 
as possible, in the "raw" table).
 
> As Dirk and I were discussing this yesterday we thought that maybe a 
> "global" map would be close to what we are doing.  Something like below:

I think a "global" namespace could be the solution: sets and maps defined 
in it could be used from any table (i.e. namespace). That is the contents 
can be matched/updated/added/deleted from anywhere, but such sets/maps can 
be defined/destroyed in "global" only.

nftables dictionaries are tightly coupled with their namespace because of 
the verdict/jump part, so those would not be allowed in this global 
namespace.

What do you think, is that a good generalization and could properly cover 
your cases?

> Create a global map from conntrack id to an application string (obviously
> these data types and syntax are made up)
> 
> nft add map global application_map { type ct_id: app_string ; }
> 
> and then some table could have a rule like this:
> 
> nft add rule inet foo bar ct id map @application_map "netflix" drop
> 
> And then something a bit more complex: map connections to users and users
> to quota status:
> 
> nft add map global user_map { type ct_id: username ; }
> nft add map global over_quota { type username: bool ; }
> nft add rule inet foo bar ct id map @user_map map @over_quota true drop

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary