Bulk loading of IP addresses or subnets in nftables?

Bulk loading of IP addresses or subnets in nftables?

[Lars Noodén]

In nftables, I'd like to apply a rule or set of rules to a very large
number of IP addresses and subnets and looking for the recommended best
practices for that: Should I make a new chain with one address or subnet
per rule, should I use a dictionary, or should I try using a set, or
does it matter?  The list would be read at boot but not usually updated
much between boots.  Which method would operate most efficiently once
the addresses and subnets are loaded?

/Lars

Re: Bulk loading of IP addresses or subnets in nftables?

[kfm]

On 26/01/2020 04:18, Lars Noodén wrote:
> In nftables, I'd like to apply a rule or set of rules to a very large
> number of IP addresses and subnets and looking for the recommended best
> practices for that: Should I make a new chain with one address or subnet
> per rule, should I use a dictionary, or should I try using a set, or
> does it matter?  The list would be read at boot but not usually updated
> much between boots.  Which method would operate most efficiently once
> the addresses and subnets are loaded?

It matters. Rather than individual rules, you should definitely use a 
set if you can.

Lately, I have been experimenting with methods to populate large sets 
from arbitrary sources. I determined that the fastest way should be to 
execute a single "add element" command that includes all of the 
elements. For newline-delimited input, here is a minimalistic script 
demonstrating one way to go about it:-

#!/bin/sh
srcfile=$1
shift
{
	echo "flush set $*"
	echo "add element $* { "
	tr '\n' ',' < "$srcfile"
	echo " }"
} | nft -f -

Assuming that this script is saved as "populate-set", it could be used 
as follows:-

populate-set mynetworks.txt ip filter mysetname

It works by considering the first parameter as the file to read the list 
of addresses/prefixes from, with the remaining parameters being passed 
to the "flush set" and "add element" commands. The tr command is used to 
convert newline characters to commas, as is required by the syntax of 
the add command. Both commands are executed with just one invocation of 
nft so that the operation is atomic (in theory).

Alas, I discovered that populating a set with a large number of elements 
can cause nft and/or netlink to choke under certain circumstances. To 
put "large" into context, I am testing with the IPv6 bogons list, which 
consists of over 116000 entries. I reported this issue as 
https://bugzilla.netfilter.org/show_bug.cgi?id=1392.

In an attempt to work around this issue, I experimented with executing 
multiple "add element" commands in smaller batches but to no avail. 
Instead, I found that the only defense is to ensure that out-of-band set 
population occurs no more than once after the last "flush ruleset" 
command. Otherwise, the population of the set is initially fast, but is 
excruciatingly slow on the second and subsequent occasions. Flushing the 
entire ruleset seems to obviate the circumstances under which this issue 
occurs. Frankly, this instability makes it a nuisance as compared to 
working with ipsets.

One other thing to keep in mind is that your set declaration will 
require the "interval" flag in order to support (CIDR) prefixes. A lot 
of people get confused by that. See 
https://bugzilla.netfilter.org/show_bug.cgi?id=1380 as an example.

-- 
Kerin Millar

Re: Bulk loading of IP addresses or subnets in nftables?

[Lars Noodén]

On 1/26/20 1:36 PM, kfm wrote:
> On 26/01/2020 04:18, Lars Noodén wrote:
>> In nftables, I'd like to apply a rule or set of rules to a very large
>> number of IP addresses and subne> [snip]
> It matters. Rather than individual rules, you should definitely use a
> set if you can.
> [snip]

Thanks for the details.  That helped and now I have a working filter
based on a set.  It needed "flags interval;" as you said.  Since I fed
it the output from a whois search on an AS number, I also had to use
"auto-merge" to fold overlapping subnets onto each other.

	set foo4 {
                type ipv4_addr
                flags interval
                auto-merge
                elements = { ..., ..., ... }
	}

There were a lot of warnings about overlap as the subnets were fed in
but the result seemed that the minimum had been calculated (I think).

	whois -h whois.radb.net '!gAS.....' \
	| tr ' ' '\n' \
	| grep -E '([0-9]{1,3}\.){3}' \
	| sort -t . -k1,1n -k2,2n -k3,3 -k4,4 \
	| xargs -I% nft element ip filter foo4 { % }

/Lars