Using iptables and ipset to DROP a list of 2 million addresses

Using iptables and ipset to DROP a list of 2 million addresses

[Mason Kaufer]

Hi,
I am currently trying to set up a firewall on an Ubuntu 22.04 machine
that will block a list of 2 million plus ip addresses without slowing
the network speed down tremendously. I have tried using ipset but I
get an error that the hash size isn't large enough. I have tried
manually setting the hash size but it only allows that option to be so
large. Is there something I am doing wrong or is there a better way to
achieve this? Any help with this would be much appreciated.
--
Mason Kaufer

Re: Using iptables and ipset to DROP a list of 2 million addresses

[Le Chevalier]

On 2024-04-11 14:39, Mason Kaufer wrote:
> Hi,
> I am currently trying to set up a firewall on an Ubuntu 22.04 machine
> that will block a list of 2 million plus ip addresses without slowing
> the network speed down tremendously. I have tried using ipset but I
> get an error that the hash size isn't large enough. I have tried
> manually setting the hash size but it only allows that option to be so
> large. Is there something I am doing wrong or is there a better way to
> achieve this? Any help with this would be much appreciated.
> --
> Mason Kaufer
>
Look at the 'list:set' feature. I have not tested this myself, but from 
the description it may act as a compound list.

https://ipset.netfilter.org/features.html


~Forza

Re: Using iptables and ipset to DROP a list of 2 million addresses

[Stephen Satchell]

On 4/11/24 5:39 AM, Mason Kaufer wrote:
> Hi,
> I am currently trying to set up a firewall on an Ubuntu 22.04 machine
> that will block a list of 2 million plus ip addresses without slowing
> the network speed down tremendously. I have tried using ipset but I
> get an error that the hash size isn't large enough. I have tried
> manually setting the hash size but it only allows that option to be so
> large. Is there something I am doing wrong or is there a better way to
> achieve this? Any help with this would be much appreciated.

There could be another way, that would work for both inbound and 
outbound blocking.  Consolidate the list into net ranges, and add the 
ranges as BLACK HOLE routes in the routing table.

I'm less and less enamored with using IP lists in a firewall when the 
routing table is optimized to handle BIG lists of addresses, if you do 
the consolidation properly.

Also, be sure to have short-circuit rules for established connections in 
your firewall list, if you do decide to lard up an IP table list.  That 
will help the speed problem, by restricting the loading to new connections.

Re: Using iptables and ipset to DROP a list of 2 million addresses

[Jozsef Kadlecsik]

On Thu, 11 Apr 2024, Le Chevalier wrote:

> On 2024-04-11 14:39, Mason Kaufer wrote:
> > I am currently trying to set up a firewall on an Ubuntu 22.04 machine 
> > that will block a list of 2 million plus ip addresses without slowing 
> > the network speed down tremendously. I have tried using ipset but I 
> > get an error that the hash size isn't large enough. I have tried 
> > manually setting the hash size but it only allows that option to be so 
> > large. Is there something I am doing wrong or is there a better way to 
> > achieve this? Any help with this would be much appreciated.

There's no upper limit in the hash size (except that the number must fit 
into u32). On my laptop:

# ipset n test hash:ip hashsize 10000000 maxelem 10000000
# ipset l
Name: test
Type: hash:ip
Revision: 5
Header: family inet hashsize 16777216 maxelem 10000000 bucketsize 12 initval 0xc61d4797
Size in memory: 393392
References: 0
Number of entries: 0
Members:

Please note, you must tune both hashsize and maxelem parameters in order 
to be able to store the given number of entries.

Best regards,
Jozsef
-- 
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.hu
PGP key : https://wigner.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics
          H-1525 Budapest 114, POB. 49, Hungary