ipset restore for bitmap:port terrible slow

ipset restore for bitmap:port terrible slow

[Reindl Harald]

Hi

the restore of a "bitmap:port" ipset with a lot of entries is *terrible*
slow, when you add a port-range like 42000–42999 it ends in 999 "add
PORTS_RESTRICTED" lines in the save-file and restore takes virtually ages

the cpu-time below is the whole systemd-unit which restores iptables,
ipset and configures the network with 3 nics, a bridge and wireguard

why is this *that much* inefficient given that the original command with
port ranges returns instantly?

on a datacenter firewall that makes the difference of 5 seconds or 15
seconds downtime at reboot

---------------------------

Name: PORTS_RESTRICTED
Type: bitmap:port
Header: range 1-55000

---------------------------

/usr/sbin/ipset -file /etc/sysconfig/ipset restore

CPU: 9.594s - Number of entries: 5192
CPU: 6.246s - Number of entries: 3192
CPU: 1.511s - Number of entries: 53

---------------------------

42000–42999 looks in /etc/sysconfig/ipset like below and frankly either
that can be speeded up or should be saved as ranges wherever it's
possible like hash:net prefers cidr

add PORTS_RESTRICTED 42000
add PORTS_RESTRICTED 42001
add PORTS_RESTRICTED 42002
add PORTS_RESTRICTED 42003
add PORTS_RESTRICTED 42004
add PORTS_RESTRICTED 42005
add PORTS_RESTRICTED 42006
add PORTS_RESTRICTED 42007
add PORTS_RESTRICTED 42008
add PORTS_RESTRICTED 42009
add PORTS_RESTRICTED 42010
add PORTS_RESTRICTED 42011
add PORTS_RESTRICTED 42012
add PORTS_RESTRICTED 42013
add PORTS_RESTRICTED 42014
add PORTS_RESTRICTED 42015
add PORTS_RESTRICTED 42016
add PORTS_RESTRICTED 42017
add PORTS_RESTRICTED 42018
add PORTS_RESTRICTED 42019
add PORTS_RESTRICTED 42020
add PORTS_RESTRICTED 42021
add PORTS_RESTRICTED 42022

---------------------------

Re: ipset restore for bitmap:port terrible slow

[Jozsef Kadlecsik]

[-- Attachment #1: Type: text/plain, Size: 1700 bytes --]

Hi,

On Wed, 17 Jun 2020, Reindl Harald wrote:

> the restore of a "bitmap:port" ipset with a lot of entries is *terrible* 
> slow, when you add a port-range like 42000–42999 it ends in 999 "add 
> PORTS_RESTRICTED" lines in the save-file and restore takes virtually 
> ages
> 
> the cpu-time below is the whole systemd-unit which restores iptables, 
> ipset and configures the network with 3 nics, a bridge and wireguard
> 
> why is this *that much* inefficient given that the original command with
> port ranges returns instantly?
> 
> on a datacenter firewall that makes the difference of 5 seconds or 15
> seconds downtime at reboot
> ---------------------------
> 
> Name: PORTS_RESTRICTED
> Type: bitmap:port
> Header: range 1-55000
> 
> ---------------------------
> 
> /usr/sbin/ipset -file /etc/sysconfig/ipset restore
> 
> CPU: 9.594s - Number of entries: 5192
> CPU: 6.246s - Number of entries: 3192
> CPU: 1.511s - Number of entries: 53
> 
> ---------------------------

I cannot reproduce the issue. What is your ipset version (both userspace 
tool and kernel modules)?
 
> 42000–42999 looks in /etc/sysconfig/ipset like below and frankly either
> that can be speeded up or should be saved as ranges wherever it's
> possible like hash:net prefers cidr

The bitmap port type does not support ranges, just individual port 
elements. 

In my test restoring a set with 10000 elements took less than 1s.

Best regars,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.hu
PGP key : https://wigner.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: ipset restore for bitmap:port terrible slow

[Reindl Harald]

Am 17.06.20 um 21:06 schrieb Jozsef Kadlecsik:
> On Wed, 17 Jun 2020, Reindl Harald wrote:
> 
>> the restore of a "bitmap:port" ipset with a lot of entries is *terrible* 
>> slow, when you add a port-range like 42000–42999 it ends in 999 "add 
>> PORTS_RESTRICTED" lines in the save-file and restore takes virtually 
>> ages
>>
>> the cpu-time below is the whole systemd-unit which restores iptables, 
>> ipset and configures the network with 3 nics, a bridge and wireguard
>>
>> why is this *that much* inefficient given that the original command with
>> port ranges returns instantly?
>>
>> on a datacenter firewall that makes the difference of 5 seconds or 15
>> seconds downtime at reboot
>> ---------------------------
>>
>> Name: PORTS_RESTRICTED
>> Type: bitmap:port
>> Header: range 1-55000
>>
>> ---------------------------
>>
>> /usr/sbin/ipset -file /etc/sysconfig/ipset restore
>>
>> CPU: 9.594s - Number of entries: 5192
>> CPU: 6.246s - Number of entries: 3192
>> CPU: 1.511s - Number of entries: 53
>>
>> ---------------------------
> 
> I cannot reproduce the issue. What is your ipset version (both userspace 
> tool and kernel modules)?

5.7.0-1.fc33.x86_64
ipset-7.5-1.fc31.x86_64

it's practically the same with 5.6.18 and i found it out by luck what
ruined my boot times that much because i saved the one with 5192 empty

>> 42000–42999 looks in /etc/sysconfig/ipset like below and frankly either
>> that can be speeded up or should be saved as ranges wherever it's
>> possible like hash:net prefers cidr
> 
> The bitmap port type does not support ranges, just individual port 
> elements. 

but it does support when you want to add 42000–42999 on the cli and that
works way faster than restore at reboot

if one could restore just the ipset definitions so that you can load the
iptables ruleset and after that load the values would improve the
situation dramatically given that 99% of the ruleset works with empty
ipsets good enough for some seconds

> In my test restoring a set with 10000 elements took less than 1s

well, that's a nested VMware ESXi within Vmware Workstation where many
things are more expensive

but 9 seconds on a single vcore to restore ipset, iptables, create a
brdige and assign two interfaces to it with a few ethtool commands on
the production vm with Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz is still
terrible given that the machine don't do anything else

i remember times where i lost 4 ping packets to the machines behind the
firewall but than i added a lot of ports to some ipsets

[root@firewall:~]$  systemd-analyze blame
9.601s network-up.service