* nft set load metrics @ 2021-09-30 11:21 Cristian Constantin 2021-09-30 14:00 ` Pablo Neira Ayuso 0 siblings, 1 reply; 9+ messages in thread From: Cristian Constantin @ 2021-09-30 11:21 UTC (permalink / raw) To: netfilter hi! is there a way to get from the kernel just some metrics about nft sets, without getting all (possibly lots of) elements of the set? e.g.: - number of full ip addresses (i.e. /32 prefix for ipv4); - number of "prefix" elements; - number of intervals; thanks, cristian ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: nft set load metrics 2021-09-30 11:21 nft set load metrics Cristian Constantin @ 2021-09-30 14:00 ` Pablo Neira Ayuso 2021-09-30 15:47 ` Cristian Constantin 0 siblings, 1 reply; 9+ messages in thread From: Pablo Neira Ayuso @ 2021-09-30 14:00 UTC (permalink / raw) To: Cristian Constantin; +Cc: netfilter On Thu, Sep 30, 2021 at 01:21:26PM +0200, Cristian Constantin wrote: > hi! > > is there a way to get from the kernel just some metrics about nft > sets, without getting > all (possibly lots of) elements of the set? > > e.g.: > > - number of full ip addresses (i.e. /32 prefix for ipv4); > - number of "prefix" elements; > - number of intervals; It's possible to extend the interface to expose this, but how useful is this? ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: nft set load metrics 2021-09-30 14:00 ` Pablo Neira Ayuso @ 2021-09-30 15:47 ` Cristian Constantin 2021-09-30 17:34 ` G.W. Haywood 2021-09-30 18:04 ` Pablo Neira Ayuso 0 siblings, 2 replies; 9+ messages in thread From: Cristian Constantin @ 2021-09-30 15:47 UTC (permalink / raw) To: Pablo Neira Ayuso; +Cc: netfilter > It's possible to extend the interface to expose this, but how useful > is this? cristian: imo, it is important from the operational point of view to monitor the size of the sets. from an implementation point of view, since the sets can grow to very large sizes, reading large packets over netlink sockets just to count the elements in the sets does not seem very efficient. the nft user space tool has a switch which turns off showing the set content: '-t', however it will only show the name of the set and the flags. I also did not check if this feature is offered at the socket layer or if the elements are actually read from the kernel but not displayed. thanks, cristian On Thu, Sep 30, 2021 at 4:00 PM Pablo Neira Ayuso <pablo@netfilter.org> wrote: > > On Thu, Sep 30, 2021 at 01:21:26PM +0200, Cristian Constantin wrote: > > hi! > > > > is there a way to get from the kernel just some metrics about nft > > sets, without getting > > all (possibly lots of) elements of the set? > > > > e.g.: > > > > - number of full ip addresses (i.e. /32 prefix for ipv4); > > - number of "prefix" elements; > > - number of intervals; > > It's possible to extend the interface to expose this, but how useful > is this? ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: nft set load metrics 2021-09-30 15:47 ` Cristian Constantin @ 2021-09-30 17:34 ` G.W. Haywood 2021-09-30 18:12 ` Pablo Neira Ayuso 2021-09-30 19:01 ` Cristian Constantin 2021-09-30 18:04 ` Pablo Neira Ayuso 1 sibling, 2 replies; 9+ messages in thread From: G.W. Haywood @ 2021-09-30 17:34 UTC (permalink / raw) To: Cristian Constantin; +Cc: netfilter Hi there, On Thu, 30 Sep 2021, Cristian Constantin wrote: > ... reading large packets over netlink sockets just to count the > elements in the sets does not seem very efficient. Agreed. It seems to me that if you need to read what you've put in the sets for the purposes of some facility, then you need to store it in RAM. It doesn't make sense to me to try to use netfilter as a kind of RAM; as you say that will be very inefficient. Presumably it was you who created the elements, can you not simply store them e.g. in a Perl hash at the time they're created? I do something similar using Net::CIDR::Lite. (I would never claim that a Perl script is the most efficient way of doing it, nor anything else for that matter, but it gets the job done. :) -- 73, Ged. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: nft set load metrics 2021-09-30 17:34 ` G.W. Haywood @ 2021-09-30 18:12 ` Pablo Neira Ayuso 2021-09-30 19:01 ` Cristian Constantin 1 sibling, 0 replies; 9+ messages in thread From: Pablo Neira Ayuso @ 2021-09-30 18:12 UTC (permalink / raw) To: G.W. Haywood; +Cc: Cristian Constantin, netfilter On Thu, Sep 30, 2021 at 06:34:52PM +0100, G.W. Haywood wrote: > Hi there, > > On Thu, 30 Sep 2021, Cristian Constantin wrote: > > > ... reading large packets over netlink sockets just to count the > > elements in the sets does not seem very efficient. > > Agreed. > > It seems to me that if you need to read what you've put in the sets > for the purposes of some facility, then you need to store it in RAM. > > It doesn't make sense to me to try to use netfilter as a kind of RAM; > as you say that will be very inefficient. Exposing the number of set elements is feasible and it sounds useful too. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: nft set load metrics 2021-09-30 17:34 ` G.W. Haywood 2021-09-30 18:12 ` Pablo Neira Ayuso @ 2021-09-30 19:01 ` Cristian Constantin 1 sibling, 0 replies; 9+ messages in thread From: Cristian Constantin @ 2021-09-30 19:01 UTC (permalink / raw) To: G.W. Haywood; +Cc: netfilter > Presumably it was you who created the elements, can you not simply > store them e.g. in a Perl hash at the time they're created? I do > something similar using Net::CIDR::Lite. (I would never claim that a > Perl script is the most efficient way of doing it, nor anything else > for that matter, but it gets the job done. :) cristian: you are basically proposing a kind of user space cache of kernel nft sets maintained by the application which pushes the rules/sets to the kernel. I have also thought of that. it can get complicated though especially when the sets are dynamic, with their entries expiring. implementing performant timers in application is yet another beast to deal with... On Thu, Sep 30, 2021 at 7:35 PM G.W. Haywood <ged@jubileegroup.co.uk> wrote: > > Hi there, > > On Thu, 30 Sep 2021, Cristian Constantin wrote: > > > ... reading large packets over netlink sockets just to count the > > elements in the sets does not seem very efficient. > > Agreed. > > It seems to me that if you need to read what you've put in the sets > for the purposes of some facility, then you need to store it in RAM. > > It doesn't make sense to me to try to use netfilter as a kind of RAM; > as you say that will be very inefficient. > > Presumably it was you who created the elements, can you not simply > store them e.g. in a Perl hash at the time they're created? I do > something similar using Net::CIDR::Lite. (I would never claim that a > Perl script is the most efficient way of doing it, nor anything else > for that matter, but it gets the job done. :) > > -- > > 73, > Ged. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: nft set load metrics 2021-09-30 15:47 ` Cristian Constantin 2021-09-30 17:34 ` G.W. Haywood @ 2021-09-30 18:04 ` Pablo Neira Ayuso 2021-09-30 18:46 ` Cristian Constantin 2021-10-02 11:54 ` Pablo Neira Ayuso 1 sibling, 2 replies; 9+ messages in thread From: Pablo Neira Ayuso @ 2021-09-30 18:04 UTC (permalink / raw) To: Cristian Constantin; +Cc: netfilter On Thu, Sep 30, 2021 at 05:47:07PM +0200, Cristian Constantin wrote: > > It's possible to extend the interface to expose this, but how useful > > is this? > > cristian: imo, it is important from the operational point of view to > monitor the size of the sets. from an implementation point of view, > since the sets can grow to very large sizes, reading large packets > over netlink sockets just to count the elements in the sets does not > seem very efficient. You mean, provide stats that allow to monitor the memory size? That might make sense, yes. > the nft user space tool has a switch which turns off showing the set > content: '-t', however it will only show the name of the set and the > flags. I also did not check if this feature is offered at the socket > layer or if the elements are actually read from the kernel but not > displayed. IIRC, they are read from the kernel, then not displayed, but it should be easy to only fetch the set, I'll prepare a patch to speed -t listing. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: nft set load metrics 2021-09-30 18:04 ` Pablo Neira Ayuso @ 2021-09-30 18:46 ` Cristian Constantin 2021-10-02 11:54 ` Pablo Neira Ayuso 1 sibling, 0 replies; 9+ messages in thread From: Cristian Constantin @ 2021-09-30 18:46 UTC (permalink / raw) To: Pablo Neira Ayuso; +Cc: netfilter > > cristian: imo, it is important from the operational point of view to > > monitor the size of the sets. from an implementation point of view, > > since the sets can grow to very large sizes, reading large packets > > over netlink sockets just to count the elements in the sets does not > > seem very efficient. > > You mean, provide stats that allow to monitor the memory size? That > might make sense, yes. cristian: basically generic resource consumption monitoring. assume a setup like this: 1. ~1M ip addresses loaded in a set; 2. there is a filter rule which drops all forwarded ip packets with dst address matching the ip addresses in the set; 3. some traffic generator (e.g. hping3) floods the respective machine/interface with UDP traffic ~400 Kpkts/s, destination ip addresses being randomized but in the range of those in the respective set; the machine running the firewall uses btw. 15%-20% CPU (with occasional spikes up to 30%) so far I can tell, the number of ip addresses in a set can affect directly the CPU consumption as well... On Thu, Sep 30, 2021 at 8:04 PM Pablo Neira Ayuso <pablo@netfilter.org> wrote: > > On Thu, Sep 30, 2021 at 05:47:07PM +0200, Cristian Constantin wrote: > > > It's possible to extend the interface to expose this, but how useful > > > is this? > > > > cristian: imo, it is important from the operational point of view to > > monitor the size of the sets. from an implementation point of view, > > since the sets can grow to very large sizes, reading large packets > > over netlink sockets just to count the elements in the sets does not > > seem very efficient. > > You mean, provide stats that allow to monitor the memory size? That > might make sense, yes. > > > the nft user space tool has a switch which turns off showing the set > > content: '-t', however it will only show the name of the set and the > > flags. I also did not check if this feature is offered at the socket > > layer or if the elements are actually read from the kernel but not > > displayed. > > IIRC, they are read from the kernel, then not displayed, but it should > be easy to only fetch the set, I'll prepare a patch to speed -t > listing. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: nft set load metrics 2021-09-30 18:04 ` Pablo Neira Ayuso 2021-09-30 18:46 ` Cristian Constantin @ 2021-10-02 11:54 ` Pablo Neira Ayuso 1 sibling, 0 replies; 9+ messages in thread From: Pablo Neira Ayuso @ 2021-10-02 11:54 UTC (permalink / raw) To: Cristian Constantin; +Cc: netfilter On Thu, Sep 30, 2021 at 08:04:08PM +0200, Pablo Neira Ayuso wrote: > IIRC, they are read from the kernel, then not displayed, but it should > be easy to only fetch the set, I'll prepare a patch to speed -t > listing. For the record, patch to speed up listing with -t: https://patchwork.ozlabs.org/project/netfilter-devel/patch/20211002115059.26792-1-pablo@netfilter.org/ ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2021-10-02 11:54 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-09-30 11:21 nft set load metrics Cristian Constantin 2021-09-30 14:00 ` Pablo Neira Ayuso 2021-09-30 15:47 ` Cristian Constantin 2021-09-30 17:34 ` G.W. Haywood 2021-09-30 18:12 ` Pablo Neira Ayuso 2021-09-30 19:01 ` Cristian Constantin 2021-09-30 18:04 ` Pablo Neira Ayuso 2021-09-30 18:46 ` Cristian Constantin 2021-10-02 11:54 ` Pablo Neira Ayuso
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.