nft show counter

nft show counter

[Frank Wunderlich]

Hi,

i want to read counters defined in nftables, but have not found a solid way to do this

nft list table ip filter

shows me the rules from specific table with all chains, rules and counters, e.g.

	chain voip1 {
		counter packets 3226 bytes 780864 accept
	}

	chain voip2 {
		counter packets 1640 bytes 119449 accept
	}

this needs to be parsed with grep/sed/awk, so maybe there is an easier way to read the counters. found nothing in manpage or google, so i just ask here. maybe there is a way to show the last x min traffic. i need this to check if connections is "alive"

regards Frank

Re: nft show counter

[Frank Myhr]

On 2021/03/31 13:47, Frank Wunderlich wrote:
> Hi,
> 
> i want to read counters defined in nftables, but have not found a solid way to do this
> 
> nft list table ip filter
> 
> shows me the rules from specific table with all chains, rules and counters, e.g.
> 
> 	chain voip1 {
> 		counter packets 3226 bytes 780864 accept
> 	}
> 
> 	chain voip2 {
> 		counter packets 1640 bytes 119449 accept
> 	}
> 
> this needs to be parsed with grep/sed/awk, so maybe there is an easier way to read the counters. found nothing in manpage or google, so i just ask here. maybe there is a way to show the last x min traffic. i need this to check if connections is "alive"
> 
> regards Frank

Hi Frank,

Just some quick ideas:

Using nft JSON output could work better for parsing:
https://wiki.nftables.org/wiki-nftables/index.php/Output_text_modifiers

You could use nftables from within a Python program to do your monitoring:
https://wiki.nftables.org/wiki-nftables/index.php/Scripting#Using_nftables_from_Python

If you don't care about actual values of packets/bytes but only that 
packets are matching certain rules at a minimum rate, you could use a 
log statement, perhaps with a limit, to log to syslog or userspace nflog 
and monitor the appropriate log file(s).
https://wiki.nftables.org/wiki-nftables/index.php/Logging_traffic
https://wiki.nftables.org/wiki-nftables/index.php/Rate_limiting_matchings

Best Wishes,
Frank

Re: nft show counter

[Florian Westphal]

Frank Wunderlich <frank-w@public-files.de> wrote:
> Hi,
> 
> i want to read counters defined in nftables, but have not found a solid way to do this
> 
> nft list table ip filter
> 
> shows me the rules from specific table with all chains, rules and counters, e.g.
> 
> 	chain voip1 {
> 		counter packets 3226 bytes 780864 accept
> 	}
> 
> 	chain voip2 {
> 		counter packets 1640 bytes 119449 accept
> 	}
> 
> this needs to be parsed with grep/sed/awk, so maybe there is an easier way to read the counters. found nothing in manpage or google, so i just ask here. maybe there is a way to show the last x min traffic. i need this to check if connections is "alive"

You could use a named counter and then fetch it via 'nft list counter
inet tablenam countername'.

Re: nft show counter

[Frank Myhr]

On 2021/03/31 17:40, Florian Westphal wrote:
> You could use a named counter and then fetch it via 'nft list counter
> inet tablenam countername'.

This is the straightforward solution!

Florian, is it correct that *all* counters are stateful objects, they're 
just anonymous stateful objects if used in a rule without first 
declaring them with a name? The wiki reads as if they're 2 separate 
things (I'm happy to fix but want to check with you first):

https://wiki.nftables.org/wiki-nftables/index.php/Counters

Thanks,
Frank M.

Aw: Re: nft show counter

[Frank Wunderlich]

> Gesendet: Donnerstag, 01. April 2021 um 01:08 Uhr
> Von: "Frank Myhr" <fmyhr@fhmtech.com>
> An: "Florian Westphal" <fw@strlen.de>, "Frank Wunderlich" <frank-w@public-files.de>
> Cc: netfilter@vger.kernel.org
> Betreff: Re: nft show counter
>
> On 2021/03/31 17:40, Florian Westphal wrote:
> > You could use a named counter and then fetch it via 'nft list counter
> > inet tablenam countername'.
>
> This is the straightforward solution!
>
> Florian, is it correct that *all* counters are stateful objects, they're
> just anonymous stateful objects if used in a rule without first
> declaring them with a name? The wiki reads as if they're 2 separate
> things (I'm happy to fix but want to check with you first):
>
> https://wiki.nftables.org/wiki-nftables/index.php/Counters

Hi,

thank you, i saw the named counters before, but i have thought i can read out the "normal" counters too...

have now changed to named counters (instead of jump to voipX-chain)

my rules-file looks like this now (just for anybody wants do do similar):

table ip filter {
    ...
    chain FORWARD {
        type filter hook forward priority 0; policy drop;
        #...
        ip saddr $ipvoipbox udp dport 5160 counter name voip1 comment "counting packets for SIP1"
        ip daddr $ipvoipbox udp dport 5160 counter name voip1 comment "counting packets for SIP1"
        ip saddr $ipvoipbox udp sport 5161 counter name voip2 comment "counting packets for SIP2"
        ip daddr $ipvoipbox udp dport 5161 counter name voip2 comment "counting packets for SIP2"
        #...
    }

    counter voip1 {
    }

    counter voip2 {
    }
}

one thing:

# nft list counter filter voip2
table ip mangle {
}
table ip nat {
}
table ip filter {
	counter voip2 {
		packets 124 bytes 7440
	}
}


tables mangle and nat should not be printed (still have them separately from converting iptables to nft)...in json-format it is right

# nft list counter filter voip2 -j
{"nftables": [{"counter": {"family": "ip", "name": "voip2", "table": "filter", "handle": 6, "packets": 136, "bytes": 8160}}]}

made a little python-script (not yet with error-handling) to read out the bytes:

import os
import json


def getCounter(countername):
    stream = os.popen('nft list counter filter '+countername+' -j')
    output = stream.read()
    #print(output)
    j=json.loads(output)
    return j["nftables"][0]["counter"]["bytes"]

print(getCounter("voip1"),"bytes")
print(getCounter("voip2"),"bytes")



regards Frank

Re: nft show counter

[Anton Rieger]

On Thu, Apr 01, 2021 at 10:08:32AM +0200, Frank Wunderlich wrote:
>made a little python-script (not yet with error-handling) to read out the bytes:
>
>import os
>import json
>
>
>def getCounter(countername):
>    stream = os.popen('nft list counter filter '+countername+' -j')
>    output = stream.read()
>    #print(output)
>    j=json.loads(output)
>    return j["nftables"][0]["counter"]["bytes"]
>
>print(getCounter("voip1"),"bytes")
>print(getCounter("voip2"),"bytes")

Here's above using the nftables python module:

```
from nftables import Nftables
from nftables import json


def getCounter(countername, family='ip'):
     nft = Nftables()
     nft.set_json_output(True)
	_, output, _ = nft.cmd(f"list counter {family} filter {countername}")
     j = json.loads(output)
     return j['nftables'][1]["counter"]["bytes"]

print(getCounter('voip1'), 'bytes')
print(getCounter('voip2'), 'bytes')
```

Greetings
Anton

Re: Re: nft show counter

[Florian Westphal]

Frank Wunderlich <frank-w@public-files.de> wrote:
> my rules-file looks like this now (just for anybody wants do do similar):
> 
> table ip filter {
>     ...
>     chain FORWARD {
>         type filter hook forward priority 0; policy drop;
>         #...
>         ip saddr $ipvoipbox udp dport 5160 counter name voip1 comment "counting packets for SIP1"
>         ip daddr $ipvoipbox udp dport 5160 counter name voip1 comment "counting packets for SIP1"
>         ip saddr $ipvoipbox udp sport 5161 counter name voip2 comment "counting packets for SIP2"
>         ip daddr $ipvoipbox udp dport 5161 counter name voip2 comment "counting packets for SIP2"
>         #...
>     }

For two counters it makes no difference but note that you can combine
named counters with maps:

map voipcounters {
 type ipv4_addr . inet_service : counter
 elements = { 10.0.1.1 . 5160 : "voip1", 10.2.1.1 . 5161 : ...

and then count with one rule only:

counter name ip saddr . udp dport map @voipcounters

> one thing:
> 
> # nft list counter filter voip2
> table ip mangle {
> }
> table ip nat {
> }
> table ip filter {
> 	counter voip2 {
> 		packets 124 bytes 7440
> 	}
> }

Can't repro so looks like this is already fixed.

> tables mangle and nat should not be printed (still have them separately from converting iptables to nft)...in json-format it is right

Yes, they should not be printed.

Re: nft show counter

[Florian Westphal]

Frank Myhr <fmyhr@fhmtech.com> wrote:
> On 2021/03/31 17:40, Florian Westphal wrote:
> > You could use a named counter and then fetch it via 'nft list counter
> > inet tablenam countername'.
> 
> This is the straightforward solution!
> 
> Florian, is it correct that *all* counters are stateful objects, they're
> just anonymous stateful objects if used in a rule without first declaring
> them with a name? The wiki reads as if they're 2 separate things (I'm happy
> to fix but want to check with you first):
> 
> https://wiki.nftables.org/wiki-nftables/index.php/Counters

The normal counters do not have a 'hidden' internal name.

Thats also the reason why simple counters do not show up in 'nft list counters',
they only exist in the context of the rule they are used in.

Re: Re: nft show counter

[Frank Wunderlich]

Am 1. April 2021 20:14:14 MESZ schrieb Florian Westphal <fw@strlen.de>:
>Frank Wunderlich <frank-w@public-files.de> wrote:
>> my rules-file looks like this now (just for anybody wants do do
>similar):
>> 
>> table ip filter {
>>     ...
>>     chain FORWARD {
>>         type filter hook forward priority 0; policy drop;
>>         #...
>>         ip saddr $ipvoipbox udp dport 5160 counter name voip1 comment
>"counting packets for SIP1"
>>         ip daddr $ipvoipbox udp dport 5160 counter name voip1 comment
>"counting packets for SIP1"
>>         ip saddr $ipvoipbox udp sport 5161 counter name voip2 comment
>"counting packets for SIP2"
>>         ip daddr $ipvoipbox udp dport 5161 counter name voip2 comment
>"counting packets for SIP2"
>>         #...
>>     }
>
>For two counters it makes no difference but note that you can combine
>named counters with maps:
>
>map voipcounters {
> type ipv4_addr . inet_service : counter
> elements = { 10.0.1.1 . 5160 : "voip1", 10.2.1.1 . 5161 : ...
>
>and then count with one rule only:
>
>counter name ip saddr . udp dport map @voipcounters

Have seen maps,but have not understood them :(

The ip is always same ($ipvoipbox),only port differs (sip1+sip2) and this bidirectional (saddr/daddr,sport/dport) to catch traffic from/to this ip separated by sip-streams.

Your example seems to match only source-addr and destination port.

Maybe its possible to match "addr" (saddr or daddr) and "port" (sport or dport),so i can have only 2 rules...with map maybe only 1

Maybe i can do further protocol filtering (filter out registrations to not count failed registrations,only working traffic).

>> one thing:
>> 
>> # nft list counter filter voip2
>> table ip mangle {
>> }
>> table ip nat {
>> }
>> table ip filter {
>> 	counter voip2 {
>> 		packets 124 bytes 7440
>> 	}
>> }
>
>Can't repro so looks like this is already fixed.
>
>> tables mangle and nat should not be printed (still have them
>separately from converting iptables to nft)...in json-format it is
>right
>
>Yes, they should not be printed.

I try this again with upstream nft
Hi,
regards Frank

Re: Re: nft show counter

[Florian Westphal]

Frank Wunderlich <frank-w@public-files.de> wrote:
> Am 1. April 2021 20:14:14 MESZ schrieb Florian Westphal <fw@strlen.de>:
> >map voipcounters {
> > type ipv4_addr . inet_service : counter
> > elements = { 10.0.1.1 . 5160 : "voip1", 10.2.1.1 . 5161 : ...
> >
> >and then count with one rule only:
> >
> >counter name ip saddr . udp dport map @voipcounters
> 
> Have seen maps,but have not understood them :(

Its just a set that returns a value instead of simple 'in set/not in
set'.

> The ip is always same ($ipvoipbox),only port differs (sip1+sip2) and this bidirectional (saddr/daddr,sport/dport) to catch traffic from/to this ip separated by sip-streams.
> 
> Your example seems to match only source-addr and destination port.

Yes, it was an example.

> Maybe its possible to match "addr" (saddr or daddr) and "port" (sport or dport),so i can have only 2 rules...with map maybe only 1

There is no 'or', so you will need two rules at a minimum if you want to
match in either direction.

map voipcounters {
	type inet_service : counter
	elements = { 5160 : "voip1", 5161 : "voip2" }
}

ip saddr $ipvoipbox counter name udp sport map @voipcounters
ip daddr $ipvoipbox counter name udp dport map @voipcounters

This says 'fetch name of counter to use from @voipcounters, with 'udp
sport/udp dport' as search key.

but with just two counters its not worth it.

Aw: Re: Re: nft show counter

[Frank Wunderlich]

> Gesendet: Donnerstag, 01. April 2021 um 20:14 Uhr
> Von: "Florian Westphal" <fw@strlen.de>
> Frank Wunderlich <frank-w@public-files.de> wrote:
> > one thing:
> >
> > # nft list counter filter voip2
> > table ip mangle {
> > }
> > table ip nat {
> > }
> > table ip filter {
> > 	counter voip2 {
> > 		packets 124 bytes 7440
> > 	}
> > }
>
> Can't repro so looks like this is already fixed.
>
> > tables mangle and nat should not be printed (still have them separately from converting iptables to nft)...in json-format it is right
>
> Yes, they should not be printed.

tested with my self-compiled nftables, and indeed it seems to be fixed:

root@bpi-r2:~# nft -f ruleset_new.nft
root@bpi-r2:~# nft list counter filter voip1    #using debian buster version
table ip mangle {
}
table ip nat {
}
table ip filter {
        counter voip1 {
                packets 0 bytes 0
        }
}
root@bpi-r2:~# /home/frank/nftables/install/sbin/nft list counter filter voip1 #self compiled
table ip filter {
        counter voip1 {
                packets 0 bytes 0
        }
}
root@bpi-r2:~#

anyway i have now moved the mangle/nat chains into my filter table in newer version of my ruleset to have only an ip(v4) and an ipv6 table (still separated as v4 is much more complicated than v6)

Thanks, i will now test the mapping for log-counters, here it will be good to have addr/port (instead of saddr/daddr + sport/dport) too. Is there a way to filter deeper into protocol stack (SIP data ignoring start registration traffic)

regards Frank

Re: Aw: Re: Re: nft show counter

[Frank Myhr]

On 2021/04/02 07:58, Frank Wunderlich wrote:
> Is there a way to filter deeper into protocol stack (SIP data ignoring start registration traffic)

There's this, on the SIP connection tracking helper:
https://home.regit.org/netfilter-en/secure-use-of-helpers/

If that's not enough, you could look at queuing packets to a userspace 
program that uses libnetfilter_queue:

https://wiki.nftables.org/wiki-nftables/index.php/Queueing_to_userspace

https://home.regit.org/netfilter-en/using-nfqueue-and-libnetfilter_queue/

For an existing app, Google found:
http://genesysguru.com/blog/blog/2013/12/11/sip-interceptor/
which may or may not have morphed into
https://docs.rhino.metaswitch.com/ocdoc/books/sis-documentation/2.6.1/sis-administration-guide/managing-the-sis/managing-interceptors/index.html
Not sure that's available at reasonable cost and/or current.


Getting beyond my depth, but some other things you could look into:

* Maybe your pbx software has some filtering ability built in? (Thinking 
of something analogous to haproxy, but for SIP/RTP.)

Looking around:

FreePBX seems to have just a pre-configured iptables setup:
https://wiki.freepbx.org/display/FPG/Firewall

Asterisk / Sangoma recommends a session border controller (SBC):
https://www.sangoma.com/articles/voip-firewall/

* ... So may be worth looking into using an SBC:
https://en.wikipedia.org/wiki/Session_border_controller

Maybe someone with more recent VoIP setup experience will have 
recommendations.

Best Wishes,
Frank M.

Re: nft show counter

[Frank Myhr]

On 2021/04/01 12:42, Anton Rieger wrote:
> On Thu, Apr 01, 2021 at 10:08:32AM +0200, Frank Wunderlich wrote:
>> made a little python-script (not yet with error-handling) to read out 
>> the bytes:
>>
>> import os
>> import json
>>
>>
>> def getCounter(countername):
>>    stream = os.popen('nft list counter filter '+countername+' -j')
>>    output = stream.read()
>>    #print(output)
>>    j=json.loads(output)
>>    return j["nftables"][0]["counter"]["bytes"]
>>
>> print(getCounter("voip1"),"bytes")
>> print(getCounter("voip2"),"bytes")
> 
> Here's above using the nftables python module:
> 
> ```
> from nftables import Nftables
> from nftables import json
> 
> 
> def getCounter(countername, family='ip'):
>      nft = Nftables()
>      nft.set_json_output(True)
>      _, output, _ = nft.cmd(f"list counter {family} filter {countername}")
>      j = json.loads(output)
>      return j['nftables'][1]["counter"]["bytes"]
> 
> print(getCounter('voip1'), 'bytes')
> print(getCounter('voip2'), 'bytes')
> ```
> 
> Greetings
> Anton

Nice!

Anton & Frank W., is it OK with you if I include your examples in the 
nftables wiki?

Thanks,
Frank M.

Re: nft show counter

[Frank Wunderlich]

Am 2. April 2021 19:16:50 MESZ schrieb Frank Myhr <fmyhr@fhmtech.com>:
>On 2021/04/01 12:42, Anton Rieger wrote:
>> On Thu, Apr 01, 2021 at 10:08:32AM +0200, Frank Wunderlich wrote:
>>> made a little python-script (not yet with error-handling) to read
>out 
>>> the bytes:
>>>
>>> import os
>>> import json
>>>
>>>
>>> def getCounter(countername):
>>>    stream = os.popen('nft list counter filter '+countername+' -j')
>>>    output = stream.read()
>>>    #print(output)
>>>    j=json.loads(output)
>>>    return j["nftables"][0]["counter"]["bytes"]
>>>
>>> print(getCounter("voip1"),"bytes")
>>> print(getCounter("voip2"),"bytes")

>
>Nice!
>Anton & Frank W., is it OK with you if I include your examples in the 
>nftables wiki?

My part is ok to include in wiki...i have posted to share :) maybe in combination with the rules (old version commented out and with map) to have full example

define ipvoipbox=192.168.0.8

table ip filter {
     map voipcounters {
         type inet_service : counter
         elements = { 5160 : "voip1", 5161 : "voip2" }
     }
    chain FORWARD {
         #ip saddr $ipvoipbox udp dport 5160 counter name voip1 comment "counting packets for SIP1"
         #ip daddr $ipvoipbox udp dport 5160 counter name voip1 comment "counting packets for SIP1"
         #ip saddr $ipvoipbox udp sport 5161 counter name voip2 comment "counting packets for SIP2"
         #ip daddr $ipvoipbox udp dport 5161 counter name voip2 comment "counting packets for SIP2"
         ip saddr $ipvoipbox counter name udp sport map @voipcounters
         ip daddr $ipvoipbox counter name udp dport map @voipcounters
     }
    counter voip1 {
     }
     counter voip2 {
     }
 }

regards Frank

Re: nft show counter

[Anton Rieger]

On Fri, Apr 02, 2021 at 01:16:50PM -0400, Frank Myhr wrote:
>On 2021/04/01 12:42, Anton Rieger wrote:
>>On Thu, Apr 01, 2021 at 10:08:32AM +0200, Frank Wunderlich wrote:
>>>made a little python-script (not yet with error-handling) to read
>>>out the bytes:
>>>
>>>import os
>>>import json
>>>
>>>
>>>def getCounter(countername):
>>>   stream = os.popen('nft list counter filter '+countername+' -j')
>>>   output = stream.read()
>>>   #print(output)
>>>   j=json.loads(output)
>>>   return j["nftables"][0]["counter"]["bytes"]
>>>
>>>print(getCounter("voip1"),"bytes")
>>>print(getCounter("voip2"),"bytes")
>>
>>Here's above using the nftables python module:
>>
>>```
>>from nftables import Nftables
>>from nftables import json
>>
>>
>>def getCounter(countername, family='ip'):
>>     nft = Nftables()
>>     nft.set_json_output(True)
>>     _, output, _ = nft.cmd(f"list counter {family} filter {countername}")
>>     j = json.loads(output)
>>     return j['nftables'][1]["counter"]["bytes"]
>>
>>print(getCounter('voip1'), 'bytes')
>>print(getCounter('voip2'), 'bytes')
>>```
>>
>>Greetings
>>Anton
>
>Nice!
>
>Anton & Frank W., is it OK with you if I include your examples in the
>nftables wiki?

Sure

Re: nft show counter

[Frank Myhr]

On 2021/04/02 13:42, Frank Wunderlich wrote:
>> Anton & Frank W., is it OK with you if I include your examples in the
>> nftables wiki?
> 
> My part is ok to include in wiki...i have posted to share :) maybe in combination with the rules (old version commented out and with map) to have full example

On 2021/04/02 18:22, Anton Rieger wrote:
> Sure 

Thanks, guys! I posted your example:
https://wiki.nftables.org/wiki-nftables/index.php/Counters#Reading_named_counters_from_Python

Please let me know if you find mistakes.

Thanks,
Frank M.

manual : CONNTRACK EXPRESSIONS

[pauloric]

Hi

Near  CONNTRACK EXPRESSIONS at manual I think that should be good to insert something like:

ct {state | direction | status | mark | expiration | helper | label}
....
ct count [over]    <-------

and just a little bellow it has already an example of it.
...
filter input tcp dport 22 meter test { ip saddr ct count over 2 } reject

Maybe a include at Table 55. Conntrack expressions  too ??? <----

Could it be inserted at https://wiki.nftables.org/wiki-nftables/index.php/Quick_reference-nftables_in_10_minutes#Ct    ???

best regards


-- 
Paulo Ricardo Bruck consultor 

Re: manual : CONNTRACK EXPRESSIONS

[Pablo Neira Ayuso]

Hi,

On Thu, Jul 22, 2021 at 04:57:37PM -0300, pauloric@contatogs.com.br wrote:
> Hi
> 
> Near  CONNTRACK EXPRESSIONS at manual I think that should be good to insert something like:
> 
> ct {state | direction | status | mark | expiration | helper | label}
> ....
> ct count [over]    <------->
> 
> and just a little bellow it has already an example of it.
> ...
> filter input tcp dport 22 meter test { ip saddr ct count over 2 } reject
> 
> Maybe a include at Table 55. Conntrack expressions  too ??? <----

Since 0.9.7, manpage contains a reference to 'ct count'.

There is also an example usage:

       restrict the number of parallel connections to a server.

           nft add set filter ssh_flood '{ type ipv4_addr; flags dynamic; }'
           nft add rule filter input tcp dport 22 add @ssh_flood '{ ip saddr ct count over 2 }' reject

> Could it be inserted at https://wiki.nftables.org/wiki-nftables/index.php/Quick_reference-nftables_in_10_minutes#Ct    ???

Updated.

Thanks