ulogd packet based logging with CT info

ulogd packet based logging with CT info

[Blažej Krajňák]

Hello,

is there any way to enable packet based logging to XML but with
connection tracking information included?

Thanks
Blažej Krajňák

Re: ulogd packet based logging with CT info

[Pablo Neira Ayuso]

On Sun, Aug 15, 2021 at 04:23:51PM +0200, Blažej Krajňák wrote:
> Hello,
> 
> is there any way to enable packet based logging to XML but with
> connection tracking information included?

From the example ulogd2 configuration file in the tree:

http://git.netfilter.org/ulogd2/tree/ulogd.conf.in#n77

# this is a stack for flow-based logging via XML
#stack=ct1:NFCT,xml1:XML

Re: ulogd packet based logging with CT info

[Blažej Krajňák]

ne 15. 8. 2021 o 16:31 Pablo Neira Ayuso <pablo@netfilter.org> wrote
>
> From the example ulogd2 configuration file in the tree:
>
> http://git.netfilter.org/ulogd2/tree/ulogd.conf.in#n77
>
> # this is a stack for flow-based logging via XML
> #stack=ct1:NFCT,xml1:XML

At first, thank you for fast response. Of course I saw this example,
but I have situation, where I need to know packet details (pktlen,
mac, ifindex) along with information of conntrack entry which this
packet triggered.
I modified one of the libnetfilter_log example utilities to print both
(packet and conntrack) information together.
https://drive.google.com/file/d/1wx_LAjH57czHyFwTBSUvSnOkMchWEiiq/view?usp=sharing

Is there any way to do the same in ulogd?

Re: ulogd packet based logging with CT info

[Blažej Krajňák]

Hello Pablo,

I'm just rewriting input plugin ulog_inppkt_NFLOG.c to include
conntrack params. I successfully included CT flags from enum
ip_conntrack_status (assured, reply seen, ...) and CT state and
direction from NFULA_CT_INFO.

However, in NFULA_CT few counters from enum nf_conntrack_attr are
still 0 value. For ex. ATTR_TIMESTAMP_START / STOP and
ATTR_ORIG/REPL_COUNTER_PACKETS/BYTES.
Is it normal, or am I missing some bug at parsing?

ne 15. 8. 2021 o 18:23 Blažej Krajňák <blazej.krajnak@gmail.com> napísal(a):
>
> ne 15. 8. 2021 o 16:31 Pablo Neira Ayuso <pablo@netfilter.org> wrote
> >
> > From the example ulogd2 configuration file in the tree:
> >
> > http://git.netfilter.org/ulogd2/tree/ulogd.conf.in#n77
> >
> > # this is a stack for flow-based logging via XML
> > #stack=ct1:NFCT,xml1:XML
>
> At first, thank you for fast response. Of course I saw this example,
> but I have situation, where I need to know packet details (pktlen,
> mac, ifindex) along with information of conntrack entry which this
> packet triggered.
> I modified one of the libnetfilter_log example utilities to print both
> (packet and conntrack) information together.
> https://drive.google.com/file/d/1wx_LAjH57czHyFwTBSUvSnOkMchWEiiq/view?usp=sharing
>
> Is there any way to do the same in ulogd?

Re: ulogd packet based logging with CT info

[Fatih USTA]

0
Hi
Did you enable sysctl parameters.

sysctl -w nf_conntrack_acct=1
sysctl -w nf_conntrack_timestamp=1

On 8/17/21, Blažej Krajňák <blazej.krajnak@gmail.com> wrote:
> Hello Pablo,
>
> I'm just rewriting input plugin ulog_inppkt_NFLOG.c to include
> conntrack params. I successfully included CT flags from enum
> ip_conntrack_status (assured, reply seen, ...) and CT state and
> direction from NFULA_CT_INFO.
>
> However, in NFULA_CT few counters from enum nf_conntrack_attr are
> still 0 value. For ex. ATTR_TIMESTAMP_START / STOP and
> ATTR_ORIG/REPL_COUNTER_PACKETS/BYTES.
> Is it normal, or am I missing some bug at parsing?
>
> ne 15. 8. 2021 o 18:23 Blažej Krajňák <blazej.krajnak@gmail.com>
> napísal(a):
>>
>> ne 15. 8. 2021 o 16:31 Pablo Neira Ayuso <pablo@netfilter.org> wrote
>> >
>> > From the example ulogd2 configuration file in the tree:
>> >
>> > http://git.netfilter.org/ulogd2/tree/ulogd.conf.in#n77
>> >
>> > # this is a stack for flow-based logging via XML
>> > #stack=ct1:NFCT,xml1:XML
>>
>> At first, thank you for fast response. Of course I saw this example,
>> but I have situation, where I need to know packet details (pktlen,
>> mac, ifindex) along with information of conntrack entry which this
>> packet triggered.
>> I modified one of the libnetfilter_log example utilities to print both
>> (packet and conntrack) information together.
>> https://drive.google.com/file/d/1wx_LAjH57czHyFwTBSUvSnOkMchWEiiq/view?usp=sharing
>>
>> Is there any way to do the same in ulogd?
>

Re: ulogd packet based logging with CT info

[Blažej Krajňák]

ut 17. 8. 2021 o 17:05 Fatih USTA <fatihusta86@gmail.com> napísal(a):
> Hi
> Did you enable sysctl parameters.
>
> sysctl -w nf_conntrack_acct=1
> sysctl -w nf_conntrack_timestamp=1
>

Yes, both are enabled. conntrack -L displays counters and delta-time correctly.

u_int32_t pkts = nfct_get_attr_u32(ct, ATTR_ORIG_COUNTER_PACKETS );
printf("  Packet counter: %d", ntohs(pkts));
printf(" (%s)\n", strerror(errno));

returns 0 (No data available)

I'm attaching draft-work of module to check.
https://drive.google.com/file/d/1KouPKrrANvOihBuQH8Uwslk3pBkmP95q/view?usp=sharing

I'm not sure about parsing

static inline int
interp_packet(struct ulogd_pluginstance *upi, uint8_t pf_family,
      struct nflog_data *ldata)
{
    struct ulogd_key *ret = upi->output.keys;

    struct nfulnl_msg_packet_hdr *ph = nflog_get_msg_packet_hdr(ldata);
    .......
    retb = nflog_nlmsg_parse(ph, attrs);
    if (retb != MNL_CB_OK) {
        printf("something went wrong");
        printf(" (%s)\n", strerror(errno));
        return retb;
    }

    nfg = mnl_nlmsg_get_payload(ph);

    print_nfct(nfg->nfgen_family, attrs[NFULA_CT_INFO], attrs[NFULA_CT], ret);

Re: ulogd packet based logging with CT info

[Pablo Neira Ayuso]

[-- Attachment #1: Type: text/plain, Size: 631 bytes --]

On Tue, Aug 17, 2021 at 04:42:12PM +0200, Blažej Krajňák wrote:
> Hello Pablo,
> 
> I'm just rewriting input plugin ulog_inppkt_NFLOG.c to include
> conntrack params. I successfully included CT flags from enum
> ip_conntrack_status (assured, reply seen, ...) and CT state and
> direction from NFULA_CT_INFO.

Great.

> However, in NFULA_CT few counters from enum nf_conntrack_attr are
> still 0 value. For ex. ATTR_TIMESTAMP_START / STOP and
> ATTR_ORIG/REPL_COUNTER_PACKETS/BYTES.
> Is it normal, or am I missing some bug at parsing?

You need this kernel patch to add this information to nfnetlink_queue,
compile-tested only.

[-- Attachment #2: x.patch --]
[-- Type: text/x-diff, Size: 1065 bytes --]

diff --git a/net/netfilter/nf_conntrack_netlink.c b/net/netfilter/nf_conntrack_netlink.c
index eb35c6151fb0..0677531ce8db 100644
--- a/net/netfilter/nf_conntrack_netlink.c
+++ b/net/netfilter/nf_conntrack_netlink.c
@@ -2669,6 +2669,8 @@ ctnetlink_glue_build_size(const struct nf_conn *ct)
 	       + nla_total_size(0) /* CTA_HELP */
 	       + nla_total_size(NF_CT_HELPER_NAME_LEN) /* CTA_HELP_NAME */
 	       + ctnetlink_secctx_size(ct)
+	       + ctnetlink_acct_size(ct)
+	       + ctnetlink_timestamp_size(ct)
 #if IS_ENABLED(CONFIG_NF_NAT)
 	       + 2 * nla_total_size(0) /* CTA_NAT_SEQ_ADJ_ORIG|REPL */
 	       + 6 * nla_total_size(sizeof(u_int32_t)) /* CTA_NAT_SEQ_OFFSET */
@@ -2726,6 +2728,10 @@ static int __ctnetlink_glue_build(struct sk_buff *skb, struct nf_conn *ct)
 	if (ctnetlink_dump_protoinfo(skb, ct, false) < 0)
 		goto nla_put_failure;
 
+	if (ctnetlink_dump_acct(skb, ct, IPCTNL_MSG_CT_GET) < 0 ||
+	    ctnetlink_dump_timestamp(skb, ct) < 0)
+		goto nla_put_failure;
+
 	if (ctnetlink_dump_helpinfo(skb, ct) < 0)
 		goto nla_put_failure;
 

Re: ulogd packet based logging with CT info

[Blažej Krajňák]

st 18. 8. 2021 o 9:23 Pablo Neira Ayuso <pablo@netfilter.org> napísal(a):
> You need this kernel patch to add this information to nfnetlink_queue,
> compile-tested only.

Hey Pablo,
patch is working like a charm. Thank you! Now I see timestamps and
packets/bytes counters in JSON output correctly.
At the end I will post customized input plugin for everyone.

The last thing I want to ask is what's correct way to get

attrs[NFULA_CT]

from

struct nfulnl_msg_packet_hdr *ph = nflog_get_msg_packet_hdr(ldata);


Now I use the following code which is working but throwing random
errors "something went wrong (Numerical result out of range)" I'm
mixing struct nfulnl_msg_packet_hdr with const struct nlmsghdr *nlh


struct nfulnl_msg_packet_hdr *ph = nflog_get_msg_packet_hdr(ldata);
struct nlattr *attrs[NFULA_MAX + 1] = { NULL };
int retb;

retb = nflog_nlmsg_parse(ph, attrs);
if (retb != MNL_CB_OK) {
    printf("something went wrong");
    printf(" (%s)\n", strerror(errno));
    return retb;
}

Re: ulogd packet based logging with CT info

[Pablo Neira Ayuso]

Hi Blažej,

On Wed, Aug 18, 2021 at 12:06:40PM +0200, Blažej Krajňák wrote:
> st 18. 8. 2021 o 9:23 Pablo Neira Ayuso <pablo@netfilter.org> napísal(a):
> > You need this kernel patch to add this information to nfnetlink_queue,
> > compile-tested only.
> 
> Hey Pablo,
> patch is working like a charm. Thank you! Now I see timestamps and
> packets/bytes counters in JSON output correctly.
> At the end I will post customized input plugin for everyone.
> 
> The last thing I want to ask is what's correct way to get
> 
> attrs[NFULA_CT]
> 
> from
> 
> struct nfulnl_msg_packet_hdr *ph = nflog_get_msg_packet_hdr(ldata);
> 
> 
> Now I use the following code which is working but throwing random
> errors "something went wrong (Numerical result out of range)" I'm
> mixing struct nfulnl_msg_packet_hdr with const struct nlmsghdr *nlh

That might be a bug in nflog_nlmsg_parse(): maybe
nflog_parse_attr_cb() is finding a mismatch in the datatype of the
attribute payload.

Could you have a look at what attribute is hitting this error?

> struct nfulnl_msg_packet_hdr *ph = nflog_get_msg_packet_hdr(ldata);
> struct nlattr *attrs[NFULA_MAX + 1] = { NULL };
> int retb;
> 
> retb = nflog_nlmsg_parse(ph, attrs);
> if (retb != MNL_CB_OK) {
>     printf("something went wrong");
>     printf(" (%s)\n", strerror(errno));
>     return retb;
> }

Re: ulogd packet based logging with CT info

[Blažej Krajňák]

I'm really confused from searching a bug.

Getting nf_conntrack via nflog_nlmsg_parse(ph, attrs); is (I think)
bad because ph parameter must be nlmsghdr not nfulnl_msg_packet_hdr

So different way. I added new getters to libnetfilter_log.c:

struct nf_conntrack *nflog_get_ct(struct nflog_data *nfad)
{
    return nfnl_get_pointer_to_data(nfad->nfa, NFULA_CT, struct nf_conntrack);
}

uint32_t nflog_get_ct_info(struct nflog_data *nfad)
{
return ntohs(nfnl_get_data(nfad->nfa, NFULA_CT_INFO, uint32_t));
}

nflog_get_ct_info works correctly. But from nflog_get_ct I'm unable to
read anything.

struct nf_conntrack *ct = nflog_get_ct(ldata);
printf("test value %" PRIu64 " /n", ct->timestamp.start);

is returning random numbers. What am I missing? How to properly parse
output of nflog_get_ct to be able to use for ex. nfct_get_attr_u32?

st 18. 8. 2021 o 13:52 Pablo Neira Ayuso <pablo@netfilter.org> napísal(a):
>
> Hi Blažej,
>
> On Wed, Aug 18, 2021 at 12:06:40PM +0200, Blažej Krajňák wrote:
> > st 18. 8. 2021 o 9:23 Pablo Neira Ayuso <pablo@netfilter.org> napísal(a):
> > > You need this kernel patch to add this information to nfnetlink_queue,
> > > compile-tested only.
> >
> > Hey Pablo,
> > patch is working like a charm. Thank you! Now I see timestamps and
> > packets/bytes counters in JSON output correctly.
> > At the end I will post customized input plugin for everyone.
> >
> > The last thing I want to ask is what's correct way to get
> >
> > attrs[NFULA_CT]
> >
> > from
> >
> > struct nfulnl_msg_packet_hdr *ph = nflog_get_msg_packet_hdr(ldata);
> >
> >
> > Now I use the following code which is working but throwing random
> > errors "something went wrong (Numerical result out of range)" I'm
> > mixing struct nfulnl_msg_packet_hdr with const struct nlmsghdr *nlh
>
> That might be a bug in nflog_nlmsg_parse(): maybe
> nflog_parse_attr_cb() is finding a mismatch in the datatype of the
> attribute payload.
>
> Could you have a look at what attribute is hitting this error?
>
> > struct nfulnl_msg_packet_hdr *ph = nflog_get_msg_packet_hdr(ldata);
> > struct nlattr *attrs[NFULA_MAX + 1] = { NULL };
> > int retb;
> >
> > retb = nflog_nlmsg_parse(ph, attrs);
> > if (retb != MNL_CB_OK) {
> >     printf("something went wrong");
> >     printf(" (%s)\n", strerror(errno));
> >     return retb;
> > }

Re: ulogd packet based logging with CT info

[Pablo Neira Ayuso]

On Wed, Aug 18, 2021 at 10:06:40PM +0200, Blažej Krajňák wrote:
> I'm really confused from searching a bug.
> 
> Getting nf_conntrack via nflog_nlmsg_parse(ph, attrs); is (I think)
> bad because ph parameter must be nlmsghdr not nfulnl_msg_packet_hdr

Right, nflog_nlmsg_parse() should take the nlh parameter.

> So different way. I added new getters to libnetfilter_log.c:
> 
> struct nf_conntrack *nflog_get_ct(struct nflog_data *nfad)
> {
>     return nfnl_get_pointer_to_data(nfad->nfa, NFULA_CT, struct nf_conntrack);
> }

This will not work (as you noticed). The kernel does not store a
struct in the NFULA_CT attribute.

Better to stick to use nflog_nlmsg_parser(), my suggestion is:

#1 msg_cb() provides struct nfgenmsg *nfmsg, you could retrieve the nlmsg
   from there since the nlmsghdr comes before nfgenmsg:

        struct nlmsghdr *nlh;

        nlh = (struct nlmsghdr *)((void *)nfg - sizeof(*nlh));

        err = nflog_nlmsg_parse(nlh, attrs);
        if (err < 0)
                ... error path

#2 once you have access to attrs[NFULA_CT], from there on:

        struct nf_conntrack *ct;

        ct = nfct_new();
        if (!ct)
                ... error path

        err = nfct_nlmsg_parse(nlh, ct);
        if (err < 0)
                ... error path

Then, you get the pointer to conntrack object.

Re: ulogd packet based logging with CT info

[Blažej Krajňák]

št 19. 8. 2021 o 12:16 Pablo Neira Ayuso <pablo@netfilter.org> napísal(a):
>
> Better to stick to use nflog_nlmsg_parser(), my suggestion is:
>
> #1 msg_cb() provides struct nfgenmsg *nfmsg, you could retrieve the nlmsg
>    from there since the nlmsghdr comes before nfgenmsg:
>
>         struct nlmsghdr *nlh;
>
>         nlh = (struct nlmsghdr *)((void *)nfg - sizeof(*nlh));
>
>         err = nflog_nlmsg_parse(nlh, attrs);
>         if (err < 0)
>                 ... error path
>
> #2 once you have access to attrs[NFULA_CT], from there on:
>
>         struct nf_conntrack *ct;
>
>         ct = nfct_new();
>         if (!ct)
>                 ... error path
>
>         err = nfct_nlmsg_parse(nlh, ct);
>         if (err < 0)
>                 ... error path
>
> Then, you get the pointer to conntrack object.

Great, your suggestions perfectly work. Thank you.
Little later I will post complete code to everyone.

Could it be useful to prepare patch to add this to ulogd2? As new
input plugin or as a upgrade to inppkt_UFLOG?

Re: ulogd packet based logging with CT info

[Pablo Neira Ayuso]

On Thu, Aug 19, 2021 at 04:05:41PM +0200, Blažej Krajňák wrote:
> št 19. 8. 2021 o 12:16 Pablo Neira Ayuso <pablo@netfilter.org> napísal(a):
> >
> > Better to stick to use nflog_nlmsg_parser(), my suggestion is:
> >
> > #1 msg_cb() provides struct nfgenmsg *nfmsg, you could retrieve the nlmsg
> >    from there since the nlmsghdr comes before nfgenmsg:
> >
> >         struct nlmsghdr *nlh;
> >
> >         nlh = (struct nlmsghdr *)((void *)nfg - sizeof(*nlh));
> >
> >         err = nflog_nlmsg_parse(nlh, attrs);
> >         if (err < 0)
> >                 ... error path
> >
> > #2 once you have access to attrs[NFULA_CT], from there on:
> >
> >         struct nf_conntrack *ct;
> >
> >         ct = nfct_new();
> >         if (!ct)
> >                 ... error path
> >
> >         err = nfct_nlmsg_parse(nlh, ct);
> >         if (err < 0)
> >                 ... error path
> >
> > Then, you get the pointer to conntrack object.
> 
> Great, your suggestions perfectly work. Thank you.
> Little later I will post complete code to everyone.

Thanks.

> Could it be useful to prepare patch to add this to ulogd2?

I think so, yes.

> As new input plugin or as a upgrade to inppkt_UFLOG?

Better if you integrate it into the existing plugin.

Please, go ahead post it for review, it might just need a few
iterations before it gets merged into master.

Thanks.

Re: ulogd packet based logging with CT info

[Blažej Krajňák]

I'm not sure if previous message was delivered to the list members (I
don't see it on web archive).
So once again:

št 19. 8. 2021 o 19:03 Pablo Neira Ayuso <pablo@netfilter.org> napísal(a):
>
> Better if you integrate it into the existing plugin.
>
> Please, go ahead post it for review, it might just need a few
> iterations before it gets merged into master.
>
> Thanks.

Okay, I will prepare my first ever patch to open source world :)

I just found an another strange behaviour of conntrack. I'm mirroring
port on switch and mirrored data are coming to Linux server. That port
on server is in bridge. In nftables I created table bridge filter with
some CT rule to enable connection tracking on bridge.
As I found I had to add another dummy interface to bridge, because
conntrack was not working at all, if just one port in bridge.
Now I see conntrack entries but all of them as UNREPLIED and just one
way byte/packet counters are increasing (see attachment). Is it
because the both ways are coming to server on the same port? Any easy
workaround?

https://drive.google.com/file/d/1-aIXA13IicHcKHIaxkC1Hz2tRckU3YDm/view?usp=sharing

Re: ulogd packet based logging with CT info

[Blažej Krajňák]

so 21. 8. 2021 o 15:03 Blažej Krajňák <blazej.krajnak@gmail.com> napísal(a):
> I just found an another strange behaviour of conntrack. I'm mirroring
> port on switch and mirrored data are coming to Linux server. That port
> on server is in bridge. In nftables I created table bridge filter with
> some CT rule to enable connection tracking on bridge.
> As I found I had to add another dummy interface to bridge, because
> conntrack was not working at all, if just one port in bridge.
> Now I see conntrack entries but all of them as UNREPLIED and just one
> way byte/packet counters are increasing (see attachment). Is it
> because the both ways are coming to server on the same port? Any easy
> workaround?

I just figure it out. Switch was sending one way of traffic tagged
with vlan. And also I had to turn off bridge MAC addresses learning
(a.k.a ageing to 0). Now it works.

Re: ulogd packet based logging with CT info

[Ken-ichirou MATSUZAWA]

 Hi,

On 08/19, Pablo Neira Ayuso wrote:
> On Thu, Aug 19, 2021 at 04:05:41PM +0200, Blažej Krajňák wrote:
> > Could it be useful to prepare patch to add this to ulogd2?
> 
> I think so, yes.
> 
> > As new input plugin or as a upgrade to inppkt_UFLOG?
> 
> Better if you integrate it into the existing plugin.

May I follow up on this?

thanks,