[libnetfilter_queue] extra data after payload

[libnetfilter_queue] extra data after payload

[U.Mutlu]

Hi,

when reading queue data via the recv() function
then one gets a return value much longer than the payload data,
so there are some extra data after the payload.
What kind of extra data is it?

for example:
   rv  = recv(fd, buf, sizeof(buf), 0);   // rv=84
   ...
   ret = nfq_get_payload(tb, &data);      // ret=40 (ie. ip + tcp pkt, both w/o options, and tcp w/o user data)

So, here, what are the extra 44 bytes after the tcp data?

Re: [libnetfilter_queue] extra data after payload

[U.Mutlu]

U.Mutlu wrote, On 2011-09-23 18:19:
> Hi,
>
> when reading queue data via the recv() function
> then one gets a return value much longer than the payload data,
> so there are some extra data after the payload.
> What kind of extra data is it?
>
> for example:
> rv = recv(fd, buf, sizeof(buf), 0); // rv=84
> ...
> ret = nfq_get_payload(tb, &data); // ret=40 (ie. ip + tcp pkt, both w/o options, and tcp w/o user data)
>
> So, here, what are the extra 44 bytes after the tcp data?

Correction: the extra data is in front of the payload.

RE: [libnetfilter_queue] extra data after payload

[Jeff Haran]

> -----Original Message-----
> From: netfilter-devel-owner@vger.kernel.org [mailto:netfilter-devel-
> owner@vger.kernel.org] On Behalf Of U.Mutlu
> Sent: Friday, September 23, 2011 9:20 AM
> To: netfilter-devel@vger.kernel.org
> Subject: [libnetfilter_queue] extra data after payload
> 
> Hi,
> 
> when reading queue data via the recv() function
> then one gets a return value much longer than the payload data,
> so there are some extra data after the payload.
> What kind of extra data is it?
> 
> for example:
>    rv  = recv(fd, buf, sizeof(buf), 0);   // rv=84
>    ...
>    ret = nfq_get_payload(tb, &data);      // ret=40 (ie. ip + tcp pkt, both w/o
> options, and tcp w/o user data)
> 
> So, here, what are the extra 44 bytes after the tcp data?

I believe you will find there is a struct nlmsghdr at  the beginning of the data, before the IP header, followed by other netlink structures. Take a look at net/netfilter/nfnetlink_queue.c:nfqnl_build_packet_message() in your kernel source tree for the details. The messages containing packets contain a 768 in the (struct nlmsghdr *)->nlmsg_type field at the beginning of the message, which corresponds to NFNL_SUBSYS_QUEUE << 8 | NFQNL_MSG_PACKET.

Note there appear to be other message types on these sockets. I've seen messages with nlmsg_type == 0 coming off these NFQUEUE sockets too, which apparently contain something other than IP packets and for which you won't get a callback when you call nfq_handle_packet(). So your code shouldn't depend on a 1 to 1 relationship between calls to nfq_handle_packet() and the callback you register via nfq_create_queue().

At least that's what I've observed after having spent the last couple of days playing with this.

Jeff Haran

Re: [libnetfilter_queue] extra data after payload

[U.Mutlu]

Jeff Haran wrote, On 2011-09-23 19:17:
>> -----Original Message-----
>> From: netfilter-devel-owner@vger.kernel.org [mailto:netfilter-devel-
>> owner@vger.kernel.org] On Behalf Of U.Mutlu
>> Sent: Friday, September 23, 2011 9:20 AM
>> To: netfilter-devel@vger.kernel.org
>> Subject: [libnetfilter_queue] extra data after payload
>>
>> Hi,
>>
>> when reading queue data via the recv() function
>> then one gets a return value much longer than the payload data,
>> so there are some extra data after the payload.
>> What kind of extra data is it?
>>
>> for example:
>>     rv  = recv(fd, buf, sizeof(buf), 0);   // rv=84
>>     ...
>>     ret = nfq_get_payload(tb,&data);      // ret=40 (ie. ip + tcp pkt, both w/o
>> options, and tcp w/o user data)
>>
>> So, here, what are the extra 44 bytes after the tcp data?
>
> I believe you will find there is a struct nlmsghdr at  the beginning of the data, before the IP header, followed by other netlink structures. Take a look at net/netfilter/nfnetlink_queue.c:nfqnl_build_packet_message() in your kernel source tree for the details. The messages containing packets contain a 768 in the (struct nlmsghdr *)->nlmsg_type field at the beginning of the message, which corresponds to NFNL_SUBSYS_QUEUE<<  8 | NFQNL_MSG_PACKET.
>
> Note there appear to be other message types on these sockets. I've seen messages with nlmsg_type == 0 coming off these NFQUEUE sockets too, which apparently contain something other than IP packets and for which you won't get a callback when you call nfq_handle_packet(). So your code shouldn't depend on a 1 to 1 relationship between calls to nfq_handle_packet() and the callback you register via nfq_create_queue().
>
> At least that's what I've observed after having spent the last couple of days playing with this.

nlmsghdr has size 16, I've not figured out yet what comes after it before the (optional) payload starts.
There are some interessting other fields besides len and type, in nlmsghdr, like pid,
but they seem to be not filled.
On my system I found it under the kernel sources:
debian/linux-headers-2.6.37.6-my1a/usr/src/linux-headers-2.6.37.6-my1a/include/linux/netlink.h

Studying the sources is not easy, too low level stuff in libnfnetlink.

Ok, I see some of the extra data can be accessed via the "Message parsing functions":
u_int32_t 	nfq_get_indev (struct nfq_data *nfad)
int 	nfq_get_indev_name (struct nlif_handle *nlif_handle, struct nfq_data *nfad, char *name)
struct nfqnl_msg_packet_hdr * 	nfq_get_msg_packet_hdr (struct nfq_data *nfad)
uint32_t 	nfq_get_nfmark (struct nfq_data *nfad)
u_int32_t 	nfq_get_outdev (struct nfq_data *nfad)
int 	nfq_get_outdev_name (struct nlif_handle *nlif_handle, struct nfq_data *nfad, char *name)
struct nfqnl_msg_packet_hw * 	nfq_get_packet_hw (struct nfq_data *nfad)
int 	nfq_get_payload (struct nfq_data *nfad, unsigned char **data)
u_int32_t 	nfq_get_physindev (struct nfq_data *nfad)
int 	nfq_get_physindev_name (struct nlif_handle *nlif_handle, struct nfq_data *nfad, char *name)
u_int32_t 	nfq_get_physoutdev (struct nfq_data *nfad)
int 	nfq_get_physoutdev_name (struct nlif_handle *nlif_handle, struct nfq_data *nfad, char *name)
int 	nfq_get_timestamp (struct nfq_data *nfad, struct timeval *tv)
int 	nfq_snprintf_xml (char *buf, size_t rem, struct nfq_data *tb, int flags)

Re: [libnetfilter_queue] extra data after payload

[Jan Engelhardt]

On Saturday 2011-09-24 10:08, U.Mutlu wrote:
>>> for example:
>>>    rv  = recv(fd, buf, sizeof(buf), 0);   // rv=84
>>>    ...
>>>    ret = nfq_get_payload(tb,&data);      // ret=40 (ie. ip + tcp pkt, both
>>> w/o
>>> options, and tcp w/o user data)
>>>
>>> So, here, what are the extra 44 bytes after the tcp data?
>>
>> I believe you will find there is a struct nlmsghdr at  the beginning of the
>> data, before the IP header
>
> nlmsghdr has size 16, I've not figured out yet what comes after it before the
> (optional) payload starts.

(It's struct nlmsghdr, struct nfgenmsg, struct nfqnl_msg_packet_hdr.)

Suffice to say that, what buf actually contains is implementation-defined, and
"uninteresting", since you chose to use a library to deal with it for you,
abstracting the whole thing.

RE: [libnetfilter_queue] extra data after payload

[Jeff Haran]

> -----Original Message-----
> From: netfilter-devel-owner@vger.kernel.org [mailto:netfilter-devel-
> owner@vger.kernel.org] On Behalf Of Jan Engelhardt
> Sent: Saturday, September 24, 2011 1:24 AM
> To: U.Mutlu
> Cc: netfilter-devel@vger.kernel.org
> Subject: Re: [libnetfilter_queue] extra data after payload
> 
> 
> On Saturday 2011-09-24 10:08, U.Mutlu wrote:
> >>> for example:
> >>>    rv  = recv(fd, buf, sizeof(buf), 0);   // rv=84
> >>>    ...
> >>>    ret = nfq_get_payload(tb,&data);      // ret=40 (ie. ip + tcp
pkt, both
> >>> w/o
> >>> options, and tcp w/o user data)
> >>>
> >>> So, here, what are the extra 44 bytes after the tcp data?
> >>
> >> I believe you will find there is a struct nlmsghdr at  the
beginning of the
> >> data, before the IP header
> >
> > nlmsghdr has size 16, I've not figured out yet what comes after it
before
> the
> > (optional) payload starts.
> 
> (It's struct nlmsghdr, struct nfgenmsg, struct nfqnl_msg_packet_hdr.)
> 
> Suffice to say that, what buf actually contains is
implementation-defined, and
> "uninteresting", since you chose to use a library to deal with it for
you,
> abstracting the whole thing.

Well, not necessarily "uninteresting". Some people like to understand
how something they are using works.

Plus, if you don't know how big these headers are, you can you know how
big a buffer you need to read the message off of the socket into without
truncating the IP packet at the end?

The sample code at
http://www.netfilter.org/projects/libnetfilter_queue/doxygen/group__Queu
e.html shows a call to recv() made into an undeclared variable named
"buf" with the amount of data to be read specified as "sizeof(buf)".

I don't see anything in the API to call to tell the user this and I
haven't seen it in the documentation either, though perhaps I've missed
it.

Thanks,

Jeff Haran

Re: [libnetfilter_queue] extra data after payload

[U.Mutlu]

Jeff Haran wrote, On 2011-09-26 19:45:
>> -----Original Message-----
>> From: netfilter-devel-owner@vger.kernel.org [mailto:netfilter-devel-
>> owner@vger.kernel.org] On Behalf Of Jan Engelhardt
>> Sent: Saturday, September 24, 2011 1:24 AM
>> To: U.Mutlu
>> Cc: netfilter-devel@vger.kernel.org
>> Subject: Re: [libnetfilter_queue] extra data after payload
>>
>> On Saturday 2011-09-24 10:08, U.Mutlu wrote:
>>>>> for example:
>>>>>     rv  = recv(fd, buf, sizeof(buf), 0);   // rv=84
>>>>>     ...
>>>>>     ret = nfq_get_payload(tb,&data);      // ret=40 (ie. ip + tcp
> pkt, both
>>>>> w/o
>>>>> options, and tcp w/o user data)
>>>>>
>>>>> So, here, what are the extra 44 bytes after the tcp data?
>>>>
>>>> I believe you will find there is a struct nlmsghdr at  the
> beginning of the
>>>> data, before the IP header
>>>
>>> nlmsghdr has size 16, I've not figured out yet what comes after it
> before
>> the
>>> (optional) payload starts.
>>
>> (It's struct nlmsghdr, struct nfgenmsg, struct nfqnl_msg_packet_hdr.)
>>
>> Suffice to say that, what buf actually contains is
> implementation-defined, and
>> "uninteresting", since you chose to use a library to deal with it for
> you,
>> abstracting the whole thing.
>
> Well, not necessarily "uninteresting". Some people like to understand
> how something they are using works.
>
> Plus, if you don't know how big these headers are, you can you know how
> big a buffer you need to read the message off of the socket into without
> truncating the IP packet at the end?
>
> The sample code at
> http://www.netfilter.org/projects/libnetfilter_queue/doxygen/group__Queu
> e.html shows a call to recv() made into an undeclared variable named
> "buf" with the amount of data to be read specified as "sizeof(buf)".
>
> I don't see anything in the API to call to tell the user this and I
> haven't seen it in the documentation either, though perhaps I've missed
> it.

One can get the header size by subtracting the payload start from the
buffer start. So, the header(s) seems to have the mentioned total len
of 44 bytes, but not sure if that's true always.

BTW, since the sample above contains this func call:
   nfq_set_mode(qh, NFQNL_COPY_PACKET, 0xffff)
it then practically means that buf must be >= 64k-1+hdrsize, ie. at least 65579 bytes...

Re: [libnetfilter_queue] extra data after payload

[U.Mutlu]

U.Mutlu wrote, On 2011-09-26 20:58:
> Jeff Haran wrote, On 2011-09-26 19:45:
>>> -----Original Message-----
>>> From: netfilter-devel-owner@vger.kernel.org [mailto:netfilter-devel-
>>> owner@vger.kernel.org] On Behalf Of Jan Engelhardt
>>> Sent: Saturday, September 24, 2011 1:24 AM
>>> To: U.Mutlu
>>> Cc: netfilter-devel@vger.kernel.org
>>> Subject: Re: [libnetfilter_queue] extra data after payload
>>>
>>> On Saturday 2011-09-24 10:08, U.Mutlu wrote:
>>>>>> for example:
>>>>>> rv = recv(fd, buf, sizeof(buf), 0); // rv=84
>>>>>> ...
>>>>>> ret = nfq_get_payload(tb,&data); // ret=40 (ie. ip + tcp
>> pkt, both
>>>>>> w/o
>>>>>> options, and tcp w/o user data)
>>>>>>
>>>>>> So, here, what are the extra 44 bytes after the tcp data?
>>>>>
>>>>> I believe you will find there is a struct nlmsghdr at the
>> beginning of the
>>>>> data, before the IP header
>>>>
>>>> nlmsghdr has size 16, I've not figured out yet what comes after it
>> before
>>> the
>>>> (optional) payload starts.
>>>
>>> (It's struct nlmsghdr, struct nfgenmsg, struct nfqnl_msg_packet_hdr.)
>>>
>>> Suffice to say that, what buf actually contains is
>> implementation-defined, and
>>> "uninteresting", since you chose to use a library to deal with it for
>> you,
>>> abstracting the whole thing.
>>
>> Well, not necessarily "uninteresting". Some people like to understand
>> how something they are using works.
>>
>> Plus, if you don't know how big these headers are, you can you know how
>> big a buffer you need to read the message off of the socket into without
>> truncating the IP packet at the end?
>>
>> The sample code at
>> http://www.netfilter.org/projects/libnetfilter_queue/doxygen/group__Queu
>> e.html shows a call to recv() made into an undeclared variable named
>> "buf" with the amount of data to be read specified as "sizeof(buf)".
>>
>> I don't see anything in the API to call to tell the user this and I
>> haven't seen it in the documentation either, though perhaps I've missed
>> it.
>
> One can get the header size by subtracting the payload start from the
> buffer start.

FIX: of course the other way around... :-)

> So, the header(s) seems to have the mentioned total len
> of 44 bytes, but not sure if that's true always.
>
> BTW, since the sample above contains this func call:
> nfq_set_mode(qh, NFQNL_COPY_PACKET, 0xffff)
> it then practically means that buf must be >= 64k-1+hdrsize, ie. at least 65579 bytes...