Packet statistics accounting module: request for comments

Packet statistics accounting module: request for comments

[Julien Lafaye]

Hello,

I created a small packet statistics accounting module based on the
xtables framework. It is based on a FIFO that is filled in netfilter
hooks and emptied by a user program reading statistics through /proc
callbacks. The goal of the module is to be able to get maximum traffic
rate with an improved resolution compared to basic techniques. Typical
expected resolution is 100ms.

The code is currently stored in github: https://github.com/jlafaye/xt_pktstat

I would like the code to be considered for inclusion in the
xtables-addons framework. Nevertheless, the code might not be mature
enough as this is the first time I do kernel programming. Although
advised by knowledgeable persons, there are probably many issues in
it.

I would greatly appreciate is somebody on this list is able to give me
pointers and people to contact to actually improve my code and get it
ready for xtables-addons inclusion.

Regards,

Julien

PS: the guide 'How to write Netfilter modules' was great.

Re: Packet statistics accounting module: request for comments

[Jan Engelhardt]

On Tuesday 2011-10-04 06:50, Julien Lafaye wrote:
>
>I created a small packet statistics accounting module based on the 
>xtables framework. It is based on a FIFO that is filled in netfilter 
>hooks and emptied by a user program reading statistics through /proc 
>callbacks. The goal of the module is to be able to get maximum traffic 
>rate with an improved resolution compared to basic techniques. Typical 
>expected resolution is 100ms.

I am thinking of how xt_quota2 (in grow mode) could have been enhanced 
to simply reset the counter when the procfs file is read. How would that 
have compared to your module?

>The code is currently stored in github: 
>https://github.com/jlafaye/xt_pktstat
>
>I would like the code to be considered for inclusion in the 
>xtables-addons framework. Nevertheless, the code might not be mature 
>enough as this is the first time I do kernel programming. Although 
>advised by knowledgeable persons, there are probably many issues in it.

The .c files need a little license-mentioning header at the least, and 
should in principle also use tab indents like LKCS.-See files from Xt-a 
for examples.

Re: Packet statistics accounting module: request for comments

[Julien Lafaye]

On Tue, Oct 4, 2011 at 1:02 PM, Jan Engelhardt <jengelh@medozas.de> wrote:
> On Tuesday 2011-10-04 06:50, Julien Lafaye wrote:
>>
>>I created a small packet statistics accounting module based on the
>>xtables framework. It is based on a FIFO that is filled in netfilter
>>hooks and emptied by a user program reading statistics through /proc
>>callbacks. The goal of the module is to be able to get maximum traffic
>>rate with an improved resolution compared to basic techniques. Typical
>>expected resolution is 100ms.
>
> I am thinking of how xt_quota2 (in grow mode) could have been enhanced
> to simply reset the counter when the procfs file is read. How would that
> have compared to your module?

My objective was to count incoming packets & bytes received during
short time frames, i.e. 100ms even 10ms. I first thought of having a
user program polling statistics but I would have to rely on accurate
scheduling of the polling process by the OS. I wanted to be able to
deploy the accounting module on Centos 5 kernels (2.6.18) which does
not have the dynticks features and no guarantee that the polling
process will be awakened when I wanted it to be. So, I chose to build
time buckets directly in the softirq handler and export them with
/proc. I agree that polling the xt_quota2 counter and resetting at
each poll will achieve the same functionality but I would run tests to
check that polling is actually scheduled at regular enough intervals.

>>The code is currently stored in github:
>>https://github.com/jlafaye/xt_pktstat
>>
>>I would like the code to be considered for inclusion in the
>>xtables-addons framework. Nevertheless, the code might not be mature
>>enough as this is the first time I do kernel programming. Although
>>advised by knowledgeable persons, there are probably many issues in it.
>
> The .c files need a little license-mentioning header at the least, and
> should in principle also use tab indents like LKCS.-See files from Xt-a
> for examples.

OK

Re: Packet statistics accounting module: request for comments

[Jan Engelhardt]

On Tuesday 2011-10-04 21:03, Julien Lafaye wrote:
>
>My objective was to count incoming packets & bytes received during 
>short time frames,[...] So, I chose to build time buckets directly in 
>the softirq handler and export them with /proc. I agree that polling 
>the xt_quota2 counter and resetting at each poll will achieve the same 
>functionality but I would run tests to check that polling is actually 
>scheduled at regular enough intervals.

I still think this could be integrated, perhaps as an extra option to 
choose a particular recording mode, to xt_quota2 (including the fifo).
Speaking of which, your module seems to lack locking on the ctx, so it's 
not very consistent on SMP as of yet.

Re: Packet statistics accounting module: request for comments

[Julien Lafaye]

Jan

> I still think this could be integrated, perhaps as an extra option to
> choose a particular recording mode, to xt_quota2 (including the fifo).
> Speaking of which, your module seems to lack locking on the ctx, so it's
> not very consistent on SMP as of yet.
>

I will fix the usage of the fifo so that there is not any odd behavior
in SMP context.

As for the inclusion into xt_quota2, I am bit skeptical, for three reasons:

- the two modules, xt_quota2  and xt_pktstat, have different features:
quota enforcement and poll-based network accounting for the first one,
and event driven high resolution network accounting for the second
one. Where I agree that the two modules could share portions of code,
I don't see how the core parts can be merged. I fear that merging the
modules would require putting the two codes side by side and not
actually merging them. But you have more experience than me on
netfilter modules, and obviously on xt_quota2, and you could give me
some hints on how to merge them in a way that would benefit the two
projects.

- xt_quota2 is clearly not a good name for the feature I propose.

- since the features provided by xt_quota2 and xt_pktstat are
separated, why trying to join them into a swiss-army knife with two
sets of parameters: one to be used for a first feature and the other
one for a second feature. This could prove to be misleading for the
user.

What do you think ?

Regards,

Julien