New/Updated L7 netfilter option - nDPI

New/Updated L7 netfilter option - nDPI

[Ed W]

Hi all.  There is an interesting project that was called opendpi 
(originally by ipoque GmbH) and recently been forked and maintained by 
the ntop guys under the nDPI label.  It offers a new and currently 
maintained layer 7 (L7) packet identification library.  It could 
definitely benefit from more eyes and development effort, but at present 
it gives much better breakdown of traffic for ntop

There is a netfilter library, originally by Elian Gidoni, that I have 
updated to use the nDPI fork

     https://github.com/ewildgoose/ndpi-netfilter



The practical upshot is that you can do stuff like:

     iptables -I FORWARD -m opendpi --WinUpdate -j LOG
or
     iptables -I FORWARD -m opendpi --skype -j REJECT

In theory you can also filter Facebook, Twitter, etc, but I concede that 
doesn't seem to work as expected right now...

Another of the clever things that nDPI does is to try and classify SSL 
traffic by examining the name on the cert.  A technique that seems 
likely to allow crude identification of significant traffic.


We could benefit from more eyes on this, both the netfilter module and 
the nDPI library

Thanks for your feedback

Ed W

Re: New/Updated L7 netfilter option - nDPI

[Andrew Beverley]

On Wed, 2012-10-24 at 23:53 +0100, Ed W wrote:
> Hi all.  There is an interesting project that was called opendpi 
> (originally by ipoque GmbH) and recently been forked and maintained by 
> the ntop guys under the nDPI label.  It offers a new and currently 
> maintained layer 7 (L7) packet identification library.

That's great news.

I had a play with l7-filter some time ago, which I assume is similar to
nDPI. How do the 2 projects compare?

I have to admit that I only had limited success with l7-filter, although
it no longer appears to be maintained anyway.

Andy

Re: New/Updated L7 netfilter option - nDPI

[Eliezer Croitoru]

On 10/28/2012 10:20 AM, Andrew Beverley wrote:
> That's great news.
>
> I had a play with l7-filter some time ago, which I assume is similar to
> nDPI. How do the 2 projects compare?
>
> I have to admit that I only had limited success with l7-filter, although
> it no longer appears to be maintained anyway.
>
> Andy
What would you want to achieve from a using l7 iptables?
filtering? scheduling?

Eliezer

-- 
Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer <at> ngtech.co.il

Re: New/Updated L7 netfilter option - nDPI

[Andrew Beverley]

On Sun, 2012-10-28 at 16:57 +0200, Eliezer Croitoru wrote:
> > I have to admit that I only had limited success with l7-filter, although
> > it no longer appears to be maintained anyway.
> >
> What would you want to achieve from a using l7 iptables?
> filtering? scheduling?

At the time I was using it to do traffic shaping, to prevent p2p
applications overloading a network with low bandwidth internet
connection. The problem was that it only needed one p2p application to
not be identified for the network to be overloaded. So in the end I took
a rather rudimentary approach and just identified any client making lots
of connections to ports above 1024:

http://www.andybev.com/index.php/Fair_traffic_shaping_an_ADSL_line_for_a_local_network_using_Linux

Andy

Re: New/Updated L7 netfilter option - nDPI

[Ed W]

On 01/11/2012 22:03, Andrew Beverley wrote:
> On Sun, 2012-10-28 at 16:57 +0200, Eliezer Croitoru wrote:
>>> I have to admit that I only had limited success with l7-filter, although
>>> it no longer appears to be maintained anyway.
>>>
>> What would you want to achieve from a using l7 iptables?
>> filtering? scheduling?
> At the time I was using it to do traffic shaping, to prevent p2p
> applications overloading a network with low bandwidth internet
> connection. The problem was that it only needed one p2p application to
> not be identified for the network to be overloaded. So in the end I took
> a rather rudimentary approach and just identified any client making lots
> of connections to ports above 1024:
>
> http://www.andybev.com/index.php/Fair_traffic_shaping_an_ADSL_line_for_a_local_network_using_Linux
>
>


I think it's safe to assume that at least a determined attacker can 
avoid these filters. Ideally you want them reasonably accurate for the 
normal situation...

I guess you just invented an "L7 Filter" yourself... It's just as good a 
match for certain requirements...!

Let me know if you measure this thing against your problem?

Cheers

Ed W

Re: New/Updated L7 netfilter option - nDPI

[Lutfi ODUNCUOGLU]

Hello,

I compiled nDPI-nefilter patch and it works fine. What I want is to 
shape the p2p traffic in my network. For this purpose i just implemented 
the nDPI-netfilter patch as two different ways for testing

  iptables -t mangle -A POSTROUTING -o XXX -m ndpi --bittorrent -j 
CONNMARK --set-mark 1


iptables -t mangle -A POSTROUTING -m connmark --mark 1  -j CLASSIFY 
--set-class 0001:0010

or

iptables -t mangle -A POSTROUTING -m ndpi --bittorrent  -j CLASSIFY 
--set-class 0001:0010


So which one is more suitable for use? I don't know if this patch 
inspects connections (marks connection) or every single packet (marks 
every single) for a match.


Regards,

Lutfi


On 11/02/2012 12:56 AM, Ed W wrote:
> On 01/11/2012 22:03, Andrew Beverley wrote:
>> On Sun, 2012-10-28 at 16:57 +0200, Eliezer Croitoru wrote:
>>>> I have to admit that I only had limited success with l7-filter,
>>>> although
>>>> it no longer appears to be maintained anyway.
>>>>
>>> What would you want to achieve from a using l7 iptables?
>>> filtering? scheduling?
>> At the time I was using it to do traffic shaping, to prevent p2p
>> applications overloading a network with low bandwidth internet
>> connection. The problem was that it only needed one p2p application to
>> not be identified for the network to be overloaded. So in the end I took
>> a rather rudimentary approach and just identified any client making lots
>> of connections to ports above 1024:
>>
>> http://www.andybev.com/index.php/Fair_traffic_shaping_an_ADSL_line_for_a_local_network_using_Linux
>>
>>
>>
>
>
> I think it's safe to assume that at least a determined attacker can
> avoid these filters. Ideally you want them reasonably accurate for the
> normal situation...
>
> I guess you just invented an "L7 Filter" yourself... It's just as good a
> match for certain requirements...!
>
> Let me know if you measure this thing against your problem?
>
> Cheers
>
> Ed W
> _______________________________________________
> Ntop-dev mailing list
> Ntop-dev@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-dev

Re: New/Updated L7 netfilter option - nDPI

[Ed W]

On 02/11/2012 12:38, Lutfi ODUNCUOGLU wrote:
> Hello,
>
> I compiled nDPI-nefilter patch and it works fine. What I want is to 
> shape the p2p traffic in my network. For this purpose i just 
> implemented the nDPI-netfilter patch as two different ways for testing
>
>  iptables -t mangle -A POSTROUTING -o XXX -m ndpi --bittorrent -j 
> CONNMARK --set-mark 1
>
>
> iptables -t mangle -A POSTROUTING -m connmark --mark 1  -j CLASSIFY 
> --set-class 0001:0010
>
> or
>
> iptables -t mangle -A POSTROUTING -m ndpi --bittorrent  -j CLASSIFY 
> --set-class 0001:0010
>
>
> So which one is more suitable for use? I don't know if this patch 
> inspects connections (marks connection) or every single packet (marks 
> every single) for a match.

I would be interested in your feedback on CPU usage for both actually?

Internally nDPI implements a "connection cache", which means that it 
asks conntrack to maintain the connection, but internally it maintains a 
cache of what protocol is matched to a given connection.  This should be 
quite similar to asking conntrack to --set-mark *if* you also have some 
iptables to subsequently skip the test once the connection is marked?

So, either way, the connection gets marked at some point and thereafter 
no CPU is expended checking the connection further.  Of course theory 
and practice aren't the same...

Just beware that the connection may only get marked several packets in, 
ie the first few packets might not match anything, then the protocol 
will be detected, thereafter the protocol is wedged as being detected as 
that protocol.  This can obviously mean several packets potentially 
going out unmarked even if the connection is subsequently marked (not 
normally a problem)

Grateful for your benchmarks!

Ed W

Re: [Ntop-dev] New/Updated L7 netfilter option - nDPI

[Eliezer Croitoru]

On 11/2/2012 2:38 PM, Lutfi ODUNCUOGLU wrote:
> Hello,
>
> I compiled nDPI-nefilter patch and it works fine. What I want is to
> shape the p2p traffic in my network. For this purpose i just implemented
> the nDPI-netfilter patch as two different ways for testing
>
>   iptables -t mangle -A POSTROUTING -o XXX -m ndpi --bittorrent -j
> CONNMARK --set-mark 1
>
>
> iptables -t mangle -A POSTROUTING -m connmark --mark 1  -j CLASSIFY
> --set-class 0001:0010
>
> or
>
> iptables -t mangle -A POSTROUTING -m ndpi --bittorrent  -j CLASSIFY
> --set-class 0001:0010
>
>
> So which one is more suitable for use? I don't know if this patch
> inspects connections (marks connection) or every single packet (marks
> every single) for a match.
>
>
> Regards,
>
> Lutfi
Hey Lutfi,

it seems to me that the set-mark is used more then classify and from 
ndpi point of view it's better then restoring marks etc since ndpi 
stores the connection status.

Regards,
Eliezer

-- 
Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer <at> ngtech.co.il

Re: New/Updated L7 netfilter option - nDPI

[Andrew Beverley]

On Thu, 2012-11-01 at 22:56 +0000, Ed W wrote:
> I guess you just invented an "L7 Filter" yourself... It's just as good a 
> match for certain requirements...!
> 
> Let me know if you measure this thing against your problem?

I no longer work on the project that required this, but I will give nDPI
a try next time I'm involved in something similar!

Thanks,

Andy

Re: New/Updated L7 netfilter option - nDPI

[Ed W]

On 28/10/2012 08:20, Andrew Beverley wrote:
> On Wed, 2012-10-24 at 23:53 +0100, Ed W wrote:
>> Hi all.  There is an interesting project that was called opendpi
>> (originally by ipoque GmbH) and recently been forked and maintained by
>> the ntop guys under the nDPI label.  It offers a new and currently
>> maintained layer 7 (L7) packet identification library.
> That's great news.
>
> I had a play with l7-filter some time ago, which I assume is similar to
> nDPI. How do the 2 projects compare?

Actually, just to augment my last answer.

The biggest thing I pick out as "interesting" in nDPI is that it has a 
go at inspecting SSL traffic and odd sub protocols of http (eg Skype, 
Windows Update).  Given that we are rapidly seeing everything start to 
look like an HTTP protocol and then there is SSL on top, it's tricky to 
classify stuff like Skype or Facebook traffic.  nDPI can do this 
(although would benefit from more work in this area). So if your SSL 
certificate says mail.google.com, then you can guess the "protocol" in 
use...

So if you want a one trick reason to try nDPI, right now you can use it 
to block/prioritise/time-restrict Skype... (or Windows Update, etc)

I have a load of users on expensive satellite connections and I need to 
help protect them from themselves so being able to prevent Windows 
Update from banging 10MB down a $30/MB connection is very helpful.  I 
also use your squid patches to do sticky per user conntrack labelling of 
traffic and hence enabling users to choose a traffic profile (so they 
can choose to do the above if they really want to...)

Cheers

Ed W

Re: New/Updated L7 netfilter option - nDPI

[Eliezer Croitoru]

On 10/28/2012 6:34 PM, Ed W wrote:
> Actually, just to augment my last answer.
>
> The biggest thing I pick out as "interesting" in nDPI is that it has a
> go at inspecting SSL traffic and odd sub protocols of http (eg Skype,
> Windows Update).  Given that we are rapidly seeing everything start to
> look like an HTTP protocol and then there is SSL on top, it's tricky to
> classify stuff like Skype or Facebook traffic.  nDPI can do this
> (although would benefit from more work in this area). So if your SSL
> certificate says mail.google.com, then you can guess the "protocol" in
> use...
>
> So if you want a one trick reason to try nDPI, right now you can use it
> to block/prioritise/time-restrict Skype... (or Windows Update, etc)
>
> I have a load of users on expensive satellite connections and I need to
> help protect them from themselves so being able to prevent Windows
> Update from banging 10MB down a $30/MB connection is very helpful.  I
> also use your squid patches to do sticky per user conntrack labelling of
> traffic and hence enabling users to choose a traffic profile (so they
> can choose to do the above if they really want to...)
>
> Cheers
>
> Ed W
Or just use WGET to download the internet into you LAN ;)

-- 
Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer <at> ngtech.co.il

Re: New/Updated L7 netfilter option - nDPI

[Eliezer Croitoru]

On 10/25/2012 12:53 AM, Ed W wrote:
> The practical upshot is that you can do stuff like:
>
>      iptables -I FORWARD -m opendpi --WinUpdate -j LOG
> or
>      iptables -I FORWARD -m opendpi --skype -j REJECT
>
> In theory you can also filter Facebook, Twitter, etc, but I concede that
> doesn't seem to work as expected right now...
>
> Another of the clever things that nDPI does is to try and classify SSL
> traffic by examining the name on the cert.  A technique that seems
> likely to allow crude identification of significant traffic.
>
>
> We could benefit from more eyes on this, both the netfilter module and
> the nDPI library
>
> Thanks for your feedback
>
> Ed W
I tried the new version which is suppose to work with:
iptables -m ndpi --help
but it seems like there is an error:
iptables v1.4.12.1: Couldn't load match `ndpi':No such file or directory

but work with opendpi.
I am using kernel 3.3.8 64 bit.

did you tested the new changes to work?
for now I cant restart the server and I was hoping to test it without 
doing it.

Thanks,
Eliezer

-- 
Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer <at> ngtech.co.il

Re: New/Updated L7 netfilter option - nDPI

[Ed W]

On 28/10/2012 15:07, Eliezer Croitoru wrote:

> I tried the new version which is suppose to work with:
> iptables -m ndpi --help
> but it seems like there is an error:
> iptables v1.4.12.1: Couldn't load match `ndpi':No such file or directory
>
> but work with opendpi.
> I am using kernel 3.3.8 64 bit.
>
> did you tested the new changes to work?

Test??  What does that word mean?  :-)


I think the error means that you don't have libxt_ndpi.so in a location 
that your iptables expects to find it?  Often it's /usr/lib/xtables or 
/lib/xtables.  Check where other things are and place the file appropriately

Does that fix it?

Ed

Re: New/Updated L7 netfilter option - nDPI

[Eliezer Croitoru]

On 10/28/2012 6:03 PM, Ed W wrote:
> Test??  What does that word mean? :-)
>
>
> I think the error means that you don't have libxt_ndpi.so in a location
> that your iptables expects to find it?  Often it's /usr/lib/xtables or
> /lib/xtables.  Check where other things are and place the file
> appropriately
>
> Does that fix it?
/usr/lib it is, /lib/ .. dosnt work for me.

also notice that the "MODULES_DIR" and "KERNEL_DIR" are needed for my 
system.
If and only if you have an idea how to make the Makefile recognize the 
proper "lib" directory and install the module it will be much more 
helpful to others.

Thanks,
Eliezer

>
> Ed


-- 
Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer <at> ngtech.co.il

Re: New/Updated L7 netfilter option - nDPI

[Ed W]

On 28/10/2012 16:51, Eliezer Croitoru wrote:
> On 10/28/2012 6:03 PM, Ed W wrote:
>> Test??  What does that word mean? :-)
>>
>>
>> I think the error means that you don't have libxt_ndpi.so in a location
>> that your iptables expects to find it?  Often it's /usr/lib/xtables or
>> /lib/xtables.  Check where other things are and place the file
>> appropriately
>>
>> Does that fix it?
> /usr/lib it is, /lib/ .. dosnt work for me.
>
> also notice that the "MODULES_DIR" and "KERNEL_DIR" are needed for my 
> system.
> If and only if you have an idea how to make the Makefile recognize the 
> proper "lib" directory and install the module it will be much more 
> helpful to others.

I compile for a non-installed (cross compiled) kernel with:

     NDPI_PATH=/usr/src/nDPI KERNEL_DIR=/usr/src/linux make

If you need MODULES_DIR (not sure why?) then I guess just add that

You can install the kernel module easily enough, either by adding 
"modules_install" to the end of the above (lib location should be picked 
up automatically) or manually with something as simple as:

     mkdir /lib/modules/your_kernel/extra
     cp src/xt_ndpi.ko /lib/modules/your_kernel/extra
     depmod -a

Incidentally, not sure why use that /extra directory?  Anyone got a 
better suggestion on location?

Obviously *also* install the netfilter module (ipt/libxt_ndpi.so) into 
your netfilter modules location

Does that get you going?

Ed W

Re: New/Updated L7 netfilter option - nDPI

[Eliezer Croitoru]

On 10/28/2012 6:57 PM, Ed W wrote:
> I compile for a non-installed (cross compiled) kernel with:
>
>      NDPI_PATH=/usr/src/nDPI KERNEL_DIR=/usr/src/linux make
>
> If you need MODULES_DIR (not sure why?) then I guess just add that
>
> You can install the kernel module easily enough, either by adding
> "modules_install" to the end of the above (lib location should be picked
> up automatically) or manually with something as simple as:
>
>      mkdir /lib/modules/your_kernel/extra
>      cp src/xt_ndpi.ko /lib/modules/your_kernel/extra
>      depmod -a
>
> Incidentally, not sure why use that /extra directory?  Anyone got a
> better suggestion on location?
>
> Obviously *also* install the netfilter module (ipt/libxt_ndpi.so) into
> your netfilter modules location
>
> Does that get you going?
>
> Ed W
I'm OK dont have any problems and done it manually already.
since it was loaded by modprobe and iptables -m ndpi -help works now I 
can test small things like marking packets to make sure it wont break 
anything.

Will look on it in the next few days.

Eliezer
-- 
Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer <at> ngtech.co.il