[LARTC] High speed traffic filtering

[LARTC] High speed traffic filtering

[Vincent Jaussaud]

Hi;

First, sorry if this question is mostly netfilter related, than lartc,
but I think you guys may have a your opinion about this.

I'm using Linux 2.4.x with netfilter packet filtering / NAT on our
front-end firewalls (P500 with 1Gb RAM), which are filtering traffic
going to our Public Web Sites.

The traffic is growing very fast since several months.. The average
traffic filtered by our firewalls, is around 30/40 Mbits/s, with peaks
around 70 Mbits/s sometimes, so that we had to switch to gigabit
technologies, to keep a good safe margin.

Our firewalls are not so high speed machines (P500 with 1Gb RAM), but
are doing good so far.

It seems, however, that we are reaching the limits, when approaching 70
Mb/s... cpu utilization is then near 100%, and the machines start
dropping packets.

So, my question is, is netfilter able to handle, let's say gigabit
traffic filtering ? What kind of hardware would be necessary to handle
such traffic ?

Have you guys any experience with filtering such high speed traffic ?

I also thought of two possible solutions, to optimize our current
firewalls, on which you may have an opinion.

1) Disabling statefull inspection, by unloading connection tracking
modules. 
I believe netfilter without connection tracking, might be much more
efficient (We don't need connection tracking actually, since we are only
filtering HTTP traffic from the others traffics at this point)

2) Replace iptables by nf-hipac for packet filtering. Have you guys any
experience with nf-hipac ? (http://www.hipac.org/)

I would be really thanksfull to hear of any solutions / workarounds /
optimization to keep our linux firewalls handling growing traffic :-)

Thanks !
Vincent.

---

Vincent Jaussaud
Kelkoo.com Security Manager 
email: tatooin@kelkoo.com

"Those who desire to give up freedom in order to gain security will not
have, nor do they deserve, either one."
    -- President Thomas Jefferson.    1743-1826


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] High speed traffic filtering

[Patrick McHardy]

Vincent Jaussaud wrote:
> Hi;
> 
> First, sorry if this question is mostly netfilter related, than lartc,
> but I think you guys may have a your opinion about this.


You should post this question to the netfilter-devel list, people there
can give you very good advice.

> 
> I'm using Linux 2.4.x with netfilter packet filtering / NAT on our
> front-end firewalls (P500 with 1Gb RAM), which are filtering traffic
> going to our Public Web Sites.
> 
> The traffic is growing very fast since several months.. The average
> traffic filtered by our firewalls, is around 30/40 Mbits/s, with peaks
> around 70 Mbits/s sometimes, so that we had to switch to gigabit
> technologies, to keep a good safe margin.
> 
> Our firewalls are not so high speed machines (P500 with 1Gb RAM), but
> are doing good so far.
> 
> It seems, however, that we are reaching the limits, when approaching 70
> Mb/s... cpu utilization is then near 100%, and the machines start
> dropping packets.
> 
> So, my question is, is netfilter able to handle, let's say gigabit
> traffic filtering ? What kind of hardware would be necessary to handle
> such traffic ?
> 
> Have you guys any experience with filtering such high speed traffic ?

Netfilter sure is able to handle 1gbit, but it doesn't depend that much
on the raw speed. The number of conntracks and simultaneos active
connections matters much more.


> I also thought of two possible solutions, to optimize our current
> firewalls, on which you may have an opinion.
> 
> 1) Disabling statefull inspection, by unloading connection tracking
> modules. 
> I believe netfilter without connection tracking, might be much more
> efficient (We don't need connection tracking actually, since we are only
> filtering HTTP traffic from the others traffics at this point)

That might help, although without stateful filtering the rules
have to be evaluated for each single packet.


> 2) Replace iptables by nf-hipac for packet filtering. Have you guys any
> experience with nf-hipac ? (http://www.hipac.org/)

nf-hipac is very good with a large number of rules, for just http
filtering I suspect iptables will do equally good or better.

> 
> I would be really thanksfull to hear of any solutions / workarounds /
> optimization to keep our linux firewalls handling growing traffic :-)

Try without conntrack if you don't need it, otherwise start with
increasing the hash table size and limit ip_conntrack_max to 2 times
the hash size. There was a thread about optimizing iptables on
netfilter-devel 1-2 month ago, it was started by Hervé Eychenne,
search the archives.

Best regards,
Patrick

> 
> Thanks !
> Vincent.
> 
> ---
> 
> Vincent Jaussaud
> Kelkoo.com Security Manager 
> email: tatooin@kelkoo.com
> 
> "Those who desire to give up freedom in order to gain security will not
> have, nor do they deserve, either one."
>     -- President Thomas Jefferson.    1743-1826
> 
> 
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] High speed traffic filtering

[Vincent Jaussaud]

On Sat, 2004-01-10 at 00:10, Patrick McHardy wrote:
> Vincent Jaussaud wrote:
> > Hi;
Hi, and thanks for you reply.

> > 

> That might help, although without stateful filtering the rules
> have to be evaluated for each single packet.
Okay, we'll give it a try. 

> 
> 
> > 2) Replace iptables by nf-hipac for packet filtering. Have you guys any
> > experience with nf-hipac ? (http://www.hipac.org/)
> 
> nf-hipac is very good with a large number of rules, for just http
> filtering I suspect iptables will do equally good or better.

To be tested then, ok I'll try to see if we could build a test network
and try to simulate such traffic.

> 
> > 
> > I would be really thanksfull to hear of any solutions / workarounds /
> > optimization to keep our linux firewalls handling growing traffic :-)
> 
> Try without conntrack if you don't need it, otherwise start with
> increasing the hash table size and limit ip_conntrack_max to 2 times
> the hash size. There was a thread about optimizing iptables on
> netfilter-devel 1-2 month ago, it was started by Hervé Eychenne,
> search the archives.

Thanks, I found the post. Indeed, there is a lot of helpful informations
within. I'll investigate these post deeper.

Thanks !
Regards,
Vincent.

> 
> Best regards,
> Patrick
> 
> > 
> > Thanks !
> > Vincent.
> > 
> > ---
> > 
> > Vincent Jaussaud
> > Kelkoo.com Security Manager 
> > email: tatooin@kelkoo.com
> > 
> > "Those who desire to give up freedom in order to gain security will not
> > have, nor do they deserve, either one."
> >     -- President Thomas Jefferson.    1743-1826
> > 
> > 
> > _______________________________________________
> > LARTC mailing list / LARTC@mailman.ds9a.nl
> > http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
-- 
Vincent Jaussaud
Kelkoo.com Security Manager 
email: tatooin@kelkoo.com

"Those who desire to give up freedom in order to gain security will not
have, nor do they deserve, either one."
    -- President Thomas Jefferson.    1743-1826


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/