Protecting against DoS

Protecting against DoS

[Pasi Kärkkäinen]

Hello!

I was thinking about the correct or best way to protect my Linux/netfilter 
box againts DoS-attacks. 

Some time ago one of the windows users in my LAN managed to get nimda (or
some other) worm to his computer. The worm started scanning the internet
for other vulnerable boxes, opening big amount of tcp-connections all the
time without closing them. 

So after a while I hit the limit of max. open connections
(/proc/sys/net/ipv4/ip_conntrack_max), and the firewall-box is basicly
DoS:ed. With the default settings, open tcp-connections stay in the state
table for 5 days, so it takes a looong time to get things running again if
you don't reload the modules or reboot the box..

Now I have a couple of questions to be sure about the facts while setting 
up the correct limits to prevent this kind of DoS-attacks..


1) Is the correct formula to calculate the maximum number of connections
(for /proc/sys/net/ipv4/ip_conntrack_max) free_memory_in_bytes / 350 ? This
is what I got from the Netfilter FAQ: "You can easily increase the number of
maximal tracked connections, but be aware that each tracked connection eats
about 350 bytes of non-swappable kernel memory!"

2) Netfilter FAQ: "To optimize performance, please also raise the number of
hash buckets by using the hashsize module loadtime parameter of the
ip_conntrack.o module." What's the correct formula to calculate good value
for hashsize?

3) Is there some problem other than the idle tcp-connections dying sooner if I
lower the the value of TCP_CONNTRACK_ESTABLISHED in
/usr/src/linux/net/ipv4/netfilter/ip_conntrack_proto_tcp.c from 5 days to 1
day or even less (to get the possible non-closed tcp-connections out from the state
table sooner) ?

4) What's the correct place to set up limits for new connections (to prevent
the state table being filled up in DoS) ? Is it better to do in the 
mangle-table/PREROUTING-chain something like "-m state --state NEW -m limit 
--limit 5/sec -j RETURN && -j DROP" than later in the filter-table/FORWARD-chain?
I'm thinking about performance here..

5) I'm thinking about measuring average "new connections per second"-rate
and setting up limits to obey that.. is this good way?

6) Do you have some other tips? What are the biggest problems in addition to
getting the state table filled up..


Thanks for your replies!

-- Pasi Kärkkäinen
       
                                   ^
                                .     .
                                 Linux
                              /    -    \
                             Choice.of.the
                           .Next.Generation.

Re: Protecting against DoS

[Michael Gale]

Hello,

	First make sure you are using tcpsyn_cookies:

echo 1 > /proc/sys/net/ipv4/tcp_syncookies -- if you have not compiled it into the kernel. This will help prevent DOS by assigning each incoming syn packet a cookie instead of a actually connection state. A connection state will be created once the three way hand shake is completed.

Second -- you should be dropping all packets on all interfaces and then only allow connections you have to pass.

Michael.


On Tue, 9 Dec 2003 17:43:34 +0200
Pasi Kärkkäinen <pasik@iki.fi> wrote:

> Hello!
> 
> I was thinking about the correct or best way to protect my Linux/netfilter 
> box againts DoS-attacks. 
> 
> Some time ago one of the windows users in my LAN managed to get nimda (or
> some other) worm to his computer. The worm started scanning the internet
> for other vulnerable boxes, opening big amount of tcp-connections all the
> time without closing them. 
> 
> So after a while I hit the limit of max. open connections
> (/proc/sys/net/ipv4/ip_conntrack_max), and the firewall-box is basicly
> DoS:ed. With the default settings, open tcp-connections stay in the state
> table for 5 days, so it takes a looong time to get things running again if
> you don't reload the modules or reboot the box..
> 
> Now I have a couple of questions to be sure about the facts while setting 
> up the correct limits to prevent this kind of DoS-attacks..
> 
> 
> 1) Is the correct formula to calculate the maximum number of connections
> (for /proc/sys/net/ipv4/ip_conntrack_max) free_memory_in_bytes / 350 ? This
> is what I got from the Netfilter FAQ: "You can easily increase the number of
> maximal tracked connections, but be aware that each tracked connection eats
> about 350 bytes of non-swappable kernel memory!"
> 
> 2) Netfilter FAQ: "To optimize performance, please also raise the number of
> hash buckets by using the hashsize module loadtime parameter of the
> ip_conntrack.o module." What's the correct formula to calculate good value
> for hashsize?
> 
> 3) Is there some problem other than the idle tcp-connections dying sooner if I
> lower the the value of TCP_CONNTRACK_ESTABLISHED in
> /usr/src/linux/net/ipv4/netfilter/ip_conntrack_proto_tcp.c from 5 days to 1
> day or even less (to get the possible non-closed tcp-connections out from the state
> table sooner) ?
> 
> 4) What's the correct place to set up limits for new connections (to prevent
> the state table being filled up in DoS) ? Is it better to do in the 
> mangle-table/PREROUTING-chain something like "-m state --state NEW -m limit 
> --limit 5/sec -j RETURN && -j DROP" than later in the filter-table/FORWARD-chain?
> I'm thinking about performance here..
> 
> 5) I'm thinking about measuring average "new connections per second"-rate
> and setting up limits to obey that.. is this good way?
> 
> 6) Do you have some other tips? What are the biggest problems in addition to
> getting the state table filled up..
> 
> 
> Thanks for your replies!
> 
> -- Pasi Kärkkäinen
>        
>                                    ^
>                                 .     .
>                                  Linux
>                               /    -    \
>                              Choice.of.the
>                            .Next.Generation.
> 


-- 
Michael Gale
Network Administrator
Utilitran Corporation

Re: Protecting against DoS

[Pasi Kärkkäinen]

On Tue, Dec 09, 2003 at 09:02:21AM -0700, Michael Gale wrote:
> 
> Hello,
> 
> 	First make sure you are using tcpsyn_cookies:
> 
> echo 1 > /proc/sys/net/ipv4/tcp_syncookies -- if you have not compiled it into the kernel. 
> This will help prevent DOS by assigning each incoming syn packet a cookie instead of a actually 
> connection state. A connection state will be created once the three way hand shake is completed.
> 
> Second -- you should be dropping all packets on all interfaces and then only allow connections 
> you have to pass.
> 

Yes.. I'm already doing both of these things. I was thinking of doing some
extra in addition of these.. Sorry I didn't mention about these already.

There are always some connections allowed that can be used to fill up the
state table..

Thanks anyway!

> Michael.
> 

-- Pasi Kärkkäinen
       
                                   ^
                                .     .
                                 Linux
                              /    -    \
                             Choice.of.the
                           .Next.Generation.

Re: Protecting against DoS

[Michael Gale]

Hello,

	Can you provide more detail on the type of traffic that caused the DOS -- this may help people in the list with suggestions on how to block it :)

Michael.



On Tue, 9 Dec 2003 18:28:20 +0200
Pasi Kärkkäinen <pasik@iki.fi> wrote:

> On Tue, Dec 09, 2003 at 09:02:21AM -0700, Michael Gale wrote:
> > 
> > Hello,
> > 
> > 	First make sure you are using tcpsyn_cookies:
> > 
> > echo 1 > /proc/sys/net/ipv4/tcp_syncookies -- if you have not compiled it into the kernel. 
> > This will help prevent DOS by assigning each incoming syn packet a cookie instead of a actually 
> > connection state. A connection state will be created once the three way hand shake is completed.
> > 
> > Second -- you should be dropping all packets on all interfaces and then only allow connections 
> > you have to pass.
> > 
> 
> Yes.. I'm already doing both of these things. I was thinking of doing some
> extra in addition of these.. Sorry I didn't mention about these already.
> 
> There are always some connections allowed that can be used to fill up the
> state table..
> 
> Thanks anyway!
> 
> > Michael.
> > 
> 
> -- Pasi Kärkkäinen
>        
>                                    ^
>                                 .     .
>                                  Linux
>                               /    -    \
>                              Choice.of.the
>                            .Next.Generation.
> 


-- 
Michael Gale
Network Administrator
Utilitran Corporation

Re: Protecting against DoS

[Pasi Kärkkäinen]

On Tue, Dec 09, 2003 at 09:40:47AM -0700, Michael Gale wrote:
> Hello,
> 
> 	Can you provide more detail on the type of traffic that caused the DOS -- this may help people in the list with suggestions on how to block it :)
> 

Yep. It was tcp-connections from the windows box (infected by the worm) to
some network-ranges on the internet. source-port was pretty much random, but
the destination was always 80. So the normal 'allow web browsing' rules
allowed the worm to DoS the linux-firewall.  

It just opened the connections all the time, but didn't close them. 

> Michael.
> 

-- Pasi Kärkkäinen
       
                                   ^
                                .     .
                                 Linux
                              /    -    \
                             Choice.of.the
                           .Next.Generation.

Re: Protecting against DoS

[Michael Gale]

Hello,

	You could try using a rate limit -- you could allow a machine to make lets say 10 outbound connections a second and then ...

Depending on your network policy you could drop or log all other outbound request.

Michael.



On Tue, 9 Dec 2003 18:51:46 +0200
Pasi Kärkkäinen <pasik@iki.fi> wrote:

> On Tue, Dec 09, 2003 at 09:40:47AM -0700, Michael Gale wrote:
> > Hello,
> > 
> > 	Can you provide more detail on the type of traffic that caused the DOS -- this may help people in the list with suggestions on how to block it :)
> > 
> 
> Yep. It was tcp-connections from the windows box (infected by the worm) to
> some network-ranges on the internet. source-port was pretty much random, but
> the destination was always 80. So the normal 'allow web browsing' rules
> allowed the worm to DoS the linux-firewall.  
> 
> It just opened the connections all the time, but didn't close them. 
> 
> > Michael.
> > 
> 
> -- Pasi Kärkkäinen
>        
>                                    ^
>                                 .     .
>                                  Linux
>                               /    -    \
>                              Choice.of.the
>                            .Next.Generation.
> 


-- 
Michael Gale
Network Administrator
Utilitran Corporation

Re: Protecting against DoS

[Pasi Kärkkäinen]

On Tue, Dec 09, 2003 at 10:06:50AM -0700, Michael Gale wrote:
> Hello,
> 
> 	You could try using a rate limit -- you could allow a machine to make lets say 10 outbound 
> connections a second and then ...
> 
> Depending on your network policy you could drop or log all other outbound request.
> 

This is what I'm planning to do.. and this is also the reason I was asking
the questions in the original mail :-)

-- Pasi Kärkkäinen
       
                                   ^
                                .     .
                                 Linux
                              /    -    \
                             Choice.of.the
                           .Next.Generation.

Re: Protecting against DoS

[Geffrey Velásquez]

Hum.. but what happen if the workstation is infected? the virus will do a
DoS to the user of the workstation, the firewall will block valid
connections.
> On Tue, Dec 09, 2003 at 10:06:50AM -0700, Michael Gale wrote:
>> Hello,
>>
>> 	You could try using a rate limit -- you could allow a machine to make
>> 	lets say 10 outbound
>> connections a second and then ...
>>
>> Depending on your network policy you could drop or log all other
>> outbound request.
>>
>
> This is what I'm planning to do.. and this is also the reason I was
> asking the questions in the original mail :-)
>
> -- Pasi Kärkkäinen
>
>                                   ^
>                                .     .
>                                 Linux
>                              /    -    \
>                             Choice.of.the
>                           .Next.Generation.

Re: Protecting against DoS

[Arnt Karlsen]

On Tue, 9 Dec 2003 14:20:09 -0500 (PET), 
"Geffrey Velásquez" <g_netfilter@netfids.com> wrote in message 
<36917.200.48.142.50.1070997609.squirrel@www.netfids.com>:

> Hum.. but what happen if the workstation is infected? the virus will
> do a DoS to the user of the workstation, the firewall will block valid
> connections.

.. ;-)  A further refinement could be force a reinstall on the infected
wintendo, or reboot it as a thin client off a boot server, so the user
can do the work he is paid to do.  ;-)

-- 
..med vennlig hilsen = with Kind Regards from Arnt... ;-)
...with a number of polar bear hunters in his ancestry...
  Scenarios always come in sets of three: 
  best case, worst case, and just in case.

Re: Protecting against DoS

[Pasi Kärkkäinen]

On Tue, Dec 09, 2003 at 07:13:22PM +0200, Pasi Kärkkäinen wrote:
> On Tue, Dec 09, 2003 at 10:06:50AM -0700, Michael Gale wrote:
> > Hello,
> > 
> > 	You could try using a rate limit -- you could allow a machine to make lets say 10 outbound 
> > connections a second and then ...
> > 
> > Depending on your network policy you could drop or log all other outbound request.
> > 
> 
> This is what I'm planning to do.. and this is also the reason I was asking
> the questions in the original mail :-)
> 

If you have some facts/suggestions, please comment my original questions.. 

Thanks!

-- Pasi Kärkkäinen
       
                                   ^
                                .     .
                                 Linux
                              /    -    \
                             Choice.of.the
                           .Next.Generation.

Re: Protecting against DoS

[Peter Frischknecht]

Hello,

I wanted to let you know that you are not alone.
We manage networks in apartment complexes.  Hundreds of students use
their computers simultaneously and we have always been able to deal with
DoS attacks.

The attack you described is identical to the one we have suffered since
sometime in September.  In short: it cannot be stopped.  But there are
many things you can do.

As you may have found in personal research, the worm has the following
characteristics:
1 - It is a Zombie.  Computers do NOT automatically start attacking web
sites.  They wait for an instruction from outside.
2 - The connection to the outside "master server" is done via IRC.
3 - The infected computer attacks 1 to a few web servers at a time. (we
have never seen more than 3).
4 - Completely random spoofed src addresses.
5 - The MAC address is (Thank You GOD) not being spoofed.
6 - The intensity of the attack deems the offending computer almost
useless during the attack.

It seemed from your description that your network was fairly small.
In order to save bandwidth, we implement a transparent caching
strategy.  So we redirect port 80 to the cache.  Guess what!!!  Our
caching server ends up the target of the attack!!!

I can tell you that I tried every pertinent module in the patch-o-matic
volume.  It was a 3 month ordeal with very frequent lockups. "connlimit"
does not work because once you enable SYN cookies, there are no
connections to limit.  "limit" draws too much CPU power, eventually
helping lock up the box.

Blocking the foreign IP sources at the FORWARD level (or INPUT in my
case) stopped the packets, but still kept my server with a very high
system use.

I have hundreds of rules in the MANGLE and NAT tables, so the bad
packets were still traveling and being matched against all those rules. 
I placed the drops for the foreign IPs in the MANGLE table.  If you look
at the docs, it is the first table the packet hits.

BTW, to stop foreign IP packets, I created one ACCEP rule for every
known IP class inside my network.  The last line simply DROPs everything
coming from the internal network.

My networks are basically functional now.  But the HIGH CPU use
persists.  It is likely caused by the interrupt handling or the packet
counters/rules checking.  I recently ran accross this page on lowering
the interrupt handling:
ftp://robur.slu.se/pub/Linux/net-development/NAPI/NAPI_HOWTO.txt

Feel free to correspond with me.  We can trade ideas on the topic.

Peter Frischknecht
Empowering Solutions, Inc.

bridge vlans in HP 2524 switch

[Computer Security]

Anyone has experience in using iptables to bridge Vlans created in the same
HP 2524 switch ?
-----------------------------------------------------------------
Gongya Yu

Re: Protecting against DoS

[Pasi Kärkkäinen]

On Sat, Jan 10, 2004 at 08:50:50PM -0500, Peter Frischknecht wrote:
> Hello,
> 
> I wanted to let you know that you are not alone.
> We manage networks in apartment complexes.  Hundreds of students use
> their computers simultaneously and we have always been able to deal with
> DoS attacks.
> 

Hello!


> The attack you described is identical to the one we have suffered since
> sometime in September.  In short: it cannot be stopped.  But there are
> many things you can do.
> 
> As you may have found in personal research, the worm has the following
> characteristics:
> 1 - It is a Zombie.  Computers do NOT automatically start attacking web
> sites.  They wait for an instruction from outside.
> 2 - The connection to the outside "master server" is done via IRC.
> 3 - The infected computer attacks 1 to a few web servers at a time. (we
> have never seen more than 3).
> 4 - Completely random spoofed src addresses.
> 5 - The MAC address is (Thank You GOD) not being spoofed.
> 6 - The intensity of the attack deems the offending computer almost
> useless during the attack.
> 

Yes, sounds like the same thing.

> It seemed from your description that your network was fairly small.
> In order to save bandwidth, we implement a transparent caching
> strategy.  So we redirect port 80 to the cache.  Guess what!!!  Our
> caching server ends up the target of the attack!!!
> 

Hehe.


> I can tell you that I tried every pertinent module in the patch-o-matic
> volume.  It was a 3 month ordeal with very frequent lockups. "connlimit"
> does not work because once you enable SYN cookies, there are no
> connections to limit.  "limit" draws too much CPU power, eventually
> helping lock up the box.
> 

Hmm.. how did you set up the limits? I wrote small application that
calculates the average new connections/sec rates and I then set up the
limits according to that..


> Blocking the foreign IP sources at the FORWARD level (or INPUT in my
> case) stopped the packets, but still kept my server with a very high
> system use.
> 
> I have hundreds of rules in the MANGLE and NAT tables, so the bad
> packets were still traveling and being matched against all those rules. 
> I placed the drops for the foreign IPs in the MANGLE table.  If you look
> at the docs, it is the first table the packet hits.
> 

Yep. It seems to be very difficult to protect against these kind of
attacks.. 

> BTW, to stop foreign IP packets, I created one ACCEP rule for every
> known IP class inside my network.  The last line simply DROPs everything
> coming from the internal network.
> 

I'm doing this also.

> My networks are basically functional now.  But the HIGH CPU use
> persists.  It is likely caused by the interrupt handling or the packet
> counters/rules checking.  I recently ran accross this page on lowering
> the interrupt handling:
> ftp://robur.slu.se/pub/Linux/net-development/NAPI/NAPI_HOWTO.txt
> 
> Feel free to correspond with me.  We can trade ideas on the topic.
> 

Sorry for the long delay in the answer :)

-- Pasi Kärkkäinen
       
                                   ^
                                .     .
                                 Linux
                              /    -    \
                             Choice.of.the
                           .Next.Generation.