RE: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified)Denial of Service Attack

RE: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified)Denial of Service Attack

[Alan Cox]

On Sul, 2004-09-12 at 19:40, Wolfpaw - Dale Corse wrote:
> This bug also exists with Apache, the default config of SSH,
> and anything controlled by inetd. This is the vast majority of
> popular services on a regular internet server.. That is bad, no?

I'm unable to duplicate any such problems with xinetd, or with thttpd,
or with apache. Apache will wait a short time then timeout connections
if you've configured it right. If you can continue making millions of
connections a second you can DoS the server the other end, not exactly
new news. The alternative is that you have an infinite number of running
services and you run out of memory instead.

Thats a high level property of any protocol which allows commitment of
resource without being able to do the security authentication first. Its
very hard to create ones that don't however, thus most devices in life
(eg your telephone) have this form or DoS attack.

My sshd also doesn't show this problem and the manual page indicates it
has a 120 second grace timeout for authentication.

The sshd manual page says:

     Gives the grace time for clients to authenticate themselves
             (default 120 seconds).

RE: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified)Denial of Service Attack

[Wolfpaw - Dale Corse]

Hi Alan,

 My inetd is not the old one, it is in Slackware 10.0..
apparently 1.79s, still works here. (the s added by
The slackware group according to the readme:

/*      $Slackware: inetd.c 1.79s 2001/02/06 13:18:00 volkerdi Exp $    */
/*      $OpenBSD: inetd.c,v 1.79 2001/01/30 08:30:57 deraadt Exp $      */ 
/*      $NetBSD: inetd.c,v 1.11 1996/02/22 11:14:41 mycroft Exp $       */
/* 

It still dies pretty fast :( Willy is probably right, in that
the bug is application level in this case. Can you explain
though, how it is appropriate to have no timeout on CLOSE_WAIT.
Lets assume (such as the case with the mysql bug), that it is
created, and for some reason never addressed.. The end case being
mysql reporting "Too many Connections". That would make me assume
these connections are still "alive" (perhaps I am assuming wrong?)
and could still cause issues with other services, if the volume of
them was to get too high.

It still seems prudent to me to have some kind of timeout (2 hours,
or something?) to have it expuldge the connection, because obviously,
it isn't going to voluntarily close.

This bug also exists with Apache, the default config of SSH,
and anything controlled by inetd. This is the vast majority of
popular services on a regular internet server.. That is bad, no?

D.

> -----Original Message-----
> From: Alan Cox [mailto:alan@lxorguk.ukuu.org.uk] 
> Sent: Sunday, September 12, 2004 11:17 AM
> To: Willy Tarreau
> Cc: Wolfpaw - Dale Corse; peter@mysql.com; Linux Kernel 
> Mailing List; netdev@oss.sgi.com
> Subject: Re: Linux 2.4.27 SECURITY BUG - TCP Local and 
> REMOTE(verified)Denial of Service Attack
> 
> 
> On Sul, 2004-09-12 at 18:59, Willy Tarreau wrote:
> > The problem is within inetd. In my case it could be because it was a
> > bit old (1999), but since you have it too,
> 
> Ancient inetd had several fd leak bugs fixed over time and 
> some other problems with built in services. Not much of a 
> suprise that a 1999 inetd has it.
> 
> Alan
> 
> 
> --------------------------------------------------------------
> --------------
> -
> This message has been scanned for Spam and Viruses by ClamAV 
> and SpamAssassin
> --------------------------------------------------------------
> --------------
> -
> 
> 
> 
> 
> 

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified)Denial of Service Attack

[Willy Tarreau]

On Sun, Sep 12, 2004 at 12:40:56PM -0600, Wolfpaw - Dale Corse wrote:
> It still seems prudent to me to have some kind of timeout (2 hours,
> or something?) to have it expuldge the connection, because obviously,
> it isn't going to voluntarily close.

Dale,

forgive me for insisting, but it would be criminal to do this. You think
that it's sort of closed while it is not. Consider CLOSE_WAIT as ESTABLISHED.
It is not because one end sends a FIN (shutdown(WRITE)) that the connection
is about to end. The culprit really is the application. It is the application
which trusts the other end too much. The system lets the application work as
it expects it, and should never try to cut the grass below the application's
feet.

You must understand how it is supposed to work, basically like this :

        A                                      B

  shutdown(WRITE)
  A enters FIN_WAIT1
          ------- ACK + FIN  ---------->  select(in[fd],out[fd],) returns in[fd]
                                          B enters CLOSE_WAIT
          <------ ACK + data -----------  read(fd) returns 0
                                          shutdown(SHUT_RD)
                                          Then remove fd from read list.
          <---- ACK + data -------------  select(NULL, out[fd],...)
                  (...)                   write(fd, data)...
          <---- ACK + data -------------  select(NULL, out[fd],...)
          <---- ACK + FIN --------------  shutdown(SHUT_WRITE) or close(fd)
 close(fd)
          ------- ACK ----------------->

If the system closed anything, on B side, since the select() does not check
read events anymore, it would only be woken up for a write even, which could
crash the application in a SIG_PIPE if the writer does not check the error
condition on the socket before writing (like most applications do).

So, let me insist, your proposal is not the right solution to this. The
right solution is to carefully check every application and fix them, the
same way you would fix them to handle time-outs on ESTABLISHED connections.

> This bug also exists with Apache, the default config of SSH,
> and anything controlled by inetd. This is the vast majority of
> popular services on a regular internet server.. That is bad, no?

You didn't wait long enough for each of them. I bet that if you wait up
to 2 minutes, SSH will close, and if you wait 5 or 10 minutes, apache
will close too. As to mysql, I have no idea, and inetd, we obviously both
have a buggy version.

I hope this clarifies a bit the situation,
Willy

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified)Denial of Service Attack

[Jurjen Oskam]

On Sun, Sep 12, 2004 at 12:40:56PM -0600, Wolfpaw - Dale Corse wrote:

> the bug is application level in this case. Can you explain
> though, how it is appropriate to have no timeout on CLOSE_WAIT.

See TCP/IP Illustrated, Volume 1 by W. Richard Stevens (ISBN
0-201-63346-9), page 238: "TCP Half-Close".

In short, you don't have a timeout on CLOSE_WAIT for the same reason you
don't have a timeout on ESTABLISHED.

-- 
Jurjen Oskam
"I often reflect that if "privileges" had been called "responsibilities" or
"duties", I would have saved thousands of hours explaining to people why
they were only gonna get them over my dead body." - Lee K. Gleason, VMS sysadmin

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Florian Weimer]

* David S. Miller:

> On Tue, 21 Sep 2004 22:04:41 +0200
> Florian Weimer <fw@deneb.enyo.de> wrote:
>
>> > Why would it be off by default?
>> 
>> Probably because PMTUD is just a DRAFT STANDARD,
>
> RFC1191 doesn't look like a draft to me.

It's not a draft document, but it's still a DRAFT STANDRD in the IETF
standards track, see RFC 3700.  (I wasn't shouting, I was using IETF
keywords. 8-)

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[David S. Miller]

On Tue, 21 Sep 2004 22:04:41 +0200
Florian Weimer <fw@deneb.enyo.de> wrote:

> > Why would it be off by default?
> 
> Probably because PMTUD is just a DRAFT STANDARD,

RFC1191 doesn't look like a draft to me.

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Florian Weimer]

* David S. Miller:

> On Tue, 21 Sep 2004 20:32:12 +0200
> Florian Weimer <fw@deneb.enyo.de> wrote:
>
>> > But then you get PMTU problems...
>> 
>> PMTU discovery is not an issue because it's turned off anyway, at
>> least by default.
>
> It's on by default, for both TCP and UDP in the kernel,
> and has been so for a long time.

Linux is not the reference TCP/IP stack for routers. 8-)

> Why would it be off by default?

Probably because PMTUD is just a DRAFT STANDARD, and these router
folks are usually extremely conservative.  Switching the default is
dangerous because it's likely to break existing setups, as Herbert
noted.

> If you disable PMTU discovery, say goodbye to TCP performance.

Indeed.  On those platforms, the CPU impact is also significant, and
the overall increase in BGP convergence time is measurable.

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[David S. Miller]

On Tue, 21 Sep 2004 20:32:12 +0200
Florian Weimer <fw@deneb.enyo.de> wrote:

> > But then you get PMTU problems...
> 
> PMTU discovery is not an issue because it's turned off anyway, at
> least by default.

It's on by default, for both TCP and UDP in the kernel,
and has been so for a long time.

Why would it be off by default?

If you disable PMTU discovery, say goodbye to TCP
performance.

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Florian Weimer]

* Herbert Xu:

> Florian Weimer <fw@deneb.enyo.de> wrote:
>> 
>>>> TCP-MD5 has no effect on ICMP based attacks.,
>>>
>>> Hmm, good point. Which attacks, and what could be done about them? 
>>> (other than IPsec protect all traffic between peers).
>> 
>> You just filter ICMP packets, in the way RST packets are already
>> filtered (i.e. rate limit).
>
> Rate-limiting has no effect on ICMP attacks unless your limit is such
> that you're effectively dropping them all.

Yes, that's the idea.  Keep in mind that all this is about traffic
destined to a router interface address, not about forwarded traffic.

> But then you get PMTU problems...

PMTU discovery is not an issue because it's turned off anyway, at
least by default.

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Herbert Xu]

Florian Weimer <fw@deneb.enyo.de> wrote:
> 
>>> TCP-MD5 has no effect on ICMP based attacks.,
>>
>> Hmm, good point. Which attacks, and what could be done about them? 
>> (other than IPsec protect all traffic between peers).
> 
> You just filter ICMP packets, in the way RST packets are already
> filtered (i.e. rate limit).

Rate-limiting has no effect on ICMP attacks unless your limit is such
that you're effectively dropping them all.  But then you get PMTU
problems...
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Alan Cox]

On Llu, 2004-09-20 at 23:03, Florian Weimer wrote:
> ndomization, quite a lot more. Of course
> > its much easier to just send "must fragment, size 68" icmp replies and
> > guess them that way.
> 
> Is this attack documented anywhere?

Bugtraq years ago and also in the discussions of the IP sec protocol
design flaws when it was being specified

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Florian Weimer]

* Alan Cox:

> On Maw, 2004-09-14 at 15:55, Paul Jakma wrote:
>> Hmm, yes, I hadnt thought of the attack-mitigating aspects of 
>> graceful restart. Though, without other measures, the session is 
>> still is open to abuse (send RST every second).
>
> Its more than that given port randomization, quite a lot more. Of course
> its much easier to just send "must fragment, size 68" icmp replies and
> guess them that way.

Is this attack documented anywhere?

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Florian Weimer]

* Paul Jakma:

>> TCP-MD5 has no effect on ICMP based attacks.,
>
> Hmm, good point. Which attacks, and what could be done about them? 
> (other than IPsec protect all traffic between peers).

You just filter ICMP packets, in the way RST packets are already
filtered (i.e. rate limit).

The only TCP desynchronization attack that has a chance of working
practice is the SYN-based one.  The rate limit for RST processing on
Cisco routers is far too low.

(Mixed Cisco/Quagga environments are a different matter, but rather
unusual and easily DoSed anyway, most of the time.)

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Willy Tarreau]

Hi Alan,

On Tue, Sep 14, 2004 at 04:10:36PM +0100, Alan Cox wrote:
> On Maw, 2004-09-14 at 15:55, Paul Jakma wrote:
> > Hmm, yes, I hadnt thought of the attack-mitigating aspects of 
> > graceful restart. Though, without other measures, the session is 
> > still is open to abuse (send RST every second).
> 
> Of course its much easier to just send "must fragment, size 68" icmp
> replies and guess them that way. This is spectacularly more effective
> and various vendors highly invalid rst acking crap won't save you.

Just wondering, I have not checked. Isn't the "must fragment" message
supposed to embed part of the packet it couldn't send in return ? If
this is the case (and if the victim processes it correctly), it would
need to guess a recent valid content. If it's not the case, I suspect
it would simply update the path mtu in the route cache, thus giving
spectacular effects :-)

Cheers,
Willy

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Alan Cox]

On Maw, 2004-09-14 at 20:41, Willy Tarreau wrote:
> Just wondering, I have not checked. Isn't the "must fragment" message
> supposed to embed part of the packet it couldn't send in return ?

You need to guess no more than for an RST attack, and furthermore in
some cases (buggy stacks) IPsec doesn't save you because the error is
from an untrusted midpoint. The proper response to such messages is to
turn off DF usage but not all stacks get it right

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Paul Jakma]

On Tue, 14 Sep 2004, Alan Cox wrote:

> TCP-MD5 has no effect on ICMP based attacks.,

Hmm, good point. Which attacks, and what could be done about them? 
(other than IPsec protect all traffic between peers).

regards,
-- 
Paul Jakma	paul@clubi.ie	paul@jakma.org	Key ID: 64A2FF6A
Fortune:
"You can't get very far in this world without your dossier being there first."
-- Arthur Miller

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Paul Jakma]

On Tue, 14 Sep 2004, Alan Cox wrote:

> guess them that way. This is spectacularly more effective and 
> various vendors highly invalid rst acking crap won't save you.

Ah, well, I dont care about various vendors. I only care about Linux, 
BSD and SunOS kernel behaviour ;)

That said, TCP-MD5 signature renders this mostly moot, and deployment 
of TCP-MD5 has increased a lot since the last round of "BGP TCP is 
insecure!" non-issues came up. Many IXes and peers now require 
TCP-MD5.

The rights and wrongs of TCP-MD5 notwithstanding, it'd be nice if 
Linux could support this. Anyone running BGP on Linux at moment must 
patch their kernel - or else just switch to Free/Open BSD.

regards,
-- 
Paul Jakma	paul@clubi.ie	paul@jakma.org	Key ID: 64A2FF6A
Fortune:
It looks like it's up to me to save our skins.  Get into that garbage chute,
flyboy!
 		-- Princess Leia Organa

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Alan Cox]

On Maw, 2004-09-14 at 17:26, Paul Jakma wrote:
> That said, TCP-MD5 signature renders this mostly moot, and deployment 
> of TCP-MD5 has increased a lot since the last round of "BGP TCP is 
> insecure!" non-issues came up. Many IXes and peers now require 
> TCP-MD5.

TCP-MD5 has no effect on ICMP based attacks.,

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Alan Cox]

On Maw, 2004-09-14 at 15:55, Paul Jakma wrote:
> Hmm, yes, I hadnt thought of the attack-mitigating aspects of 
> graceful restart. Though, without other measures, the session is 
> still is open to abuse (send RST every second).

Its more than that given port randomization, quite a lot more. Of course
its much easier to just send "must fragment, size 68" icmp replies and
guess them that way. This is spectacularly more effective and various
vendors highly invalid rst acking crap won't save you.

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Paul Jakma]

On Mon, 13 Sep 2004, Ville Hallivuori wrote:

> Actually you can treat TCP session failure as transient error. Just 
> use BGP graceful restart (witch basically allows re-opening TCP 
> connection without losing routing tables).
>
> http://www.ietf.org/internet-drafts/draft-ietf-idr-restart-10.txt

Hmm, yes, I hadnt thought of the attack-mitigating aspects of 
graceful restart. Though, without other measures, the session is 
still is open to abuse (send RST every second).

regards,
-- 
Paul Jakma	paul@clubi.ie	paul@jakma.org	Key ID: 64A2FF6A
Fortune:
Wit, n.:
 	The salt with which the American Humorist spoils his cookery
 	... by leaving it out.
 		-- Ambrose Bierce, "The Devil's Dictionary"

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Ville Hallivuori]

On Mon, Sep 13, 2004 at 04:30:36AM +0100, Paul Jakma wrote:

> More specifically, BGP should have treated TCP resets as a transient 
> error, to be expected (indeed, they /cant/ be a sign that a link is 

Actually you can treat TCP session failure as transient error. Just
use BGP graceful restart (witch basically allows re-opening TCP
connection without losing routing tables).

http://www.ietf.org/internet-drafts/draft-ietf-idr-restart-10.txt

-- 
[Ville Hallivuori][vph@iki.fi][http://www.iki.fi/vph/]
[ID 8E1AD461][FP16=C9 50 E2 DF 48 F6 33 62  5D 87 47 9D 3F 2B 07 5D]
[ID 58543419][FP20=8731 941D 15AB D4A0 88A0  FC8F B55C F4C4 5854 3419]
[ID 8061C24E][FP20=C722 12DA 841E D811 DBFE  2FB3 174C E291 8061 C24E]

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Paul Jakma]

On Mon, 13 Sep 2004, Tonnerre wrote:

> Problem there: you can fake connection errors almost as easily as 
> sending an RST packet, so the DoS might reappear, might it not?

Sure, but TCP just isnt going to solve this for you.

> 				Tonnerre

regards,
-- 
Paul Jakma	paul@clubi.ie	paul@jakma.org	Key ID: 64A2FF6A
Fortune:
All progress is based upon a universal innate desire of every organism
to live beyond its income.
 		-- Samuel Butler, "Notebooks"

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Willy Tarreau]

On Mon, Sep 13, 2004 at 09:07:41PM +0200, Tonnerre wrote:
> Salut,
> 
> On Mon, Sep 13, 2004 at 06:18:47AM +0200, Willy Tarreau wrote:
> > > The BGP state machine should instead, in normal operation, have 
> > > only treated Hold time expired as the definitive sign of "peer is 
> > > down" and allowed reconnects.
> > 
> > It should not necessarily wait for the time-out, but at least wait for
> > a few reconnect errors.
> 
> Problem  there: you  can fake  connection errors  almost as  easily as
> sending an RST packet, so the DoS might reappear, might it not?
> 

No, as long as you don't keep the routes from the old session until the
new one establishes and fills up (or you reach the timeout). And when I
spoke about "connection errors", I really spoke about connection
establishment. I bet you'll have more difficulties trying to send the
right RST just after a SYN (or an ICMP unreachable with the right payload)
than sending them once the session is already established. It does make
a big difference.

Willy

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 511 bytes --]

Salut,

On Mon, Sep 13, 2004 at 06:18:47AM +0200, Willy Tarreau wrote:
> > The BGP state machine should instead, in normal operation, have 
> > only treated Hold time expired as the definitive sign of "peer is 
> > down" and allowed reconnects.
> 
> It should not necessarily wait for the time-out, but at least wait for
> a few reconnect errors.

Problem  there: you  can fake  connection errors  almost as  easily as
sending an RST packet, so the DoS might reappear, might it not?

				Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Paul Jakma]

On Mon, 13 Sep 2004, Willy Tarreau wrote:

> It should not necessarily wait for the time-out, but at least wait for
> a few reconnect errors.

No, it should wait for the timeout. (how many reconnects? maybe use a 
time for that? well, you already have one, so use that. if you want 
to timeout quicker, lower it.)

Alas though, it wouldnt be BGP.

> Regards,
> willy

regards,
-- 
Paul Jakma	paul@clubi.ie	paul@jakma.org	Key ID: 64A2FF6A
Fortune:
You will be called upon to help a friend in trouble.

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Willy Tarreau]

On Mon, Sep 13, 2004 at 04:30:36AM +0100, Paul Jakma wrote:
 
> More specifically, BGP should have treated TCP resets as a transient 
> error, to be expected (indeed, they /cant/ be a sign that a link is 
> down - if you can receive a RST the link or path is patently quite 
> ok).

The application level does not always distinguish between TCP RST and
error generated by the local system because of a "network unreachable"
due to a link down and a lost route.

> The BGP state machine should instead, in normal operation, have 
> only treated Hold time expired as the definitive sign of "peer is 
> down" and allowed reconnects.

It should not necessarily wait for the time-out, but at least wait for
a few reconnect errors.

Regards,
willy

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Paul Jakma]

On Mon, 13 Sep 2004, Paul Jakma wrote:

> I think he means that BGP treating TCP connections as if they could 
> reliably and securely indicate link/path status (ie connection 
> reset/timeout == link down) status was, in retrospect, a very dumb 
> idea on the part of BGP.

More specifically, BGP should have treated TCP resets as a transient 
error, to be expected (indeed, they /cant/ be a sign that a link is 
down - if you can receive a RST the link or path is patently quite 
ok). The BGP state machine should instead, in normal operation, have 
only treated Hold time expired as the definitive sign of "peer is 
down" and allowed reconnects.

> regards,

regards,
-- 
Paul Jakma	paul@clubi.ie	paul@jakma.org	Key ID: 64A2FF6A
Fortune:
You will attract cultured and artistic people to your home.

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Paul Jakma]

On Sun, 12 Sep 2004, Toon van der Pas wrote:

>> knowing to a resonable degree what arrived. TCP does not proide a
>> security service.
>>
>> (The core of this problem arises because certain people treat TCP
>> connection down on the peering session as link down)
>
> Alan, could you please elaborate on this last statement?
> I don't understand what you mean, and am very interested.

I think he means that BGP treating TCP connections as if they could 
reliably and securely indicate link/path status (ie connection 
reset/timeout == link down) status was, in retrospect, a very dumb 
idea on the part of BGP.

> Thanks,
> Toon.

regards,
-- 
Paul Jakma	paul@clubi.ie	paul@jakma.org	Key ID: 64A2FF6A
Fortune:
This restaurant was advertising breakfast any time. So I ordered
french toast in the renaissance.
- Steven Wright, comedian

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Willy Tarreau]

On Sun, Sep 12, 2004 at 01:42:26PM -0600, Wolfpaw - Dale Corse wrote:
 
> MD5 encryption was added to the sessions between
> routers to make hijacking the stream more difficult
> (if not next to impossible)

Correction : MD5 *signature* was added from the beginning since the problem
was identified from start, but seeing that certain people did not implement
it, others found interesting to turn this into a "generic TCP vulnerability"
to get some credits, or perhaps to make them react positively.

Regards,
Willy

RE: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Wolfpaw - Dale Corse]

Hey,

 I'm not Alan, just trying to save him some typing :) The 
issue being referenced is that BGP4 as you know uses TCP 
communication to check link status. If the TCP session is 
severed in any way, BGP assumes the link down, and drops 
the advertisements, or the table depending which side 
(at least in Cisco's implementation).

This basically leaves you dead in the water for a few
seconds while the BGP session is re-established, and
the advertisements are send out, and table rebuilt. It
can also cause other routers on the net to see you as
"flapping", and dampen your routes.. Which again leaves
you dead in the water (at least from things behind them).

This is accomplished by guessing the correct TCP Sequence
number, and sending RST packets to drop the TCP connection.

BGP does not actually check the layer 2 status of the
connection to make sure the link is still UP before
it assumes you have dropped. I believe this is the
poor implementation he is referring to.

MD5 encryption was added to the sessions between
routers to make hijacking the stream more difficult
(if not next to impossible)

D.

> -----Original Message-----
> From: Dale Corse [mailto:admin@wolfpaw.net] 
> Sent: Sunday, September 12, 2004 1:36 PM
> To: 'Dale'
> Subject: FW: Linux 2.4.27 SECURITY BUG - TCP Local and 
> REMOTE(verified) Denial of Service Attack
> 
> 
> 
> 
> -----Original Message-----
> From: Toon van der Pas [mailto:toon@hout.vanvergehaald.nl] 
> Sent: Sunday, September 12, 2004 1:24 PM
> To: Alan Cox
> Cc: Wolfpaw - Dale Corse; kaukasoi@elektroni.ee.tut.fi; Linux 
> Kernel Mailing List
> Subject: Re: Linux 2.4.27 SECURITY BUG - TCP Local and 
> REMOTE(verified) Denial of Service Attack
> 
> 
> On Sun, Sep 12, 2004 at 06:04:53PM +0100, Alan Cox wrote:
> > 
> > This is not a TCP flaw, its a combination of poor design by certain
> > vendors, poor BGP implementation and a lack of 
> understanding of what 
> > TCP does and does not do. See IPSec. TCP gets stuff from A to B in 
> > order and knowing to a resonable degree what arrived. TCP does not 
> > proide a security service.
> > 
> > (The core of this problem arises because certain people treat TCP
> > connection down on the peering session as link down)
> 
> Alan, could you please elaborate on this last statement?
> I don't understand what you mean, and am very interested.
> 
> Thanks,
> Toon.
> -- 
> "Debugging is twice as hard as writing the code in the first 
> place. Therefore, if you write the code as cleverly as 
> possible, you are, by definition, not smart enough to debug 
> it." - Brian W. Kernighan
> 
> --------------------------------------------------------------
> --------------
> -
> This message has been scanned for Spam and Viruses by ClamAV 
> and SpamAssassin
> --------------------------------------------------------------
> --------------
> -
> 
> 
> 
> 

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Toon van der Pas]

On Sun, Sep 12, 2004 at 06:04:53PM +0100, Alan Cox wrote:
> 
> This is not a TCP flaw, its a combination of poor design by certain
> vendors, poor BGP implementation and a lack of understanding of what TCP
> does and does not do. See IPSec. TCP gets stuff from A to B in order and
> knowing to a resonable degree what arrived. TCP does not proide a
> security service.
> 
> (The core of this problem arises because certain people treat TCP
> connection down on the peering session as link down)

Alan, could you please elaborate on this last statement?
I don't understand what you mean, and am very interested.

Thanks,
Toon.
-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

RE: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified)Denial of Service Attack

[Wolfpaw - Dale Corse]

Hi Alan,

> On Sul, 2004-09-12 at 18:29, Wolfpaw - Dale Corse wrote:
> > A fair comment :) But look at it this way:
> > 
> > - The TCP RFC was last updated when?
> 
> About 2 months ago. The 793 RFC isn't updated instead new 
> ones are added for the additional features/discoveries.

My fault there :( Sorry.

> > - What is the average time for a tcp packet to fly even across
> >   the world these days? Maybe 300 ms? 1 second? 5?
> > - It is not a secret that the TCP protocol has flaws, take for
> >   example the RST bug, which required among other things, BGP4
> >   to use MD5 encryption to avoid being potentially attacked.
> 
> This is not a TCP flaw, its a combination of poor design by 
> certain vendors, poor BGP implementation and a lack of 
> understanding of what TCP does and does not do. See IPSec. 
> TCP gets stuff from A to B in order and knowing to a 
> resonable degree what arrived. TCP does not proide a security service.
> 
> (The core of this problem arises because certain people treat 
> TCP connection down on the peering session as link down)

Good point :)

> > So this brings me to:
> > 
> > A) Why are the timeouts so long?
> 
> So you don't get random corruption

Hmm. I'll take your word for it, I'm not quite grasping it,
but you know quite a bit more about it then I do :) I would
have thought once a close is sent, the data has all been received
and/or sent anyway, so what would corrupt?

> > C) Socket still re-uses an FD before it is actually completely
> 
> Pardon ?

It _appears_ that when we close a socket (ie with mysql_close) on
the client side, the client side closes the FD properly (though Mysql
doesn't), and then if we call connect (which it does a lot, being a proxy
server) it will reuse that FD. At this point, the Mysql side still hasn't
closed it, and it is sitting in CLOSE_WAIT, where it remains forever, since
it is in use by the client side elsewhere already. Should connect be
checking the "list of not connected, but state other then CLOSED" list
before it decides to use a particular FD? Or is this behavior intentional?

> > sending something to the other side is required, but I can't see why
> > having the other side send something back is part of the protocol. 
> > This could be
> 
> Because packet sizes are finite and not doing so requires an 
> infinite sequence space and thus infinite packet sizes. 
> Reread the TCP specifications more carefully, also look at 
> RFC1337 which discusses some of the real world cases of 
> getting this wrong.

Fair enough, thank you for clarifying it for me :)

Keep up the good work :) (which you do a lot of I might add :)
D.

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Willy Tarreau]

Hi again, Dale,

I forgot to say that you don't need to fear releasing your exploit. I
developped its equivalent 4 years ago to stress-test web servers and
proxies, and if I launch it against victim:23, I get the exact same
result within seconds : a CLOSE_WAIT socket :

attacker> ./connectdata 10.0.3.2 23 200 1
ERROR: connect()=-1, nbconn=134 : Connection refused
ERROR: connect()=-1, nbconn=135 : Connection refused
ERROR: connect()=-1, nbconn=136 : Connection refused
ERROR: connect()=-1, nbconn=137 : Connection refused

The program connects 200 sockets to the same IP:port, and sends the begining
of an HTTP request.

victim> sudo netstat -atnp|grep -v LISTEN
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   
tcp       17      0 10.0.3.2:23             10.0.3.1:38214          CLOSE_WAIT  1333/inetd          

It's even not necessary to send data, then even faster to block my very old
inetd :

attacker> ./connectdata-nb 10.0.3.2 23 200
200 connections established.
Press any key so exit.

This time, it sends 200 non-blocking connect() calls without any data. It
takes a fraction of a second with the same result. Hopefully, it'll will
help Peter and you reproduce the problem faster on mysql.

Both programs have been freely available here for two years ; I didn't think
they would be useful again !

   http://w.ods.org/tools/connect/

Regards,
Willy

RE: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified)Denial of Service Attack

[Alan Cox]

On Sul, 2004-09-12 at 19:52, Wolfpaw - Dale Corse wrote:
> > > A) Why are the timeouts so long?
> > 
> > So you don't get random corruption
> 
> Hmm. I'll take your word for it, I'm not quite grasping it,
> but you know quite a bit more about it then I do :) I would
> have thought once a close is sent, the data has all been received
> and/or sent anyway, so what would corrupt?

It has all arrived at least once. However you don't know if a
retransmitted packet took a long trip round the world and popped out
later. When that happens you need to be sure you can tell old and new
apart. The TIME_WAIT state is basically "don't use this identical set of
port/address data for long enough to be sure any prior use has exited
the internet in its entirety.

> It _appears_ that when we close a socket (ie with mysql_close) on
> the client side, the client side closes the FD properly (though Mysql

socket != fd. If you close an fd and open you may get the same fd but
its a different socket. If its getting stuck closing could you have
another copy of the fd left open somewhere or have passed it to a
process you forked (thats a classic nasty to track down error) ?

> doesn't), and then if we call connect (which it does a lot, being a proxy
> server) it will reuse that FD. At this point, the Mysql side still hasn't
> closed it, and it is sitting in CLOSE_WAIT, where it remains forever, since
> it is in use by the client side elsewhere already. Should connect be
> checking the "list of not connected, but state other then CLOSED" list
> before it decides to use a particular FD? Or is this behavior intentional?

Sockets are two way at the TCP layer. "close" really means "have
finished sending". When both ends have finished sending the connection
is complete but not before. Thus if Mysql daemon says "I am done" but
you have open references to the handle then it will stay open one way
still.

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Willy Tarreau]

Hi Dale,

I've tried your code right here.
The "attacker" was 10.0.3.1, and the victim 10.0.3.2.

I could successfully generate 1 CLOSE_WAIT on the victim with your program.
It was on port 23 and attached to inetd as fd #3. So I killed inetd, the
connection was then freed, and restarted it.

I changed the code slightly to be able to pass IP/ports as arguments.
On the victim, I straced inetd (pid 1013), and captured all TCP traffic
on port 23.

attacker> ./tcpnclose2 10.0.3.2 22 10.0.3.2 23

I stopped it when it was shouting at me :
socket failed.Connecting to 10.0.3.2:22 (FD: -1)...  FAILED: UNKNOWN ERROR.
socket failed.Connecting to 10.0.3.2:23 (FD: -1)...  FAILED: UNKNOWN ERROR.

Then, on the victim :

victim> sudo netstat -atnp|grep -v LISTEN
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   
tcp        1      0 10.0.3.2:23             10.0.3.1:34058          CLOSE_WAIT  1013/inetd          

victim> tcpdump -Svnr capture-victim.cap tcp port 34058
reading from file capture-victim.cap, link-type EN10MB (Ethernet)
19:05:10.360728 IP (tos 0x0, ttl  64, id 8168, offset 0, flags [DF], length: 48) 10.0.3.1.34058 > 10.0.3.2.23: S [tcp sum ok] 2882867180:2882867180(0) win 15920 <mss 7960,nop,nop,sackOK>
19:05:10.360764 IP (tos 0x0, ttl  64, id 0, offset 0, flags [DF], length: 48) 10.0.3.2.23 > 10.0.3.1.34058: S [tcp sum ok] 2614211278:2614211278(0) ack 2882867181 win 5840 <mss 1460,nop,nop,sackOK>
19:05:10.360863 IP (tos 0x0, ttl  64, id 8169, offset 0, flags [DF], length: 40) 10.0.3.1.34058 > 10.0.3.2.23: . [tcp sum ok] ack 2614211279 win 15920
19:06:17.668670 IP (tos 0x0, ttl  64, id 8170, offset 0, flags [DF], length: 40) 10.0.3.1.34058 > 10.0.3.2.23: F [tcp sum ok] 2882867181:2882867181(0) ack 2614211279 win 15920
19:06:17.671102 IP (tos 0x0, ttl  64, id 11127, offset 0, flags [DF], length: 40) 10.0.3.2.23 > 10.0.3.1.34058: . [tcp sum ok] ack 2882867182 win 5840

==> We see that the victim (10.0.3.2) did not send the FIN in return.

Now let's take a closer look at inetd :

victim> cat /proc/net/tcp 
  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode                                                     
  16: 0203000A:0017 0103000A:850A 08 00000000:00000001 00:00000000 00000000     0        0 6420 1 d5dac400 1500 20 0 2 -1                            

==> The socket (state 8 = CLOSE_WAIT) is bound to inode 6420.

victim> sudo ls -l /proc/1013/fd/|grep 6420
lrwx------    1 root     root           64 Sep 12 19:28 3 -> socket:[6420]

==> Again, it's FD #3.

I restarted strace on inetd, and noticed that fd#3 was not in the select fd
list anymore (remember one of the two cases I spoke about a few hours ago ?) :
victim> strace -p 1013
select(22, [4 5 6 7 8 9 11 12 13 14 15 16 17 18 19 20 21], NULL, NULL, NULL <unfinished ...>

Then, I took a look at the strace capture (184 MB !), to which I inserted line
numbers for better readability :

1:1013  accept(10, 0, NULL)               = 3
2:1013  fcntl64(10, F_SETFL, O_RDONLY)    = 0
3:1013  rt_sigprocmask(SIG_BLOCK, [HUP ALRM CHLD], NULL, 8) = 0
4:1013  fork()                            = 1108
5:1013  rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
6:1013  close(3)                          = 0

This was the last but one connection assigned to fd #3. As you see, it's
finally closed. But a few lines later :

7:1013  select(22, [4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21], NULL, NULL, NULL) = 1 (in [10])
8:1013  fcntl64(10, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
9:1013  accept(10, 0, NULL)               = 3
10:1013  fcntl64(10, F_SETFL, O_RDONLY)    = 0
11:1013  rt_sigprocmask(SIG_BLOCK, [HUP ALRM CHLD], NULL, 8) = 0
12:1013  gettimeofday({1095008773, 685550}, NULL) = 0

The FD gets re-used, but is never scanned anymore, so never closed either :

35:1013  select(22, [4 5 6 7 8 9 11 12 13 14 15 16 17 18 19 20 21], NULL, NULL, NULL <unfinished ...>

Conclusion :
============

The problem is within inetd. In my case it could be because it was a bit
old (1999), but since you have it too, it might indicate an old bug. The
fact that it affects mysql too does not prove that the problem is in the
kernel, and I suspect that for whatever reason, there are some race
conditions in these two programs if the connection is either reused or
closed very quickly.

To demonstrate this, I've run your program against my reverse-proxy,
haproxy, which I fortunately happen to know better than these other
programs. I could not manage to get even a CLOSE_WAIT session after
several attempts.  All connections are closed normally, and as you'll
see with this extract from strace, the polled file-descriptors are
active once you kill the attacker :

(...)
close(593)                              = 0
select(684, [3 5 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617
618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636
637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655
656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674
675 676 677 678 679 680 681 682 683], NULL, NULL, {4, 835000}) = 81 (in [603
604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622
623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641
642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660
661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679
680 681 682 683], left {4, 836000})
gettimeofday({1095011266, 506966}, NULL) = 0
recv(603, "", 4096, 0x4000)             = 0
recv(604, "", 4096, 0x4000)             = 0
recv(605, "", 4096, 0x4000)             = 0
(...)
close(605)                              = 0
close(604)                              = 0
close(603)                              = 0
select(6, [3 5], NULL, NULL, NULL <unfinished ...>

So I believe you'll have to dig into some programs because at least you found
a vulnerability in both inetd and mysql :-)

Regards,
Willy

RE: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Wolfpaw - Dale Corse]

> On Sun, Sep 12, 2004 at 09:45:38AM -0600, Wolfpaw - Dale Corse wrote:
> >  No problem :) I run the following, against SSH as the target, and I
> > can also kill it. (using telnet as the other side of the attack)
> > 
> > root@maximus:/home/admin# telnet XXXXXXXXXXXXXXX 22
> > Trying XXXXXXXXXXXXXX...
> > Connected to XXXXXXXXXXXXXXXXX.
> > Escape character is '^]'.
> > Connection closed by foreign host.
> > root@maximus:/home/admin#
> 
> > Now.. Do you really want me to post the source code for it?
> 
> With default sshd_config you can DOS sshd trivially by 
> opening ten connections using ten times "telnet XXXXXXXXXXXXXXX 22".

A fair comment :) But look at it this way:

- The TCP RFC was last updated when?
- What is the average time for a tcp packet to fly even across
  the world these days? Maybe 300 ms? 1 second? 5?
- It is not a secret that the TCP protocol has flaws, take for
  example the RST bug, which required among other things, BGP4
  to use MD5 encryption to avoid being potentially attacked.

So this brings me to:

A) Why are the timeouts so long?
B) CLOSE_WAIT having _no timeout at all_ is still using the
   assumption the other side is honest, and will actually
   reply. This is a very bad assumption.
C) Socket still re-uses an FD before it is actually completely
   closed. This is bad, because by calling a second close in
   the case of mysql, you can get the connection to go away,
   but in that case, it closes whatever else is on that FD
   too. (A more likely analysis is that it closes the current
   connection, and then cleans the CLOSE_WAIT on that FD out
   of the other pool)

All I am trying to point out is that the Internet in general, and
The Open Source movement has survived, and evolved because of innovation,
and the ability to meet upcoming threats quickly. TCP has some issues
which are blindingly obvious, and they are issues that, in my view (flawed
as it may be) can be at least somewhat minimized by a few simple changes.

I realize daemons have connection queues, and timeouts for a reason, but
really, if a daemon wishes to close a connection, for whatever reason,
sending something to the other side is required, but I can't see why having
the other side send something back is part of the protocol. This could be
implemented with KEEPALIVE much easier, and would avoid the flaws.. No reply
from the host in say 10 seconds, then drop the connection. You could still
clog queues, to which I would say the application needs to cope with one
client
filling the whole queue as best it can, and it wouldn't stop a DDOS (not much
does), but it might help some at least.

Anyway.. That's my 2 cents. I will continue my conversation with Peter from
mysql in regards to mysql_close, which was really, the entire point. It is
sad however to see a maintainer come across in the manner which David has
during the course of this discussion. It doesn't bode well for the future
of open source to tell someone off, whom likely has a valid point.. whether
or not it is a repairable fault.

Regards,
Dale.
--------------------------------
Dale Corse
System Administrator
Wolfpaw Services Inc.
http://www.wolfpaw.net
(780) 474-4095

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Alan Cox]

On Sul, 2004-09-12 at 18:59, Willy Tarreau wrote:
> The problem is within inetd. In my case it could be because it was a bit
> old (1999), but since you have it too,

Ancient inetd had several fd leak bugs fixed over time and some other
problems with built in services. Not much of a suprise that a 1999 inetd
has it.

Alan

RE: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Alan Cox]

On Sul, 2004-09-12 at 18:29, Wolfpaw - Dale Corse wrote:
> A fair comment :) But look at it this way:
> 
> - The TCP RFC was last updated when?

About 2 months ago. The 793 RFC isn't updated instead new ones are added
for the additional features/discoveries.

> - What is the average time for a tcp packet to fly even across
>   the world these days? Maybe 300 ms? 1 second? 5?
> - It is not a secret that the TCP protocol has flaws, take for
>   example the RST bug, which required among other things, BGP4
>   to use MD5 encryption to avoid being potentially attacked.

This is not a TCP flaw, its a combination of poor design by certain
vendors, poor BGP implementation and a lack of understanding of what TCP
does and does not do. See IPSec. TCP gets stuff from A to B in order and
knowing to a resonable degree what arrived. TCP does not proide a
security service.

(The core of this problem arises because certain people treat TCP
connection down on the peering session as link down)

> So this brings me to:
> 
> A) Why are the timeouts so long?

So you don't get random corruption

> C) Socket still re-uses an FD before it is actually completely

Pardon ?

> sending something to the other side is required, but I can't see why having
> the other side send something back is part of the protocol. This could be

Because packet sizes are finite and not doing so requires an infinite
sequence space and thus infinite packet sizes. Reread the TCP
specifications more carefully, also look at RFC1337 which discusses some
of the real world cases of getting this wrong.

Re: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Petri Kaukasoina]

On Sun, Sep 12, 2004 at 09:45:38AM -0600, Wolfpaw - Dale Corse wrote:
>  No problem :) I run the following, against SSH as the target, and I
> can also kill it. (using telnet as the other side of the attack)
> 
> root@maximus:/home/admin# telnet XXXXXXXXXXXXXXX 22
> Trying XXXXXXXXXXXXXX...
> Connected to XXXXXXXXXXXXXXXXX.
> Escape character is '^]'.
> Connection closed by foreign host.
> root@maximus:/home/admin# 

> Now.. Do you really want me to post the source code for it? 

With default sshd_config you can DOS sshd trivially by opening ten
connections using ten times "telnet XXXXXXXXXXXXXXX 22".

RE: Linux 2.4.27 SECURITY BUG - TCP Local and REMOTE(verified) Denial of Service Attack

[Wolfpaw - Dale Corse]

Hi Willy,

 No problem :) I run the following, against SSH as the target, and I
can also kill it. (using telnet as the other side of the attack)

root@magik:/etc# telnet 0.0.0.0 22
Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.
Connection closed by foreign host.
root@magik:/etc# telnet 0.0.0.0 23
Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.
 
Magik login: Connection closed by foreign host.
root@magik:/etc# 

And from a remote host:

root@maximus:/home/admin# telnet XXXXXXXXXXXXXXX 22
Trying XXXXXXXXXXXXXX...
Connected to XXXXXXXXXXXXXXXXX.
Escape character is '^]'.
Connection closed by foreign host.
root@maximus:/home/admin# 

And it gets worse now..:

root@avalon:/root# telnet XXXXXXXXXXXXXXX
Trying XXXXXXXXXXXXX...
Connected to XXXXXXXXXXXXX.
Escape character is '^]'.
telnetd: All network ports in use.
Connection closed by foreign host.
root@avalon:/root# telnet XXXXXXXXXXXXXX 22
Trying XXXXXXXXXXXXXX...
Connected to XXXXXXXXXXXXXXXXX.
Escape character is '^]'.
Connection closed by foreign host.
root@avalon:/root# 

Well well.. We have ourselves a Remote Denial of Service tool.

Now.. Do you really want me to post the source code for it?

I wouldn't want to upset David again. This has basically
disabled interactive administration on that machine, by taking
out both ssh and telnet at the same time. If you want to demo
it, I can send it to you privately.. I'm a bit apprehensive
about releasing a 'ready-to-rock' remote DoS exploit on a
list though :(

Just think here a moment.. Lets say I modify it a bit more,
and turn it into a DDOS utility, so now you have (pardon my
language) .. An assload of these coming at your server, how
are you going to stop it? Simply - you can't, and your server
will run out of sockets long before all the remote hosts do.

This one in essence takes a nice working tcp connection application,
and removes the close statements, which as you mentioned will cause
the sockets to end up in that state. What I am attempting to demonstrate
here, is the fact it can relatively easily take out any tcp based app,
and simply saying, something should be done about it. What the actual
bug is, I think I know (and have said), but I will leave that determination 
to the actual kernel developers.

I was not attempting to say you should break TCP with the timeouts, or
even make them short, I just would tend to think no timeout at ALL is
a bit of a design flaw, because if the other end is no longer there,
the work will never get done, but one of the ends expects the response
indefinitely, which to me looks like an assumption. I think we have all
learned these days that you can't make anymore assumptions, people use
those to break things.

I also would like to thank you for engaging in a discussion about the
bug with me, in a polite manner, instead of simply writing me off as
some loud mouth neophyte.

Anyway - from my view, this is a bug in the OS, because it should not
occur, if it does, we need to find a way to ensure it doesn't. I know
a few firewall tricks that might stop it, but I'm not sure. If a regular
user can invoke this kind of response so easily, I would say it's a bad
thing.

Regards,
Dale.
> -----Original Message-----
> From: Dale Corse [mailto:admin@wolfpaw.net] 
> Sent: Sunday, September 12, 2004 8:58 AM
> To: Dale
> Subject: FW: Linux 2.4.27 SECURITY BUG - TCP Local (probable 
> Remote) Denial of Service
> 
> 
> 
> 
> -----Original Message-----
> From: Willy Tarreau [mailto:willy@w.ods.org] 
> Sent: Sunday, September 12, 2004 4:36 AM
> To: Wolfpaw - Dale Corse
> Cc: peter@mysql.com; linux-kernel@vger.kernel.org
> Subject: Re: Linux 2.4.27 SECURITY BUG - TCP Local (probable 
> Remote) Denial of Service
> 
> 
> On Sun, Sep 12, 2004 at 03:24:11AM -0600, Wolfpaw - Dale Corse wrote:
>  
> > This is the odd part, try the exploit,
> 
> I have nothing to try it right here.
> 
> > they are detached in
> > the list, but it appears apache isn't aware of that. If you run the
> > code, and do multiple telnets from another window, you will 
> see that 
> > there are occurrences where a connection can't be established, and 
> > this is where the problem is. I used a stock version of Mysql 3 
> > (latest stable), stock apache, on an unmoded Linux box 
> (except it had 
> > GrSecurity) and I was able to see a noticeable slowdown in web 
> > transactions with a browser. I was also the only person hitting the 
> > machine.
> 
> How can you be sure that your problem is not simply related 
> to either apache or mysql not freeing the connection fast 
> enough ? Apache is very limited in terms of simultaneous 
> connections, and it is trivial for anyone to block an apache 
> server by establishing as many connections as it can handle, 
> sending the start of a request and doing nothing more (and it 
> has a very long default time out BTW). It might be the same 
> with mysql.
> 
> > I am not saying you are incorrect, I'm simply clarifying 
> what seems to
> > be occurring with the issue I found.
> > 
> > Do you happen to know of any solution for sockets stuck in 
> CLOSE_WAIT,
> > they seem to stick around forever.
> 
> Yes, the only solution is to debug the process and make it 
> sanely close the socket once it does not need it anymore. 
> Usually, in such circumstances, you'll find that an strace on 
> the process shows either :
>   - a select loop with your socket in the list of the active FDs, but
>     nothing in the process will do anything on this FD and the process
>     will go back to the select loop => bug in FD handling
>  - a select loop which does not include the FD while it has 
> not been released
>    => bug in FD releasing code (usually a missing close).
> 
> > This bug may be more Mysql then kernel, I don't know - I still would
> > tend to think these connections should not be clogging up the 
> > applications connection queue, and that CLOSE_WAIT should have a 
> > settable timeout, regardless of what the RFC says about it.
> 
> No, CLOSE_WAIT means that the application still has some work 
> to do. Under no circumstances, the kernel should destroy its 
> ability to work normally !
> 
> > I did experience more CLOSE_WAIT's stuck at one point with 
> Mysql.. we
> > had an issue wherein after calling mysql_close with the C 
> API it was 
> > still leaving the sessions established, so I had moved the 
> timeout on 
> > that sql daemon to 20 seconds (its all fast transactions) .. This 
> > caused a lot of CLOSE_WAIT issues for some reason.
> 
> So you've just demonstrated that it's mysql_close which is 
> the culprit. If it does not really close the connection while 
> you expected it to, it is the real problem. If you lower the 
> mysql timeout, mysql will close on its end, but as long as 
> the code using mysql_close() will not close, of course the 
> socket will remain close_wait. And to be clear, even if you 
> would have a short CLOSE_WAIT time-out, it would not help 
> because you would still be running out of file-descriptors 
> after a moment.
> 
> > We then
> > added something that would go through and use 'close' on 
> the fd of the
> > Mysql connection, after mysql_close was called. This had the odd 
> > effect of the fd being reused by a connection, before it was out of 
> > CLOSE_WAIT and actually closed, so it would close the new 
> Connection, 
> > and also the old one :P which led us to this discovery that 
> connect() 
> > appears to reuse FD's before they are actually fully 
> closed.. This is 
> > how it appears anyway. Thus my use of specifically mysql 
> and connect 
> > in the PoC code.
> 
> If you manage to write a PoC code which does not involve 
> either apache not mysql, and which still exhibits the 
> described behaviour, then perhaps kernel developpers will 
> listen a bit more, but at the moment, you only showed us how 
> you could trigger a DoS by connecting to a buggy application.
> 
> Cheers,
> Willy
> 
> 
> --------------------------------------------------------------
> --------------
> -
> This message has been scanned for Spam and Viruses by ClamAV 
> and SpamAssassin
> --------------------------------------------------------------
> --------------
> -
> 
> 
> 
> 
>