More on ip_conntrack problems

More on ip_conntrack problems

[alex]

I've seen numerous references to percieved problems with default
timeouts and potential DoS attacks on ip_conntrack but I'm starting to
think is possible to ip_conntrack just to miss connection closures.

I'm running on a memory constrained machine (28Mb Physical RAM) which
shouldn't prove too problamatic as I'm only on a DSL connection. I've
been running a Freenet node on my internal machine with the appropraite
port forwarding magic with iptables. The problem I'm seeing is a slow
climb in the number of connections until it reaches saturation and
connectivity becomes very dicey (so I now open a ssh shell to my gateway
box before I do anything, just to ensure I can still talk to it).

The one piece of evidence that makes me suspicious of ip_conntracks
ability to detect connection tear down is the disparity between the
numbers I get from the gateway box to what I get on my internal box

gateway:
cat /proc/net/ip_conntrack | wc -l

give 1688 connections

internal box:

netstat -t | wc -l

gives 41 connections

I eyeballed the ip_conntrack output and all the connections tracked are
Freenet connections so I'm discounting other server connections like web
etc as in the minority. Once I stop freenet the count stays quite
static, I hope it will of cleared down once I get back to my box this
afternoon but I assume this is due to the long timeouts on closing TCP
connections.

A few questions:

a) Am I measuring things correctly?
b) Should the number of connections on ip_conntrack be broadly the same
as the internal machines understanding of connections (netstat output)?
c) Has this come up before? 
d) Are there any patches I could try that alter ipconntracks end of
connection heuristics?

Cheers,

-- 
Alex@Bennee.com
http://www.bennee.com/~alex/

Re: More on ip_conntrack problems

[Patrick Schaaf]

Hi Alex,

[snip good description of general situation]

> A few questions:
> 
> a) Am I measuring things correctly?

You are, but one thing is missing, if I'm correct in the explanation below.
We need to know the distribution of conntrack states from your gateway:

grep ^tcp /proc/net/ip_conntrack | awk '{print $4}' | sort | uniq -c

Read on:

> b) Should the number of connections on ip_conntrack be broadly the same
> as the internal machines understanding of connections (netstat output)?

No. I'll explain why:

How many new TCP connections per seconds is that freenet thing doing?
Let's call this CPS.

How long living are the individual connections, in seconds, as seen
by the server? Let's call this CT (connection time.)

These two numbers are essential for analysis. You want to know them
for any connection tracking system you are responsible for.

Now, assume that I know how long conntrack is supposed to keep record
of a connection, after the server and client are finished with it.
Let's call that ET (extra time.)

This means that the conntracking box sees each CT connection as
a CT+ET connection. On the server, you can expect, in netstat,
to see CPS*CT connections.  And on the conntracking box, you
expect CPS*(CT+ET).

Let nS := CPS * CT, be the number of connections on the server.
Let nC := nS + CPS * ET, be the number of connections on the conntracker.

What can we do, given only nS and nC? Well, (nC - nS) appears to
be equal to CPS * ET. Thus, knowing ET to be X seconds, and given
your values of nS := 41, nC := 1688, we can estimate CPS to be 1647/X.

Let's assume I know X to be 10 seconds. Then, you should have about 170
connections per second. A good load - what are these doing? On the other
extreme, still assuming normal operation, you could have X as 120 seconds,
i.e. 13 connections per second. Still a good load - what is that freenet
thing doing to your home network???

To know the real answer, i.e. get a handle on X, you need to learn
the conntrack state distribution on your gateway, as per the commands
given above. The normal closing states have these (inactivity) timeouts:

	2 MINS,     FIN_WAIT
	2 MINS,     TIME_WAIT
	10 SECS,    CLOSE
	60 SECS,    CLOSE_WAIT
	30 SECS,    LAST_ACK

For example, on one of my transproxy machines, I currently see this
when I run the given commands:

208	CLOSE
156	CLOSE_WAIT
14905	ESTABLISHED
3	FIN_WAIT
55	SYN_RECV
39	SYN_SENT
11953	TIME_WAIT

The dominant terminating state is the TIME_WAIT state, so I'm at 120
seconds ET, and estimate CPS to be 99 connections per second. That's
even true...

If other closing states dominate for your freenet thing, you can do
a weighted calculation based on the frequency data and timeouts,
and arrive at an "average X".

Alternatively, you can measure CPS by other means, and calculate X
from that side.

You can see that the extra knowledge of the distribution of closing
states, if statistically significant (*), gives you all the knowledge
to understand the situation fully.

I hope this text could help you.

best regards
  Patrick

(*) this comment is significant. I have the luxury of large numbers.
You could need to sample several times, and create aggregate averages.

Re: More on ip_conntrack problems

[Henrik Nordstrom]

On Wednesday 10 July 2002 09.10, alex wrote:
> I've seen numerous references to percieved problems with default
> timeouts and potential DoS attacks on ip_conntrack but I'm starting
> to think is possible to ip_conntrack just to miss connection
> closures.

It can.. see the archives. Posted a relatively detailed description on 
when conntrack will miss connection closures some weeks 
ago..<http://lists.samba.org/pipermail/netfilter-devel/2002-June/004906.html> 
Quite likely to happen if you have clients on unreliable connections.

Most people running larger conntrack setups probably won't notice 
unless they are under attack as the frequency this happens in normal 
traffic is very low, but if you have a small conntrack table and 
relatively many rapid connections then you may experience problems as 
each forgotten connection occupies a conntrack slot for a 
considerable amount of time..

To tell if this is your problem you need to do what Patrick Schaaf suggested. 
If you see that your conntrack box thinks there is many more 
CONNECTED TCP connections than there is ESTABLISHED connections on your 
server then you are bitten by this lost connection closures.

If you see that there is no big difference then your problem is simply
that your conntrack table is too small for the traffic you are seeing.

In any event, you most likely need to increase the conntrack hash 
table if you haven't already.. see the FAQ/Howto documents..

Regards
Henrik Nordström
MARA Systems AB, Sweden