Memory usage for ip_conntrack

Memory usage for ip_conntrack

[Dax Kelson]

I'm teaching a Linux class and the book says "Using ip_conntrack will
use much more memory".

A student asked me how much is "much".

So on a 2.4.20+  how much memory does it take to track the state of a
connection?

If I echo 102400 > /proc/sys/net/ipv4/ip_conntrack_max, what is my worst
case memory usage?

TIA,

Dax Kelson

Re: Memory usage for ip_conntrack

[Martin Josefsson]

On Fri, 2003-07-18 at 22:07, Dax Kelson wrote:
> I'm teaching a Linux class and the book says "Using ip_conntrack will
> use much more memory".
> 
> A student asked me how much is "much".
> 
> So on a 2.4.20+  how much memory does it take to track the state of a
> connection?

This depends on which patches you've applied and if you have selected
NAT support when you compiled it or not. You can see how much memory it
uses by looking at the kernel output when ip_conntrack is initialized.
It looks like:

ip_conntrack version 2.1 (5632 buckets, 45056 max) - 304 bytes per conntrack

That's the number of bytes ip_conntrack will try to allocate for each
connection. But it isn't neccessarily the real number of bytes
allocated. You'll have to look at /proc/slabinfo for that. Look at
column 4, that's the object size. In my case it's 320 bytes.

And each bucket in the hashtable will use 8 bytes on a 32bit machine, 16
bytes on a 64bit machine.

> If I echo 102400 > /proc/sys/net/ipv4/ip_conntrack_max, what is my worst
> case memory usage?

Don't do this. This will increase the maximum number of connections it
will track, but not the number of buckets. Which means that it will be
slower due to longer collision-chains. Instead increase the number of
buckets. modprobe ip_conntrack hashsize=131072 (or any number here. If
it's a < 2.4.21 kernel 2^n sizes aren't recommended due to a poor
hashfunction. Instead you should use 2^n-1)

In my case the memory-usage for the above numbers would be:

5632 buckets * 8 bytes (32bit machine)
+
102400 * 320 bytes (object size from slabinfo)
= 45056 + 32768000 ~= 31.3 MB

Increasing the number of buckets doesn't cost much memory compared to
the actual connections and it gives you a nice performanceboost if you
are trying to handle lots of connections. (the default is based on the
amount of memory in the machine and it's normally ok for desktop
machines and small servers/routers)

ip_conntrack is a memory-hog, we are working on reducing the
memory-usage.

-- 
/Martin

Memory usage/tuning for ip_conntrack

[Dax Kelson]

On Fri, 2003-07-18 at 15:28, Martin Josefsson wrote:

> Increasing the number of buckets doesn't cost much memory compared to
> the actual connections and it gives you a nice performanceboost if you
> are trying to handle lots of connections. (the default is based on the
> amount of memory in the machine and it's normally ok for desktop
> machines and small servers/routers)

I just wanted to followup publicly for archival purposes.

OK, so getting more into the realm of "best practices", by default the
maximum number of connections tracked will be 8x the number of buckets.

The number of buckets is determined at boot time based on amount of ram
(this in turns determines max connections that can be tracked -- (8 *
buckets see above)).

This is fine for desktop, small routers, etc.

However, both numbers can be tuned independently of each other.

On a box with lots of connections flowing through it (ie, a dedicated
high volume NAT/firewall/router), you can get better performance if your
max connections per bucket ratio is 2:1 instead of the default 8:1.

On Red Hat Linux, to do this tuning, figure out how many maximum
connections you want to track, divide this number in half (make sure it
isn't a 2^n number on kernels < 2.4.21).  This is how many buckets you
should have.  To configure your system with this many buckets, add the
following lines to your /etc/modules.conf file:

(The 44000 used below is just an example, estimate your worst case
scenario and add 10% to it).

# I want to have a 2:1 bucket to connection ratio for good performance
# Since I want to have a maximum of 44000 connections tracked, I
# set the number of buckets to 1/2 that value for a 2:1 ratio. 
options ip_conntrack hashsize=22000

Now by default you'll have 8 * 22000 this number of maximum connections
which is much higher than the  To readjust this down to the good 2:1
ratio add the following line to your /etc/sysctl.conf file:

net.ipv4.ip_conntrack_max = 44000

Now to figure out maximum possible memory consumption, do the following:

(memory used per buckets * number of buckets) + (memory used per tracked
connection * maximum number of tracked connections)

Memory used per bucket is 8 bytes on 32bit hardware, 16 bytes on 64bit
hardware.

To determine "memory used per tracked connection", run this command:

grep ip_conntrack /proc/slabinfo  | tr -s " " | cut -d " " -f 4

Did I miss anything or do you have anything to add Martin?

Dax Kelson
Guru Labs

Re: Memory usage for ip_conntrack

[Carlos Carvalho]

Martin Josefsson (gandalf@wlug.westbo.se) wrote on 18 July 2003 23:28:
 >> If I echo 102400 > /proc/sys/net/ipv4/ip_conntrack_max, what is my worst
 >> case memory usage?
 >
 >Don't do this. This will increase the maximum number of connections it
 >will track, but not the number of buckets. Which means that it will be
 >slower due to longer collision-chains. Instead increase the number of
 >buckets. modprobe ip_conntrack hashsize=131072 (or any number here.

How can we increase the number of buckets with a monolithic kernel?

Re: Memory usage for ip_conntrack

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 1167 bytes --]

On Sat, Jul 19, 2003 at 09:10:40PM -0300, Carlos Carvalho wrote:
> Martin Josefsson (gandalf@wlug.westbo.se) wrote on 18 July 2003 23:28:
>  >> If I echo 102400 > /proc/sys/net/ipv4/ip_conntrack_max, what is my worst
>  >> case memory usage?
>  >
>  >Don't do this. This will increase the maximum number of connections it
>  >will track, but not the number of buckets. Which means that it will be
>  >slower due to longer collision-chains. Instead increase the number of
>  >buckets. modprobe ip_conntrack hashsize=131072 (or any number here.
> 
> How can we increase the number of buckets with a monolithic kernel?

For 2.4: by altering the default in the kernel source, sorry.
For 2.5/2.6: there is now a generic way of specifying module parameters
from the boot command line.

-- 
- Harald Welte <laforge@netfilter.org>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]