2.4.20-ac2 Memory Leak?

2.4.20-ac2 Memory Leak?

[Gregory Stark]

My router box has a problem, it seems to be running out of memory. Programs
that worked fine earlier are now swapping like crazy. 

What confuses me is that if I add up all the RSS of the processes I get 5.9M,
a number drastically lower than the available RAM on the machine (24M) and
drastically lower than the amount of RAM "free" says is taken (22M).

It seems something in kernel space has taken a ton of memory out of play?
Or is my diagnosis wrong?


USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.8  1216  196 ?        S    Mar09   0:06 init [2] 
root         2  0.0  0.0     0    0 ?        SW   Mar09   0:02 [keventd]
root         3  0.0  0.0     0    0 ?        SWN  Mar09   0:01 [ksoftirqd_CPU0]
root         4  0.0  0.0     0    0 ?        SW   Mar09   0:25 [kswapd]
root         5  0.0  0.0     0    0 ?        SW   Mar09   0:00 [bdflush]
root         6  0.0  0.0     0    0 ?        SW   Mar09   0:00 [kupdated]
root       111  0.0  0.0     0    0 ?        SW   Mar09   0:00 [eth0]
root       118  0.0  0.8  1408  192 ?        S    Mar09   0:08 dhclient -pf /var/run/dhclient.eth0.pid eth0
root       182  0.0  1.1  1284  248 ?        S    Mar09   0:18 /sbin/syslogd
root       185  0.0  0.7  1840  168 ?        S    Mar09   0:04 /sbin/klogd
root       205  0.0  0.6  1236  156 ?        S    Mar09   0:00 /usr/sbin/inetd
root       212  0.0  1.0  2688  240 ?        S    Mar09   0:00 /usr/sbin/sshd
root       224  0.0  0.6  2308  140 tty1     S    Mar09   0:03 -bash
stark      225  0.0  0.5  2276  116 tty2     S    Mar09   0:01 -bash
stark      226  0.0  0.6  2300  144 tty3     S    Mar09   0:06 -bash
root       227  0.0  0.5  1200  116 tty4     S    Mar09   0:00 /sbin/getty 38400 tty4
root       228  0.0  0.5  1200  116 tty5     S    Mar09   0:00 /sbin/getty 38400 tty5
root       229  0.0  0.5  1200  116 tty6     S    Mar09   0:00 /sbin/getty 38400 tty6
root       236  0.0  0.6  1736  144 tty1     S    Mar09   0:03 tail -100f /var/log/ppp.log
root       245  0.0  0.8  2000  184 ?        S    Mar09   1:09 pppd call provider
root       253  0.0  0.0     0    0 ?        SW   Mar09   0:00 [eth1]
root       303  0.0  0.7  2088  160 ?        S    Mar09   0:25 /usr/sbin/zhm arilinn.mit.edu neskaya.mit.edu
root       494  0.0  0.5  5772  128 ?        S    Mar09   0:00 /usr/sbin/sshd
stark      496  0.0  1.5  5904  356 ?        S    Mar09   1:47 /usr/sbin/sshd
stark      497  0.5  2.7  3412  620 ?        S    Mar09  69:08 zwgc -ttymode -nofork -f /home/stark/.zwgc.desc
root      1315  0.0  0.7  1440  160 ?        S    Mar10   0:07 /usr/sbin/dhcpd -q eth1
root     11109  0.0  1.9  2316  432 tty2     S    14:02   0:01 bash
root     11367  0.0  1.9  2316  432 tty2     S    18:07   0:00 bash
root     11368  0.0  3.3  2488  744 tty2     R    18:07   0:00 ps auxww



        total:    used:    free:  shared: buffers:  cached:
Mem:  22962176 22597632   364544        0   245760  3223552
Swap: 67489792  4120576 63369216
MemTotal:        22424 kB
MemFree:           356 kB
MemShared:           0 kB
Buffers:           240 kB
Cached:           2352 kB
SwapCached:        796 kB
Active:           3008 kB
Inact_dirty:       192 kB
Inact_clean:       340 kB
Inact_target:      708 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:        22424 kB
LowFree:           356 kB
SwapTotal:       65908 kB
SwapFree:        61884 kB
Committed_AS:     8552 kB



Module                  Size  Used by    Not tainted
ipt_MASQUERADE          1784   0  (autoclean)
ipt_LOG                 3384   1  (autoclean)
ipt_state                536   1  (autoclean)
ipt_TCPMSS              2360   1  (autoclean)
ip_nat_ftp              3568   0  (unused)
iptable_nat            21016   2  [ipt_MASQUERADE ip_nat_ftp]
ip_conntrack_irc        3152   0  (unused)
ip_conntrack_ftp        4208   1  [ip_nat_ftp]
ip_conntrack           27776   4  [ipt_MASQUERADE ipt_state ip_nat_ftp iptable_nat ip_conntrack_irc ip_conntrack_ftp]
ethertap                3108   1 
iptable_filter          1644   1  (autoclean)
ip_tables              14296   8  [ipt_MASQUERADE ipt_LOG ipt_state ipt_TCPMSS iptable_nat iptable_filter]
pppoe                   7820   1 
pppox                   1176   1  [pppoe]
ppp_generic            19836   3  (autoclean) [pppoe pppox]
slhc                    5104   0  (autoclean) [ppp_generic]
8139too                14216   2 
rtc                     6588   0  (autoclean)


-- 
greg

Re: 2.4.20-ac2 Memory Leak?

[Martin Josefsson]

[-- Attachment #1: Type: text/plain, Size: 888 bytes --]

On Wed, 2003-03-19 at 00:17, Gregory Stark wrote:
> My router box has a problem, it seems to be running out of memory. Programs
> that worked fine earlier are now swapping like crazy. 
> 
> What confuses me is that if I add up all the RSS of the processes I get 5.9M,
> a number drastically lower than the available RAM on the machine (24M) and
> drastically lower than the amount of RAM "free" says is taken (22M).
> 
> It seems something in kernel space has taken a ton of memory out of play?
> Or is my diagnosis wrong?

2.4.20 changed the linked list handling and ip_conntrack relied on the
old way. I've attached a patch that removes this assumption from
ip_conntrack.

This can be the source of your problems, connections can get very long
timeouts and stay in ip_conntrack.

-- 
/Martin

Never argue with an idiot. They drag you down to their level, then beat you with experience.

[-- Attachment #2: 10_confirm_fix.patch --]
[-- Type: text/x-patch, Size: 6426 bytes --]

diff -urN --exclude-from=diff.exclude linux-2.4.20-base/include/linux/netfilter_ipv4/ip_conntrack.h linux-2.4.20-del/include/linux/netfilter_ipv4/ip_conntrack.h
--- linux-2.4.20-base/include/linux/netfilter_ipv4/ip_conntrack.h	Fri Nov 29 00:53:15 2002
+++ linux-2.4.20-del/include/linux/netfilter_ipv4/ip_conntrack.h	Fri Feb 21 17:01:38 2003
@@ -6,6 +6,7 @@
 
 #include <linux/config.h>
 #include <linux/netfilter_ipv4/ip_conntrack_tuple.h>
+#include <linux/bitops.h>
 #include <asm/atomic.h>
 
 enum ip_conntrack_info
@@ -41,6 +42,10 @@
 	/* Conntrack should never be early-expired. */
 	IPS_ASSURED_BIT = 2,
 	IPS_ASSURED = (1 << IPS_ASSURED_BIT),
+
+	/* Connection is confirmed: originating packet has left box */
+	IPS_CONFIRMED_BIT = 3,
+	IPS_CONFIRMED = (1 << IPS_CONFIRMED_BIT),
 };
 
 #include <linux/netfilter_ipv4/ip_conntrack_tcp.h>
@@ -159,7 +164,7 @@
 	struct ip_conntrack_tuple_hash tuplehash[IP_CT_DIR_MAX];
 
 	/* Have we seen traffic both ways yet? (bitset) */
-	volatile unsigned long status;
+	unsigned long status;
 
 	/* Timer function; drops refcnt when it goes off. */
 	struct timer_list timeout;
@@ -254,7 +259,7 @@
 /* It's confirmed if it is, or has been in the hash table. */
 static inline int is_confirmed(struct ip_conntrack *ct)
 {
-	return ct->tuplehash[IP_CT_DIR_ORIGINAL].list.next != NULL;
+	return test_bit(IPS_CONFIRMED_BIT, &ct->status);
 }
 
 extern unsigned int ip_conntrack_htable_size;
diff -urN --exclude-from=diff.exclude linux-2.4.20-base/net/ipv4/netfilter/ip_conntrack_core.c linux-2.4.20-del/net/ipv4/netfilter/ip_conntrack_core.c
--- linux-2.4.20-base/net/ipv4/netfilter/ip_conntrack_core.c	Tue Feb 18 17:08:21 2003
+++ linux-2.4.20-del/net/ipv4/netfilter/ip_conntrack_core.c	Fri Feb 21 17:01:39 2003
@@ -292,9 +292,6 @@
 {
 	DEBUGP("clean_from_lists(%p)\n", ct);
 	MUST_BE_WRITE_LOCKED(&ip_conntrack_lock);
-	/* Remove from both hash lists: must not NULL out next ptrs,
-           otherwise we'll look unconfirmed.  Fortunately, LIST_DELETE
-           doesn't do this. --RR */
 	LIST_DELETE(&ip_conntrack_hash
 		    [hash_conntrack(&ct->tuplehash[IP_CT_DIR_ORIGINAL].tuple)],
 		    &ct->tuplehash[IP_CT_DIR_ORIGINAL]);
@@ -467,6 +464,7 @@
 		ct->timeout.expires += jiffies;
 		add_timer(&ct->timeout);
 		atomic_inc(&ct->ct_general.use);
+		set_bit(IPS_CONFIRMED_BIT, &ct->status);
 		WRITE_UNLOCK(&ip_conntrack_lock);
 		return NF_ACCEPT;
 	}
@@ -585,7 +583,7 @@
    connection.  Too bad: we're in trouble anyway. */
 static inline int unreplied(const struct ip_conntrack_tuple_hash *i)
 {
-	return !(i->ctrack->status & IPS_ASSURED);
+	return !(test_bit(IPS_ASSURED_BIT, &i->ctrack->status));
 }
 
 static int early_drop(struct list_head *chain)
@@ -720,7 +718,7 @@
 			conntrack, expected);
 		/* Welcome, Mr. Bond.  We've been expecting you... */
 		IP_NF_ASSERT(master_ct(conntrack));
-		conntrack->status = IPS_EXPECTED;
+		__set_bit(IPS_EXPECTED_BIT, &conntrack->status);
 		conntrack->master = expected;
 		expected->sibling = conntrack;
 		LIST_DELETE(&ip_conntrack_expect_list, expected);
@@ -768,11 +766,11 @@
 		*set_reply = 1;
 	} else {
 		/* Once we've had two way comms, always ESTABLISHED. */
-		if (h->ctrack->status & IPS_SEEN_REPLY) {
+		if (test_bit(IPS_SEEN_REPLY_BIT, &h->ctrack->status)) {
 			DEBUGP("ip_conntrack_in: normal packet for %p\n",
 			       h->ctrack);
 		        *ctinfo = IP_CT_ESTABLISHED;
-		} else if (h->ctrack->status & IPS_EXPECTED) {
+		} else if (test_bit(IPS_EXPECTED_BIT, &h->ctrack->status)) {
 			DEBUGP("ip_conntrack_in: related packet for %p\n",
 			       h->ctrack);
 			*ctinfo = IP_CT_RELATED;
diff -urN --exclude-from=diff.exclude linux-2.4.20-base/net/ipv4/netfilter/ip_conntrack_proto_tcp.c linux-2.4.20-del/net/ipv4/netfilter/ip_conntrack_proto_tcp.c
--- linux-2.4.20-base/net/ipv4/netfilter/ip_conntrack_proto_tcp.c	Tue Feb 18 17:07:26 2003
+++ linux-2.4.20-del/net/ipv4/netfilter/ip_conntrack_proto_tcp.c	Fri Feb 21 17:03:35 2003
@@ -192,7 +192,7 @@
 	   have an established connection: this is a fairly common
 	   problem case, so we can delete the conntrack
 	   immediately.  --RR */
-	if (!(conntrack->status & IPS_SEEN_REPLY) && tcph->rst) {
+	if (!test_bit(IPS_SEEN_REPLY_BIT, &conntrack->status) && tcph->rst) {
 		WRITE_UNLOCK(&tcp_lock);
 		if (del_timer(&conntrack->timeout))
 			conntrack->timeout.function((unsigned long)conntrack);
diff -urN --exclude-from=diff.exclude linux-2.4.20-base/net/ipv4/netfilter/ip_conntrack_proto_udp.c linux-2.4.20-del/net/ipv4/netfilter/ip_conntrack_proto_udp.c
--- linux-2.4.20-base/net/ipv4/netfilter/ip_conntrack_proto_udp.c	Fri Nov 29 00:53:15 2002
+++ linux-2.4.20-del/net/ipv4/netfilter/ip_conntrack_proto_udp.c	Fri Feb 21 17:01:39 2003
@@ -51,7 +51,7 @@
 {
 	/* If we've seen traffic both ways, this is some kind of UDP
 	   stream.  Extend timeout. */
-	if (conntrack->status & IPS_SEEN_REPLY) {
+	if (test_bit(IPS_SEEN_REPLY_BIT, &conntrack->status)) {
 		ip_ct_refresh(conntrack, UDP_STREAM_TIMEOUT);
 		/* Also, more likely to be important, and not a probe */
 		set_bit(IPS_ASSURED_BIT, &conntrack->status);
diff -urN --exclude-from=diff.exclude linux-2.4.20-base/net/ipv4/netfilter/ip_conntrack_standalone.c linux-2.4.20-del/net/ipv4/netfilter/ip_conntrack_standalone.c
--- linux-2.4.20-base/net/ipv4/netfilter/ip_conntrack_standalone.c	Fri Nov 29 00:53:15 2002
+++ linux-2.4.20-del/net/ipv4/netfilter/ip_conntrack_standalone.c	Fri Feb 21 21:10:37 2003
@@ -77,7 +77,7 @@
 }
 
 static unsigned int
-print_conntrack(char *buffer, const struct ip_conntrack *conntrack)
+print_conntrack(char *buffer, struct ip_conntrack *conntrack)
 {
 	unsigned int len;
 	struct ip_conntrack_protocol *proto
@@ -95,12 +95,12 @@
 	len += print_tuple(buffer + len,
 			   &conntrack->tuplehash[IP_CT_DIR_ORIGINAL].tuple,
 			   proto);
-	if (!(conntrack->status & IPS_SEEN_REPLY))
+	if (!(test_bit(IPS_SEEN_REPLY_BIT, &conntrack->status)))
 		len += sprintf(buffer + len, "[UNREPLIED] ");
 	len += print_tuple(buffer + len,
 			   &conntrack->tuplehash[IP_CT_DIR_REPLY].tuple,
 			   proto);
-	if (conntrack->status & IPS_ASSURED)
+	if (test_bit(IPS_ASSURED_BIT, &conntrack->status))
 		len += sprintf(buffer + len, "[ASSURED] ");
 	len += sprintf(buffer + len, "use=%u ",
 		       atomic_read(&conntrack->ct_general.use));

Re: 2.4.20-ac2 Memory Leak?

[Greg Stark]

Martin Josefsson <gandalf@wlug.westbo.se> writes:

> This can be the source of your problems, connections can get very long
> timeouts and stay in ip_conntrack.

Is there a way to list the connections and confirm this is the problem?
It seems it would require an awful lot of connections to consume megabytes of
memory.

Also, I've looked high and low and can't find this anywhere, how do i tune the
timeouts connections get? I have certain protocols that potentially receive
very little traffic and I want to make sure they don't time out.

--
greg

Re: 2.4.20-ac2 Memory Leak?

[Martin Josefsson]

On Wed, 2003-03-19 at 00:46, Greg Stark wrote:
> Martin Josefsson <gandalf@wlug.westbo.se> writes:
> 
> > This can be the source of your problems, connections can get very long
> > timeouts and stay in ip_conntrack.
> 
> Is there a way to list the connections and confirm this is the problem?
> It seems it would require an awful lot of connections to consume megabytes of
> memory.

cat /proc/net/ip_conntrack

The third field is the timeout in seconds.
The fourth field is the state of the connection, if it's TIME_WAIT with
a large timeout then it's the list handling bug. (iirc it was TIME_WAIT
that showed the problem...)

> Also, I've looked high and low and can't find this anywhere, how do i tune the
> timeouts connections get? I have certain protocols that potentially receive
> very little traffic and I want to make sure they don't time out.

The default tcp timeout is 5 days for esatblished connections.

There's a patch in patch-o-matic that enables you to tune the timeouts
without having to edit the source. Instructions on how to get
patch-o-matic are availiable on http://www.netfilter.org

-- 
/Martin

Never argue with an idiot. They drag you down to their level, then beat you with experience.

Re: 2.4.20-ac2 Memory Leak?

[Greg Stark]

Martin Josefsson <gandalf@wlug.westbo.se> writes:

> On Wed, 2003-03-19 at 00:46, Greg Stark wrote:
> > Martin Josefsson <gandalf@wlug.westbo.se> writes:
> > 
> > > This can be the source of your problems, connections can get very long
> > > timeouts and stay in ip_conntrack.
> > 
> > Is there a way to list the connections and confirm this is the problem?
> > It seems it would require an awful lot of connections to consume megabytes of
> > memory.
> 
> cat /proc/net/ip_conntrack
> 
> The third field is the timeout in seconds.
> The fourth field is the state of the connection, if it's TIME_WAIT with
> a large timeout then it's the list handling bug. (iirc it was TIME_WAIT
> that showed the problem...)

Nope, a total of 16 entries in ip_conntrack, one of which is in TIME_WAIT with
a timeout of 2m.



> The default tcp timeout is 5 days for esatblished connections.
> 
> There's a patch in patch-o-matic that enables you to tune the timeouts
> without having to edit the source. Instructions on how to get
> patch-o-matic are availiable on http://www.netfilter.org

Thanks.

-- 
greg

Re: 2.4.20-ac2 Memory Leak?

[Greg Stark]

> On Wed, 2003-03-19 at 00:17, Gregory Stark wrote:
> > My router box has a problem, it seems to be running out of memory. Programs
> > that worked fine earlier are now swapping like crazy. 
> > 
> > What confuses me is that if I add up all the RSS of the processes I get 5.9M,
> > a number drastically lower than the available RAM on the machine (24M) and
> > drastically lower than the amount of RAM "free" says is taken (22M).
> > 
> > It seems something in kernel space has taken a ton of memory out of play?
> > Or is my diagnosis wrong?

One thing I find suspicious is that my ISP was down yesterday morning, causing
pppd to try repeatedly to connect. It seems it attempted to connect over 3,000
times while the ISP was down. Every connection failed at the PAP
authentication stage.

Perhaps there's a memory leak in the pppoe connection initiation process?
Even so that would be about 4k per pppoe connection attempt. Hm, a page?

--
greg