Finding memory leak

Finding memory leak

[Holger Kiehl]

Hello

I have several systems where memory usage goes up and up until the systems
just hangup. I assume that the leak is somewhere in kernel since killing all
applications does not give me any memory back. Looking at /proc/slabinfo I
can see that size-2048 is always constantly going up, here a full output:

    cat /proc/slabinfo
    slabinfo - version: 1.1 (SMP)
    kmem_cache            64     64    244    4    4    1 :  252  126
    urb_priv               0      0     64    0    0    1 :  252  126
    tcp_tw_bucket         30     30    128    1    1    1 :  252  126
    tcp_bind_bucket      149    336     32    3    3    1 :  252  126
    tcp_open_request       0      0     64    0    0    1 :  252  126
    inet_peer_cache        1     58     64    1    1    1 :  252  126
    ip_fib_hash           15    224     32    2    2    1 :  252  126
    ip_dst_cache      163890 163890    256 10926 10926    1 :  252  126
    arp_cache             90     90    128    3    3    1 :  252  126
    blkdev_requests     3072   3090    128  103  103    1 :  252  126
    journal_head         392  47894     48   17  622    1 :  252  126
    revoke_table           2    250     12    1    1    1 :  252  126
    revoke_record        210    336     32    3    3    1 :  252  126
    dnotify_cache          0      0     20    0    0    1 :  252  126
    file_lock_cache      160    160     96    4    4    1 :  252  126
    fasync_cache           0      0     16    0    0    1 :  252  126
    uid_cache            137    224     32    2    2    1 :  252  126
    skbuff_head_cache 164571 164610    256 10974 10974    1 :  252  126
    sock                 132    132    896   33   33    1 :  124   62
    sigqueue              87     87    132    3    3    1 :  252  126
    kiobuf                 0      0     64    0    0    1 :  252  126
    cdev_cache           143    232     64    4    4    1 :  252  126
    bdev_cache            10     58     64    1    1    1 :  252  126
    mnt_cache             14     58     64    1    1    1 :  252  126
    inode_cache         1526   1526    512  218  218    1 :  124   62
    dentry_cache        2165   2460    128   82   82    1 :  252  126
    filp                1869   1890    128   63   63    1 :  252  126
    names_cache            8      8   4096    8    8    1 :   60   30
    buffer_head        74538  93690    128 2790 3123    1 :  252  126
    mm_struct            270    270    256   18   18    1 :  252  126
    vm_area_struct      4027   4380    128  146  146    1 :  252  126
    fs_cache             343    348     64    6    6    1 :  252  126
    files_cache          182    182    512   26   26    1 :  124   62
    signal_act           164    187   1408   17   17    4 :   60   30
    size-131072(DMA)       0      0 131072    0    0   32 :    0    0
    size-131072            5      5 131072    5    5   32 :    0    0
    size-65536(DMA)        0      0  65536    0    0   16 :    0    0
    size-65536             0      0  65536    0    0   16 :    0    0
    size-32768(DMA)        0      0  32768    0    0    8 :    0    0
    size-32768             2      3  32768    2    3    8 :    0    0
    size-16384(DMA)        0      0  16384    0    0    4 :    0    0
    size-16384            19     19  16384   19   19    4 :    0    0
    size-8192(DMA)         0      0   8192    0    0    2 :    0    0
    size-8192             10     12   8192   10   12    2 :    0    0
    size-4096(DMA)         0      0   4096    0    0    1 :   60   30
    size-4096            126    126   4096  126  126    1 :   60   30
    size-2048(DMA)         0      0   2048    0    0    1 :   60   30
    size-2048         164232 164232   2048 82116 82116    1 :   60   30
    size-1024(DMA)         0      0   1024    0    0    1 :  124   62
    size-1024            212    212   1024   53   53    1 :  124   62
    size-512(DMA)          0      0    512    0    0    1 :  124   62
    size-512             264    264    512   33   33    1 :  124   62
    size-256(DMA)          0      0    256    0    0    1 :  252  126
    size-256             210    210    256   14   14    1 :  252  126
    size-128(DMA)          3     30    128    1    1    1 :  252  126
    size-128           83404  84510    128 2817 2817    1 :  252  126
    size-64(DMA)           0      0    128    0    0    1 :  252  126
    size-64             1572   1650    128   55   55    1 :  252  126
    size-32(DMA)          51     58     64    1    1    1 :  252  126
    size-32             1241   1276     64   22   22    1 :  252  126

This happens with any of the 2.4.2? kernels and have also tried different
hardware (also with no SMP). But size-2048 is always going up. I think
I narrowed this down to a DVB card driver which is currently not in kernel
but the source of the driver is available.

There are some kernel values that are changed:

   net.core.rmem_max=1500000
   net.core.wmem_max=1500000
   net.ipv4.route.max_size=30000000

But even with default values memory it is still leaking.

The question that I have are: 

   - Is it correct to assume that an application cannot be the cause
     of the leak, ie. it can only be in kernel or a driver?
   - How can I really prove that its the driver leaking the memory?
   - What is the meaning of size-2048 in slabinfo?

Thanks,
Holger

PS: Please cc me since I am not on the list.

Re: Finding memory leak

[Andrew Morton]

Holger Kiehl <Holger.Kiehl@dwd.de> wrote:
>
>    - Is it correct to assume that an application cannot be the cause
>       of the leak, ie. it can only be in kernel or a driver?

Yes.

>     - How can I really prove that its the driver leaking the memory?

Look for instances of kmalloc(n, ...) for n in the range 1025 to 2048 which
do not have a matching kfree.

>     - What is the meaning of size-2048 in slabinfo?

It is the pool which is used to satisfy kmalloc requests in the range 1025
to 2048 bytes inclusive.

Re: Finding memory leak

[Holger Kiehl]

On Wed, 15 Oct 2003, Andrew Morton wrote:

> Holger Kiehl <Holger.Kiehl@dwd.de> wrote:
> >
> >    - Is it correct to assume that an application cannot be the cause
> >       of the leak, ie. it can only be in kernel or a driver?
> 
> Yes.
> 
> >     - How can I really prove that its the driver leaking the memory?
> 
> Look for instances of kmalloc(n, ...) for n in the range 1025 to 2048 which
> do not have a matching kfree.
> 
Could only find two instances one allocating 14 bytes and the other 4096.
Looking further I found several calls of dev_alloc_skb(). But I assume
these calls are also not the cause, because I put in some printk and
the size-2048 values do not go up when this is called.

Since the only thing running at that time is some application that receives
a multicast stream from the DVB card and writes the data to the disk, I put
in printk's at every kmalloc() or allocation functions in net/core and
net/ipv4. After many trials I could narrow this down to every time the
function igmpv3_newpack() in net/ipv4/igmp.c is called. This is always
called twice very quickly (there where never any other printk's between
these two calls). The size allocated by alloc_skb() in igmpv3_newpack()
when size-2048 in slabinfo goes up is 1529 bytes (in skbuff.c this allocates
1596 bytes).

The kernel network code is way beyond my knowledge and I cannot determine
how it frees up the memory again. Also, it is very very likely that the leak
is at some completely different place and igmpv3_newpack() is just called
shortly before or afterwards.

So the questions I have are:

  - Could someone please check igmpv3_newpack() and assure me that there
    is no leak.

  - Are there any tools or kernel switches that I can use that allow me
    to capture the leak faster? Currently I am using 2.4.22.

  - Which system call might cause a igmpv3_newpack().

Thanks,
Holger

Re: Finding memory leak

[Jörn Engel]

On Fri, 17 October 2003 13:26:44 +0000, Holger Kiehl wrote:
> On Wed, 15 Oct 2003, Andrew Morton wrote:
> > Holger Kiehl <Holger.Kiehl@dwd.de> wrote:
> > >
> > >    - Is it correct to assume that an application cannot be the cause
> > >       of the leak, ie. it can only be in kernel or a driver?
> > 
> > Yes.
> > 
> > >     - How can I really prove that its the driver leaking the memory?
> > 
> > Look for instances of kmalloc(n, ...) for n in the range 1025 to 2048 which
> > do not have a matching kfree.
> > 
> Could only find two instances one allocating 14 bytes and the other 4096.
> Looking further I found several calls of dev_alloc_skb(). But I assume
> these calls are also not the cause, because I put in some printk and
> the size-2048 values do not go up when this is called.
> 
> Since the only thing running at that time is some application that receives
> a multicast stream from the DVB card and writes the data to the disk, I put
> in printk's at every kmalloc() or allocation functions in net/core and
> net/ipv4. After many trials I could narrow this down to every time the
> function igmpv3_newpack() in net/ipv4/igmp.c is called. This is always
> called twice very quickly (there where never any other printk's between
> these two calls). The size allocated by alloc_skb() in igmpv3_newpack()
> when size-2048 in slabinfo goes up is 1529 bytes (in skbuff.c this allocates
> 1596 bytes).
> 
> The kernel network code is way beyond my knowledge and I cannot determine
> how it frees up the memory again. Also, it is very very likely that the leak
> is at some completely different place and igmpv3_newpack() is just called
> shortly before or afterwards.
> 
> So the questions I have are:
> 
>   - Could someone please check igmpv3_newpack() and assure me that there
>     is no leak.

There was a leak, found by the stanford checker team.  I've provided a
broken fix, DaveM wanted to write a decent one.  Not sure if it has
already found it's way into the official kernel.

Jörn

-- 
Happiness isn't having what you want, it's wanting what you have.
-- unknown

Re: Finding memory leak

[Jörn Engel]

On Sat, 18 October 2003 19:23:53 +0200, Jörn Engel wrote:
> > 
> >   - Could someone please check igmpv3_newpack() and assure me that there
> >     is no leak.
> 
> There was a leak, found by the stanford checker team.  I've provided a
> broken fix, DaveM wanted to write a decent one.  Not sure if it has
> already found it's way into the official kernel.

If DaveM hasn't fixed it yet, you can also try this patch.  Since I'm
pretty unaware of the networking code, this may be broken again, but
it also can't make things much worse for you.

Jörn

-- 
Write programs that do one thing and do it well. Write programs to work
together. Write programs to handle text streams, because that is a
universal interface. 
-- Doug MacIlroy

--- linux-2.6.0-test5/net/ipv4/igmp.c~igmp_memleak	2003-09-17 15:21:02.000000000 +0200
+++ linux-2.6.0-test5/net/ipv4/igmp.c	2003-10-18 19:46:17.000000000 +0200
@@ -70,6 +70,8 @@
  *		Alexey Kuznetsov:	Accordance to igmp-v2-06 draft.
  *		David L Stevens:	IGMPv3 support, with help from
  *					Vinay Kulkarni
+ *		Jörn Engel:		Fix memleak in igmpv3_newpack, reported
+ *					by David Yu Chen (stanford checker)
  */
 
 
@@ -280,8 +282,10 @@
 				    .nl_u = { .ip4_u = {
 				    .daddr = IGMPV3_ALL_MCR } },
 				    .proto = IPPROTO_IGMP };
-		if (ip_route_output_key(&rt, &fl))
+		if (ip_route_output_key(&rt, &fl)) {
+			kfree_skb(skb);
 			return 0;
+		}
 	}
 	if (rt->rt_src == 0) {
 		ip_rt_put(rt);

Re: Finding memory leak

[Jörn Engel]

On Sat, 18 October 2003 19:46:36 +0200, Jörn Engel wrote:
> On Sat, 18 October 2003 19:23:53 +0200, Jörn Engel wrote:
> > > 
> > >   - Could someone please check igmpv3_newpack() and assure me that there
> > >     is no leak.
> > 
> > There was a leak, found by the stanford checker team.  I've provided a
> > broken fix, DaveM wanted to write a decent one.  Not sure if it has
> > already found it's way into the official kernel.
> 
> If DaveM hasn't fixed it yet, you can also try this patch.  Since I'm
> pretty unaware of the networking code, this may be broken again, but
> it also can't make things much worse for you.

Or this one.  igmpv3_newpack() is to a large degree identical to
igmp_send_report().  The latter doesn't have a memleak, so I basically
adjusted the former to behave identical.  Should be foolproof, even
for me.

The space was added to make those two lines identical in both
functions as well.  If someone prefers 80 columns to 81, please adjust
both lines.

Andrew, DaveM, can this go into -test9?

Jörn

-- 
Data expands to fill the space available for storage.
-- Parkinson's Law

--- linux-2.6.0-test5/net/ipv4/igmp.c~igmp_memleak	2003-09-17 15:21:02.000000000 +0200
+++ linux-2.6.0-test5/net/ipv4/igmp.c	2003-10-18 19:51:37.000000000 +0200
@@ -70,6 +70,8 @@
  *		Alexey Kuznetsov:	Accordance to igmp-v2-06 draft.
  *		David L Stevens:	IGMPv3 support, with help from
  *					Vinay Kulkarni
+ *		Jörn Engel:		Fix memleak in igmpv3_newpack, reported
+ *					by David Yu Chen (stanford checker)
  */
 
 
@@ -271,10 +273,6 @@
 	struct iphdr *pip;
 	struct igmpv3_report *pig;
 
-	skb = alloc_skb(size + dev->hard_header_len + 15, GFP_ATOMIC);
-	if (skb == NULL)
-		return 0;
-
 	{
 		struct flowi fl = { .oif = dev->ifindex,
 				    .nl_u = { .ip4_u = {
@@ -288,12 +286,18 @@
 		return 0;
 	}
 
+	skb = alloc_skb(size + dev->hard_header_len + 15, GFP_ATOMIC);
+	if (skb == NULL) {
+		ip_rt_put(rt);
+		return 0;
+	}
+
 	skb->dst = &rt->u.dst;
 	skb->dev = dev;
 
 	skb_reserve(skb, (dev->hard_header_len+15)&~15);
 
-	skb->nh.iph = pip =(struct iphdr *)skb_put(skb, sizeof(struct iphdr)+4);
+	skb->nh.iph = pip = (struct iphdr *)skb_put(skb, sizeof(struct iphdr)+4);
 
 	pip->version  = 4;
 	pip->ihl      = (sizeof(struct iphdr)+4)>>2;

Re: Finding memory leak

[Holger Kiehl]

On Sat, 18 Oct 2003, Jörn Engel wrote:

> On Sat, 18 October 2003 19:23:53 +0200, Jörn Engel wrote:
> > > 
> > >   - Could someone please check igmpv3_newpack() and assure me that there
> > >     is no leak.
> > 
> > There was a leak, found by the stanford checker team.  I've provided a
> > broken fix, DaveM wanted to write a decent one.  Not sure if it has
> > already found it's way into the official kernel.
> 
> If DaveM hasn't fixed it yet, you can also try this patch.  Since I'm
> pretty unaware of the networking code, this may be broken again, but
> it also can't make things much worse for you.
> 
I cannot use this patch since I am using 2.4.22, and there igmpv3_newpack()
looks a bit different. I also cannot test 2.6.0-test since the driver for
the DVB card is only for 2.4.x.

But your patch did go in, looking at 2.6.0-test8 there now is a
kfree_skb(skb).

Holger

Re: Finding memory leak

[Holger Kiehl]

On Sat, 18 Oct 2003, Jörn Engel wrote:

> On Sat, 18 October 2003 19:46:36 +0200, Jörn Engel wrote:
> > On Sat, 18 October 2003 19:23:53 +0200, Jörn Engel wrote:
> > > > 
> > > >   - Could someone please check igmpv3_newpack() and assure me that there
> > > >     is no leak.
> > > 
> > > There was a leak, found by the stanford checker team.  I've provided a
> > > broken fix, DaveM wanted to write a decent one.  Not sure if it has
> > > already found it's way into the official kernel.
> > 
> > If DaveM hasn't fixed it yet, you can also try this patch.  Since I'm
> > pretty unaware of the networking code, this may be broken again, but
> > it also can't make things much worse for you.
> 
> Or this one.  igmpv3_newpack() is to a large degree identical to
> igmp_send_report().  The latter doesn't have a memleak, so I basically
> adjusted the former to behave identical.  Should be foolproof, even
> for me.
> 
Did try this, but the leak is still there. The size-2048 values is always
slowly going up.

Holger

Re: Finding memory leak

[David S. Miller]

On Sat, 18 Oct 2003 19:59:12 +0200
Jörn Engel <joern@wohnheim.fh-wedel.de> wrote:

> Andrew, DaveM, can this go into -test9?

Please have a look at current sources before asking questions
like this ok? :-)

Even as of -test7 or -test8 your patch is in there.  It's been in
there for a long time.

Re: Finding memory leak

[Holger Kiehl]

On Sun, 19 Oct 2003, David S. Miller wrote:

> On Sat, 18 Oct 2003 19:59:12 +0200
> Jörn Engel <joern@wohnheim.fh-wedel.de> wrote:
> 
> > Andrew, DaveM, can this go into -test9?
> 
> Please have a look at current sources before asking questions
> like this ok? :-)
> 
But what about 2.4.22? I am having a leak there, my size-2048 value
is always going up. As mentioned earlier this seems to happen
everytime igmpv3_newpack() gets called and allocates 1529 bytes.

I took igmpv3_newpack() from 2.6.0-test8, had to made some changes

        {
                struct rt_key key = {};

                key.dst = IGMPV3_ALL_MCR;
                key.oif = dev->ifindex;
                if (ip_route_output_key(&rt, &key)) {
                        kfree_skb(skb);
                        return 0;
                }
        }

since there is no struct flowi in 2.4.x. Don't know if the above is
correct, but the leak is still there.

The size-2048 values always go up when we receive data via a DVB card from
multiple multicast streams. Depending on the amount of memory the system
has, they always hangup after a certain time.

What else can I do find the leak?

Please help.

Thanks,
Holger