Memory leak with dev_add_pack()

Memory leak with dev_add_pack()

[Spiro Trikaliotis]

Hello,

I have been tasked with writing a kernel module to handle an own
protocol (with an own EtherType) in the Linux kernel.

While I am not a complete newbie in the Linux kernel, having to handle
Ethernet traffic there is new to me.

I have looked at several tutorials and also at sample code (for example
http://www.phrack.org/archives/55/p55_0x0c_Building%20Into%20The%20Linux%20Network%20Layer_by_lifeline%20&%20kossak.txt,
and some tutorials I cannot find at the moment...) and tried to cut it
down to the bare essentials.

I came up with the following code utilising Linux Protocol Module:

   http://www.trikaliotis.net/testmodule/test_kernel.c

   the correspondig Makefile is in
   http://www.trikaliotis.net/testmodule/Makefile

Note: This is a cut-down version of the actual code which shows the
problem. Additionally, I register for ETH_P_ALL so the problem occurs as
fast as possible.

While it essentially works as I want it to, I found that I am leaking
memory this way. Having this module in a highly loaded network, 
"head -n5 /proc/meminfo" shows that MemFree is steadily falling, until
the kernel OOPS with an out of memory condition.

"Of course", if I undefine DO_MEMLEAK, the memory leak does not occur
anymore.

I tried for almost two weeks now to look at different tutorials, and I
also looked in working code (for example, the IP LPM) and tried to
compare with my code, but I could not find out what I am doing wrong
here. I am sure it must be something obvious, and very silly -
unfortunaely, for me, it is not obvious at the moment.

Can anyone help me here, please?

Best regards,
Spiro.

-- 
Spiro R. Trikaliotis                              http://opencbm.sf.net/
http://www.trikaliotis.net/                     http://www.viceteam.org/

Memory leak with dev_add_pack()

[Michael Blizek]

Hi!

On 21:54 Fri 21 Jan     , Spiro Trikaliotis wrote:
...
> While it essentially works as I want it to, I found that I am leaking
> memory this way. Having this module in a highly loaded network, 
> "head -n5 /proc/meminfo" shows that MemFree is steadily falling, until
> the kernel OOPS with an out of memory condition.

I could not see anything wrong here either. Some things which came to my
mind where:

- Does this also happen, if you do not call PromiscuousMode_Enter?
- Which user grows is /proc/slabinfo? (If this file is emply or does not
  exist, you may meed to recompile the kernel to use slab instead of
  sl[b-z]b)
- Which kernel version are you using? Have you tried different versions?

	-Michi
-- 
programing a layer 3+4 network protocol for mesh networks
see http://michaelblizek.twilightparadox.com

Memory leak with dev_add_pack()

[Spiro Trikaliotis]

Hallo Michi,

first of all, thank you for your answer.

* On Sat, Jan 22, 2011 at 08:18:05AM +0100 Michael Blizek wrote:
 
> On 21:54 Fri 21 Jan     , Spiro Trikaliotis wrote:
 
> - Does this also happen, if you do not call PromiscuousMode_Enter?

Yes, it does.

In fact, I only included it in my minimal example so the memory leak
happens faster, in case someone wants to give it a try himself.

> - Which user grows is /proc/slabinfo? (If this file is emply or does not
>   exist, you may meed to recompile the kernel to use slab instead of
>   sl[b-z]b)

Thank you for the pointer, it might help.

The objects which have changed the most on a mildly loaded network are:

25000 buffer_head
 5000 dentry
 5000 ext3_inode_cache
 5000 size-64

The number if the number of more active object after approx. 20h of
letting it run on the mildly loaded network.

Note that the ext3_inode_cache might have grown because I was regularly
writing a new log file from slabinfo (slabinfo --once > slabinfo.`date
+...`)

Thus, I would expect I am leaking buffer_head.


> - Which kernel version are you using? Have you tried different versions?

I am running this on a Debian Lenny (32 bit), Kernel 2.6.26-2-686, and
an Ubuntu 10.04, also 32 bit. Unfortunately this machine is currently
out of my reach, but I think it is a variant of 2.6.32 or 2.6.33.

Both kernels are from the distribution, that is, no vanilla kernels.
 

Beste Gr??e
   Spiro.

-- 
Spiro R. Trikaliotis                              http://opencbm.sf.net/
http://www.trikaliotis.net/                     http://www.viceteam.org/

Memory leak with dev_add_pack()

[Spiro Trikaliotis]

Hello Anish,

* On Sat, Jan 22, 2011 at 11:02:12PM +0530 anish kumar wrote:

>> I have been tasked with writing a kernel module to handle an own
>> protocol (with an own EtherType) in the Linux kernel.
>
> are you writing a kernel based sniffer??

Not really. I should take a custom protocol (with its own EtherType) and
tunnel it in a specific way to another machine.

In this sense: Yes, it is a sniffer.

However, as I wrote in my original mail, I used ETH_P_ALL instead of the
EtherType I need in order to make sure that I am leaking more memory, so
people can see it themselves faster.

>>   http://www.trikaliotis.net/testmodule/test_kernel.c
>
> which kernel version are you using and above code would not be sufficient 
> to find out the problem.Can
> you send the above file and makefile as well??

The problem occurrs with different kernel versions, as I wrote in my
other mail. It is a 2.6.26-2-686 from Debian Lenny, and a kernel
on Ubuntu 10.04, also 32 bit. Unfortunately this machine is currently
out of my reach, but I think it is a variant of 2.6.32 or 2.6.33.

The Makefile is already there, see the next part of my original message:

>>   the correspondig Makefile is in
>>   http://www.trikaliotis.net/testmodule/Makefile


>> While it essentially works as I want it to, I found that I am leaking
>
> if it is sniffer are you getting all the packets???

I would say yes. The fully functional version works as expected, if I
neglect the memory leak. Thus, I would expect I get all of the packets I
need.

Best regards,
Spiro.

-- 
Spiro R. Trikaliotis                              http://opencbm.sf.net/
http://www.trikaliotis.net/                     http://www.viceteam.org/

Memory leak with dev_add_pack()

[Michael Blizek]

Hi!

On 11:40 Sun 23 Jan     , Spiro Trikaliotis wrote:
...
> * On Sat, Jan 22, 2011 at 08:18:05AM +0100 Michael Blizek wrote:
...
> > - Which user grows is /proc/slabinfo? (If this file is emply or does not
> >   exist, you may meed to recompile the kernel to use slab instead of
> >   sl[b-z]b)
> 
> Thank you for the pointer, it might help.
> 
> The objects which have changed the most on a mildly loaded network are:
> 
> 25000 buffer_head
>  5000 dentry
>  5000 ext3_inode_cache
>  5000 size-64
> 
> The number if the number of more active object after approx. 20h of
> letting it run on the mildly loaded network.
> 
> Note that the ext3_inode_cache might have grown because I was regularly
> writing a new log file from slabinfo (slabinfo --once > slabinfo.`date
> +...`)
> 
> Thus, I would expect I am leaking buffer_head.

The funny point is that buffer_head belongs to the filesystem subsystem. What
file systems are you using? Could it be that your code just triggers the
memory leak, because data is logged to disk, e.g. to /var/log/kern.log ?

I have tried your program on my virtual machine (2.6.28) and could see any
leak, but maybe data is leaked very slowly...

	-Michi
-- 
programing a layer 3+4 network protocol for mesh networks
see http://michaelblizek.twilightparadox.com

Memory leak with dev_add_pack()

[Daniel Baluta]

Hi Spiro,

> While it essentially works as I want it to, I found that I am leaking
> memory this way. Having this module in a highly loaded network,
> "head -n5 /proc/meminfo" shows that MemFree is steadily falling, until
> the kernel OOPS with an out of memory condition.

Have you considered using kmemleak [1] to help your debugging?

thanks,
Daniel.

[1] http://lxr.linux.no/linux+v2.6.37/Documentation/kmemleak.txt

Memory leak with dev_add_pack()

[anish kumar]

> Hi!
>
> On 11:40 Sun 23 Jan     , Spiro Trikaliotis wrote:
> ...
>> * On Sat, Jan 22, 2011 at 08:18:05AM +0100 Michael Blizek wrote:
> ...
>> > - Which user grows is /proc/slabinfo? (If this file is emply or does 
>> > not
>> >   exist, you may meed to recompile the kernel to use slab instead of
>> >   sl[b-z]b)
>>
>> Thank you for the pointer, it might help.
>>
>> The objects which have changed the most on a mildly loaded network are:
>>
>> 25000 buffer_head
>>  5000 dentry
>>  5000 ext3_inode_cache
>>  5000 size-64
>>
>> The number if the number of more active object after approx. 20h of
>> letting it run on the mildly loaded network.
>>
>> Note that the ext3_inode_cache might have grown because I was regularly
>> writing a new log file from slabinfo (slabinfo --once > slabinfo.`date
>> +...`)
>>
>> Thus, I would expect I am leaking buffer_head.
>
> The funny point is that buffer_head belongs to the filesystem subsystem. 
> What
> file systems are you using? Could it be that your code just triggers the
> memory leak, because data is logged to disk, e.g. to /var/log/kern.log ?
I did small experiment with your code.I removed all your logs which were 
getting
logged in kernel buffers(kern.log).With this change i checked the "meminfo" 
and found that
the memory leaking is almost same as compared to normal case(with no change 
in your code).

>
> I have tried your program on my virtual machine (2.6.28) and could see any
> leak, but maybe data is leaked very slowly...
i confirm that data is leaking very slowly (below is the output i got on my 
ubuntu machine
with removed logs from your code).

$date
Sun Jan 23 19:13:47 RET 2011

$ head -n5 /proc/meminfo
MemTotal:        1018172 kB
MemFree:          373708 kB
Buffers:           32232 kB
Cached:           296004 kB
SwapCached:            0 kB

$ date
Sun Jan 23 19:15:03 RET 2011

$ head -n5 /proc/meminfo
MemTotal:        1018172 kB
MemFree:          373584 kB
Buffers:           32248 kB
Cached:           296004 kB
SwapCached:            0 kB

With logs enabled in your code i can see marginal increase in leaking 
memory.
I can defintely see the memory leaking with your ko and will investigate 
further as to the reason.
Hopefully kmemleak will lead us somewhere.

>
> -Michi
> -- 
> programing a layer 3+4 network protocol for mesh networks
> see http://michaelblizek.twilightparadox.com
>
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies 

Memory leak with dev_add_pack()

[Michael Blizek]

Hi!

On 21:22 Sun 23 Jan     , anish kumar wrote:
> > The funny point is that buffer_head belongs to the filesystem subsystem. 
> > What
> > file systems are you using? Could it be that your code just triggers the
> > memory leak, because data is logged to disk, e.g. to /var/log/kern.log ?
> I did small experiment with your code.I removed all your logs which were 
> getting
> logged in kernel buffers(kern.log).With this change i checked the "meminfo" 
> and found that
> the memory leaking is almost same as compared to normal case(with no change 
> in your code).
> 
> >
> > I have tried your program on my virtual machine (2.6.28) and could see any
> > leak, but maybe data is leaked very slowly...
> i confirm that data is leaking very slowly (below is the output i got on my 
> ubuntu machine
> with removed logs from your code).
> 
> $date
> Sun Jan 23 19:13:47 RET 2011
> 
> $ head -n5 /proc/meminfo
> MemTotal:        1018172 kB
> MemFree:          373708 kB
> Buffers:           32232 kB
> Cached:           296004 kB
> SwapCached:            0 kB
> 
> $ date
> Sun Jan 23 19:15:03 RET 2011
> 
> $ head -n5 /proc/meminfo
> MemTotal:        1018172 kB
> MemFree:          373584 kB
> Buffers:           32248 kB
> Cached:           296004 kB
> SwapCached:            0 kB
> 
> With logs enabled in your code i can see marginal increase in leaking 
> memory.

This small increase could easily be caused by background activity. Please try
to reproduce this in single user mode with all file systems mounted read-only
(use e.g. "mount -o remount,ro /") and no processes running except init and
your shell. Then see if memory increase is both correlated to network traffic
and reason of system crashes. If it takes too long, you can add e.g.
"mem=32M" to the kernel boot parameter list in the bootloader.

	-Michi
-- 
programing a layer 3+4 network protocol for mesh networks
see http://michaelblizek.twilightparadox.com

Memory leak with dev_add_pack()

[anish kumar]

> Hi!
>
> On 21:22 Sun 23 Jan     , anish kumar wrote:
>> > The funny point is that buffer_head belongs to the filesystem 
>> > subsystem.
>> > What
>> > file systems are you using? Could it be that your code just triggers 
>> > the
>> > memory leak, because data is logged to disk, e.g. to /var/log/kern.log 
>> > ?
>> I did small experiment with your code.I removed all your logs which were
>> getting
>> logged in kernel buffers(kern.log).With this change i checked the 
>> "meminfo"
>> and found that
>> the memory leaking is almost same as compared to normal case(with no 
>> change
>> in your code).
>>
>> >
>> > I have tried your program on my virtual machine (2.6.28) and could see 
>> > any
>> > leak, but maybe data is leaked very slowly...
>> i confirm that data is leaking very slowly (below is the output i got on 
>> my
>> ubuntu machine
>> with removed logs from your code).
>>
>> $date
>> Sun Jan 23 19:13:47 RET 2011
>>
>> $ head -n5 /proc/meminfo
>> MemTotal:        1018172 kB
>> MemFree:          373708 kB
>> Buffers:           32232 kB
>> Cached:           296004 kB
>> SwapCached:            0 kB
>>
>> $ date
>> Sun Jan 23 19:15:03 RET 2011
>>
>> $ head -n5 /proc/meminfo
>> MemTotal:        1018172 kB
>> MemFree:          373584 kB
>> Buffers:           32248 kB
>> Cached:           296004 kB
>> SwapCached:            0 kB
>>
>> With logs enabled in your code i can see marginal increase in leaking
>> memory.
>
> This small increase could easily be caused by background activity. Please 
> try
> to reproduce this in single user mode with all file systems mounted 
> read-only
> (use e.g. "mount -o remount,ro /") and no processes running except init 
> and
> your shell. Then see if memory increase is both correlated to network 
> traffic
> and reason of system crashes. If it takes too long, you can add e.g.
> "mem=32M" to the kernel boot parameter list in the bootloader.
Thanks for this suggestions.

After persuading my kernel to use only 100M(with 32M system not booting,it 
is not vanilla kernel)
of my physical memory and after installing your ko i tried to check meminfo 
and found
that the claim made by you(.ko is leaking memory) may not be true as i can 
see in the
output of /proc/meminfo,MemFree is sometimes increasing and sometimes 
decreasing(could
be because of several other components).
I can say with certainty that your ko looks fine to me after this testing.
I couldn't see any panic happening in my system( system is running fine 
under such low memory).
However i see that several other applications(xserv) is crashing because of 
out of memory which i think is fine.

ps:I dont have vanilla kernel where i could have changed the intitialisation 
files to remount while booting and could have
controlled the list of process running on the system.
In android we are doing that in init.rc file need to check in ubuntu.

> -Michi
> -- 
> programing a layer 3+4 network protocol for mesh networks
> see http://michaelblizek.twilightparadox.com
> 

Memory leak with dev_add_pack()

[Michael Blizek]

Hi!

On 22:59 Sun 23 Jan     , anish kumar wrote:
...
> ps:I dont have vanilla kernel where i could have changed the intitialisation 
> files to remount while booting and could have
> controlled the list of process running on the system.
> In android we are doing that in init.rc file need to check in ubuntu.

Why not type "init 1" in a root shell?

	-Michi
-- 
programing a layer 3+4 network protocol for mesh networks
see http://michaelblizek.twilightparadox.com

Memory leak with dev_add_pack()

[anish singh]

On Mon, Jan 24, 2011 at 9:38 PM, Michael Blizek <
michi1@michaelblizek.twilightparadox.com> wrote:

> Hi!
>
> On 22:59 Sun 23 Jan     , anish kumar wrote:
> ...
> > ps:I dont have vanilla kernel where i could have changed the
> intitialisation
> > files to remount while booting and could have
> > controlled the list of process running on the system.
> > In android we are doing that in init.rc file need to check in ubuntu.
>
> Why not type "init 1" in a root shell?
>

Thanks for this information but looks like this is not properly supported in
my ubuntu system as
after this command it is dropping to recovery menu.


>        -Michi
> --
> programing a layer 3+4 network protocol for mesh networks
> see http://michaelblizek.twilightparadox.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110125/3e371618/attachment.html