Hi, Needs suggestions for finding and fixing stack/memory corruption when calling a function

Hi, Needs suggestions for finding and fixing stack/memory corruption when calling a function

[Fawad Lateef]

Hello,

I need some suggestions about how-to approach, find and fix a memory
corruption issue which is happening in a C/C++ very complex and large
code (code evolved over several years). Code is running on AT91SAM9260
(armv5l architecture; single processor with preemption enabled) and
completely in Linux user-space.

The problem is:

-- We are calling a function which has three integer arguments.
With-in that function 2nd and 3rd arguments always gets corrupted
while 1st argument is fine. Just before calling that function printing
arguments is fine.

Now it will be good if I can get some suggestions about whats
happening and how-to look into this problem. I am thinking that there
is some memory/stack corruption happening somewhere.

Thanks in advance.

-- Fawad Lateef

Re: Hi, Needs suggestions for finding and fixing stack/memory corruption when calling a function

[Andrej Gelenberg]

Hi,


heap, stack, buffer overflow? multiple threads? It is bit difficult to
suggest something without some code. Have you tried to use gdb or
valgrind? (do someone know if valgrind work on arm?)

Regards,
Andrej Gelenberg

On 05/23/2011 04:41 PM, Fawad Lateef wrote:
> Hello,
> 
> I need some suggestions about how-to approach, find and fix a memory
> corruption issue which is happening in a C/C++ very complex and large
> code (code evolved over several years). Code is running on AT91SAM9260
> (armv5l architecture; single processor with preemption enabled) and
> completely in Linux user-space.
> 
> The problem is:
> 
> -- We are calling a function which has three integer arguments.
> With-in that function 2nd and 3rd arguments always gets corrupted
> while 1st argument is fine. Just before calling that function printing
> arguments is fine.
> 
> Now it will be good if I can get some suggestions about whats
> happening and how-to look into this problem. I am thinking that there
> is some memory/stack corruption happening somewhere.
> 
> Thanks in advance.
> 
> -- Fawad Lateef
> --
> To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Hi, Needs suggestions for finding and fixing stack/memory corruption when calling a function

[Zhongye Jia]

would you please provide your problem code? according to your
description, it highly seems that some of your pointers accidentally
write that memory, but i'm not sure.
you may use a debugger to watch that block of memory and see when it changed.

On Mon, May 23, 2011 at 10:41 PM, Fawad Lateef <fawadlateef@gmail.com> wrote:
> Hello,
>
> I need some suggestions about how-to approach, find and fix a memory
> corruption issue which is happening in a C/C++ very complex and large
> code (code evolved over several years). Code is running on AT91SAM9260
> (armv5l architecture; single processor with preemption enabled) and
> completely in Linux user-space.
>
> The problem is:
>
> -- We are calling a function which has three integer arguments.
> With-in that function 2nd and 3rd arguments always gets corrupted
> while 1st argument is fine. Just before calling that function printing
> arguments is fine.
>
> Now it will be good if I can get some suggestions about whats
> happening and how-to look into this problem. I am thinking that there
> is some memory/stack corruption happening somewhere.
>
> Thanks in advance.
>
> -- Fawad Lateef
> --
> To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Hi, Needs suggestions for finding and fixing stack/memory corruption when calling a function

[Fawad Lateef]

Hi Andrej,


On Mon, May 23, 2011 at 3:45 PM, Andrej Gelenberg
<andrej.gelenberg@udo.edu> wrote:
> Hi,
>
>
> heap, stack, buffer overflow? multiple threads? It is bit difficult to
> suggest something without some code. Have you tried to use gdb or
> valgrind? (do someone know if valgrind work on arm?)
>

Thanks for replying.

There is _no_ threading only while(1) loop, we have multiple processes
communicating through pipes but this problem is happening only in one
process hence I am assuming that something local to that process is
corrupting memory.

Our root-filesystem don't have gdb support and I think valgrind is
_not_ supported on arm :(

Is there any possibility that the problem is related to some compiler
optimization or something along that line ? We are using gcc-4.2.0
based tool-chain and Linux kernel 2.6.29.

I can't post the code due to two main reasons: -- Its closed source
and its too big and I don't think that anyone wants to look at code
with around 20 cpp files and each file has hundreds of lines of code.

Regards,

Fawad Lateef

> Regards,
> Andrej Gelenberg
>
> On 05/23/2011 04:41 PM, Fawad Lateef wrote:
>> Hello,
>>
>> I need some suggestions about how-to approach, find and fix a memory
>> corruption issue which is happening in a C/C++ very complex and large
>> code (code evolved over several years). Code is running on AT91SAM9260
>> (armv5l architecture; single processor with preemption enabled) and
>> completely in Linux user-space.
>>
>> The problem is:
>>
>> -- We are calling a function which has three integer arguments.
>> With-in that function 2nd and 3rd arguments always gets corrupted
>> while 1st argument is fine. Just before calling that function printing
>> arguments is fine.
>>
>> Now it will be good if I can get some suggestions about whats
>> happening and how-to look into this problem. I am thinking that there
>> is some memory/stack corruption happening somewhere.
>>
>> Thanks in advance.
>>
>> -- Fawad Lateef
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Hi, Needs suggestions for finding and fixing stack/memory corruption when calling a function

[Mahavir Jain]

On Mon, May 23, 2011 at 8:29 PM, Fawad Lateef <fawadlateef@gmail.com> wrote:
> Hi Andrej,
>
>
> On Mon, May 23, 2011 at 3:45 PM, Andrej Gelenberg
> <andrej.gelenberg@udo.edu> wrote:
>> Hi,
>>
>>
>> heap, stack, buffer overflow? multiple threads? It is bit difficult to
>> suggest something without some code. Have you tried to use gdb or
>> valgrind? (do someone know if valgrind work on arm?)
>>
>
> Thanks for replying.
>
> There is _no_ threading only while(1) loop, we have multiple processes
> communicating through pipes but this problem is happening only in one
> process hence I am assuming that something local to that process is
> corrupting memory.
>
> Our root-filesystem don't have gdb support and I think valgrind is
> _not_ supported on arm :(

Try gdb-server that should fit in small root filesystem.
http://www.delorie.com/gnu/docs/gdb/gdbserver.1.html

>
> Is there any possibility that the problem is related to some compiler
> optimization or something along that line ? We are using gcc-4.2.0
> based tool-chain and Linux kernel 2.6.29.
>
> I can't post the code due to two main reasons: -- Its closed source
> and its too big and I don't think that anyone wants to look at code
> with around 20 cpp files and each file has hundreds of lines of code.
>
> Regards,
>
> Fawad Lateef
>
>> Regards,
>> Andrej Gelenberg
>>
>> On 05/23/2011 04:41 PM, Fawad Lateef wrote:
>>> Hello,
>>>
>>> I need some suggestions about how-to approach, find and fix a memory
>>> corruption issue which is happening in a C/C++ very complex and large
>>> code (code evolved over several years). Code is running on AT91SAM9260
>>> (armv5l architecture; single processor with preemption enabled) and
>>> completely in Linux user-space.
>>>
>>> The problem is:
>>>
>>> -- We are calling a function which has three integer arguments.
>>> With-in that function 2nd and 3rd arguments always gets corrupted
>>> while 1st argument is fine. Just before calling that function printing
>>> arguments is fine.
>>>
>>> Now it will be good if I can get some suggestions about whats
>>> happening and how-to look into this problem. I am thinking that there
>>> is some memory/stack corruption happening somewhere.
>>>
>>> Thanks in advance.
>>>
>>> -- Fawad Lateef
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Thanks,
MJ
--
To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Hi, Needs suggestions for finding and fixing stack/memory corruption when calling a function

[Fawad Lateef]

On Tue, May 24, 2011 at 7:08 AM, ern0 <ern0@linkbroker.hu> wrote:
> Sounds like wrong compilation. Have you recompiled both the caller and the
> callee functions? Is the compiling target set correctly (btw ARM integers
> are always put on 32-bit boundaries)? What are the types of the parameters?
> Try to change its order. Finally, if nothing works, pass them different way,
> one-by-one (setFirst(), setSecond(), SetThirdAndperform()) or with pointers,
> I mean foo(int*) instead of foo(int).
>

Thanks for reply and suggestions. Both functions are part of single
application and I tried rebuilding all of them again. For now I added
two more temporary arguments between 1st and 2nd arguments and the
required arguments getting correct values thats why I am assuming that
by adding temporary variables we moved the memory corruption from real
arguments into temporary arguments. After adding these arguments my
function is behaving correctly so for now this is a work-around _but_
I want to figure-out its proper fix (as the same sort of issue we
faced in our cgi based web-pages code some weeks earlier).

Hence I needs suggestion that is this really a memory corruption or
some compiler issues ? If its a compiler issue then its not a big
problem but if its memory corruption issue then we have to _fix_ it
asap.

My compilation arguments are:

* Compilation ----- armv5l-linux-g++ -Wall -Wno-write-strings
-mcpu=arm9 -Os -DARM9_LINUX_GCC -D_GNU_SOURCE -DUNIX -pipe -fPIC -c -o
xxxxxxxxxxx.o xxxxxxxxxxx.cc

* Linking Shared Library ----- armv5l-linux-g++ -shared -o xxxxxx.so

* Linking Executable ----- armv5l-linux-g++ -Wl,-rpath,/mnt/apps/lib
-Wall -Wno-write-strings -mcpu=arm9 -Os -DARM9_LINUX_GCC -D_GNU_SOURCE
-DUNIX -pipe -s ..........


Thanks,

Fawad Lateef


> On Mon, May 23, 2011 at 16:41, Fawad Lateef <fawadlateef@gmail.com> wrote:
>>
>> Hello,
>>
>> I need some suggestions about how-to approach, find and fix a memory
>> corruption issue which is happening in a C/C++ very complex and large
>> code (code evolved over several years). Code is running on AT91SAM9260
>> (armv5l architecture; single processor with preemption enabled) and
>> completely in Linux user-space.
>>
>> The problem is:
>>
>> -- We are calling a function which has three integer arguments.
>> With-in that function 2nd and 3rd arguments always gets corrupted
>> while 1st argument is fine. Just before calling that function printing
>> arguments is fine.
>>
>> Now it will be good if I can get some suggestions about whats
>> happening and how-to look into this problem. I am thinking that there
>> is some memory/stack corruption happening somewhere.
>>
>> Thanks in advance.
>>
>> -- Fawad Lateef
>> --
>> To unsubscribe from this list: send the line "unsubscribe
>> linux-c-programming" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> ern0
> Haben Sie Fragen?
>
--
To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Hi, Needs suggestions for finding and fixing stack/memory corruption when calling a function

[Glynn Clements]

Fawad Lateef wrote:

> For now I added two more temporary arguments between 1st and 2nd
> arguments and the required arguments getting correct values thats
> why I am assuming that by adding temporary variables we moved the
> memory corruption from real arguments into temporary arguments. 
> After adding these arguments my function is behaving correctly so
> for now this is a work-around _but_ I want to figure-out its proper
> fix (as the same sort of issue we faced in our cgi based web-pages
> code some weeks earlier).
> 
> Hence I needs suggestion that is this really a memory corruption or
> some compiler issues ? If its a compiler issue then its not a big
> problem but if its memory corruption issue then we have to _fix_ it
> asap.

Check the assembler output ("gcc -S ...").

Personally, I'd be more inclined to suspect register corruption. Are
you calling a library function which use a different ABI? E.g. if your
code is compiled for an ABI where certain registers are preserved but
calling a function using an ABI which doesn't preserver those
registers, then you'll have problems.

-- 
Glynn Clements <glynn@gclements.plus.com>

Re: Hi, Needs suggestions for finding and fixing stack/memory corruption when calling a function

[Fawad Lateef]

Hi Glynn,


On Tue, May 24, 2011 at 1:48 PM, Glynn Clements
<glynn@gclements.plus.com> wrote:
>
> Fawad Lateef wrote:
>
>> For now I added two more temporary arguments between 1st and 2nd
>> arguments and the required arguments getting correct values thats
>> why I am assuming that by adding temporary variables we moved the
>> memory corruption from real arguments into temporary arguments.
>> After adding these arguments my function is behaving correctly so
>> for now this is a work-around _but_ I want to figure-out its proper
>> fix (as the same sort of issue we faced in our cgi based web-pages
>> code some weeks earlier).
>>
>> Hence I needs suggestion that is this really a memory corruption or
>> some compiler issues ? If its a compiler issue then its not a big
>> problem but if its memory corruption issue then we have to _fix_ it
>> asap.
>
> Check the assembler output ("gcc -S ...").
>

Nice idea, will look into this.

> Personally, I'd be more inclined to suspect register corruption. Are
> you calling a library function which use a different ABI? E.g. if your
> code is compiled for an ABI where certain registers are preserved but
> calling a function using an ABI which doesn't preserver those
> registers, then you'll have problems.
>

humm, Can you give some suggestion about how to verify this ? Using
readelf or objdump on different libraries and executables can give
some clue ? As we are using some pre-compiled  libraries provided by
some stack we are using.

Thanks for the help.

Regards,

Fawad Lateef


> --
> Glynn Clements <glynn@gclements.plus.com>
>

Re: Hi, Needs suggestions for finding and fixing stack/memory corruption when calling a function

[Glynn Clements]

Fawad Lateef wrote:

> > Personally, I'd be more inclined to suspect register corruption. Are
> > you calling a library function which use a different ABI? E.g. if your
> > code is compiled for an ABI where certain registers are preserved but
> > calling a function using an ABI which doesn't preserver those
> > registers, then you'll have problems.
> 
> humm, Can you give some suggestion about how to verify this ? Using
> readelf or objdump on different libraries and executables can give
> some clue ? As we are using some pre-compiled  libraries provided by
> some stack we are using.

The files themselves may not contain sufficient information. E.g. if
someone builds a library using e.g. -mregparm=3, that isn't going to
show up in the objdump (etc) output. First, check the documentation
for the libraries; if that doesn't help, try disassembling them.

-- 
Glynn Clements <glynn@gclements.plus.com>

Re: Hi, Needs suggestions for finding and fixing stack/memory corruption when calling a function

[Fawad Lateef]

Hi,

Just want to say thanks for helping me and want to post solution which
worked for me; so that anyone facing similar issue can try this.

I used -O2 rather than -Os in my compiler options and the problem
vanished from all components. So somehow using option -Os optimizes
the size _but_ seems to be messing with function arguments (I looked
at assembler code generated by armv5l-linux-gcc and found big
differences in argument passing between -Os and -O2).
So my current compiler options are:

armv5l-linux-g++ -Wall -Wno-write-strings -mcpu=arm9 -O2
-DARM9_LINUX_GCC -D_GNU_SOURCE -DUNIX -pipe -fPIC -c -o xxxxxxx.o
xxxxxxx.cc

Regards,

-- Fawad Lateef

P.S. I did top-posting rather than bottom-posting because I am not
replying to some comments.


On Wed, May 25, 2011 at 12:07 PM, Glynn Clements
<glynn@gclements.plus.com> wrote:
>
> Fawad Lateef wrote:
>
> > > Personally, I'd be more inclined to suspect register corruption. Are
> > > you calling a library function which use a different ABI? E.g. if your
> > > code is compiled for an ABI where certain registers are preserved but
> > > calling a function using an ABI which doesn't preserver those
> > > registers, then you'll have problems.
> >
> > humm, Can you give some suggestion about how to verify this ? Using
> > readelf or objdump on different libraries and executables can give
> > some clue ? As we are using some pre-compiled  libraries provided by
> > some stack we are using.
>
> The files themselves may not contain sufficient information. E.g. if
> someone builds a library using e.g. -mregparm=3, that isn't going to
> show up in the objdump (etc) output. First, check the documentation
> for the libraries; if that doesn't help, try disassembling them.
>
> --
> Glynn Clements <glynn@gclements.plus.com>
--
To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html