Debugging the linux kernel core dump

Debugging the linux kernel core dump

[Akash]

Hello,

I have a core dump from following machine :
<http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies>
# uname -a
Linux sing-vto2-dst 2.6.34.8-0.2-desktop #1 SMP PREEMPT 2011-04-06 18:11:26
+0200 x86_64 x86_64 x86_64 GNU/Linux

I have opened this core in gdb

# gdb vmlinux-2.6.34.8-0.2-desktop kernel.core0
GNU gdb 6.6 (build 2009-10-05)
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
(no debugging symbols found)
Program terminated with signal 11, Segmentation fault.
#0  0xffffffff8100ba60 in default_idle ()


How can I add symbols to gdb ?
When I try to add symbols from System.map I get following error :

(gdb) add-symbol-file System.map-2.6.34.8-0.2-desktop
The address where System.map-2.6.34.8-0.2-desktop has been loaded is missing


At what address should the System.map be added ? How can I find out that
address ?

Thanks for any help.

Akash
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110804/4edac619/attachment.html 

Debugging the linux kernel core dump

[amit mehta]

Not sure about the symbol table, but was your kernel compiled
with -g flag (-g to the CFLAGS variable in the kernel Makefile) ?

-Amit

On Fri, Aug 5, 2011 at 10:42 AM, Akash <email2akashjain@gmail.com> wrote:
> Hello,
>
> I have a core dump from following machine :
>
> # uname -a
> Linux sing-vto2-dst 2.6.34.8-0.2-desktop #1 SMP PREEMPT 2011-04-06 18:11:26
> +0200 x86_64 x86_64 x86_64 GNU/Linux
>
> I have opened this core in gdb
>
> # gdb vmlinux-2.6.34.8-0.2-desktop kernel.core0
> GNU gdb 6.6 (build 2009-10-05)
> Copyright (C) 2006 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.? Type "show warranty" for details.
> (no debugging symbols found)
> Program terminated with signal 11, Segmentation fault.
> #0? 0xffffffff8100ba60 in default_idle ()
>
>
> How can I add symbols to gdb ?
> When I try to add symbols from System.map I get following error :
>
> (gdb) add-symbol-file System.map-2.6.34.8-0.2-desktop
> The address where System.map-2.6.34.8-0.2-desktop has been loaded is missing
>
>
> At what address should the System.map be added ? How can I find out that
> address ?
>
> Thanks for any help.
>
> Akash
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>
>

Debugging the linux kernel core dump

[Akash]

I think it was not built with -g otherwise the symbols would have been there
in the vmlinux itself, right ?
That is the reason I need to add symbols externally.
Please dont ask me to rebuild the kernel, this is vanilla OpenSUSE 11.3
symbols and debug kernel should be available somewhere. I just need to know
how/where to add the symbols. The problem is not reproducible, I need to
debug the core which I have on hand.

Thanks.
Akash


On Thu, Aug 4, 2011 at 10:19 PM, amit mehta <gmate.amit@gmail.com> wrote:

> Not sure about the symbol table, but was your kernel compiled
> with -g flag (-g to the CFLAGS variable in the kernel Makefile) ?
>
> -Amit
>
> On Fri, Aug 5, 2011 at 10:42 AM, Akash <email2akashjain@gmail.com> wrote:
> > Hello,
> >
> > I have a core dump from following machine :
> >
> > # uname -a
> > Linux sing-vto2-dst 2.6.34.8-0.2-desktop #1 SMP PREEMPT 2011-04-06
> 18:11:26
> > +0200 x86_64 x86_64 x86_64 GNU/Linux
> >
> > I have opened this core in gdb
> >
> > # gdb vmlinux-2.6.34.8-0.2-desktop kernel.core0
> > GNU gdb 6.6 (build 2009-10-05)
> > Copyright (C) 2006 Free Software Foundation, Inc.
> > GDB is free software, covered by the GNU General Public License, and you
> are
> > welcome to change it and/or distribute copies of it under certain
> > conditions.
> > Type "show copying" to see the conditions.
> > There is absolutely no warranty for GDB.  Type "show warranty" for
> details.
> > (no debugging symbols found)
> > Program terminated with signal 11, Segmentation fault.
> > #0  0xffffffff8100ba60 in default_idle ()
> >
> >
> > How can I add symbols to gdb ?
> > When I try to add symbols from System.map I get following error :
> >
> > (gdb) add-symbol-file System.map-2.6.34.8-0.2-desktop
> > The address where System.map-2.6.34.8-0.2-desktop has been loaded is
> missing
> >
> >
> > At what address should the System.map be added ? How can I find out that
> > address ?
> >
> > Thanks for any help.
> >
> > Akash
> > _______________________________________________
> > Kernelnewbies mailing list
> > Kernelnewbies at kernelnewbies.org
> > http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110804/fe16f503/attachment.html 

Debugging the linux kernel core dump

[amit mehta]

On Fri, Aug 5, 2011 at 10:53 AM, Akash <email2akashjain@gmail.com> wrote:
> I think it was not built with -g otherwise the symbols would have been there
> in the vmlinux itself, right ?
Right.
> That is the reason I need to add symbols externally.
> Please dont ask me to rebuild the kernel, this is vanilla OpenSUSE 11.3
> symbols and debug kernel should be available somewhere. I just need to know
> how/where to add the symbols. The problem is not reproducible, I need to
> debug the core which I have on hand.
>
> Thanks.
> Akash
>
I "think" System.map file is not the file which gbd understands , On
my system, i tried this:
goon:/boot # gdb -q
(gdb) symbol-file System.map-2.6.32.12-0.7-default -readnow
`/boot/System.map-2.6.32.12-0.7-default': can't read symbols: File
format not recognized.

I might be missing something else as well.

-Amit

>
> On Thu, Aug 4, 2011 at 10:19 PM, amit mehta <gmate.amit@gmail.com> wrote:
>>
>> Not sure about the symbol table, but was your kernel compiled
>> with -g flag (-g to the CFLAGS variable in the kernel Makefile) ?
>>
>> -Amit
>>
>> On Fri, Aug 5, 2011 at 10:42 AM, Akash <email2akashjain@gmail.com> wrote:
>> > Hello,
>> >
>> > I have a core dump from following machine :
>> >
>> > # uname -a
>> > Linux sing-vto2-dst 2.6.34.8-0.2-desktop #1 SMP PREEMPT 2011-04-06
>> > 18:11:26
>> > +0200 x86_64 x86_64 x86_64 GNU/Linux
>> >
>> > I have opened this core in gdb
>> >
>> > # gdb vmlinux-2.6.34.8-0.2-desktop kernel.core0
>> > GNU gdb 6.6 (build 2009-10-05)
>> > Copyright (C) 2006 Free Software Foundation, Inc.
>> > GDB is free software, covered by the GNU General Public License, and you
>> > are
>> > welcome to change it and/or distribute copies of it under certain
>> > conditions.
>> > Type "show copying" to see the conditions.
>> > There is absolutely no warranty for GDB.? Type "show warranty" for
>> > details.
>> > (no debugging symbols found)
>> > Program terminated with signal 11, Segmentation fault.
>> > #0? 0xffffffff8100ba60 in default_idle ()
>> >
>> >
>> > How can I add symbols to gdb ?
>> > When I try to add symbols from System.map I get following error :
>> >
>> > (gdb) add-symbol-file System.map-2.6.34.8-0.2-desktop
>> > The address where System.map-2.6.34.8-0.2-desktop has been loaded is
>> > missing
>> >
>> >
>> > At what address should the System.map be added ? How can I find out that
>> > address ?
>> >
>> > Thanks for any help.
>> >
>> > Akash
>> > _______________________________________________
>> > Kernelnewbies mailing list
>> > Kernelnewbies at kernelnewbies.org
>> > http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>> >
>> >
>
>

Debugging the linux kernel core dump

[Mulyadi Santosa]

Hi...

On Fri, Aug 5, 2011 at 12:23, Akash <email2akashjain@gmail.com> wrote:
> I think it was not built with -g otherwise the symbols would have been there
> in the vmlinux itself, right ?
> That is the reason I need to add symbols externally.
> Please dont ask me to rebuild the kernel, this is vanilla OpenSUSE 11.3
> symbols and debug kernel should be available somewhere. I just need to know
> how/where to add the symbols. The problem is not reproducible, I need to
> debug the core which I have on hand.

try to check opensuse repo, see which one provide kinda "debuginfo"
package for your kernel (at least that's how Fedora/RHEL call it).
Usually it will be located under /boot with quite obvious name
(suffixed with "debug" or similar) or you could just try to use
vmlinux as the symbol file

NB: make sure you use the very same debuginfo package with your
current kernel version. Any slight difference could yield very
different symbol addresses.

-- 
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com

Debugging the linux kernel core dump

[Akash]

On Fri, Aug 5, 2011 at 9:23 PM, Mulyadi Santosa
<mulyadi.santosa@gmail.com>wrote:

> Hi...
>
> On Fri, Aug 5, 2011 at 12:23, Akash <email2akashjain@gmail.com> wrote:
> > I think it was not built with -g otherwise the symbols would have been
> there
> > in the vmlinux itself, right ?
> > That is the reason I need to add symbols externally.
> > Please dont ask me to rebuild the kernel, this is vanilla OpenSUSE 11.3
> > symbols and debug kernel should be available somewhere. I just need to
> know
> > how/where to add the symbols. The problem is not reproducible, I need to
> > debug the core which I have on hand.
>
> try to check opensuse repo, see which one provide kinda "debuginfo"
> package for your kernel (at least that's how Fedora/RHEL call it).
> Usually it will be located under /boot with quite obvious name
> (suffixed with "debug" or similar) or you could just try to use
> vmlinux as the symbol file
>
> NB: make sure you use the very same debuginfo package with your
> current kernel version. Any slight difference could yield very
> different symbol addresses.
>


Thanks Malyudi. Thats sounds like a reasonable approach. I did try to search
kernel-debug RPMs for my kernel could not find any. May be I should just
keep looking.

On that note, I wonder what is the difference between kernel-base,
kernel-default, kernel-devel, and similar other RPMs. I guess some contain
binaries, some contain unstripped binaries and some contain soruces/headers.
But which is which ?? I hate distros for creating so many RPMs. They could
have included ALL of this in one RPM and users would not have minded.

->Akash


> --
> regards,
>
> Mulyadi Santosa
> Freelance Linux trainer and consultant
>
> blog: the-hydra.blogspot.com
> training: mulyaditraining.blogspot.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20110805/c3c46102/attachment-0001.html 

Debugging the linux kernel core dump

[Mulyadi Santosa]

Hi Akash...

On Sat, Aug 6, 2011 at 12:06, Akash <email2akashjain@gmail.com> wrote:
> Thanks Malyudi. Thats sounds like a reasonable approach. I did try to search
> kernel-debug RPMs for my kernel could not find any. May be I should just
> keep looking.

AFAIK too, in Ubuntu and its derivatives, kernel related packages are
likely named "linux-image".

So maybe you could try that name prefix too.... a hint: since you use
openSuse, try to use zypper "search" facility, so that you could
reduce mistakes

> On that note, I wonder what is the difference between kernel-base,
> kernel-default, kernel-devel, and similar other RPMs.

most of the times, anything named with "kernel" and "devel" usually
means a package that contains the kernel headers file and others
needed to compile a kernel module or anything related to kernel stuff.
it's a stripped down version of full blown kernel source.

kernel base? kernel default? no clear hint there...


-- 
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com