Debugging GRUB2 with GDB and QEMU

Debugging GRUB2 with GDB and QEMU

[Lubomir Kundrak]

I wrote some GDB macros that might be helpful for people
that want to debug GRUB 2 modules with QEMU. [1]

[1] http://NetBSD.sk/~lkundrak/misc/grub2-gdb/

The file genmk.rb.diff is a patch to makefile-generator
script, that makes the build system leave output files with
debugging information. Of course, one has have ruby interpreter
to regenerate the makefiles and compile GRUB with -g
afterwards.

Second file is .gdbinit, the GDB initialization file.  It is
commented, so there is no need to describe it here. It basically
passes the information extracted from list headed by grub_dl_head to
the perl script, gmodule.pl, and loads its output. It also
sets a breakpoint whose command list contains macro for loading
proper symbol file each time a module is loaded. (I know the work
could be done without the help of a Perl script, but it would
be far more complicated, I guess)

Another note worth mentioning is that you'll probably want
to add aliases for some functions, so that all gdb facilities
will work correctly. You will at least want to define the symbol
'main' for backtraces to stop at the right place. Some facilities
also want the function 'malloc' to be defined. So, you'll
probably have to add something like

.globl main
main = codestart

to assembly language sources or

malloc() __attribute__ ((alias("grub_malloc")));

to C files.

I hope this will be useful to at least some of you. Best regards!
Lubo.

-- 
  o	Lubomir Kundrak
 *O*	<lkundrak@skosi.org>, http://skosi.org/~lkundrak/
 (_)	SKOSI -- Slovak OpenSource Initiative

Re: Debugging GRUB2 with GDB and QEMU

[Yoshinori K. Okuji]

On Friday 12 May 2006 15:20, Lubomir Kundrak wrote:
> I wrote some GDB macros that might be helpful for people
> that want to debug GRUB 2 modules with QEMU. [1]

Great!

I guess it is necessary to compile the binary with -O0 or -O1 to get a 
reasonable backtrace... I'd appreciate if you can write a HOWTO. :)

Okuji

RE : Debugging GRUB2 with GDB and QEMU

[Eric Salomé]

Hi Lubomir,

Thanks for the scripts. It is just what I was looking for.
Meanwhile, since my last email, I have done something foolish. 
I've coded in grub's sources just the same algorithm you provide but
I've clumbered the sources with new functions those purpose is only for
debugging while you are doing all the job outside of the sources, and
one might think it's better the way you do.

1) Got Ruby and change the makefiles just the way you do to get .elf
files with debugging info. (I've setup for a different extension but
I'll change for .elf)

2) Added a very few lines to grub_load_segments() to get the name of the
section along with its load address in mod->segment.
(You might want that to get ride of the readelf and perl script)

3) Build in a buffer, during build_load_core() and grub_load_segments()
processing, the string 
"filename.elf load_addr -s .text 0x23888 ... "
print it on the screen with grub_dprintf.

4) Call a function with that string argument to be caught by a .gdbinit
script to launch the add-symbol-file command before the call to
(mod->init(mod)).

I first thought I could just get along with add-symbol-file command to
gdb, but I was frustrated I could not set breakpoint ahead of time and
then have it resolved when the module is loaded .... I was in the mood
of doing C development so I've been overkilling again :
Before I even finished the .gdbinit script, I've changed my mind and
done "one" step further : 

5) Added the "filename.elf -s .text 0x23234 ... " to the list of
link_maps maintained by ld.so (actually ld.so complains a little but
I'll fix that later) and do the call to _ld_debug_state() to generate a
shlib event in gdb.

6) Changed the solib-svr4.c in gdb 6.1 so that it can handle loading ELF
file with instructions to load each segment separately at the right
place.

So I am running with a modified gdb that understand grub's way of
loading modules on the fly.

If you think it can be of general interest, we can start from here to
define a GDB interface to Grub 2. Otherwise, I'll use Ludomir external
scripts since I don't want to maintain separate source files from the
official delivery.

Note that if we change somewhat the way grub load modules and allocate
first the memory to contain all the segments, lying right one after
another in memory, we won't have to change gdb anymore (step 6).

Best regards,

_________________________________________
Eric Salomé - Paris, France

Re: Debugging GRUB2 with GDB and QEMU

[devShadow]

Hi, thanks for the help with debugging modules.  I have a module that I 
created and still need a little assistance in getting gdb to break in my 
  module's grub_cmd_*.  I am able to connect to QEMU with gdb and when I 
load the module, gdb loads the symbol table for my module.

But, when I set a breakpoint at the address of the first operation in my 
module, gdb doesn't break when I execute the command in GRUB.  If it has 
something to do with the aliases you mention at the bottom of your 
message, could you elaborate a little more or direct me in the right 
direction?

Thanks

David Johnson

Lubomir Kundrak wrote:
> I wrote some GDB macros that might be helpful for people
> that want to debug GRUB 2 modules with QEMU. [1]
> 
> [1] http://NetBSD.sk/~lkundrak/misc/grub2-gdb/
> 
> The file genmk.rb.diff is a patch to makefile-generator
> script, that makes the build system leave output files with
> debugging information. Of course, one has have ruby interpreter
> to regenerate the makefiles and compile GRUB with -g
> afterwards.
> 
> Second file is .gdbinit, the GDB initialization file.  It is
> commented, so there is no need to describe it here. It basically
> passes the information extracted from list headed by grub_dl_head to
> the perl script, gmodule.pl, and loads its output. It also
> sets a breakpoint whose command list contains macro for loading
> proper symbol file each time a module is loaded. (I know the work
> could be done without the help of a Perl script, but it would
> be far more complicated, I guess)
> 
> Another note worth mentioning is that you'll probably want
> to add aliases for some functions, so that all gdb facilities
> will work correctly. You will at least want to define the symbol
> 'main' for backtraces to stop at the right place. Some facilities
> also want the function 'malloc' to be defined. So, you'll
> probably have to add something like
> 
> .globl main
> main = codestart
> 
> to assembly language sources or
> 
> malloc() __attribute__ ((alias("grub_malloc")));
> 
> to C files.
> 
> I hope this will be useful to at least some of you. Best regards!
> Lubo.
> 

RE : Debugging GRUB2 with GDB and QEMU

[Eric Salomé]

Hi Lubomir,

I've tested your scripts.
They are great.
So I changed my code in kern/dl.c to reflect your .elf convention and I
have all ready to use 

With -DGRUB_COMPAT_GDB
You need to recompile gdb with a modified solib-svr4.c file that allows
an  ELF file to be loaded at different non-contigus memory locations,
one for each section.

All is automatic using ld.so interface to gdb and you can set pending
breakpoints waiting for the "shared lib" (the grub module) to be loaded.

(ld.so doesn't complain anymore when the program quits, I've added a
grub_unload_all() before exiting grub-emu for modules loaded with the
"insmod" command)

Without -DGRUB_COMPAT_GDB
Every thing is setup so that before calling grub_dl_mod_init() there is
a call to grub_gdb_load_mod(mod) and mod->elf contains the string 
Filename.elf -s .text 0x233 -s .rodata 0x236 
Ready to be feed to a .gdbinit script that generates the add-symbol-file
command to be swallowed by any standard gdb.

I've noticed that the attachment files I've sent with my previous emails
were not received on Grub 2 mailing list, so that I can send you (and to
others) my source code if you wish to your personal email address.

Source code is ok, should work on both 32-bits and 64-bits
architectures, though I don't do too many checks to prevent buffer
overflow in grub-emu.

I hope you might find these source extensions to kern/dl.c useful.
It's a great pleasure to be able to dynamically load modules and do the
debugging task at ease.

Thanks again Lubomir. Your scripts have been a great help to finalize
this GRUB / GDB interface.

PS: the following lines in genmk.rb didn't work too well on my pc
(Ubuntu) :
+#{exec}: #{pre_obj} #{mod_obj}
 	-rm -f $@
 	$(CC) $(#{prefix}_LDFLAGS) $(LDFLAGS) -Wl,-r,-d -o $@ $^

ld (when called by gcc) complains it can't find -lgcc_s, so I changed
back to 
	$(LD) $(#{prefix}_LDFLAGS) $(LDFLAGS) -r -d -o $@ $^

Best Regards,
_______________________________________
Eric Salomé - Paris, France


> -----Message d'origine-----
> De : grub-devel-bounces+esalome=ctx.net@gnu.org [mailto:grub-devel-
> bounces+esalome=ctx.net@gnu.org] De la part de Lubomir Kundrak
> Envoyé : vendredi 12 mai 2006 15:20
> À : grub-devel@gnu.org
> Objet : Debugging GRUB2 with GDB and QEMU
> 
> I wrote some GDB macros that might be helpful for people
> that want to debug GRUB 2 modules with QEMU. [1]
> 
> [1] http://NetBSD.sk/~lkundrak/misc/grub2-gdb/
> 
> The file genmk.rb.diff is a patch to makefile-generator
> script, that makes the build system leave output files with
> debugging information. Of course, one has have ruby interpreter
> to regenerate the makefiles and compile GRUB with -g
> afterwards.
> 
> Second file is .gdbinit, the GDB initialization file.  It is
> commented, so there is no need to describe it here. It basically
> passes the information extracted from list headed by grub_dl_head to
> the perl script, gmodule.pl, and loads its output. It also
> sets a breakpoint whose command list contains macro for loading
> proper symbol file each time a module is loaded. (I know the work
> could be done without the help of a Perl script, but it would
> be far more complicated, I guess)
> 
> Another note worth mentioning is that you'll probably want
> to add aliases for some functions, so that all gdb facilities
> will work correctly. You will at least want to define the symbol
> 'main' for backtraces to stop at the right place. Some facilities
> also want the function 'malloc' to be defined. So, you'll
> probably have to add something like
> 
> .globl main
> main = codestart
> 
> to assembly language sources or
> 
> malloc() __attribute__ ((alias("grub_malloc")));
> 
> to C files.
> 
> I hope this will be useful to at least some of you. Best regards!
> Lubo.
> 
> --
>   o	Lubomir Kundrak
>  *O*	<lkundrak@skosi.org>, http://skosi.org/~lkundrak/
>  (_)	SKOSI -- Slovak OpenSource Initiative
> 
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel

Re: Debugging GRUB2 with GDB and QEMU

[Lubomir Kundrak]

I noticed, that when any segment is 0 bytes long and starts
at 0x0, gdb desn't resolve symbols correctly.
I have motified the gmodule.pl to reflect this.
Please check an updated version.

regards,
lubo

On Sat, May 13, 2006 at 02:45:37PM -0400, devShadow wrote:
> X-Original-To: lkundrak@skosi.org
> Delivered-To: lkundrak@morph.sk
> Date: Sat, 13 May 2006 14:45:37 -0400
> From: devShadow <devShadow@xacks.com>
> User-Agent: Mozilla Thunderbird 1.0.7 (Windows/20050923)
> X-Accept-Language: en-us, en
> To: The development of GRUB 2 <grub-devel@gnu.org>
> In-Reply-To: <20060512132007.GA88582@meta.morph.sk>
> X-Virus-Scanned: Symantec AntiVirus Scan Engine
> Subject: Re: Debugging GRUB2 with GDB and QEMU
> X-BeenThere: grub-devel@gnu.org
> X-Mailman-Version: 2.1.5
> Precedence: list
> Reply-To: The development of GRUB 2 <grub-devel@gnu.org>
> List-Id: The development of GRUB 2 <grub-devel.gnu.org>
> List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/grub-devel>,
> 	<mailto:grub-devel-request@gnu.org?subject=unsubscribe>
> List-Archive: <http://lists.gnu.org/pipermail/grub-devel>
> List-Post: <mailto:grub-devel@gnu.org>
> List-Help: <mailto:grub-devel-request@gnu.org?subject=help>
> List-Subscribe: <http://lists.gnu.org/mailman/listinfo/grub-devel>,
> 	<mailto:grub-devel-request@gnu.org?subject=subscribe>
> Errors-To: grub-devel-bounces+lkundrak=skosi.org@gnu.org
> 
> Hi, thanks for the help with debugging modules.  I have a module that I 
> created and still need a little assistance in getting gdb to break in my 
>  module's grub_cmd_*.  I am able to connect to QEMU with gdb and when I 
> load the module, gdb loads the symbol table for my module.
> 
> But, when I set a breakpoint at the address of the first operation in my 
> module, gdb doesn't break when I execute the command in GRUB.  If it has 
> something to do with the aliases you mention at the bottom of your 
> message, could you elaborate a little more or direct me in the right 
> direction?
> 
> Thanks
> 
> David Johnson
> 
> Lubomir Kundrak wrote:
> >I wrote some GDB macros that might be helpful for people
> >that want to debug GRUB 2 modules with QEMU. [1]
> >
> >[1] http://NetBSD.sk/~lkundrak/misc/grub2-gdb/
> >
> >The file genmk.rb.diff is a patch to makefile-generator
> >script, that makes the build system leave output files with
> >debugging information. Of course, one has have ruby interpreter
> >to regenerate the makefiles and compile GRUB with -g
> >afterwards.
> >
> >Second file is .gdbinit, the GDB initialization file.  It is
> >commented, so there is no need to describe it here. It basically
> >passes the information extracted from list headed by grub_dl_head to
> >the perl script, gmodule.pl, and loads its output. It also
> >sets a breakpoint whose command list contains macro for loading
> >proper symbol file each time a module is loaded. (I know the work
> >could be done without the help of a Perl script, but it would
> >be far more complicated, I guess)
> >
> >Another note worth mentioning is that you'll probably want
> >to add aliases for some functions, so that all gdb facilities
> >will work correctly. You will at least want to define the symbol
> >'main' for backtraces to stop at the right place. Some facilities
> >also want the function 'malloc' to be defined. So, you'll
> >probably have to add something like
> >
> >.globl main
> >main = codestart
> >
> >to assembly language sources or
> >
> >malloc() __attribute__ ((alias("grub_malloc")));
> >
> >to C files.
> >
> >I hope this will be useful to at least some of you. Best regards!
> >Lubo.
> >
> 
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel

-- 
  o	Lubomir Kundrak
 *O*	<lkundrak@skosi.org>, http://skosi.org/~lkundrak/
 (_)	SKOSI -- Slovak OpenSource Initiative

Re: Debugging GRUB2 with GDB and QEMU

[David Johnson]

Thanks, I now have gdb running as I expect in my module.

David

Lubomir Kundrak wrote:
> I noticed, that when any segment is 0 bytes long and starts
> at 0x0, gdb desn't resolve symbols correctly.
> I have motified the gmodule.pl to reflect this.
> Please check an updated version.
> 
> regards,
> lubo
> 

Re: Debugging GRUB2 with GDB and QEMU

[Lubomir Kundrak]

On Sat, May 13, 2006 at 06:38:26AM +0200, Yoshinori K. Okuji wrote:
> Message-Id: <200605130638.26578.okuji@enbug.org>
> 
> Great!
> 
> I guess it is necessary to compile the binary with -O0 or -O1 to get a 
> reasonable backtrace... I'd appreciate if you can write a HOWTO. :)

On i386 GCC generates full stack frames even with -O optizations.
It only omits frame pointer on architectures on which it doesn't
interfere with debugging.

I started a small HOWTO recently, so here is it: [1]. Enjoy :o)
[1] http://NetBSD.sk/~lkundrak/misc/grub2-gdb/howto.html

> 
> Okuji

Lubo

-- 
  o	Lubomir Kundrak
 *O*	<lkundrak@skosi.org>, http://skosi.org/~lkundrak/
 (_)	SKOSI -- Slovak OpenSource Initiative

Re: Debugging GRUB2 with GDB and QEMU

[Hollis Blanchard]

On Thu, 2006-05-18 at 14:27 +0200, Lubomir Kundrak wrote:
> On Sat, May 13, 2006 at 06:38:26AM +0200, Yoshinori K. Okuji wrote:
> 
> On i386 GCC generates full stack frames even with -O optizations.
> It only omits frame pointer on architectures on which it doesn't
> interfere with debugging.
> 
> I started a small HOWTO recently, so here is it: [1]. Enjoy :o)
> [1] http://NetBSD.sk/~lkundrak/misc/grub2-gdb/howto.html

This looks like very interesting stuff, but wouldn't it be better to put
this in the wiki? Please see http://grub.enbug.org.

-Hollis

Re: Debugging GRUB2 with GDB and QEMU

[Lubomir Kundrak]

On Thu, May 18, 2006 at 08:35:25AM -0500, Hollis Blanchard wrote:
> 
> On Thu, 2006-05-18 at 14:27 +0200, Lubomir Kundrak wrote:
> > On Sat, May 13, 2006 at 06:38:26AM +0200, Yoshinori K. Okuji wrote:
> > 
> > On i386 GCC generates full stack frames even with -O optizations.
> > It only omits frame pointer on architectures on which it doesn't
> > interfere with debugging.
> > 
> > I started a small HOWTO recently, so here is it: [1]. Enjoy :o)
> > [1] http://NetBSD.sk/~lkundrak/misc/grub2-gdb/howto.html
> 
> This looks like very interesting stuff, but wouldn't it be better to put
> this in the wiki? Please see http://grub.enbug.org.
> 
> -Hollis

Yeah, I thought about it, but I don't have Internet connection at home,
so I am unable to edit it in wiki when offline, so it's easier for me
to have it as a standalone file. If anyone wants to change anything,
I will accept a patch for that hapilly.

-- 
  o	Lubomir Kundrak
 *O*	<lkundrak@skosi.org>, http://skosi.org/~lkundrak/
 (_)	SKOSI -- Slovak OpenSource Initiative

Re: Debugging GRUB2 with GDB and QEMU

[Yoshinori K. Okuji]

On Thursday 18 May 2006 14:27, Lubomir Kundrak wrote:
> On Sat, May 13, 2006 at 06:38:26AM +0200, Yoshinori K. Okuji wrote:
> > Message-Id: <200605130638.26578.okuji@enbug.org>
> >
> > Great!
> >
> > I guess it is necessary to compile the binary with -O0 or -O1 to get a
> > reasonable backtrace... I'd appreciate if you can write a HOWTO. :)
>
> On i386 GCC generates full stack frames even with -O optizations.
> It only omits frame pointer on architectures on which it doesn't
> interfere with debugging.

The problem is not the frame pointers. With -Os, gcc inlines functions used 
only once. For example, grub_dl_add is eliminated in gcc 4.x.

Okuji