Debugging gdb built by Yocto

Debugging gdb built by Yocto

[Patrick Doyle]

Does anybody have any tips or tricks for how I might debug the
(cross-canadian) gdb built by Yocto's SDK?

I need to add some printf's to the gdb code to help track down why
something isn't working, but none of my traditional
get-ready-to-debug-this-code techniques are working.

How can I run the gdb that I just built?  Note that I am presuming that I can

$ bitbake gdb-cross-canadian-mipsel -ccompile -f

to rebuild gdb after I add a printf or two to it... but I can't figure
out how to run gdb without going through the sdk installation step.

$ bitbake gdb-cross-canadian-mipsel -cdevshell
# ../build-mipsel-poky-linux/gdb/gdb
bash: ../build-mipsel-poky-linux/gdb/gdb: No such file or directory
# file ../build-mipsel-poky-linux/gdb/gdb
../build-mipsel-poky-linux/gdb/gdb: ELF 64-bit LSB shared object,
x86-64, version 1 (GNU/Linux), dynamically linked, interpreter
/opt/iro, BuildID[sha1]=7f985bbe4cb6c97558b159860b2498f6389b254e, for
GNU/Linux 3.2.0, not stripped
# ldd ../build-mipsel-poky-linux/gdb/gdb
../build-mipsel-poky-linux/gdb/gdb: /lib/x86_64-linux-gnu/libm.so.6:
version `GLIBC_2.29' not found (required by
../build-mipsel-poky-linux/gdb/gdb)
    linux-vdso.so.1 =>  (0x00007fff8a0c2000)
 ...

# LD_LIBRARY_PATH=../recipe-sysroot-native/usr/libexec:../recipe-sysroot-native/usr/lib
../build-mipsel-poky-linux/gdb/gdb
bash: ../build-mipsel-poky-linux/gdb/gdb: No such file or directory

None of the techniques from my bag-of-tricks works.

I guess I could go grab the source myself, manually apply the patches
myself, build it, and see if that works.

Or I could sit down real hard and think about why I am trying to debug
the canadian-cross built tool on my development host... perhaps I
should try debugging the native (cross)-gdb on my native host.  I'll
go try that now, but, in the mean time, I thought it was about time
for me to ask others for some clues.

Any clues or pointers?

--wpd

Re: [yocto] Debugging gdb built by Yocto

[Khem Raj]

On 2/18/20 8:26 AM, Patrick Doyle wrote:
> Does anybody have any tips or tricks for how I might debug the
> (cross-canadian) gdb built by Yocto's SDK?
> 
> I need to add some printf's to the gdb code to help track down why
> something isn't working, but none of my traditional
> get-ready-to-debug-this-code techniques are working.
> 
> How can I run the gdb that I just built?  Note that I am presuming that I can
> 
> $ bitbake gdb-cross-canadian-mipsel -ccompile -f
> 
> to rebuild gdb after I add a printf or two to it... but I can't figure
> out how to run gdb without going through the sdk installation step.
> 
> $ bitbake gdb-cross-canadian-mipsel -cdevshell
> # ../build-mipsel-poky-linux/gdb/gdb
> bash: ../build-mipsel-poky-linux/gdb/gdb: No such file or directory
> # file ../build-mipsel-poky-linux/gdb/gdb
> ../build-mipsel-poky-linux/gdb/gdb: ELF 64-bit LSB shared object,
> x86-64, version 1 (GNU/Linux), dynamically linked, interpreter
> /opt/iro, BuildID[sha1]=7f985bbe4cb6c97558b159860b2498f6389b254e, for
> GNU/Linux 3.2.0, not stripped
> # ldd ../build-mipsel-poky-linux/gdb/gdb
> ../build-mipsel-poky-linux/gdb/gdb: /lib/x86_64-linux-gnu/libm.so.6:
> version `GLIBC_2.29' not found (required by
> ../build-mipsel-poky-linux/gdb/gdb)
>      linux-vdso.so.1 =>  (0x00007fff8a0c2000)
>   ...
> 
> # LD_LIBRARY_PATH=../recipe-sysroot-native/usr/libexec:../recipe-sysroot-native/usr/lib
> ../build-mipsel-poky-linux/gdb/gdb
> bash: ../build-mipsel-poky-linux/gdb/gdb: No such file or directory
> 
> None of the techniques from my bag-of-tricks works.
> 
> I guess I could go grab the source myself, manually apply the patches
> myself, build it, and see if that works.
> 
> Or I could sit down real hard and think about why I am trying to debug
> the canadian-cross built tool on my development host... perhaps I
> should try debugging the native (cross)-gdb on my native host.  I'll
> go try that now, but, in the mean time, I thought it was about time
> for me to ask others for some clues.
> 
> Any clues or pointers?

its perhaps due to the fact that host where this will run is sdkhost and 
not your normal buildhost, so perhaps you can use uninative tarball 
provided glibc to run it if your sdkhost is similar to buildhost and 
that might work.

> 
> --wpd
> 
> 
> 
> 

Re: [yocto] Debugging gdb built by Yocto

[Richard Purdie]

On Tue, 2020-02-18 at 11:26 -0500, Patrick Doyle wrote:
> Does anybody have any tips or tricks for how I might debug the
> (cross-canadian) gdb built by Yocto's SDK?
> 
> I need to add some printf's to the gdb code to help track down why
> something isn't working, but none of my traditional
> get-ready-to-debug-this-code techniques are working.
> 
> How can I run the gdb that I just built?  Note that I am presuming
> that I can
> 
> $ bitbake gdb-cross-canadian-mipsel -ccompile -f

Do you perhaps want gdb-cross-mipsel ?

cross-canadian is designed to be run as part of the SDK.

Cheers,

Richard

Re: [yocto] Debugging gdb built by Yocto

[Patrick Doyle]

On Wed, Feb 19, 2020 at 5:19 PM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Tue, 2020-02-18 at 11:26 -0500, Patrick Doyle wrote:
> > Does anybody have any tips or tricks for how I might debug the
> > (cross-canadian) gdb built by Yocto's SDK?
> Do you perhaps want gdb-cross-mipsel ?
>
> cross-canadian is designed to be run as part of the SDK.
Thank you.  That is, indeed, what I want.  It demonstrates the same
problem, and I can debug it.

Aside... it's quite a mind bending experience to debug gdb with gdb :-)

Just imagine...

$ gdb gdb-cross-mipsel
(gdb) break main
(gdb) run
(gdb)

That last "(gdb)" prompt is from the gdb I'm debugging :-)

Also aside... and totally off topic, except that I'm sure some might
be wondering why I feel the need to debug gdb...

I am trying to understand why I can't get stack traces from cores file
on a mipsel system.  At the moment (after strategic additions of
breakpoints, printf statements, and more breakpoints, and lots of
internet combing), I am chasing down a rabbit hole related to the
facts that the MIPS build uses/produces PIE code (Position Independent
Executables, which is somehow different than Position Independent
Code), a new ELF tag (MIPS_RLD_MAP_REL) was added 5 years ago to
binutils, gdb looks for that tag, but the musl dynamic loader is not
aware of it.  I don't know if this is the root cause of my problem or
just (another) rabbit hole.  If anybody has any suggestions, I'm all
ears.

--wpd

Re: [yocto] Debugging gdb built by Yocto

[Khem Raj]

On Thu, 2020-02-20 at 08:52 -0500, Patrick Doyle wrote:
> On Wed, Feb 19, 2020 at 5:19 PM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > On Tue, 2020-02-18 at 11:26 -0500, Patrick Doyle wrote:
> > > Does anybody have any tips or tricks for how I might debug the
> > > (cross-canadian) gdb built by Yocto's SDK?
> > Do you perhaps want gdb-cross-mipsel ?
> > 
> > cross-canadian is designed to be run as part of the SDK.
> Thank you.  That is, indeed, what I want.  It demonstrates the same
> problem, and I can debug it.
> 
> Aside... it's quite a mind bending experience to debug gdb with gdb
> :-)
> 
> Just imagine...
> 
> $ gdb gdb-cross-mipsel
> (gdb) break main
> (gdb) run
> (gdb)
> 
> That last "(gdb)" prompt is from the gdb I'm debugging :-)
> 
> Also aside... and totally off topic, except that I'm sure some might
> be wondering why I feel the need to debug gdb...
> 
> I am trying to understand why I can't get stack traces from cores
> file
> on a mipsel system.  At the moment (after strategic additions of
> breakpoints, printf statements, and more breakpoints, and lots of
> internet combing), I am chasing down a rabbit hole related to the
> facts that the MIPS build uses/produces PIE code (Position
> Independent
> Executables, which is somehow different than Position Independent
> Code), a new ELF tag (MIPS_RLD_MAP_REL) was added 5 years ago to
> binutils, gdb looks for that tag, but the musl dynamic loader is not
> aware of it.  I don't know if this is the root cause of my problem or
> just (another) rabbit hole.  If anybody has any suggestions, I'm all
> ears.

You must be on to something here, since musl does not do anything with
MIPS_RLD_MAP*, does this all work well with glibc/mips in same settings
?

> 
> --wpd
> 

Re: [yocto] Debugging gdb built by Yocto

[Patrick Doyle]

[-- Attachment #1: Type: text/plain, Size: 1604 bytes --]

On Thu, Feb 20, 2020 at 4:18 PM Khem Raj <raj.khem@gmail.com> wrote:
> On Thu, 2020-02-20 at 08:52 -0500, Patrick Doyle wrote:
> > I am trying to understand why I can't get stack traces from cores
> > file
> > on a mipsel system.  At the moment (after strategic additions of
> > breakpoints, printf statements, and more breakpoints, and lots of
> > internet combing), I am chasing down a rabbit hole related to the
> > facts that the MIPS build uses/produces PIE code (Position
> > Independent
> > Executables, which is somehow different than Position Independent
> > Code), a new ELF tag (MIPS_RLD_MAP_REL) was added 5 years ago to
> > binutils, gdb looks for that tag, but the musl dynamic loader is not
> > aware of it.  I don't know if this is the root cause of my problem or
> > just (another) rabbit hole.  If anybody has any suggestions, I'm all
> > ears.
>
> You must be on to something here, since musl does not do anything with
> MIPS_RLD_MAP*, does this all work well with glibc/mips in same settings
> ?
I haven't tried this with glibc/mips with the same settings, but I
expect it would work in that scenario, as I see support for
MIPS_RLD_MAP_REL in the glibc sources.

If anybody is interested, I have attached the two patches we applied
to get core dumps to work.  I am planning to clean them up a bit and
submit them upstream and/or to Yocto/OE's repo, but, in case I don't
(or I forget), and since you expressed an interest in this, I am
including them here.

I'm not sure if my email will make it to the list with attachments or
not.  We'll see.

--wpd

[-- Attachment #2: 0007-Teach-dynlink.c-about-DT_MIPS_RLD_MAP_REL.patch --]
[-- Type: text/x-patch, Size: 1787 bytes --]

From 26172a25aec6434f78a1ac21d5587dc8192e6ba6 Mon Sep 17 00:00:00 2001
From: Patrick Doyle <pdoyle@irobot.com>
Date: Thu, 20 Feb 2020 17:03:30 -0500
Subject: [PATCH] Teach dynlink.c about DT_MIPS_RLD_MAP_REL

Record the address of the debug structure in a location indicated by
DT_MIPS_RLD_MAP_REL so that debuggers can properly debug core files with
dynamic libraries.

See https://binutils.sourceware.narkive.com/aTb1ofXN/patch-mips-support-shared-library-debug-with-mips-pie
for the definition of DT_MIPS_RLD_MAP_REL.
---
 ldso/dynlink.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/ldso/dynlink.c b/ldso/dynlink.c
index 6325d8eb..842e7b34 100644
--- a/ldso/dynlink.c
+++ b/ldso/dynlink.c
@@ -1866,6 +1866,23 @@ _Noreturn void __dls3(size_t *sp)
 			size_t *ptr = (size_t *) app.dynv[i+1];
 			*ptr = (size_t)&debug;
 		}
+		/* According to
+		 * https://binutils.sourceware.narkive.com/aTb1ofXN/patch-mips-support-shared-library-debug-with-mips-pie
+		 * The definition of DT_MIPS_RLD_MAP_REL is:
+		 *
+		 * This member is used by debugging. It contains a relative offset from the tag's runtime
+		 * location of a 32-bit word in the .data section which is supplied by the compilation
+		 * environment. The word's contents are not specified and programs using this value are not
+		 * ABI - compliant.
+		 *
+		 * We need to supply the address of the `debug` structure, relative to the address of the
+		 * DT_MIPS_RLD_MAP_REL tag, in order for gdb to successfully load the share libraries
+		 * referenced by a core file.
+		 */
+		if (app.dynv[i]==DT_MIPS_RLD_MAP_REL) {
+			void **ptr = (void **)((char *)&app.dynv[i] + (unsigned)app.dynv[i+1]);
+			*ptr = &debug;
+		}
 	}
 
 	/* This must be done before final relocations, since it calls
-- 
2.24.1


[-- Attachment #3: 0006-Add-some-CFI-directives-to-MIPS-asm-sources.patch --]
[-- Type: text/x-patch, Size: 2595 bytes --]

From 84cfdd63cac7917689cb30805757070b50b9bf19 Mon Sep 17 00:00:00 2001
From: Daniele Tamino <dtamino@irobot.com>
Date: Wed, 29 Jan 2020 18:43:22 -0800
Subject: [PATCH] Add some CFI directives to MIPS asm sources

---
 src/internal/mips/syscall.s  | 4 ++++
 src/ldso/mips/dlsym.s        | 6 ++++++
 src/thread/mips/syscall_cp.s | 9 +++++++++
 3 files changed, 19 insertions(+)

diff --git a/src/internal/mips/syscall.s b/src/internal/mips/syscall.s
index 5d0def52..f9bc599d 100644
--- a/src/internal/mips/syscall.s
+++ b/src/internal/mips/syscall.s
@@ -4,6 +4,7 @@
 .hidden __syscall
 .type   __syscall,@function
 __syscall:
+    .cfi_startproc
 	move    $2, $4
 	move    $4, $5
 	move    $5, $6
@@ -13,6 +14,7 @@ __syscall:
 	lw      $9, 24($sp)
 	lw      $10,28($sp)
 	subu    $sp, $sp, 32
+	.cfi_adjust_cfa_offset 32
 	sw      $8, 16($sp)
 	sw      $9, 20($sp)
 	sw      $10,24($sp)
@@ -21,6 +23,8 @@ __syscall:
 	syscall
 	beq     $7, $0, 1f
 	addu    $sp, $sp, 32
+    .cfi_adjust_cfa_offset -32
 	subu    $2, $0, $2
 1:	jr      $ra
 	nop
+	.cfi_endproc
diff --git a/src/ldso/mips/dlsym.s b/src/ldso/mips/dlsym.s
index 1573e519..f1036621 100644
--- a/src/ldso/mips/dlsym.s
+++ b/src/ldso/mips/dlsym.s
@@ -3,15 +3,21 @@
 .hidden __dlsym
 .type dlsym,@function
 dlsym:
+    .cfi_startproc
 	lui $gp, %hi(_gp_disp)
 	addiu $gp, %lo(_gp_disp)
 	addu $gp, $gp, $25
 	move $6, $ra
 	lw $25, %call16(__dlsym)($gp)
 	addiu $sp, $sp, -16
+	.cfi_adjust_cfa_offset 16
 	sw $ra, 12($sp)
+    .cfi_rel_offset $ra, 12
 	jalr $25
 	nop
 	lw $ra, 12($sp)
+    .cfi_restore $ra
 	jr $ra
 	addiu $sp, $sp, 16
+	.cfi_adjust_cfa_offset -16
+	.cfi_endproc
diff --git a/src/thread/mips/syscall_cp.s b/src/thread/mips/syscall_cp.s
index d2846264..ab173496 100644
--- a/src/thread/mips/syscall_cp.s
+++ b/src/thread/mips/syscall_cp.s
@@ -14,9 +14,12 @@
 .hidden __syscall_cp_asm
 .type   __syscall_cp_asm,@function
 __syscall_cp_asm:
+    .cfi_startproc
 	subu    $sp, $sp, 32
+	.cfi_adjust_cfa_offset 32
 __cp_begin:
 	lw      $4, 0($4)
+	.cfi_remember_state
 	bne     $4, $0, __cp_cancel
 	move    $2, $5
 	move    $4, $6
@@ -35,14 +38,18 @@ __cp_begin:
 __cp_end:
 	beq     $7, $0, 1f
 	addu    $sp, $sp, 32
+	.cfi_adjust_cfa_offset -32
 	subu    $2, $0, $2
 1:	jr      $ra
 	nop
 
 __cp_cancel:
+    .cfi_restore_state
 	move    $2, $ra
+	.cfi_register $ra, $2
 	bal     1f
 	addu    $sp, $sp, 32
+	.cfi_adjust_cfa_offset -32
 	.gpword .
 	.gpword __cancel
 1:	lw      $3, ($ra)
@@ -51,3 +58,5 @@ __cp_cancel:
 	addu    $25, $25, $3
 	jr      $25
 	move    $ra, $2
+	.cfi_restore $ra
+	.cfi_endproc
-- 
2.24.0