What is the recommended method for debugging applications with Yocto

What is the recommended method for debugging applications with Yocto

[Patrick Doyle]

I know I can do

$ bitbake base-image -cpopulate_sdk

and distribute the resulting SDK to developers (myself included).

But suppose I've just built and deployed an image and want to debug a
core file generated by one of the applications.

I don't see .debug directories anywhere in $TMP, so when I try to tell
gdb to use the sysroot from the application, that works, but I don't
get debugging symbols libc, nor do I get stack traces (on my MIPS
target).
(I guess it's possible that I would have found the .debug directories
if I had performed a complete build from scratch without the benefit
of the sstate-cache, but I rely upon the sstate-cache and routinely
start with a blank $TMP)

So, I can populate the sdk, and end up with a pile of .debug directories:

$ find . -name .debug -type d
./tmp/work/sundial-poky-linux-musl/base-image/1.0-r0/sdk/image/opt/irobot-mt/0.2/sysroots/mips32r2-24kec-nf-poky-linux-musl/usr/lib/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/bin/mipsel-poky-linux/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/plugin/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/bin/mipsel-poky-linux/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/.debug
./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/plugin/.debug
./tmp/work/mips32r2-24kec-nf-poky-linux-musl/musl-utils/20170421-r0/packages-split/musl-utils-dbg/usr/bin/.debug
./tmp/work/mips32r2-24kec-nf-poky-linux-musl/musl-utils/20170421-r0/package/usr/bin/.debug

Which sysroot directory is the best one at which I can point gdb for
getting debug symbols?

If I build this with an autobuilder and want to archive the sysroot
for use by gdb later, which one should I archive?

I could archive the generated SDK, but that will require that I (or
others) install the SDK just to debug a core file, which seems a
little heavy to me.  But I can do that, if that's "the Yocto way".

I wonder if there is a lighter weight mechanism for accessing the
debugging symbols during development, and for archiving said symbols
for post-development support.

Any ideas?  Best practices?  Recommended approaches?

Thanks

--wpd

Re: [yocto] What is the recommended method for debugging applications with Yocto

[Khem Raj]

On 5/15/20 8:25 AM, Patrick Doyle wrote:
> I know I can do
> 
> $ bitbake base-image -cpopulate_sdk
> 
> and distribute the resulting SDK to developers (myself included).
> 
> But suppose I've just built and deployed an image and want to debug a
> core file generated by one of the applications.
> 
> I don't see .debug directories anywhere in $TMP, so when I try to tell
> gdb to use the sysroot from the application, that works, but I don't
> get debugging symbols libc, nor do I get stack traces (on my MIPS
> target).
> (I guess it's possible that I would have found the .debug directories
> if I had performed a complete build from scratch without the benefit
> of the sstate-cache, but I rely upon the sstate-cache and routinely
> start with a blank $TMP)
> 
> So, I can populate the sdk, and end up with a pile of .debug directories:
> 
> $ find . -name .debug -type d
> ./tmp/work/sundial-poky-linux-musl/base-image/1.0-r0/sdk/image/opt/irobot-mt/0.2/sysroots/mips32r2-24kec-nf-poky-linux-musl/usr/lib/.debug
> ./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/bin/mipsel-poky-linux/.debug
> ./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/.debug
> ./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/plugin/.debug
> ./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/bin/mipsel-poky-linux/.debug
> ./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/.debug
> ./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/plugin/.debug
> ./tmp/work/mips32r2-24kec-nf-poky-linux-musl/musl-utils/20170421-r0/packages-split/musl-utils-dbg/usr/bin/.debug
> ./tmp/work/mips32r2-24kec-nf-poky-linux-musl/musl-utils/20170421-r0/package/usr/bin/.debug
> 
> Which sysroot directory is the best one at which I can point gdb for
> getting debug symbols?
> 
> If I build this with an autobuilder and want to archive the sysroot
> for use by gdb later, which one should I archive?
> 
> I could archive the generated SDK, but that will require that I (or
> others) install the SDK just to debug a core file, which seems a
> little heavy to me.  But I can do that, if that's "the Yocto way".
> 
> I wonder if there is a lighter weight mechanism for accessing the
> debugging symbols during development, and for archiving said symbols
> for post-development support.
> 
> Any ideas?  Best practices?  Recommended approaches?
> 

I would think of using IMAGE_GEN_DEBUGFS set to 1 which should generate 
a debug archive of image

see following for detailed steps

https://docs.windriver.com/bundle/Wind_River_Linux_Platform_Developers_Guide_9_1/page/fvn1520619692704.html

https://developer.ridgerun.com/wiki/index.php?title=Preparing_Yocto_Development_Environment_for_Debugging

> Thanks
> 
> --wpd
> 
> 
> 
> 

Re: [yocto] What is the recommended method for debugging applications with Yocto

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 3448 bytes --]

The three debugging sections in the yocto manual should help?

https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#platdev-gdb-remotedebug

Alex

On Fri, 15 May 2020 at 17:26, Patrick Doyle <wpdster@gmail.com> wrote:

> I know I can do
>
> $ bitbake base-image -cpopulate_sdk
>
> and distribute the resulting SDK to developers (myself included).
>
> But suppose I've just built and deployed an image and want to debug a
> core file generated by one of the applications.
>
> I don't see .debug directories anywhere in $TMP, so when I try to tell
> gdb to use the sysroot from the application, that works, but I don't
> get debugging symbols libc, nor do I get stack traces (on my MIPS
> target).
> (I guess it's possible that I would have found the .debug directories
> if I had performed a complete build from scratch without the benefit
> of the sstate-cache, but I rely upon the sstate-cache and routinely
> start with a blank $TMP)
>
> So, I can populate the sdk, and end up with a pile of .debug directories:
>
> $ find . -name .debug -type d
>
> ./tmp/work/sundial-poky-linux-musl/base-image/1.0-r0/sdk/image/opt/irobot-mt/0.2/sysroots/mips32r2-24kec-nf-poky-linux-musl/usr/lib/.debug
>
> ./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/bin/mipsel-poky-linux/.debug
>
> ./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/.debug
>
> ./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/plugin/.debug
>
> ./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/bin/mipsel-poky-linux/.debug
>
> ./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/.debug
>
> ./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/plugin/.debug
>
> ./tmp/work/mips32r2-24kec-nf-poky-linux-musl/musl-utils/20170421-r0/packages-split/musl-utils-dbg/usr/bin/.debug
>
> ./tmp/work/mips32r2-24kec-nf-poky-linux-musl/musl-utils/20170421-r0/package/usr/bin/.debug
>
> Which sysroot directory is the best one at which I can point gdb for
> getting debug symbols?
>
> If I build this with an autobuilder and want to archive the sysroot
> for use by gdb later, which one should I archive?
>
> I could archive the generated SDK, but that will require that I (or
> others) install the SDK just to debug a core file, which seems a
> little heavy to me.  But I can do that, if that's "the Yocto way".
>
> I wonder if there is a lighter weight mechanism for accessing the
> debugging symbols during development, and for archiving said symbols
> for post-development support.
>
> Any ideas?  Best practices?  Recommended approaches?
>
> Thanks
>
> --wpd
> 
>

[-- Attachment #2: Type: text/html, Size: 4006 bytes --]

Re: [yocto] What is the recommended method for debugging applications with Yocto

[Andreas M?ller]

On Fri, May 15, 2020 at 5:36 PM Khem Raj <raj.khem@gmail.com> wrote:
>
> On 5/15/20 8:25 AM, Patrick Doyle wrote:
> > I know I can do
> >
> > $ bitbake base-image -cpopulate_sdk
> >
> > and distribute the resulting SDK to developers (myself included).
> >
> > But suppose I've just built and deployed an image and want to debug a
> > core file generated by one of the applications.
> >
> > I don't see .debug directories anywhere in $TMP, so when I try to tell
> > gdb to use the sysroot from the application, that works, but I don't
> > get debugging symbols libc, nor do I get stack traces (on my MIPS
> > target).
> > (I guess it's possible that I would have found the .debug directories
> > if I had performed a complete build from scratch without the benefit
> > of the sstate-cache, but I rely upon the sstate-cache and routinely
> > start with a blank $TMP)
> >
> > So, I can populate the sdk, and end up with a pile of .debug directories:
> >
> > $ find . -name .debug -type d
> > ./tmp/work/sundial-poky-linux-musl/base-image/1.0-r0/sdk/image/opt/irobot-mt/0.2/sysroots/mips32r2-24kec-nf-poky-linux-musl/usr/lib/.debug
> > ./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/bin/mipsel-poky-linux/.debug
> > ./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/.debug
> > ./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/packages-split/gcc-cross-canadian-mipsel-dbg/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/plugin/.debug
> > ./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/bin/mipsel-poky-linux/.debug
> > ./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/.debug
> > ./tmp/work/x86_64-nativesdk-pokysdk-linux/gcc-cross-canadian-mipsel/8.3.0-r0/package/opt/irobot-mt/0.2/sysroots/x86_64-pokysdk-linux/usr/libexec/mipsel-poky-linux/gcc/mipsel-poky-linux/8.3.0/plugin/.debug
> > ./tmp/work/mips32r2-24kec-nf-poky-linux-musl/musl-utils/20170421-r0/packages-split/musl-utils-dbg/usr/bin/.debug
> > ./tmp/work/mips32r2-24kec-nf-poky-linux-musl/musl-utils/20170421-r0/package/usr/bin/.debug
> >
> > Which sysroot directory is the best one at which I can point gdb for
> > getting debug symbols?
> >
> > If I build this with an autobuilder and want to archive the sysroot
> > for use by gdb later, which one should I archive?
> >
> > I could archive the generated SDK, but that will require that I (or
> > others) install the SDK just to debug a core file, which seems a
> > little heavy to me.  But I can do that, if that's "the Yocto way".
> >
> > I wonder if there is a lighter weight mechanism for accessing the
> > debugging symbols during development, and for archiving said symbols
> > for post-development support.
> >
> > Any ideas?  Best practices?  Recommended approaches?
> >
>
> I would think of using IMAGE_GEN_DEBUGFS set to 1 which should generate
> a debug archive of image
>
> see following for detailed steps
>
> https://docs.windriver.com/bundle/Wind_River_Linux_Platform_Developers_Guide_9_1/page/fvn1520619692704.html
>
> https://developer.ridgerun.com/wiki/index.php?title=Preparing_Yocto_Development_Environment_for_Debugging
>
> > Thanks
> >
> > --wpd
> >
I created [1] in meta-mortsgna. It is not recommended (bit of a hack /
nobody but me and my colleagues use it) but it

* works on the fly: Once enabled building the recipe you want to debug
is good enough
* not bound to SDK or image which tends to last
* no manual copying necessary

[1] https://github.com/schnitzeltony/meta-mortsgna/blob/master/classes/instant-sysroot-target.bbclass

Andreas

Re: [yocto] What is the recommended method for debugging applications with Yocto

[Patrick Doyle]

Thanks folks, IMAGE_GEN_DEBUGFS was exactly what I was looking for.

--wpd